The parallel-build failure gate only inspected $exit_code in run_on_finish. A
ForkManager child killed by a signal -- SIGKILL, or the OOM-killer under the
concurrent build load -- is reaped with $exit_code == 0 but $exit_signal != 0
(and possibly $core_dump). Such a worker therefore was NOT recorded as a
failure, so the parent could still index and GPG-sign a repository that is
missing the package that worker was building -- the exact partial-repo hazard
the gate was added to prevent, via a path it did not cover.
Capture $exit_signal and $core_dump from the run_on_finish callback and fail
the build when any of $exit_code, $exit_signal, or $core_dump is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses three concurrency-safety defects raised in review of the parallel
buildrpms.pl work:
1. Per-target build lock was released immediately. The flock filehandle was a
lexical (my $blk) scoped to the guard block, so it was destroyed -- and the
lock dropped -- as soon as that block exited, before any worker forked. The
"intentionally leaked" comment did not match the code. Hold the handle in a
file-scoped $BUILD_LOCK_FH so the fd (and the lock) live for the whole
process; forked children inherit the fd but their exits never release it.
2. Build failures were silently swallowed. buildspkgs()/buildpkgs() called
sh_retry() in void context, so a mock build that failed all retries returned
non-zero into the void; the child then exited 0 and the parent's run_on_finish
ignored the exit code. The parent could therefore index and GPG-sign a repo
that was missing packages and still exit 0. Now sh_retry failures die in the
child, run_on_finish records any non-zero child, and the run aborts before
update_repo and again before signing if any child failed -- never publishing
a partial core.
3. Abort cleanup unmounted unrelated builds. sweep_mock_mounts() lazy-unmounted
every bind under /var/lib/mock, which on a shared host tears out the live
chroots of concurrent, unrelated builds. Scope it to this run's own chroots
(each chroot dir plus its -bootstrap sibling), passed in from abort_builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
merge_core_repos already assembles the multi-arch core and runs the full
finalize tail (index -> sign -> final metadata -> write_release_alias, in the
required order so the xCAT-release-latest alias lands AFTER metadata and stays
out of the repo index). The preceding commit restored a separate finalize_core
sub purely to satisfy xcat_release_package.t, which grepped for
'sub finalize_core { ... $dir ... }'. That left two overlapping entry points:
--finalize-core (finalize one pre-assembled dir) and --merge-core-repos
(assemble N per-arch dirs THEN finalize) -- the former a strict subset of the
latter, with no in-tree or CI caller.
Consolidate on the single assemble+finalize path: inline the finalize tail into
merge_core_repos, drop sub finalize_core and the --finalize-core getopt/dispatch,
and update xcat_release_package.t to assert the stable alias is created after the
final metadata pass inside merge_core_repos ($out). Behaviour is unchanged (merge
already wrote the alias); only the redundant finalize_core interface is removed.
xcat_release_package.t 26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The CI buildrpms.pl port (d2d98b724) renamed the upstream sub finalize_core($dir)
to merge_core_repos($out) and inlined its index/sign/metadata/alias tail. The
xCAT-release logic (write the xCAT-release-latest stable bootstrap alias AFTER the
final metadata pass, so it stays out of the repo index) was preserved, but the rename
broke xCAT-test/unit/xcat_release_package.t, which statically pins:
sub finalize_core { ... write_repo_metadata_dir($dir); ... write_release_alias($dir); }
so 'assembled core repository creates the stable alias after final metadata' failed
(1 of 26), turning xcat_pr_test red on PR #7701.
Restore finalize_core($dir) as the shared finalize primitive (index -> sign ->
final metadata -> stable alias) and have merge_core_repos delegate to it on the
assembled multi-arch dir. Re-expose the --finalize-core CLI (single already-populated
dir) it also feeds. Behaviour is unchanged for both --merge-core-repos (CI) and
--finalize-core; only the shared code path is named again. xcat_release_package.t now
26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl built every package with mock -N (--no-clean), reusing the per-package
<pkg>-<target><ext> chroot across runs for flat disk. But a build aborted or killed
mid-flight leaves that chroot half-initialised with a corrupt rpmdb; the NEXT run
reused it and failed (cannot open Packages database .../usr/lib/sysimage/rpm),
producing an incomplete core (e.g. missing xCAT-test) that fails the deploy-time
completeness gate.
Re-init the buildroot (mock --init) right before building each package, after the
diskcache skip so it only runs when actually building. --init restores from mock's
root-cache tarball (cheap) so disk stays flat and builds stay fast; -N is kept on the
srpm/binary calls so they still reuse the freshly initialised root within the run.
This stabilises builds against corrupt state left by any previous failed/aborted build.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 3 EL CD pipelines (xcat-core-devel-cd, xcat-core-stable-cd, xcat-dep-el-cd)
overlaid a pinned $CI/buildrpms.pl at build time because the tree's buildrpms.pl
lacked the options they depend on:
- a per-target flock guard alongside --mock-uniqueext, so concurrent same-target
builds do not corrupt each other's /var/lib/mock chroot namespace;
- --native-only (build only arch-native pkgs on the secondary arch) plus
--merge-core-repos/--output-dir/--input-core-repos, replacing --finalize-core,
to assemble one signed flat multi-arch core from per-arch build outputs;
- sh_retry() to absorb transient mock/nspawn flakes;
- a single --target guard and graceful mock cancellation
(sweep_mock_mounts/abort_builds) that unmounts chroots on abort.
The xCAT-release repository package (master/2.19 only) is preserved: its
write_release_alias() is invoked from the new merge_core_repos() and per-target.
Porting them in-tree lets CI drop the $CI/buildrpms.pl pin and run the three
pipelines in parallel without the cross-job serialize lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
write_release_alias() finds the release rpm with
glob("$repodir/xCAT-release-$VERSION-$RELEASE.noarch.rpm"). That pattern has
no wildcard, and glob() returns a wildcard-free pattern verbatim even when the
file does not exist -- so @release_rpms == 1 is true and cp runs against a
nonexistent file. A partial build that does not produce xCAT-release (e.g.
buildrpms.pl --package xCAT-genesis-base, the way the xcat-dep pipeline builds
the OS-dependent genesis image) then dies with:
Can't cp('dist/alma+epel-8-x86_64/rpms/xCAT-release-2.19.0-snap202607261133.noarch.rpm',
'dist/alma+epel-8-x86_64/rpms/xCAT-release-latest.noarch.rpm'):
No such file or directory at .../buildrpms.pl line 659
Guard the alias write on the rpm actually existing (grep { -f }), and die with
the real error if the cp itself fails. A build that does not produce
xCAT-release now simply skips the alias instead of failing.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcatprobe subcommands (xcatmn, discovery, osdeploy) and hierarchy.pm load
xCAT:: perl modules but only add $XCATROOT/probe/lib/perl to @INC, not the
main $XCATROOT/lib/perl where perl-xCAT ships them. The RPM builder stopped
embedding those helpers, so /opt/xcat/probe/lib/perl/xCAT/ was absent on
installed systems and the subcommands were reported "Unsupported sub command".
Stage GlobalDef.pm, NetworkUtils.pm and ServiceNodeUtils.pm into a temporary
xCAT-probe tree and build its source tarball once, before the parallel target
builds fork, so workers only read a complete archive. Add package-payload
regression tests. Adopts the approach from upstream PR #7581.
The remaining buildrpms.pl changes are deployment-compliance refactoring
(signed, self-contained core repo output).
Closes#7579
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl stages source tarballs into $HOME/rpmbuild/SOURCES but only runs
rpmdev-setuptree in its one-time env-setup path. On a host where that never ran
(or $HOME/rpmbuild was cleaned) source staging fails with "SOURCES/...: No such
file or directory", no srpms/rpms are produced, and the run still exits 0 -- so
the deployable repo silently ends up empty. Create the rpmbuild tree up front so
a build no longer depends on prior manual setup.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl now writes xcat-core.repo, mklocalrepo.sh and buildinfo.txt into
dist/<target>/rpms (templates ported from buildcore.sh), so the built tree is
directly deployable to xcat.org and cluster-test.pl no longer needs to collect
and re-createrepo the dist output.
The full package set is built on every arch (x86_64 and ppc64le alike), so each
arch produces a complete, self-contained xcat-core repo -- no need to copy the
noarch packages from the x86_64 build into the ppc repo. --package now replaces
the default set (so --package xCAT-genesis-base builds only genesis-base for the
dep pipeline). xCATsn added to \@native_pkgs so its arch rpm is located correctly.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The single flat xcat-core repo serves el8/el9/el10 from one build, but two spec
decisions were resolved at BUILD time and so were wrong for the other ELs that
share the same repo.
1. DHCP backend. xCAT.spec and xCATsn.spec selected the dhcp provider with a
build-time %if (rhel >= 10 -> kea, else dhcpd), so an el10 build wrongly
required kea on el8/el9 (and an el8/el9 build wrongly required dhcpd on
el10). Replace it with an install-time RPM rich dependency:
Requires: (kea if (system-release >= 10) else /usr/sbin/dhcpd)
Requires: (kea-hooks if (system-release >= 10))
dnf now resolves it per client: kea on el10+ (which dropped ISC dhcp from the
distro), dhcpd on el8/el9. SLES has no "system-release" provide, so the
condition is false there and it falls back to /usr/sbin/dhcpd, preserving
prior behavior. system-release is versioned per release package
(el10=10.x, el9=9.x, el8=8.x).
2. openssl. Make the xCAT-server openssl Requires uniform across EL (non-SUSE)
instead of el10-only, so the produced package set does not depend on which EL
the build host happened to be.
Also drop xCAT-genesis-base from the default @PACKAGES set in buildrpms.pl: its
initramfs bundles the build-chroot kernel/glibc and is therefore OS- and
arch-dependent, so it cannot ship in the single flat core. It is now built per
target by the xcat-dep pipeline (mockbuild-all.pl, via
`buildrpms.pl --package xCAT-genesis-base`) and ships in xcat-dep/rh<N>. The
explicit `--package xCAT-genesis-base` build path is retained.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
This fixes a problem where the dracut image was pulling artifacts from
the build host and not the mock image, e.g. building for EL9 in EL10.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
* Motivation: Before this commit we hardcoded "rhel" in the mock chroot
names, but these depend running over RHEL host. After this commit the
distribution name is read from /etc/os-release ID variable. So it
should work on any EL clone now.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
Also add --regennginxconfig to reset nginx config, and
ensure nginx configuration is generated for all targets
not only the ones provided in the command line. This avoid
the anoying effect of overriding the repositories of previous
builds.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
Add buildrpms.pl to build RPMs in parallel using mock
Add xCAT-buildkit to the build list
Fix build dependency in xCAT-buildkit.spec
Add fallback in /etc/init.d/xcatd for /etc/rc.d/init.d/functions
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão