build-ubunturepo guarded builds with a single host-global, fail-fast lock at
/var/lock/xcatbld.lock (`flock -n`). Two builds on one host -- e.g. the devel and
stable Ubuntu CD lanes on xcat-master-ub -- therefore collided on that one lock and
the loser exited 1 with "Can't get lock ...", failing the whole pipeline, even though
each lane builds from its own checkout into its own DEST and they share nothing.
build-ubunturepo builds its packages in-place in its own source checkout (it rewrites
debian/changelog and debian/control, drops *.orig.tar.gz at the checkout root and runs
dpkg-buildpackage inside the package dirs), so the resource two concurrent builds
actually contend for is the checkout, not the host. Key the lock on the checkout path
($curdir): builds of the SAME checkout still fail-fast (they would corrupt each other
in-place), while builds of DISTINCT checkouts get distinct locks and run in parallel.
The lock file stays on the local /var/lock (reliable flock, unlike the NFS/virtiofs
checkout) and the source tree is left byte-pristine.
Add xCAT-test/unit/build_ubunturepo_lock.t, which extracts the lock block from the
script verbatim and asserts: the lock path is /var/lock/xcatbld-<hash-of-checkout>.lock
and deterministic per checkout; a second build of the same checkout fails fast; two
distinct checkouts acquire their locks concurrently.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
makedhcp using the Kea backend fails for any node whose mac table entry
uses the *NOIP* sentinel to mark a secondary NIC that has no IP address
(e.g. "mac1|mac2!*NOIP*|mac3!*NOIP*"). kea_node_reservations() and
kea_node_reservations6() split each NIC's hostname and pass it straight
to getipaddr(), so the literal string "*NOIP*" is treated as a host to
resolve. It cannot resolve, and the reservation is reported as unresolved
-- which aborts the whole makedhcp run, leaving the node with no Kea
reservation at all and therefore no DHCP lease.
This breaks provisioning on distros that use the Kea backend (e.g. EL10,
which has no ISC dhcp-server): the node never gets an address and is
unreachable, while the ISC backend has always handled *NOIP* and is
unaffected.
Skip *NOIP* NICs in both the DHCPv4 and DHCPv6 reservation builders, the
same way the ISC path does -- there is no address to reserve for them.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Add a regression case to dhcp_kea_plugin_intent.t for a node whose mac
table entry uses the *NOIP* sentinel on a secondary NIC
("mac1|mac2!*NOIP*"). The mocked getipaddr resolves every name (including
the literal *NOIP*), so the only thing that can keep the node to a single
reservation is an explicit *NOIP* skip in kea_node_reservations() and
kea_node_reservations6(). The test asserts exactly one v4 and one v6
reservation, for the real NIC, with no reservation carrying *NOIP* as a
hostname -- covering both the "bogus second reservation" and the
"unresolved reservation aborts makedhcp" failure modes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The parallel-build failure gate only inspected $exit_code in run_on_finish. A
ForkManager child killed by a signal -- SIGKILL, or the OOM-killer under the
concurrent build load -- is reaped with $exit_code == 0 but $exit_signal != 0
(and possibly $core_dump). Such a worker therefore was NOT recorded as a
failure, so the parent could still index and GPG-sign a repository that is
missing the package that worker was building -- the exact partial-repo hazard
the gate was added to prevent, via a path it did not cover.
Capture $exit_signal and $core_dump from the run_on_finish callback and fail
the build when any of $exit_code, $exit_signal, or $core_dump is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses three concurrency-safety defects raised in review of the parallel
buildrpms.pl work:
1. Per-target build lock was released immediately. The flock filehandle was a
lexical (my $blk) scoped to the guard block, so it was destroyed -- and the
lock dropped -- as soon as that block exited, before any worker forked. The
"intentionally leaked" comment did not match the code. Hold the handle in a
file-scoped $BUILD_LOCK_FH so the fd (and the lock) live for the whole
process; forked children inherit the fd but their exits never release it.
2. Build failures were silently swallowed. buildspkgs()/buildpkgs() called
sh_retry() in void context, so a mock build that failed all retries returned
non-zero into the void; the child then exited 0 and the parent's run_on_finish
ignored the exit code. The parent could therefore index and GPG-sign a repo
that was missing packages and still exit 0. Now sh_retry failures die in the
child, run_on_finish records any non-zero child, and the run aborts before
update_repo and again before signing if any child failed -- never publishing
a partial core.
3. Abort cleanup unmounted unrelated builds. sweep_mock_mounts() lazy-unmounted
every bind under /var/lib/mock, which on a shared host tears out the live
chroots of concurrent, unrelated builds. Scope it to this run's own chroots
(each chroot dir plus its -bootstrap sibling), passed in from abort_builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
merge_core_repos already assembles the multi-arch core and runs the full
finalize tail (index -> sign -> final metadata -> write_release_alias, in the
required order so the xCAT-release-latest alias lands AFTER metadata and stays
out of the repo index). The preceding commit restored a separate finalize_core
sub purely to satisfy xcat_release_package.t, which grepped for
'sub finalize_core { ... $dir ... }'. That left two overlapping entry points:
--finalize-core (finalize one pre-assembled dir) and --merge-core-repos
(assemble N per-arch dirs THEN finalize) -- the former a strict subset of the
latter, with no in-tree or CI caller.
Consolidate on the single assemble+finalize path: inline the finalize tail into
merge_core_repos, drop sub finalize_core and the --finalize-core getopt/dispatch,
and update xcat_release_package.t to assert the stable alias is created after the
final metadata pass inside merge_core_repos ($out). Behaviour is unchanged (merge
already wrote the alias); only the redundant finalize_core interface is removed.
xcat_release_package.t 26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The CI buildrpms.pl port (d2d98b724) renamed the upstream sub finalize_core($dir)
to merge_core_repos($out) and inlined its index/sign/metadata/alias tail. The
xCAT-release logic (write the xCAT-release-latest stable bootstrap alias AFTER the
final metadata pass, so it stays out of the repo index) was preserved, but the rename
broke xCAT-test/unit/xcat_release_package.t, which statically pins:
sub finalize_core { ... write_repo_metadata_dir($dir); ... write_release_alias($dir); }
so 'assembled core repository creates the stable alias after final metadata' failed
(1 of 26), turning xcat_pr_test red on PR #7701.
Restore finalize_core($dir) as the shared finalize primitive (index -> sign ->
final metadata -> stable alias) and have merge_core_repos delegate to it on the
assembled multi-arch dir. Re-expose the --finalize-core CLI (single already-populated
dir) it also feeds. Behaviour is unchanged for both --merge-core-repos (CI) and
--finalize-core; only the shared code path is named again. xcat_release_package.t now
26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl built every package with mock -N (--no-clean), reusing the per-package
<pkg>-<target><ext> chroot across runs for flat disk. But a build aborted or killed
mid-flight leaves that chroot half-initialised with a corrupt rpmdb; the NEXT run
reused it and failed (cannot open Packages database .../usr/lib/sysimage/rpm),
producing an incomplete core (e.g. missing xCAT-test) that fails the deploy-time
completeness gate.
Re-init the buildroot (mock --init) right before building each package, after the
diskcache skip so it only runs when actually building. --init restores from mock's
root-cache tarball (cheap) so disk stays flat and builds stay fast; -N is kept on the
srpm/binary calls so they still reuse the freshly initialised root within the run.
This stabilises builds against corrupt state left by any previous failed/aborted build.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 3 EL CD pipelines (xcat-core-devel-cd, xcat-core-stable-cd, xcat-dep-el-cd)
overlaid a pinned $CI/buildrpms.pl at build time because the tree's buildrpms.pl
lacked the options they depend on:
- a per-target flock guard alongside --mock-uniqueext, so concurrent same-target
builds do not corrupt each other's /var/lib/mock chroot namespace;
- --native-only (build only arch-native pkgs on the secondary arch) plus
--merge-core-repos/--output-dir/--input-core-repos, replacing --finalize-core,
to assemble one signed flat multi-arch core from per-arch build outputs;
- sh_retry() to absorb transient mock/nspawn flakes;
- a single --target guard and graceful mock cancellation
(sweep_mock_mounts/abort_builds) that unmounts chroots on abort.
The xCAT-release repository package (master/2.19 only) is preserved: its
write_release_alias() is invoked from the new merge_core_repos() and per-target.
Porting them in-tree lets CI drop the $CI/buildrpms.pl pin and run the three
pipelines in parallel without the cross-job serialize lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #7677 added xCAT-server/share/xcat/netboot/rh/compute.rhels9.ppc64le.pkglist
with a stray empty line at the end of the file. The reviewer asked for it to be
removed before merge, but it slipped through. A trailing blank line in a pkglist
is meaningless and inconsistent with the rest of the netboot package lists, so
remove it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Vinícius Ferrão
Vinícius Ferrão
Daniel Hilst