makedhcp using the Kea backend fails for any node whose mac table entry
uses the *NOIP* sentinel to mark a secondary NIC that has no IP address
(e.g. "mac1|mac2!*NOIP*|mac3!*NOIP*"). kea_node_reservations() and
kea_node_reservations6() split each NIC's hostname and pass it straight
to getipaddr(), so the literal string "*NOIP*" is treated as a host to
resolve. It cannot resolve, and the reservation is reported as unresolved
-- which aborts the whole makedhcp run, leaving the node with no Kea
reservation at all and therefore no DHCP lease.
This breaks provisioning on distros that use the Kea backend (e.g. EL10,
which has no ISC dhcp-server): the node never gets an address and is
unreachable, while the ISC backend has always handled *NOIP* and is
unaffected.
Skip *NOIP* NICs in both the DHCPv4 and DHCPv6 reservation builders, the
same way the ISC path does -- there is no address to reserve for them.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 9125d4da3c)
Add a regression case to dhcp_kea_plugin_intent.t for a node whose mac
table entry uses the *NOIP* sentinel on a secondary NIC
("mac1|mac2!*NOIP*"). The mocked getipaddr resolves every name (including
the literal *NOIP*), so the only thing that can keep the node to a single
reservation is an explicit *NOIP* skip in kea_node_reservations() and
kea_node_reservations6(). The test asserts exactly one v4 and one v6
reservation, for the real NIC, with no reservation carrying *NOIP* as a
hostname -- covering both the "bogus second reservation" and the
"unresolved reservation aborts makedhcp" failure modes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 52cdf7fe36)
The parallel-build failure gate only inspected $exit_code in run_on_finish. A
ForkManager child killed by a signal -- SIGKILL, or the OOM-killer under the
concurrent build load -- is reaped with $exit_code == 0 but $exit_signal != 0
(and possibly $core_dump). Such a worker therefore was NOT recorded as a
failure, so the parent could still index and GPG-sign a repository that is
missing the package that worker was building -- the exact partial-repo hazard
the gate was added to prevent, via a path it did not cover.
Capture $exit_signal and $core_dump from the run_on_finish callback and fail
the build when any of $exit_code, $exit_signal, or $core_dump is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses three concurrency-safety defects raised in review of the parallel
buildrpms.pl work:
1. Per-target build lock was released immediately. The flock filehandle was a
lexical (my $blk) scoped to the guard block, so it was destroyed -- and the
lock dropped -- as soon as that block exited, before any worker forked. The
"intentionally leaked" comment did not match the code. Hold the handle in a
file-scoped $BUILD_LOCK_FH so the fd (and the lock) live for the whole
process; forked children inherit the fd but their exits never release it.
2. Build failures were silently swallowed. buildspkgs()/buildpkgs() called
sh_retry() in void context, so a mock build that failed all retries returned
non-zero into the void; the child then exited 0 and the parent's run_on_finish
ignored the exit code. The parent could therefore index and GPG-sign a repo
that was missing packages and still exit 0. Now sh_retry failures die in the
child, run_on_finish records any non-zero child, and the run aborts before
update_repo and again before signing if any child failed -- never publishing
a partial core.
3. Abort cleanup unmounted unrelated builds. sweep_mock_mounts() lazy-unmounted
every bind under /var/lib/mock, which on a shared host tears out the live
chroots of concurrent, unrelated builds. Scope it to this run's own chroots
(each chroot dir plus its -bootstrap sibling), passed in from abort_builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT 2.18 defaults the DNS/DHCP OMAPI TSIG key to hmac-md5. On EL9/EL10 the newer
bind/Net::DNS reject md5-signed dynamic updates (TSIG BADSIG), so makedns fails
(FORMERR) and node DNS setup / install cases fail on a DEFAULT install, needing a
manual 'chdef -t site dhcpomapialgorithm=hmac-sha256' workaround.
Backport xcat2/xcat-core#7597: xcatconfig initDB detects a fresh install and, via
OmapiPolicy::new_install_default_algorithm, seeds site.dhcpomapialgorithm=hmac-sha256
for EL9+/Ubuntu 20.04+ (EL8/older keep hmac-md5, which works there). Existing sites
are untouched. Default installs on EL9/EL10 now work with no intervention.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl built every package with mock -N (--no-clean), reusing the per-package
<pkg>-<target><ext> chroot across runs for flat disk. But a build aborted or killed
mid-flight leaves that chroot half-initialised with a corrupt rpmdb; the NEXT run
reused it and failed (cannot open Packages database .../usr/lib/sysimage/rpm),
producing an incomplete core (e.g. missing xCAT-test) that fails the deploy-time
completeness gate.
Re-init the buildroot (mock --init) right before building each package, after the
diskcache skip so it only runs when actually building. --init restores from mock's
root-cache tarball (cheap) so disk stays flat and builds stay fast; -N is kept on the
srpm/binary calls so they still reuse the freshly initialised root within the run.
This stabilises builds against corrupt state left by any previous failed/aborted build.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 3 EL CD pipelines (xcat-core-devel-cd, xcat-core-stable-cd, xcat-dep-el-cd)
overlaid a pinned $CI/buildrpms.pl at build time because the tree's buildrpms.pl
lacked the options they depend on:
- a per-target flock guard alongside --mock-uniqueext, so concurrent same-target
builds do not corrupt each other's /var/lib/mock chroot namespace;
- --native-only (build only arch-native pkgs on the secondary arch) plus
--merge-core-repos/--output-dir/--input-core-repos, replacing --finalize-core,
to assemble one signed flat multi-arch core from per-arch build outputs;
- sh_retry() to absorb transient mock/nspawn flakes;
- a single --target guard and graceful mock cancellation
(sweep_mock_mounts/abort_builds) that unmounts chroots on abort.
Porting them in-tree lets CI drop the $CI/buildrpms.pl pin and run the three
pipelines in parallel without the cross-job serialize lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
wait_for_provision caps how long retry_install polls for a node to reach
'booted' after each rinstall. Measured healthy provisions in devel-cd #22 boot
well inside this window (diskless ~6-8 min, diskfull ~8-16 min), so 30 min was
mostly slack that only lengthened the give-up time on a genuinely failing node.
Drop it to 20 min: still above the ~16-min diskfull install, while shrinking each
failed attempt's cost (6 min fixed sleep + 20 poll = 26 min vs 36). Combined with
the 3->2 attempt default, a fully-failing case's retry stack drops substantially.
Trade-off noted: the margin over a slow-under-load diskfull install is now tighter.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 851284067d)
retry_install.sh retries a node provision $times (default 3) x 30 min. The flat
diskless/diskfull cases each call it multiple times, so when a node genuinely
fails to come up the retries stack to ~3.5-4 h per case (measured: a single
contention-flaked el8-x86 diskless burned 14181s in devel-cd #22), which both
balloons the CD wall-clock and delays the red verdict.
Two attempts still absorbs a one-off transient (a single slow/failed netboot)
while halving the worst-case retry time (3->2 tries per call). Callers that pass
an explicit count (e.g. the negative-provision check that passes 1) are
unaffected -- only the default changes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit b9336448f7)
copycds only defines a <osver>-<arch>-netboot-compute osimage when the netboot
compute pkglist/exlist/postinstall for that distro+arch exist under
share/xcat/netboot/<distro>/. For AlmaLinux the ppc64le set was present only for
alma10, so on an el8/el9 ppc64le management node copycds created just the
install-* osimages and no netboot one.
The stateless provisioning test then fails in a confusing way: the case's own
`chdef -t osimage -o <osver>-ppc64le-netboot-compute synclists=...` auto-creates
a bare object, so lsdef reports the image exists (imagetype=NIM and nothing
else), while genimage rejects it with
Error: Cannot find image '<osver>-ppc64le-netboot-compute' from the osimage table.
and packimage plus the install that follows fail with it.
Two gaps are filled:
- share/xcat/netboot/rh/compute.rhels9.ppc64le.pkglist did not exist at all
(rhels8 and rhels10 both ship one). Add it with the same content as the
rhels9 x86_64 list, matching rhels10 where the two arches are identical.
- alma8/alma9 ppc64le compute exlist/pkglist/postinstall are added as symlinks
into the rh/ equivalents, exactly as alma10 ppc64le and rocky10 ppc64le
already do.
With these present, copycds produces a fully populated netboot-compute osimage
(provmethod=netboot, pkglist/exlist/postinstall/rootimgdir set) as it already
does on alma10 ppc64le.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 8bb0606120)
Goconserver.pm shells out to `service goconserver start|stop|restart` (and
`service conserver stop`). That command is provided by the initscripts package,
which is NOT installed on a minimal EL9/EL10 management node -- there, every one
of these calls fails with rc!=0.
The damaging case is restart_service(). makegocons calls build_conf(), which
rewrites /etc/goconserver/server.conf with the cert-enabled configuration
(global.ssl_key_file/ssl_cert_file/ssl_ca_cert_file), and then calls
restart_service() so the daemon picks it up. When the restart fails, xCAT only
logs "Could not restart goconserver service." and the daemon keeps running with
the configuration it read at boot -- the packaged default, which has no ssl_*
fields. goconserver's TLS is gated on those fields being set (sslEnable stays
false), so it silently serves plain HTTP on the api port while Goconserver.pm
always talks to it over https. Every subsequent request then dies with
SSL connect attempt failed error:0A0000C6:SSL routines::packet length too long
so makegocons/makegocons -d report "Failed to send delete request." and the
console can never be registered. On EL8 the bug is invisible because initscripts
happens to be installed there.
Use the xCAT::Utils service helpers (startservice/stopservice/restartservice),
which resolve the unit through servicemap() and issue `systemctl <action> <unit>`
on systemd hosts, falling back to `service`/`initctl` only where appropriate.
This is the same mechanism the rest of the tree (AAsn.pm and friends) already
uses for named, dhcpd, nfs and others.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit b42bbe0e02)
envvar() resolves a template variable purely from %ENV, so a template that
references $XCATROOT expands to an empty string whenever XCATROOT is not
present in the environment of the process doing the substitution. $::XCATROOT
is set as a package global at module load and is reliably available, so use it
for that one variable before falling back to %ENV.
Recovered from the unmerged lenovobuild branch (original da72c6b8).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit 34866ebc23)
`ip -6 route` output for ECMP/multipath routes carries `nexthop` continuation
lines, and a `default` route has no network prefix. donets() treated both as
networks and added bogus entries to the networks table. Skip them alongside
the existing link-local/unreachable/via/lo filtering.
Recovered from the unmerged lenovobuild branch (original 7094ba0c).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit c97e973f36)
donets() only skips the exact address 'fe80::/64' when scanning routes, so a
link-local route with any other prefix (e.g. a longer fe80:: subnet) is added
to the networks table as a bogus network. Match any fe80:: prefix instead of
the single literal value.
Recovered from the unmerged lenovobuild branch (original 8b79cf85).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit d2be0c3865)
nodech's ^= operator removed only a single literal value, so
"nodech n1 groups^=a,b" tried to strip the combined string "a,b" and left the
individual groups in place. Split the value on commas and remove each piece.
Recovered from the unmerged lenovobuild branch (original 95c78b33).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit f71ddef6ba)
nodech's ,= operator appended the whole right-hand side as one value, so
"nodech n1 groups,=a,b" added the literal "a,b" and its duplicate check only
compared against that combined string. Split the value on commas and add each
piece individually, skipping any already present.
Recovered from the unmerged lenovobuild branch (original 086b0c0d).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit 6e33661fc0)
run_fast_regression_test() prints a case's output only when it fails.
For 250 cases that is the right default, but it leaves no way to tell
whether a passing case did real work or skipped everything. That is not
academic for cases wrapping prove: prove exits 0 both when tests pass
and when every test skips, so integration_tests reports green either
way and the log cannot distinguish them.
Add @verbose_cases. A case named there has its output printed on a pass
as well, and the failure branch no longer prints a second copy. Seed it
with integration_tests to find out which of the three integration tests
actually run on a runner -- in particular whether
dhcp_kea_config_validation.t validates from /etc/kea now that the case
runs as root, or still skips. Emptying the list restores the previous
behaviour exactly.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 53e2b0bd7b)
The case gated only on rc==0, but prove exits 0 both when tests pass and
when every test skips, so the case could report green having run nothing
at all. That is not hypothetical: all three tests carry environment
guards, and on the pull request run the harness discards a passing
case's output, so there was no way to tell from the log whether anything
had executed.
Add check:output=~Files=3, which proves prove found all three files. It
still tolerates a legitimate skip on a node without Kea, and it catches
a packaging regression or a rename. A missing directory was already
caught, since prove -r on a nonexistent path exits 2.
The count has to be maintained by hand when tests are added here, which
the README now says.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 764e667e0a)
The integration testcase was added without the ci_test label on the
grounds that the pull request workflow has no management node. That was
wrong: github_action_xcat_test.pl builds xCAT, installs it, runs
setup-local-client.sh and chtab, and leaves xcatd running, which makes
the runner a single node management node. Running the ci_test cases
against it is exactly what the fast regression stage already does.
Label the case ci_test so the integration tests get pull request
coverage as well. This also unlocks a test that had been skipping for a
fixable reason: each case is invoked through sudo, and
dhcp_kea_config_validation.t skips only because /etc/kea is not writable
by the unprivileged runner user, so as root it validates instead of
skipping.
Running the integration suite as root and the unit suite unprivileged is
the right way round. Integration tests legitimately need to write to
places like /etc/kea, whereas running the unit tests as root would let
permission-related assertions pass for the wrong reason.
Correct both READMEs, which carried the mistaken claim that the
integration tests do not run in CI.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit de17401338)
Three of the files under xCAT-test/unit are not unit tests. They need an
installed management node rather than a checkout: copycds_packages_integrity.t
wants an /install populated by a real copycds, dhcp_kea_config_validation.t
wants a kea-dhcp4 binary that can read the config it generates, and
dhcp_kea_control_agent_smoke.t wants live kea-dhcp4 and kea-ctrl-agent
daemons running as root.
On a GitHub runner none of that exists, so all three plan skip_all. They were
the only three files skipping in the pull request run, which is not a
coincidence -- the skip is the symptom of them being filed in the wrong place.
A skipped test reports neither pass nor fail, so leaving them mixed in with
the unit tests trains the reader to scroll past skips in a directory where a
skip should mean something is wrong.
Move them to xCAT-test/integration, ship that directory alongside unit in
both the rpm and the deb, and drive it from a new xcattest testcase that
proves the installed copy on an MN. The case is deliberately not labelled
ci_test: the pull request workflow has no management node and must not pick
it up. check:rc==0 is the right gate for it -- prove exits non-zero on a real
failure, exits 0 when a test legitimately skips on a node without Kea, and
exits 2 if the directory is missing entirely, so a packaging regression still
fails the case.
Add a README.md to each directory recording which side of the line a new test
belongs on and how each suite is run.
xCAT-test/unit is now 44 files and 802 assertions with no skips at all; the
assertion count is unchanged, confirming the three moved files were
contributing nothing but skips.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 6ab9aca0b7)
xcat2-backport-automation[bot]
Vinícius Ferrão
Daniel Hilst
Vinícius Ferrão