retry_install.sh retries a node provision $times (default 3) x 30 min. The flat
diskless/diskfull cases each call it multiple times, so when a node genuinely
fails to come up the retries stack to ~3.5-4 h per case (measured: a single
contention-flaked el8-x86 diskless burned 14181s in devel-cd #22), which both
balloons the CD wall-clock and delays the red verdict.
Two attempts still absorbs a one-off transient (a single slow/failed netboot)
while halving the worst-case retry time (3->2 tries per call). Callers that pass
an explicit count (e.g. the negative-provision check that passes 1) are
unaffected -- only the default changes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit b9336448f7)
copycds only defines a <osver>-<arch>-netboot-compute osimage when the netboot
compute pkglist/exlist/postinstall for that distro+arch exist under
share/xcat/netboot/<distro>/. For AlmaLinux the ppc64le set was present only for
alma10, so on an el8/el9 ppc64le management node copycds created just the
install-* osimages and no netboot one.
The stateless provisioning test then fails in a confusing way: the case's own
`chdef -t osimage -o <osver>-ppc64le-netboot-compute synclists=...` auto-creates
a bare object, so lsdef reports the image exists (imagetype=NIM and nothing
else), while genimage rejects it with
Error: Cannot find image '<osver>-ppc64le-netboot-compute' from the osimage table.
and packimage plus the install that follows fail with it.
Two gaps are filled:
- share/xcat/netboot/rh/compute.rhels9.ppc64le.pkglist did not exist at all
(rhels8 and rhels10 both ship one). Add it with the same content as the
rhels9 x86_64 list, matching rhels10 where the two arches are identical.
- alma8/alma9 ppc64le compute exlist/pkglist/postinstall are added as symlinks
into the rh/ equivalents, exactly as alma10 ppc64le and rocky10 ppc64le
already do.
With these present, copycds produces a fully populated netboot-compute osimage
(provmethod=netboot, pkglist/exlist/postinstall/rootimgdir set) as it already
does on alma10 ppc64le.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 8bb0606120)
Goconserver.pm shells out to `service goconserver start|stop|restart` (and
`service conserver stop`). That command is provided by the initscripts package,
which is NOT installed on a minimal EL9/EL10 management node -- there, every one
of these calls fails with rc!=0.
The damaging case is restart_service(). makegocons calls build_conf(), which
rewrites /etc/goconserver/server.conf with the cert-enabled configuration
(global.ssl_key_file/ssl_cert_file/ssl_ca_cert_file), and then calls
restart_service() so the daemon picks it up. When the restart fails, xCAT only
logs "Could not restart goconserver service." and the daemon keeps running with
the configuration it read at boot -- the packaged default, which has no ssl_*
fields. goconserver's TLS is gated on those fields being set (sslEnable stays
false), so it silently serves plain HTTP on the api port while Goconserver.pm
always talks to it over https. Every subsequent request then dies with
SSL connect attempt failed error:0A0000C6:SSL routines::packet length too long
so makegocons/makegocons -d report "Failed to send delete request." and the
console can never be registered. On EL8 the bug is invisible because initscripts
happens to be installed there.
Use the xCAT::Utils service helpers (startservice/stopservice/restartservice),
which resolve the unit through servicemap() and issue `systemctl <action> <unit>`
on systemd hosts, falling back to `service`/`initctl` only where appropriate.
This is the same mechanism the rest of the tree (AAsn.pm and friends) already
uses for named, dhcpd, nfs and others.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit b42bbe0e02)
envvar() resolves a template variable purely from %ENV, so a template that
references $XCATROOT expands to an empty string whenever XCATROOT is not
present in the environment of the process doing the substitution. $::XCATROOT
is set as a package global at module load and is reliably available, so use it
for that one variable before falling back to %ENV.
Recovered from the unmerged lenovobuild branch (original da72c6b8).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit 34866ebc23)
`ip -6 route` output for ECMP/multipath routes carries `nexthop` continuation
lines, and a `default` route has no network prefix. donets() treated both as
networks and added bogus entries to the networks table. Skip them alongside
the existing link-local/unreachable/via/lo filtering.
Recovered from the unmerged lenovobuild branch (original 7094ba0c).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit c97e973f36)
donets() only skips the exact address 'fe80::/64' when scanning routes, so a
link-local route with any other prefix (e.g. a longer fe80:: subnet) is added
to the networks table as a bogus network. Match any fe80:: prefix instead of
the single literal value.
Recovered from the unmerged lenovobuild branch (original 8b79cf85).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit d2be0c3865)
nodech's ^= operator removed only a single literal value, so
"nodech n1 groups^=a,b" tried to strip the combined string "a,b" and left the
individual groups in place. Split the value on commas and remove each piece.
Recovered from the unmerged lenovobuild branch (original 95c78b33).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit f71ddef6ba)
nodech's ,= operator appended the whole right-hand side as one value, so
"nodech n1 groups,=a,b" added the literal "a,b" and its duplicate check only
compared against that combined string. Split the value on commas and add each
piece individually, skipping any already present.
Recovered from the unmerged lenovobuild branch (original 086b0c0d).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
(cherry picked from commit 6e33661fc0)
run_fast_regression_test() prints a case's output only when it fails.
For 250 cases that is the right default, but it leaves no way to tell
whether a passing case did real work or skipped everything. That is not
academic for cases wrapping prove: prove exits 0 both when tests pass
and when every test skips, so integration_tests reports green either
way and the log cannot distinguish them.
Add @verbose_cases. A case named there has its output printed on a pass
as well, and the failure branch no longer prints a second copy. Seed it
with integration_tests to find out which of the three integration tests
actually run on a runner -- in particular whether
dhcp_kea_config_validation.t validates from /etc/kea now that the case
runs as root, or still skips. Emptying the list restores the previous
behaviour exactly.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 53e2b0bd7b)
The case gated only on rc==0, but prove exits 0 both when tests pass and
when every test skips, so the case could report green having run nothing
at all. That is not hypothetical: all three tests carry environment
guards, and on the pull request run the harness discards a passing
case's output, so there was no way to tell from the log whether anything
had executed.
Add check:output=~Files=3, which proves prove found all three files. It
still tolerates a legitimate skip on a node without Kea, and it catches
a packaging regression or a rename. A missing directory was already
caught, since prove -r on a nonexistent path exits 2.
The count has to be maintained by hand when tests are added here, which
the README now says.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 764e667e0a)
The integration testcase was added without the ci_test label on the
grounds that the pull request workflow has no management node. That was
wrong: github_action_xcat_test.pl builds xCAT, installs it, runs
setup-local-client.sh and chtab, and leaves xcatd running, which makes
the runner a single node management node. Running the ci_test cases
against it is exactly what the fast regression stage already does.
Label the case ci_test so the integration tests get pull request
coverage as well. This also unlocks a test that had been skipping for a
fixable reason: each case is invoked through sudo, and
dhcp_kea_config_validation.t skips only because /etc/kea is not writable
by the unprivileged runner user, so as root it validates instead of
skipping.
Running the integration suite as root and the unit suite unprivileged is
the right way round. Integration tests legitimately need to write to
places like /etc/kea, whereas running the unit tests as root would let
permission-related assertions pass for the wrong reason.
Correct both READMEs, which carried the mistaken claim that the
integration tests do not run in CI.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit de17401338)
Three of the files under xCAT-test/unit are not unit tests. They need an
installed management node rather than a checkout: copycds_packages_integrity.t
wants an /install populated by a real copycds, dhcp_kea_config_validation.t
wants a kea-dhcp4 binary that can read the config it generates, and
dhcp_kea_control_agent_smoke.t wants live kea-dhcp4 and kea-ctrl-agent
daemons running as root.
On a GitHub runner none of that exists, so all three plan skip_all. They were
the only three files skipping in the pull request run, which is not a
coincidence -- the skip is the symptom of them being filed in the wrong place.
A skipped test reports neither pass nor fail, so leaving them mixed in with
the unit tests trains the reader to scroll past skips in a directory where a
skip should mean something is wrong.
Move them to xCAT-test/integration, ship that directory alongside unit in
both the rpm and the deb, and drive it from a new xcattest testcase that
proves the installed copy on an MN. The case is deliberately not labelled
ci_test: the pull request workflow has no management node and must not pick
it up. check:rc==0 is the right gate for it -- prove exits non-zero on a real
failure, exits 0 when a test legitimately skips on a node without Kea, and
exits 2 if the directory is missing entirely, so a packaging regression still
fails the case.
Add a README.md to each directory recording which side of the line a new test
belongs on and how each suite is run.
xCAT-test/unit is now 44 files and 802 assertions with no skips at all; the
assertion count is unchanged, confirming the three moved files were
contributing nothing but skips.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 6ab9aca0b7)
The unit test stage failed on the first CI run with
Cannot detect source of 'xCAT-test/unit/*.t'
Files=0, Tests=0
Result: NOTESTS
the glob reached prove unexpanded because it matched nothing. The source
tree is gone by the time the tests run: build-ubunturepo sets
local_core_repo_path="$curdir/../../xcat-core"
and rm -rf's it before creating the apt repository there. GitHub checks
out into work/<repo>/<repo>, so for /home/runner/work/xcat-core/xcat-core
that path resolves to the checkout's own parent and the build wipes the
checkout, leaving an empty directory of the same name behind. The cd
still succeeds, which is why prove was handed a literal glob rather than
failing outright. This is also why every testcase that predates this
change proves /opt/xcat/share/xcat/tools/autotest/unit: after the build
the installed copy is the only one left.
Copy the checkout aside in preserve_source_tree() before the build and
prove that copy, so FindBin still resolves to a real source tree. Switch
to `prove -r xCAT-test/unit` as well, so a missing directory fails loudly
instead of silently degrading to a no-op the way an unmatched glob does.
Reproduced and verified by replaying the build under GitHub's directory
layout: the checkout drops to 0 test files, the preserved copy keeps all
47 and proves clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 63505793db)
The 47 test files under xCAT-test/unit/ were shipped but almost never
executed on a pull request. Only three hand-written xcattest cases
reached them -- dhcp_unit, ipmi_unit and xcatprobe_unit -- and each
proved a single glob against the installed copy, so the majority of the
suite had never run at all. Real drift went unnoticed as a result:
ubuntu_subiquity_template.t still asserted the pre-86e77bcd7 shape of
compute.subiquity.tmpl and failed against the current template.
Run `prove xCAT-test/unit/*.t` directly from github_action_xcat_test.pl.
The tests resolve xCAT modules and fixture files relative to the repo
root through FindBin, so they must be proved from the checkout and not
from /opt/xcat/share/xcat/tools/autotest/unit; install_xcat() chdir's
away, hence the getcwd() captured up front. The step runs after the
install because the suite needs the perl dependencies xCAT pulls in
(Net::DNS, XML::Simple) and a usable xCAT database.
Drop the three prove testcases so their tests do not run twice, and
refresh the two stale ubuntu_subiquity_template.t assertions: the
identity section is now intentional (86e77bcd7) and the MAC
normalization gained cut filters ahead of the tr (c6e38483f), which the
loosened regex plus a new assertion for the suffix stripping now cover.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit a0d2322b71)
Daniel Hilst
Vinícius Ferrão
xcat2-backport-automation[bot]
Vinícius Ferrão