The ppc64el and riscv64 xcat debs depend on xcat-genesis-scripts-amd64, and
xcat-genesis-scripts-ppc64 depends on xcat-genesis-base-ppc64, a package no
repository publishes. Nothing reports either one: the amd64 scripts package is
Architecture: all, so it installs on any architecture, and the broken ppc64
dependency is never reached because nothing pulls that package.
Extend debian_control_arch_coverage.t. It now reads the Depends field of
xCAT/debian/control and xCATsn/debian/control, applies each architecture
restriction the way dpkg-gencontrol does, and asserts that the genesis scripts
a given architecture receives are that architecture's own. It also asserts that
xCAT-genesis-scripts/debian/control-<arch> builds xcat-genesis-scripts-<arch>
and depends on xcat-genesis-base-<arch>.
Eight of the eighteen assertions fail on this tree.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The nodeset_shell_incorrectmasterip case ran "nodeset testnode shell", the command failed
with "/tftpboot/boot/grub2/grub2.x86_64 does not exits", and the case still passed.
check_destiny in xCAT-test/autotest/testcase/genesis/test.sh discards the return value of
runcmd and greps the boot configuration file, which grub2.pm writes before it stops on the
missing boot loader. The sub-case asserts nothing.
wait_for_boot in xCAT-test/autotest/testcase/genesis/genesistest.pl waits for
nodelist.status "booted". A Genesis node reports its destiny with getdestiny and xcatd
writes "shell", "configuring" or "booting" from it, never "booted". Every caller discards
the return value, so each case rests on its xdsh probes alone.
genesis_incorrectmasterip_check.t now runs the check with a nodeset that fails, and reads
whether the grub2 boot loader for the node arch is present when nodeset runs.
genesis_testcase_helpers.t drives the status wait with lsdef shadowed, and drives the shell
case with every command it runs shadowed. genesis_payload_verification.t reads a payload
without mktemp, which getdestiny needs to make its request file.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 3597760645ddf38f110457934982e2cc9a113f84)
doxcat names dhclient at six call sites. AlmaLinux 10 and EPEL 10 package no ISC
dhcp-client, so the legacy Genesis image for el10 carries no dhclient binary. A node
that boots that image reports "dhclient: command not found" on its console and never
acquires an address.
The test lifts the client selection out of doxcat and runs it with the clients shadowed
by stubs that record their own argv. doxcat cannot be sourced, so the routines are
extracted and driven on their own. It also reads the spec and the dracut module, which
decide what client reaches the image.
It fails on the current source for nine reasons: doxcat carries no selection routine and
no runner, it still starts command lines with dhclient and still chains into dhclient
from the secondary NIC loop, the spec build-requires no client on the releases that drop
the ISC one and does not check the payload for one, and the dracut module installs
dhclient alone.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 5d2801f400978c07f9073c7b920f327e990be25b)
A compute node fetches the legacy Genesis kernel and initramfs, the kernel starts, and then
nothing else happens: doxcat never runs, the node acquires no address, sshd refuses every
connection and the node stays at status=powering-on. The five genesis test cases in
xCAT-test/autotest/testcase/genesis have never passed. VersatusHPC/xcat-internal#78.
Three holes in the image, each fatal on its own. dracut_105/el/xcat-cmdline.sh ends in
`while :; do tmux attach-session -t doxcat || tmux new-session -s doxcat doxcat; done`, and the
image carries no locale data, so tmux exits with "need UTF-8 locale" and the loop spins
without ever reaching doxcat. module-setup.sh does not install
/usr/libexec/openssh/sshd-session, which OpenSSH 9.8 and later exec for every connection and
which EL9 now ships. xCAT-genesis-base.spec does not BuildRequire dhcp-client, so dhclient is
absent from the build chroot; dracut_install reports the missing binary and returns, and the
module install function keeps going, so the image ships without it.
xcat-cmdline.sh now resolves xcat_console_mode() once and runs doxcat directly when the
terminal multiplexer cannot start a session; the same shape replaces the screen loop on
Ubuntu. module-setup.sh installs the OpenSSH session helpers and the C.utf8 locale where they
exist. The spec BuildRequires dhcp-client on the releases that package it, and runs the new
xCAT-genesis-builder/verify-genesis-payload over the extracted payload, which fails the build
when sshd needs a helper the image lacks, when tmux has no UTF-8 locale, or when a binary the
caller named is missing.
The same runs exposed four defects in the test cases themselves. test.sh defined its synthetic
node as ppc64le whatever the management node was, so nodeset could not find a genesis kernel on
x86_64. genesistest.pl get_os() matched neither AlmaLinux nor Rocky and reported the OS as
unsupported. The -g check read $? instead of check_genesis_file()'s return value, so it could
never fail. And testxdsh() met "REMOTE HOST IDENTIFICATION HAS CHANGED" from the second boot
on, because Genesis makes new host keys every boot and nothing dropped the stale known_hosts
entry. test.sh now derives the node arch from uname and takes the tftp root from TFTPDIR,
get_os() recognises the redhat family, report_genesis_files() carries the result to an exit
status, and forget_host_keys() runs makeknownhosts -r before each probe.
Tests: genesis_console_mode.t drives xcat_console_mode() with the multiplexer shadowed;
genesis_payload_verification.t drives the verifier over payload trees carrying each hole;
genesis_testcase_helpers.t drives get_os(), check_genesis_file(), report_genesis_files() and
testxdsh(); genesis_incorrectmasterip_check.t runs test.sh against a scratch tftp root. Each
fails on the parent commit. The verifier also reports all three holes against the released
xCAT-genesis-base-x86_64-2.19.0-snap202609021858 payload.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit cb6021eb3cdb3abc75e4dd6704cb42b28074e140)
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Three case files ship a check line the harness cannot use. pscp/cases0 asks for
"$$CN: done" with no operator, and load_case drops a check whose content does not
start with a word character, so the two lines go without a message and the case
asserts less than it reads. rscan/cases0 writes a command as a check, and
ngpfb/cases0 compares rc against a pattern, which no operator accepts, so both cases
report "Unrecognized testcase syntax" and fail on every run.
Each line is repaired to what the case around it says it means. pscp prints
"<node>: done" for each node it copied, in xCAT-client/bin/pscp, so the two lines
become "output=~$$CN: done". rscan runs its check against the definitions that
"rscan -z -w" wrote, so "check:lsdef -l $$CN" becomes a cmd and keeps the two checks
that follow it. rmhwconn is asked for output without "state=LINE UP", which is what
the lshwconn checks in the same case assert the other way round.
xCAT-test/unit/autotest_check_lines_are_understood.t reads every check line under
xCAT-test/autotest/testcase and asserts the harness reports one result for each of
them, and that none uses an operator it does not know. Without this change it names
all five: two lines from ngpfb, one from rscan, and pscp reporting 37 results for
the 39 check lines it ships.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A failing case named one failed check and no result at all for the checks after it,
although the commands of the case kept running. The first [Failed] line was read as
the cause of the failure three times this week, and each time the real fault was a
later check: a riscv64 cell reported a makedns check 160 lines before rpower could
not start the domain.
run_case in xCAT-test/xcattest used one variable, $failflag, for two facts: the
result of the case, and the result of the check being reported. Every branch read
$failflag to decide whether to print [Pass] or [Failed], so a check that ran after a
failed one always read as failed. The guard "last if ($failflag)" at the top of the
check loop hid that, and hid every later check with it.
The result of a check is now $checkfail, set and read inside one iteration. A
continue block carries it into $failflag, which keeps the result of the case. The
guard and the per-branch "last" statements are gone, so each check reports what it
found. The output ~~ branch no longer clears $failflag on a match, which without the
guard would have turned a failed case into a passing one.
xCAT-test/unit/xcattest_report_every_check.t runs the harness over a fixture case and
asserts on the CHECK lines it writes. Without this change it reports two of four
checks, and one of two failed checks. A case whose checks all pass logs the same text
before and after: no truncation could happen while $failflag stayed 0.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The name of the volume of a node, and the bus of a file-backed disk, could come from a
match made by a routine on the call path. A riscv64 node breaks on it: a leaked value
that is neither scsi nor virtio gives the node an hd* volume, and the riscv64 virt
machine has no IDE controller for that disk.
createstorage and build_diskstruct in xCAT-server/lib/xcat/plugins/kvm.pm read the model
of the disk out of the vmstorage value with s/=(.*)//, then read $1. The substitution is
allowed to fail, because most vmstorage values state no model, and a failed match leaves
$1 as the last successful capture. dohyp gives every node the storage model scsi before
mkvm runs, and a captured value takes priority over it, so a leaked value can only
replace the default that keeps a riscv64 node on sd*.
The leak follows the call path, not the history of the process. Perl restores $1 when the
block that set it ends, so a match made in a routine that has returned cannot reach
createstorage; only a match still live in an enclosing block can, and a later successful
match without a group empties $1 again. A long-running xcatd is not what makes this
happen, and looking for one is a wrong turn.
Both routines now read $1 only when their own substitution matches. A vmstorage value
that states a model, and vmstoragemodel, name the volume as before.
The default itself moves into default_storagemodel, which dohyp calls, so a test can hold
it. It sat inline with a comment, and changing it to ide left every assertion passing.
kvm_createstorage_model.t runs each node twice, once with a capture left live in the
calling block, because a case that leaves $1 empty passes against the defect. Five of its
eleven assertions fail without this change, and a sixth fails if the default changes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A node whose vmstorage is a libvirt storage pool (dir://, nfs:// or lvm://) got a <disk>
element with no bus attribute. libvirt then chose the controller from the name of the
device alone, so the disk of a riscv64 node worked only while its volume was named sd*.
build_diskstruct in xCAT-server/lib/xcat/plugins/kvm.pm matched the pool entry, a hash
reference, against /^vd/, /^hd/ and /^sd/. A reference in a match is its address as a
string, so no branch ran and the bus was never set. The name of the device is in the
device field of that entry.
The three tests now read that field. The bus each one sets is the bus libvirt gives an
hd*, sd* or vd* name, so no domain changes: a riscv64 node keeps the sd* name its volume
has, and keeps the scsi controller the riscv64 virt machine provides. libvirt stores the
domain built before this change with bus="scsi" on that disk, which is what the domain
built after it states.
kvm_diskstruct_bus.t drives build_diskstruct in a scratch package, with a stub storage
pool in place of the one routine that reaches libvirt, and asserts the bus of an hd*, an
sd* and a vd* volume. It also asserts that a riscv64 node keeps the sd* name of its
volume. Four of its seven assertions fail without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs.pl replaced build-ubunturepo, and both CD pipelines prefer it: the
fallback to build-ubunturepo fires only for refs that predate builddebs.pl, and
such a ref carries its own copy. Nothing on this branch runs the script, so its
presence only invites edits that never reach a build. The developer guide said
it was kept as a differential oracle until the CD pipelines moved over. They
have.
Remove the script, and record the removal in the build guide beside the
buildcore.sh, makerpm and buildlocal.sh entries.
xcat_probe_package_payload.t asserted the Debian staging by matching a `cp -f`
line in build-ubunturepo. builddebs.pl stages the helpers through
XCAT::BuildUtils::stage_probe_helpers, so the test now calls that function and
checks the files it produced. Verified by making stage_probe_helpers skip a
helper: the assertion fails.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A node with arch=riscv64 got an x86_64 libvirt domain from mkvm. The node took a
DHCP lease, received the riscv64 GRUB binary that nodeset staged, and could not run
it. The firmware fell through to the empty disk and stopped, so both flat
provisioning cases of the riscv64 cell failed with a node that never installed.
build_xmldesc and build_diskstruct in xCAT-server/lib/xcat/plugins/kvm.pm read the
architecture from the hypervisor cpumodel. The arch of the node was never read while
the domain XML was built, so on an x86_64 hypervisor every guest was an x86_64
guest, whatever the node said.
guest_arch_profile now takes the arch of the node as well, and returns the domain
type, the <os> arch and machine, the firmware and the device settings that follow
from them. A riscv64 node becomes a qemu domain with the virt machine type and UEFI
firmware. It drops the parts the riscv64 virt machine has no controller for, or that
libvirt refuses there: the pae, acpi and apic features, the SeaBIOS serial option,
the ich6 sound card, the USB tablet, and the ide disk and hd* optical drive. libvirt
resolves the emulator and the UEFI firmware files itself. POWER and x86_64 domains
do not change.
kvm_guest_arch.t drives build_xmldesc and build_diskstruct in a scratch package,
stubbing only the routines that reach libvirt or the xCAT database, and asserts the
domain and the disks of each architecture. Ten of its twenty assertions fail without
this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 8d149c856302c8016fb0ead31e9859fd5a1e9dff)
mknb stages the Genesis payload before building a netboot image, and those copies are the
only point at which it learns that an installed Genesis image is unusable. The legacy branch
runs two of them and keeps only the second exit status, so an unreadable root tree is
invisible: mknb exits 0 having built an initramfs from nothing, and the node never boots
with no error naming the cause. When the kernel copy is the one that fails, the message
blames the root tree instead.
Extract the staging decision as stage_genesis_payload, preserving today's behaviour exactly,
so the outcome can be driven with an injected runner instead of a real Genesis tree.
The test fails on this commit, 2 of 10: 'an unreadable root tree fails the step' and 'the
failure names the kernel, not the root tree'.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 989deaa31eb1f3979a8db030d3ae8c404abda2fb)
builddebs.pl replaced build-ubunturepo as the Ubuntu builder, and the arch support
riscv64 has in build-ubunturepo did not come with it. The pipeline prefers builddebs.pl
whenever the ref carries it, so on this branch the switch silently stops producing riscv64
debs: no xcat_*_riscv64.deb in the pool, and a published Release that says
'Architectures: amd64 ppc64el'. apt on a riscv64 management node then reports 'Unable to
locate package xcat', which is the same failure build-ubunturepo was fixed for.
builddebs.pl reads its architectures from BuildUtils, so unlike build-ubunturepo -- which
hardcoded the pair in three places -- riscv64 goes in one: @DEB_ARCHES.
xcat-genesis-scripts is the exception and needs its own rule. Its per-arch deb Depends on
xcat-genesis-base-<arch>, and no riscv64 genesis-base deb exists, because riscv64 takes the
OpenEmbedded Genesis image from the shared xcat-dep pool. Built for riscv64 it would be
uninstallable, so deb_package_arches excludes it.
build_utils.t covers both: removing riscv64 from @DEB_ARCHES fails four assertions,
including the reprepro Architectures line, and removing the genesis-scripts exclusion fails
its own.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 90f9156411b11b33496a470574f37fcdc2a7330f)
Installing xCAT on a riscv64 Ubuntu management node fails before it starts:
E: Unable to locate package xcat
E: Unable to locate package xcat-test
xCAT/debian/control and xCATsn/debian/control name their architectures
explicitly, as "amd64 ppc64el". riscv64 is absent, so no riscv64 deb is ever
produced and apt has nothing to install -- while the rest of the tree already
carries riscv64 install templates, DHCP boot policy, mknb support and a Genesis
machine configuration.
The test reads both control files and asserts the explicit list covers every
Debian architecture xCAT ships. It fails on both files today.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 059c23486d8f15e8a0a224ef7dfe05cd7fe123a4)
A riscv64 build of xcat-core fails at xCAT-genesis-scripts:
ERROR: Cannot find/open srpm: dist/rocky-10-riscv64-xcat/rpms/SRPMS/
xCAT-genesis-scripts-riscv64-2.19.0-snap202609020458.src.rpm
The srpm on disk is named xCAT-genesis-scripts-%{tarch}-2.19.0-....src.rpm.
xCAT-genesis-scripts.spec and xCAT-genesis-base.spec take the package name from
%{tarch}, which an %ifarch ladder sets. That ladder has no riscv64 branch, so
%{tarch} stays literal and rpm builds a package with a macro in its name.
The test expands both specs with rpmspec for every arch xCAT supports and
asserts the Name carries that arch. It fails on riscv64 for both specs.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit 07a0e79e42ec39eaf1ed6e9291ab6b3f6f869e97)
Drive the assignment genimage makes and check the name debootstrap receives for
each architecture, including the POWER LE spelling that debootstrap rejects.
Pin both directions for every architecture xCAT supports on Ubuntu, the
pass-through for names Debian shares, and the round trip that copycd and
debootstrap depend on agreeing about.
Build each Ubuntu media layout on disk and ask the resolver for its kernel and
initrd: the netboot trees, the flat netboot layout, both casper images, and the
POWER layouts where the kernel and the initrd sit in different directories.
Pin the precedence the installer depends on, a netboot tree over a live image
and a hardware-enablement kernel over the release one, and pin the three ways
media resolve to nothing.
getipaddr answers from %::hostiphash before it resolves. The bypass tests
OnlyV6 and GetAllAddresses and does not test OnlyV4, so a caller that asks for
IPv4 is handed whatever the first lookup cached.
An unrestricted lookup passes AF_UNSPEC to getaddrinfo, so on a dual-stack
management node with an AAAA record it caches the IPv6 address. xcatd is
long-lived and the hash is a global, so one earlier caller poisons every OnlyV4
caller after it.
debian.pm then writes nfsroot=2001:db8::1:/install, which is not a parseable
nfsroot, and the Subiquity install never mounts. dhcp.pm and mknb.pm hold four
more OnlyV4 callers with the same exposure.
The test also pins what the fix must not break: an IPv4 cache entry is still
served to an OnlyV4 caller, and an unrestricted caller still gets its cache hit
whatever family it holds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Subiquity diskful path builds three values from configuration, and nothing
checks that the value it builds is one the consumer can use.
The boot flip addresses port 3002. xcatd's install monitor listens on
site.xcatiport, so a cluster that moves the port loses the flip and every node
PXE-loops. The exchange also counts any reply as an accepted request, so a
different service on that port reads as a flipped node. xcatd greets with
"ready" and answers every request with "done".
subiquity_nfsroot_server resolves the install server without a family. A
dual-stack management node answers with its IPv6 address, and nfsroot=<v6>:<path>
is unparseable, because klibc nfsmount takes everything after the first colon as
the path.
The DNS setup writes the xcatmaster name as a nameserver when getent fails,
which is the case the step exists to prevent.
ubuntu_subiquity_boot_flip.t drives the template's own late-command against a
stand-in monitor. ubuntu_resolvconf_ip.t runs the DNS step and reads the file it
writes. debian_subiquity_boot_params.t drives the helpers with a dual-stack
resolver. All three fail on this commit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The static host query reads dhcpd.conf when no configuration is in memory. When the
read fails the query returns nothing, and listnode reports "no DHCP reservation
found" -- the answer for a node that has no reservation. The operator cannot tell
the two apart.
The same query only reads a "hardware ethernet" line. An InfiniBand node declares
"hardware infiniband", so its query answer carries no hardware address. A twin
declaration inside the same markers must not replace the primary one either.
The new assertions drive the writer to build both InfiniBand shapes, set the path of
dhcpd.conf to a file that does not exist, and call listnode. A deletion that names a
hostname is asserted to keep the other declarations of the node.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
makentp reads site.ntpbackend to pick the NTP daemon, so an admin has to set it, but the
attribute appears in no help text. lsdef -t site -h and tabdump -d print the site table
description from xCAT::Schema, which documents site.dhcpbackend and says nothing about
ntpbackend. The makentp man page lists the site attributes the command honors and does not
list it either.
The selector test now reads the site help from the loaded schema and the makentp pod.
Five assertions fail: the attribute name, its three valid values, and the man page entry.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT::NTP::Backend->available reports chrony as available when chronyd exists. makentp
configures chrony only when systemctl exists as well, and setupntp hands over to ntpd without
it. On a host with chronyd and no systemctl the selector answers chrony, makentp takes the ntpd
path anyway, and the admin is told nothing.
ntp_backend_selection.t drives available and choose with the command probe injected, so a case
can hold chronyd present and systemctl absent without depending on what the test host has
installed.
Six assertions fail: available reports chrony on chronyd alone, and choose neither downgrades
nor asks for an install.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
setupntp treats the requested backend as a preference in one direction only. A node given
--backend chrony without chronyd uses ntpd, but a node given --backend ntpd without ntpd still
hands over to setupntp.traditional, which writes /etc/ntp.conf and starts a daemon that is not
installed.
The selection cases in makentp_ntp_deps.t now also make ntpd absent. The stub for check_executes
takes a list of absent commands, so a case can hide ntpd, or ntpd and chronyd together.
Two cases fail: --backend ntpd and --use-ntpd both keep ntpd where only chrony is present.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The respawn tests cover the pacing in xCAT::RespawnUtils. Two things the daemon
itself has to do are untested.
A SIGCHLD handler that does not clear $pid_MON leaves xcatd holding a dead pid,
so the service loop never re-forks the monitor. Only ssl_reaper clears it, and
generic_reaper is the handler at startup and again while connections are
throttled.
The respawn block also runs from the middle of the service loop, so the child
inherits the client connections the parent has accepted but not yet dispatched.
xcatd cannot be loaded in a unit test, so the two reapers and the respawn fork
block are lifted out of the program text and run in a scratch package against
stand-in descriptors. generic_reaper fails, and so does the pending-connection
assertion; ssl_reaper passes and guards the path that already works.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
DHCP backend auto-selection fails hard when the backend it prefers is not installed.
The xcat metapackage's "isc-dhcp-server | kea" Depends guarantees isc-dhcp-server,
while auto-selection prefers kea on 22.04+, where kea arrives only through Recommends.
An apt run that skips recommends therefore leaves a node with isc only, auto picks the
absent kea, and makedhcp errors out on a freshly installed management node (#7710).
makedhcp -q re-enters omshell on the very releases whose write paths avoid it. xCAT
already records each reservation as a static host block in dhcpd.conf on Ubuntu's
ISC-limited releases because their ISC DHCP 4.4 omshell can wedge at 100% CPU and
never be reaped, but listnode still called _omshell_query_host unconditionally.
Both host-block scans also match the node name loosely: /\Q$node\E\b.* start$/ matches
at a hyphen, so "compute" matches the "compute-01" marker -- a query can answer with
another node's address and a delete can remove another node's reservation.
Cover all three: the auto-selection fallback, a query answered from dhcpd.conf without
omshell, and exact node-name matching for both the query and the delete.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Replace source-shape extraction with real preprocess and command execution. Model missing table rows with the same return shape as xCAT::Table.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão