Two ways the Subiquity install could fail without saying anything useful.
The boot flip read from the install monitor with no timeout. A monitor that accepts the
connection and then never answers -- which #7759 shows is a state it gets into -- blocked the read
forever, and with it the Subiquity late-command and the install. The retry loop could not help:
it never reached the retry. Both reads now take -t 10, so five attempts are bounded at roughly two
minutes and end in the failure that is already logged. The regression test stands up a listener
that accepts and holds the connection; removing the timeouts fails it.
mkinstall resolved the install server for nfsroot and fell back to the name when that failed:
"getipaddr($instserver) || $instserver". The name is the original defect -- klibc's nfsmount
cannot resolve one -- so the node panicked "can't parse IP address" at boot, on the node, with
nothing reported on the management node. The management node knows at template time, so it says
so there and skips the node, as the other unrecoverable per-node conditions in this routine do.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A node may list syncfiles in both postscripts and postbootscripts -- once for the install and
once for the booted node. The deferral prepended its own copy regardless, so the booted node ran
syncfiles twice in a row inside a marker block that made it look like xCAT had generated the
duplicate.
Keep the entry the node already has and still remove the install-time one, which is the copy
that cannot work in the in-target chroot.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Three passes of the same reasoning had accumulated: in the code, in the POD of the routines the
previous commit extracted, and again in the test headers. Say each once, where the reader needs it.
The POD blocks were the worst of it -- extracting subiquity_kcmdline() and
defer_syncfiles_to_postboot() moved the essays out of the routines but did not shorten them. The
toram paragraph ran eleven lines for one kernel argument; the exact systemd-shutdown message and
the size of the 24.04 layers are colour, not reasoning.
The template's two comments and the apt-sources pair are trimmed the same way: the sources.list
explanation lives in Template.pm, and the test says which case it is checking and points there.
That the tests still pass after rewriting the template is the point of the previous commit -- a
test matching its text would have broken here.
47 comment and POD lines removed, no behaviour change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The tests added with this fix matched regexes against the text of debian.pm, Postage.pm and
compute.subiquity.tmpl. A source match cannot tell whether the code it found ever runs: moving
the Subiquity command line into its own routine leaves every one of those assertions passing,
and reformatting a line fails them while the behaviour is untouched. One of them pinned Perl
syntax outright, qr/\$kcmdline\s*\.=\s*" ---";/, and another matched the text of a substitution.
Run the code instead.
Two decisions were lifted out of the routines that had grown around them, so a test can call
them: subiquity_kcmdline() in debian.pm builds the installer command line from its inputs, and
defer_syncfiles_to_postboot() in Postage.pm returns the adjusted postscript lists. Both are pure
and carry the reasoning that used to sit inline. The callers keep their behaviour exactly.
The template's two shell fragments are extracted and executed: the boot flip runs against a
stand-in for xcatd on the install-monitor port, and the resolv.conf step runs with a getent that
answers as the case requires.
Every assertion now fails when the behaviour it describes is removed, which is what the source
matches only appeared to do:
boot=casper dropped from the command line 1 assertion fails
toram dropped 1 assertion fails
nfsroot given the host name instead of the IP 2 assertions fail
the deferral made a no-op 7 assertions fail
the wrong token sent to xcatd 1 assertion fails
the failed-flip log line removed 3 assertions fail
the retry loop reduced to one attempt 2 assertions fail
resolv.conf given the host name 3 assertions fail
the resolution fallback removed 2 assertions fail
ubuntu_subiquity_bootparams.t is removed. Its three matches are covered by execution in
debian_subiquity_netboot.t, and it demonstrated the problem: it still passed after the command
line moved into another routine.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Boot the live installer correctly. Add boot=casper so casper actually processes
netboot=nfs instead of scanning local disks and panicking, resolve the install server
to a literal IP because casper mounts the live filesystem with klibc's nfsmount which
has no resolver, and add toram so casper copies the squashfs into RAM and unmounts the
NFS source. That last one is what lets the node reboot at all: with the NFS root still
mounted, systemd-shutdown waits forever on an lvm/pvscan wedged in uninterruptible I/O
on it and the node never power-cycles into the disk it just installed. casper has no
cmdline knob for NFS mount options -- it parses only nfsroot= and takes the whole value
as the path -- so toram is its supported way to avoid a network root.
Write the installer's resolv.conf nameserver as an IP. glibc's resolver discards a
hostname given on a nameserver line, so the xcatmaster name left the installer, and
the in-target apt that inherits the file, with no DNS at all.
Add the online archive through apt `sources:` on classic-sources releases, where
Subiquity renders the target sources.list from the install media alone and in-target
apt cannot find packages the ISO does not carry. Deb822 releases are excluded: their
primary mirror already lands in ubuntu.sources, so legacy .list files would configure
the same suites twice.
Flip the node to local-disk boot from the live installer over bash's /dev/tcp instead
of relying on updateflag.awk, which needs gawk's |& coprocess while Ubuntu's
/usr/bin/awk is mawk. The exchange is checked, and a failure is recorded in the
install log rather than silently PXE-looping into another install.
Defer syncfiles to the postbootscripts on the diskful install path, so it runs on the
booted node with sshd up rather than inside the in-target chroot where the MN cannot
reach it.
Make the test harness's provision wait overridable through WAIT_FOR_PROVISION; the
default stays short so a boot loop still fails fast.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The gating reg_linux_diskfull_installation_flat case cannot pass on Ubuntu: the
compute node never reaches a booted, installed OS, and the case only ever reports
"ssh: connect ... port 22: Connection refused". It is not one defect but a chain, each
reachable only once the one before it is fixed.
The installer never boots. Without boot=casper, casper never processes netboot=nfs --
it scans the local disks, finds no live media and panics "Unable to find a medium
containing a live file system". nfsroot must also be a literal IP: casper mounts the
live filesystem with klibc's nfsmount, which cannot resolve a hostname. And with the
NFS root still mounted at end of install, systemd-shutdown blocks forever on an
lvm/pvscan wedged in uninterruptible I/O on it, so the node never power-cycles into
the disk it just installed.
The installer has no usable DNS. A nameserver line in /etc/resolv.conf must hold an IP
-- glibc's resolver discards a hostname written there -- so writing the xcatmaster name
leaves the installer, and the in-target apt that inherits the file, hanging on
archive.ubuntu.com.
In-target apt cannot find its packages on classic-sources releases, because Subiquity
renders the target sources.list from the install media alone.
The node never leaves the installer. The boot flip to local disk goes through
updateflag.awk, which needs gawk's |& /inet coprocess, but Ubuntu's /usr/bin/awk is
mawk -- so the flip fails silently and the node reinstalls forever.
And syncfiles runs inside the in-target chroot, asking the MN to scp files into a node
that has no sshd yet, so it times out and the node reports failed on a good install.
Cover each stage. All fail today.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Add a unit test for the option that selects fping instead of nmap.
The test takes the specification out of the plugin source and gives it to
Getopt::Long with the settings that the daemon uses, so it drives the
specification that the plugin ships.
It shows that -f, --usefping and the older --useping each select fping, that
--use and --us still select usemon and do not select fping, that the bundles
-mf and -fm select both options, and that both places parse through the one
specification.
Add a unit test for the routine that chooses the compression program. The
test lifts the routine out of the plugin source, because the plugin needs a
management node to load.
The test shows that lzma is used when it is there, that xz stands in when it
is not, and that xz is asked for the lzma container rather than its own. It
also shows that the caller takes the command from the routine, that the file
keeps its name and its suffix, and that the gzip fallback and the rename into
place both remain.
The install disk autotests read the log of a provisioned node. The
choice files no longer carry the identifier in their name, and the
selection message names the driver group and the identifier instead of
the previous wording, so read the new lines. The reinstall case reads
the record of its disk without naming a group, as it did before.
Cover a RAID volume that reports a WWN against a direct attached disk
that reports none, in both scan orders, which the previous readback
decided by identifier. Keep the identifier rules of one group under
test as well: the disk that reports a WWN wins, the lower WWN wins
between two, and a path wins over no identifier at all.
Cover a guest whose only disk is a Xen disk, which the scan has to
select rather than leave to the fallback, and a guest with two Xen
disks, where the driver group decides. Against the previous filter both
cases fail.
Assert that the RHEL 10 copy is gone, that the RHEL 10 installer
includes the common script, and that the common script keeps the VROC
fallback, the Xen fallback and the guarded failure log.
The RHEL 10 copy of the script is about to go away, so stop naming it
here first. The cases keep running against the common script, so the
coverage does not change.
The install disk autotest reads the log of a node whose disks sit
behind a SAS host adapter, and that driver group moved from the second
choice to the third. Read the third group instead.
Run the real scripts in a sandbox. A stub udevadm serves the device
properties from fixture files, and the partition list and the output
paths move into the sandbox. Every case runs against the common script
and against the copy the RHEL 10 installer includes.
Cover the direct attached disk against a RAID volume, a RAID only
server, the host adapter against a direct attached disk and against an
unknown driver, an NVMe device from the last group, and the default
fallback. Against the previous scripts the RAID cases fail, so they
discriminate.
Daniel Hilst
Vinícius Ferrão