mirror of
https://github.com/xcat2/xcat-core.git
synced 2026-09-05 04:27:55 +00:00
test(xcat-core): capture the Ubuntu Subiquity diskful install never completing
The gating reg_linux_diskfull_installation_flat case cannot pass on Ubuntu: the compute node never reaches a booted, installed OS, and the case only ever reports "ssh: connect ... port 22: Connection refused". It is not one defect but a chain, each reachable only once the one before it is fixed. The installer never boots. Without boot=casper, casper never processes netboot=nfs -- it scans the local disks, finds no live media and panics "Unable to find a medium containing a live file system". nfsroot must also be a literal IP: casper mounts the live filesystem with klibc's nfsmount, which cannot resolve a hostname. And with the NFS root still mounted at end of install, systemd-shutdown blocks forever on an lvm/pvscan wedged in uninterruptible I/O on it, so the node never power-cycles into the disk it just installed. The installer has no usable DNS. A nameserver line in /etc/resolv.conf must hold an IP -- glibc's resolver discards a hostname written there -- so writing the xcatmaster name leaves the installer, and the in-target apt that inherits the file, hanging on archive.ubuntu.com. In-target apt cannot find its packages on classic-sources releases, because Subiquity renders the target sources.list from the install media alone. The node never leaves the installer. The boot flip to local disk goes through updateflag.awk, which needs gawk's |& /inet coprocess, but Ubuntu's /usr/bin/awk is mawk -- so the flip fails silently and the node reinstalls forever. And syncfiles runs inside the in-target chroot, asking the MN to scp files into a node that has no sshd yet, so it times out and the node reports failed on a good install. Cover each stage. All fail today. Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
This commit is contained in:
Daniel Hilst
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
use Test::More;
|
||||
|
||||
# Regression: the Ubuntu (subiquity) diskful install netboots the live installer over NFS.
|
||||
# The kernel command line built in debian.pm must satisfy TWO casper requirements or the
|
||||
# installer never boots and the node PXE-loops forever (surfacing to the test as
|
||||
# "ssh: connect ... port 22: Connection refused"):
|
||||
#
|
||||
# 1. boot=casper -- without it casper does not process netboot=nfs at all; it scans the
|
||||
# local disks, finds nothing, and panics "Unable to find a medium containing a live
|
||||
# file system", dropping to the initramfs emergency shell.
|
||||
#
|
||||
# 2. nfsroot must be a literal IP, NOT a hostname. casper mounts the live filesystem with
|
||||
# klibc's nfsmount, which does NOT resolve hostnames -- it fails with
|
||||
# "nfsmount: can't parse IP address '<mn-hostname>'". (busybox/util-linux `mount -t nfs`
|
||||
# resolves names, which is why manual mounts work while casper's does not.) The instserver
|
||||
# must therefore be passed through xCAT::NetworkUtils->getipaddr() before it is put in
|
||||
# nfsroot=. The ds=...http URL is fetched later by cloud-init in the booted live system,
|
||||
# where normal DNS works, so only nfsroot needs the IP.
|
||||
|
||||
sub slurp { my ($p) = @_; local $/; open my $fh, '<', $p or return undef; <$fh> }
|
||||
|
||||
my $deb = slurp('xCAT-server/lib/xcat/plugins/debian.pm');
|
||||
plan skip_all => 'debian.pm not found' unless defined $deb;
|
||||
|
||||
# The exact netboot fragment is unique to the subiquity install cmdline, so match it directly
|
||||
# instead of trying to slice one of the several using_subiquity() blocks.
|
||||
like($deb, qr/\bboot=casper\b/,
|
||||
'subiquity netboot kcmdline includes boot=casper (else casper scans local disks and panics)');
|
||||
like($deb, qr/autoinstall ip=dhcp boot=casper netboot=nfs nfsroot=\$\{nfsip\}:/,
|
||||
'subiquity kcmdline is: autoinstall ip=dhcp boot=casper netboot=nfs nfsroot=<ip>');
|
||||
like($deb, qr/getipaddr\(\$instserver\)/,
|
||||
'instserver is resolved to an IP via getipaddr for nfsroot (klibc nfsmount cannot resolve hostnames)');
|
||||
unlike($deb, qr/nfsroot=\$\{instserver\}:/,
|
||||
'nfsroot does NOT use the bare instserver hostname (would break klibc nfsmount)');
|
||||
|
||||
# The subiquity netboot cmdline must include 'toram' so casper copies the squashfs into RAM and
|
||||
# unmounts the NFS source -- otherwise the end-of-install reboot wedges in systemd-shutdown on a
|
||||
# D-state process doing I/O to the still-mounted NFS root, and the node never boots the installed disk.
|
||||
like($deb, qr/\btoram\b/,
|
||||
'subiquity netboot cmdline includes toram (run from RAM, no NFS root at shutdown -> no reboot hang)');
|
||||
unlike($deb, qr/nfsroot=[^ ]*,soft/,
|
||||
'nfsroot does NOT append ,soft (casper takes the whole nfsroot value as the path, which breaks the mount)');
|
||||
|
||||
done_testing();
|
||||
@@ -0,0 +1,59 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
|
||||
use File::Spec;
|
||||
use FindBin;
|
||||
use Test::More;
|
||||
|
||||
# Regression: on Ubuntu/Debian the DISKFUL install runs a node's postscripts inside the
|
||||
# installer's in-target chroot -- before the node has booted as itself. The syncfiles
|
||||
# postscript works by asking the management node to scp files INTO the running node, which
|
||||
# cannot happen in that phase: the not-yet-booted node has no sshd for the MN to reach, so the
|
||||
# push times out, syncfiles exits 1, and the node reports status=failed even though the OS
|
||||
# installed perfectly.
|
||||
#
|
||||
# syncfiles must therefore move to the postbootscripts set, which runs on the booted node
|
||||
# where ssh is already listening. Two things bound that move:
|
||||
#
|
||||
# * It applies to the DISKFUL install path only. netboot and statelite already run their
|
||||
# postscripts on the booted node, so moving syncfiles there changes working behaviour for
|
||||
# no reason. EL/SLES are unaffected either way -- their postscripts already run on the
|
||||
# booted node.
|
||||
# * syncfiles must run BEFORE the existing postbootscripts, since a postbootscript may
|
||||
# consume the files it synchronises. Appending it would invert that.
|
||||
|
||||
my $repo_root = File::Spec->rel2abs(
|
||||
File::Spec->catdir( $FindBin::Bin, '..', '..' )
|
||||
);
|
||||
my $path = File::Spec->catfile( $repo_root, 'xCAT-server', 'lib', 'perl', 'xCAT', 'Postage.pm' );
|
||||
plan skip_all => "Postage.pm not found" unless -f $path;
|
||||
|
||||
my $src = do { local $/; open my $fh, '<', $path or die $!; <$fh> };
|
||||
|
||||
# Isolate the deferral block so the assertions below cannot accidentally match code elsewhere.
|
||||
my ($block) = $src =~ /(\n[^\n]*ubuntu\|debian[^\n]*\n(?:.*?\n)*?[^\n]*syncfiles(?:.*?\n)*?\s*\}\n\s*\}\n)/;
|
||||
ok( defined $block, 'found the syncfiles deferral block in makescript' )
|
||||
or do { done_testing(); exit };
|
||||
|
||||
like( $block, qr/\$os\s*=~\s*.\^\(\?:ubuntu\|debian\)/,
|
||||
'the move is guarded to ubuntu/debian nodes only' );
|
||||
|
||||
like( $block, qr/\$diskful_install|\bnodesetstate\b|\bprovmethod\b/,
|
||||
'the move is guarded to the diskful install path (not netboot/statelite)' );
|
||||
|
||||
like( $block, qr/\$postscripts\s*=~\s*s\/\^\[[^\]]*\]\*syncfiles/,
|
||||
'syncfiles is stripped from the in-target postscripts list' );
|
||||
|
||||
like( $block, qr/\$postbootscripts\s*=\s*"[^"]*syncfiles[^"]*"\s*\.\s*\$postbootscripts/,
|
||||
'syncfiles is PREPENDED to postbootscripts, so it runs before scripts that consume its files' );
|
||||
|
||||
unlike( $block, qr/\$postbootscripts\s*\.=\s*"syncfiles/,
|
||||
'syncfiles is not appended after the existing postbootscripts' );
|
||||
|
||||
# The strip and the re-add must be paired, so a node with no syncfiles postscript never
|
||||
# acquires a spurious one.
|
||||
like( $block, qr/if\s*\(.*?\$postscripts.*?syncfiles.*?\)\s*\{\s*.*?\$postbootscripts/s,
|
||||
'syncfiles is only added to postbootscripts when it was present in postscripts' );
|
||||
|
||||
done_testing();
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
use Test::More;
|
||||
|
||||
# Regression: the Ubuntu (subiquity) diskful install writes the installer's /etc/resolv.conf
|
||||
# from compute.subiquity.tmpl's "DNS setup" early-command. A nameserver line in
|
||||
# /etc/resolv.conf MUST be an IP address -- glibc's resolver does NOT resolve a hostname
|
||||
# given on a nameserver line, it simply discards the entry. The template used to write the
|
||||
# xcatmaster *name* there:
|
||||
#
|
||||
# echo "nameserver #TABLE:noderes:$NODE:xcatmaster#" >/etc/resolv.conf
|
||||
#
|
||||
# which left the installer (and the in-target apt-get update, which inherits this resolv.conf)
|
||||
# with no usable DNS. The install then hangs resolving archive.ubuntu.com -- the node never
|
||||
# finishes, and the test sees "ssh: connect ... port 22: Connection refused". (Everything the
|
||||
# early-command did BEFORE this point still worked because the live installer's DHCP-provided
|
||||
# resolv.conf can resolve the xcatmaster name -- so the failure only surfaces at the very next
|
||||
# DNS-dependent step, the in-target apt.)
|
||||
#
|
||||
# The fix resolves the xcatmaster hostname to an IPv4 address (via getent, while DNS still
|
||||
# works) and writes that IP into resolv.conf.
|
||||
|
||||
sub slurp { my ($p) = @_; local $/; open my $fh, '<', $p or return undef; <$fh> }
|
||||
|
||||
my $tmpl = slurp('xCAT-server/share/xcat/install/ubuntu/compute.subiquity.tmpl');
|
||||
plan skip_all => 'compute.subiquity.tmpl not found' unless defined $tmpl;
|
||||
|
||||
# The old, broken form -- a bare hostname token written straight into the nameserver line --
|
||||
# must be gone.
|
||||
unlike($tmpl, qr/nameserver \s+ \#TABLE:noderes:\$NODE:xcatmaster\# /x,
|
||||
'resolv.conf nameserver is NOT the bare xcatmaster hostname token (glibc cannot resolve a name there)');
|
||||
|
||||
# The xcatmaster name must be resolved to an IPv4 address before it is used as a nameserver.
|
||||
like($tmpl, qr/getent \s+ ahostsv4 /x,
|
||||
'DNS setup resolves the xcatmaster to an IPv4 address via getent ahostsv4');
|
||||
|
||||
# And the nameserver line must be written from that resolved IP variable.
|
||||
like($tmpl, qr/nameserver \s+ \$xcatmaster_ip/x,
|
||||
'resolv.conf nameserver is written from the resolved xcatmaster IP');
|
||||
|
||||
done_testing();
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
|
||||
use FindBin;
|
||||
use lib "$FindBin::Bin/../../perl-xCAT";
|
||||
use lib "$FindBin::Bin/../../xCAT-server/lib/perl";
|
||||
|
||||
use Test::More;
|
||||
|
||||
# Regression: on an ONLINE Subiquity install of a classic-sources release (20.04 / 22.04),
|
||||
# Subiquity renders the target's /etc/apt/sources.list from the install media alone
|
||||
# ("deb file:///cdrom"). A live-server ISO does not carry everything xCAT's postscripts need --
|
||||
# chrony, for one -- so curtin's in-target apt fails with "E: Unable to locate package chrony"
|
||||
# and the install crashes. `sources_list:` is a curtin key that Subiquity's autoinstall schema
|
||||
# ignores, so the online archive has to be added through `sources:`, which Subiquity honours by
|
||||
# writing /etc/apt/sources.list.d/*.list.
|
||||
#
|
||||
# That fix belongs to the classic-sources releases ONLY. On a Deb822 release (24.04 / 26.04)
|
||||
# the primary mirror already lands in /etc/apt/sources.list.d/ubuntu.sources, so adding legacy
|
||||
# .list files on top produces duplicate apt entries for the same suites -- exactly the source
|
||||
# conflict the Deb822 branch elsewhere in this function goes out of its way to avoid.
|
||||
|
||||
require xCAT::Template;
|
||||
|
||||
my $MIRROR = 'http://br.archive.ubuntu.com/ubuntu';
|
||||
|
||||
sub apt_config_for {
|
||||
my (%opt) = @_;
|
||||
no warnings 'redefine';
|
||||
local *xCAT::Template::ubuntu_subiquity_apt_mirror = sub { $opt{mirror} };
|
||||
local *xCAT::Template::ubuntu_subiquity_uses_deb822_sources = sub { $opt{deb822} };
|
||||
local *xCAT::Template::ubuntu_subiquity_otherpkg_sources = sub { () };
|
||||
local *xCAT::Template::ubuntu_subiquity_uses_generated_cdrom_source = sub { 0 };
|
||||
return xCAT::Template::ubuntu_subiquity_apt_config('/some/media/dir');
|
||||
}
|
||||
|
||||
# --- online, classic sources (20.04 / 22.04): the archive must be added via sources: ---
|
||||
my $classic = apt_config_for( mirror => $MIRROR, deb822 => 0 );
|
||||
|
||||
like( $classic, qr/^\s*mirror-selection:/m,
|
||||
'classic online install still sets the primary mirror' );
|
||||
like( $classic, qr/^\s*sources:/m,
|
||||
'classic online install adds apt sources (sources_list is ignored by Subiquity)' );
|
||||
like( $classic, qr/xcat-ubuntu-archive\.list:/,
|
||||
'classic online install writes an xCAT-owned archive source' );
|
||||
like( $classic, qr/xcat-ubuntu-updates\.list:/,
|
||||
'classic online install writes an xCAT-owned updates source' );
|
||||
like( $classic, qr/deb \Q$MIRROR\E \$RELEASE main restricted universe multiverse/,
|
||||
'the archive source uses the configured mirror and curtin\'s $RELEASE token' );
|
||||
|
||||
# --- online, Deb822 (24.04 / 26.04): no legacy .list files on top of ubuntu.sources ---
|
||||
my $deb822 = apt_config_for( mirror => $MIRROR, deb822 => 1 );
|
||||
|
||||
like( $deb822, qr/^\s*mirror-selection:/m,
|
||||
'Deb822 online install still sets the primary mirror' );
|
||||
unlike( $deb822, qr/xcat-ubuntu-archive\.list:/,
|
||||
'Deb822 online install does NOT add a legacy archive .list (it would duplicate ubuntu.sources)' );
|
||||
unlike( $deb822, qr/xcat-ubuntu-updates\.list:/,
|
||||
'Deb822 online install does NOT add a legacy updates .list' );
|
||||
|
||||
# --- offline is untouched by any of this ---
|
||||
my $offline = apt_config_for( mirror => '', deb822 => 0 );
|
||||
like( $offline, qr/fallback: offline-install/,
|
||||
'the offline path is unchanged' );
|
||||
unlike( $offline, qr/xcat-ubuntu-archive\.list:/,
|
||||
'the offline path adds no online archive source' );
|
||||
|
||||
done_testing();
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env perl
|
||||
use strict;
|
||||
use warnings;
|
||||
|
||||
use File::Spec;
|
||||
use FindBin;
|
||||
use Test::More;
|
||||
|
||||
# Regression: after Subiquity installs the OS the node MUST be flipped to local-disk boot, or it
|
||||
# PXE-loops straight back into the installer and the installed OS -- with its sshd -- never
|
||||
# boots. The diskful case then only ever reports
|
||||
# "ssh: connect to host <cn> port 22: Connection refused".
|
||||
#
|
||||
# xCAT performs that flip when the node reports "next" to xcatd on the install-monitor port,
|
||||
# which makes xcatd run "nodeset <node> next" and rewrite the node's xNBA script to fall through
|
||||
# to the local disk. The in-target post-script tries this through updateflag.awk, but that
|
||||
# depends on gawk's |& / /inet coprocess and on /usr/bin/awk being gawk -- on Ubuntu
|
||||
# /usr/bin/awk is normally mawk, which has neither, so the flip fails silently.
|
||||
#
|
||||
# The template must therefore trigger the flip itself from the live installer, with no gawk
|
||||
# dependency, and must NOT report success when it did not happen.
|
||||
|
||||
my $repo_root = File::Spec->rel2abs(
|
||||
File::Spec->catdir( $FindBin::Bin, '..', '..' )
|
||||
);
|
||||
my $path = File::Spec->catfile(
|
||||
$repo_root, 'xCAT-server', 'share', 'xcat', 'install', 'ubuntu', 'compute.subiquity.tmpl'
|
||||
);
|
||||
plan skip_all => 'compute.subiquity.tmpl not found' unless -f $path;
|
||||
|
||||
my $tmpl = do { local $/; open my $fh, '<', $path or die $!; <$fh> };
|
||||
|
||||
like( $tmpl, qr{/dev/tcp/\$xm/3002},
|
||||
'the flip contacts xcatd on the install-monitor port from the installer' );
|
||||
like( $tmpl, qr{printf "next},
|
||||
'it sends the "next" token that makes xcatd run "nodeset <node> next"' );
|
||||
like( $tmpl, qr{xm=#XCATVAR:XCATMASTER#},
|
||||
"the flip targets the node's own xcatmaster" );
|
||||
like( $tmpl, qr{^\s*-\s*\['bash',\s*'-c',}m,
|
||||
'the flip runs under bash (dash has no /dev/tcp) via an argv list command' );
|
||||
|
||||
# A failed flip must be visible. The original form broke out of its retry loop as soon as the
|
||||
# socket connected -- never on a confirmed exchange -- and ended in `true`, so a total failure
|
||||
# looked exactly like success and the node silently reinstalled forever.
|
||||
like( $tmpl, qr{ok=1},
|
||||
'success is recorded only after the exchange completes, not merely on connect' );
|
||||
like( $tmpl, qr{FAILED to flip},
|
||||
'a failed flip is reported into the install log rather than passing silently' );
|
||||
|
||||
# The magic-SysRq forced reboot must be gone: toram already unmounts the NFS live root, so the
|
||||
# shutdown hang it worked around cannot occur, and an unconditional timed reboot would race a
|
||||
# slow install.
|
||||
unlike( $tmpl, qr{sysrq-trigger},
|
||||
'no unconditional magic-SysRq reboot (toram removes the hang it worked around)' );
|
||||
|
||||
done_testing();
|
||||
@@ -11,7 +11,8 @@ plan skip_all => "debian.pm not found" unless -f $plugin_path;
|
||||
|
||||
my $src = do { local $/; open my $fh, '<', $plugin_path or die $!; <$fh> };
|
||||
|
||||
like($src, qr/autoinstall ip=dhcp netboot=nfs/, 'subiquity bootparams enable autoinstall');
|
||||
like($src, qr/autoinstall ip=dhcp boot=casper netboot=nfs/,
|
||||
'subiquity bootparams enable autoinstall (boot=casper is required for casper to process netboot=nfs)');
|
||||
like($src, qr/ds=nocloud-net;s=http:\/\//, 'subiquity bootparams point at NoCloud seed');
|
||||
like($src, qr/\$kcmdline\s*\.=\s*" ---";/, 'subiquity bootparams end with installer argument separator');
|
||||
|
||||
|
||||
Reference in New Issue
Block a user