ddns.pm's DNS-update send has a 3-try retry loop, but it only retried on
NOTAUTH and -- on the final attempt -- the `next` fell out of the loop without
reporting failure. makedns then printed "Completed updating DNS records." and
"DNS setup is completed" even when the update was never accepted.
Two problems this causes:
- SERVFAIL was never retried. named routinely replies SERVFAIL transiently
right after a zone (re)load, before the zone is ready to accept dynamic
updates; the update succeeds a second or two later. This reliably breaks real
flat-cluster provisioning on el9/el10, where the test case's makedns -n runs
right after named reloads the zone with the new compute node.
- A persistent NOTAUTH/SERVFAIL (retries exhausted) was silently swallowed and
reported as success, so a genuinely failed DNS update looked like it worked.
Extract the batch send into send_ddns_update(), which retries NOTAUTH and
SERVFAIL (pausing on SERVFAIL), returns 0 only on NOERROR, and otherwise emits
the error and returns 1. Both batch-send sites now `return 1 if
send_ddns_update(...)`, so a real failure aborts add_or_delete_records instead
of falsely reporting success.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Modern x86-64 UEFI firmware (including OVMF and recent iPXE) reports DHCP
client architecture 00:07, but the xNBA network-boot conditionals only matched
00:09. A UEFI node would load xnba.efi, re-request with user-class xNBA and
arch 00:07, fail to match, and be handed xnba.efi again -- an endless boot
loop. Accept 00:07 alongside 00:09 in the xNBA UEFI branches of both the
network-level (addnet) and node-level (addnode) dhcpd statements.
Validated on an OVMF UEFI client: it progresses from xnba.efi to the .uefi
genesis config and boots, instead of looping on xnba.efi.
Recovered from the unmerged lenovobuild branch (5d697b5c, 86b0e12e).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
The generated /opt/xcat/xcatinstallpost ran the install postscripts first and
only disabled the xcatpostinit1 service afterwards. If one of those postscripts
rebooted the node (for example a firmware or kernel update), the service was
still enabled, so on the next boot the whole postscript set ran again from the
start. Disable the service *before* running the postscripts so a mid-postscript
reboot cannot re-trigger them.
The disable decision reads OSVER (base variant only), RUNBOOTSCRIPTS and
NODESTATUS, which the included xcatinstallpost script would otherwise set.
Moving the INCLUDE below the disable block leaves those variables unset, so read
them up front (the included script re-reads them) and source xcatlib.sh so
msgutil_r is available; the disable decision is then identical to before.
The original change (lenovobuild 9184cbe0) touched only the legacy post.xcat.
Modern EL uses the systemd variants: post.xcat.ng (RHEL/CentOS 8 and 9) and
post.xcat.rhels10 (RHEL 10). Apply the same reorder to all three so the fix
actually takes effect on current platforms.
Lab-validated with a full AlmaLinux 9.8 stateful install (post.xcat.ng path):
the generated xcatinstallpost disables the service before running the
postscripts, the postscripts run exactly once, the node reaches "booted", and
with the default NODESTATUS=1 the service is left enabled just as before.
Recovered from the unmerged lenovobuild branch (original 9184cbe0), adapted and
extended to the .ng and .rhels10 variants.
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
replaycons chose the confluent console log directory
(/var/log/confluent/consoles) whenever the confetty binary happened to be
present. On a host that has confluent installed but runs conserver as the
active console service, that directory is empty and replaycons could not find
the log. Gate on the site table's consoleservice attribute instead -- the
same setting rcons uses -- so replaycons reads /var/log/confluent/consoles
only when confluent is actually the configured console service.
Recovered from the unmerged lenovobuild branch (originals 3ce4d81b3, 57f6ddd38).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
The DNS-managed-networks check used my_ip_facing($net->{net}), which only
returns local addresses on the interface directly facing that network. A
nameserver that is a local address reachable via a different (routed)
interface was therefore treated as external, and the network was wrongly
dropped from ddns management.
Use thishostisnot(), which recognizes any local address regardless of the
interface, and keep the existing <xcatmaster> and site-nameserver special
cases.
Recovered from the unmerged lenovobuild branch (original 3c343f83), adapted
to preserve those special cases: lab validation showed the original replaced
them wholesale, which would exclude any network whose nameserver is the
literal <xcatmaster> token (thishostisnot('<xcatmaster>') is true).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
The diskless/statelite xcatroot scripts only recognize an http:// rootfs
image URL; an https:// imgurl falls through to the "unsupported" path even
though the download itself already uses wget/curl on the full $imgurl, both
of which handle TLS.
Accept xhttps alongside xhttp in the protocol test. Applied to every current
xcatroot variant (rh dracut/dracut_033/dracut_047/dracut_105 stateless and
statelite, plus fedora, sles and ubuntu), so HTTPS works uniformly rather
than only on the three rh scripts the original touched.
Recovered from the unmerged lenovobuild branch (original 751bce24, extended
to the dracut generations and distros that postdate it).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
mknb writes the compressed genesis filesystem by streaming cpio output
straight onto the final path in $tftpdir/xcat (genesis.fs.$arch.lzma or .gz).
When two mknb/nodeset runs target a shared $tftpdir at once, one run reads or
overwrites the other's half-written image, producing a corrupt genesis.fs and
failed netboots.
Write each image to a uniquely-suffixed temporary file
(xCAT::Utils::genpassword(24)) and File::Copy::move() it into place. move()
within the same directory is a rename, so the final path only ever appears
complete. Both modules are already imported by mknb.pm.
Recovered from the unmerged lenovobuild branch (originals c651e7b8 and its
fix da0723f2, which switched the suffix to genpassword after the first
attempt only produced a single character). master lacks the branch's xz
path, so only the lzma and gzip paths are adjusted here.
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
xCAT-server.spec sets "AutoReqProv: no", so RPM does not auto-generate
dependencies from the Perl modules this package ships. Modules that are
loaded (compile-time "use") only by xCAT-server code are therefore neither
auto-required here nor pulled in transitively by perl-xCAT (which does have
AutoReqProv on and already covers SNMP, Expect, JSON, Net::Ping, Time::HiRes
and Text::Balanced). On a host without them the package installs cleanly and
the affected subcommands then fail at load time with "Can't locate <M>.pm".
Add the five that are genuinely uncovered, each a compile-time "use" in a
module shipped by xCAT-server and used nowhere in perl-xCAT:
perl-Net-Telnet xCAT/SSHInteract.pm (telnet switch/console)
perl-Net-DNS plugins/activedirectory.pm
perl-Crypt-CBC xCAT/IPMI.pm (IPMI 2.0 RMCP+ crypto)
perl-Crypt-Rijndael xCAT/IPMI.pm (IPMI 2.0 RMCP+ crypto)
perl-DB_File Confluent/Client.pm
The Crypt::CBC / Crypt::Rijndael gap is the notable one: IPMI.pm fails to
load without them, disabling all IPMI-based hardware control on a host where
those modules are absent.
Recovered from the unmerged lenovobuild branch (Jarrod Johnson; originals
807d30d3, c2e22f6f, 792e41727), reduced to the subset that master's current
packaging does not already satisfy.
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
go-xcat's install list still named yaboot-xcat, which was obsoleted in the
xcat-dep 2.18 rebuild and is no longer published. It also lists the cross-arch
ppc64 genesis packages, which the arch-split 2.18 dep repo no longer places in
the x86_64 view an x86 management node reads. Because EL dnf is strict by
default, any one of these unresolvable names aborts the entire `dnf install`
transaction, so `go-xcat install` fails outright on EL9/EL10 even though every
package the node actually needs is available.
Drop yaboot-xcat from the install list (kept in the uninstall list so existing
installs are still cleaned up) and pass `dnf --setopt=strict=0` so a package
missing from the enabled repository is warned and skipped instead of failing
the whole install. The existing post-install smoke test still catches a
genuinely broken install.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
cons/kvm drives the KVM guest serial console through a shared tmux session on
the hypervisor (tmux, unlike screen, is present on every libvirt host we target
incl. ppc64le hosts that ship no screen). The KVM host package lists were never
updated for that switch, so an xCAT-provisioned KVM host got 'screen' (now
unused) -- or nothing -- where tmux is required, leaving rcons broken.
Ship tmux in the KVM host pkglists:
- install/rh, install/ubuntu: screen -> tmux / add tmux.
- netboot/{alma,centos,rocky}: were symlinks to the RHEL 5.4 list; make them
real lists with tmux (the rhel5/6/7 lists keep screen).
- install/suse: add a KVM host list (SUSE had none) carrying tmux.
screen stays where it is still used: the Xen console, and the EOL rhel5/6/7 and
centos6/7 lists.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The diskful service-node otherpkgs pkglist exists for every EL version and
both arches EXCEPT rhels10 ppc64le: install/rh/ has
service.rhels10.x86_64.otherpkgs.pkglist (and install/alma/ symlinks it) but
no ppc64le counterpart, while rhels5/6/7/8/9 and the netboot/ tree carry both
arches. xcat-test's SN_setup_case points a ppc64le service node's osimage
otherpkglist at install/<distro>/service.<ver>.ppc64le.otherpkgs.pkglist; with
the file absent the list resolves empty, so the otherpkgs postscript installs
nothing -- xCATsn never lands, xcatd never starts on the service node, and
every hierarchical dispatch fails with "<sn>:3001 No route to host /
Connection refused", cascading the whole *_hierarchy test suite.
Add install/rh/service.rhels10.ppc64le.otherpkgs.pkglist (xCATsn + the rh10
ppc64le goconserver, mirroring rhels9) and the install/alma symlink to it,
matching the existing x86_64 layout.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
cons/kvm ran `ssh -t <vmhost> screen ... <pty>`, which requires screen on every
hypervisor. Bare libvirt hosts (notably ppc64le power servers) ship tmux but not
screen, so goconserver's console fork failed and no console was captured.
Use tmux (present on all our libvirt hosts) for the shared detach/reattach
session, wrapping `virsh console` keyed on the domain name (== the xCAT node name
for KVM) -- stable across guest reboots, unlike the raw serial pty. The session
status bar is disabled so the captured console log stays clean serial.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
build_xmldesc only recognised cpumodel "ppc64" (big-endian) when deciding to set
<os type arch='ppc64' machine='pseries'>, but ppc64le hosts report cpumodel
"ppc64le" (virNodeGetInfo model). A guest on a ppc64le hypervisor was therefore
emitted as an x86-style domain -- no machine type and with pae/acpi/apic -- and
libvirt rejected it: "machine type 'pseries-*' does not support ACPI".
Recognise "ppc64le" alongside "ppc64" for the arch/machine block, and omit the
x86-only pae/acpi/apic features on both. Little- and big-endian pseries guests
both use libvirt <os> arch "ppc64".
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The single flat xcat-core repo serves el8/el9/el10 from one build, but two spec
decisions were resolved at BUILD time and so were wrong for the other ELs that
share the same repo.
1. DHCP backend. xCAT.spec and xCATsn.spec selected the dhcp provider with a
build-time %if (rhel >= 10 -> kea, else dhcpd), so an el10 build wrongly
required kea on el8/el9 (and an el8/el9 build wrongly required dhcpd on
el10). Replace it with an install-time RPM rich dependency:
Requires: (kea if (system-release >= 10) else /usr/sbin/dhcpd)
Requires: (kea-hooks if (system-release >= 10))
dnf now resolves it per client: kea on el10+ (which dropped ISC dhcp from the
distro), dhcpd on el8/el9. SLES has no "system-release" provide, so the
condition is false there and it falls back to /usr/sbin/dhcpd, preserving
prior behavior. system-release is versioned per release package
(el10=10.x, el9=9.x, el8=8.x).
2. openssl. Make the xCAT-server openssl Requires uniform across EL (non-SUSE)
instead of el10-only, so the produced package set does not depend on which EL
the build host happened to be.
Also drop xCAT-genesis-base from the default @PACKAGES set in buildrpms.pl: its
initramfs bundles the build-chroot kernel/glibc and is therefore OS- and
arch-dependent, so it cannot ship in the single flat core. It is now built per
target by the xcat-dep pipeline (mockbuild-all.pl, via
`buildrpms.pl --package xCAT-genesis-base`) and ships in xcat-dep/rh<N>. The
explicit `--package xCAT-genesis-base` build path is retained.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Ubuntu netboot image inherited the build host's /etc/hostname --
genimage's chroot shares the build host's UTS namespace, so /etc/hostname
ended up as the MN's name and every diskless node booted as that name
instead of its own ("hostname poisoning"). xCAT keys on IP/MAC so
provisioning still succeeds, but the node's local identity (hostname,
prompt, syslog) is wrong and N nodes share one name. The image also
shipped no netplan, so systemd-networkd-wait-online failed. In
compute.postinstall, clear /etc/hostname so each node takes its name at
boot from DHCP (udhcpc option host-name) / xCAT, and drop in a DHCP
netplan (optional, dhcp-identifier mac) so systemd-networkd configures the
boot NIC. Ubuntu-only.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Ubuntu live-server ISO copycds imports is not a complete apt pool, so
both Ubuntu install paths fail without an external mirror: the netboot
genimage's debootstrap from the ISO-only pkgdir cannot resolve packages,
and the Subiquity diskful install (Template.pm) fell back to the airgapped
file:///cdrom config. Default site.ubuntu_apt_mirror to
http://archive.ubuntu.com/ubuntu when unset (overridable for local/geo/
ports mirrors). In the Ubuntu genimage, when pkgdir has no http mirror,
build the debootstrap source and chroot sources.list from that mirror with
the codename derived from osvers and the release/-updates/-security pockets
(main + universe) -- the compute pkglist needs universe (busybox-static,
dracut*) and the security-updated openssh. Ubuntu-only; EL netboot
(netboot/rh) and copycds/packimage are untouched.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
When a node's install NIC is selected by MAC (installnic=mac), the generated
netplan was invalid: it named the ethernet device the literal string 'mac' and
set its MAC match to the full pipe-delimited list of all node MACs. The result
had no valid interface, so neither the installer environment nor the installed
system came up on the network. Match the install NIC by interface-name glob in
the installer, and in the target use only the primary MAC and omit set-name
when the install NIC is selected by MAC.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The generated Subiquity autoinstall config defined no identity section. Newer
Subiquity treats a config without an identity as incomplete and stops at an
interactive "Continue with autoinstall?" confirmation, so the install never
proceeds unattended (the prompt's advice to add 'autoinstall' to the kernel
command line is a red herring -- it was already present). Provide an identity
with a password sourced from the cluster passwd table so the install runs
non-interactively.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Ubuntu live-server install media is not a complete apt package mirror.
With the generated apt configuration pinned to the media only, an install that
needs a package absent from the media stalls instead of fetching it. Emit an
online-mirror apt configuration when one is configured (site.ubuntu_apt_mirror)
so missing packages are retrieved from the archive; offline behaviour is
unchanged when it is unset.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão