A compute node booted into legacy Genesis on Ubuntu refuses every ssh connection:
[xcat100-mn]: xcat100-cn: ssh: connect to host xcat100-cn port 22: Connection refused
and on the node's own console, once per boot:
Missing privilege separation directory
Debian and Ubuntu build sshd with --with-privsep-path=/run/sshd. The payload's own sshd
carries that path and the format string "Missing privilege separation directory: %s", and the
image ships /run holding only rpcbind. So sshd exits at startup and the node is unreachable.
nodeset_shell, nodeset_cmdline, nodeset_runimg and genesis_boot_proof_legacy all fail on it,
while seq_discovery, which needs no ssh, passes.
doxcat already does this for the other family: it creates /usr/share/empty.sshd, the EL
privsep path, with the comment "On some systems /usr/share/empty.sshd needs to be created
before calling sshd". The Debian path was never added, although the tree knows about it twice
-- remoteshell says "prepare the /var/run/sshd for ubuntu" and etc/init.d/functions creates
it. This adds the same lines for /run/sshd and /var/run/sshd.
Created immediately before sshd starts, not baked into the image: /run is a tmpfs on a
running system, so a directory under it does not survive the mount. The mode is set whether
or not the directory had to be created, because sshd refuses a group or world writable
privsep directory with the SAME message as an absent one, so a wrong mode is
indistinguishable from this bug.
genesis_sshd_privsep.bats asserts what sshd imposes -- the path is a directory, owned by the
user that will run sshd, and not group or world writable -- read at the moment sshd runs. The
recorder is bound over /usr/sbin/sshd because doxcat calls that absolute path, and because
recording from a later command cannot see the ORDER: with the recorder on a PATH command
instead, moving the directories after the sshd line still passed.
Five mutations, each red: removing the loop fails 2, never setting the mode fails 1, dropping
/var/run/sshd fails 2, a group-writable mode fails 2, and creating the directories after sshd
starts fails 2. The whole bats suite goes from 21 failures to 19 with this change, the
difference being these two tests; the other 19 are unrelated and fail on the base.
doxcat_sandbox.bash gains chmod, and a third test keeps the next one from being missed. The
sandbox curates PATH without /usr/bin, so a command it does not copy resolves to nothing and
fails SILENTLY: doxcat's chmod did nothing under test for as long as this sandbox has
existed, and any assertion about a mode could never pass. The guard installs bash's
command_not_found_handle through BASH_ENV and fails naming whatever doxcat invoked and could
not resolve, so it needs no list to keep current. With chmod removed again it reports exactly
"chmod". Its reach is the commands doxcat runs on the path the test drives, which is a floor
rather than a completeness check on the helper's list.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A compute node that boots the legacy Genesis image on el10 x86_64 never acquires an
address. Its serial console reports "/usr/bin/doxcat: line 293: dhclient: command not
found" and then "It seems to be taking a while to acquire an IPv4 address". The DHCP
server side is sound: Kea leases the address and the node never asks for it.
AlmaLinux 10 and EPEL 10 package no ISC dhcp-client. dracut_install reports a missing
binary and returns, so module-setup.sh named dhclient, the build kept going and the image
shipped without a client. doxcat then named dhclient at six call sites with no
alternative.
doxcat now chooses its client at run time. genesis_dhcp_command() returns the command
line for one interface and one address family, and genesis_start_dhcp() runs it and
reports when the image carries none. The ISC client keeps its command line where a
release packages it, so el8 and el9 are unchanged. Where it is absent, dhcpcd stands in:
AlmaLinux 10 baseos packages it at 236 KB, and it carries its own resolv.conf, hostname
and ntp hooks, so it needs no dhclient-script. dhcpcd on a single interface exits when
its 30 second timeout expires and de-configures the interface as it goes, so it is asked
for -t 0 and -p.
The dracut module installs whichever client the build root carries, together with the
hooks dhcpcd runs on every lease. The spec build-requires dhcpcd from rhel 10 on, and
verify-genesis-payload now requires usr/sbin/dhcpcd there, so an image that ships with no
client fails the build instead of reaching a node.
genesis_dhcp_client.t drives both routines with the clients shadowed by recording stubs.
It fails on the parent commit, and deleting the dhcpcd branch turns five of its
assertions red.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
(cherry picked from commit b9329998c80383bb0397d16a558f66a6bb9db399)
Legacy Genesis excludes every secondary interface with IFF_UP, even when firmware only raised it and no address is configured. Let eligible interfaces join the existing secondary-interface DHCP path when they are physical, not enslaved, unaddressed, and not owned by the TSM or IMM management paths. Existing DOWN interfaces keep their prior path.
Recovered from the unmerged lenovobuild branch (original 9a1381f6), adapted to leave preconfigured and management-owned interfaces untouched.
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
When a node boots from an InfiniBand interface, doxcat could not match BOOTIF
to the netdev, so boot-nic detection failed. Load ib_ipoib and, if the normal
Ethernet lookup finds nothing, fall back to matching the InfiniBand device by
its (shorter) IPoIB address.
modprobe ib_ipoib is harmless where there is no IB hardware (the module simply
loads no devices), and the IB lookup only runs when the Ethernet lookup fails,
so non-IB nodes are unaffected.
Not lab-validated (no InfiniBand fabric available; the SR635's Mellanox card is
in Ethernet mode).
Recovered from the unmerged lenovobuild branch (aeeebd5f).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
For each secondary NIC in NICSTOBRINGUP, doxcat backgrounds a loop that polls
`ethtool $nic` until the link reports "detected: yes" before running dhclient.
A NIC that is never connected keeps that loop (and its `sleep 5`) running for
the life of the genesis environment, leaking background shells and needlessly
churning.
Add a second exit condition so each loop also stops once /tmp/netinitted
exists, and touch that file once the boot NIC has completed network init. A
link that never appears no longer waits indefinitely.
Recovered from the unmerged lenovobuild branch (original 9ece12e4).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
* Add ondiscover support for chain table
* For ondiscover, run nextdestiny after the current operation is done
* Don't set OpenPower server boot from network if boot
RH7/Centos7 and recent versions of fedora relocate `/*bin/*` into `/usr/*bin/`. This causes
conflicts for upgrades against xCAT-genesis-scripts-* packages which expect the `/bin` a to be
a directory while `xCAT-genesis-base-*` packages provide a link. Relocating all files into `/usr`
fixes that conflict and allows a clean upgrade from all old versions.