The unit tests drive xCAT::RespawnUtils, which is where the pacing lives, but nothing
connects that to the daemon. Replacing the respawn condition in xcatd's service loop with
"if (0)" -- so a dead install monitor is never re-forked -- leaves the whole suite green.
The behaviour the PR exists to deliver is unverified.
That gap cannot be closed in a unit test: xcatd needs the database, SSL, the plugin tree
and /var/run/xcat before it will start, which is why the pacing was extracted in the first
place. It belongs in xCAT-test, where there is a running daemon to kill things in. Add a
case that kills the install monitor and requires that a new one appears, that it reclaims
xcatiport rather than merely existing, and that the SSL listener keeps its pid throughout
-- surviving without a restart being the entire point.
The process titles are matched anchored. An unanchored "pgrep -f xcatd: install monitor"
also matches the running test's own command line, and the kill would then take out the
test; that was observed on a live MN, not guessed.
Two smaller test defects go with it. The fork test kept every pid it forked in @spawned
and had its END block signal all of them, including ones it had already reaped -- verified
as 3 of 3 -- so a recycled pid would take a signal meant for a process that no longer
exists, and the suite runs as root in CI. Reaped pids now leave the list. And the tunables
reach policy() straight from %ENV, where they can be empty or misspelt; assert that none
of those shapes produces a Perl warning, since xcatd runs under use warnings and would put
one in the daemon log on every start.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The pacing that keeps the install monitor alive is written inline in xcatd, and xcatd
cannot be run in a unit test: it needs the database, SSL, the plugin tree and
/var/run/xcat before it will start at all. So the test reached for the only thing left and
matched regular expressions against the script's source -- that a respawn branch exists,
that it mentions an interval, that it names a cap. Every one of those assertions passes
against pacing that is subtly wrong, and none of them would notice the retry budget
running out and never being refilled, which is the actual defect under review. Grepping
the implementation also pins its shape, so the code cannot be rearranged without editing
the test that is supposed to be guarding it.
State the pacing instead as an interface a test can execute: xCAT::RespawnUtils, pure
functions that take a state and a time and return the next state, with no clock, no
globals and no I/O of their own. Passing the time in is what lets the schedule be checked
over a virtual clock rather than in real seconds.
Drive it for the delay backing off to a ceiling and holding there, for the never-give-up
property (three hours into a continuous failure the daemon is still forking monitors), for
the reset (a monitor that stayed up long enough to serve clears the backoff when it later
dies), for the guards on a policy that could not back off, and for purity itself. Then
drive it for real against a genuinely held TCP port: fail several times, release the port,
and require that a respawned monitor binds it and stays up without the daemon being
restarted.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The respawn added for the install monitor is paced by a retry budget, and once that
budget is spent the daemon stops trying. That reintroduces the failure the respawn
exists to remove: with no monitor alive there is nothing left to reset the counter, so
xcatiport stays dead until the whole daemon is restarted, and a port that becomes free
a minute later is never picked back up. Pacing the retries is necessary -- an unguarded
re-fork spins as fast as fork allows while the port is held, and keeps re-entering
do_installm_service's USR2 socket-takeover handshake -- but pacing must not decay into
giving up.
The property that matters is therefore behavioural, not structural: the monitor comes
back on its own, at a bounded rate, no matter how long it has been failing. Assert it by
driving xcatd's real pacing code rather than grepping for it -- extract the marked
mon-respawn-policy region from the script verbatim, the way build_ubunturepo_lock.t
drives build-ubunturepo's real lock, and run it. Over a virtual clock, check that the
delay backs off to a ceiling and holds there, that the daemon is still forking monitors
three hours into a failure, and that a monitor which stayed up long enough to serve
resets the pacing when it later dies. Then do it for real against a genuinely held TCP
port: fail several times, release the port, and require that a respawned monitor binds
it and stays up without the daemon being restarted.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Re-fork the install monitor from the main service loop when $pid_MON has been cleared
and xcatiport is still configured, so a single death of that child no longer leaves
the port dead until the whole daemon is restarted. The forked child closes the SSL
listener and the UDP control socket before re-entering do_installm_service, so it
serves only the install monitor.
Rate limit the respawn. do_installm_service dies when it cannot bind the port after
its own retries, which is exactly the case where an unguarded re-fork would spin as
fast as fork allows and keep re-entering that function's USR2 socket-takeover
handshake against whatever still holds the socket. Consecutive attempts are separated
by XCATD_MON_RESPAWN_INTERVAL seconds (default 5) and capped at XCATD_MON_RESPAWN_MAX
(default 10), after which xcatd logs that it is giving up on the port rather than
retrying forever. A monitor that stayed up long enough to outlast the whole retry
budget resets the counter, so an unrelated death much later gets a full budget again.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install monitor -- the child listening on xcatiport for node install-status
updates and the "next" boot-flip request -- is forked exactly once at daemon startup.
When it dies the SIGCHLD reaper only clears $pid_MON and nothing re-forks it, so a
single death of that child (a stray signal, or a lost socket takeover during an xcatd
restart) leaves xcatiport permanently dead while the main daemon keeps running.
Installing nodes can then no longer report booted or request the boot flip until the
whole daemon is restarted, which is disruptive to any concurrent operation.
A respawn must also be rate limited. do_installm_service dies when it cannot bind the
port after its own retries, so an unguarded re-fork in the main loop would spin as
fast as fork allows for as long as the port stays held, and would collide with that
same function's USR2 socket-takeover handshake.
Assert that the main loop re-forks the monitor, that the child re-enters
do_installm_service, and that respawns are spaced, capped, and reported on exhaustion.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Cover the order that names the install device: installnic, then primarynic,
then mac.mac. Either attribute may name an interface or carry an address, and
the keyword mac returns to mac.mac.
Cover the device the kickstart names for each of those inputs. A node that
sets neither attribute keeps the address it has today. Cover the defect the
change closes, where a mac.mac entry that holds several untagged addresses
resolves to the last of them.
Cover that Ubuntu keeps its own pair of a name and an address over the same
resolution, and that the unique local address still comes from the hardware
address.
Pin the riscv64 %ifarch blocks in the xCAT, xCATsn and genesis specs
(genesis token, ipmitool-xcat, no x86 PXE loaders) and the riscv64
entries in buildcore.sh, buildlocal.sh and the buildrpms.pl notes.
Resolve the rocky10/rhels10 point releases through
imgutils::get_profile_def_filename with the OS search list emulated, and
assert the riscv64 files exist, win the lookup, match the x86_64 package
lists and point service images at the riscv64 dependency repository.
Pin the riscv64 installer kernel lookup in anaconda.pm and
geninitrd.pm, and evaluate the rh/genimage default driver and resolver
library blocks for riscv64 and the existing architectures.
Add an autotest nodeset case with bogus riscv64 grub2 nodes and a
rhels10.99 riscv64 osimage, checking the per-node grub2 files, the
grub2.riscv64 loader link and the hex-ip/mac links.
Drive mknb riscv64 --configfileonly through the existing stubs and
assert the grub2 network configuration: name, cpu guard, kernel, initrd,
xcatd endpoint, serial console, lzma preference, no PXELINUX/xNBA/
petitboot output, removal without artifacts and for :noboot interfaces.
Also install a riscv64 Genesis export and reject it for another
architecture.
Exercise _default_netboot() with the database-backed modules stubbed:
riscv64 defaults to grub2 and keeps any grub2 variant, the x86, PowerNV,
ppc, onie and aarch64 outcomes are unchanged, and undefined inputs do
not warn.
Assert the Kea xcat-riscv64 class (architecture 0x001b only, grub2
boot file, present with and without xNBA loaders) and pin the ISC
subnet block so the riscv64 branch stays ahead of the /yaboot
fallback that would otherwise shadow it.
Pin lookupNetboot, the profiled-node netboot rule table and the schema
descriptions for riscv64 without loading the database-backed modules:
the shipped subroutines are extracted from the source and evaluated
directly, so the test runs without DBI.
Daniel Hilst
Vinícius Ferrão