Three defects found by running the respawn against a live xcatd on an MN rather than only
against its unit tests.
A monitor whose child dies between xfork() returning and the assignment to $pid_MON is
lost for good. ssl_reaper matches $CHILDPID against $pid_MON, so a child reaped in that
window is compared against a stale value and missed, and $pid_MON is then left naming a
pid that no longer exists. The service loop reads !$pid_MON to decide whether to respawn,
so it never respawns again -- the same permanently dead xcatiport this whole change exists
to prevent, reached by a different route. Block SIGCHLD across the fork and the assignment
at both fork sites; the child unblocks on the same line, since it needs to reap its own
children. Reproduced with a widened window before the fix and confirmed closed after.
Recovery took 30 seconds on an idle daemon. The respawn only gets a turn when the service
loop comes round, and the loop parks in $bothwatcher->can_read(30) when there is nothing
to serve, so the full select timeout was being added to the respawn delay. Wait in 5s hops
while the monitor is down and at the usual 30s otherwise, so an idle daemon pays a few
extra wakeups only while xcatiport is actually dead. Measured on the MN afterwards: a
killed monitor returns in 5s, then 10s, then 21s across three kills in a row -- the
backoff, visible in wall-clock time -- reclaiming the port each time, with the SSL listener
holding the same pid throughout.
The tunables are read from %ENV and were compared before being validated, so an empty or
misspelt XCATD_MON_RESPAWN_* put "Argument isn't numeric" in the daemon log at every start.
Anything that is not a plain non-negative integer is now treated as unset.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The backoff that decides when to re-fork the install monitor is arithmetic over a handful
of counters, but it lives inline in xcatd among the daemon's globals, its signal handlers
and its fork. xcatd needs the database, SSL, the plugin tree and /var/run/xcat before it
will run, so nothing in a unit test can execute that arithmetic; a test can only match
patterns against the script's source and hope the shape it finds behaves. That is how a
retry budget which ran out and could never be refilled passed a green test run.
Move the pacing to xCAT::RespawnUtils as pure functions: each takes the current state and
the current time and returns the next state, reading no clock, no globals and no files.
Passing the time in is what makes the schedule checkable over a virtual clock instead of
in real seconds, and returning a new state rather than mutating one is what makes it safe
to call from the SIGCHLD handler -- the result is built before the caller installs it, so
a signal arriving partway through cannot leave the pacing half-updated.
The behaviour is unchanged from the previous commit and stays covered by
xCAT-test/unit/xcatd_monitor_respawn.t, which now executes these functions instead of
grepping for them: the delay doubles from XCATD_MON_RESPAWN_MIN_INTERVAL (5s) to
XCATD_MON_RESPAWN_MAX_INTERVAL (300s) and holds there without ever refusing a retry, and a
monitor that stayed up XCATD_MON_RESPAWN_HEALTHY seconds (60s) resets the backoff when it
later dies. policy() now also refuses a floor below one second, which would double to
itself and give a fork storm rather than a backoff, and a ceiling under the floor.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Firmware configured for UEFI HTTP boot sends client architecture 28
(0x001c) and only accepts an offer whose boot file is a URL and whose
reply carries the HTTPClient vendor class; it ignores the PXE offer that
answers architecture 27. riscv64 nodes therefore could not boot at all
from firmware set up that way.
Answer them with the same grub2 image over HTTP. The URL has to name the
management node address of the network the request came in on, so the
class belongs to the subnet, like the POWER OPAL and xNBA network classes
next to it, and it is only offered while the boot loader is actually
published. The ISC backend renders the matching subnet branch.
This covers every client without a reservation, which is the discovery
case. A node that nodeset has configured keeps its per-node boot loader
over TFTP, as on the other architectures.
The option flags of a client class -- always-send and its siblings -- were
passed through as whatever the caller set, so a plain Perl 1 reached the
configuration as the number 1 and Kea refuses to parse that. The class
flag next to it is already normalised; do the same for the option data, so
callers can stay free of JSON.
RISC-V 64-bit UEFI firmware identifies itself with DHCP option 93
client-system-architecture 27 (0x001b, IANA processor architecture
types). Neither DHCP backend knew the value: Kea handed such clients no
boot file and ISC dhcpd fell through to the /yaboot catch-all.
Add an xcat-riscv64 Kea client class and an ISC subnet branch that send
them boot/grub2/grub2.riscv64, the same shape as the aarch64 entries.
The UEFI HTTP boot id (0x001c) is left alone: it needs a URL boot file
and the HTTPClient vendor class, which is a separate change.
riscv64 nodes boot through UEFI and grub2 only, like ppc64le PowerKVM
guests and aarch64. Teach the three places that declare which
noderes.netboot methods an architecture accepts about it:
- xCAT::Utils::lookupNetboot returns grub2,grub2-tftp,grub2-http for
riscv64 so nodeset and rinstall accept those methods.
- the profiled-node rule table in xCAT::ProfiledNodeUtils defaults
riscv64 nodes to grub2.
- the nodetype.arch, osimage.osarch and noderes.netboot schema
descriptions document riscv64.
The arch token is riscv64 everywhere (rpm, dpkg and uname -m agree), so
no alias handling is added.
Add a new pdutype (genpdu) for PDUs implementing the Raritan PDU2-MIB.
A single MIB covers the Raritan PX2/PX3/PX4/PXC/SRC/PXO/BCM series, the
Server Technology PRO3X/PRO4X series, the Legrand intelligent PDUs, and
all PDUs following the PDU2-MIB.
Supports rpower (whole-PDU and per-outlet), rinv and rvitals over SNMP
v1, v2c or v3, with credentials read from the pdu table).
Unlike the existing types, sensor units and decimal precision are read
from the MIB per sensor rather than hardcoded, so readings are correct
across models that report differing precision for the same sensor.
Outlet switching capability is probed at connect time, so metered-only
models report a single "unsupported" message instead of a per-outlet error.
rspconfig is not supported for genpdu; PDU linking and BCM2/PMC power
meters are out of scope.
Tested on Raritan PX4-5851-E7V2 (fw 4.2.10.5-50400, switched),
PX3-1901U-N1 and PX3-1901U-N1A6 (fw 4.0.20.5-49038, metered), and
PX2-1901U-N1A6 (fw 4.0.20.5-49038, metered).
Signed-off-by: Kilian Cavalotti <kilian@stanford.edu>
The ^ operator in a noderange names a file, and noderange() reads it.
Add a nofile option. When nofile is set, noderange() does not open the file. It
records that a ^file atom was present, so a caller can fail closed even when the
range also holds plain nodes. file_operator_rejected() reports this.
noderange() also expands site.excludenodes through a nested call. That range is
trusted site data. Expand it without nofile and keep the request's rejection
state across the call. A configured exclusion cannot clear the flag, and a
^file in site.excludenodes cannot set it.
The default behavior does not change. Forward the option through extnoderange().
xcatd warns "xCAT Version mismatch!" when a node's xCAT version differs
from the server's. It compared the full version strings, which include a
build-specific suffix such as " (git commit <hash>)". Two nodes at the
same release built from different snapshots then reported a version
mismatch on every request, even though the same release is ABI
compatible.
Keep warning when the versions differ, but tell the two cases apart. A
different release is still "xCAT Version mismatch!". The same release
built from a different commit now reports "xCAT build level differs (same
release):" instead, so the build difference is still visible without
being called a mismatch. Both messages show the full version strings.
Add xCAT::Version->Release, which returns the version without the
build-specific suffix, to make that distinction.
This was recovered from the lenovobuild branch, which stripped the older
"built <date>" suffix and dropped the same-release warning entirely; this
reimplements it for the current version format and keeps the build
difference visible.
The ^ noderange operator reads node names from a file. NodeRange opened
that file with a two-argument open. A two-argument open reads shell
metacharacters in the path, so a noderange such as ^"id|" ran a command.
xcatd expands a noderange while it processes a request, so the command
ran on the management node.
Use a three-argument open with an explicit read mode. The value is then
only ever a file name. The ^ operator keeps working: ^/tmp/nodes still
reads the file.
This fix was recovered from the lenovobuild branch. The original there
(commit for "Remove load from file in noderange support") removed the ^
operator. This keeps the documented operator and closes the command path
instead.
Boot URLs handed to nodes always carried an explicit :80 even when the HTTP
server listens on the default port. grub2.pm already omits the port when it is
80; make the xNBA, genesis and DHCP boot URLs consistent with it.
$portsuffix is empty when site.httpport is unset or 80 and ":<port>" otherwise,
so a non-default port is unchanged. Covers the ISC and Kea paths in dhcp.pm,
xnba.pm, mknb.pm, and the Kea xNBA client classes in BootPolicy.pm.
dhcp_boot_policy.t and mknb_xcatd_address.t follow the new URLs and gain cases
for a non-default and an unset port. The floating-address guard in
mknb_xcatd_address.t also drops :80 so that it keeps matching.
Recovered from the unmerged lenovobuild branch (cd61fd9d, a2f77ea9, 1035841b,
b7db0e2, 792fe6d).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
Cumulus switch MAC discovery matched the fdb port with a regex that only
accepted a plain numeric swp name (dev swp([0-9]+)), then guessed among
swp5/05/swp05 formats. A node on a breakout port (swp1s0, swp1s1, ...) never
matched, so it was silently not discovered. Match any swp name (dev (swp[^ ]+))
and use it directly, dropping the format-guessing.
Gated: this whole block runs only for switchtype eq 'onie' (Cumulus/ONIE
switches); SNMP switches use a separate path and are unaffected.
Validated against real 'bridge fdb show' output (the stock iproute2 command
Cumulus runs over SSH): the old regex parses only 'dev swp1' and drops
'dev swp1s0'; the new regex parses both. Confirmed the environment on a booted
NVIDIA Cumulus VX 5.10.
Recovered from the unmerged lenovobuild branch (72d68bc7).
Recover several man-page clarifications that never merged from the lenovobuild
branch. All are documentation-only -- the man .pod sources with their
generated .rst, plus the networks-table description in Schema.pm:
* makedhcp / nodeset: note that UEFI boot directives are intentionally
omitted when nodetype.os names an OS without UEFI support, even for
'nodeset shell'.
* networks table dhcpserver: warn that it should be set only for service
node operation and that <xcatmaster> is not a valid value.
* noderange: clarify that the syntax is for xCAT.
* reventlog: note that unimplemented sensor types yield 'No Mappings Found'
and point to confluent's nodeeventlog facility.
* rcons: document confluent console auto-reconnect (a randomized 2-4 minute
retry, or 'ctrl-e o a' to force one); the original's stray backtick markup
is corrected.
Recovered from the unmerged lenovobuild branch (originals 2d4f12f4, 760f2f8b,
23d63d59, 44b47c44, c63d780f, db4dc42b).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
formatNetmask() uses its first argument immediately (inet_aton($mask),
2**$mask, hex $mask) with no check that it is defined. Callers that pass an
undefined mask trigger "Use of uninitialized value" warnings and a
meaningless result instead of a clean failure.
Return undef up front when $mask is not defined.
Recovered from the unmerged lenovobuild branch (original acbbeb86).
Co-authored-by: Jarrod Johnson <10814490+jjohnson42@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão
Kilian Cavalotti