The comments around the install monitor respawn retell the failure, defend the
design and repeat the same causal chain in three places. Reduce them to the
facts that are not visible at the site: the ordering rules, why there is no
attempt limit, and what each fork site inherits. The rest is in the commit
messages and the PR.
Comment only. RespawnUtils.pm loses 26 lines and no code changes; xcatd loses
comment lines only. Both unit test files still pass.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The respawn is forked from the middle of the service loop, so the child
inherits @pendingconnections -- the client sockets the parent has accepted and
not yet handed to a worker. The monitor never serves one, and it outlives the
worker that does, so its copy keeps that client's socket open until the daemon
exits.
Close them in the child, next to the listener and the rescanplugins channel it
already drops.
xcatd_install_monitor.t runs the lifted respawn block against stand-in
descriptors and requires every pending connection to be closed. It fails
without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install monitor is forked while generic_reaper is the SIGCHLD handler.
ssl_reaper is only installed once the main service loop starts, and
generic_reaper comes back whenever connections are throttled.
Only ssl_reaper cleared $pid_MON. A death reaped by generic_reaper left
$pid_MON holding a dead pid, and the service loop re-forks only when $pid_MON
is clear, so xcatiport stayed dead for the life of the daemon.
Move that accounting into reap_install_monitor and call it from both reapers.
xcatd_install_monitor.t runs both reapers over a dead child and requires each
to clear $pid_MON and fold the death into the pacing. The generic_reaper case
fails without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
supervise() blocks SIGCHLD across the fork but unblocks it before returning, and the caller
installs the pid afterwards:
($mon_respawn, $pid_MON) = xCAT::RespawnUtils::supervise { ... } ...;
so the assignment is outside the blocked region -- the same unprotected window that existed
before e0b0ac6, moved from xcatd into the helper that was meant to make it impossible to get
wrong. ssl_reaper matches the dead child against $pid_MON and folds the death into
$mon_respawn; a monitor dying in that gap is compared against a pid still holding 0, missed,
and the caller then overwrites both with a pid that no longer exists. !$pid_MON never fires
again, so the respawn loop never runs and xcatiport stays dead until xcatd is restarted --
the failure this PR exists to remove.
Have supervise() install them itself, which is why `state` and `pid` are now passed by
reference: the pacing state is recorded and the pid assigned while SIGCHLD is still blocked,
and only then is it unblocked, so there is no point at which a reaper can run and see either
of them stale. Nothing is left for the caller to do afterwards, so both call sites become
plain statements that read $pid_MON when they need it. The child unblocks before running its
body, as it did when the unblock sat ahead of the fork's branch. The new pid is returned as
well, for a caller that wants it inline.
Verified on a live MN (xcat54-mn, AlmaLinux 10.2, xCAT 2.19.0): the startup fork produces a
monitor holding xcatiport 3002; killing it is recovered in 5s, killing the replacement at
once in 11s -- the backoff -- and killing one that had served past the healthy interval is
recovered in 1s, with the port reclaimed and xcatd active throughout. The unit test's window
subtest, red in the preceding commit, now passes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both places that fork the install monitor repeat the same careful sequence: record the
attempt, block SIGCHLD, fork, unblock, and on failure record the exit so the next attempt
backs off. Two of those steps are ordering requirements rather than steps -- the attempt
must be recorded before the fork, because the child can die and be reaped before fork()
returns, and SIGCHLD must be blocked across the fork and the assignment, or the reaper
compares the dead child against a stale pid and misses it. Neither is apparent from
reading the code, and both were got wrong at least once while writing it. Leaving them
open-coded means the next caller -- $pid_UDP has the same never-respawned shape -- gets to
rediscover them.
Move the sequence into xCAT::RespawnUtils::supervise(), which takes the child body as a
block and the rest as named arguments:
($mon_respawn, $pid_MON) = supervise {
...the child...
} state => $mon_respawn, pid => $pid_MON, now => time();
The (&@) prototype is what allows the leading block, and it applies to a fully qualified
call, so no Exporter machinery is needed. It does require the module to be loaded with
`use` rather than `require`: under `require` the sub is unknown when the call is compiled,
the block is then read as a bare block and its value arrives as the first argument, which
fails at runtime rather than at compile time. Both call sites and the test use `use`, and
the constraint is written down next to the sub. Passing a live pid is a no-op, so a caller
that forgets to check does not end up with two children.
The module gains its first impure function, which is why it sits under its own heading with
the pure ones stated to be pure above it: those return new state and touch nothing, which is
what keeps them testable on a made-up clock and safe inside a signal handler. supervise()
forks, so it is tested by the fork-and-port case instead, which now drives it rather than
its own copy of the same sequence. POSIX and xCAT::Utils are required inside supervise()
rather than at the top, so loading the module for the pure functions still pulls in nothing.
xcatd loses $mon_chldmask and its :signal_h import along with the duplication.
Verified on a live MN: the startup fork goes through supervise() and produces a monitor
holding xcatiport, and two consecutive kills are recovered in 5s then 10s -- the backoff --
with the port reclaimed and the SSL listener holding its pid throughout.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT::RespawnUtils::policy() was called near the top of the file, some twenty lines above
the "use xCAT::RespawnUtils" that loads the module. It works, because use is compile-time
and perl compiles the whole file before running any of it, so the import has already
happened by the time that statement executes. But nothing at the call site says so. It
reads as a plain ordering mistake, and it stops working the moment someone converts the
import to require -- a routine thing to do to a daemon that loads this many modules -- with
the failure being an undefined subroutine at startup.
Move the declaration below the imports, next to the osver() call that already makes a
runtime call to a use'd module there. The only constraint on where it can go is that
ssl_reaper closes over $mon_respawn and so must be compiled after it is declared; the new
position clears that by a thousand lines, and compiling under strict is what proves it,
since a lexical declared after the sub would fail to compile rather than silently bind
elsewhere.
Pure relocation: the moved block is byte-identical and no behaviour changes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The respawn forks from the main service loop, much further down the program than the fork
at startup, so it inherits everything the parent has opened in between. That is the
rescanplugins socketpair from further up this file -- the channel a subcommand process uses
to hand a reloaded cmd_handlers hash back to the parent. The child closes the SSL listener
and the UDP control socket but not those two, so a respawned monitor holds both ends of a
channel it never reads or writes, for as long as it lives.
Measured on a live MN by diffing /proc/<pid>/fd between a monitor forked at startup and one
respawned after being killed: the respawned process carried one extra socket, and both ends
of that pair were also held by the SSL listener parent. The leak is two descriptors and it
does not accumulate, since each respawn forks afresh from the parent; the reason to fix it
is that the block is commented "serve only the install monitor" and no longer did, so a
monitor's file descriptors depended on whether it was the first one or a replacement. That
is the kind of difference that makes a later problem reproduce only on one path.
Close both ends in the respawn child. The monitor's own plugin-rescan channel is a
different socketpair, created before either fork, and is untouched. Verified afterwards on
the same MN: the respawned monitor no longer shares a socketpair with the parent, and still
binds xcatiport and serves it, with the SSL listener holding its pid throughout.
Not covered by a test. Both the unit suite and the xCAT-test case format work at the level
of processes and ports; this is an invariant about file descriptors that needs /proc on a
running daemon, and asserting it there would be more fragile than the line it guards.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Three defects found by running the respawn against a live xcatd on an MN rather than only
against its unit tests.
A monitor whose child dies between xfork() returning and the assignment to $pid_MON is
lost for good. ssl_reaper matches $CHILDPID against $pid_MON, so a child reaped in that
window is compared against a stale value and missed, and $pid_MON is then left naming a
pid that no longer exists. The service loop reads !$pid_MON to decide whether to respawn,
so it never respawns again -- the same permanently dead xcatiport this whole change exists
to prevent, reached by a different route. Block SIGCHLD across the fork and the assignment
at both fork sites; the child unblocks on the same line, since it needs to reap its own
children. Reproduced with a widened window before the fix and confirmed closed after.
Recovery took 30 seconds on an idle daemon. The respawn only gets a turn when the service
loop comes round, and the loop parks in $bothwatcher->can_read(30) when there is nothing
to serve, so the full select timeout was being added to the respawn delay. Wait in 5s hops
while the monitor is down and at the usual 30s otherwise, so an idle daemon pays a few
extra wakeups only while xcatiport is actually dead. Measured on the MN afterwards: a
killed monitor returns in 5s, then 10s, then 21s across three kills in a row -- the
backoff, visible in wall-clock time -- reclaiming the port each time, with the SSL listener
holding the same pid throughout.
The tunables are read from %ENV and were compared before being validated, so an empty or
misspelt XCATD_MON_RESPAWN_* put "Argument isn't numeric" in the daemon log at every start.
Anything that is not a plain non-negative integer is now treated as unset.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The backoff that decides when to re-fork the install monitor is arithmetic over a handful
of counters, but it lives inline in xcatd among the daemon's globals, its signal handlers
and its fork. xcatd needs the database, SSL, the plugin tree and /var/run/xcat before it
will run, so nothing in a unit test can execute that arithmetic; a test can only match
patterns against the script's source and hope the shape it finds behaves. That is how a
retry budget which ran out and could never be refilled passed a green test run.
Move the pacing to xCAT::RespawnUtils as pure functions: each takes the current state and
the current time and returns the next state, reading no clock, no globals and no files.
Passing the time in is what makes the schedule checkable over a virtual clock instead of
in real seconds, and returning a new state rather than mutating one is what makes it safe
to call from the SIGCHLD handler -- the result is built before the caller installs it, so
a signal arriving partway through cannot leave the pacing half-updated.
The behaviour is unchanged from the previous commit and stays covered by
xCAT-test/unit/xcatd_monitor_respawn.t, which now executes these functions instead of
grepping for them: the delay doubles from XCATD_MON_RESPAWN_MIN_INTERVAL (5s) to
XCATD_MON_RESPAWN_MAX_INTERVAL (300s) and holds there without ever refusing a retry, and a
monitor that stayed up XCATD_MON_RESPAWN_HEALTHY seconds (60s) resets the backoff when it
later dies. policy() now also refuses a floor below one second, which would double to
itself and give a fork storm rather than a backoff, and a ceiling under the floor.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The respawn of the install monitor was paced by a retry budget that, once spent, made
the daemon stop trying for good. That put xcatiport back in the state the respawn was
added to fix: with no monitor alive there is nothing left to reset the counter, so the
port stays dead until the whole daemon is restarted, and a port that frees up a minute
later is never picked back up. It only reached that state more slowly than before.
Pacing itself is needed. do_installm_service dies when it cannot bind the port, so an
unguarded re-fork spins as fast as fork allows while something else holds it, and keeps
re-entering that function's USR2 socket-takeover handshake. Replace the budget with an
exponential backoff that has a ceiling but no end: the delay doubles from
XCATD_MON_RESPAWN_MIN_INTERVAL (default 5s) to XCATD_MON_RESPAWN_MAX_INTERVAL (default
300s) and stays there. A monitor that cannot start therefore costs one fork per five
minutes for as long as that lasts, and is back within five minutes of the port becoming
free, with no restart and no operator action.
A monitor that ran for XCATD_MON_RESPAWN_HEALTHY seconds (default 60) plainly got the
socket and served, so its eventual death resets the delay: an isolated death is retried
at once and the backoff only builds up during a real streak of failures to start. The
ceiling is reported once per streak rather than on every attempt, and says that xcatd is
still retrying instead of that it has stopped.
The pacing lives in a marked mon-respawn-policy region, free of forking and of daemon
state, so xCAT-test/unit/xcatd_monitor_respawn.t drives the real code rather than a copy
of it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Re-fork the install monitor from the main service loop when $pid_MON has been cleared
and xcatiport is still configured, so a single death of that child no longer leaves
the port dead until the whole daemon is restarted. The forked child closes the SSL
listener and the UDP control socket before re-entering do_installm_service, so it
serves only the install monitor.
Rate limit the respawn. do_installm_service dies when it cannot bind the port after
its own retries, which is exactly the case where an unguarded re-fork would spin as
fast as fork allows and keep re-entering that function's USR2 socket-takeover
handshake against whatever still holds the socket. Consecutive attempts are separated
by XCATD_MON_RESPAWN_INTERVAL seconds (default 5) and capped at XCATD_MON_RESPAWN_MAX
(default 10), after which xcatd logs that it is giving up on the port rather than
retrying forever. A monitor that stayed up long enough to outlast the whole retry
budget resets the counter, so an unrelated death much later gets a full budget again.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Own collection, sensitivity, finalization, and reset as one request-scoped state object so xcatd only forwards callbacks and appends the finalized text.
The commands.log response classifier used a "passw" text match on the
request arguments. A secret whose name has no such text passed the
check, so a read of an authentication key, a privacy key or the snmpc
site value logged its bare value in the response. A command that
expands an argument also passed the check: nodels with a table name
returns every column of the table, and lsdef returns attributes that
the request never names. The daemon also ran redact_password over the
whole connection log on each request, so the redactor split at the
first request of the connection and the change signal swept the text of
earlier requests and responses.
Add secret_in_request. The routine reports a request that names a
secret attribute, selects a secret site key, or dumps a table that owns
a secret column through tabdump or nodels, from the same secret set
that the argument redaction uses. The response classifier calls it, so
the response of such a request logs as redacted.
Add secret_in_response. The routine reports response text that holds
"passw" or a secret attribute name in assignment or column form. The
response finalizer calls it in place of the bare text match, so an
expanded listing that carries an authentication key or a product key
logs as redacted even when the request never names it. The lsvm
response is the directory entry, whose passwords are positional, so
the classifier marks the command itself.
Build each request segment alone, redact the segment, and then append
it to the connection log. The redactor now always sees the current
command, and the change signal covers only the current request.
The daemon redacted secret attributes on the joined command string. The
match failed when a value held a space. The match also failed for a "+="
splice assignment. The validate() path did not quote the arguments, so a
multi-word secret value kept its later words in syslog and in the
auditlog table. A password that a command receives through an option or
a positional operand was not redacted at all. The debug dispatch trace
wrote the raw arguments to syslog when site.xcatdebugmode was set.
Redact the argument vector before the daemon joins it. Add
redact_password_args for this task. The routine masks the value of a
secret attribute in any argument, at the start or embedded after another
token. An embedded secret assignment masks to the end of the argument,
because a shell value may hold quotes and spaces. The routine allows
spaces around the operator. It accepts the "=", "+=", ",=", "^=", "!=",
"=~" and "!~" operators that chdef, nodech and node selection use. It
masks a password option value in each form that Getopt::Long accepts: a
separate argument, a compact short option, a bundle of short options
with the "?" help letter, a "+" option prefix, a single-letter option
with two dashes, a long option, a long option with an equals sign, and
an abbreviated long option. The long-name match runs first, so a long
option keeps its name and masks its value. A walk over each bundle then
finds the first secret letter, so the mask always starts at the option
and the result does not depend on hash order. The walk knows which
other letters of a command take a value, so a secret letter inside such
a value does not redact and the audit text stays correct. The walk also
knows which letters take an integer, because the z/VM cpu option
consumes only its signed digits and the parser then continues the
bundle into the password option. The value stops match letter case,
because a bundle keeps short options case sensitive and an unknown
capital letter does not absorb the rest. The mkvm secret match ignores
letter case, because the z/VM parser keeps the Getopt::Long default for
long names. The mkhwconn match keeps letter case, because -p
is the hardware control point and -P is the password. The routine knows
the password options of bmcdiscover, switchdiscover, mkhwconn, mkvm,
createvcluster, lsvcluster and rmvcluster, the rspconfig password
assignments, the mkvm clone pw= operand, and the positional password
operands of chvm. It masks the site.value argument of tabch and chtab
when a selector or a site.key assignment names snmpc. An exact short
option that takes a non-secret value stays visible, so the PPC mkvm -p
profile is not an abbreviation of --password. The dispatch trace builds
its text from the redacted vector.
Add snmpc, productkey, prodkey.key, tokenid and token.tokenid to the
secret list, with community and pdu.community. The secret list holds
only attributes that map to a secret column, so key and sshkeydir stay
visible.
redact_password keeps a second pass over the joined string. This pass
masks an embedded secret assignment to the end of the line, because the
argument boundaries are gone after the join.
The commands.log response classifier marks a response sensitive when the
request was redacted. The argument vector pass sets that signal, so a
secret whose name has no "passw" text still marks its response.
xcatd redacts the request in commands.log but appends the command response
verbatim. A command whose output holds a secret writes it in clear text.
Examples are tabdump passwd, gettab of a passwd column, and getcredentials.
Collect the response into a per-command buffer. Set a sensitive flag when the
command is getcredentials, an argument names a password, or the request was
redacted. When the command finishes, replace the whole buffer if the flag is
set or the buffer still holds password content, then append the buffer. A
connection can carry more than one command, so the buffer is finalized at the
next command's start and at the end of the connection.
The buffer holds the full response, so a secret split across several callbacks
is also redacted. A per-callback check cannot do this.
The word-content check is a fallback. The request classification is the main
signal. A secret with no password marker, such as the output of an xdsh cat of
a shadow file, is a pre-existing leak of the root-only log. It is out of scope.
Recovered from the lenovobuild branch. Reimplemented against master.
xcatd expands the request noderange before it authorizes the caller: once to
count the nodes, and once in validate() to match the policy rules. The ^
operator makes xcatd open a caller-named file at that point. A client can
connect without a certificate, because the listener does not require one, and
such a client has no peername.
Expand these two pre-authorization noderanges with nofile when the caller has
no peername (checked with defined, so the identity "0" still counts as
authenticated). If validate() finds a rejected ^file atom on such a request,
deny it. An authenticated caller expands ^file as before.
xcatd warns "xCAT Version mismatch!" when a node's xCAT version differs
from the server's. It compared the full version strings, which include a
build-specific suffix such as " (git commit <hash>)". Two nodes at the
same release built from different snapshots then reported a version
mismatch on every request, even though the same release is ABI
compatible.
Keep warning when the versions differ, but tell the two cases apart. A
different release is still "xCAT Version mismatch!". The same release
built from a different commit now reports "xCAT build level differs (same
release):" instead, so the build difference is still visible without
being called a mismatch. Both messages show the full version strings.
Add xCAT::Version->Release, which returns the version without the
build-specific suffix, to make that distinction.
This was recovered from the lenovobuild branch, which stripped the older
"built <date>" suffix and dropped the same-release warning entirely; this
reimplements it for the current version format and keeps the build
difference visible.
Docker container lifecycle management (mgt=docker, mkdocker, rmdocker,
lsdocker) was added in 2015-2016 as an experiment targeting Docker API
v1.22 on Ubuntu only. Documentation and man pages were deliberately
removed in 2019 (PRs #6222 and #6324) with the original developer's
approval, noting that "the interface of Docker has become very simple
right now, so there is no value for xCAT to offer such functions."
The plugin was still being shipped but has had no functional code changes
since April 2016, was never listed as a valid mgt value in Schema.pm,
and no user ever filed an issue about it.
Removed:
- xCAT-server/lib/xcat/plugins/docker.pm (1,142 lines)
- xCAT/postscripts/setupdockerhost
- xCAT-server/share/xcat/scripts/setup-dockerhost-cert.sh
- xCAT-test/autotest/testcase/dockercommand/ (test cases)
- Docker attribute definitions in Schema.pm
- Client symlinks (mkdocker, rmdocker, lsdocker)
- Usage entries and dockerhost cert handling in credentials.pm
- Docker attribute documentation in man7 pages
The "Running xCAT in Docker" documentation (dockerized_xcat/) is
retained as it documents containerizing xCAT itself, not the removed
mgt=docker feature.
Closes#7518
* Fix silent failure when site.master is not set (#6157)
Hardware control commands (rpower, rinv, etc.) silently return no output
and exit 0 when site.master is empty. The original fix (#6074) was
reverted (#6158) because it warned per-node with the wrong hostname.
Check once in plugin_command before dispatching to plugins, so the error
appears exactly once with the correct command name.
* Also reject empty site.master, not only undef
The else branch handles a rare case where XS libraries (Sys::Virt,
Net::SNMP) crash without setting $@. This IS a plugin bug, so keep
that label and the debug hint. Only the common case (die with $@)
gets the clean passthrough.
When a plugin dies during request processing, xcatd wrapped the error
in a misleading "plugin bug" message that hid the real cause (e.g.
"No space left on device"). Now passes through the actual error from
the eval, making the output useful for any failure, not just disk full.
Fixes#2719
This commit adds an early return to xcatconfig settunnables function.
This function set parameters at
/proc/sys/net/ipv4/neigh/default/gc_thresh1
/proc/sys/net/ipv4/neigh/default/gc_thresh2
/proc/sys/net/ipv4/neigh/default/gc_thresh3
And set sysctl attributes by writing to /etc/sysctl.d/ and
/etc/sysctl.conf
These are tunning network parameters for running on production
and should not affect the overall function for testing purposes.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
On Debian-based systems, /etc/sysctl.conf doesn't exist and
/etc/sysctl.d/ directory is used instead. Modified xcatconfig
to prefer /etc/sysctl.d/99-xcat.conf when the directory exists,
falling back to /etc/sysctl.conf for backward compatibility.
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão
Markus Hilger
Kilian Cavalotti
Markus Hilger
Mark Gurevich
besawn
Kurt H Maier