Extracts the three command-log response subs and drives them. It checks the
request classification, the redaction of a bare passwd value, a secret split
across callbacks in either order, the password-content fallback, that a benign
response is kept, and that the finalizer preserves an earlier command's response
on a shared connection. It fails hard if a sub cannot be extracted.
xcatd redacts the request in commands.log but appends the command response
verbatim. A command whose output holds a secret writes it in clear text.
Examples are tabdump passwd, gettab of a passwd column, and getcredentials.
Collect the response into a per-command buffer. Set a sensitive flag when the
command is getcredentials, an argument names a password, or the request was
redacted. When the command finishes, replace the whole buffer if the flag is
set or the buffer still holds password content, then append the buffer. A
connection can carry more than one command, so the buffer is finalized at the
next command's start and at the end of the connection.
The buffer holds the full response, so a secret split across several callbacks
is also redacted. A per-callback check cannot do this.
The word-content check is a fallback. The request classification is the main
signal. A secret with no password marker, such as the output of an xdsh cat of
a shadow file, is a pre-existing leak of the root-only log. It is out of scope.
Recovered from the lenovobuild branch. Reimplemented against master.
The test runs the real NodeRange code. It sends a ^ range that a
two-argument open would run as a command and checks the command does not
run. It also reads a real comment-only file to show the operator still
works.
The command assertion fails against the previous behavior.
The ^ noderange operator reads node names from a file. NodeRange opened
that file with a two-argument open. A two-argument open reads shell
metacharacters in the path, so a noderange such as ^"id|" ran a command.
xcatd expands a noderange while it processes a request, so the command
ran on the management node.
Use a three-argument open with an explicit read mode. The value is then
only ever a file name. The ^ operator keeps working: ^/tmp/nodes still
reads the file.
This fix was recovered from the lenovobuild branch. The original there
(commit for "Remove load from file in noderange support") removed the ^
operator. This keeps the documented operator and closes the command path
instead.
build-ubunturepo guarded builds with a single host-global, fail-fast lock at
/var/lock/xcatbld.lock (`flock -n`). Two builds on one host -- e.g. the devel and
stable Ubuntu CD lanes on xcat-master-ub -- therefore collided on that one lock and
the loser exited 1 with "Can't get lock ...", failing the whole pipeline, even though
each lane builds from its own checkout into its own DEST and they share nothing.
build-ubunturepo builds its packages in-place in its own source checkout (it rewrites
debian/changelog and debian/control, drops *.orig.tar.gz at the checkout root and runs
dpkg-buildpackage inside the package dirs), so the resource two concurrent builds
actually contend for is the checkout, not the host. Key the lock on the checkout path
($curdir): builds of the SAME checkout still fail-fast (they would corrupt each other
in-place), while builds of DISTINCT checkouts get distinct locks and run in parallel.
The lock file stays on the local /var/lock (reliable flock, unlike the NFS/virtiofs
checkout) and the source tree is left byte-pristine.
Add xCAT-test/unit/build_ubunturepo_lock.t, which extracts the lock block from the
script verbatim and asserts: the lock path is /var/lock/xcatbld-<hash-of-checkout>.lock
and deterministic per checkout; a second build of the same checkout fails fast; two
distinct checkouts acquire their locks concurrently.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Run redact_password rather than inspecting its source, so that a secret which
survives is a failure here rather than something the shape of the code hides.
Each supported way of writing an assignment is exercised, including the table
qualified form and spaces around the equals sign, together with the positional
flag that the per-command table handles.
The expected set is derived from Schema.pm rather than repeated, so an attribute
added there with a secret column fails this test instead of quietly reaching the
logs, and the test bails if that mapping cannot be parsed so it cannot pass
vacuously. Detail that is not secret is asserted to survive, since redacting it
would cost the log its usefulness without protecting anything.
Only syslog received the redacted arguments. The auditlog table was given the
raw string, so a password removed from syslog and from commands.log was still
written to the database, which persists and is readable by anything with access
to it.
Use the redacted text for both. This also covers the mkvm --password and -w
masking a few lines above, which the table did not have either.
redact_password only knew about bmcdiscover, mkhwconn and rspconfig, so setting
a password the ordinary way left it in the clear:
[Request] chdef node01 'bmcpassword=SEKRET'
Those secrets are carried as an assignment on whichever command happens to set
them, so match them by name rather than by command. The names are every
attribute Schema.pm maps to a password, passwd, authkey or privkey column, and
the columns themselves, since a table qualified assignment such as
passwd.password= is accepted too. An assignment may be written with spaces
around the equals sign and the value may contain spaces, so a quoted argument is
redacted to its closing quote.
Attributes such as key, which names a monitoring attribute, and sshkeydir, which
is a directory, are not secrets and are left readable. The existing per-command
table is kept for a flag like bmcdiscover -p, which carries its secret in the
following argument where there is no name to match.
makedhcp using the Kea backend fails for any node whose mac table entry
uses the *NOIP* sentinel to mark a secondary NIC that has no IP address
(e.g. "mac1|mac2!*NOIP*|mac3!*NOIP*"). kea_node_reservations() and
kea_node_reservations6() split each NIC's hostname and pass it straight
to getipaddr(), so the literal string "*NOIP*" is treated as a host to
resolve. It cannot resolve, and the reservation is reported as unresolved
-- which aborts the whole makedhcp run, leaving the node with no Kea
reservation at all and therefore no DHCP lease.
This breaks provisioning on distros that use the Kea backend (e.g. EL10,
which has no ISC dhcp-server): the node never gets an address and is
unreachable, while the ISC backend has always handled *NOIP* and is
unaffected.
Skip *NOIP* NICs in both the DHCPv4 and DHCPv6 reservation builders, the
same way the ISC path does -- there is no address to reserve for them.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Add a regression case to dhcp_kea_plugin_intent.t for a node whose mac
table entry uses the *NOIP* sentinel on a secondary NIC
("mac1|mac2!*NOIP*"). The mocked getipaddr resolves every name (including
the literal *NOIP*), so the only thing that can keep the node to a single
reservation is an explicit *NOIP* skip in kea_node_reservations() and
kea_node_reservations6(). The test asserts exactly one v4 and one v6
reservation, for the real NIC, with no reservation carrying *NOIP* as a
hostname -- covering both the "bogus second reservation" and the
"unresolved reservation aborts makedhcp" failure modes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The parallel-build failure gate only inspected $exit_code in run_on_finish. A
ForkManager child killed by a signal -- SIGKILL, or the OOM-killer under the
concurrent build load -- is reaped with $exit_code == 0 but $exit_signal != 0
(and possibly $core_dump). Such a worker therefore was NOT recorded as a
failure, so the parent could still index and GPG-sign a repository that is
missing the package that worker was building -- the exact partial-repo hazard
the gate was added to prevent, via a path it did not cover.
Capture $exit_signal and $core_dump from the run_on_finish callback and fail
the build when any of $exit_code, $exit_signal, or $core_dump is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses three concurrency-safety defects raised in review of the parallel
buildrpms.pl work:
1. Per-target build lock was released immediately. The flock filehandle was a
lexical (my $blk) scoped to the guard block, so it was destroyed -- and the
lock dropped -- as soon as that block exited, before any worker forked. The
"intentionally leaked" comment did not match the code. Hold the handle in a
file-scoped $BUILD_LOCK_FH so the fd (and the lock) live for the whole
process; forked children inherit the fd but their exits never release it.
2. Build failures were silently swallowed. buildspkgs()/buildpkgs() called
sh_retry() in void context, so a mock build that failed all retries returned
non-zero into the void; the child then exited 0 and the parent's run_on_finish
ignored the exit code. The parent could therefore index and GPG-sign a repo
that was missing packages and still exit 0. Now sh_retry failures die in the
child, run_on_finish records any non-zero child, and the run aborts before
update_repo and again before signing if any child failed -- never publishing
a partial core.
3. Abort cleanup unmounted unrelated builds. sweep_mock_mounts() lazy-unmounted
every bind under /var/lib/mock, which on a shared host tears out the live
chroots of concurrent, unrelated builds. Scope it to this run's own chroots
(each chroot dir plus its -bootstrap sibling), passed in from abort_builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
merge_core_repos already assembles the multi-arch core and runs the full
finalize tail (index -> sign -> final metadata -> write_release_alias, in the
required order so the xCAT-release-latest alias lands AFTER metadata and stays
out of the repo index). The preceding commit restored a separate finalize_core
sub purely to satisfy xcat_release_package.t, which grepped for
'sub finalize_core { ... $dir ... }'. That left two overlapping entry points:
--finalize-core (finalize one pre-assembled dir) and --merge-core-repos
(assemble N per-arch dirs THEN finalize) -- the former a strict subset of the
latter, with no in-tree or CI caller.
Consolidate on the single assemble+finalize path: inline the finalize tail into
merge_core_repos, drop sub finalize_core and the --finalize-core getopt/dispatch,
and update xcat_release_package.t to assert the stable alias is created after the
final metadata pass inside merge_core_repos ($out). Behaviour is unchanged (merge
already wrote the alias); only the redundant finalize_core interface is removed.
xcat_release_package.t 26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The CI buildrpms.pl port (d2d98b724) renamed the upstream sub finalize_core($dir)
to merge_core_repos($out) and inlined its index/sign/metadata/alias tail. The
xCAT-release logic (write the xCAT-release-latest stable bootstrap alias AFTER the
final metadata pass, so it stays out of the repo index) was preserved, but the rename
broke xCAT-test/unit/xcat_release_package.t, which statically pins:
sub finalize_core { ... write_repo_metadata_dir($dir); ... write_release_alias($dir); }
so 'assembled core repository creates the stable alias after final metadata' failed
(1 of 26), turning xcat_pr_test red on PR #7701.
Restore finalize_core($dir) as the shared finalize primitive (index -> sign ->
final metadata -> stable alias) and have merge_core_repos delegate to it on the
assembled multi-arch dir. Re-expose the --finalize-core CLI (single already-populated
dir) it also feeds. Behaviour is unchanged for both --merge-core-repos (CI) and
--finalize-core; only the shared code path is named again. xcat_release_package.t now
26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl built every package with mock -N (--no-clean), reusing the per-package
<pkg>-<target><ext> chroot across runs for flat disk. But a build aborted or killed
mid-flight leaves that chroot half-initialised with a corrupt rpmdb; the NEXT run
reused it and failed (cannot open Packages database .../usr/lib/sysimage/rpm),
producing an incomplete core (e.g. missing xCAT-test) that fails the deploy-time
completeness gate.
Re-init the buildroot (mock --init) right before building each package, after the
diskcache skip so it only runs when actually building. --init restores from mock's
root-cache tarball (cheap) so disk stays flat and builds stay fast; -N is kept on the
srpm/binary calls so they still reuse the freshly initialised root within the run.
This stabilises builds against corrupt state left by any previous failed/aborted build.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 3 EL CD pipelines (xcat-core-devel-cd, xcat-core-stable-cd, xcat-dep-el-cd)
overlaid a pinned $CI/buildrpms.pl at build time because the tree's buildrpms.pl
lacked the options they depend on:
- a per-target flock guard alongside --mock-uniqueext, so concurrent same-target
builds do not corrupt each other's /var/lib/mock chroot namespace;
- --native-only (build only arch-native pkgs on the secondary arch) plus
--merge-core-repos/--output-dir/--input-core-repos, replacing --finalize-core,
to assemble one signed flat multi-arch core from per-arch build outputs;
- sh_retry() to absorb transient mock/nspawn flakes;
- a single --target guard and graceful mock cancellation
(sweep_mock_mounts/abort_builds) that unmounts chroots on abort.
The xCAT-release repository package (master/2.19 only) is preserved: its
write_release_alias() is invoked from the new merge_core_repos() and per-target.
Porting them in-tree lets CI drop the $CI/buildrpms.pl pin and run the three
pipelines in parallel without the cross-job serialize lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #7677 added xCAT-server/share/xcat/netboot/rh/compute.rhels9.ppc64le.pkglist
with a stray empty line at the end of the file. The reviewer asked for it to be
removed before merge, but it slipped through. A trailing blank line in a pkglist
is meaningless and inconsistent with the rest of the netboot package lists, so
remove it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Vinícius Ferrão
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão