genesis_ubuntu_build_root.t cut REQUIRED_PACKAGES out of
builddeb-genesis-base with a regex and evaluated it through bash -c. It
then matched the apt-cache fallback calls as text. Reformatting the
script broke it, and a wrong choice between renamed packages passed it.
genesis_payload_verification.t ran the verifier script and parsed its
stderr.
genesis_ubuntu_build_root.t now calls required_packages() with a chosen
set of carried packages and asserts the exact result: the amd64 extras,
the name picked for each renamed package, the order apt is asked,
tzdata-legacy, and the error for a release that carries neither name.
It reads the mandatory commands from XCAT::GenesisPayload, the code the
build uses. genesis_payload_verification.t calls the XCAT::GenesisPayload
functions with chosen payload trees and asserts the exact missing paths
and results.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Brings in #7866, which removes --database from the createrepo call. Without it the
branch cannot index a repository on the shared build tree: createrepo_c tries to
write primary.sqlite there and the NFS re-export answers
Cannot open .repodata/primary.sqlite: Can not create db_info table: disk I/O error
which failed every EL build in xcat-ci #44 while the Ubuntu builds, which do not use
createrepo, passed.
Killing the command is not killing the build. sh() ran the command through /bin/sh,
and cancellation signalled that shell alone -- but dpkg-buildpackage starts workers
of its own, and those survive their shell. The lock was then released while they
were still writing debian/changelog and debian/control, which is the state the lock
exists to prevent: the next build takes the checkout and the two rewrite it
together.
The command now runs in its own process group, so cancellation can take all of it.
Both sides call setpgid, so neither depends on which runs first, and INT and TERM
are blocked across the fork so cancellation cannot land in the window before the
group exists.
Cancellation escalates from the caught signal to KILL, and then CHECKS: a shell that
has exited is not a build that has stopped, so it waits for the whole group to
disappear rather than for the leader to be reaped. If the group is still there after
that, the locks are RETAINED and the process exits non-zero. Releasing a lock while
a worker may still be writing is worse than leaving a lock behind for a person to
clear -- the first corrupts a build, the second stops one.
cancel_build ignores INT and TERM while it runs, so a second Ctrl-C cannot interrupt
the cleanup half way and release the lock early.
sh() also reports a signalled command as 128+signal instead of 0. $? >> 8 is zero
for a child killed by a signal, so a build stopped mid-way looked to its caller like
one that had succeeded.
Two cases added to builddebs_lock_cancellation.t: a build whose worker is a
grandchild, and a command killed by a signal. Verified by signalling the pid instead
of the group, which leaves the worker running and turns the first red.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install monitor holds the later connections for a node whose handler
is still running, and forks the next one when it reaps that handler.
SIGCHLD is what brings the parent back to look: it interrupts the accept.
A handler can exit after the parent checks its queue and before the
accept begins. The signal is then handled where there is no accept to
interrupt, and the parent blocks in accept with a connection already
queued and ready to run. That connection waits until some other node
calls in. A single node retrying on its own waits until it times out.
do_installm_service now waits through wait_for_installm_connection, which
selects on the listening socket. The wait is bounded by
$installm_wakeup_seconds while connections are queued, so the parent
looks at its queue again instead of waiting for another client. An idle
monitor with an empty queue still waits without a bound, because a
handler that exits then leaves nothing to do.
The new case asserts the wait ends on its own bound with nothing to
accept, and ends at once when a connection is already there. It fails
when the bound is ignored.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install monitor answers a node's requests in the order they arrived.
Two ways to lose that order had no test.
The first is a queued client that gives up. The parent holds the later
connections for a busy node unread, so a client that closes its socket
must not let the request behind it overtake the request that is running.
The new case runs one request, queues two more, drops the middle client,
and asserts the last request starts after the running one ends. It fails
when the per-node queue is removed.
The second is a fork that fails. The monitor then answers the node
itself, in line. The new case makes one fork fail and asserts the request
is answered and that the node's next request starts only after it. It
fails when the fallback drops the connection instead.
The file also asserts the lifted service holds no literal /var/run path.
Every access to the pid file goes through $installm_pidfile, so pointing
that variable at the scratch tree redirects all of them, and a path
written out again inside the routine would reach the host file.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcatd_install_monitor_concurrency.t failed about one run in five. The
assertion "the node is released only after the destiny advance finished"
compares the time the test read "done" from the socket against the time
the plugin stand-in recorded when it finished. The stand-in wrote that
time with %.3f, which rounds up, so a recorded time can be later than the
moment it was taken. The test then failed on the rounding and not on the
order:
'1790220468.11892' >= '1790220468.119'
The events file now holds whole microseconds from gettimeofday, and the
comparisons read the same clock. gettimeofday rounds nothing.
Each case also writes an events file of its own. A handler forked by one
case outlives the monitor that forked it, so it could append to the case
that runs next.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is released in DESTROY, and perl does not run DESTROY when a signal
ends the process. A build stopped with SIGTERM or SIGINT therefore left its lock
directory behind, and the next build of that checkout died on
FATAL: another build of <path> already holds <dir> (held by [pid=NNNN])
naming a pid that had already exited. Nothing clears it but a person. One such
directory blocked an openSUSE target across three consecutive CI runs before anyone
looked at what the lock actually said.
buildrpms.pl has released its lock on cancellation for some time, through an END
block and an abort handler. This is the Debian builder catching up.
The order matters, and is the reason this is not simply an END block. The command in
flight is stopped BEFORE the lock is released: handing the checkout to a second build
while dpkg-buildpackage is still rewriting debian/changelog and debian/control in it
is worse than holding the lock a moment longer. The wait for that command is bounded,
so a subprocess that ignores the signal cannot hold the lock for ever either.
sh() now forks and execs rather than calling system(), because system() gives no pid
and a handler cannot stop what it cannot name. The child _exits rather than exits, so
it never runs the parent's END block and releases a lock the parent still holds.
The handler is installed by XCAT::BuildUtils::install_build_cancellation rather than
written inline in the builder, so a test can use the same wiring the builder uses. A
test that installs an equivalent handler of its own proves the helper works while
saying nothing about whether anything calls it -- the first version of this test did
exactly that, and passed with the wiring removed.
Release is idempotent: a signal handler and then DESTROY both reach it, and the
second must not remove a directory a LATER build has since taken.
builddebs_lock_cancellation.t terminates the holder, then takes the lock again, and
checks no build subprocess was orphaned. Verified by removing the wiring: assertions
9 through 12 fail, naming the leaked lock, the refused build and the stray process.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build tree can live on an NFS re-export. The kernel refuses locks on one --
"Clients are not allowed to get file locks or delegations from a reexport
server" -- so every flock() there answers errno 524, and a build that takes one
dies before it starts.
buildrpms.pl's per-target lock and BuildUtils.pm's take_build_lock, which
builddebs.pl calls for the Ubuntu core build, are both atomic mkdir claims now.
Each records its owner and names it when it refuses.
A directory is not released by a filehandle closing, which is how both locks
were freed before. buildrpms.pl releases from END, and again in abort_builds
because that handler re-raises the signal with DEFAULT and END blocks do not run
then -- a killed build would otherwise strand the lock for every later one.
BuildUtils returns a small object whose DESTROY releases it, preserving the
caller's "hold the returned value" contract.
Both releases are guarded by owning pid: both scripts fork, and the flock they
replace could not be released by a child.
builddebs_lock.t closed the returned value to prove the lock is released, which
is "Not a GLOB reference" against the new contract. It now lets the value go out
of scope. What it asserts is unchanged: a second build of the same checkout is
refused, and the next one succeeds once the first releases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
master changed the same three files this branch changes, so the pull request could not
merge. Conflicts and how each was resolved:
xCAT-genesis-builder/builddeb-genesis-base. Both sides teach the build root to cope with
a package whose name moved between releases. master added optional_packages(), which
keeps a package only where apt has a candidate, and called it for util-linux-extra. This
branch added add_first_available(), which takes the first name apt carries and fails when
it carries none, and calls it for bind9-dnsutils/dnsutils and util-linux-extra/util-linux,
plus add_if_available() for tzdata-legacy. The branch covers master's case and two more,
so its helpers are kept and optional_packages() goes with its only caller. Every other
master change to this file, including the DHCP client fix, is preserved.
xCAT-test/unit/genesis_payload_verification.t. master has four assertions this branch does
not: two payloads missing an absolute path. Its version is kept. The branch replaced
plan skip_all with a fail(), because skipping covers nothing when the file under test is
the gate itself, and that change is applied to master's version.
xCAT-test/unit/genesis_ubuntu_build_root.t. master's added assertions drive
optional_packages() directly, which the resolved builder no longer has. This branch's
version matches the implementation that survives, and it already dies rather than skipping
when the builder is missing, so it is kept whole.
prove -j4 -r xCAT-test/unit passes: 212 files, 5892 tests.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The test extracted and evaluated part of nextdestiny, so it could not
detect missing database writes or downstream dispatch.
Load the complete plugin and exercise request routing, persisted chain
updates and destiny handoffs with database and command fixtures.
The dispatch test matched source patterns without checking returned
requests. Load the complete plugin and exercise its request preprocessing
with fixture database rows and the real service-node mapping code.
Check destination filtering, network regeneration, self-dispatch guards,
request contents and hierarchical network errors. Run each case in a fresh
process so lexical option state cannot leak between fixtures.
The identity test checked source text without observing the request sent
to xdcp.
Run the complete syncfiles plugin and inspect each outgoing request.
Keep production code unchanged.
The ordering between two requests for one node was a chain of pipes: each child
held the write end and the next child for that node read the previous one to end
of file. End of file there means the previous process is gone, not that its work
finished, so a child that died released the one behind it -- and released the
wrong one, because each child waits on its immediate predecessor rather than on
the request actually in flight. The fork-failure path closed the predecessor and
served the request in line without waiting at all.
The parent now owns the order. %installm_busy names the handler serving a node,
%installm_queue holds the connections accepted for that node meanwhile, and the
next one is forked when the handler ahead of it is reaped. A handler that dies
cannot release the one behind it, and the fork-failure path has nothing to fall
back over, because it is reached only when the node has no handler.
Three other changes the same design makes possible or necessary:
- The answer to a destiny advance now follows the advance. Every other request
is still answered before it runs, because its result does not change what
the node does next. Holding one node costs no other node anything now, and
it lets the node retry an advance whose handler died -- which the old order
could not, because "done" was already on the wire.
- The monitor drains its handlers before it exits. Without this a restart
orphans them into the systemd service cgroup, where anything still running
at TimeoutStopSec is killed after its answer was already sent.
- SIGCHLD is caught, so a handler exiting interrupts accept and the parent
comes back to look for a connection queued for that node.
The pid file path is a variable, so the test can point the lifted routine at a
scratch file instead of the one a restarting xcatd reads to tell the running
monitor to let go of the port.
xcatd_install_monitor_concurrency.t grew the cases for all of it. Four
mutations, each caught by one assertion: forking every connection at once turns
the ordering case red; answering a destiny advance before the plugin turns the
release case red; removing the drain turns the stand-down case red; and leaving
the emptied queue entry behind turns the leak case red. The full unit suite is
194 files, 5662 tests, green.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The step that fills lib/firmware in the Ubuntu netboot initrd asked modinfo
about $rootimg_dir/$module and looked a firmware name up under lib/firmware.
The copy step beside it takes a module from $customdir or $pathtofiles first,
and the kernel looks a firmware name up under updates/<kernel>, updates/,
<kernel>/ and lib/firmware. So a custom driver reached the initrd with no
firmware, even when the root image carried it, and a firmware override was left
out of the initrd altogether.
initrd_firmware_files now takes the module directories the copy step searches
and the kernel release. It resolves each module in that order before asking
modinfo, and keeps every firmware file that exists in the four directories the
kernel searches, so the override still wins on the node.
ubuntu_genimage_initrd_firmware.t covers both: its two new cases are red on the
commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t drove the firmware step over a root image that
holds every module and every firmware file. It covered neither of the two places
genimage reads from beside the root image.
The test now puts a driver in the custom directory, with its firmware in the
root image, and a firmware override under lib/firmware/updates/<kernel>. Both
are red: the step asks modinfo about the module under the root image, where a
custom driver is not, and it looks for a firmware name under lib/firmware only,
where an override is not.
The five assertions that were there stay green.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
deb_belongs_to_dist reads any trailing ~word in a package version as the
codename the deb was built for. Debian uses ~ for a prerelease, and --release
takes whatever the caller gives it, so `--release 1~rc1` puts one in every
package name:
xcat-client_2.19.0-1~rc1_all.deb noble=0 focal=0
The deb is then published into no suite at all, with --genesis nowhere in the
command, and the run still reports how many packages it published.
Only the Genesis image is built per codename, so only it can be excluded by
one. The new cases assert a prerelease version reaches every suite, that a
Genesis SCRIPTS deb is not treated as the image, and that the image itself is
still confined to its own suite when its version carries both.
The second half asserts genesis_dists(), which does not exist yet: a plain
--genesis run takes the release list the rest of the build uses, and focal is
on it. focal ships debhelper 12.10 -- measured on the focal management node --
against the package's debhelper-compat (= 13), so sbuild stops on the build
dependencies and the run ends on its first release.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
REQUIRED_PACKAGES named util-linux-extra for every release. focal and jammy
have no such package -- apt reports "Candidate: (none)" -- so apt-get install
exits non-zero and, under set -euo pipefail, the build stops before dracut
runs. hwclock is in util-linux there, which is essential and already present.
optional_packages() keeps a package only where apt has a candidate for it, and
util-linux-extra goes through it. The unconditional list keeps isc-dhcp-client
and ifenslave, which every release has and neither of which the build root
carries by itself.
The call to verify-genesis-payload goes with it. That script is added by the
genesis payload branch, not this one, so the line stopped the build at the
point it was meant to guard.
Also corrects the plan count and a dereference in the test committed before
this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddeb-genesis-base names util-linux-extra in REQUIRED_PACKAGES for every
release. Measured on the four Ubuntu management nodes: focal and jammy report
"Candidate: (none)" for that package and carry hwclock in util-linux, which is
essential and already in the build root; noble and resolute carry it in
util-linux-extra. apt-get install with a package it cannot locate exits
non-zero, and the script runs under set -euo pipefail, so the build stops on
two supported targets before dracut runs.
util-linux only Suggests util-linux-extra, and the install passes
--no-install-recommends, so a release that split the package has to name it.
The test asserts the unconditional list does not name it, and drives the
selector that decides, with apt-cache shadowed for a release that has the
package and one that does not.
It also drops the assertion that matched "verify-genesis-payload" against the
text of the build script. That proved the string was present, not that the
verifier ran, ran before packaging, or stopped the build -- and the script it
names does not exist on this branch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcatd_install_monitor_concurrency.t called plan skip_all when xCAT-server/sbin/xcatd was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/sbin/xcatd moved aside the file now exits 2 and prints "xcatd not found at <path>";
before this change it exited 0 and printed "1..0 # SKIP xcatd not found at <path>". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t called plan skip_all when xCAT-server/share/xcat/netboot/ubuntu/genimage was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/share/xcat/netboot/ubuntu/genimage moved aside the file now exits 2 and prints "genimage not found at <path>";
before this change it exited 0 and printed "1..0 # SKIP genimage not found at <path>". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
postscript_heredoc_embedding.t called plan skip_all when xCAT-server/share/xcat/install/scripts was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/share/xcat/install/scripts moved aside the file now exits 2 and prints "<scriptdir> not found";
before this change it exited 0 and printed "1..0 # SKIP <scriptdir> not found". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão