Brings in #7866, which removes --database from the createrepo call. Without it the
branch cannot index a repository on the shared build tree: createrepo_c tries to
write primary.sqlite there and the NFS re-export answers
Cannot open .repodata/primary.sqlite: Can not create db_info table: disk I/O error
which failed every EL build in xcat-ci #44 while the Ubuntu builds, which do not use
createrepo, passed.
Killing the command is not killing the build. sh() ran the command through /bin/sh,
and cancellation signalled that shell alone -- but dpkg-buildpackage starts workers
of its own, and those survive their shell. The lock was then released while they
were still writing debian/changelog and debian/control, which is the state the lock
exists to prevent: the next build takes the checkout and the two rewrite it
together.
The command now runs in its own process group, so cancellation can take all of it.
Both sides call setpgid, so neither depends on which runs first, and INT and TERM
are blocked across the fork so cancellation cannot land in the window before the
group exists.
Cancellation escalates from the caught signal to KILL, and then CHECKS: a shell that
has exited is not a build that has stopped, so it waits for the whole group to
disappear rather than for the leader to be reaped. If the group is still there after
that, the locks are RETAINED and the process exits non-zero. Releasing a lock while
a worker may still be writing is worse than leaving a lock behind for a person to
clear -- the first corrupts a build, the second stops one.
cancel_build ignores INT and TERM while it runs, so a second Ctrl-C cannot interrupt
the cleanup half way and release the lock early.
sh() also reports a signalled command as 128+signal instead of 0. $? >> 8 is zero
for a child killed by a signal, so a build stopped mid-way looked to its caller like
one that had succeeded.
Two cases added to builddebs_lock_cancellation.t: a build whose worker is a
grandchild, and a command killed by a signal. Verified by signalling the pid instead
of the group, which leaves the worker running and turns the first red.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is released in DESTROY, and perl does not run DESTROY when a signal
ends the process. A build stopped with SIGTERM or SIGINT therefore left its lock
directory behind, and the next build of that checkout died on
FATAL: another build of <path> already holds <dir> (held by [pid=NNNN])
naming a pid that had already exited. Nothing clears it but a person. One such
directory blocked an openSUSE target across three consecutive CI runs before anyone
looked at what the lock actually said.
buildrpms.pl has released its lock on cancellation for some time, through an END
block and an abort handler. This is the Debian builder catching up.
The order matters, and is the reason this is not simply an END block. The command in
flight is stopped BEFORE the lock is released: handing the checkout to a second build
while dpkg-buildpackage is still rewriting debian/changelog and debian/control in it
is worse than holding the lock a moment longer. The wait for that command is bounded,
so a subprocess that ignores the signal cannot hold the lock for ever either.
sh() now forks and execs rather than calling system(), because system() gives no pid
and a handler cannot stop what it cannot name. The child _exits rather than exits, so
it never runs the parent's END block and releases a lock the parent still holds.
The handler is installed by XCAT::BuildUtils::install_build_cancellation rather than
written inline in the builder, so a test can use the same wiring the builder uses. A
test that installs an equivalent handler of its own proves the helper works while
saying nothing about whether anything calls it -- the first version of this test did
exactly that, and passed with the wiring removed.
Release is idempotent: a signal handler and then DESTROY both reach it, and the
second must not remove a directory a LATER build has since taken.
builddebs_lock_cancellation.t terminates the holder, then takes the lock again, and
checks no build subprocess was orphaned. Verified by removing the wiring: assertions
9 through 12 fail, naming the leaked lock, the refused build and the stray process.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The project had no written release process, and the 2.18 and 2.19
releases missed steps: the release table in the documentation, the
docs version, signed tags, and the wiki and website index pages.
A new Releases section describes the branch and version model, the
label and milestone that each pull request needs, and a checklist for
release candidates, publishing, the signed tag, the GitHub release, the
release notes, the website and the announcement. The build hosts, the
signing key and the publish procedure stay in the private repository of
the maintainers.
The OpenEmbedded metadata job ran on every pull request and took about
20 minutes. It reads only xCAT-genesis-builder/oe and
xCAT-genesis-scripts, and it pins its upstream sources to fixed commits.
A pull request that changes neither directory gets the same result each
time.
The job moves unchanged to its own workflow, which runs only when those
paths or the workflow file change. xcat_pr_test stays in xcat_test.yml
and runs on every pull request, because a required check that does not
run blocks the merge.
2.19.0 is released, and master still builds packages that call themselves
2.19.0. Every snapshot built from master since then carries the released
version, so a candidate cannot be told from the release it follows.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build tree can live on an NFS re-export. The kernel refuses locks on one --
"Clients are not allowed to get file locks or delegations from a reexport
server" -- so every flock() there answers errno 524, and a build that takes one
dies before it starts.
buildrpms.pl's per-target lock and BuildUtils.pm's take_build_lock, which
builddebs.pl calls for the Ubuntu core build, are both atomic mkdir claims now.
Each records its owner and names it when it refuses.
A directory is not released by a filehandle closing, which is how both locks
were freed before. buildrpms.pl releases from END, and again in abort_builds
because that handler re-raises the signal with DEFAULT and END blocks do not run
then -- a killed build would otherwise strand the lock for every later one.
BuildUtils returns a small object whose DESTROY releases it, preserving the
caller's "hold the returned value" contract.
Both releases are guarded by owning pid: both scripts fork, and the flock they
replace could not be released by a child.
builddebs_lock.t closed the returned value to prove the lock is released, which
is "Not a GLOB reference" against the new contract. It now lets the value go out
of scope. What it asserts is unchanged: a second build of the same checkout is
refused, and the next one succeeds once the first releases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
createrepo_dir passed --database, which writes *.sqlite.bz2. Building those
needs SQLite, and SQLite needs POSIX locks. A build tree can live on an NFS
re-export, where the kernel refuses locks outright: every attempt answers
errno 524.
So every target died with "Cannot open .repodata/primary.sqlite: Can not create
db_info table: disk I/O error", and the build staged nothing.
Measured on such a share, with a local control: a bare sqlite3 connect fails
there and succeeds on local disk; createrepo_c fails with --database and
succeeds without it, emitting primary/filelists/other as *.xml.zst.
Nothing this project ships reads the sqlite metadata. dnf on el8+ and zypper
both read the XML.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The step that fills lib/firmware in the Ubuntu netboot initrd asked modinfo
about $rootimg_dir/$module and looked a firmware name up under lib/firmware.
The copy step beside it takes a module from $customdir or $pathtofiles first,
and the kernel looks a firmware name up under updates/<kernel>, updates/,
<kernel>/ and lib/firmware. So a custom driver reached the initrd with no
firmware, even when the root image carried it, and a firmware override was left
out of the initrd altogether.
initrd_firmware_files now takes the module directories the copy step searches
and the kernel release. It resolves each module in that order before asking
modinfo, and keeps every firmware file that exists in the four directories the
kernel searches, so the override still wins on the node.
ubuntu_genimage_initrd_firmware.t covers both: its two new cases are red on the
commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t drove the firmware step over a root image that
holds every module and every firmware file. It covered neither of the two places
genimage reads from beside the root image.
The test now puts a driver in the custom directory, with its firmware in the
root image, and a firmware override under lib/firmware/updates/<kernel>. Both
are red: the step asks modinfo about the module under the root image, where a
custom driver is not, and it looks for a firmware name under lib/firmware only,
where an override is not.
The five assertions that were there stay green.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
reg_linux_diskless_installation_flat corrupts the KVM machine type, proves the
node fails to boot, restores the machine type and then removes it again. On
ppc64 the two ladders name different machine types: the restore writes
machine:pseries-rhel7.6.0 and the cleanup removes machine:pseries-7.6.0. The sed
matches nothing, so the node keeps machine:pseries-rhel7.6.0 after the case ends
and the next case runs against a node the previous one changed.
The cleanup ladder now names the machine type the restore ladder writes.
diskless_flat_vmothersetting_machine.bats covers it: the ppc64le cleanup cases
are red on the commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
diskless_flat_vmothersetting_machine.bats checked the restore with a substring
match and checked the cleanup only for the absence of "unary operator
expected". A cleanup that writes the machine type back, or leaves it in place,
passed both.
The test now reads the value chdef receives. The restore must write exactly
machine:<type>, and must keep a setting the node already carries. The cleanup
must write an empty value when the machine type is all there is, and must leave
the other setting behind when there is one. The chdef stub brackets its
arguments so an empty value is not the same as no call.
ppc64le is red on the cleanup: the restore ladder writes
machine:pseries-rhel7.6.0 and the cleanup ladder removes machine:pseries-7.6.0,
so the node keeps the machine type. x86_64 and riscv64 pass.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
REQUIRED_PACKAGES named util-linux-extra for every release. focal and jammy
have no such package -- apt reports "Candidate: (none)" -- so apt-get install
exits non-zero and, under set -euo pipefail, the build stops before dracut
runs. hwclock is in util-linux there, which is essential and already present.
optional_packages() keeps a package only where apt has a candidate for it, and
util-linux-extra goes through it. The unconditional list keeps isc-dhcp-client
and ifenslave, which every release has and neither of which the build root
carries by itself.
The call to verify-genesis-payload goes with it. That script is added by the
genesis payload branch, not this one, so the line stopped the build at the
point it was meant to guard.
Also corrects the plan count and a dereference in the test committed before
this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddeb-genesis-base names util-linux-extra in REQUIRED_PACKAGES for every
release. Measured on the four Ubuntu management nodes: focal and jammy report
"Candidate: (none)" for that package and carry hwclock in util-linux, which is
essential and already in the build root; noble and resolute carry it in
util-linux-extra. apt-get install with a package it cannot locate exits
non-zero, and the script runs under set -euo pipefail, so the build stops on
two supported targets before dracut runs.
util-linux only Suggests util-linux-extra, and the install passes
--no-install-recommends, so a release that split the package has to name it.
The test asserts the unconditional list does not name it, and drives the
selector that decides, with apt-cache shadowed for a release that has the
package and one that does not.
It also drops the assertion that matched "verify-genesis-payload" against the
text of the build script. That proved the string was present, not that the
verifier ran, ran before packaging, or stopped the build -- and the script it
names does not exist on this branch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t called plan skip_all when xCAT-server/share/xcat/netboot/ubuntu/genimage was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/share/xcat/netboot/ubuntu/genimage moved aside the file now exits 2 and prints "genimage not found at <path>";
before this change it exited 0 and printed "1..0 # SKIP genimage not found at <path>". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
debian_install_prescript.t called plan skip_all when xCAT-server/lib/xcat/plugins/debian.pm was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/lib/xcat/plugins/debian.pm moved aside the file now exits 2 and prints "debian.pm not found";
before this change it exited 0 and printed "1..0 # SKIP debian.pm not found". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs_release_placeholder.t called plan skip_all when builddebs.pl was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With builddebs.pl moved aside the file now exits 2 and prints "builddebs.pl not found";
before this change it exited 0 and printed "1..0 # SKIP builddebs.pl not found". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
makedns reports "error was FORMERR" for an update that named rejected with
NOTAUTH. The FORMERR is the answer to the retry, not to the first attempt.
send_ddns_update in ddns.pm signs the packet the caller built, and signs that
same packet again on each attempt. Net::DNS::Packet::sign_tsig appends the TSIG
to the additional section, so the second attempt sends two TSIG records and
named answers FORMERR. FORMERR is neither NOTAUTH nor SERVFAIL, so the routine
stops and reports it. The NOTAUTH and SERVFAIL retry can never be accepted, on
any algorithm.
Each attempt now signs a request of its own. A packet cannot be unsigned again,
so ddns_update_request copies the prerequisite and update records into a new
Net::DNS::Update instead, and the caller keeps the unsigned original.
ddns_update_retry.t fails before this change: the second attempt carries two
TSIG records, and an update that the retry answers with NOERROR still reports
failure.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
send_ddns_update signs the same packet on every attempt. Net::DNS appends the
TSIG to the additional section, so the second attempt carries two TSIG records.
named answers FORMERR to that message, which is neither NOTAUTH nor SERVFAIL, so
the routine stops and reports FORMERR. The retry path can never be accepted.
ddns_update_retry.t drives send_ddns_update with a resolver that answers FORMERR
to a message with more than one TSIG record, as named does, and otherwise
answers a scripted rcode. It asserts that every attempt carries exactly one TSIG
record and the same update records, and that a retry answered NOERROR reports
success. Both subtests fail before the fix.
The header of each new test records that XCATROOT must name the tree under test,
because xCAT::Table adds the installed /opt/xcat/lib/perl to @INC.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
makedns exits 1 on a management node that has Net::DNS below 1.36 and an
hmac-sha256 key, and reports "Failure encountered updating <zone> with entry
'', error was FORMERR".
update_namedconf in ddns.pm rewrites the named.conf key stanza to hmac-md5
whenever Net::DNS is below 1.36, and ddns_tsig_algorithm returns hmac-md5 for
the same reason. ddns_sign_update signs with site.dhcpomapialgorithm, which
xcatconfig sets to hmac-sha256 on EL9 and later. named matches a TSIG key by
name and by algorithm, so it answers NOTAUTH. The retry signs the same packet
a second time, and named answers FORMERR to the two signatures.
The version test protected the two-argument sign_tsig($name, $secret), which
produces an HMAC-MD5 signature only. ddns_sign_update signs every other
algorithm through a KEY RR, so the Net::DNS version no longer selects the
algorithm. This change deletes the rewrite and the version test, and signs with
the algorithm the key stanza declares. OmapiPolicy->algorithm_rr_type maps that
algorithm to its KEY RR number.
ddns_named_key_algorithm.t fails before this change: it reads the stanza as
hmac-md5 where the key was hmac-sha256.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
On a management node with Net::DNS below 1.36, makedns rewrites the named.conf
key stanza to hmac-md5. It then signs the update with the algorithm the site
table selects. named matches a TSIG key by name and by algorithm, so it rejects
every update and makedns exits 1.
ddns_named_key_algorithm.t drives update_namedconf over a scratch named.conf and
then signs one update with the context that run produced. It asserts that the
stanza keeps the algorithm the key was generated with, that the signature uses
that algorithm, and that named is not restarted. Two of its five subtests fail
before the fix.
ddns_omapi_policy.t pinned the rewrite as correct, so its expectations move to
the algorithm the key already has.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
genesis_base_deb_arch.t stopped with a die when debuild-xcat-genesis-base was
absent, because a checkout without the converter has no deb rename to measure.
The BATS file skipped there instead, and a skip reads green.
setup() now asserts the script is readable.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
diskless_flat_vmothersetting_machine.t measures shell. It lifts two commands
out of reg_linux_diskless_installation_flat, renders the xcattest placeholders,
writes a wrapper that shadows lsdef and chdef, and shells out. The Perl adds
nothing the shell cannot state, and the case file it reads is shell itself.
The test moves to xCAT-test/bats, which the xcat_test workflow already runs. It
keeps what it proved: on ppc64le, x86_64 and riscv64 the restore writes a
machine type rather than an empty vmothersetting, the check that follows it
passes, and the remove path compares two defined strings. A command that stops
matching fails the test instead of covering nothing.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Eight unit tests measure shell code: the Genesis dracut cmdline hooks, doxcat,
getcert, the two Genesis deb builders, go-xcat and the genesis test case. The
Perl in each one is scaffolding. It reads the script, lifts a block out with a
regular expression, writes a wrapper, shells out and reads the files back. A
reader follows two languages to reach one assertion, and the scaffolding is
longer than the assertion.
xCAT-test/bats already states this kind of assertion in the language of the
thing under test, and the xcat_test workflow runs it. The eight files move
there. Each one keeps what it proved: the rpm architecture becomes the Debian
architecture and names the deb it supersedes, the dracut hook picks the console
mode the multiplexer can provide, the hook gives root the home directory /,
getcert stops when the image ships no openssl, the genesis case defines its
node with the architecture of the management node and fails when nodeset fails,
doxcat picks dhcpcd where the release drops the ISC client, and go-xcat names
the Genesis packages the packaging builds.
helpers/shell_source.bash gains refute_grep. bash ignores errexit for a command
inverted with "!", so "! grep" anywhere but the last line of a test can never
fail it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The comment above the /usr/bin/awk case in genesis_payload_verification.t
said what the verifier used to do with an absolute path. The rule the
reader needs is what dracut_install does with one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Thirteen test files this branch adds call BAIL_OUT at fifty-one places:
an extraction that stopped matching, a fixture that is not there, a
harness that wrote no log. prove stops every remaining file on a
bail-out, not only the file that called it, so one of them hides the
results of every test that would have run after it. die is just as loud
and costs only its own file.
Fifteen comments the branch added also carried the incident rather than
the constraint. Three pasted an error transcript, five traced a failure
from a macro or a missing file out to a node that never boots, and the
rest counted call sites, package sizes or dracut build numbers. Each now
states the one fact the reader cannot re-derive from the code.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
debian_install_prescript.t and ubuntu_subiquity_storage.t called BAIL_OUT
at eight places where an extraction or a fixture was not what they
expected. prove stops every remaining file on a bail-out, so one changed
script hides the results of every test that would have run after it. die
is just as loud and costs only its own file.
Three comments also carried the incident rather than the constraint. The
prescript test header named a build number, a cell, a case and the ssh
error the node produced nine times; the boot-file test repeated the
nodeset message and the media table below it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
utils_gettimezone.t called BAIL_OUT when it could not lift the two
routines out of Utils.pm. prove stops every remaining file on a bail-out,
so a rename in Utils.pm that breaks the regex in this file also hides
every test that would have run after it. die is just as loud and costs
only this file.
The header also carried the incident report: a node name, an autoinst
file and line, the anaconda message and the minutes the node spent in
status=installing. The reader needs the contract, which is that the value
must be one token.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The comment on the use-domains assertion in ubuntu_subiquity_installnic.t
followed the missing search domain from systemd-networkd through
updateflag.awk to a node stuck in postbooting. The assertion below it
already names what is checked. The comment now carries the one fact the
code does not show: networkd defaults UseDomains to no.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
diskless_flat_vmothersetting_machine.t called BAIL_OUT at four places
where it could not read or match the case file. prove stops every
remaining file on a bail-out, so an edit to the case that breaks a regex
in this file also hides every test that would have run after it. die is
just as loud and costs only this file.
The header also traced the empty vmothersetting out to the red cell. What
the reader needs is the ladder that names two architectures.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs_release_placeholder.t called BAIL_OUT when its extraction of
the release decision stopped matching. prove stops every remaining file
on a bail-out, so a rewrite of builddebs.pl that breaks the regex in this
file also hides every test that would have run after it. die is just as
loud and costs only this file.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
genesis_ubuntu_build_root.t called BAIL_OUT at four places where an
extraction stopped matching. prove stops every remaining file on a
bail-out, so one stale regex in this file hides the results of the tests
that would have run after it. die is just as loud and costs only this
file.
Four comments in the same file also explained more than the code hides:
the header retold the failure the test exists for, two narrated the two
helper subroutines, and one traced the lease failure across doxcat, the
image and the node.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Three comments in debian_install_boot_files.t explain more than the code
hides. One narrates the media table the assertions already list. One
repeats the nodeset error message and the releases that produce it. One
describes what mkinstall used to refuse before the change.
A comment states why, once, where the reader cannot re-derive it. Each of
the three now carries only the fact the assertions do not show.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão
Gary Skouson