Brings in #7866, which removes --database from the createrepo call. Without it the
branch cannot index a repository on the shared build tree: createrepo_c tries to
write primary.sqlite there and the NFS re-export answers
Cannot open .repodata/primary.sqlite: Can not create db_info table: disk I/O error
which failed every EL build in xcat-ci #44 while the Ubuntu builds, which do not use
createrepo, passed.
Killing the command is not killing the build. sh() ran the command through /bin/sh,
and cancellation signalled that shell alone -- but dpkg-buildpackage starts workers
of its own, and those survive their shell. The lock was then released while they
were still writing debian/changelog and debian/control, which is the state the lock
exists to prevent: the next build takes the checkout and the two rewrite it
together.
The command now runs in its own process group, so cancellation can take all of it.
Both sides call setpgid, so neither depends on which runs first, and INT and TERM
are blocked across the fork so cancellation cannot land in the window before the
group exists.
Cancellation escalates from the caught signal to KILL, and then CHECKS: a shell that
has exited is not a build that has stopped, so it waits for the whole group to
disappear rather than for the leader to be reaped. If the group is still there after
that, the locks are RETAINED and the process exits non-zero. Releasing a lock while
a worker may still be writing is worse than leaving a lock behind for a person to
clear -- the first corrupts a build, the second stops one.
cancel_build ignores INT and TERM while it runs, so a second Ctrl-C cannot interrupt
the cleanup half way and release the lock early.
sh() also reports a signalled command as 128+signal instead of 0. $? >> 8 is zero
for a child killed by a signal, so a build stopped mid-way looked to its caller like
one that had succeeded.
Two cases added to builddebs_lock_cancellation.t: a build whose worker is a
grandchild, and a command killed by a signal. Verified by signalling the pid instead
of the group, which leaves the worker running and turns the first red.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is released in DESTROY, and perl does not run DESTROY when a signal
ends the process. A build stopped with SIGTERM or SIGINT therefore left its lock
directory behind, and the next build of that checkout died on
FATAL: another build of <path> already holds <dir> (held by [pid=NNNN])
naming a pid that had already exited. Nothing clears it but a person. One such
directory blocked an openSUSE target across three consecutive CI runs before anyone
looked at what the lock actually said.
buildrpms.pl has released its lock on cancellation for some time, through an END
block and an abort handler. This is the Debian builder catching up.
The order matters, and is the reason this is not simply an END block. The command in
flight is stopped BEFORE the lock is released: handing the checkout to a second build
while dpkg-buildpackage is still rewriting debian/changelog and debian/control in it
is worse than holding the lock a moment longer. The wait for that command is bounded,
so a subprocess that ignores the signal cannot hold the lock for ever either.
sh() now forks and execs rather than calling system(), because system() gives no pid
and a handler cannot stop what it cannot name. The child _exits rather than exits, so
it never runs the parent's END block and releases a lock the parent still holds.
The handler is installed by XCAT::BuildUtils::install_build_cancellation rather than
written inline in the builder, so a test can use the same wiring the builder uses. A
test that installs an equivalent handler of its own proves the helper works while
saying nothing about whether anything calls it -- the first version of this test did
exactly that, and passed with the wiring removed.
Release is idempotent: a signal handler and then DESTROY both reach it, and the
second must not remove a directory a LATER build has since taken.
builddebs_lock_cancellation.t terminates the holder, then takes the lock again, and
checks no build subprocess was orphaned. Verified by removing the wiring: assertions
9 through 12 fail, naming the leaked lock, the refused build and the stray process.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The project had no written release process, and the 2.18 and 2.19
releases missed steps: the release table in the documentation, the
docs version, signed tags, and the wiki and website index pages.
A new Releases section describes the branch and version model, the
label and milestone that each pull request needs, and a checklist for
release candidates, publishing, the signed tag, the GitHub release, the
release notes, the website and the announcement. The build hosts, the
signing key and the publish procedure stay in the private repository of
the maintainers.
The OpenEmbedded metadata job ran on every pull request and took about
20 minutes. It reads only xCAT-genesis-builder/oe and
xCAT-genesis-scripts, and it pins its upstream sources to fixed commits.
A pull request that changes neither directory gets the same result each
time.
The job moves unchanged to its own workflow, which runs only when those
paths or the workflow file change. xcat_pr_test stays in xcat_test.yml
and runs on every pull request, because a required check that does not
run blocks the merge.
2.19.0 is released, and master still builds packages that call themselves
2.19.0. Every snapshot built from master since then carries the released
version, so a candidate cannot be told from the release it follows.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build tree can live on an NFS re-export. The kernel refuses locks on one --
"Clients are not allowed to get file locks or delegations from a reexport
server" -- so every flock() there answers errno 524, and a build that takes one
dies before it starts.
buildrpms.pl's per-target lock and BuildUtils.pm's take_build_lock, which
builddebs.pl calls for the Ubuntu core build, are both atomic mkdir claims now.
Each records its owner and names it when it refuses.
A directory is not released by a filehandle closing, which is how both locks
were freed before. buildrpms.pl releases from END, and again in abort_builds
because that handler re-raises the signal with DEFAULT and END blocks do not run
then -- a killed build would otherwise strand the lock for every later one.
BuildUtils returns a small object whose DESTROY releases it, preserving the
caller's "hold the returned value" contract.
Both releases are guarded by owning pid: both scripts fork, and the flock they
replace could not be released by a child.
builddebs_lock.t closed the returned value to prove the lock is released, which
is "Not a GLOB reference" against the new contract. It now lets the value go out
of scope. What it asserts is unchanged: a second build of the same checkout is
refused, and the next one succeeds once the first releases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
createrepo_dir passed --database, which writes *.sqlite.bz2. Building those
needs SQLite, and SQLite needs POSIX locks. A build tree can live on an NFS
re-export, where the kernel refuses locks outright: every attempt answers
errno 524.
So every target died with "Cannot open .repodata/primary.sqlite: Can not create
db_info table: disk I/O error", and the build staged nothing.
Measured on such a share, with a local control: a bare sqlite3 connect fails
there and succeeds on local disk; createrepo_c fails with --database and
succeeds without it, emitting primary/filelists/other as *.xml.zst.
Nothing this project ships reads the sqlite metadata. dnf on el8+ and zypper
both read the XML.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
site.cleanupdiskfullxcatpost names the node type it applies to, and site.5.rst
says "on the diskfull nodes". xcatdsklspost serves every type: updatenode calls
it for a diskless or statelite node as well, and the site value is one row that
reaches all of them. With cleanupdiskfullxcatpost=yes and cleanupxcatpost=no,
append_xcatpost_cleanup still appended the delete for those nodes.
The diskful branch now reads NODESETSTATE from the generated mypostscript and
appends nothing for netboot or statelite. That is the signal remoteshell,
hardeths, configeth, otherpkgs and servicenode already use for this question,
and mypostscript.tmpl exports it.
cleanupxcatpost is unchanged. It names no node type and keeps applying to every
one, which xcatdsklspost_xcatpost_cleanup.bats now asserts so a guard added to
the wrong branch shows up.
Four cases cover it: a netboot node and a statelite node keep their postscripts,
a diskful node is still cleaned, and cleanupxcatpost still empties a netboot
node. The first two fail against the parent commit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The step that fills lib/firmware in the Ubuntu netboot initrd asked modinfo
about $rootimg_dir/$module and looked a firmware name up under lib/firmware.
The copy step beside it takes a module from $customdir or $pathtofiles first,
and the kernel looks a firmware name up under updates/<kernel>, updates/,
<kernel>/ and lib/firmware. So a custom driver reached the initrd with no
firmware, even when the root image carried it, and a firmware override was left
out of the initrd altogether.
initrd_firmware_files now takes the module directories the copy step searches
and the kernel release. It resolves each module in that order before asking
modinfo, and keeps every firmware file that exists in the four directories the
kernel searches, so the override still wins on the node.
ubuntu_genimage_initrd_firmware.t covers both: its two new cases are red on the
commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t drove the firmware step over a root image that
holds every module and every firmware file. It covered neither of the two places
genimage reads from beside the root image.
The test now puts a driver in the custom directory, with its firmware in the
root image, and a firmware override under lib/firmware/updates/<kernel>. Both
are red: the step asks modinfo about the module under the root image, where a
custom driver is not, and it looks for a firmware name under lib/firmware only,
where an override is not.
The five assertions that were there stay green.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
reg_linux_diskless_installation_flat corrupts the KVM machine type, proves the
node fails to boot, restores the machine type and then removes it again. On
ppc64 the two ladders name different machine types: the restore writes
machine:pseries-rhel7.6.0 and the cleanup removes machine:pseries-7.6.0. The sed
matches nothing, so the node keeps machine:pseries-rhel7.6.0 after the case ends
and the next case runs against a node the previous one changed.
The cleanup ladder now names the machine type the restore ladder writes.
diskless_flat_vmothersetting_machine.bats covers it: the ppc64le cleanup cases
are red on the commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
diskless_flat_vmothersetting_machine.bats checked the restore with a substring
match and checked the cleanup only for the absence of "unary operator
expected". A cleanup that writes the machine type back, or leaves it in place,
passed both.
The test now reads the value chdef receives. The restore must write exactly
machine:<type>, and must keep a setting the node already carries. The cleanup
must write an empty value when the machine type is all there is, and must leave
the other setting behind when there is one. The chdef stub brackets its
arguments so an empty value is not the same as no call.
ppc64le is red on the cleanup: the restore ladder writes
machine:pseries-rhel7.6.0 and the cleanup ladder removes machine:pseries-7.6.0,
so the node keeps the machine type. x86_64 and riscv64 pass.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
run_zypper_local_repo extracted six lines of the zypper branch of otherpkgs and
evaluated each one in turn. The branch it measures is an if/else: the repository
is deleted only when the refresh fails. Evaluating the lines separately ran the
delete every time, so the test showed that zypper sd is reachable and never that
a repository which refreshes is kept.
The helper now evaluates the whole branch, from the #use zypper comment to the
apt branch that follows, with pmatch lifted from the same file. The success case
asserts rc=0 and no zypper sd; the failure case keeps its delete assertion.
Flipping the refresh test in otherpkgs to "if [ $? -ne 0 ]" turns both cases
red. The same mutation left the previous helper green, because it never
evaluated that line.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
REQUIRED_PACKAGES named util-linux-extra for every release. focal and jammy
have no such package -- apt reports "Candidate: (none)" -- so apt-get install
exits non-zero and, under set -euo pipefail, the build stops before dracut
runs. hwclock is in util-linux there, which is essential and already present.
optional_packages() keeps a package only where apt has a candidate for it, and
util-linux-extra goes through it. The unconditional list keeps isc-dhcp-client
and ifenslave, which every release has and neither of which the build root
carries by itself.
The call to verify-genesis-payload goes with it. That script is added by the
genesis payload branch, not this one, so the line stopped the build at the
point it was meant to guard.
Also corrects the plan count and a dereference in the test committed before
this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddeb-genesis-base names util-linux-extra in REQUIRED_PACKAGES for every
release. Measured on the four Ubuntu management nodes: focal and jammy report
"Candidate: (none)" for that package and carry hwclock in util-linux, which is
essential and already in the build root; noble and resolute carry it in
util-linux-extra. apt-get install with a package it cannot locate exits
non-zero, and the script runs under set -euo pipefail, so the build stops on
two supported targets before dracut runs.
util-linux only Suggests util-linux-extra, and the install passes
--no-install-recommends, so a release that split the package has to name it.
The test asserts the unconditional list does not name it, and drives the
selector that decides, with apt-cache shadowed for a release that has the
package and one that does not.
It also drops the assertion that matched "verify-genesis-payload" against the
text of the build script. That proved the string was present, not that the
verifier ran, ran before packaging, or stopped the build -- and the script it
names does not exist on this branch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t called plan skip_all when xCAT-server/share/xcat/netboot/ubuntu/genimage was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/share/xcat/netboot/ubuntu/genimage moved aside the file now exits 2 and prints "genimage not found at <path>";
before this change it exited 0 and printed "1..0 # SKIP genimage not found at <path>". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
postscript_heredoc_embedding.t called plan skip_all when xCAT-server/share/xcat/install/scripts was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/share/xcat/install/scripts moved aside the file now exits 2 and prints "<scriptdir> not found";
before this change it exited 0 and printed "1..0 # SKIP <scriptdir> not found". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
debian_install_prescript.t called plan skip_all when xCAT-server/lib/xcat/plugins/debian.pm was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With xCAT-server/lib/xcat/plugins/debian.pm moved aside the file now exits 2 and prints "debian.pm not found";
before this change it exited 0 and printed "1..0 # SKIP debian.pm not found". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs_release_placeholder.t called plan skip_all when builddebs.pl was absent, so a checkout that lost
the file reported 0 tests and exit 0. A test that cannot fail measures nothing.
Die instead, which is what makentp_ntp_deps.t already does for setupntp.
With builddebs.pl moved aside the file now exits 2 and prints "builddebs.pl not found";
before this change it exited 0 and printed "1..0 # SKIP builddebs.pl not found". With the file
present the test passes either way.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Four cases in xCAT-test/bats/postscripts_otherpkgs.bats fail on the integrated
tree. extract_shell_if_block reports that the upgrade anchor occurs 4 times and
the url repository anchor occurs 2 times, where 1 is expected.
"if [ $hasyum -eq 1 ]; then" opens the upgrade block and also the yum branch of
the preremove, the install and the postremove blocks. "OTHERPKGDIR_INTERNET"
matches the guard that opens the url repository block and an assignment inside
that block. The earlier helper took the first match and said nothing.
otherpkgs_block now passes NTH and TOTAL through to extract_shell_if_block. The
two upgrade cases take occurrence 1 of 4, so a fifth identical line fails the
test instead of moving it. The two url repository cases anchor on the guard
line itself, which occurs once.
Both blocks the earlier helper took are the blocks the assertions describe, so
no case measured the wrong code. On the integrated tree bats -r xCAT-test/bats
gives 113 ok and 0 not ok. On this branch, where the helper is not yet
hardened, the file gives 18 ok.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
makedns reports "error was FORMERR" for an update that named rejected with
NOTAUTH. The FORMERR is the answer to the retry, not to the first attempt.
send_ddns_update in ddns.pm signs the packet the caller built, and signs that
same packet again on each attempt. Net::DNS::Packet::sign_tsig appends the TSIG
to the additional section, so the second attempt sends two TSIG records and
named answers FORMERR. FORMERR is neither NOTAUTH nor SERVFAIL, so the routine
stops and reports it. The NOTAUTH and SERVFAIL retry can never be accepted, on
any algorithm.
Each attempt now signs a request of its own. A packet cannot be unsigned again,
so ddns_update_request copies the prerequisite and update records into a new
Net::DNS::Update instead, and the caller keeps the unsigned original.
ddns_update_retry.t fails before this change: the second attempt carries two
TSIG records, and an update that the retry answers with NOERROR still reports
failure.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
send_ddns_update signs the same packet on every attempt. Net::DNS appends the
TSIG to the additional section, so the second attempt carries two TSIG records.
named answers FORMERR to that message, which is neither NOTAUTH nor SERVFAIL, so
the routine stops and reports FORMERR. The retry path can never be accepted.
ddns_update_retry.t drives send_ddns_update with a resolver that answers FORMERR
to a message with more than one TSIG record, as named does, and otherwise
answers a scripted rcode. It asserts that every attempt carries exactly one TSIG
record and the same update records, and that a retry answered NOERROR reports
success. Both subtests fail before the fix.
The header of each new test records that XCATROOT must name the tree under test,
because xCAT::Table adds the installed /opt/xcat/lib/perl to @INC.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
makedns exits 1 on a management node that has Net::DNS below 1.36 and an
hmac-sha256 key, and reports "Failure encountered updating <zone> with entry
'', error was FORMERR".
update_namedconf in ddns.pm rewrites the named.conf key stanza to hmac-md5
whenever Net::DNS is below 1.36, and ddns_tsig_algorithm returns hmac-md5 for
the same reason. ddns_sign_update signs with site.dhcpomapialgorithm, which
xcatconfig sets to hmac-sha256 on EL9 and later. named matches a TSIG key by
name and by algorithm, so it answers NOTAUTH. The retry signs the same packet
a second time, and named answers FORMERR to the two signatures.
The version test protected the two-argument sign_tsig($name, $secret), which
produces an HMAC-MD5 signature only. ddns_sign_update signs every other
algorithm through a KEY RR, so the Net::DNS version no longer selects the
algorithm. This change deletes the rewrite and the version test, and signs with
the algorithm the key stanza declares. OmapiPolicy->algorithm_rr_type maps that
algorithm to its KEY RR number.
ddns_named_key_algorithm.t fails before this change: it reads the stanza as
hmac-md5 where the key was hmac-sha256.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
On a management node with Net::DNS below 1.36, makedns rewrites the named.conf
key stanza to hmac-md5. It then signs the update with the algorithm the site
table selects. named matches a TSIG key by name and by algorithm, so it rejects
every update and makedns exits 1.
ddns_named_key_algorithm.t drives update_namedconf over a scratch named.conf and
then signs one update with the context that run produced. It asserts that the
stanza keeps the algorithm the key was generated with, that the signature uses
that algorithm, and that named is not restarted. Two of its five subtests fail
before the fix.
ddns_omapi_policy.t pinned the rewrite as correct, so its expectations move to
the algorithm the key already has.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A failed package install through the rpm or dpkg fallback leaves no
record in syslog. The same holds for the four package removal branches
that run after the install. The postscript sets its return code, but a
reader of the node log sees nothing, because the package manager output
goes to the console only when VERBOSE is set.
In xCAT/postscripts/otherpkgs the fallback install block, and the yum,
zypper, apt and rpm removal blocks, set RETURNVAL on a non-zero status
and stop there. Only the three repository install branches send a
message with local4.err.
This change adds the same local4.err message to the five branches that
have none. Each message names the command that ran, as the repository
install branches do.
xCAT-test/bats/postscripts_otherpkgs.bats drives each block with a
package manager that returns a failure. Two cases assert the message is
present, and both fail on the parent commit. Two more assert the
success path still logs "installed." and "removed." alone.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The otherpkgs postscript writes two diagnostic lines for each package
sublist: the split package array with its size, and the detected package
manager. On a list with tens of entries these lines fill the updatenode
output and the node log, and hide the install results.
The two echo commands sit in xCAT/postscripts/otherpkgs, after the
IFS split that builds pkgsarray, and no condition guards them. Every
other diagnostic in the script runs only when VERBOSE is set.
This change puts the two lines in the same "if [ $VERBOSE ]" block the
rest of the script uses.
xCAT-test/bats/postscripts_otherpkgs.bats runs the split and the lines
that follow it. One case asserts no output when VERBOSE is empty, and
fails on the parent commit. A second case asserts both lines are still
printed when VERBOSE is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every diskfull install stopped in the installer. The node installed the OS and
answered ping, but no postscript ran, so remoteshell never wrote the root key
and the management node got "Permission denied (publickey,password)". The node
stayed at status powering-on and reinstalled in a loop.
append_xcatpost_cleanup in xCAT/postscripts/xcatdsklspost writes the cleanup
with a here-document whose delimiter is EOF, so the file holds a line "EOF".
post.xcat, post.xcat.ng, post.xcat.rhels10, post.debian and the two s390x
scripts embed the whole file with "#INCLUDE:<path>#" inside "cat
>/opt/xcat/xcatdsklspost << 'EOF'". That line ends the outer here-document
early, and the remaining 890 lines of xcatdsklspost become shell code in the
generated install script. bash reports a syntax error and curtin fails the
late-command that runs it.
The here-document now ends at XCATPOST_CLEANUP, and a comment states the
constraint the file is under.
postscript_heredoc_embedding.t assembles each embedding and parses it. It fails
on the six install scripts without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install-time post script embeds xcatdsklspost, xcatinstallpost and
xcatpostinit1 with "#INCLUDE:<path>#" inside a here-document. The template
copies each file verbatim, so a line in the postscript that equals the
here-document delimiter ends that here-document early, and the rest of the
postscript becomes shell code in the generated install script.
Nothing measured that. The new test assembles every such embedding the six
post.* install scripts declare, and asserts that bash parses the result and
that no line of the embedded file equals the delimiter.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The local otherpkgs repository is written with the alias xcat-otherpkgs$localrepoindex, where
localrepoindex is urlrepoindex plus the array index. Lines 833 and 843 of
xCAT/postscripts/otherpkgs then refresh and delete xcat-otherpkgs$index. With http OTHERPKGDIR
entries present urlrepoindex is not zero, so zypper names a repository that block did not add:
the refresh fails for a repository that exists, and the delete removes another one.
Both lines now use $localrepoindex. The log and the echo of a failed SDK repository add name
$bname, which is what zypper ar used; they said bname without the $.
postscripts_otherpkgs.bats drives the add, the refresh and the delete with zypper shadowed and
urlrepoindex set to 2, and reads the alias back out of the repository file the script wrote.
Against the unfixed script the refresh and the delete name xcat-otherpkgs0 while the file says
xcat-otherpkgs2, and the SDK log carries no repository name.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão
Gary Skouson