The apt publish reads staging/<codename>/<arch> of every expected arch,
but takes only the run lock of the host arch. A ppc64el run can refill
its staging while the publish assembles from it.
Finalize runs on one host and takes the cell locks of both arches. If it
dies, only that host can reclaim the locks of the other arch's cells,
and the builds that own those cells wait on them and fail.
These tests fail until the fix: the publish must refuse while any
expected arch holds its run lock, and finalize --finalize-arch must
write and lock only the cells of that arch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
XCAT::NFSLock recovered a dead owner's lock under a second lock,
lock.break. A process that died after it created lock.break left it in
place, and every later acquire timed out even when both processes were
dead. The lock and its record were also put in place with rename, which
replaces an empty directory, so acquire took over a lock.d that had no
owner record.
The lock now follows the NFS lock protocol at the top of NFSLock.pm. It
uses mkdir, rmdir, unlink and plain writes, and no rename. lock.d/metadata
holds the owner identity and a hash, and invalid metadata only causes a
retry. Takeover and release both hold lock.borrow, beside lock.d. Only a
process on the owner's machine takes over a lock, after it reads the
same identity twice and proves that owner dead. acquire retries R times
with a wait of T +/- jitter, and --try-unlock-timeout maps to R.
Records written by the previous format are invalid metadata and are not
taken over. A process that dies while it holds lock.borrow still blocks
the lock. The protocol assumes that this does not happen.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
payload.sha256 differs from the bare release tree in the two shims, and
matches the tree once the committed ipxe/shim shims replace them, as the
builders install them. SHA256SUMS covers the shims with the archives.
The next commit installs the ipxe/shim 16.1 shims over the shims of the
release tree, so payload.sha256 no longer describes the bare tree. The
commit after it checks the tree with the shims in place.
The per-arch runs of one dep build share --repo-dep and each took
<repo-dep>/.lock. The first run won and the others died, so the
pipeline passed --force-unlock. That option removed any lock it found:
a rerun of the same refs could take a cell from a run still writing it,
and common recovery could remove the staging tree of a live publisher.
mockbuild-all.pl now takes XCAT::NFSLock locks: <output>/.lock, one
<repo-dep>/rh<N>/.<arch>.lock per target, and .common-publish.lock while
it recovers or publishes common. --finalize-xcat-dep locks the cells it
rewrites. --try-unlock-timeout N replaces --force-unlock. A run that is
not a dry run recovers an interrupted common publication when no other
run holds the common lock.
sbuild-all.pl takes its per-arch run lock and its apt publish lock as
XCAT::NFSLock locks instead of flock, which fails on the shared tree.
createrepo_c runs without --database. SQLite needs locks, which a client
of an NFS re-export cannot take.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The dep builds lock trees on a shared NFS tree that the hypervisors
re-export. The kernel refuses flock and fcntl locks to clients of a
re-export, so every such lock fails with errno 524.
XCAT::NFSLock builds a lock as a directory under a private name, with an
owner record and the caller's metadata, and renames it into place. The
record holds the machine id, boot id, pid, process start time and a
random token. The owner removes the lock, or a process on the owner's
host that proves the owner dead: another boot, no such pid, or a pid
with another start time. That process takes a breaker lock and proves
the owner dead again before it removes the lock. Removal renames the
lock away first. Any other acquire waits up to a timeout, retrying every
3 s by default, then fails with the mv command that moves the lock away.
A lock left by mockbuild-all.pl before this module is taken when its
pid is gone on this host.
t/nfslock.t covers each rule without NFS.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Require ipxe-xcat in every EL target that lists xnba-undi, at the same pin
on riscv64 as on ppc64le, and on every Debian target. Hold its Debian pin
to its changelog, and check that its control file keeps it an
Architecture: all package built on amd64 only.
Damage a copy of a fixture tree one way at a time: a changed byte, a
missing file, an extra file or directory, a symlink with another target,
and a symlink replaced by a copy of its target. Each must fail the check
and name the path. Also hold the committed payload.sha256 and SHA256SUMS
to the committed archives.
An AlmaLinux host resolves to the alma config file, an id that names its
file resolves to it, the os-release id wins when both files exist, and a
release with no config file is an error that names the files it tried.
The riscv64 goconserver build never finished. It installs a Go toolchain built FOR the
chroot architecture, so on riscv64 the compiler itself runs under qemu-user: three
xcat-dep-ubuntu-cd cells (jammy, noble, resolute) each burned the full 9000s budget, two
of them with no CPU ticks at all in the stall sample.
Nothing asserted which toolchain the build fetches, or which architecture it compiles for.
This test lifts the build shell out of goconserver/sbuild.pl and runs it with the commands
it calls shadowed, then asserts on what the run asked for: the toolchain tarball must name
the build host, and the real debian/rules must reach `go build` with GOARCH set to the
chroot architecture. The recorders refuse any write outside the scratch tree and report it,
so the build's `rm -rf /usr/local/go` is an assertion here rather than damage to the host.
It fails on the current builder: the toolchain is fetched for riscv64, GOARCH is unset and
two writes escape the build tree.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Five files under t/ call BAIL_OUT at eleven places: a missing command, a
manifest section that is not there, an extraction that stopped matching,
a run_bounded that never returned. prove stops every remaining file on a
bail-out, not only the file that called it, so one of these hides the
results of every test that would have run after it. die is just as loud
and costs only its own file.
The header of genesis_native_deb.t also retold how an EL image reached an
Ubuntu node. The fallback and its effect are one sentence.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
convert_genesis_rpm extracted xCAT-genesis-base-<arch>.noarch.rpm with rpm2cpio
and repackaged it as a deb. That rpm carries an EL kernel and EL kernel modules,
so every Ubuntu management node installed an image built for another
distribution. The pipeline took that path on every run, because GENESIS_BASE_RPM
is what it passes.
xcat-core builds the deb natively now, one per Ubuntu codename. Drop
--genesis-rpm, --genesis-rpm-ppc, --require-ppc-genesis, convert_genesis_rpm,
maintained_genesis_control and genesis_deb_control, and --xcat-source with them:
the maintained control and the maintainer scripts come from the real
dpkg-buildpackage now, not from a hand-assembled DEBIAN/control.
The images differ per release, so build_genesis stages each one into the suite it
was built for. genesis_debs_for_codename reads the codename back from the deb
version, and a run that publishes a codename with no image for it stops instead
of serving another release's image.
genesis_native_deb.t fails without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
sbuild-all.pl gets its Genesis deb one of two ways. --genesis-deb ingests a
native deb. --genesis-rpm and --genesis-rpm-ppc convert the EL rpm with rpm2cpio,
and that is the path the pipeline takes, so an Ubuntu management node installs an
image built from an EL kernel.
xcat-core now builds one Genesis deb per Ubuntu codename, each carrying that
release's kernel, with the codename in the version. build_genesis stages every
Genesis deb it holds into every suite, so three images would land in each one and
apt would serve the newest, which belongs to another release.
genesis_native_deb.t asserts the deb each suite takes, and that the three rpm
options are gone. It fails on this commit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
master added the common repository gate to the workflow and rewrote the
riscv64 comment of packages-manifest.conf after this branch started.
The workflow keeps both new prove lines. The two tests are unrelated.
The riscv64 comment takes master's text. This branch said the x86 boot
components are not built for riscv64; master then listed elilo-xcat,
syslinux-xcat and xnba-undi in the cell, because a riscv64 management
node serves the x86 nodes of a mixed cluster, so the branch sentence is
no longer true. The last line keeps this branch's statement: the perl
set is the EL10 one plus perl-HTML-Form. The pin itself merged without
a conflict, and t/riscv64_perl_cell.t passes on the merged cell.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
master added the per-cell architecture rule (rpm_arch, rpm_in_cell) after
this branch started, so the two sides collide in three places.
The MockBuildUtils and t/mockbuild-all.t import lists take both sets of
names. Neither side removes a name the other needs.
MockBuildUtils.pm now held two definitions of rpm_arch, one from each
side, and Perl kept the later one. master's definition stands: it reads
the header of a file and falls back to the name suffix otherwise, which
is what its own tests and rpm_in_cell need. The branch definition is
removed. carry_over_rpms takes rpm_arch as an argument, so it keeps its
own architecture gate and its own message.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
skipped_builder must place a source package with the builder required_pkgs
would skip and never claim the OpenEmbedded Genesis. carry_over_rpms must
keep every binary rpm of a skipped build whose source package the target
manifest still names, leave out whole any build the run already carries a
member of, and
die on an unreadable header, an unsigned or foreign-architecture member of
a selected build, or a package published at two versions. Paths with
spaces are covered. Both fail to import against the previous module.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
The riscv64 cell must carry elilo-xcat, grub2-xcat, syslinux-xcat and
xnba-undi at the pins of the EL10 ppc64le cell. 3 assertions fail against
the previous manifest.
The manifest consistency check covered the amd64 and ppc64el sections. It
now covers the riscv64 sections too, so the 4 boot components must be listed
there. 12 assertions fail against the previous manifest.
The assertion fails against the previous manifest, naming perl-HTML-Form as
the package the cell lacked. It reads the set from the builder, so a package
added to the builder later fails here until the cell names it.
The probe starts a descendant, kills the leader with SIGKILL, and asserts the
descendant is gone when the call returns. It fails against the previous
run_bounded, where the descendant survived.
The assertions cover a normal exit, the highest exit code, and death by TERM,
KILL and a core-dumping signal. run_bounded is driven through the same helper,
so the decoding and its caller cannot disagree.
The probe sends itself a signal while the mask is held and asserts it arrives
only once the mask is restored, which is what makes the fork and the pid
registration a single uninterruptible step.
The race itself is not reproducible without a hook in the production path, so
the assertions pin the invariants the fix establishes: the build starts with the
handled signals unblocked, and the caller keeps them unblocked afterwards.
Two real processes stand in for a worker and the build it runs. The probe fails
when the handler forwards to nobody, which is what the orchestrators did.
Six of the assertions fail against the previous build_deb_in_chroot, which
accepted a smoke argument it never acted on: a wrong version, a binary that
exits non-zero, and a smoke naming a deb the build never produced all passed.
The two assertions fail against the previous package list and pass with it,
so a later edit cannot drop the binfmt handler that a foreign chroot needs.
The command records its own pid and sleeps well inside the budget, so the
bound cannot be what ends it; the wrapper is then terminated and the pid
probed. Without the forwarding the build survives.
Drives the check extracted from sbuild-all.pl with the handler node
redirected into a temporary tree, covering the native case, a foreign
target with no handler, and the same target once one is registered.
Drives the scan extracted from sbuild-all.pl against a staged tree holding
every supported architecture plus an unsupported one, so the assertions
read the set publish would carry forward rather than a copy of the rule.
A riscv64 goconserver `go build` sat 26 minutes with zero CPU ticks across a
20-second sample, both Go pids in futex_wait and no socket open. Nothing bounds
a build step, so the cell did not fail -- it hung, and a hung run reads as
"still running" rather than as a defect.
t/build_timeout.t drives the bounded path with a command that hangs and asserts
that the call returns, reports a timeout, and prints the process tree, each
pid's wchan, the open socket count and a CPU-tick sample. A second case drives a
spinning command and asserts the report calls it slow, not deadlocked, so the
sample means something.
Each call under test runs in a forked child whose stdio is detached to a file,
and the parent bounds that child. An unbounded run must fail this test, not
block prove.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Ubuntu build fails at the end, after compiling every package:
FATAL: manifest validation failed:
[noble-amd64] grub2-xcat: built 2.12-2, manifest pins 2.12-1
debs-manifest.conf pins the exact deb version each package must produce, and that
version comes from the package's own debian/changelog. Bumping the changelog
without the pin costs a whole build to discover a one-line edit.
The test compares every non-glob pin with the first line of that package's
debian/changelog. Globbed pins are deliberate -- goconserver's revision is the CD
stamp and xcat-genesis-base is not versioned by xcat-dep -- and are skipped. It
fails on grub2-xcat today.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The architecture-coverage block was appended at the end of t/sbuild-all.t, past
done_testing. Test::More had already declared the plan, so the run ended with
"planned 194 tests but ran 197" and the three assertions counted for nothing.
Move the block above done_testing.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ipmitool-xcat fails to build on riscv64:
dh: warning: No packages to build. Possible architecture mismatch:
riscv64, want: i386 amd64 ia64 ppc64el
make: ./configure: No such file or directory
Its debian/control names architectures explicitly, and riscv64 is not in the
list, so debhelper builds nothing and the build dies at configure. conserver and
goconserver say Architecture: any and are unaffected.
The test reads each compiled dep's debian/control and asserts that an explicit
architecture list covers every arch BuildUtils::supported_arches names. It fails
on ipmitool today.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão