The race itself is not reproducible without a hook in the production path, so
the assertions pin the invariants the fix establishes: the build starts with the
handled signals unblocked, and the caller keeps them unblocked afterwards.
Two real processes stand in for a worker and the build it runs. The probe fails
when the handler forwards to nobody, which is what the orchestrators did.
Six of the assertions fail against the previous build_deb_in_chroot, which
accepted a smoke argument it never acted on: a wrong version, a binary that
exits non-zero, and a smoke naming a deb the build never produced all passed.
The two assertions fail against the previous package list and pass with it,
so a later edit cannot drop the binfmt handler that a foreign chroot needs.
The command records its own pid and sleeps well inside the budget, so the
bound cannot be what ends it; the wrapper is then terminated and the pid
probed. Without the forwarding the build survives.
Drives the check extracted from sbuild-all.pl with the handler node
redirected into a temporary tree, covering the native case, a foreign
target with no handler, and the same target once one is registered.
Drives the scan extracted from sbuild-all.pl against a staged tree holding
every supported architecture plus an unsupported one, so the assertions
read the set publish would carry forward rather than a copy of the rule.
A riscv64 goconserver `go build` sat 26 minutes with zero CPU ticks across a
20-second sample, both Go pids in futex_wait and no socket open. Nothing bounds
a build step, so the cell did not fail -- it hung, and a hung run reads as
"still running" rather than as a defect.
t/build_timeout.t drives the bounded path with a command that hangs and asserts
that the call returns, reports a timeout, and prints the process tree, each
pid's wchan, the open socket count and a CPU-tick sample. A second case drives a
spinning command and asserts the report calls it slow, not deadlocked, so the
sample means something.
Each call under test runs in a forked child whose stdio is detached to a file,
and the parent bounds that child. An unbounded run must fail this test, not
block prove.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Ubuntu build fails at the end, after compiling every package:
FATAL: manifest validation failed:
[noble-amd64] grub2-xcat: built 2.12-2, manifest pins 2.12-1
debs-manifest.conf pins the exact deb version each package must produce, and that
version comes from the package's own debian/changelog. Bumping the changelog
without the pin costs a whole build to discover a one-line edit.
The test compares every non-glob pin with the first line of that package's
debian/changelog. Globbed pins are deliberate -- goconserver's revision is the CD
stamp and xcat-genesis-base is not versioned by xcat-dep -- and are skipped. It
fails on grub2-xcat today.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The architecture-coverage block was appended at the end of t/sbuild-all.t, past
done_testing. Test::More had already declared the plan, so the run ended with
"planned 194 tests but ran 197" and the three assertions counted for nothing.
Move the block above done_testing.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ipmitool-xcat fails to build on riscv64:
dh: warning: No packages to build. Possible architecture mismatch:
riscv64, want: i386 amd64 ia64 ppc64el
make: ./configure: No such file or directory
Its debian/control names architectures explicitly, and riscv64 is not in the
list, so debhelper builds nothing and the build dies at configure. conserver and
goconserver say Architecture: any and are unaffected.
The test reads each compiled dep's debian/control and asserts that an explicit
architecture list covers every arch BuildUtils::supported_arches names. It fails
on ipmitool today.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Master gained the EL10 riscv64 forcearch target (PR #66) and the Perl Ubuntu
build (PR #63). Both touch the files this branch rewrites, so the merge is
resolved per file:
mockbuild-all.pl, mockbuild-perl-packages.pl keep the forcearch target profile,
the noarch chroot and --epel-gap from master, and this branch's manifest gate,
atomic per-cell deploy and per-package chroot scrub. assert_required_deps is
dropped: verify_target_repo replaced it. The post-join bootstrap scrub now reads
the uniqueext and the config recorded when the chroot was made, because the wave
loop of master no longer numbers packages in @packages order.
packages-manifest.conf gains a [rocky-10-riscv64-xcat] section. A target with no
section is fatal, so without it the riscv64 target cannot run.
goconserver/mockbuild.pl builds in the mock chroot for the host arch and cross-
compiles on the host for a foreign --target-arch. A forcearch chroot would run
the Go toolchain under qemu. Both paths overlay the pinned go.mod/go.sum and
ship server.conf as YAML.
The host-install smoke stays removed (it corrupts the build host rpm database).
The checks that do not install on the host are kept: the chroot install of a
cross-built ipmitool-xcat and XS perl module, and the binfmt run of the cross-
built goconserver binaries.
goconserver/gomod/ takes the pin of master, whose `go` directive is the lower of
the two, so both the EL10 chroot and the Ubuntu sbuild toolchain accept it.
t/genesis_openembedded_consumer.t: the skip count of the RPM block is 64, the
number of tests it runs. Both sides carried a stale number.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A CD run died at compile time inside XCAT::BuildUtils because xcat-master-ub was
missing File::Slurper: "Can't locate File/Slurper.pm in @INC", in the middle of a
build. It was fixed by hand, so the next unprovisioned host fails the same way
and the documented install line can drift from what the code actually loads.
--install-deps installs this host's prerequisites and exits: the sbuild/schroot
toolchain plus the modules. It then LOADS each module and fails naming any that
is still missing, rather than trusting apt's exit code.
That probe earned its place immediately: the first list named libipc-cmd-perl,
which does not exist on Ubuntu -- IPC::Cmd is core there -- and apt failed the
whole install over it. The package is gone from the list and the module is
asserted by loading instead, with a test that pins both halves of that reasoning.
The list and the command are pure functions in BuildUtils, so the decision is
unit-tested and the side effect stays in the caller. Run on xcat-master-ub and
xcat-master-ub-ppc: both report every module present.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Two CD runs died at compile time inside XCAT::BuildUtils because a builder was
missing a Perl module the script loads: perl-File-Slurper on xcat-master-ub and
perl-IPC-Cmd on xcat-master-ppc. Both surfaced as "Can't locate ... in @INC" in
the middle of a build, and both were fixed by hand -- so the next unprovisioned
host fails the same way, and BUILD.md's install line can drift from what the code
actually requires.
--install-deps installs this host's prerequisites and exits: the toolchain plus
the modules, through dnf or zypper as the host's ID dictates. It then LOADS each
module and fails naming any that is still missing, rather than trusting the
package manager's exit code -- a package that installs cleanly but leaves the
module unusable is exactly the failure this exists to prevent.
The list and the command are pure functions in MockBuildUtils, so the decision is
unit-tested (package sets per family, the right installer non-interactively, and
the probe reporting only what genuinely cannot be loaded); the side effect stays
in the caller. Run on xcat-master and xcat-master-ppc: both report every module
present. xcat-master-suse is unreachable and still needs it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every package in a suite pool is gated against debs-manifest.conf, but the
OpenEmbedded Genesis release is published into pool/main/xcat-genesis-openembedded
-- one pool every suite indexes, described by no [<codename>-<arch>] section. So
nothing asserted the published pool was complete: its packages were checked only
as they were copied, against the release checksums, and a pool that lost one
afterwards would publish quietly.
[shared] describes that pool -- all seven architectures, pinned '2.*' like
xcat-genesis-base, because they are built FROM xcat-core and their version walks
with it. A glob rather than an EVR floor: pins in this manifest are exact-or-glob
(version_matches), and the '>= epoch:version-release' form is an EL-side feature
of packages-manifest.conf. verify_shared_pool runs on the side tree before the
swap, so an incomplete pool is never published.
[shared] is not a build target, so the manifest now has two kinds of section. No
code iterates sections blindly, but t/sbuild-all.t did -- twice -- so it now
selects <codename>-<arch> sections and asserts the shared-pool section is not
treated as a target. The consumer fixtures carry the shipped [shared] section
verbatim: publishing a release against a manifest that lacks it is refused, not
silently ungated.
Verified the gate fails when removed, and when [shared] and the pool disagree.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every package in the per-EL cells is gated against packages-manifest.conf, but
the OpenEmbedded Genesis release is published into xcat-dep/common, which sits
BESIDE those cells and is described by no [<target>] section. Nothing asserted
the published shared repository was complete: its packages were checked only as
they were copied, against the release checksums, so a repository that lost one
afterwards would publish quietly.
[common] describes that repository -- all seven architectures, floored at the
paired xcat-core version (>= 2.18.0; these carry no Epoch, unlike
xCAT-genesis-base). verify_common_repo runs on the STAGE, before the atomic swap,
so an incomplete shared repo is never published. Completeness only: the release
checksums cover the bytes and the deploy asserts every signature.
[common] is not a build target, so the manifest now has two kinds of section.
No code iterates sections blindly, but t/mockbuild-all.t did, and asserted
conserver-xcat in every one; it now selects target-named sections and asserts
the shared-repo section is NOT treated as a target.
t/common-repo-gate.t drives the real publish path and asserts on the repository
left behind. Verified it fails without the gate, and that dropping an
architecture from [common] is caught.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The manifest pinned most packages by %{VERSION} alone, so the gate accepted an
rpm with the right Version and a Release older than xCAT will install against --
xCAT states several of these as ">= version-release" Requires (PR #62 review).
Converted, in every section, to the floors taken verbatim from xcat-core's specs:
goconserver >= 0.3.3-snap202011021058, xnba-undi >= 1.21.1-1,
syslinux-xcat >= 6.03-1, ipmitool-xcat >= 1.8.18-4 (xCAT.spec / xCATsn.spec),
perl-HTTP-Async >= 0.30-3, perl-Net-HTTPS-NB >= 0.14-3 (xCAT-server.spec). Where
two specs disagree the stronger floor is used. Each was checked against the EVR
this repository actually builds, using the gate's own comparator, so none of them
reds a build that is in fact correct.
grub2-xcat is deliberately left on its Version pin. xCAT-server asks for
'>= 2.02-0.76.el7.1.snap201905160255', but the grub2-xcat built here -- and shipped
by both published channels today -- is 1.0-2, which cannot satisfy it. Encoding
that Requires would fail every build over a discrepancy that lives in xcat-core,
so it is documented in the manifest header and reported upstream instead.
t/mockbuild-all.t now guards the shipped manifest: every release-sensitive
package keeps an EVR floor in every section, and grub2-xcat stays the documented
exception. Verified the guard fails when a floor is regressed to a bare version.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
verify_target_repo filtered the manifest through required_pkgs() with the
invocation's --skip-genesis / --skip-perl / --skip-xcat-dep, so the flags that
describe what a run BUILT also decided what the verified repository was allowed
to lack: a repo with no xCAT-genesis-base passed whenever the verifying run
carried --skip-genesis (PR #62 review). Those flags mean "this invocation did not
build it", never "the repository may ship without it" -- a package an earlier run
produced is still expected to be present. The gate now takes the manifest whole.
No change for the CD pipeline, which passes no package-selection skips; it closes
the hole for the documented skip-mode and finalize invocations.
The Genesis-release consumer fixtures now pass --no-verify-repo. Their dependency
packages are copies of a single rpm, so no manifest describes them the way a real
one describes a real build -- with the gate honest, a fixture manifest could only
be satisfied by lying about what the cell contains. The gate is covered instead
against purpose-built rpms in t/verify-repo-el.t, and those runs still need a
manifest SECTION to exist, which is all they ever needed.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
verify_target_repo filters the manifest through required_pkgs() with the
invocation's --skip-genesis / --skip-perl / --skip-xcat-dep, so the flags that
say what THIS run built also decide what the verified repository is allowed to
be missing. A repo with no xCAT-genesis-base passes when the verifying run was
given --skip-genesis (PR #62 review).
Drives the real `mockbuild-all.pl --verify-repo` over fixture repos built from
two minimal rpms, so the gate reads real header names. The assertions are on the
reported problems rather than the exit code: a standalone --verify-repo demands a
repomd signature by contract and these fixtures are unsigned, so it exits
non-zero either way -- what separates a working gate from a broken one is
whether the missing package is NAMED. The complete-repo baseline is asserted too,
so the test cannot pass by the gate simply always complaining.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The documented publish-only invocation is
`sbuild-all.pl --skip-build --skip-genesis --publish`, and verify_assembled_repo
passes those same flags to required_pkgs() when deciding what the PUBLISHED
repository must contain. So the flags that say what this INVOCATION built also
decide what the repository is allowed to be missing, and a repo carrying no
xcat-genesis-base passes its own publication gate.
Assert the gate on a repo missing Genesis (with --skip-genesis) and on one
missing a compiled dep (with --skip-xcat-dep). Both fixtures keep native stanzas
for the arch, so the failure under test is the missing PACKAGE and not the arch
reading as absent. Both fail on the current gate.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A CI run of this branch died on resolute/ppc64el with a 404 fetching
libssl-dev_3.5.5-1ubuntu3.4_ppc64el.deb: a development suite rolled openssl and
dropped that version from the pool while the chroot's index still named it.
mk-build-deps was the one apt operation in the in-chroot script NOT wrapped in
apt_retry, so a single transient mirror inconsistency failed the package -- and,
with the matrix running failFast, took the other architecture's in-flight builds
down with it.
Retry it the same way the rest of the script retries apt, refreshing the index
between attempts, since a stale index is precisely what produces this. It stays
FATAL once the attempts are spent: a package must never build against whatever
the chroot happens to carry. The refresh goes through apt_retry, so every
apt-get in the script still runs under the fatal helper.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #65 changed where the OpenEmbedded Genesis packages live. On the RPM side
they are published once under xcat-dep/common; on the APT side, once under
pool/main/xcat-genesis-openembedded, with every suite's Packages index pointing
at that one copy instead of each suite carrying its own. It made that change in
build-apt-repo.sh -- the script this branch deletes, having absorbed apt assembly
and signing into sbuild-all.pl -- so the layout is ported here rather than lost.
sbuild-all.pl:
- --genesis-release now rebuilds a single shared pool inside the side tree and
indexes it into every suite, so a release is stored once rather than once per
codename. The debs are Architecture:all and identical everywhere; the previous
per-suite copy multiplied hundreds of megabytes by the number of suites.
- Because every suite's index points into that pool, publishing a release must
cover every suite: a run whose --dists omits one is refused, instead of leaving
that suite indexing files the new release retired.
- OpenEmbedded Genesis debs are dropped from suite pools unconditionally now,
not only when a release is being published -- they belong to the shared pool.
- Published files get an explicit mode 0644: they are served by a web server
running as another user, and inheriting the builder's umask is how that breaks.
master's transaction machinery (per-file backups, --force-unlock recovery of an
interrupted publisher) is deliberately NOT ported: it exists because the shell
publisher writes into the live repository. sbuild-all.pl assembles a complete
side tree, gates it, and renames it into place under one global publish lock, so
a failed or killed run leaves the published repository untouched and there is no
half-written state to recover. The tests assert that guarantee directly.
The APT consumer tests now drive sbuild-all.pl's publish path with master's new
expectations: the shared pool holds one complete release, suite pools hold none
of it, every suite/arch index carries the shared Filename, a later single-suite
rebuild keeps using the pool, a partial-suite release is refused, and a
publication that cannot be signed leaves the packages, indexes and key exactly
as they were.
Full suite green: 416 tests on xcat-master (rome01, EL10) and 414 on
xcat-master-ub, where the APT cases actually run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #65 moved the OpenEmbedded Genesis release out of the per-EL cells: it is now
published once into <repo-dep>/common, under its own repository lock, and each
per-EL deploy only drops stale OpenEmbedded packages. That rewrites the same
deploy path this branch rewrites, so the merge had to reconcile the two.
Resolution:
- deploy_target keeps this branch's staged, verified, atomically swapped cell,
and adopts master's model inside it: the release is no longer installed here,
so the stage only calls remove_genesis_packages before it is signed. master's
sign/index/write-metadata already run on the stage, and assert_required_deps
stays gone -- verify_target_repo, which also asserts pinned versions and every
rpm signature, replaced it.
- master's lock and publication state (@HELD_LOCKS, the common stage/backup
globals, the INT/TERM/HUP handlers) replaces this branch's single $HELD_LOCK,
which the merged lock helpers no longer use.
- rpmsign takes master's explicit --define %__gpg, and the redundant bare
`use FindBin` is dropped: this branch already imports $RealBin from it.
- BUILD.md keeps master's note that per-target tarballs exclude xcat-dep/common;
the --skip-xcat section stays out, that flag being gone from this branch.
master's new consumer tests needed the same two adaptations as the ones merged
last time, for the same reasons: test_signed_common_rpm_repository and
test_rpm_repository_lock point --repo-root at a scratch tree, which on this
branch must carry a packages-manifest.conf section for the synthetic target, and
every run's --skip-xcat became --skip-genesis -- this branch removed --skip-xcat
(Getopt::Long silently prefix-matched it to --skip-xcat-dep) and --skip-genesis
is what makes xcat-core's buildrpms.pl unnecessary.
Green as root: 325 tests on xcat-master (rome01, EL10) and 327 on
xcat-master-ub, which also runs the APT cases. perlcritic is clean over the
gated file list, and build-apt-repo.sh still passes bash -n.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #64 landed on master and rewrote the same regions of mockbuild-all.pl this
branch rewrites: it moved the shared helpers into lib/XCAT/BuildUtils.pm
(sh_quote -> shell_quote, print_step, require_command, run_command), added the
--genesis-release publication path to deploy_target, and added
assert_build_progress. Left unmerged, this branch cannot land, and merging it
naively would drop the Genesis release packages from the deployed cell.
Resolution:
- Imports: MockBuildUtils (this branch) and XCAT::BuildUtils / XCAT::GenesisRelease
(master) coexist. print_step is imported only from MockBuildUtils -- the two
implementations are identical, and importing both would redefine it. The local
require_command is dropped in favour of the imported one.
- deploy_target: master's --genesis-release steps (remove_genesis_packages and
verify_genesis_release_packages) now run inside this branch's staged, atomic
cell, before sign_and_index_repo, because rpmsign rewrites the very bytes the
release checksums cover. Files are staged with master's mode-preserving
publish_file.
- assert_build_progress is kept, and is called from both the serial and the
parallel path before the failed step ids are returned. The zero-tolerance
manifest gate (verify_target_repo) still runs afterwards, so a run whose
builders all failed now says exactly that instead of naming missing packages.
- assert_required_deps and have_rpm are gone: verify_target_repo, which asserts
the manifest, the pinned versions and every rpm signature, replaces them.
t/genesis_openembedded_consumer.t needed three adaptations to run against this
branch at all, none of them optional:
- Its runs use a synthetic target, and this branch builds and gates every target
against <repo-root>/packages-manifest.conf, where a target with no section is
fatal. Each run now writes that section into its scratch repo root.
- They passed --skip-xcat, which this branch removed (Getopt::Long silently
prefix-matched it to --skip-xcat-dep) and which was what let buildrpms.pl be
absent. They pass --skip-genesis instead: that is the only step needing it now.
- --skip-build collects from this target's own build-results tree here, where it
used to collect from the legacy build-output/list* directories. The
"a run that built nothing" fixture had left an rpm in build-results, which is
now genuinely collected, so that case leaves the tree empty and a new case
covers --skip-build keeping (and collecting) the results it reads.
Full suite green as root: 238 tests on xcat-master (rome01, EL10) and 244 on
xcat-master-ub, which additionally runs the APT consumer cases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #64 landed on master and added --genesis-release to build-apt-repo.sh -- the
very script this branch deletes, having absorbed the apt assembly + signing phase
into sbuild-all.pl. A plain merge would either resurrect the shell publisher or
silently drop the OpenEmbedded Genesis release from every apt suite, so the
feature is ported to where apt publication now lives.
sbuild-all.pl --genesis-release <dir>:
- The release is validated once at startup, before any build or publish, with the
same checksum-verify-checksum sequence mockbuild-all.pl uses on the rpm side, so
a release rewritten together with its SHA256SUMS while the verifier runs is
rejected. It must be complete (every supported architecture) and carry debs.
- During assemble_into, each release deb is copied into the codename's pool and the
flat per-version directory and re-checked against the verified checksums. That
happens with the publish lock held, between the pool wipe and apt-ftparchive, so
the bytes that are indexed and signed are the bytes that were verified -- the
separate re-verification pass build-apt-repo.sh ran before indexing has no
window left to cover here.
- Copies are plain copies, never link(): a pool file sharing an inode with the
release would let a write through either path change what the other holds.
- Anything staged under the OpenEmbedded Genesis package name is dropped when the
option is given; the verified release is the only source of those packages.
- XCAT::GenesisRelease is loaded on demand rather than imported at compile time. It
pulls in XCAT::BuildUtils, which needs File::Slurper, and xcat-master-ub does not
carry it: a compile-time import made every apt build -- including the ones that
never pass --genesis-release -- die with "Can't locate File/Slurper.pm".
Also here:
- --publish-lock-wait <seconds> makes the 1800s publish-lock wait settable, so a
caller that would rather fail fast than queue can, and so the lock is testable.
- t/genesis_openembedded_consumer.t: the four APT consumer tests now drive
sbuild-all.pl's real publish path (staging tree, publish lock, atomic swap)
instead of build-apt-repo.sh, including the new flock-based lock behaviour.
- The workflow compiles sbuild-all.pl and BuildUtils.pm instead of shellchecking
the removed script; BUILD.md and genesis-openembedded/README.md document the apt
invocation.
Full suite green on both build hosts: 345 tests on xcat-master-ub (Ubuntu 24.04,
where the APT and RPM consumer tests actually run) and 341 on xcat-master.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Pin that a locked APT directory is refused with a message naming it, that
--force-unlock takes over and the lock is released at the end, and that a run
which skips building keeps the build results it exists to collect.
Pin the rule that decides whether an invocation produced anything: every
attempted step failing is a total failure, one survivor is not, and a run with
no steps to attempt is unaffected.
A package left in the staging repository by an earlier run is invisible to
collection but visible to createrepo and deploy. Pin that the run clears it, so
the empty-collection guard cannot be satisfied by an earlier invocation's output.
Pin that every pooled Genesis package is checked against the release manifest
again before apt-ftparchive reads the pool, so the check cannot be dropped
without a failing test.
The pooled package carries the release bytes, which the existing digest check
covers, but it must not be the release file itself; assert the identity is
distinct so a link cannot come back unnoticed.
Vinícius Ferrão
Daniel Hilst