A CD run died at compile time inside XCAT::BuildUtils because xcat-master-ub was
missing File::Slurper: "Can't locate File/Slurper.pm in @INC", in the middle of a
build. It was fixed by hand, so the next unprovisioned host fails the same way
and the documented install line can drift from what the code actually loads.
--install-deps installs this host's prerequisites and exits: the sbuild/schroot
toolchain plus the modules. It then LOADS each module and fails naming any that
is still missing, rather than trusting apt's exit code.
That probe earned its place immediately: the first list named libipc-cmd-perl,
which does not exist on Ubuntu -- IPC::Cmd is core there -- and apt failed the
whole install over it. The package is gone from the list and the module is
asserted by loading instead, with a test that pins both halves of that reasoning.
The list and the command are pure functions in BuildUtils, so the decision is
unit-tested and the side effect stays in the caller. Run on xcat-master-ub and
xcat-master-ub-ppc: both report every module present.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every package in a suite pool is gated against debs-manifest.conf, but the
OpenEmbedded Genesis release is published into pool/main/xcat-genesis-openembedded
-- one pool every suite indexes, described by no [<codename>-<arch>] section. So
nothing asserted the published pool was complete: its packages were checked only
as they were copied, against the release checksums, and a pool that lost one
afterwards would publish quietly.
[shared] describes that pool -- all seven architectures, pinned '2.*' like
xcat-genesis-base, because they are built FROM xcat-core and their version walks
with it. A glob rather than an EVR floor: pins in this manifest are exact-or-glob
(version_matches), and the '>= epoch:version-release' form is an EL-side feature
of packages-manifest.conf. verify_shared_pool runs on the side tree before the
swap, so an incomplete pool is never published.
[shared] is not a build target, so the manifest now has two kinds of section. No
code iterates sections blindly, but t/sbuild-all.t did -- twice -- so it now
selects <codename>-<arch> sections and asserts the shared-pool section is not
treated as a target. The consumer fixtures carry the shipped [shared] section
verbatim: publishing a release against a manifest that lacks it is refused, not
silently ungated.
Verified the gate fails when removed, and when [shared] and the pool disagree.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
verify_assembled_repo filtered the manifest through required_pkgs() with this
invocation's --skip-genesis / --skip-xcat-dep. Since the documented publish-only
run IS `--skip-build --skip-genesis --publish`, the flags that describe what this
invocation built were also deciding what the published repository was allowed to
lack -- so a repository with no xcat-genesis-base passed its own publication
gate (PR #63 review).
Those flags mean "this invocation did not build it", never "the repository may
ship without it": a package built by an earlier run is still expected in the
tree, which the side tree is seeded from. The gate now takes the manifest whole.
The other two required_pkgs() call sites are unchanged and correct -- they choose
what to BUILD and what to validate in THIS arch's staging.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The documented publish-only invocation is
`sbuild-all.pl --skip-build --skip-genesis --publish`, and verify_assembled_repo
passes those same flags to required_pkgs() when deciding what the PUBLISHED
repository must contain. So the flags that say what this INVOCATION built also
decide what the repository is allowed to be missing, and a repo carrying no
xcat-genesis-base passes its own publication gate.
Assert the gate on a repo missing Genesis (with --skip-genesis) and on one
missing a compiled dep (with --skip-xcat-dep). Both fixtures keep native stanzas
for the arch, so the failure under test is the missing PACKAGE and not the arch
reading as absent. Both fail on the current gate.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A CI run of this branch died on resolute/ppc64el with a 404 fetching
libssl-dev_3.5.5-1ubuntu3.4_ppc64el.deb: a development suite rolled openssl and
dropped that version from the pool while the chroot's index still named it.
mk-build-deps was the one apt operation in the in-chroot script NOT wrapped in
apt_retry, so a single transient mirror inconsistency failed the package -- and,
with the matrix running failFast, took the other architecture's in-flight builds
down with it.
Retry it the same way the rest of the script retries apt, refreshing the index
between attempts, since a stale index is precisely what produces this. It stays
FATAL once the attempts are spent: a package must never build against whatever
the chroot happens to carry. The refresh goes through apt_retry, so every
apt-get in the script still runs under the fatal helper.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #65 changed where the OpenEmbedded Genesis packages live. On the RPM side
they are published once under xcat-dep/common; on the APT side, once under
pool/main/xcat-genesis-openembedded, with every suite's Packages index pointing
at that one copy instead of each suite carrying its own. It made that change in
build-apt-repo.sh -- the script this branch deletes, having absorbed apt assembly
and signing into sbuild-all.pl -- so the layout is ported here rather than lost.
sbuild-all.pl:
- --genesis-release now rebuilds a single shared pool inside the side tree and
indexes it into every suite, so a release is stored once rather than once per
codename. The debs are Architecture:all and identical everywhere; the previous
per-suite copy multiplied hundreds of megabytes by the number of suites.
- Because every suite's index points into that pool, publishing a release must
cover every suite: a run whose --dists omits one is refused, instead of leaving
that suite indexing files the new release retired.
- OpenEmbedded Genesis debs are dropped from suite pools unconditionally now,
not only when a release is being published -- they belong to the shared pool.
- Published files get an explicit mode 0644: they are served by a web server
running as another user, and inheriting the builder's umask is how that breaks.
master's transaction machinery (per-file backups, --force-unlock recovery of an
interrupted publisher) is deliberately NOT ported: it exists because the shell
publisher writes into the live repository. sbuild-all.pl assembles a complete
side tree, gates it, and renames it into place under one global publish lock, so
a failed or killed run leaves the published repository untouched and there is no
half-written state to recover. The tests assert that guarantee directly.
The APT consumer tests now drive sbuild-all.pl's publish path with master's new
expectations: the shared pool holds one complete release, suite pools hold none
of it, every suite/arch index carries the shared Filename, a later single-suite
rebuild keeps using the pool, a partial-suite release is refused, and a
publication that cannot be signed leaves the packages, indexes and key exactly
as they were.
Full suite green: 416 tests on xcat-master (rome01, EL10) and 414 on
xcat-master-ub, where the APT cases actually run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #64 landed on master and added --genesis-release to build-apt-repo.sh -- the
very script this branch deletes, having absorbed the apt assembly + signing phase
into sbuild-all.pl. A plain merge would either resurrect the shell publisher or
silently drop the OpenEmbedded Genesis release from every apt suite, so the
feature is ported to where apt publication now lives.
sbuild-all.pl --genesis-release <dir>:
- The release is validated once at startup, before any build or publish, with the
same checksum-verify-checksum sequence mockbuild-all.pl uses on the rpm side, so
a release rewritten together with its SHA256SUMS while the verifier runs is
rejected. It must be complete (every supported architecture) and carry debs.
- During assemble_into, each release deb is copied into the codename's pool and the
flat per-version directory and re-checked against the verified checksums. That
happens with the publish lock held, between the pool wipe and apt-ftparchive, so
the bytes that are indexed and signed are the bytes that were verified -- the
separate re-verification pass build-apt-repo.sh ran before indexing has no
window left to cover here.
- Copies are plain copies, never link(): a pool file sharing an inode with the
release would let a write through either path change what the other holds.
- Anything staged under the OpenEmbedded Genesis package name is dropped when the
option is given; the verified release is the only source of those packages.
- XCAT::GenesisRelease is loaded on demand rather than imported at compile time. It
pulls in XCAT::BuildUtils, which needs File::Slurper, and xcat-master-ub does not
carry it: a compile-time import made every apt build -- including the ones that
never pass --genesis-release -- die with "Can't locate File/Slurper.pm".
Also here:
- --publish-lock-wait <seconds> makes the 1800s publish-lock wait settable, so a
caller that would rather fail fast than queue can, and so the lock is testable.
- t/genesis_openembedded_consumer.t: the four APT consumer tests now drive
sbuild-all.pl's real publish path (staging tree, publish lock, atomic swap)
instead of build-apt-repo.sh, including the new flock-based lock behaviour.
- The workflow compiles sbuild-all.pl and BuildUtils.pm instead of shellchecking
the removed script; BUILD.md and genesis-openembedded/README.md document the apt
invocation.
Full suite green on both build hosts: 345 tests on xcat-master-ub (Ubuntu 24.04,
where the APT and RPM consumer tests actually run) and 341 on xcat-master.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Pin that a locked APT directory is refused with a message naming it, that
--force-unlock takes over and the lock is released at the end, and that a run
which skips building keeps the build results it exists to collect.
Everything from the pool wipe to the signature is one transaction over a shared
tree, and the Genesis packages are verified inside it, so a second writer between
that verification and apt-ftparchive would be indexed and signed unchecked --
or, arriving later, would leave clients with checksum failures against metadata
this run signed.
Take the tree for the duration with an atomic mkdir lock, the same NFS-safe
scheme mockbuild-all.pl uses for its output, released on exit and overridable
with --force-unlock when a killed run left one behind.
The staging repositories are cleared per invocation, but the builder results are
not: a reused --run-id leaves the previous run's packages under build-results,
where a step that fails this time is collected from the last time it succeeded.
Individual failures are tolerated by design, so this can mix two invocations in
one signed repository.
Start the result tree empty whenever the run builds. --skip-build collects the
repository-level build-output tree instead and keeps what is there.
Pin the rule that decides whether an invocation produced anything: every
attempted step failing is a total failure, one survivor is not, and a run with
no steps to attempt is unaffected.
Tolerating individual dep-builder failures is deliberate: some packages are el-
or arch-pinned and are expected to fail on some targets. Tolerating all of them
is not -- it means the builder is unusable, the invocation produced nothing, and
whatever the run publishes came from somewhere other than this build.
Count the failures on both the serial and the parallel path and stop when they
account for every attempted step, before collection can take an earlier run's
artifacts for this one's. The rule itself lives in BuildUtils, where it can be
exercised without a builder.
A package left in the staging repository by an earlier run is invisible to
collection but visible to createrepo and deploy. Pin that the run clears it, so
the empty-collection guard cannot be satisfied by an earlier invocation's output.
The collection guard counts what this run copied, but the staging repositories
were never cleared: packages left by an earlier run with the same --run-id sat
there unseen by collection, were indexed by createrepo and published by
deploy_target, where the name-only dependency assertion accepted them. A run
whose builders all failed could therefore ship a previous run's packages.
Empty the staging repositories before collection, so everything they hold
afterwards comes from this invocation.
Pin that every pooled Genesis package is checked against the release manifest
again before apt-ftparchive reads the pool, so the check cannot be dropped
without a failing test.
apt-ftparchive indexes and the signature covers whatever is in the pool at that
moment, while the packages were checked when they were copied. Verify them again
against the release manifest immediately before the indexes are generated, so a
package that changed in between cannot be published as a verified one.
Daniel Hilst
Vinícius Ferrão