PR #65 changed where the OpenEmbedded Genesis packages live. On the RPM side
they are published once under xcat-dep/common; on the APT side, once under
pool/main/xcat-genesis-openembedded, with every suite's Packages index pointing
at that one copy instead of each suite carrying its own. It made that change in
build-apt-repo.sh -- the script this branch deletes, having absorbed apt assembly
and signing into sbuild-all.pl -- so the layout is ported here rather than lost.
sbuild-all.pl:
- --genesis-release now rebuilds a single shared pool inside the side tree and
indexes it into every suite, so a release is stored once rather than once per
codename. The debs are Architecture:all and identical everywhere; the previous
per-suite copy multiplied hundreds of megabytes by the number of suites.
- Because every suite's index points into that pool, publishing a release must
cover every suite: a run whose --dists omits one is refused, instead of leaving
that suite indexing files the new release retired.
- OpenEmbedded Genesis debs are dropped from suite pools unconditionally now,
not only when a release is being published -- they belong to the shared pool.
- Published files get an explicit mode 0644: they are served by a web server
running as another user, and inheriting the builder's umask is how that breaks.
master's transaction machinery (per-file backups, --force-unlock recovery of an
interrupted publisher) is deliberately NOT ported: it exists because the shell
publisher writes into the live repository. sbuild-all.pl assembles a complete
side tree, gates it, and renames it into place under one global publish lock, so
a failed or killed run leaves the published repository untouched and there is no
half-written state to recover. The tests assert that guarantee directly.
The APT consumer tests now drive sbuild-all.pl's publish path with master's new
expectations: the shared pool holds one complete release, suite pools hold none
of it, every suite/arch index carries the shared Filename, a later single-suite
rebuild keeps using the pool, a partial-suite release is refused, and a
publication that cannot be signed leaves the packages, indexes and key exactly
as they were.
Full suite green: 416 tests on xcat-master (rome01, EL10) and 414 on
xcat-master-ub, where the APT cases actually run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #64 landed on master and added --genesis-release to build-apt-repo.sh -- the
very script this branch deletes, having absorbed the apt assembly + signing phase
into sbuild-all.pl. A plain merge would either resurrect the shell publisher or
silently drop the OpenEmbedded Genesis release from every apt suite, so the
feature is ported to where apt publication now lives.
sbuild-all.pl --genesis-release <dir>:
- The release is validated once at startup, before any build or publish, with the
same checksum-verify-checksum sequence mockbuild-all.pl uses on the rpm side, so
a release rewritten together with its SHA256SUMS while the verifier runs is
rejected. It must be complete (every supported architecture) and carry debs.
- During assemble_into, each release deb is copied into the codename's pool and the
flat per-version directory and re-checked against the verified checksums. That
happens with the publish lock held, between the pool wipe and apt-ftparchive, so
the bytes that are indexed and signed are the bytes that were verified -- the
separate re-verification pass build-apt-repo.sh ran before indexing has no
window left to cover here.
- Copies are plain copies, never link(): a pool file sharing an inode with the
release would let a write through either path change what the other holds.
- Anything staged under the OpenEmbedded Genesis package name is dropped when the
option is given; the verified release is the only source of those packages.
- XCAT::GenesisRelease is loaded on demand rather than imported at compile time. It
pulls in XCAT::BuildUtils, which needs File::Slurper, and xcat-master-ub does not
carry it: a compile-time import made every apt build -- including the ones that
never pass --genesis-release -- die with "Can't locate File/Slurper.pm".
Also here:
- --publish-lock-wait <seconds> makes the 1800s publish-lock wait settable, so a
caller that would rather fail fast than queue can, and so the lock is testable.
- t/genesis_openembedded_consumer.t: the four APT consumer tests now drive
sbuild-all.pl's real publish path (staging tree, publish lock, atomic swap)
instead of build-apt-repo.sh, including the new flock-based lock behaviour.
- The workflow compiles sbuild-all.pl and BuildUtils.pm instead of shellchecking
the removed script; BUILD.md and genesis-openembedded/README.md document the apt
invocation.
Full suite green on both build hosts: 345 tests on xcat-master-ub (Ubuntu 24.04,
where the APT and RPM consumer tests actually run) and 341 on xcat-master.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Pin that a locked APT directory is refused with a message naming it, that
--force-unlock takes over and the lock is released at the end, and that a run
which skips building keeps the build results it exists to collect.
Pin the rule that decides whether an invocation produced anything: every
attempted step failing is a total failure, one survivor is not, and a run with
no steps to attempt is unaffected.
A package left in the staging repository by an earlier run is invisible to
collection but visible to createrepo and deploy. Pin that the run clears it, so
the empty-collection guard cannot be satisfied by an earlier invocation's output.
Pin that every pooled Genesis package is checked against the release manifest
again before apt-ftparchive reads the pool, so the check cannot be dropped
without a failing test.
The pooled package carries the release bytes, which the existing digest check
covers, but it must not be the release file itself; assert the identity is
distinct so a link cannot come back unnoticed.
A run with --genesis-release whose builders produced nothing must stop at the
collection guard rather than reach the deployable repository, so pin the exit
status, the message and the untouched deployment directory.
The manifest pinned only the UPSTREAM version: deb_version() and
resolve_present_names() both reduced the version with deb_upstream_version()
before comparing, dropping the epoch and the debian_revision. A package can
carry the right upstream version and still be the wrong package:
* the debian_revision is the PACKAGING revision -- elilo-xcat 3.14-5 and
3.14-6 are different builds of the same upstream 3.14;
* the epoch overrides version comparison outright -- an un-epoched 2.19.0
sorts BELOW 2:2.18.0.
That matters here because xCAT's own debian/control declares versioned
dependencies on these packages -- goconserver (>= 0.3.3-snap...), ipmitool-xcat
(>= 1.8.17-1), grub2-xcat (>= 2.02-...) in xCAT, xCATsn and xCAT-server -- so an
upstream-only pin can accept a deb that the gate calls good and that
`apt install xCAT` then refuses.
The pin is now matched against the FULL [epoch:]upstream[-revision], exactly as
it appears in the built .deb and in the published Packages index. No new
operator was needed: the existing exact/glob/'*' grammar simply applies to the
whole version, which is stricter than a >= floor because it also rejects an
epoch the pin does not name. Six pins become fully exact; two stay globbed
because their revision is not owned by the checkout:
* goconserver=0.3.3-snap* -- upstream exact; the revision is the CD stamp
snap<SOURCE_DATE_EPOCH>, which changes every run. The glob still REQUIRES a
snap-stamped revision, so an unstamped deb is rejected.
* xcat-genesis-base=2.* -- its version is not owned by xcat-dep; it walks
with whatever xcat-core the genesis was built from (XCAT_CORE_REF), and the
two arches legitimately differ since each is converted from its own rpm.
The gate still enforces exactly one genesis FOR THIS ARCH.
deb_version() gained an optional $arch for that logical name: both arch-suffixed
genesis debs are staged on the amd64 host (the cross-arch ppc one for #7610) and
carry different revisions, so without it the pair would look like a version
conflict. This mirrors resolve_present_names, which already refuses to borrow
another arch's genesis.
Comparing full versions also exposed a pre-existing defect it had been masking:
the collector treated PREBUILT .deb files checked into the source tree as build
output. elilo/ ships elilo-xcat_3.14-5_all.deb, elilo-xcat_3.14-6_all.deb and
gnu-efi_3.0v-5_amd64.deb; the first and third were being published into the apt
repo under this run's name, and the stale 3.14-5 collided with the freshly built
3.14-6 once revisions were compared. The build now snapshots the .debs present
before the build and subtracts them afterwards -- a file the build overwrites
changes size/mtime and still counts as output. Verified in the noble chroot: only
elilo-xcat_3.14-6_all.deb is collected; gnu-efi and 3.14-5 are reported and left
behind.
deb_upstream_version() has no remaining callers and is removed.
Tests: version_matches over whole versions (stale revision rejected, unnamed
epoch rejected, both globs still enforcing what they should); deb_version
returning the full version, dying on two revisions of one upstream version (the
elilo case), and selecting the per-arch genesis; resolve_present_names handing
the full version to the comparator; and two end-to-end cases in t/verify-repo.t
asserting the gate now fails a stale packaging revision and an unnamed epoch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses the three blocking issues from the PR #63 review.
1. THE TWO ARCH JOBS COULD BOTH REWRITE THE PUBLISHED REPOSITORY.
The amd64 and ppc64el runs execute concurrently on their two hosts against the
same --apt-dir. The run lock does not protect the repository -- it is per-arch by
design, precisely so the two arches CAN build in parallel -- so both were free to
wipe and repopulate the same pool/, dists/, Release, InRelease and tarball at the
same time and interleave into a corrupt but green repository.
Split by role, so architecture jobs produce staging artifacts and one locked
finalization step publishes atomically:
* A run that builds no longer publishes. It fills staging/<codename>/<arch>/
and stops, saying so and printing the finalize command. Publishing happens
only with --publish, or implicitly on a run that builds nothing
(--skip-build), which IS the finalization step; --skip-createrepo still
forces it off.
* publish_repo() takes ONE GLOBAL publish lock (.sbuild-all.publish.lock), not
the per-arch build lock, so even a cron run racing a manual one on the same
host serializes instead of interleaving.
* It publishes atomically. assemble_into() builds and signs the complete tree
in a side directory (<apt-dir>.publish-<run-id>.<pid>), the gate runs against
THAT tree, and swap_into_place() then renames it onto --apt-dir. A reader --
the deploy rsync, an apt client on a served tree -- sees either the previous
complete repo or the new complete repo, never a half-wiped pool or an index
that disagrees with its Release. A failure anywhere before the swap leaves
the published tree untouched and removes the side tree. The side tree is
seeded from the current published one, so codenames outside --dists survive.
* The tarball moved inside publish, under the same lock: previously every
per-arch build tarred the shared apt tree while the other arch rewrote it.
2. build_deb_in_chroot() WAS NOT FAIL-HARD, AND THE ENVIRONMENT WAS NOT CLEAN.
The common build-tooling install ended in `|| true`; a failed Build-Depends
installation only warned; and the Build-Depends came from a sed pipeline over
debian/control that stripped version constraints `(>= 12)`, could not express
alternatives `a | b`, and mangled arch qualifiers. Worse, the per-codename
chroots are long-lived and shared by all seven packages, and nothing asserted
that a schroot session is throwaway -- so on a chroot without a union mount,
package N's build-dependencies stayed installed for package N+1 and a package
whose debian/control forgets a Build-Depends would build green on a sibling's
leftovers.
Making dependency setup fatal is only half the fix; it means nothing if a stale
environment can satisfy an undeclared dependency. Both halves are addressed:
* The in-chroot program is now generated by the pure, unit-tested
BuildUtils::chroot_build_script() and runs under `set -euo pipefail`.
apt-get update and the common tooling go through an apt_retry helper that
retries a transient mirror hiccup and then FAILS the build. Build-Depends are
installed with mk-build-deps (devscripts + equivs), which hands
debian/control's relationships to apt verbatim, and a failure is fatal. The
mk-build-deps dummy package is excluded from deb collection alongside dbgsym.
* ensure_disposable_chroot() repairs a chroot.d entry that lacks
union-type=overlay, and build_deb_in_chroot() re-reads `schroot --config` and
hard-fails on a chroot that is still not disposable, naming the fix.
BuildUtils::chroot_is_disposable() is the pure predicate.
3. THE REPOSITORY GATE STILL HAD FALSE PASSES.
* Standalone --verify-repo skipped signature verification whenever --gpg-home
was not passed, so the common `--verify-repo <dir>` invocation silently
checked completeness only. Signatures are now verified BY DEFAULT there;
--no-verify-signature is the explicit opt-out. (The automatic pre-swap gate
still requires a signature iff --gpg-sign was used, so an intentionally
unsigned tree does not false-fail -- and publishing unsigned now warns.)
* The expected architecture set was inferred from what happened to be present,
so an entirely missing secondary architecture read as "this run did not build
it" and passed. It is now always a CLAIM: --expect-arch (repeatable, accepts
a space/comma list) if given, else the staged arch set when publishing, else
each codename's own Release "Architectures:" line when verifying standalone
(BuildUtils::parse_release_architectures). The pure
BuildUtils::verify_repo_arches then reports an expected arch with no NATIVE
package as MISSING-ARCH and natives for an unexpected arch as
UNEXPECTED-ARCH (a stale architecture). Native detection matters because the
Architecture:all packages ride into every arch's index, so a non-empty
binary-<arch>/Packages is not evidence that arch was built.
* Release now advertises exactly the expected arch set and only those
binary-<arch> indices are written, so an amd64-only build no longer
advertises a ppc64el it cannot serve.
* An expected cell with no manifest section used to be skipped with a note --
a third free pass. It is now a hard NO-MANIFEST error.
TESTS
t/sbuild-all.t gains verify_repo_arches (both directions), the
parse_release_architectures parser, chroot_is_disposable, regression guards on
the generated in-chroot script (no `|| true`, no warn-only build-deps, no sed
extraction, mk-build-deps present, every apt-get behind the fatal helper), and
the build_deb_in_chroot disposability guard driven through a stub schroot.
t/verify-repo.t is new: it drives the real `sbuild-all.pl --verify-repo` against
fixture apt trees and asserts each former false pass now fails -- missing
secondary architecture (with and without --expect-arch), arch:all-only index,
unsigned repo with no --gpg-home, missing manifest section -- and that an honest
single-arch repo still passes.
Validated on the build hosts: a full focal+noble amd64 build of all seven
packages through the new fatal dependency path (mk-build-deps installs the
declared Build-Depends inside the oldest, focal, chroot), and the publish path
end to end -- a staging-only run leaves the repo untouched; publish assembles
aside, gates, and swaps; an unrelated codename survives the swap; stale debs are
dropped; a failed gate leaves the published repo byte-identical and cleans up the
side tree.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão