xCAT/xcat-dep
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-dep.git synced 2026-09-12 04:26:25 +00:00
Code Issues Projects Releases Wiki Activity

fix(xcat-dep): unsafe repo publication, non-fail-hard builds, repo-gate false passes

Browse Source 
Addresses the three blocking issues from the PR #63 review.

1. THE TWO ARCH JOBS COULD BOTH REWRITE THE PUBLISHED REPOSITORY.

The amd64 and ppc64el runs execute concurrently on their two hosts against the
same --apt-dir. The run lock does not protect the repository -- it is per-arch by
design, precisely so the two arches CAN build in parallel -- so both were free to
wipe and repopulate the same pool/, dists/, Release, InRelease and tarball at the
same time and interleave into a corrupt but green repository.

Split by role, so architecture jobs produce staging artifacts and one locked
finalization step publishes atomically:

  * A run that builds no longer publishes. It fills staging/<codename>/<arch>/
    and stops, saying so and printing the finalize command. Publishing happens
    only with --publish, or implicitly on a run that builds nothing
    (--skip-build), which IS the finalization step; --skip-createrepo still
    forces it off.
  * publish_repo() takes ONE GLOBAL publish lock (.sbuild-all.publish.lock), not
    the per-arch build lock, so even a cron run racing a manual one on the same
    host serializes instead of interleaving.
  * It publishes atomically. assemble_into() builds and signs the complete tree
    in a side directory (<apt-dir>.publish-<run-id>.<pid>), the gate runs against
    THAT tree, and swap_into_place() then renames it onto --apt-dir. A reader --
    the deploy rsync, an apt client on a served tree -- sees either the previous
    complete repo or the new complete repo, never a half-wiped pool or an index
    that disagrees with its Release. A failure anywhere before the swap leaves
    the published tree untouched and removes the side tree. The side tree is
    seeded from the current published one, so codenames outside --dists survive.
  * The tarball moved inside publish, under the same lock: previously every
    per-arch build tarred the shared apt tree while the other arch rewrote it.

2. build_deb_in_chroot() WAS NOT FAIL-HARD, AND THE ENVIRONMENT WAS NOT CLEAN.

The common build-tooling install ended in `|| true`; a failed Build-Depends
installation only warned; and the Build-Depends came from a sed pipeline over
debian/control that stripped version constraints `(>= 12)`, could not express
alternatives `a | b`, and mangled arch qualifiers. Worse, the per-codename
chroots are long-lived and shared by all seven packages, and nothing asserted
that a schroot session is throwaway -- so on a chroot without a union mount,
package N's build-dependencies stayed installed for package N+1 and a package
whose debian/control forgets a Build-Depends would build green on a sibling's
leftovers.

Making dependency setup fatal is only half the fix; it means nothing if a stale
environment can satisfy an undeclared dependency. Both halves are addressed:

  * The in-chroot program is now generated by the pure, unit-tested
    BuildUtils::chroot_build_script() and runs under `set -euo pipefail`.
    apt-get update and the common tooling go through an apt_retry helper that
    retries a transient mirror hiccup and then FAILS the build. Build-Depends are
    installed with mk-build-deps (devscripts + equivs), which hands
    debian/control's relationships to apt verbatim, and a failure is fatal. The
    mk-build-deps dummy package is excluded from deb collection alongside dbgsym.
  * ensure_disposable_chroot() repairs a chroot.d entry that lacks
    union-type=overlay, and build_deb_in_chroot() re-reads `schroot --config` and
    hard-fails on a chroot that is still not disposable, naming the fix.
    BuildUtils::chroot_is_disposable() is the pure predicate.

3. THE REPOSITORY GATE STILL HAD FALSE PASSES.

  * Standalone --verify-repo skipped signature verification whenever --gpg-home
    was not passed, so the common `--verify-repo <dir>` invocation silently
    checked completeness only. Signatures are now verified BY DEFAULT there;
    --no-verify-signature is the explicit opt-out. (The automatic pre-swap gate
    still requires a signature iff --gpg-sign was used, so an intentionally
    unsigned tree does not false-fail -- and publishing unsigned now warns.)
  * The expected architecture set was inferred from what happened to be present,
    so an entirely missing secondary architecture read as "this run did not build
    it" and passed. It is now always a CLAIM: --expect-arch (repeatable, accepts
    a space/comma list) if given, else the staged arch set when publishing, else
    each codename's own Release "Architectures:" line when verifying standalone
    (BuildUtils::parse_release_architectures). The pure
    BuildUtils::verify_repo_arches then reports an expected arch with no NATIVE
    package as MISSING-ARCH and natives for an unexpected arch as
    UNEXPECTED-ARCH (a stale architecture). Native detection matters because the
    Architecture:all packages ride into every arch's index, so a non-empty
    binary-<arch>/Packages is not evidence that arch was built.
  * Release now advertises exactly the expected arch set and only those
    binary-<arch> indices are written, so an amd64-only build no longer
    advertises a ppc64el it cannot serve.
  * An expected cell with no manifest section used to be skipped with a note --
    a third free pass. It is now a hard NO-MANIFEST error.

TESTS

t/sbuild-all.t gains verify_repo_arches (both directions), the
parse_release_architectures parser, chroot_is_disposable, regression guards on
the generated in-chroot script (no `|| true`, no warn-only build-deps, no sed
extraction, mk-build-deps present, every apt-get behind the fatal helper), and
the build_deb_in_chroot disposability guard driven through a stub schroot.

t/verify-repo.t is new: it drives the real `sbuild-all.pl --verify-repo` against
fixture apt trees and asserts each former false pass now fails -- missing
secondary architecture (with and without --expect-arch), arch:all-only index,
unsigned repo with no --gpg-home, missing manifest section -- and that an honest
single-arch repo still passes.

Validated on the build hosts: a full focal+noble amd64 build of all seven
packages through the new fatal dependency path (mk-build-deps installs the
declared Build-Depends inside the oldest, focal, chroot), and the publish path
end to end -- a staging-only run leaves the repo untouched; publish assembles
aside, gates, and swaps; an unrelated codename survives the swap; stale debs are
dropped; a failed gate leaves the published repo byte-identical and cleans up the
side tree.

Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
This commit is contained in:
Daniel Hilst
2026-08-21 20:25:51 -03:00
parent 0266df91f3
commit ca051e46c4
5 changed files with 1092 additions and 217 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BUILD.md
+77 -28
View File
@@ -282,6 +282,23 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of
- **Fresh staging + promote-on-success.** Everything is built + validated into a per-run staging tree
first; the published apt repo is (re)assembled from staging ONLY after the complete expected set
validates — a partial/failed build never reaches the repo and stale debs never accumulate.
- **Build runs stage; publishing is a separate, locked, atomic step.** The two arches build
*concurrently* on their two hosts against the same `--apt-dir`, so an arch build run **never
publishes**: it fills staging and stops. Publishing happens with **`--publish`** (implied by
`--skip-build`, i.e. the finalization run). That step takes **one global publish lock** — not the
per-arch build lock — assembles the whole tree into a **side directory**, runs the repo gate against
*that* tree, and only then swaps it onto `--apt-dir` with a single `rename(2)`. Readers therefore
see either the previous complete repo or the new complete repo, never a half-wiped `pool/` or an
index that disagrees with its `Release`; a failed gate leaves the published tree untouched.
Codenames outside `--dists` survive the swap. `--skip-createrepo` forces "do not publish".
- **Clean, disposable build environment per package.** Every package builds in its own `schroot`
session, and the chroot must hand out a **throwaway** session (`union-type=overlay`, or a
snapshot/tarball chroot). `sbuild-all.pl` repairs a chroot that lacks one and hard-fails if it still
is not disposable. That is what makes the fail-hard dependency handling mean something: inside the
session `apt-get update`, the common build tooling and the package's `Build-Depends` (resolved with
`mk-build-deps`, so version constraints, `a | b` alternatives and arch qualifiers are honoured) are
all **fatal** on failure — and since nothing survives the session, a package whose `debian/control`
forgets a `Build-Depends` cannot build green on a sibling package's leftovers.
- **Per-arch package sets (`debs-manifest.conf`).** One `[<codename>-<arch>]` section per target. The
noarch boot components (`syslinux-xcat`/`grub2-xcat`/`elilo-xcat`/`xnba-undi`, `Architecture:all`)
are built ONCE on amd64 — single producer, their source is x86-only — and assembled into every
@@ -307,7 +324,9 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of
- **`<dep>/sbuild.pl`** ×7 — per-package builders (mirror `<dep>/mockbuild.pl`); each drives its
maintained `debian/` in the chroot and collects the `.deb`(s). Invoked by `sbuild-all.pl`.
- **`debs-manifest.conf`** — per `[<codename>-<arch>]` required set + version pins.
- **`t/sbuild-all.t`** — fixture tests (`prove t/sbuild-all.t`).
- **`t/sbuild-all.t`** — unit tests for the pure helpers (`prove t/`).
- **`t/verify-repo.t`** — end-to-end tests of the repo gate against fixture apt trees (missing
secondary arch, arch:all-only index, unsigned repo, missing manifest section).
## Usage (per arch, as root on the matching build host)
@@ -316,43 +335,58 @@ ppc64el on the ppc Ubuntu host). The Ubuntu version(s) to build are selected wit
space/comma list of codenames) or, for exactly one, **`--target <codename>-<arch>`**. Version ↔
codename: `20.04`=`focal`, `22.04`=`jammy`, `24.04`=`noble`, `26.04`=`resolute`.
### Build ALL supported Ubuntu versions
The full flow is **two steps**: each arch builds into staging on its own host, then **one**
finalization step publishes the assembled repo atomically.
### Step 1 — build each arch into staging (no publishing)
```bash
# amd64 host — build focal+jammy+noble+resolute, sign, assemble the apt tree:
# amd64 host — build focal+jammy+noble+resolute into staging:
./sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \
--xcat-source ../xcat-core --genesis-rpm <xCAT-genesis-base-x86_64 rpm> \
--genesis-rpm-ppc <xCAT-genesis-base-ppc64 rpm> \
--genesis-rpm-ppc <xCAT-genesis-base-ppc64 rpm>
# ppc64el host — arch-specific deps only (the Architecture:all boot components and both genesis
# debs come from the amd64 build):
./sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" --skip-genesis
```
These runs touch **only** `staging/<codename>/<arch>/`; the apt tree at `--apt-dir` is left alone, so
the two hosts can run at the same time. `--dists` may be omitted entirely — with no
`--dists`/`--target`, **all supported codenames** are built (`focal jammy noble resolute`).
### Step 2 — publish once, after every arch has staged
```bash
./sbuild-all.pl --skip-build --skip-genesis \
--publish --expect-arch "amd64 ppc64el" \
--gpg-sign --gpg-key-id xcat@example.com --gpg-home <gpg-home>
```
`--dists` may be omitted entirely — with no `--dists`/`--target`, **all supported codenames** are
built (the default is `focal jammy noble resolute`).
This takes the global publish lock, assembles + signs both arches' staging into a side tree, gates it,
and swaps it onto `--apt-dir` atomically. `--expect-arch` states which architectures the published
repo must serve — omit it and the staged arch set is used instead. (`--publish` is implied here
because the run builds nothing; state it explicitly if you want to build **and** publish in one go.)
### Build ONE specific Ubuntu version
```bash
# just 24.04 (noble) on amd64 — two equivalent forms:
./sbuild-all.pl --arch amd64 --dists noble --xcat-source ../xcat-core --genesis-rpm <rpm> --gpg-sign ...
./sbuild-all.pl --target noble-amd64 --xcat-source ../xcat-core --genesis-rpm <rpm> --gpg-sign ...
./sbuild-all.pl --arch amd64 --dists noble --xcat-source ../xcat-core --genesis-rpm <rpm>
./sbuild-all.pl --target noble-amd64 --xcat-source ../xcat-core --genesis-rpm <rpm>
# just 20.04 (focal):
./sbuild-all.pl --arch amd64 --dists focal ...
```
### ppc64el host
```bash
# arch-specific deps only (the Architecture:all boot components come from the amd64 build):
./sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" \
--xcat-source ../xcat-core --genesis-rpm <xCAT-genesis-base-ppc64 rpm> --gpg-sign ...
```
### Handy variants
```bash
./sbuild-all.pl --dry-run --arch amd64 --dists noble # print the plan, do nothing
./sbuild-all.pl --skip-build --skip-genesis --gpg-sign ... # assemble-only (re-index/re-sign staging)
./sbuild-all.pl --dry-run --arch amd64 --dists noble # print the plan, do nothing
./sbuild-all.pl --skip-build --skip-genesis --gpg-sign ... # publish-only (re-index/re-sign staging)
# single host: build AND publish in one go
./sbuild-all.pl --arch amd64 --dists noble --genesis-rpm <rpm> \
--publish --expect-arch amd64 --gpg-sign --gpg-key-id <id> --gpg-home <dir>
```
`sbuild-all.pl --help` lists every option and `sbuild-all.pl --man` (or `perldoc sbuild-all.pl`)
@@ -362,24 +396,39 @@ prints the full manual; the shared flags (`--repo-root`, `--manifest`,
# Repository verification gate
After the apt repo is assembled + signed, `sbuild-all.pl` runs a **manifest-driven gate** that fails
the build if the published repo is incomplete or mis-signed. It uses `debs-manifest.conf` as the
single source of truth and is layered so the decision logic is pure and unit-tested
(`BuildUtils::verify_repo_packages` / `verify_repo_signature`; `parse_packages_index` /
`resolve_present_names` for parsing/resolution), separate from the disk/gpg I/O.
Before an assembled repo is published, `sbuild-all.pl` runs a **manifest-driven gate** that fails the
build if the repo is incomplete, serves the wrong architectures, or is mis-signed. It uses
`debs-manifest.conf` as the single source of truth and is layered so the decision logic is pure and
unit-tested (`BuildUtils::verify_repo_arches` / `verify_repo_packages` / `verify_repo_signature`;
`parse_packages_index` / `parse_release_architectures` / `resolve_present_names` for
parsing/resolution), separate from the disk/gpg I/O.
- **Runs automatically** at the end of `assemble_apt`, **per codename × arch**. Suppress with
`--no-verify-repo`; skipped under `--dry-run`. Verify an assembled tree out of band with
`--verify-repo=<apt_dir>` (manifest/dists/key from `--manifest`/`--dists`/`--gpg-key-id`/`--gpg-home`).
- **Runs automatically** inside `--publish`, against the **side tree, before the swap**, per codename ×
expected arch — so a repo that fails the gate is never published. Suppress with `--no-verify-repo`;
skipped under `--dry-run`. Verify an already-published tree out of band with
`--verify-repo=<apt_dir>` (manifest/dists/arches/key from
`--manifest`/`--dists`/`--expect-arch`/`--gpg-key-id`/`--gpg-home`).
- **Architectures:** the expected set is always a **claim**, never an inference from what happens to
be present — `--expect-arch` if given, else the staged arch set when publishing, else each
codename's own `Release` `Architectures:` line when verifying standalone. An expected arch with no
*native* package is `MISSING-ARCH`; natives for an arch outside the expected set are
`UNEXPECTED-ARCH` (a stale architecture). This is what makes an **entirely missing secondary
architecture** a failure instead of reading as "this run did not build it". Note that a non-empty
`binary-<arch>/Packages` is *not* evidence the arch was built: the `Architecture:all` packages
(`grub2-xcat`, the genesis debs) ride into every arch's index, so `index_has_native_arch` is what
counts.
- **Completeness:** for each cell, every package that codename×arch's manifest section requires (after
`required_pkgs` skip-filtering) must appear in the published `binary-<arch>/Packages` with a version
satisfying its pin. The arch-suffixed genesis (`xcat-genesis-base`) is resolved to **this cell's
arch** (`xcat-genesis-base-<arch>`) — never a different arch, so a missing native genesis is caught.
An expected cell with **no manifest section** is a hard error (`NO-MANIFEST`), not a free pass.
- **Signature:** each `dists/<cn>/InRelease` (or detached `Release`+`Release.gpg`) must be a *good*
signature whose **primary-key fingerprint equals the fingerprint of `--gpg-key-id`** — the repo was
signed by exactly the CLI key. Expired/revoked keys and expired signatures are rejected; if the CLI
key does not resolve to a fingerprint the gate fails (`SIGKEY`), never passes. A signature is only
*required* when `--gpg-sign` was used (an intentionally-unsigned repo does not false-fail).
key does not resolve to a fingerprint the gate fails (`SIGKEY`), never passes. In the automatic
pre-swap run a signature is required iff `--gpg-sign` was used (an intentionally-unsigned repo does
not false-fail); in the **standalone `--verify-repo` mode signatures are checked by default**, and
`--no-verify-signature` is the explicit opt-out.
**Semantic idiosyncrasies (intentional, and mirrored in the EL `mockbuild-all.pl` gate):**
BuildUtils.pm
+167 -26
View File
@@ -24,10 +24,11 @@ use MIME::Base64 qw(encode_base64);
our @EXPORT_OK = qw(
sh_quote print_step
version_matches required_pkgs read_manifest standard_options
verify_repo_packages verify_repo_signature parse_packages_index resolve_present_names
verify_repo_packages verify_repo_signature verify_repo_arches
parse_packages_index parse_release_architectures resolve_present_names
index_has_native_arch control_binary_arch skip_arch_all_on
codename_to_version version_to_codename known_codenames
chroot_name chroot_sources_list
chroot_name chroot_sources_list chroot_is_disposable chroot_build_script
control_field genesis_deb_control
deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb
build_deb_in_chroot
@@ -197,6 +198,58 @@ sub verify_repo_signature {
return @problems;
}
# verify_repo_arches(\@expected, \%native) -> @problems
# @expected : the architecture set the published repo is REQUIRED to serve. It comes from an
# EXPLICIT claim -- --expect-arch, the staged arch set that was just promoted, or the
# repo's own Release "Architectures:" line -- NEVER from "whichever binary-<arch>
# directories happen to exist". That is the whole point: if the expected set were
# inferred from presence, an entirely missing secondary architecture would read as
# "this run did not build it" instead of "the repository is incomplete" (the false-PASS
# the PR #63 review caught).
# %native : arch => boolean, "the published binary-<arch>/Packages carries at least one stanza
# built FOR that arch" (index_has_native_arch) -- collected by the IO layer for every
# arch it looked at, expected or not.
# Returns human-readable problem strings (empty list = the arch set is exactly right):
# "MISSING-ARCH <a> (expected, but the published index carries no native <a> package)"
# "UNEXPECTED-ARCH <a> (native <a> packages published, but <a> is not in the expected set <set>)"
# The UNEXPECTED direction matters too: it catches a stale architecture left behind in a tree that is
# no longer built for it. Pure: no I/O, deterministic (problems in sorted arch order).
sub verify_repo_arches {
my ($expected, $native) = @_;
my %want = map { $_ => 1 } @{ $expected || [] };
my @problems;
for my $a (sort keys %want) {
push @problems, "MISSING-ARCH $a (expected, but the published index carries no native $a package)"
unless $native->{$a};
}
for my $a (sort keys %$native) {
next unless $native->{$a};
next if $want{$a};
push @problems, "UNEXPECTED-ARCH $a (native $a packages published, but $a is not in the "
. "expected set [" . join(' ', sort keys %want) . "])";
}
return @problems;
}
# parse_release_architectures($release_text) -> @arches
# The architecture set an apt Release file CLAIMS to serve (its "Architectures:" line), in file order
# and de-duplicated; an empty list when the field is absent. This is the repository's own published
# claim, so it is the right FALLBACK expected-arch set for a standalone verification of a tree whose
# build-time --expect-arch is not known: verifying a repo against what it advertises to apt clients
# catches "Release says amd64 ppc64el but binary-ppc64el is empty/missing". Pure: text in, list out.
sub parse_release_architectures {
my ($text) = @_;
return () unless defined $text;
my ($line) = $text =~ /^Architectures:[ \t]*(.*?)[ \t]*$/m;
return () unless defined $line;
my (@a, %seen);
for my $x (split /\s+/, $line) {
next unless length $x;
push @a, $x unless $seen{$x}++;
}
return @a;
}
# parse_packages_index($text) -> \%{ package_name => version }
# Parse a Debian 'Packages' index: RFC822 stanzas separated by blank line(s); each carries a
# 'Package:' and a 'Version:'. Returns name => version (the FULL Debian version verbatim, epoch +
@@ -315,6 +368,35 @@ sub chroot_sources_list {
);
}
# chroot_is_disposable($schroot_config_text): true iff a `schroot -c <name> -- ...` session against
# this chroot gets a THROWAWAY filesystem -- i.e. everything the build installs or writes is discarded
# when the session ends, so the NEXT package starts from the pristine base.
#
# This is the load-bearing precondition of the per-package build (PR #63 review concern #2): the
# per-codename chroots are long-lived and shared by all seven packages, so without a disposable
# session, package N's build-dependencies stay installed for package N+1 -- and a package whose
# debian/control forgets a Build-Depends builds anyway, silently, because a sibling happened to pull
# the dependency in. Making dependency installation fatal is only half the fix; it means nothing if a
# stale environment can satisfy an undeclared dependency in the first place.
#
# Disposable configurations, per schroot(1):
# * union-type = overlay | overlayfs | aufs | unionfs -- a per-session union mount over the base
# directory; writes go to the (discarded) overlay. This is what sbuild-createchroot sets up.
# * type = file -- the session unpacks a fresh tarball.
# * type = {btrfs,lvm,zfs}-snapshot -- the session gets its own snapshot.
# A plain `type=directory` with `union-type=none` is NOT disposable: the session bind-mounts the base
# directory read-write and every build mutates it permanently.
# Pure: `schroot --config -c <name>` text in, boolean out.
sub chroot_is_disposable {
my ($config_text) = @_;
return 0 unless defined $config_text && $config_text ne '';
my ($union) = $config_text =~ /^union-type=[ \t]*(\S+)/m;
return 1 if defined $union && $union =~ /^(?:overlay|overlayfs|aufs|unionfs)$/;
my ($type) = $config_text =~ /^type=[ \t]*(\S+)/m;
return 1 if defined $type && $type =~ /^(?:file|btrfs-snapshot|lvm-snapshot|zfs-snapshot)$/;
return 0;
}
# ---------------------------------------------------------------------------------------------------
# Debian control-metadata helpers (concern #2: preserve the maintained packaging's semantics).
# ---------------------------------------------------------------------------------------------------
@@ -488,15 +570,84 @@ sub cross_copy_genesis_deb {
# MockBuildUtils' helpers do for the per-package mockbuild.pl builders).
# ---------------------------------------------------------------------------------------------------
# chroot_build_script: the bash program that runs INSIDE the per-package schroot session. Returned as
# text (pure) so t/sbuild-all.t can assert its fail-hard properties without a chroot.
#
# Fail-hard contract (PR #63 review concern #2) -- every step below is FATAL, none is best-effort:
# * `set -euo pipefail`: any unchecked command failure aborts the build.
# * apt-get update / the common build tooling install are retried (transient mirror hiccups) and
# then FATAL. They used to end in `|| true`, which let a package build with, say, no `quilt` and
# produce a silently-wrong .deb.
# * Build-Depends are installed with `mk-build-deps` (devscripts + equivs) instead of a sed
# extraction, and a failure is FATAL. mk-build-deps feeds the control file's relationships to apt
# verbatim, so version constraints `(>= 12)`, alternatives `a | b` and arch qualifiers `[!ppc64el]`
# are honoured -- the sed pipeline dropped all three, and, paired with the swallowed error, a
# too-old or missing build dependency produced a green build.
# The build environment is disposable: the caller asserts the chroot gives each session a throwaway
# overlay/snapshot (chroot_is_disposable), so nothing this script installs can leak into the next
# package's build and satisfy an undeclared dependency.
sub chroot_build_script {
return <<'INNER';
set -euo pipefail
PKGSRC="$1"; OUT="$2"; SDE="$3"; EXTRA="$4"; B64="$5"
export DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS=nocheck SOURCE_DATE_EPOCH="$SDE"
# apt_retry: run apt-get, retrying a few times for a transient mirror/network hiccup, then FATAL.
apt_retry() {
local i
for i in 1 2 3; do
if apt-get "$@"; then return 0; fi
echo "[warn] 'apt-get $*' failed (attempt $i/3); retrying in 5s" >&2
sleep 5
done
echo "FATAL: 'apt-get $*' failed after 3 attempts" >&2
return 1
}
apt_retry update -q
# Common build tooling. FATAL: a missing tool silently changes what gets built.
# shellcheck disable=SC2086 # $EXTRA is a deliberate word-split package list
apt_retry install -y --no-install-recommends \
git wget curl ca-certificates devscripts equivs quilt fakeroot build-essential $EXTRA
W=$(mktemp -d)
cp -a "$PKGSRC" "$W/pkg"
cd "$W/pkg"
# Declared Build-Depends, resolved by mk-build-deps: it hands debian/control's relationships to apt
# verbatim, so versions/alternatives/arch-qualifiers are honoured. FATAL on failure -- an unsatisfied
# build dependency must stop the build, never be papered over by whatever the chroot already carries.
if [ -f debian/control ]; then
echo "== installing Build-Depends from debian/control (mk-build-deps) =="
mk-build-deps --install --remove \
--tool 'apt-get -y --no-install-recommends' debian/control
fi
printf '%s' "$B64" | base64 -d > "$W/pkgbuild.sh"
( cd "$W/pkg" && bash "$W/pkgbuild.sh" )
# Collect the built binaries. The mk-build-deps dummy package (<src>-build-deps_*.deb) and debug
# symbols are not build output and must never reach the repo.
mapfile -t found < <(find "$W" -maxdepth 3 -name '*.deb' \
! -name '*-dbgsym_*' ! -name '*-build-deps_*' -print | sort)
if [ "${#found[@]}" -eq 0 ]; then
echo "FATAL: the package build produced no .deb" >&2
exit 1
fi
mkdir -p "$OUT"
for d in "${found[@]}"; do cp -v "$d" "$OUT/"; done
INNER
}
# build_deb_in_chroot: build ONE package inside its <codename>-<arch>-sbuild chroot and collect the
# produced .deb(s) into $result_dir. The COMMON orchestration lives here -- an ephemeral schroot
# produced .deb(s) into $result_dir. The COMMON orchestration lives here -- a disposable schroot
# session, apt update, install of common tools + the package's debian/control Build-Depends, an
# OUT-OF-TREE copy of the package dir (the checkout is never mutated), SOURCE_DATE_EPOCH for
# reproducible builds, deb collection, and a host-side check that the debs actually landed (a
# chroot-local --result-dir would otherwise be a silent no-output). Each <dep>/sbuild.pl supplies
# only its package-specific $build snippet (the source prep + dpkg-buildpackage that used to live in
# make_deb.sh); $build runs with CWD = the copied package dir and must leave its .deb(s) somewhere
# under the build work tree. Dies on any failure.
# under the build work tree. Dies on any failure -- including a chroot that is not disposable.
# %args: pkg, chroot, pkg_dir, result_dir, build_timestamp, build (required); extra_tools (arrayref, optional)
sub build_deb_in_chroot {
my (%a) = @_;
@@ -505,33 +656,23 @@ sub build_deb_in_chroot {
my $pkg = $a{pkg};
die "FATAL: chroot $a{chroot} missing (run sbuild-all.pl to auto-init it)\n"
if system("schroot -l 2>/dev/null | grep -qx chroot:$a{chroot}") != 0;
# HARD precondition: each package must build in a CLEAN, throwaway environment. A shared
# `type=directory` chroot with no union mount would carry the previous package's build-deps into
# this one, so an undeclared Build-Depends would build green here and fail for everyone else.
my $cfg = `schroot --config -c ${\ sh_quote($a{chroot}) } 2>/dev/null` // '';
die "FATAL: chroot $a{chroot} is NOT disposable -- a schroot session against it would mutate the\n"
. " shared base filesystem, so one package's build-dependencies would leak into the next and\n"
. " hide a missing Build-Depends. Add 'union-type=overlay' to its /etc/schroot/chroot.d/ entry\n"
. " (or delete the chroot and let sbuild-all.pl re-create it).\n"
unless chroot_is_disposable($cfg);
make_path($a{result_dir});
my $extra = join(' ', @{ $a{extra_tools} || [] });
my $b64 = encode_base64($a{build}, '');
# The whole per-package build runs in ONE schroot session (ephemeral overlay). schroot SANITIZES
# the environment, so values are passed as POSITIONAL ARGS to the inner bash; the package-specific
# The whole per-package build runs in ONE disposable schroot session. schroot SANITIZES the
# environment, so values are passed as POSITIONAL ARGS to the inner bash; the package-specific
# build is passed base64-encoded to avoid any quoting interplay through schroot.
my $inner = <<'INNER';
set -uo pipefail
PKGSRC="$1"; OUT="$2"; SDE="$3"; EXTRA="$4"; B64="$5"
export DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS=nocheck SOURCE_DATE_EPOCH="$SDE"
for t in 1 2 3; do apt-get update -q && break; sleep 5; done
apt-get install -y --no-install-recommends \
git wget curl ca-certificates devscripts quilt fakeroot build-essential $EXTRA >/dev/null 2>&1 || true
W=$(mktemp -d); cp -a "$PKGSRC" "$W/pkg"; cd "$W/pkg"
if [ -f debian/control ]; then
BD=$(sed -n '/^Build-Depends:/,/^\S/p' debian/control | tr ',' '\n' \
| sed -E 's/^Build-Depends://; s/\(.*\)//; s/\[.*\]//; s/[[:space:]]//g' \
| grep -E '^[a-z0-9]' | grep -v '^debhelper-compat' | sort -u | tr '\n' ' ')
[ -n "$BD" ] && { apt-get install -y $BD >/dev/null 2>&1 || echo "[warn] some build-deps failed to install"; }
fi
printf '%s' "$B64" | base64 -d > "$W/pkgbuild.sh"
( cd "$W/pkg" && bash "$W/pkgbuild.sh" ) || { echo "package build FAILED"; exit 1; }
found=$(find "$W" -maxdepth 3 -name '*.deb' ! -name '*-dbgsym_*' -print)
[ -n "$found" ] || { echo "build produced no .deb"; exit 1; }
mkdir -p "$OUT"; echo "$found" | while read -r d; do cp -v "$d" "$OUT/"; done
INNER
my $inner = chroot_build_script();
my $cmd = 'schroot -c ' . sh_quote($a{chroot}) . ' -u root -d / -- bash -c '
. sh_quote($inner) . ' bash '
sbuild-all.pl
+488 -160
View File
@@ -11,9 +11,12 @@
# Design (mirrors mockbuild-all.pl + fixes the PR #63 review):
# 1. One host = one arch (dpkg --print-architecture / --arch); a set of codenames (--dists).
# Each (codename,arch) is a TARGET named "<codename>-<arch>" with a section in debs-manifest.conf.
# 2. Everything is built + validated into a FRESH per-run STAGING tree first; the published apt repo
# is (re)assembled from staging ONLY after the complete expected set validates -- so a partial or
# failed build never reaches the published repo, and stale debs never accumulate (concern #1).
# 2. Everything is built + validated into a FRESH per-run STAGING tree first. An architecture build
# run STOPS THERE: publishing is a separate --publish step that takes ONE GLOBAL lock, assembles
# into a side tree, gates it, and swaps it onto the published path with a single rename(2). The
# two arches build concurrently on their two hosts, so a per-arch build that also published would
# interleave wipes of the same pool/dists/Release; and a partial or failed build must never reach
# the published repo, nor stale debs accumulate in it (concern #1).
# 3. Per-arch package sets come from the manifest: the x86 boot components (syslinux/elilo/xnba,
# Architecture:all) are built once on amd64 (single producer); ppc64el builds only the genuinely
# arch-specific compiled deps (concern #3).
@@ -37,9 +40,11 @@ use Fcntl qw(:flock);
use FindBin qw($RealBin);
use lib $RealBin;
use BuildUtils qw(sh_quote print_step version_matches required_pkgs read_manifest standard_options
verify_repo_packages verify_repo_signature parse_packages_index resolve_present_names
verify_repo_packages verify_repo_signature verify_repo_arches
parse_packages_index parse_release_architectures resolve_present_names
index_has_native_arch control_binary_arch skip_arch_all_on
codename_to_version known_codenames chroot_name chroot_sources_list
chroot_is_disposable
control_field genesis_deb_control
deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb);
@@ -64,12 +69,24 @@ my $parallel_targets = 0;
my ($skip_build, $skip_install, $skip_genesis, $skip_xcat_dep) = (0,0,0,0);
my ($skip_createrepo, $skip_tarball) = (0,0);
my $dry_run = 0;
# --publish: run the FINALIZATION phase (assemble + sign + gate + tarball). See the "Publishing"
# comment block below: an architecture build run stages only; publishing is a separate, singly-locked,
# atomic step. undef = not specified -> defaulted from --skip-build.
my $publish;
# --expect-arch: the architecture set the PUBLISHED repo must serve, stated EXPLICITLY (repeatable,
# and each value may be a space/comma list). This is what stops "an entirely missing secondary
# architecture" from reading as "this run simply did not build it" -- see resolve_expect_arches().
my @expect_arch;
# Completeness+signature gate on the PUBLISHED apt index (what apt clients see). $verify_repo_arg set
# (--verify-repo=<apt_dir>) runs the gate STANDALONE against that assembled apt dir and exits (no lock,
# no build). $no_verify_repo suppresses the AUTOMATIC post-assembly gate that otherwise runs at the end
# of assemble_apt. Default: automatic gate ON.
# no build). $no_verify_repo suppresses the AUTOMATIC post-publish gate that otherwise runs before the
# assembled tree is swapped into place. Default: automatic gate ON.
my $verify_repo_arg = '';
my $no_verify_repo = 0;
# Signature verification is ON by default for the standalone gate (--verify-repo): an unsigned or
# wrongly-signed repo is a real defect, and silently skipping the check because no --gpg-home happened
# to be passed is a false PASS. --no-verify-signature is the explicit, deliberate opt-out.
my $no_verify_signature = 0;
my $gpg_sign = 0;
my $gpg_key_id = 'xcat@megware.com';
my $gpg_home = '';
@@ -142,8 +159,11 @@ $spec{'genesis-deb=s'} = \@genesis_debs;
$spec{'genesis-rpm=s'} = \$genesis_rpm;
$spec{'genesis-rpm-ppc=s'} = \$genesis_rpm_ppc;
$spec{'require-ppc-genesis!'} = \$require_ppc_genesis;
$spec{'publish!'} = \$publish; # run the finalization (assemble+sign+gate+tarball)
$spec{'expect-arch=s'} = \@expect_arch; # repeatable; each value may be a space/comma list
$spec{'verify-repo=s'} = \$verify_repo_arg; # standalone gate: --verify-repo=<apt_dir>
$spec{'no-verify-repo!'} = \$no_verify_repo; # suppress the automatic post-assembly gate
$spec{'no-verify-repo!'} = \$no_verify_repo; # suppress the automatic pre-swap gate
$spec{'no-verify-signature!'} = \$no_verify_signature; # explicit opt-out of the signature check
$spec{'output=s'} = \$output_root; # --output alias
$spec{'help|h'} = sub { pod2usage(-verbose => 1, -exitval => 0); };
$spec{'man'} = sub { pod2usage(-verbose => 2, -exitval => 0); };
@@ -179,6 +199,29 @@ for my $cn (@dist_list) {
die "FATAL: unknown codename '$cn' (known: @{[known_codenames()]})\n" unless codename_to_version($cn);
}
@expect_arch = grep { length } map { split /[\s,]+/ } @expect_arch;
for my $a (@expect_arch) {
die "FATAL: unsupported --expect-arch '$a' (amd64|ppc64el)\n" unless $a =~ /^(amd64|ppc64el)$/;
}
# ---------------------------------------------------------------------------------------------------
# Publishing: an ARCH BUILD RUN STAGES ONLY; publishing is a SEPARATE, SINGLY-LOCKED, ATOMIC step.
#
# The two arches build CONCURRENTLY on their two hosts against the same --apt-dir on the shared tree.
# If both were allowed to assemble, they would wipe and repopulate the same pool/, dists/, Release,
# InRelease and tarball at the same time and interleave into a corrupt (but green) repository. The
# per-arch run lock does NOT prevent that -- it is per-arch by design, so the two arches can build in
# parallel (PR #63 review concern #1).
#
# So the phases are split by ROLE:
# * a run that BUILDS (no --skip-build) produces staging artifacts and stops there;
# * publishing happens only when asked for with --publish -- or implicitly on a run that builds
# nothing (--skip-build), which IS the finalization step -- and it takes ONE GLOBAL publish lock
# and swaps a fully-assembled, fully-verified tree into place with a single rename().
# --skip-createrepo (the shared CLI's "do not build a repo") always wins and suppresses publishing.
$publish = ($skip_build ? 1 : 0) unless defined $publish;
$publish = 0 if $skip_createrepo;
$output_root ||= "$repo_root/build-output/sbuild-all";
$apt_dir ||= "$repo_root/repos/apt";
$run_id ||= strftime("%Y%m%d-%H%M%S", localtime());
@@ -190,13 +233,24 @@ my %MANIFEST = read_manifest($manifest);
# Standalone gate: --verify-repo=<apt_dir> checks an already-assembled apt tree (completeness +
# Release signatures) using THIS script's manifest resolution (--manifest or the default
# debs-manifest.conf), --dists, and --gpg-key-id/--gpg-home, then exits. It takes NO run lock and does
# NOT build. Dispatched here (after manifest + @dist_list are resolved) so it never trips the
# build-only per-target section check below and never reaches the lock/build phases.
# debs-manifest.conf), --dists, --expect-arch, and --gpg-key-id/--gpg-home, then exits. It takes NO run
# lock and does NOT build. Dispatched here (after manifest + @dist_list are resolved) so it never trips
# the build-only per-target section check below and never reaches the lock/build phases.
#
# Two deliberate STRICTNESS choices here (PR #63 review concern #3), both of which used to be
# false-PASSes:
# * The signature IS verified by default. It used to be skipped unless --gpg-home happened to be
# given, so the common `--verify-repo <dir>` invocation silently checked completeness only.
# --no-verify-signature is the explicit opt-out.
# * The expected architecture set is a CLAIM, never "whatever is present": --expect-arch if given,
# else the arch set each codename's own Release advertises to apt clients. An entirely missing
# binary-<arch> therefore reads as MISSING-ARCH, not as "that arch was not built this run".
if (length $verify_repo_arg) {
die "FATAL: --verify-repo apt dir not found: $verify_repo_arg\n" unless -d $verify_repo_arg;
print_step('Standalone repo verification (no build, no lock)');
verify_assembled_repo(\%MANIFEST, abs_path($verify_repo_arg), \@dist_list, [$arch]);
my $adir = abs_path($verify_repo_arg);
verify_assembled_repo(\%MANIFEST, $adir, \@dist_list,
resolve_expect_arches('standalone', $adir), !$no_verify_signature);
exit 0;
}
@@ -238,6 +292,11 @@ print " apt-dir: $apt_dir\n";
print " staging: $staging\n";
print " run-id: $run_id snap-ts: $snap_ts build-number: " . (defined $build_number ? $build_number : '(none)') . "\n";
print " gpg-sign: " . ($gpg_sign ? "yes (key $gpg_key_id)" : "no") . "\n";
print " publish: " . ($publish
? "yes (assemble + sign + gate + swap into $apt_dir, under the global publish lock)"
: "no (STAGING ONLY -- re-run with --publish, or run the separate finalization step)") . "\n";
print " expect-arch: " . (@expect_arch ? "@expect_arch" : '(derive from the staged arch set)') . "\n"
if $publish;
print " dry-run: " . ($dry_run ? "yes" : "no") . "\n";
# ---------------------------------------------------------------------------------------------------
@@ -294,6 +353,35 @@ sub have_chroot {
my $out = `schroot -l 2>/dev/null`;
return $out =~ /^chroot:\Q$name\E$/m ? 1 : 0;
}
# ensure_disposable_chroot($name): make sure a `schroot -c <name> -- ...` session gets a THROWAWAY
# filesystem, so each package builds in a clean environment and one package's build-dependencies can
# never leak into the next (PR #63 review concern #2 -- see BuildUtils::chroot_is_disposable).
# sbuild-createchroot normally writes union-type=overlay; a chroot created before that (or by hand)
# may not have it. Repair the chroot.d entry in place, then re-check and hard-fail if it still is not
# disposable -- silently building in a shared, mutable chroot is exactly what must not happen.
sub ensure_disposable_chroot {
my ($name) = @_;
return 1 if $dry_run;
my $cfg = `schroot --config -c ${\ sh_quote($name) } 2>/dev/null` // '';
return 1 if chroot_is_disposable($cfg);
my ($file) = grep { -f $_ && do { local $/; open my $fh, '<', $_ or 0;
my $t = <$fh>; close $fh; $t =~ /^\[\Q$name\E\]/m } }
glob('/etc/schroot/chroot.d/*');
if ($file) {
print " chroot $name: NOT disposable -> adding union-type=overlay to $file\n";
run("sed -i " . sh_quote("/^union-type=/d") . " " . sh_quote($file));
run("printf '%s\\n' 'union-type=overlay' >> " . sh_quote($file));
$cfg = `schroot --config -c ${\ sh_quote($name) } 2>/dev/null` // '';
}
die "FATAL: chroot $name is not disposable (no union/snapshot session): a build would mutate the\n"
. " shared base filesystem and leak build-dependencies into the next package. Add\n"
. " 'union-type=overlay' to its /etc/schroot/chroot.d/ entry, or delete the chroot so this\n"
. " script re-creates it.\n"
unless chroot_is_disposable($cfg);
return 1;
}
sub ensure_chroots {
print_step('Ensure sbuild chroots (auto-init on first run)');
die "FATAL: chroot init requires root (uid=$>)\n" if $> != 0 && !$dry_run;
@@ -305,7 +393,11 @@ sub ensure_chroots {
."echo '/opt/xcat-ci-shared /opt/xcat-ci-shared none rw,bind 0 0' >> $fstab", nofail => 1);
for my $cn (@dist_list) {
my $name = chroot_name($cn, $arch);
if (have_chroot($name)) { print " chroot $name: present\n"; next; }
if (have_chroot($name)) {
print " chroot $name: present\n";
ensure_disposable_chroot($name);
next;
}
print " chroot $name: MISSING -> creating\n";
# debootstrap may lack a script for a new codename -> fall back to the generic one.
run("[ -e /usr/share/debootstrap/scripts/$cn ] || ln -sf gutsy /usr/share/debootstrap/scripts/$cn", nofail => 1);
@@ -321,6 +413,8 @@ sub ensure_chroots {
} else { print "+ write $root/etc/apt/sources.list (main+universe)\n"; }
run("mkdir -p $root/opt/xcat-ci-shared", nofail => 1); # bind-mount target for the shared tree
die "FATAL: chroot $name still absent after create\n" if !$dry_run && !have_chroot($name);
# Each package MUST build in a throwaway session (see ensure_disposable_chroot).
ensure_disposable_chroot($name);
print " chroot $name: created\n";
}
}
@@ -644,59 +738,120 @@ sub sig_observed_key {
return $obs_fpr ne '' ? $obs_fpr : undef;
}
# verify_assembled_repo($manifest_href, $apt_dir, $dists_aref): the ONE completeness+signature gate,
# shared by the automatic post-assembly run (end of assemble_apt) and the standalone --verify-repo mode.
# It is the IO layer: it PARSES the repository (parse_packages_index of each published
# binary-<arch>/Packages -> %present; gpg --verify of each dists/<cn>/InRelease -> %observed signer),
# PARSES the manifest (-> %expected pkg pins per cell) and resolves the GPG key (--gpg-key-id -> expected
# signer), then delegates the DECISION to the pure verify_repo_packages (per codename x arch) and
# verify_repo_signature (per codename). Package problems are [<cn>/<arch>]-prefixed; the pure signature
# problems already carry the codename unit. Any problem dies non-zero.
# release_claimed_arches($adir, $cn): the architecture set dists/<cn>/Release ADVERTISES to apt
# clients ("Architectures:"), or the empty list when there is no Release. The repo's own claim, which
# is what the standalone gate holds it to when no --expect-arch is given.
sub release_claimed_arches {
my ($adir, $cn) = @_;
my $rel = "$adir/dists/$cn/Release";
$rel = "$adir/dists/$cn/InRelease" unless -f $rel;
return () unless -f $rel;
open my $fh, '<', $rel or return ();
local $/; my $t = <$fh>; close $fh;
return parse_release_architectures($t);
}
# resolve_expect_arches($mode, $adir): the architecture set the published repo is REQUIRED to serve.
#
# This is the crux of PR #63 review concern #3. It must be a CLAIM someone made, never an inference
# from "which binary-<arch> directories happen to be populated" -- because inferring it from presence
# is precisely what turns an entirely missing secondary architecture into a silent PASS ("ppc64el is
# absent, so this run must not have built ppc64el"). Sources, in order:
# 1. --expect-arch -- explicit; always wins. This is what the CD pipeline passes.
# 2. mode 'publish' -- the arch set actually STAGED under staging/<cn>/<arch> (every one of
# which passed validate_manifest), unioned with this host's --arch.
# That is the set this publish intends to ship.
# 3. mode 'standalone' -- the union of what each codename's own Release advertises, unioned
# with --arch. Verifying a repo against its published claim is what
# catches "Release says amd64 ppc64el, but binary-ppc64el is missing".
sub resolve_expect_arches {
my ($mode, $adir) = @_;
if (@expect_arch) {
my %u = map { $_ => 1 } @expect_arch;
return [sort keys %u];
}
my %u = ($arch => 1);
if ($mode eq 'publish') {
for my $cn (@dist_list) {
for my $d (glob("$staging/$cn/*")) {
next unless -d $d;
my $a = basename($d);
$u{$a} = 1 if $a =~ /^(amd64|ppc64el)$/;
}
}
print " expected arches (from the staged set): " . join(' ', sort keys %u) . "\n";
} else {
for my $cn (@dist_list) { $u{$_} = 1 for release_claimed_arches($adir, $cn); }
print " expected arches (from each codename's Release 'Architectures:'): "
. join(' ', sort keys %u) . "\n";
}
return [sort keys %u];
}
# verify_assembled_repo($manifest_href, $apt_dir, $dists_aref, $expect_arches_aref, $sig_required):
# the ONE completeness+signature gate, shared by the automatic pre-swap run (inside publish_repo) and
# the standalone --verify-repo mode. It is the IO layer: it PARSES the repository (parse_packages_index
# of each published binary-<arch>/Packages -> %present; gpg --verify of each dists/<cn>/InRelease ->
# %observed signer), PARSES the manifest (-> %expected pkg pins per cell) and resolves the GPG key
# (--gpg-key-id -> expected signer), then delegates the DECISION to the pure verify_repo_arches (the
# arch set), verify_repo_packages (per codename x arch) and verify_repo_signature (per codename).
# Package problems are [<cn>/<arch>]-prefixed; the pure arch/signature problems already carry their
# unit. Any problem dies non-zero.
#
# $expect_arches_aref is the EXPECTED arch set from resolve_expect_arches -- required and non-empty.
# Every expected arch is checked (a missing index is MISSING-INDEX, an index with no native package is
# MISSING-ARCH), and any arch that published natives WITHOUT being expected is UNEXPECTED-ARCH. The
# gate never derives what it should demand from what it happens to find.
sub verify_assembled_repo {
my ($man, $adir, $dists, $arches, $sig_enabled) = @_;
my ($man, $adir, $dists, $arches, $sig_required) = @_;
my @all;
# $sig_enabled: whether a valid signature is REQUIRED. The post-assembly auto-run passes $gpg_sign
# -- a repo assembled WITHOUT --gpg-sign is intentionally unsigned, so don't demand a signature and
# false-fail. Standalone --verify-repo passes undef -> fall back to "a gpg key/home is configured".
$sig_enabled = ($gpg_home ne '' || $gpg_sign) unless defined $sig_enabled;
my $expected_key = $sig_enabled ? resolve_expected_key() : undef;
my @expected = sort @{ $arches || [] };
die "FATAL: verify_assembled_repo: no expected architecture set (pass --expect-arch)\n"
unless @expected;
# $sig_required: whether a valid signature is DEMANDED. The pre-swap auto-run passes $gpg_sign --
# a repo assembled WITHOUT --gpg-sign is intentionally unsigned, so don't demand a signature and
# false-fail. Standalone --verify-repo passes !--no-verify-signature, i.e. ON unless opted out.
my $expected_key = $sig_required ? resolve_expected_key() : undef;
my $expected_is_fpr = defined($expected_key) ? 1 : 0;
print " apt-dir: $adir\n";
print " signature check: " . ($sig_enabled
print " expected arches: " . join(' ', @expected) . "\n";
print " signature check: " . ($sig_required
? "on (expected key " . ($expected_key // $gpg_key_id) . ($expected_is_fpr ? '' : ' [UNRESOLVED -> hard fail]') . ")"
: "SKIPPED (no --gpg-home/--gpg-sign)") . "\n";
: "OFF (" . ($no_verify_signature ? '--no-verify-signature'
: 'repo assembled without --gpg-sign') . ")") . "\n";
# STRICT: if signing is expected but --gpg-key-id does not resolve to a fingerprint (not in the
# keyring), we CANNOT confirm the signer -- that is a hard failure, never a presence-only pass.
push @all, "SIGKEY: cannot resolve --gpg-key-id '$gpg_key_id' to a fingerprint (in the $gpg_home keyring?)"
if $sig_enabled && !$expected_is_fpr;
push @all, "SIGKEY: cannot resolve --gpg-key-id '$gpg_key_id' to a fingerprint (in the "
. ($gpg_home ne '' ? $gpg_home : 'ambient GNUPGHOME') . " keyring?)"
if $sig_required && !$expected_is_fpr;
my (%exp_sig, %obs_sig, %checked_arch);
my (%exp_sig, %obs_sig);
for my $cn (@$dists) {
# Arches to verify for THIS codename: the caller's --arch set (the required FLOOR -- always
# checked, so an explicitly-requested arch whose index is empty/missing is still caught)
# UNIONed with any arch that actually published a non-empty binary-<arch>/Packages. This lets
# the multi-arch assemble (invoked with a single --arch amd64) still verify the ppc64el debs it
# carries, while a genuinely single-arch run (e.g. BUILD_PPC=false, amd64 only) never
# false-fails demanding an arch it did not build.
my %want = map { $_ => 1 } @{ $arches || [] };
# Add an arch iff it published NATIVE debs (a Packages stanza with Architecture == that arch),
# NOT merely a non-empty index: every binary-<arch>/Packages carries the Architecture:all debs
# (grub2-xcat, genesis), so a non-empty ppc index does NOT imply ppc was built. Native-arch
# detection keeps a genuine single-arch run (BUILD_PPC=false) from demanding the ppc section.
for my $a (qw(amd64 ppc64el)) {
# Which arches carry NATIVE debs here (a Packages stanza with Architecture == that arch), as
# opposed to only the Architecture:all debs (grub2-xcat, genesis) that ride into EVERY
# binary-<arch> index. Collected for the EXPECTED set plus the other supported arches, so the
# pure verify_repo_arches can report both directions: expected-but-absent and present-but-
# unexpected (a stale arch left behind in the tree).
my %native;
for my $a (do { my %s = map { $_ => 1 } (@expected, qw(amd64 ppc64el)); sort keys %s }) {
my $idx = "$adir/dists/$cn/main/binary-$a/Packages";
$native{$a} = 0;
next unless -f $idx;
open my $ifh, '<', $idx or next;
local $/; my $body = <$ifh>; close $ifh;
$want{$a} = 1 if index_has_native_arch($body, $a);
$native{$a} = index_has_native_arch($body, $a) ? 1 : 0;
}
push @all, map { "[$cn] $_" } verify_repo_arches(\@expected, \%native);
# completeness: manifest (source of truth) vs the PUBLISHED index, per codename x arch.
for my $a (sort keys %want) {
$checked_arch{$a} = 1;
for my $a (@expected) {
my $tgt = "$cn-$a";
my $req = $man->{$tgt};
# A cell we are REQUIRED to serve but have no manifest section for cannot be verified at
# all -- that is a configuration error, not a reason to pass it. (It used to be silently
# skipped, which made an unlisted target a free PASS.)
unless ($req && %$req) {
print " [$cn/$a] no manifest section [$tgt] -- skipping (codename does not target this arch)\n";
push @all, "[$cn/$a] NO-MANIFEST section [$tgt] in $manifest, but $a is expected";
next;
}
my @names = required_pkgs([sort keys %$req], $skip_genesis, $skip_xcat_dep);
@@ -711,73 +866,111 @@ sub verify_assembled_repo {
}
# signature IO: record the expected + observed signer for this codename (compared in bulk below).
# Only when the expected key resolved to a fingerprint (else the SIGKEY hard-fail above stands).
if ($sig_enabled && $expected_is_fpr) {
if ($sig_required && $expected_is_fpr) {
$exp_sig{$cn} = $expected_key;
$obs_sig{$cn} = sig_observed_key($adir, $cn);
}
}
push @all, verify_repo_signature(\%exp_sig, \%obs_sig) if $sig_enabled && $expected_is_fpr;
push @all, verify_repo_signature(\%exp_sig, \%obs_sig) if $sig_required && $expected_is_fpr;
if (@all) {
print "$_\n" for @all;
die "FATAL: apt repo INCOMPLETE (" . scalar(@all) . " problem(s))\n";
}
print "[verify-repo] complete: all required packages present + version-pinned"
. ($sig_enabled ? " + Release signatures valid (key $expected_key)" : "")
. " for [" . join(' ', @$dists) . "] x {" . join(',', sort keys %checked_arch) . "}\n";
. ($sig_required ? " + Release signatures valid (key $expected_key)" : "")
. " for [" . join(' ', @$dists) . "] x {" . join(',', @expected) . "}\n";
return;
}
# ---------------------------------------------------------------------------------------------------
# Phase: assemble + sign the apt repo (absorbed build-apt-repo.sh; promote-on-success)
# Phase: PUBLISH -- the single, locked, atomic finalization (absorbed build-apt-repo.sh).
#
# Concern #1 of the PR #63 review: the amd64 and ppc64el build runs execute CONCURRENTLY on their two
# hosts against the same --apt-dir. Two publishers there would interleave their wipes and rewrites of
# pool/, dists/, Release, InRelease and the tarball. The fix has three parts:
#
# 1. Build runs do not publish at all (see the $publish decision above): they only fill staging.
# Publishing is a separate step, run once, after every arch has staged and validated.
# 2. That step takes ONE GLOBAL publish lock -- not the per-arch build lock -- so even a mistaken
# second publisher (cron racing a manual run on the same host) serializes instead of interleaving.
# 3. It assembles into a SIDE TREE and swaps it in with rename(2) once the gate has passed. A reader
# (deploy.sh's rsync, an apt client on a served tree) therefore only ever sees the previous
# complete repo or the new complete repo -- never a half-wiped pool or an index that does not
# match its Release. A failed gate leaves the published tree untouched.
#
# The lock file lives on the shared tree; within a host flock() is authoritative, which is what
# matters, since the pipeline's finalization step always runs on one host (the amd64 Ubuntu builder).
# ---------------------------------------------------------------------------------------------------
sub assemble_apt {
return if $skip_createrepo;
print_step('Assemble apt repo (promote validated staging -> published repo)');
run("command -v apt-ftparchive >/dev/null 2>&1 || { echo 'need apt-utils' >&2; exit 1; }", always => 1) unless $dry_run;
my $PUBLISH_LOCK_FH;
my $PUBLISH_LOCK_WAIT = 1800; # seconds to wait for a concurrent publisher before giving up
sub acquire_publish_lock {
make_path($output_root);
my $lockfile = "$output_root/.sbuild-all.publish.lock";
open($PUBLISH_LOCK_FH, '>', $lockfile) or die "FATAL: cannot open publish lock $lockfile: $!\n";
unless (flock($PUBLISH_LOCK_FH, LOCK_EX | LOCK_NB)) {
print " publish lock is held by another run -- waiting up to ${PUBLISH_LOCK_WAIT}s: $lockfile\n";
local $SIG{ALRM} = sub {
die "FATAL: timed out after ${PUBLISH_LOCK_WAIT}s waiting for the publish lock $lockfile\n";
};
alarm($PUBLISH_LOCK_WAIT);
my $ok = flock($PUBLISH_LOCK_FH, LOCK_EX);
alarm(0);
die "FATAL: cannot take the publish lock $lockfile: $!\n" unless $ok;
}
print " publish lock acquired: $lockfile\n";
return $lockfile;
}
# assemble_into($dir, $expect_arches): (re)assemble every --dists codename inside $dir from the
# validated staging tree, index it per expected binary-<arch>, write + sign Release. $dir is the SIDE
# tree, never the published one.
sub assemble_into {
my ($dir, $expect) = @_;
for my $cn (@dist_list) {
my $ver = codename_to_version($cn);
my $pool = "$apt_dir/pool/main/$cn";
# wipe ONLY this codename's published pool+dists, then repopulate from validated staging
# (both arches: staging/<cn>/{amd64,ppc64el}/*.deb). Wiping first is what removes stale debs
# from a prior run so the published repo never carries a mixture (concern #1).
wipe_tree($pool, "$apt_dir/dists/$cn", "$apt_dir/$ver") unless $dry_run;
make_path($pool, "$apt_dir/$ver") unless $dry_run;
unless ($dry_run) {
# Collect this codename's staged debs across both arches, deduping on binary package
# NAME+ARCH: if two files resolve to the same package+arch (e.g. a native ppc genesis and
# an amd64-host cross-converted one both claiming xcat-genesis-base-ppc64el/all) only ONE
# may reach the pool. Keep the highest version and warn naming both -- a safety net that
# holds regardless of the --skip-genesis single-producer contract (concern #4).
my %best; # "name|arch" => { file => path, ver => version }
for my $deb (glob("$staging/$cn/*/*.deb")) {
my $name = deb_field($deb, 'Package');
my $darch = deb_field($deb, 'Architecture');
my $dver = deb_field($deb, 'Version');
my $key = "$name|$darch";
if (my $cur = $best{$key}) {
my $new_wins = deb_ver_gt($dver, $cur->{ver});
my ($win, $lose) = $new_wins ? ($deb, $cur->{file}) : ($cur->{file}, $deb);
warn "WARN: duplicate binary $name/$darch in staging for $cn -- keeping "
. basename($win) . ", dropping " . basename($lose) . "\n";
$best{$key} = { file => $deb, ver => $dver } if $new_wins;
next;
}
$best{$key} = { file => $deb, ver => $dver };
}
for my $key (sort keys %best) {
my $deb = $best{$key}{file};
my $b = basename($deb);
link($deb, "$pool/$b") or copy($deb, "$pool/$b");
copy($deb, "$apt_dir/$ver/$b");
my $pool = "$dir/pool/main/$cn";
# wipe ONLY this codename's pool+dists in the side tree, then repopulate from validated
# staging (all arches: staging/<cn>/{amd64,ppc64el}/*.deb). Wiping first is what removes stale
# debs from a prior run so the published repo never carries a mixture.
wipe_tree($pool, "$dir/dists/$cn", "$dir/$ver");
make_path($pool, "$dir/$ver");
# Collect this codename's staged debs across both arches, deduping on binary package
# NAME+ARCH: if two files resolve to the same package+arch (e.g. a native ppc genesis and
# an amd64-host cross-converted one both claiming xcat-genesis-base-ppc64el/all) only ONE
# may reach the pool. Keep the highest version and warn naming both -- a safety net that
# holds regardless of the --skip-genesis single-producer contract.
my %best; # "name|arch" => { file => path, ver => version }
for my $deb (glob("$staging/$cn/*/*.deb")) {
my $name = deb_field($deb, 'Package');
my $darch = deb_field($deb, 'Architecture');
my $dver = deb_field($deb, 'Version');
my $key = "$name|$darch";
if (my $cur = $best{$key}) {
my $new_wins = deb_ver_gt($dver, $cur->{ver});
my ($win, $lose) = $new_wins ? ($deb, $cur->{file}) : ($cur->{file}, $deb);
warn "WARN: duplicate binary $name/$darch in staging for $cn -- keeping "
. basename($win) . ", dropping " . basename($lose) . "\n";
$best{$key} = { file => $deb, ver => $dver } if $new_wins;
next;
}
$best{$key} = { file => $deb, ver => $dver };
}
# Packages index per binary-<arch>: an arch's index carries that arch's debs + all Architecture:all.
for my $a (qw(amd64 ppc64el)) {
my $bindir = "$apt_dir/dists/$cn/main/binary-$a";
make_path($bindir) unless $dry_run;
next if $dry_run;
my $all = `cd ${\ sh_quote($apt_dir)} && apt-ftparchive packages pool/main/$cn`;
for my $key (sort keys %best) {
my $deb = $best{$key}{file};
my $b = basename($deb);
link($deb, "$pool/$b") or copy($deb, "$pool/$b");
copy($deb, "$dir/$ver/$b");
}
# Packages index per EXPECTED binary-<arch>: an arch's index carries that arch's debs + all
# Architecture:all. Only the expected arches get an index -- writing a binary-ppc64el index
# containing nothing but the Architecture:all debs would advertise a ppc64el repo that cannot
# actually satisfy a ppc64el client (and would then be mistaken for "ppc was published").
my $all = `cd ${\ sh_quote($dir)} && apt-ftparchive packages pool/main/$cn`;
for my $a (@$expect) {
my $bindir = "$dir/dists/$cn/main/binary-$a";
make_path($bindir);
open my $pf, '>', "$bindir/Packages" or die "write Packages: $!\n";
for my $para (split /\n\n+/, $all) {
next unless $para =~ /\S/;
@@ -787,61 +980,130 @@ sub assemble_apt {
close $pf;
run("gzip -9 -kf -n " . sh_quote("$bindir/Packages"));
}
next if $dry_run;
# Advertise ONLY the arches actually staged for this codename: an arch counts iff its
# binary-<arch>/Packages is non-empty. A single-arch run must not claim a missing arch in
# Release (apt would then error on the absent index).
my @staged_arches = grep { -s "$apt_dir/dists/$cn/main/binary-$_/Packages" } qw(amd64 ppc64el);
@staged_arches = ('amd64') unless @staged_arches; # never emit an empty Architectures line
# Release + sign
# Release advertises EXACTLY the expected arch set -- the same claim the gate then holds the
# tree to, and the claim a later standalone --verify-repo falls back on.
my @rel = ('apt-ftparchive',
'-o', 'APT::FTPArchive::Release::Origin=xCAT',
'-o', 'APT::FTPArchive::Release::Label=xcat-dep',
'-o', "APT::FTPArchive::Release::Suite=$cn",
'-o', "APT::FTPArchive::Release::Codename=$cn",
'-o', 'APT::FTPArchive::Release::Architectures=' . join(' ', @staged_arches),
'-o', 'APT::FTPArchive::Release::Architectures=' . join(' ', @$expect),
'-o', 'APT::FTPArchive::Release::Components=main',
'-o', "APT::FTPArchive::Release::Description=xCAT dependency packages for $ver",
'release', "$apt_dir/dists/$cn/");
run(join(' ', map { sh_quote($_) } @rel) . " > " . sh_quote("$apt_dir/dists/$cn/Release"));
'release', "$dir/dists/$cn/");
run(join(' ', map { sh_quote($_) } @rel) . " > " . sh_quote("$dir/dists/$cn/Release"));
my $det = strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime($build_timestamp));
run("sed -i " . sh_quote("s/^Date: .*/Date: $det/") . " " . sh_quote("$apt_dir/dists/$cn/Release"), nofail => 1);
run("sed -i " . sh_quote("s/^Date: .*/Date: $det/") . " " . sh_quote("$dir/dists/$cn/Release"), nofail => 1);
if ($gpg_sign) {
my $g = $gpg_home ? "GNUPGHOME=" . sh_quote($gpg_home) . " " : '';
my $rel = "$apt_dir/dists/$cn/Release";
my $rel = "$dir/dists/$cn/Release";
run("${g}gpg --default-key " . sh_quote($gpg_key_id) . " --batch --yes --armor --detach-sign -o "
. sh_quote("$rel.gpg") . " " . sh_quote($rel));
run("${g}gpg --default-key " . sh_quote($gpg_key_id) . " --batch --yes --armor --clearsign -o "
. sh_quote("$apt_dir/dists/$cn/InRelease") . " " . sh_quote($rel));
. sh_quote("$dir/dists/$cn/InRelease") . " " . sh_quote($rel));
}
print " assembled + " . ($gpg_sign ? 'signed' : 'UNSIGNED') . " apt tree for $cn\n";
print " assembled + " . ($gpg_sign ? 'signed' : 'UNSIGNED') . " apt tree for $cn ("
. join(' ', @$expect) . ")\n";
}
# export the signing pubkey for clients
if ($gpg_sign && !$dry_run) {
if ($gpg_sign) {
my $g = $gpg_home ? "GNUPGHOME=" . sh_quote($gpg_home) . " " : '';
my $keysrc = "$repo_root/repomd.xml.key";
if (-f $keysrc) { copy($keysrc, "$apt_dir/xcat-dep.asc"); }
else { run("${g}gpg --armor --export " . sh_quote($gpg_key_id) . " > " . sh_quote("$apt_dir/xcat-dep.asc"), nofail => 1); }
}
# Automatic post-assembly GATE on the just-published index (completeness + Release signatures).
# Runs once every codename's dists/<cn>/.../Packages + signed Release are written. Suppressed with
# --no-verify-repo (iteration/debug); skipped under --dry-run (nothing was published).
unless ($no_verify_repo || $dry_run) {
print_step('Verify published apt repo (post-assembly completeness + signature gate)');
# Require a valid signature iff we actually signed (--gpg-sign); a repo assembled without it is
# intentionally unsigned and must not false-fail. (Standalone --verify-repo omits this arg.)
verify_assembled_repo(\%MANIFEST, $apt_dir, \@dist_list, [$arch], $gpg_sign);
if (-f $keysrc) { copy($keysrc, "$dir/xcat-dep.asc"); }
else { run("${g}gpg --armor --export " . sh_quote($gpg_key_id) . " > " . sh_quote("$dir/xcat-dep.asc"), nofail => 1); }
}
}
# swap_into_place($tmp, $target): publish $tmp AS $target with rename(2) -- the only moment the
# published repo changes, and it changes all at once. Both paths are siblings, so both renames are
# same-filesystem and atomic. If the second rename fails the previous tree is put straight back.
sub swap_into_place {
my ($tmp, $target) = @_;
my $old = "$target.old-$run_id.$$";
wipe_tree($old) if -d $old;
my $had_old = 0;
if (-d $target) {
rename($target, $old) or die "FATAL: cannot move the published repo aside ($target -> $old): $!\n";
$had_old = 1;
}
unless (rename($tmp, $target)) {
my $err = $!;
rename($old, $target) if $had_old; # put the previous repo back; nothing was lost
die "FATAL: cannot swap the assembled repo into place ($tmp -> $target): $err\n";
}
print " published atomically: $target\n";
wipe_tree($old) if $had_old;
}
sub publish_repo {
unless ($publish) {
print_step('Publish SKIPPED -- this run produced STAGING ONLY');
print " staged under $staging (arch $arch)\n";
print " the apt repo at $apt_dir is untouched: publishing is a separate, singly-locked,\n"
. " atomic step so two concurrent arch builds can never rewrite it at the same time.\n"
. " Finalize with: sbuild-all.pl --skip-build --skip-genesis --publish "
. "--expect-arch \"<arches>\" --gpg-sign ...\n";
return;
}
print_step('Publish apt repo (locked, assembled aside, swapped in atomically)');
my $expect = resolve_expect_arches('publish', $apt_dir);
if ($dry_run) {
print " [dry-run] would lock $output_root/.sbuild-all.publish.lock, assemble @dist_list for "
. join(' ', @$expect) . " into $apt_dir.publish-<run-id>.<pid>, gate it, then rename it "
. "onto $apt_dir\n";
return;
}
run("command -v apt-ftparchive >/dev/null 2>&1 || { echo 'need apt-utils' >&2; exit 1; }", always => 1);
warn "WARN: publishing an UNSIGNED apt repo (no --gpg-sign) -- apt clients will reject it\n"
unless $gpg_sign;
acquire_publish_lock();
# Side-tree name carries the pid too: --run-id has second granularity, and two runs that start in
# the same second must not pick the same scratch path.
my $tmp = "$apt_dir.publish-$run_id.$$";
wipe_tree($tmp) if -e $tmp;
make_path(dirname($apt_dir), $tmp);
# Seed the side tree from the currently published one so codenames OUTSIDE --dists survive the
# swap (a --dists noble run must not drop focal/jammy/resolute). A hardlink copy is cheap and
# safe here: assemble_into only unlinks and re-creates files, it never writes through a link.
if (-d $apt_dir) {
run("cp -al " . sh_quote("$apt_dir/.") . " " . sh_quote("$tmp/") . " 2>/dev/null || "
. "cp -a " . sh_quote("$apt_dir/.") . " " . sh_quote("$tmp/"));
}
# Assemble, then GATE BEFORE PUBLISH: the completeness + signature check runs against the side
# tree, so an incomplete or mis-signed repo is never swapped in -- the previously published tree
# stays exactly as it was. The gate is suppressed with --no-verify-repo (iteration/debug).
# Anything that fails here takes the side tree with it and leaves $apt_dir untouched.
eval {
assemble_into($tmp, $expect);
unless ($no_verify_repo) {
print_step('Verify assembled apt repo (completeness + signature gate, before publishing)');
# Require a valid signature iff we actually signed (--gpg-sign); a repo assembled without
# it is intentionally unsigned and must not false-fail.
verify_assembled_repo(\%MANIFEST, $tmp, \@dist_list, $expect, $gpg_sign);
}
1;
} or do {
my $err = $@ || "unknown error\n";
print " NOT publishing: $apt_dir keeps its previous contents\n";
wipe_tree($tmp) if -d $tmp;
die $err;
};
swap_into_place($tmp, $apt_dir);
make_tarball();
}
# ---------------------------------------------------------------------------------------------------
# Phase: tarball (optional)
# Phase: tarball (optional) -- part of PUBLISH, under the same lock. A per-arch build run must not tar
# the shared apt tree: with both arches running it would archive a tree the other arch is rewriting.
# ---------------------------------------------------------------------------------------------------
sub make_tarball {
return if $skip_tarball;
print_step('Tarball');
my $tb = "$output_root/$run_id/xcat-dep-$arch-$run_id.tar.gz";
make_path(dirname($tb)) unless $dry_run; # the run dir may not exist yet (e.g. an assemble-only run)
make_path(dirname($tb)); # the run dir may not exist yet (e.g. a publish-only run)
run("tar -C " . sh_quote(dirname($apt_dir)) . " -czf " . sh_quote($tb) . " " . sh_quote(basename($apt_dir)), nofail => 1);
print " $tb\n";
}
@@ -855,9 +1117,8 @@ unless ($skip_build) {
}
build_genesis() unless $skip_genesis;
validate_manifest();
assemble_apt();
make_tarball();
print_step("Completed ($arch: @dist_list)");
publish_repo(); # no-op unless --publish (or a --skip-build finalization run); tarball is inside
print_step("Completed ($arch: @dist_list)" . ($publish ? '' : ' -- staging only, not published'));
__END__
@@ -869,17 +1130,26 @@ sbuild-all.pl - build, validate, sign and assemble the xcat-dep Ubuntu/Debian ap
sbuild-all.pl [options]
# build ALL supported Ubuntu versions for this host's arch, sign + assemble the apt tree:
sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \
--xcat-source ../xcat-core --genesis-rpm <xCAT-genesis-base rpm> \
# STEP 1 -- per arch, on that arch's build host: build + validate into staging (does NOT publish):
sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \
--xcat-source ../xcat-core --genesis-rpm <xCAT-genesis-base rpm>
sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" --skip-genesis
# STEP 2 -- ONCE, after every arch has staged: assemble, sign, gate and publish atomically:
sbuild-all.pl --skip-build --skip-genesis --publish --expect-arch "amd64 ppc64el" \
--gpg-sign --gpg-key-id xcat@megware.com --gpg-home <gpg-home>
# build ONE Ubuntu version only:
sbuild-all.pl --arch amd64 --dists noble ...
sbuild-all.pl --target noble-amd64 ... # equivalent single-target form
# assemble-only (re-sign/re-index from already-built staging):
sbuild-all.pl --skip-build --skip-genesis --gpg-sign --gpg-key-id <id> --gpg-home <dir>
# single host, build AND publish in one go (add --publish explicitly):
sbuild-all.pl --arch amd64 --dists noble --genesis-rpm <rpm> --publish --expect-arch amd64 \
--gpg-sign --gpg-key-id <id> --gpg-home <dir>
# verify an already-published tree out of band (signatures checked by DEFAULT):
sbuild-all.pl --verify-repo <apt_dir> --dists "focal jammy noble resolute" \
--expect-arch "amd64 ppc64el" --gpg-key-id <id> --gpg-home <dir>
sbuild-all.pl --help # option summary
sbuild-all.pl --man # this manual
@@ -897,10 +1167,32 @@ packaging (never re-implemented) via its per-package C<< <dep>/sbuild.pl >> buil
One host builds one architecture (C<--arch>, default C<dpkg --print-architecture>) for a set of
Ubuntu codenames (C<--dists>). Each C<< <codename>-<arch> >> is a B<target> with a section in
C<debs-manifest.conf>. Everything is built and validated into a fresh, per-arch B<staging> tree
first; the published apt repo is (re)assembled from validated staging only after the complete
expected set validates -- so a partial or failed build never reaches the repo and stale debs never
accumulate. Any missing chroot / package / artifact, or any version-pin mismatch, fails the whole
run non-zero.
first -- so a partial or failed build never reaches the repo and stale debs never accumulate. Any
missing chroot / package / artifact, or any version-pin mismatch, fails the whole run non-zero.
=head2 Build runs stage; publishing is a separate, locked, atomic step
An architecture build run B<does not publish>. The two arches build concurrently on their two hosts
against the same C<--apt-dir>, so a build run that also assembled would interleave its wipe and
rewrite of C<pool/>, C<dists/>, C<Release>, C<InRelease> and the tarball with the other arch's.
Publishing happens only with C<--publish> -- or implicitly on a run that builds nothing
(C<--skip-build>), which B<is> the finalization step. It takes B<one global publish lock> (not the
per-arch build lock), assembles the whole tree into a side directory, runs the completeness +
signature gate against B<that> tree, and only then swaps it onto C<--apt-dir> with a single
C<rename(2)>. Readers therefore see either the previous complete repo or the new complete repo, never
a half-written one, and a failed gate leaves the published tree untouched. Codenames outside
C<--dists> survive the swap (the side tree is seeded from the current published one).
=head2 Each package builds in a clean, disposable chroot
Every package is built in its own C<schroot> session, and the session must be B<throwaway> (an
overlay/union mount or a snapshot). C<sbuild-all.pl> repairs a chroot that lacks one
(C<union-type=overlay>) and hard-fails if it still is not disposable. That is what makes the fail-hard
dependency handling meaningful: inside the session, C<apt-get update>, the common build tooling and
the package's C<Build-Depends> (resolved with C<mk-build-deps>, so versions/alternatives/arch
qualifiers are honoured) are all B<fatal> on failure -- and because nothing survives the session, a
package whose C<debian/control> forgets a C<Build-Depends> cannot build green on a sibling's leftovers.
=head1 PHASES
@@ -909,12 +1201,14 @@ run non-zero.
=item Ensure chroots
Auto-initializes any missing C<< <codename>-<arch>-sbuild >> chroot on first run (main + universe,
fast mirror, shared-tree bind-mount); idempotent. Skipped with C<--skip-build>.
fast mirror, shared-tree bind-mount) and ensures each one hands out B<disposable> sessions
(C<union-type=overlay>); idempotent. Skipped with C<--skip-build>.
=item Build
Runs each manifest package's C<< <dep>/sbuild.pl >> in the matching chroot into
C<staging/E<lt>codenameE<gt>/E<lt>archE<gt>/>.
C<staging/E<lt>codenameE<gt>/E<lt>archE<gt>/>. Dependency installation inside the chroot is fatal on
failure (see L</"Each package builds in a clean, disposable chroot">).
=item Genesis
@@ -927,29 +1221,42 @@ ppc64el genesis (issue #7610) unless C<--require-ppc-genesis> gates it. Skipped
Asserts every manifest-required package is present at its pinned version (zero tolerance).
=item Assemble
=item Publish (C<--publish>; locked + atomic)
Wipes+repopulates each codename's published C<pool>/C<dists> from validated staging, indexes per
C<binary-E<lt>archE<gt>> (Architecture:all packages land in every arch index) and gpg-signs
C<Release>/C<InRelease>. Skipped with C<--skip-createrepo>.
Takes the global publish lock, seeds a side tree from the currently published repo, wipes+repopulates
each C<--dists> codename's C<pool>/C<dists> there from validated staging, indexes per B<expected>
C<binary-E<lt>archE<gt>> (Architecture:all packages land in every expected arch index), advertises
exactly the expected arch set in C<Release>, gpg-signs C<Release>/C<InRelease>, runs the gate below
and -- only if it passes -- swaps the side tree onto C<--apt-dir> with C<rename(2)>. Suppressed with
C<--skip-createrepo>; not run at all on a build run without C<--publish>.
=item Verify (published-repo gate)
=item Verify (repo gate)
After assembly, a manifest-driven gate asserts -- per codename E<times> arch, against the B<published>
C<binary-E<lt>archE<gt>/Packages> index apt clients actually see (not the staging pool) -- that every
manifest-required package is present at its pinned upstream version, and that each codename's
C<Release> is validly gpg-signed by the expected key (C<InRelease>, or detached C<Release.gpg>). Any
missing package, version mismatch, missing index, or unsigned/wrong-key signature fails the run. The
pure decisions (C<BuildUtils::verify_repo_packages> for completeness, C<BuildUtils::verify_repo_signature>
for the signer) and the index parsing (C<BuildUtils::parse_packages_index>) are unit-tested; this
script's IO layer parses the repository + resolves the gpg key and feeds those pure deciders. This
automatic gate is suppressed with C<--no-verify-repo>; the same check runs standalone against an
already-assembled tree via C<--verify-repo=E<lt>apt_dirE<gt>>.
A manifest-driven gate asserts -- per codename E<times> expected arch, against the
C<binary-E<lt>archE<gt>/Packages> index apt clients will actually see (not the staging pool) -- that
every manifest-required package is present at its pinned upstream version, that the repo serves
exactly the B<expected architecture set>, and that each codename's C<Release> is validly gpg-signed by
the expected key (C<InRelease>, or detached C<Release.gpg>). Any missing package, version mismatch,
missing index, missing/unexpected architecture, missing manifest section, or unsigned/wrong-key
signature fails the run. The pure decisions (C<BuildUtils::verify_repo_arches> for the arch set,
C<verify_repo_packages> for completeness, C<verify_repo_signature> for the signer) and the parsing
(C<parse_packages_index>, C<parse_release_architectures>) are unit-tested; this script's IO layer
parses the repository + resolves the gpg key and feeds those pure deciders.
The expected architecture set is always a B<claim>, never an inference from what happens to be
present: C<--expect-arch> if given, else the staged arch set when publishing, else each codename's own
C<Release> C<Architectures:> line when verifying standalone. An entirely missing secondary
architecture is therefore reported (C<MISSING-ARCH>), not read as "this run did not build it".
The gate runs automatically before the swap (suppress with C<--no-verify-repo>) and standalone against
an already-published tree via C<--verify-repo=E<lt>apt_dirE<gt>>, where signatures are checked B<by
default> (opt out with C<--no-verify-signature>).
=item Tarball
A repo tarball build artifact (the deployable offline FRS dep bundle is produced by the pipeline's
C<deploy.sh --tarball-kind dep>). Skipped with C<--skip-tarball>.
A repo tarball build artifact of the published tree, taken under the publish lock (the deployable
offline FRS dep bundle is produced by the pipeline's C<deploy.sh --tarball-kind dep>). Skipped with
C<--skip-tarball>.
=back
@@ -1015,20 +1322,41 @@ the default gives 8 concurrent build streams for a 4-codename matrix (4 per host
=item B<--skip-build> B<--skip-install> B<--skip-genesis> B<--skip-xcat-dep> B<--skip-createrepo> B<--skip-tarball>
Skip the corresponding phase(s). C<--skip-build --skip-genesis> gives an assemble-only run.
Skip the corresponding phase(s). C<--skip-build --skip-genesis> is the finalization run (it publishes
by default; see C<--publish>). C<--skip-createrepo> forces "do not publish" and always wins.
=item B<--publish>
Run the finalization phase: take the global publish lock, assemble + sign into a side tree, gate it and
swap it onto C<--apt-dir> atomically. B<A build run does not publish unless this is given>, so the two
arches can build concurrently without racing each other on the shared repo. Defaults to on for a run
that builds nothing (C<--skip-build>), which is the finalization step. C<--no-publish> forces it off.
=item B<--expect-arch> C<< amd64|ppc64el >>
The architecture set the published repo must serve, stated explicitly. Repeatable, and each value may
be a space/comma list (C<--expect-arch "amd64 ppc64el">). Used by the gate: an expected arch with no
native package is C<MISSING-ARCH>, an unexpected arch that published natives is C<UNEXPECTED-ARCH>,
and C<Release> advertises exactly this set. Without it the gate falls back to the staged arch set when
publishing, or to each codename's C<Release> C<Architectures:> when verifying standalone.
=item B<--verify-repo> C<< =<apt_dir> >>
Standalone mode: verify an already-assembled apt tree at C<< <apt_dir> >> (completeness + Release
signatures) using this script's manifest resolution (C<--manifest> or the default
C<debs-manifest.conf>), C<--dists>, and C<--gpg-key-id>/C<--gpg-home>, then exit. Takes no run lock and
builds nothing.
Standalone mode: verify an already-published apt tree at C<< <apt_dir> >> (architecture set +
completeness + Release signatures) using this script's manifest resolution (C<--manifest> or the
default C<debs-manifest.conf>), C<--dists>, C<--expect-arch>, and C<--gpg-key-id>/C<--gpg-home>, then
exit. Signatures are verified B<by default>. Takes no run lock and builds nothing.
=item B<--no-verify-repo>
Suppress the B<automatic> post-assembly completeness+signature gate (for iteration/debug). The gate is
Suppress the B<automatic> pre-swap completeness+signature gate (for iteration/debug). The gate is
ON by default.
=item B<--no-verify-signature>
Explicitly skip the Release signature check in the standalone C<--verify-repo> mode (for an
intentionally unsigned local tree). Without it, an unsigned or wrongly-signed repo fails the gate.
=item B<--dry-run>
Print the planned actions without executing them.
t/sbuild-all.t
+151 -3
View File
@@ -13,12 +13,14 @@ use File::Temp qw(tempdir);
use File::Path qw(make_path);
use File::Basename qw(basename);
use BuildUtils qw(required_pkgs version_matches read_manifest standard_options
verify_repo_packages verify_repo_signature parse_packages_index resolve_present_names
verify_repo_packages verify_repo_signature verify_repo_arches
parse_packages_index parse_release_architectures resolve_present_names
index_has_native_arch control_binary_arch skip_arch_all_on
codename_to_version version_to_codename known_codenames
chroot_name chroot_sources_list
chroot_name chroot_sources_list chroot_is_disposable chroot_build_script
control_field genesis_deb_control
deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb);
deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb
build_deb_in_chroot);
# Run a printing sub with STDOUT muted so its progress lines do not pollute TAP.
sub quiet(&) {
@@ -382,4 +384,150 @@ SKIP: {
ok(!skip_arch_all_on($ctl, 'syslinux-xcat', undef), 'undef arch -> not skipped (no crash)');
}
# ---- verify_repo_arches: the expected arch SET is a claim, never "whatever is present" ------------
# PR #63 review concern #3: with the set inferred from presence, an entirely missing secondary
# architecture read as "this run did not build it" and PASSED. These are the two directions that must
# now be reported.
{
my @expect = qw(amd64 ppc64el);
is_deeply([verify_repo_arches(\@expect, { amd64 => 1, ppc64el => 1 })], [],
'every expected arch has native packages -> no problems');
# The false-PASS: a two-arch repo whose ppc64el side is entirely absent.
my @m = verify_repo_arches(\@expect, { amd64 => 1, ppc64el => 0 });
is(scalar(@m), 1, 'an expected arch with no native package -> one problem');
like($m[0], qr/^MISSING-ARCH ppc64el /, '... reported as MISSING-ARCH, not silently passed');
# An arch:all-only ppc index (grub2-xcat/genesis ride into every binary-<arch>) is NOT evidence
# that ppc was published -- index_has_native_arch is what feeds %native, so it stays MISSING-ARCH.
my $allonly = "Package: grub2-xcat\nVersion: 2.12-1\nArchitecture: all\n";
my @m2 = verify_repo_arches(\@expect,
{ amd64 => 1, ppc64el => (index_has_native_arch($allonly, 'ppc64el') ? 1 : 0) });
like($m2[0], qr/^MISSING-ARCH ppc64el /,
'an arch:all-only index does not satisfy an expected arch');
# The other direction: a stale arch left in a tree that is no longer built for it.
my @u = verify_repo_arches(['amd64'], { amd64 => 1, ppc64el => 1 });
is(scalar(@u), 1, 'a native arch outside the expected set -> one problem');
like($u[0], qr/^UNEXPECTED-ARCH ppc64el .*expected set \[amd64\]/,
'... reported as UNEXPECTED-ARCH naming the expected set');
# A genuine single-arch run (BUILD_PPC=false) expects only amd64 and must stay clean.
is_deeply([verify_repo_arches(['amd64'], { amd64 => 1, ppc64el => 0 })], [],
'single-arch expectation with no ppc natives -> no problems (no false MISSING/UNEXPECTED)');
}
# ---- parse_release_architectures: the repo's own claim (standalone-verify fallback) ---------------
{
my $rel = "Origin: xCAT\nLabel: xcat-dep\nSuite: noble\nCodename: noble\n"
. "Architectures: amd64 ppc64el\nComponents: main\nDate: Thu, 21 Aug 2026 00:00:00 +0000\n";
is_deeply([parse_release_architectures($rel)], [qw(amd64 ppc64el)],
'Architectures: line parsed in file order');
is_deeply([parse_release_architectures("Architectures: amd64 amd64 ppc64el \n")],
[qw(amd64 ppc64el)], 'duplicates collapsed, trailing whitespace ignored');
is_deeply([parse_release_architectures("Origin: xCAT\nComponents: main\n")], [],
'no Architectures: field -> empty list (no claim)');
is_deeply([parse_release_architectures(undef)], [], 'undef input -> empty list (no crash)');
}
# ---- chroot_is_disposable: each package must build in a THROWAWAY environment (concern #2) --------
# A shared type=directory chroot with no union mount keeps package N's build-deps installed for
# package N+1, so a missing Build-Depends builds green. build_deb_in_chroot hard-fails on that.
{
my $overlay = "[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\n"
. "union-type=overlay\nprofile=sbuild\n";
ok(chroot_is_disposable($overlay), 'union-type=overlay -> disposable (what sbuild-createchroot sets)');
ok(chroot_is_disposable("type=directory\nunion-type=aufs\n"), 'union-type=aufs -> disposable');
ok(chroot_is_disposable("type=file\nfile=/srv/chroot/noble.tar.gz\n"),
'type=file (tarball unpacked per session) -> disposable');
ok(chroot_is_disposable("type=btrfs-snapshot\n"), 'snapshot chroot -> disposable');
ok(!chroot_is_disposable("[x]\ntype=directory\ndirectory=/srv/chroot/x\nunion-type=none\n"),
'type=directory with union-type=none -> NOT disposable (build-deps would leak)');
ok(!chroot_is_disposable("[x]\ntype=directory\ndirectory=/srv/chroot/x\n"),
'type=directory with no union-type at all -> NOT disposable');
ok(!chroot_is_disposable(''), 'empty config -> not disposable (fail closed)');
ok(!chroot_is_disposable(undef), 'undef config -> not disposable (no crash, fail closed)');
}
# ---- chroot_build_script: the in-chroot build is FAIL-HARD (concern #2) --------------------------
# Regression guard for the exact defects the review named: the common package installation ended in
# `|| true`, and a failed Build-Depends installation only warned -- so a package could build without
# its declared dependencies and ship.
{
my $s = chroot_build_script();
like($s, qr/^set -euo pipefail$/m, 'the in-chroot script aborts on any unchecked failure');
unlike($s, qr/apt-get[^\n]*\|\|\s*true/, 'no `apt-get ... || true` (installation is fatal)');
unlike($s, qr/apt_retry[^\n]*\|\|\s*true/, 'no `apt_retry ... || true`');
unlike($s, qr/\[warn\][^\n]*build-dep/i, 'a failed build-dep install is not downgraded to a warning');
unlike($s, qr/^\s*BD=\$\(sed/m, 'the sed Build-Depends extraction is gone');
like($s, qr/mk-build-deps --install --remove/,
'Build-Depends come from mk-build-deps (honours versions/alternatives/arch qualifiers)');
like($s, qr/apt_retry update -q/, 'apt-get update is retried then fatal');
like($s, qr/apt_retry install -y/, 'the common tooling install is retried then fatal');
like($s, qr/\bequivs\b/, 'equivs is installed (mk-build-deps needs it)');
# the mk-build-deps dummy package must never be collected as build output
like($s, qr/!\s*-name\s+'\*-build-deps_\*'/, 'the *-build-deps dummy deb is excluded from collection');
like($s, qr/!\s*-name\s+'\*-dbgsym_\*'/, 'debug symbols are excluded from collection');
# Every apt-get in the script is either the ONE inside the fatal apt_retry helper, the helper's
# own diagnostics, or the resolver mk-build-deps is told to use -- never a direct, best-effort
# `apt-get ... || true` call.
my @apt = grep { /\bapt-get\b/ && !/^\s*#/ } split /\n/, $s;
my @unaccounted = grep { !/if apt-get "\$\@"; then return 0; fi/
&& !/^\s*echo /
&& !/--tool 'apt-get / } @apt;
is_deeply(\@unaccounted, [],
'every apt-get goes through the fatal apt_retry helper (or is mk-build-deps\' --tool)')
or diag("unaccounted apt-get line(s):\n" . join("\n", @unaccounted));
}
# ---- build_deb_in_chroot REFUSES a non-disposable chroot (the guard, not just the predicate) ------
# Driven with a stub `schroot` on PATH so the guard can be exercised without a real chroot: it answers
# `-l` (the chroot exists), `--config` (the configuration under test) and swallows the build itself.
{
my $fakebin = tempdir(CLEANUP => 1);
open my $fh, '>', "$fakebin/schroot" or die "write stub schroot: $!";
print $fh <<'STUB';
#!/bin/bash
case "$1" in
-l) echo "chroot:noble-amd64-sbuild"; exit 0 ;;
--config) printf '%s\n' "$FAKE_SCHROOT_CONFIG"; exit 0 ;;
esac
exit 0
STUB
close $fh;
chmod 0755, "$fakebin/schroot";
my $work = tempdir(CLEANUP => 1);
make_path("$work/pkg", "$work/out");
my @args = (pkg => 'fixture', chroot => 'noble-amd64-sbuild', pkg_dir => "$work/pkg",
result_dir => "$work/out", build_timestamp => 1755000000, build => "true\n");
{
local $ENV{PATH} = "$fakebin:$ENV{PATH}";
local $ENV{FAKE_SCHROOT_CONFIG} =
"[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\n";
my $ok = eval { quiet { build_deb_in_chroot(@args) }; 1 };
ok(!$ok, 'build_deb_in_chroot refuses to build in a non-disposable chroot');
like($@, qr/is NOT disposable/, '... with an explicit "NOT disposable" error');
like($@, qr/union-type=overlay/, '... telling the operator how to fix it');
}
{
# Same stub, now reporting an overlay: the guard passes, so the failure must be the LATER
# host-side "the debs did not land" check -- proving the disposability gate is not blanket-red.
local $ENV{PATH} = "$fakebin:$ENV{PATH}";
local $ENV{FAKE_SCHROOT_CONFIG} =
"[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\nunion-type=overlay\n";
my $ok = eval { quiet { build_deb_in_chroot(@args) }; 1 };
ok(!$ok, 'a disposable chroot gets past the guard (and then fails for another reason)');
unlike($@, qr/is NOT disposable/, '... the failure is NOT the disposability guard');
like($@, qr/no \.deb is visible/, '... it is the host-side "debs did not land" check');
}
}
done_testing;
t/verify-repo.t
+209
View File
@@ -0,0 +1,209 @@
#!/usr/bin/perl
# Focused end-to-end tests for the PUBLISHED-REPO GATE, driving the real `sbuild-all.pl --verify-repo`
# against hand-built fixture apt trees. The pure decisions are unit-tested in t/sbuild-all.t; what is
# tested HERE is the wiring that the PR #63 review found to have false-PASS cases (concern #3):
#
# * the standalone gate must verify Release SIGNATURES BY DEFAULT -- it used to skip the check
# silently unless --gpg-home happened to be passed, so `--verify-repo <dir>` on an unsigned tree
# reported success;
# * an entirely MISSING secondary architecture must be reported, not read as "this run did not build
# that arch". The expected arch set is a claim (--expect-arch, else the repo's own Release
# "Architectures:" line), never an inference from which binary-<arch> dirs happen to be populated.
#
# The fixtures are plain text (Packages/Release), so no dpkg/apt/gpg tooling is required.
use strict;
use warnings;
use Test::More;
use FindBin qw($RealBin);
use File::Temp qw(tempdir);
use File::Path qw(make_path remove_tree);
my $SCRIPT = "$RealBin/../sbuild-all.pl";
plan skip_all => "sbuild-all.pl not found at $SCRIPT" unless -f $SCRIPT;
my $tmp = tempdir(CLEANUP => 1);
# An empty GNUPGHOME makes the signature outcome deterministic wherever this runs: --gpg-key-id can
# never resolve to a fingerprint here, so "signatures are checked" is observable as a hard failure
# rather than depending on the developer's keyring.
my $gpghome = "$tmp/gpg-home";
make_path($gpghome);
chmod 0700, $gpghome;
sub write_file {
my ($path, $body) = @_;
make_path($path =~ m{^(.*)/[^/]+$} ? $1 : '.');
open my $fh, '>', $path or die "write $path: $!";
print $fh $body;
close $fh;
}
sub stanza {
my ($pkg, $ver, $arch) = @_;
return "Package: $pkg\nVersion: $ver\nArchitecture: $arch\n"
. "Filename: pool/main/noble/${pkg}_${ver}_${arch}.deb\nDescription: fixture\n\n";
}
# The two Architecture:all packages ride into EVERY binary-<arch> index, exactly as apt-ftparchive
# emits them -- which is why a non-empty ppc index is not evidence that ppc64el was built.
sub arch_all_stanzas {
my ($genesis_arch) = @_;
return stanza('grub2-xcat', '2.12-1', 'all')
. stanza("xcat-genesis-base-$genesis_arch", '2.19.0-snap202608211200', 'all');
}
sub native_stanzas {
my ($a) = @_;
return stanza('ipmitool-xcat', "1.8.18-snap202608211200", $a)
. stanza('goconserver', "0.3.3-snap202608211200", $a);
}
# make_repo(%opt): a fixture apt tree for codename 'noble'.
# arches => the arches to write a binary-<arch>/Packages for (default both)
# release_arches => what Release advertises (default: the same list)
# native => arches that get NATIVE stanzas (default: all of `arches`)
my $repo_seq = 0;
sub make_repo {
my (%o) = @_;
my @arches = @{ $o{arches} // ['amd64', 'ppc64el'] };
my @rel = @{ $o{release_arches} // [@arches] };
my %native = map { $_ => 1 } @{ $o{native} // [@arches] };
my $dir = "$tmp/repo" . (++$repo_seq);
for my $a (@arches) {
my $body = ($native{$a} ? native_stanzas($a) : '') . arch_all_stanzas($a);
write_file("$dir/dists/noble/main/binary-$a/Packages", $body);
}
write_file("$dir/dists/noble/Release",
"Origin: xCAT\nLabel: xcat-dep\nSuite: noble\nCodename: noble\n"
. "Architectures: " . join(' ', @rel) . "\nComponents: main\n"
. "Date: Thu, 21 Aug 2026 12:00:00 +0000\n");
return $dir;
}
# The manifest is the source of truth for what each codename x arch must carry.
my $manifest = "$tmp/debs-manifest.conf";
write_file($manifest, <<'MAN');
[noble-amd64]
ipmitool-xcat=1.8.18
goconserver=0.3.3
grub2-xcat=*
xcat-genesis-base=2.*
[noble-ppc64el]
ipmitool-xcat=1.8.18
goconserver=0.3.3
grub2-xcat=*
xcat-genesis-base=2.*
MAN
# run_cmd(@argv) -> ($exit_code, $combined_output)
sub run_cmd {
my (@cmd) = @_;
my $line = join(' ', map { my $x = $_; $x =~ s/'/'"'"'/g; "'$x'" } @cmd) . ' 2>&1';
local $ENV{GNUPGHOME} = $gpghome;
open my $ph, '-|', $line or die "cannot run: $line: $!";
my $out = do { local $/; <$ph> };
close $ph;
return ($? >> 8, defined $out ? $out : '');
}
# run_gate($repo, @extra) -> ($exit_code, $output). --arch amd64 is always pinned so the result does
# not depend on the host running the test. Note that --gpg-home is deliberately NOT passed: the
# reviewer's report was that omitting it silently disabled the signature check, so the tests below
# exercise exactly that invocation. (run_cmd still points GNUPGHOME at an empty keyring, so the
# outcome does not depend on the developer's own keys.)
sub run_gate {
my ($repo, @extra) = @_;
return run_cmd($^X, $SCRIPT, '--verify-repo', $repo, '--manifest', $manifest,
'--dists', 'noble', '--arch', 'amd64', @extra);
}
# ---- a complete two-arch repo passes (the baseline: the gate is not simply always-red) -----------
{
my $repo = make_repo();
my ($rc, $out) = run_gate($repo, '--no-verify-signature');
is($rc, 0, 'complete two-arch repo passes the gate') or diag($out);
like($out, qr/\[verify-repo\] complete/, '... and says so');
like($out, qr/expected arches: amd64 ppc64el/,
'... having taken the expected arch set from the repo\'s own Release claim');
}
# ---- THE FALSE-PASS: an entirely missing secondary architecture -----------------------------------
# The repo advertises amd64 + ppc64el but ships no binary-ppc64el at all. Verified with --arch amd64
# and no --expect-arch -- the exact invocation that used to report success.
{
my $repo = make_repo(arches => ['amd64'], release_arches => ['amd64', 'ppc64el']);
my ($rc, $out) = run_gate($repo, '--no-verify-signature');
isnt($rc, 0, 'a repo missing an advertised architecture FAILS (was a silent pass)') or diag($out);
like($out, qr/MISSING-ARCH ppc64el|\[noble\/ppc64el\] MISSING-INDEX/,
'... naming the missing architecture');
}
# ... and the same tree with the expected set passed EXPLICITLY, which is what the CD pipeline does.
{
my $repo = make_repo(arches => ['amd64'], release_arches => ['amd64']);
my ($rc, $out) = run_gate($repo, '--no-verify-signature', '--expect-arch', 'amd64 ppc64el');
isnt($rc, 0, '--expect-arch demands the arch even when Release does not advertise it') or diag($out);
like($out, qr/MISSING-ARCH ppc64el|\[noble\/ppc64el\] MISSING-INDEX/, '... naming it');
}
# An index that exists but carries ONLY the Architecture:all packages is not a built architecture:
# grub2-xcat and the genesis debs land in every binary-<arch>/Packages.
{
my $repo = make_repo(arches => ['amd64', 'ppc64el'], native => ['amd64']);
my ($rc, $out) = run_gate($repo, '--no-verify-signature');
isnt($rc, 0, 'an arch:all-only ppc index does not count as a built ppc64el') or diag($out);
like($out, qr/MISSING-ARCH ppc64el/, '... reported as MISSING-ARCH');
like($out, qr/MISSING ipmitool-xcat|MISSING goconserver/,
'... and its native packages are reported missing');
}
# A genuinely single-arch repo (BUILD_PPC=false) that advertises only amd64 must NOT false-fail.
{
my $repo = make_repo(arches => ['amd64'], release_arches => ['amd64']);
my ($rc, $out) = run_gate($repo, '--no-verify-signature');
is($rc, 0, 'an honest single-arch repo passes (no false MISSING-ARCH)') or diag($out);
}
# A stale architecture left in a tree that is no longer expected is also a problem.
{
my $repo = make_repo();
my ($rc, $out) = run_gate($repo, '--no-verify-signature', '--expect-arch', 'amd64');
isnt($rc, 0, 'natives for an arch outside --expect-arch FAIL (stale architecture)') or diag($out);
like($out, qr/UNEXPECTED-ARCH ppc64el/, '... reported as UNEXPECTED-ARCH');
}
# ---- THE OTHER FALSE-PASS: standalone verification skipped signatures ----------------------------
# Same complete tree, without --no-verify-signature AND without --gpg-home -- the invocation that used
# to report success because the signature check was silently skipped. The fixture has no
# InRelease/Release.gpg, so a gate that actually checks signatures must fail.
{
my $repo = make_repo();
my ($rc, $out) = run_gate($repo);
isnt($rc, 0, 'an UNSIGNED repo fails standalone verification by default') or diag($out);
like($out, qr/signature check: on/, '... the signature check is on by default');
like($out, qr/UNSIGNED noble|SIGKEY/, '... and the problem is reported');
}
# ... and the opt-out is explicit, not accidental.
{
my $repo = make_repo();
my ($rc, $out) = run_gate($repo, '--no-verify-signature');
is($rc, 0, '--no-verify-signature is the deliberate opt-out') or diag($out);
like($out, qr/signature check: OFF \(--no-verify-signature\)/,
'... and the log names the flag that turned it off');
}
# ---- a cell with no manifest section is a configuration error, not a free pass --------------------
{
my $thin = "$tmp/thin-manifest.conf";
write_file($thin, "[noble-amd64]\nipmitool-xcat=1.8.18\ngoconserver=0.3.3\n"
. "grub2-xcat=*\nxcat-genesis-base=2.*\n");
my $repo = make_repo();
my ($rc, $out) = run_cmd($^X, $SCRIPT, '--verify-repo', $repo, '--manifest', $thin,
'--dists', 'noble', '--arch', 'amd64', '--no-verify-signature');
isnt($rc, 0, 'an expected cell with no manifest section FAILS (was silently skipped)') or diag($out);
like($out, qr/NO-MANIFEST section \[noble-ppc64el\]/, '... naming the missing section');
}
done_testing;