diff --git a/BUILD.md b/BUILD.md index 0f205bc..a9c866f 100644 --- a/BUILD.md +++ b/BUILD.md @@ -282,6 +282,23 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of - **Fresh staging + promote-on-success.** Everything is built + validated into a per-run staging tree first; the published apt repo is (re)assembled from staging ONLY after the complete expected set validates — a partial/failed build never reaches the repo and stale debs never accumulate. +- **Build runs stage; publishing is a separate, locked, atomic step.** The two arches build + *concurrently* on their two hosts against the same `--apt-dir`, so an arch build run **never + publishes**: it fills staging and stops. Publishing happens with **`--publish`** (implied by + `--skip-build`, i.e. the finalization run). That step takes **one global publish lock** — not the + per-arch build lock — assembles the whole tree into a **side directory**, runs the repo gate against + *that* tree, and only then swaps it onto `--apt-dir` with a single `rename(2)`. Readers therefore + see either the previous complete repo or the new complete repo, never a half-wiped `pool/` or an + index that disagrees with its `Release`; a failed gate leaves the published tree untouched. + Codenames outside `--dists` survive the swap. `--skip-createrepo` forces "do not publish". +- **Clean, disposable build environment per package.** Every package builds in its own `schroot` + session, and the chroot must hand out a **throwaway** session (`union-type=overlay`, or a + snapshot/tarball chroot). `sbuild-all.pl` repairs a chroot that lacks one and hard-fails if it still + is not disposable. That is what makes the fail-hard dependency handling mean something: inside the + session `apt-get update`, the common build tooling and the package's `Build-Depends` (resolved with + `mk-build-deps`, so version constraints, `a | b` alternatives and arch qualifiers are honoured) are + all **fatal** on failure — and since nothing survives the session, a package whose `debian/control` + forgets a `Build-Depends` cannot build green on a sibling package's leftovers. - **Per-arch package sets (`debs-manifest.conf`).** One `[-]` section per target. The noarch boot components (`syslinux-xcat`/`grub2-xcat`/`elilo-xcat`/`xnba-undi`, `Architecture:all`) are built ONCE on amd64 — single producer, their source is x86-only — and assembled into every @@ -307,7 +324,9 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of - **`/sbuild.pl`** ×7 — per-package builders (mirror `/mockbuild.pl`); each drives its maintained `debian/` in the chroot and collects the `.deb`(s). Invoked by `sbuild-all.pl`. - **`debs-manifest.conf`** — per `[-]` required set + version pins. -- **`t/sbuild-all.t`** — fixture tests (`prove t/sbuild-all.t`). +- **`t/sbuild-all.t`** — unit tests for the pure helpers (`prove t/`). +- **`t/verify-repo.t`** — end-to-end tests of the repo gate against fixture apt trees (missing + secondary arch, arch:all-only index, unsigned repo, missing manifest section). ## Usage (per arch, as root on the matching build host) @@ -316,43 +335,58 @@ ppc64el on the ppc Ubuntu host). The Ubuntu version(s) to build are selected wit space/comma list of codenames) or, for exactly one, **`--target -`**. Version ↔ codename: `20.04`=`focal`, `22.04`=`jammy`, `24.04`=`noble`, `26.04`=`resolute`. -### Build ALL supported Ubuntu versions +The full flow is **two steps**: each arch builds into staging on its own host, then **one** +finalization step publishes the assembled repo atomically. + +### Step 1 — build each arch into staging (no publishing) ```bash -# amd64 host — build focal+jammy+noble+resolute, sign, assemble the apt tree: +# amd64 host — build focal+jammy+noble+resolute into staging: ./sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \ --xcat-source ../xcat-core --genesis-rpm \ - --genesis-rpm-ppc \ + --genesis-rpm-ppc + +# ppc64el host — arch-specific deps only (the Architecture:all boot components and both genesis +# debs come from the amd64 build): +./sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" --skip-genesis +``` + +These runs touch **only** `staging///`; the apt tree at `--apt-dir` is left alone, so +the two hosts can run at the same time. `--dists` may be omitted entirely — with no +`--dists`/`--target`, **all supported codenames** are built (`focal jammy noble resolute`). + +### Step 2 — publish once, after every arch has staged + +```bash +./sbuild-all.pl --skip-build --skip-genesis \ + --publish --expect-arch "amd64 ppc64el" \ --gpg-sign --gpg-key-id xcat@example.com --gpg-home ``` -`--dists` may be omitted entirely — with no `--dists`/`--target`, **all supported codenames** are -built (the default is `focal jammy noble resolute`). +This takes the global publish lock, assembles + signs both arches' staging into a side tree, gates it, +and swaps it onto `--apt-dir` atomically. `--expect-arch` states which architectures the published +repo must serve — omit it and the staged arch set is used instead. (`--publish` is implied here +because the run builds nothing; state it explicitly if you want to build **and** publish in one go.) ### Build ONE specific Ubuntu version ```bash # just 24.04 (noble) on amd64 — two equivalent forms: -./sbuild-all.pl --arch amd64 --dists noble --xcat-source ../xcat-core --genesis-rpm --gpg-sign ... -./sbuild-all.pl --target noble-amd64 --xcat-source ../xcat-core --genesis-rpm --gpg-sign ... +./sbuild-all.pl --arch amd64 --dists noble --xcat-source ../xcat-core --genesis-rpm +./sbuild-all.pl --target noble-amd64 --xcat-source ../xcat-core --genesis-rpm # just 20.04 (focal): ./sbuild-all.pl --arch amd64 --dists focal ... ``` -### ppc64el host - -```bash -# arch-specific deps only (the Architecture:all boot components come from the amd64 build): -./sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" \ - --xcat-source ../xcat-core --genesis-rpm --gpg-sign ... -``` - ### Handy variants ```bash -./sbuild-all.pl --dry-run --arch amd64 --dists noble # print the plan, do nothing -./sbuild-all.pl --skip-build --skip-genesis --gpg-sign ... # assemble-only (re-index/re-sign staging) +./sbuild-all.pl --dry-run --arch amd64 --dists noble # print the plan, do nothing +./sbuild-all.pl --skip-build --skip-genesis --gpg-sign ... # publish-only (re-index/re-sign staging) +# single host: build AND publish in one go +./sbuild-all.pl --arch amd64 --dists noble --genesis-rpm \ + --publish --expect-arch amd64 --gpg-sign --gpg-key-id --gpg-home ``` `sbuild-all.pl --help` lists every option and `sbuild-all.pl --man` (or `perldoc sbuild-all.pl`) @@ -362,24 +396,39 @@ prints the full manual; the shared flags (`--repo-root`, `--manifest`, # Repository verification gate -After the apt repo is assembled + signed, `sbuild-all.pl` runs a **manifest-driven gate** that fails -the build if the published repo is incomplete or mis-signed. It uses `debs-manifest.conf` as the -single source of truth and is layered so the decision logic is pure and unit-tested -(`BuildUtils::verify_repo_packages` / `verify_repo_signature`; `parse_packages_index` / -`resolve_present_names` for parsing/resolution), separate from the disk/gpg I/O. +Before an assembled repo is published, `sbuild-all.pl` runs a **manifest-driven gate** that fails the +build if the repo is incomplete, serves the wrong architectures, or is mis-signed. It uses +`debs-manifest.conf` as the single source of truth and is layered so the decision logic is pure and +unit-tested (`BuildUtils::verify_repo_arches` / `verify_repo_packages` / `verify_repo_signature`; +`parse_packages_index` / `parse_release_architectures` / `resolve_present_names` for +parsing/resolution), separate from the disk/gpg I/O. -- **Runs automatically** at the end of `assemble_apt`, **per codename × arch**. Suppress with - `--no-verify-repo`; skipped under `--dry-run`. Verify an assembled tree out of band with - `--verify-repo=` (manifest/dists/key from `--manifest`/`--dists`/`--gpg-key-id`/`--gpg-home`). +- **Runs automatically** inside `--publish`, against the **side tree, before the swap**, per codename × + expected arch — so a repo that fails the gate is never published. Suppress with `--no-verify-repo`; + skipped under `--dry-run`. Verify an already-published tree out of band with + `--verify-repo=` (manifest/dists/arches/key from + `--manifest`/`--dists`/`--expect-arch`/`--gpg-key-id`/`--gpg-home`). +- **Architectures:** the expected set is always a **claim**, never an inference from what happens to + be present — `--expect-arch` if given, else the staged arch set when publishing, else each + codename's own `Release` `Architectures:` line when verifying standalone. An expected arch with no + *native* package is `MISSING-ARCH`; natives for an arch outside the expected set are + `UNEXPECTED-ARCH` (a stale architecture). This is what makes an **entirely missing secondary + architecture** a failure instead of reading as "this run did not build it". Note that a non-empty + `binary-/Packages` is *not* evidence the arch was built: the `Architecture:all` packages + (`grub2-xcat`, the genesis debs) ride into every arch's index, so `index_has_native_arch` is what + counts. - **Completeness:** for each cell, every package that codename×arch's manifest section requires (after `required_pkgs` skip-filtering) must appear in the published `binary-/Packages` with a version satisfying its pin. The arch-suffixed genesis (`xcat-genesis-base`) is resolved to **this cell's arch** (`xcat-genesis-base-`) — never a different arch, so a missing native genesis is caught. + An expected cell with **no manifest section** is a hard error (`NO-MANIFEST`), not a free pass. - **Signature:** each `dists//InRelease` (or detached `Release`+`Release.gpg`) must be a *good* signature whose **primary-key fingerprint equals the fingerprint of `--gpg-key-id`** — the repo was signed by exactly the CLI key. Expired/revoked keys and expired signatures are rejected; if the CLI - key does not resolve to a fingerprint the gate fails (`SIGKEY`), never passes. A signature is only - *required* when `--gpg-sign` was used (an intentionally-unsigned repo does not false-fail). + key does not resolve to a fingerprint the gate fails (`SIGKEY`), never passes. In the automatic + pre-swap run a signature is required iff `--gpg-sign` was used (an intentionally-unsigned repo does + not false-fail); in the **standalone `--verify-repo` mode signatures are checked by default**, and + `--no-verify-signature` is the explicit opt-out. **Semantic idiosyncrasies (intentional, and mirrored in the EL `mockbuild-all.pl` gate):** diff --git a/BuildUtils.pm b/BuildUtils.pm index 3fbd2f7..964cb2f 100644 --- a/BuildUtils.pm +++ b/BuildUtils.pm @@ -24,10 +24,11 @@ use MIME::Base64 qw(encode_base64); our @EXPORT_OK = qw( sh_quote print_step version_matches required_pkgs read_manifest standard_options - verify_repo_packages verify_repo_signature parse_packages_index resolve_present_names + verify_repo_packages verify_repo_signature verify_repo_arches + parse_packages_index parse_release_architectures resolve_present_names index_has_native_arch control_binary_arch skip_arch_all_on codename_to_version version_to_codename known_codenames - chroot_name chroot_sources_list + chroot_name chroot_sources_list chroot_is_disposable chroot_build_script control_field genesis_deb_control deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb build_deb_in_chroot @@ -197,6 +198,58 @@ sub verify_repo_signature { return @problems; } +# verify_repo_arches(\@expected, \%native) -> @problems +# @expected : the architecture set the published repo is REQUIRED to serve. It comes from an +# EXPLICIT claim -- --expect-arch, the staged arch set that was just promoted, or the +# repo's own Release "Architectures:" line -- NEVER from "whichever binary- +# directories happen to exist". That is the whole point: if the expected set were +# inferred from presence, an entirely missing secondary architecture would read as +# "this run did not build it" instead of "the repository is incomplete" (the false-PASS +# the PR #63 review caught). +# %native : arch => boolean, "the published binary-/Packages carries at least one stanza +# built FOR that arch" (index_has_native_arch) -- collected by the IO layer for every +# arch it looked at, expected or not. +# Returns human-readable problem strings (empty list = the arch set is exactly right): +# "MISSING-ARCH (expected, but the published index carries no native package)" +# "UNEXPECTED-ARCH (native packages published, but is not in the expected set )" +# The UNEXPECTED direction matters too: it catches a stale architecture left behind in a tree that is +# no longer built for it. Pure: no I/O, deterministic (problems in sorted arch order). +sub verify_repo_arches { + my ($expected, $native) = @_; + my %want = map { $_ => 1 } @{ $expected || [] }; + my @problems; + for my $a (sort keys %want) { + push @problems, "MISSING-ARCH $a (expected, but the published index carries no native $a package)" + unless $native->{$a}; + } + for my $a (sort keys %$native) { + next unless $native->{$a}; + next if $want{$a}; + push @problems, "UNEXPECTED-ARCH $a (native $a packages published, but $a is not in the " + . "expected set [" . join(' ', sort keys %want) . "])"; + } + return @problems; +} + +# parse_release_architectures($release_text) -> @arches +# The architecture set an apt Release file CLAIMS to serve (its "Architectures:" line), in file order +# and de-duplicated; an empty list when the field is absent. This is the repository's own published +# claim, so it is the right FALLBACK expected-arch set for a standalone verification of a tree whose +# build-time --expect-arch is not known: verifying a repo against what it advertises to apt clients +# catches "Release says amd64 ppc64el but binary-ppc64el is empty/missing". Pure: text in, list out. +sub parse_release_architectures { + my ($text) = @_; + return () unless defined $text; + my ($line) = $text =~ /^Architectures:[ \t]*(.*?)[ \t]*$/m; + return () unless defined $line; + my (@a, %seen); + for my $x (split /\s+/, $line) { + next unless length $x; + push @a, $x unless $seen{$x}++; + } + return @a; +} + # parse_packages_index($text) -> \%{ package_name => version } # Parse a Debian 'Packages' index: RFC822 stanzas separated by blank line(s); each carries a # 'Package:' and a 'Version:'. Returns name => version (the FULL Debian version verbatim, epoch + @@ -315,6 +368,35 @@ sub chroot_sources_list { ); } +# chroot_is_disposable($schroot_config_text): true iff a `schroot -c -- ...` session against +# this chroot gets a THROWAWAY filesystem -- i.e. everything the build installs or writes is discarded +# when the session ends, so the NEXT package starts from the pristine base. +# +# This is the load-bearing precondition of the per-package build (PR #63 review concern #2): the +# per-codename chroots are long-lived and shared by all seven packages, so without a disposable +# session, package N's build-dependencies stay installed for package N+1 -- and a package whose +# debian/control forgets a Build-Depends builds anyway, silently, because a sibling happened to pull +# the dependency in. Making dependency installation fatal is only half the fix; it means nothing if a +# stale environment can satisfy an undeclared dependency in the first place. +# +# Disposable configurations, per schroot(1): +# * union-type = overlay | overlayfs | aufs | unionfs -- a per-session union mount over the base +# directory; writes go to the (discarded) overlay. This is what sbuild-createchroot sets up. +# * type = file -- the session unpacks a fresh tarball. +# * type = {btrfs,lvm,zfs}-snapshot -- the session gets its own snapshot. +# A plain `type=directory` with `union-type=none` is NOT disposable: the session bind-mounts the base +# directory read-write and every build mutates it permanently. +# Pure: `schroot --config -c ` text in, boolean out. +sub chroot_is_disposable { + my ($config_text) = @_; + return 0 unless defined $config_text && $config_text ne ''; + my ($union) = $config_text =~ /^union-type=[ \t]*(\S+)/m; + return 1 if defined $union && $union =~ /^(?:overlay|overlayfs|aufs|unionfs)$/; + my ($type) = $config_text =~ /^type=[ \t]*(\S+)/m; + return 1 if defined $type && $type =~ /^(?:file|btrfs-snapshot|lvm-snapshot|zfs-snapshot)$/; + return 0; +} + # --------------------------------------------------------------------------------------------------- # Debian control-metadata helpers (concern #2: preserve the maintained packaging's semantics). # --------------------------------------------------------------------------------------------------- @@ -488,15 +570,84 @@ sub cross_copy_genesis_deb { # MockBuildUtils' helpers do for the per-package mockbuild.pl builders). # --------------------------------------------------------------------------------------------------- +# chroot_build_script: the bash program that runs INSIDE the per-package schroot session. Returned as +# text (pure) so t/sbuild-all.t can assert its fail-hard properties without a chroot. +# +# Fail-hard contract (PR #63 review concern #2) -- every step below is FATAL, none is best-effort: +# * `set -euo pipefail`: any unchecked command failure aborts the build. +# * apt-get update / the common build tooling install are retried (transient mirror hiccups) and +# then FATAL. They used to end in `|| true`, which let a package build with, say, no `quilt` and +# produce a silently-wrong .deb. +# * Build-Depends are installed with `mk-build-deps` (devscripts + equivs) instead of a sed +# extraction, and a failure is FATAL. mk-build-deps feeds the control file's relationships to apt +# verbatim, so version constraints `(>= 12)`, alternatives `a | b` and arch qualifiers `[!ppc64el]` +# are honoured -- the sed pipeline dropped all three, and, paired with the swallowed error, a +# too-old or missing build dependency produced a green build. +# The build environment is disposable: the caller asserts the chroot gives each session a throwaway +# overlay/snapshot (chroot_is_disposable), so nothing this script installs can leak into the next +# package's build and satisfy an undeclared dependency. +sub chroot_build_script { + return <<'INNER'; +set -euo pipefail +PKGSRC="$1"; OUT="$2"; SDE="$3"; EXTRA="$4"; B64="$5" +export DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS=nocheck SOURCE_DATE_EPOCH="$SDE" + +# apt_retry: run apt-get, retrying a few times for a transient mirror/network hiccup, then FATAL. +apt_retry() { + local i + for i in 1 2 3; do + if apt-get "$@"; then return 0; fi + echo "[warn] 'apt-get $*' failed (attempt $i/3); retrying in 5s" >&2 + sleep 5 + done + echo "FATAL: 'apt-get $*' failed after 3 attempts" >&2 + return 1 +} + +apt_retry update -q +# Common build tooling. FATAL: a missing tool silently changes what gets built. +# shellcheck disable=SC2086 # $EXTRA is a deliberate word-split package list +apt_retry install -y --no-install-recommends \ + git wget curl ca-certificates devscripts equivs quilt fakeroot build-essential $EXTRA + +W=$(mktemp -d) +cp -a "$PKGSRC" "$W/pkg" +cd "$W/pkg" + +# Declared Build-Depends, resolved by mk-build-deps: it hands debian/control's relationships to apt +# verbatim, so versions/alternatives/arch-qualifiers are honoured. FATAL on failure -- an unsatisfied +# build dependency must stop the build, never be papered over by whatever the chroot already carries. +if [ -f debian/control ]; then + echo "== installing Build-Depends from debian/control (mk-build-deps) ==" + mk-build-deps --install --remove \ + --tool 'apt-get -y --no-install-recommends' debian/control +fi + +printf '%s' "$B64" | base64 -d > "$W/pkgbuild.sh" +( cd "$W/pkg" && bash "$W/pkgbuild.sh" ) + +# Collect the built binaries. The mk-build-deps dummy package (-build-deps_*.deb) and debug +# symbols are not build output and must never reach the repo. +mapfile -t found < <(find "$W" -maxdepth 3 -name '*.deb' \ + ! -name '*-dbgsym_*' ! -name '*-build-deps_*' -print | sort) +if [ "${#found[@]}" -eq 0 ]; then + echo "FATAL: the package build produced no .deb" >&2 + exit 1 +fi +mkdir -p "$OUT" +for d in "${found[@]}"; do cp -v "$d" "$OUT/"; done +INNER +} + # build_deb_in_chroot: build ONE package inside its --sbuild chroot and collect the -# produced .deb(s) into $result_dir. The COMMON orchestration lives here -- an ephemeral schroot +# produced .deb(s) into $result_dir. The COMMON orchestration lives here -- a disposable schroot # session, apt update, install of common tools + the package's debian/control Build-Depends, an # OUT-OF-TREE copy of the package dir (the checkout is never mutated), SOURCE_DATE_EPOCH for # reproducible builds, deb collection, and a host-side check that the debs actually landed (a # chroot-local --result-dir would otherwise be a silent no-output). Each /sbuild.pl supplies # only its package-specific $build snippet (the source prep + dpkg-buildpackage that used to live in # make_deb.sh); $build runs with CWD = the copied package dir and must leave its .deb(s) somewhere -# under the build work tree. Dies on any failure. +# under the build work tree. Dies on any failure -- including a chroot that is not disposable. # %args: pkg, chroot, pkg_dir, result_dir, build_timestamp, build (required); extra_tools (arrayref, optional) sub build_deb_in_chroot { my (%a) = @_; @@ -505,33 +656,23 @@ sub build_deb_in_chroot { my $pkg = $a{pkg}; die "FATAL: chroot $a{chroot} missing (run sbuild-all.pl to auto-init it)\n" if system("schroot -l 2>/dev/null | grep -qx chroot:$a{chroot}") != 0; + # HARD precondition: each package must build in a CLEAN, throwaway environment. A shared + # `type=directory` chroot with no union mount would carry the previous package's build-deps into + # this one, so an undeclared Build-Depends would build green here and fail for everyone else. + my $cfg = `schroot --config -c ${\ sh_quote($a{chroot}) } 2>/dev/null` // ''; + die "FATAL: chroot $a{chroot} is NOT disposable -- a schroot session against it would mutate the\n" + . " shared base filesystem, so one package's build-dependencies would leak into the next and\n" + . " hide a missing Build-Depends. Add 'union-type=overlay' to its /etc/schroot/chroot.d/ entry\n" + . " (or delete the chroot and let sbuild-all.pl re-create it).\n" + unless chroot_is_disposable($cfg); make_path($a{result_dir}); my $extra = join(' ', @{ $a{extra_tools} || [] }); my $b64 = encode_base64($a{build}, ''); - # The whole per-package build runs in ONE schroot session (ephemeral overlay). schroot SANITIZES - # the environment, so values are passed as POSITIONAL ARGS to the inner bash; the package-specific + # The whole per-package build runs in ONE disposable schroot session. schroot SANITIZES the + # environment, so values are passed as POSITIONAL ARGS to the inner bash; the package-specific # build is passed base64-encoded to avoid any quoting interplay through schroot. - my $inner = <<'INNER'; -set -uo pipefail -PKGSRC="$1"; OUT="$2"; SDE="$3"; EXTRA="$4"; B64="$5" -export DEBIAN_FRONTEND=noninteractive DEB_BUILD_OPTIONS=nocheck SOURCE_DATE_EPOCH="$SDE" -for t in 1 2 3; do apt-get update -q && break; sleep 5; done -apt-get install -y --no-install-recommends \ - git wget curl ca-certificates devscripts quilt fakeroot build-essential $EXTRA >/dev/null 2>&1 || true -W=$(mktemp -d); cp -a "$PKGSRC" "$W/pkg"; cd "$W/pkg" -if [ -f debian/control ]; then - BD=$(sed -n '/^Build-Depends:/,/^\S/p' debian/control | tr ',' '\n' \ - | sed -E 's/^Build-Depends://; s/\(.*\)//; s/\[.*\]//; s/[[:space:]]//g' \ - | grep -E '^[a-z0-9]' | grep -v '^debhelper-compat' | sort -u | tr '\n' ' ') - [ -n "$BD" ] && { apt-get install -y $BD >/dev/null 2>&1 || echo "[warn] some build-deps failed to install"; } -fi -printf '%s' "$B64" | base64 -d > "$W/pkgbuild.sh" -( cd "$W/pkg" && bash "$W/pkgbuild.sh" ) || { echo "package build FAILED"; exit 1; } -found=$(find "$W" -maxdepth 3 -name '*.deb' ! -name '*-dbgsym_*' -print) -[ -n "$found" ] || { echo "build produced no .deb"; exit 1; } -mkdir -p "$OUT"; echo "$found" | while read -r d; do cp -v "$d" "$OUT/"; done -INNER + my $inner = chroot_build_script(); my $cmd = 'schroot -c ' . sh_quote($a{chroot}) . ' -u root -d / -- bash -c ' . sh_quote($inner) . ' bash ' diff --git a/sbuild-all.pl b/sbuild-all.pl index 97368b5..41efb3c 100755 --- a/sbuild-all.pl +++ b/sbuild-all.pl @@ -11,9 +11,12 @@ # Design (mirrors mockbuild-all.pl + fixes the PR #63 review): # 1. One host = one arch (dpkg --print-architecture / --arch); a set of codenames (--dists). # Each (codename,arch) is a TARGET named "-" with a section in debs-manifest.conf. -# 2. Everything is built + validated into a FRESH per-run STAGING tree first; the published apt repo -# is (re)assembled from staging ONLY after the complete expected set validates -- so a partial or -# failed build never reaches the published repo, and stale debs never accumulate (concern #1). +# 2. Everything is built + validated into a FRESH per-run STAGING tree first. An architecture build +# run STOPS THERE: publishing is a separate --publish step that takes ONE GLOBAL lock, assembles +# into a side tree, gates it, and swaps it onto the published path with a single rename(2). The +# two arches build concurrently on their two hosts, so a per-arch build that also published would +# interleave wipes of the same pool/dists/Release; and a partial or failed build must never reach +# the published repo, nor stale debs accumulate in it (concern #1). # 3. Per-arch package sets come from the manifest: the x86 boot components (syslinux/elilo/xnba, # Architecture:all) are built once on amd64 (single producer); ppc64el builds only the genuinely # arch-specific compiled deps (concern #3). @@ -37,9 +40,11 @@ use Fcntl qw(:flock); use FindBin qw($RealBin); use lib $RealBin; use BuildUtils qw(sh_quote print_step version_matches required_pkgs read_manifest standard_options - verify_repo_packages verify_repo_signature parse_packages_index resolve_present_names + verify_repo_packages verify_repo_signature verify_repo_arches + parse_packages_index parse_release_architectures resolve_present_names index_has_native_arch control_binary_arch skip_arch_all_on codename_to_version known_codenames chroot_name chroot_sources_list + chroot_is_disposable control_field genesis_deb_control deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb); @@ -64,12 +69,24 @@ my $parallel_targets = 0; my ($skip_build, $skip_install, $skip_genesis, $skip_xcat_dep) = (0,0,0,0); my ($skip_createrepo, $skip_tarball) = (0,0); my $dry_run = 0; +# --publish: run the FINALIZATION phase (assemble + sign + gate + tarball). See the "Publishing" +# comment block below: an architecture build run stages only; publishing is a separate, singly-locked, +# atomic step. undef = not specified -> defaulted from --skip-build. +my $publish; +# --expect-arch: the architecture set the PUBLISHED repo must serve, stated EXPLICITLY (repeatable, +# and each value may be a space/comma list). This is what stops "an entirely missing secondary +# architecture" from reading as "this run simply did not build it" -- see resolve_expect_arches(). +my @expect_arch; # Completeness+signature gate on the PUBLISHED apt index (what apt clients see). $verify_repo_arg set # (--verify-repo=) runs the gate STANDALONE against that assembled apt dir and exits (no lock, -# no build). $no_verify_repo suppresses the AUTOMATIC post-assembly gate that otherwise runs at the end -# of assemble_apt. Default: automatic gate ON. +# no build). $no_verify_repo suppresses the AUTOMATIC post-publish gate that otherwise runs before the +# assembled tree is swapped into place. Default: automatic gate ON. my $verify_repo_arg = ''; my $no_verify_repo = 0; +# Signature verification is ON by default for the standalone gate (--verify-repo): an unsigned or +# wrongly-signed repo is a real defect, and silently skipping the check because no --gpg-home happened +# to be passed is a false PASS. --no-verify-signature is the explicit, deliberate opt-out. +my $no_verify_signature = 0; my $gpg_sign = 0; my $gpg_key_id = 'xcat@megware.com'; my $gpg_home = ''; @@ -142,8 +159,11 @@ $spec{'genesis-deb=s'} = \@genesis_debs; $spec{'genesis-rpm=s'} = \$genesis_rpm; $spec{'genesis-rpm-ppc=s'} = \$genesis_rpm_ppc; $spec{'require-ppc-genesis!'} = \$require_ppc_genesis; +$spec{'publish!'} = \$publish; # run the finalization (assemble+sign+gate+tarball) +$spec{'expect-arch=s'} = \@expect_arch; # repeatable; each value may be a space/comma list $spec{'verify-repo=s'} = \$verify_repo_arg; # standalone gate: --verify-repo= -$spec{'no-verify-repo!'} = \$no_verify_repo; # suppress the automatic post-assembly gate +$spec{'no-verify-repo!'} = \$no_verify_repo; # suppress the automatic pre-swap gate +$spec{'no-verify-signature!'} = \$no_verify_signature; # explicit opt-out of the signature check $spec{'output=s'} = \$output_root; # --output alias $spec{'help|h'} = sub { pod2usage(-verbose => 1, -exitval => 0); }; $spec{'man'} = sub { pod2usage(-verbose => 2, -exitval => 0); }; @@ -179,6 +199,29 @@ for my $cn (@dist_list) { die "FATAL: unknown codename '$cn' (known: @{[known_codenames()]})\n" unless codename_to_version($cn); } +@expect_arch = grep { length } map { split /[\s,]+/ } @expect_arch; +for my $a (@expect_arch) { + die "FATAL: unsupported --expect-arch '$a' (amd64|ppc64el)\n" unless $a =~ /^(amd64|ppc64el)$/; +} + +# --------------------------------------------------------------------------------------------------- +# Publishing: an ARCH BUILD RUN STAGES ONLY; publishing is a SEPARATE, SINGLY-LOCKED, ATOMIC step. +# +# The two arches build CONCURRENTLY on their two hosts against the same --apt-dir on the shared tree. +# If both were allowed to assemble, they would wipe and repopulate the same pool/, dists/, Release, +# InRelease and tarball at the same time and interleave into a corrupt (but green) repository. The +# per-arch run lock does NOT prevent that -- it is per-arch by design, so the two arches can build in +# parallel (PR #63 review concern #1). +# +# So the phases are split by ROLE: +# * a run that BUILDS (no --skip-build) produces staging artifacts and stops there; +# * publishing happens only when asked for with --publish -- or implicitly on a run that builds +# nothing (--skip-build), which IS the finalization step -- and it takes ONE GLOBAL publish lock +# and swaps a fully-assembled, fully-verified tree into place with a single rename(). +# --skip-createrepo (the shared CLI's "do not build a repo") always wins and suppresses publishing. +$publish = ($skip_build ? 1 : 0) unless defined $publish; +$publish = 0 if $skip_createrepo; + $output_root ||= "$repo_root/build-output/sbuild-all"; $apt_dir ||= "$repo_root/repos/apt"; $run_id ||= strftime("%Y%m%d-%H%M%S", localtime()); @@ -190,13 +233,24 @@ my %MANIFEST = read_manifest($manifest); # Standalone gate: --verify-repo= checks an already-assembled apt tree (completeness + # Release signatures) using THIS script's manifest resolution (--manifest or the default -# debs-manifest.conf), --dists, and --gpg-key-id/--gpg-home, then exits. It takes NO run lock and does -# NOT build. Dispatched here (after manifest + @dist_list are resolved) so it never trips the -# build-only per-target section check below and never reaches the lock/build phases. +# debs-manifest.conf), --dists, --expect-arch, and --gpg-key-id/--gpg-home, then exits. It takes NO run +# lock and does NOT build. Dispatched here (after manifest + @dist_list are resolved) so it never trips +# the build-only per-target section check below and never reaches the lock/build phases. +# +# Two deliberate STRICTNESS choices here (PR #63 review concern #3), both of which used to be +# false-PASSes: +# * The signature IS verified by default. It used to be skipped unless --gpg-home happened to be +# given, so the common `--verify-repo ` invocation silently checked completeness only. +# --no-verify-signature is the explicit opt-out. +# * The expected architecture set is a CLAIM, never "whatever is present": --expect-arch if given, +# else the arch set each codename's own Release advertises to apt clients. An entirely missing +# binary- therefore reads as MISSING-ARCH, not as "that arch was not built this run". if (length $verify_repo_arg) { die "FATAL: --verify-repo apt dir not found: $verify_repo_arg\n" unless -d $verify_repo_arg; print_step('Standalone repo verification (no build, no lock)'); - verify_assembled_repo(\%MANIFEST, abs_path($verify_repo_arg), \@dist_list, [$arch]); + my $adir = abs_path($verify_repo_arg); + verify_assembled_repo(\%MANIFEST, $adir, \@dist_list, + resolve_expect_arches('standalone', $adir), !$no_verify_signature); exit 0; } @@ -238,6 +292,11 @@ print " apt-dir: $apt_dir\n"; print " staging: $staging\n"; print " run-id: $run_id snap-ts: $snap_ts build-number: " . (defined $build_number ? $build_number : '(none)') . "\n"; print " gpg-sign: " . ($gpg_sign ? "yes (key $gpg_key_id)" : "no") . "\n"; +print " publish: " . ($publish + ? "yes (assemble + sign + gate + swap into $apt_dir, under the global publish lock)" + : "no (STAGING ONLY -- re-run with --publish, or run the separate finalization step)") . "\n"; +print " expect-arch: " . (@expect_arch ? "@expect_arch" : '(derive from the staged arch set)') . "\n" + if $publish; print " dry-run: " . ($dry_run ? "yes" : "no") . "\n"; # --------------------------------------------------------------------------------------------------- @@ -294,6 +353,35 @@ sub have_chroot { my $out = `schroot -l 2>/dev/null`; return $out =~ /^chroot:\Q$name\E$/m ? 1 : 0; } + +# ensure_disposable_chroot($name): make sure a `schroot -c -- ...` session gets a THROWAWAY +# filesystem, so each package builds in a clean environment and one package's build-dependencies can +# never leak into the next (PR #63 review concern #2 -- see BuildUtils::chroot_is_disposable). +# sbuild-createchroot normally writes union-type=overlay; a chroot created before that (or by hand) +# may not have it. Repair the chroot.d entry in place, then re-check and hard-fail if it still is not +# disposable -- silently building in a shared, mutable chroot is exactly what must not happen. +sub ensure_disposable_chroot { + my ($name) = @_; + return 1 if $dry_run; + my $cfg = `schroot --config -c ${\ sh_quote($name) } 2>/dev/null` // ''; + return 1 if chroot_is_disposable($cfg); + my ($file) = grep { -f $_ && do { local $/; open my $fh, '<', $_ or 0; + my $t = <$fh>; close $fh; $t =~ /^\[\Q$name\E\]/m } } + glob('/etc/schroot/chroot.d/*'); + if ($file) { + print " chroot $name: NOT disposable -> adding union-type=overlay to $file\n"; + run("sed -i " . sh_quote("/^union-type=/d") . " " . sh_quote($file)); + run("printf '%s\\n' 'union-type=overlay' >> " . sh_quote($file)); + $cfg = `schroot --config -c ${\ sh_quote($name) } 2>/dev/null` // ''; + } + die "FATAL: chroot $name is not disposable (no union/snapshot session): a build would mutate the\n" + . " shared base filesystem and leak build-dependencies into the next package. Add\n" + . " 'union-type=overlay' to its /etc/schroot/chroot.d/ entry, or delete the chroot so this\n" + . " script re-creates it.\n" + unless chroot_is_disposable($cfg); + return 1; +} + sub ensure_chroots { print_step('Ensure sbuild chroots (auto-init on first run)'); die "FATAL: chroot init requires root (uid=$>)\n" if $> != 0 && !$dry_run; @@ -305,7 +393,11 @@ sub ensure_chroots { ."echo '/opt/xcat-ci-shared /opt/xcat-ci-shared none rw,bind 0 0' >> $fstab", nofail => 1); for my $cn (@dist_list) { my $name = chroot_name($cn, $arch); - if (have_chroot($name)) { print " chroot $name: present\n"; next; } + if (have_chroot($name)) { + print " chroot $name: present\n"; + ensure_disposable_chroot($name); + next; + } print " chroot $name: MISSING -> creating\n"; # debootstrap may lack a script for a new codename -> fall back to the generic one. run("[ -e /usr/share/debootstrap/scripts/$cn ] || ln -sf gutsy /usr/share/debootstrap/scripts/$cn", nofail => 1); @@ -321,6 +413,8 @@ sub ensure_chroots { } else { print "+ write $root/etc/apt/sources.list (main+universe)\n"; } run("mkdir -p $root/opt/xcat-ci-shared", nofail => 1); # bind-mount target for the shared tree die "FATAL: chroot $name still absent after create\n" if !$dry_run && !have_chroot($name); + # Each package MUST build in a throwaway session (see ensure_disposable_chroot). + ensure_disposable_chroot($name); print " chroot $name: created\n"; } } @@ -644,59 +738,120 @@ sub sig_observed_key { return $obs_fpr ne '' ? $obs_fpr : undef; } -# verify_assembled_repo($manifest_href, $apt_dir, $dists_aref): the ONE completeness+signature gate, -# shared by the automatic post-assembly run (end of assemble_apt) and the standalone --verify-repo mode. -# It is the IO layer: it PARSES the repository (parse_packages_index of each published -# binary-/Packages -> %present; gpg --verify of each dists//InRelease -> %observed signer), -# PARSES the manifest (-> %expected pkg pins per cell) and resolves the GPG key (--gpg-key-id -> expected -# signer), then delegates the DECISION to the pure verify_repo_packages (per codename x arch) and -# verify_repo_signature (per codename). Package problems are [/]-prefixed; the pure signature -# problems already carry the codename unit. Any problem dies non-zero. +# release_claimed_arches($adir, $cn): the architecture set dists//Release ADVERTISES to apt +# clients ("Architectures:"), or the empty list when there is no Release. The repo's own claim, which +# is what the standalone gate holds it to when no --expect-arch is given. +sub release_claimed_arches { + my ($adir, $cn) = @_; + my $rel = "$adir/dists/$cn/Release"; + $rel = "$adir/dists/$cn/InRelease" unless -f $rel; + return () unless -f $rel; + open my $fh, '<', $rel or return (); + local $/; my $t = <$fh>; close $fh; + return parse_release_architectures($t); +} + +# resolve_expect_arches($mode, $adir): the architecture set the published repo is REQUIRED to serve. +# +# This is the crux of PR #63 review concern #3. It must be a CLAIM someone made, never an inference +# from "which binary- directories happen to be populated" -- because inferring it from presence +# is precisely what turns an entirely missing secondary architecture into a silent PASS ("ppc64el is +# absent, so this run must not have built ppc64el"). Sources, in order: +# 1. --expect-arch -- explicit; always wins. This is what the CD pipeline passes. +# 2. mode 'publish' -- the arch set actually STAGED under staging// (every one of +# which passed validate_manifest), unioned with this host's --arch. +# That is the set this publish intends to ship. +# 3. mode 'standalone' -- the union of what each codename's own Release advertises, unioned +# with --arch. Verifying a repo against its published claim is what +# catches "Release says amd64 ppc64el, but binary-ppc64el is missing". +sub resolve_expect_arches { + my ($mode, $adir) = @_; + if (@expect_arch) { + my %u = map { $_ => 1 } @expect_arch; + return [sort keys %u]; + } + my %u = ($arch => 1); + if ($mode eq 'publish') { + for my $cn (@dist_list) { + for my $d (glob("$staging/$cn/*")) { + next unless -d $d; + my $a = basename($d); + $u{$a} = 1 if $a =~ /^(amd64|ppc64el)$/; + } + } + print " expected arches (from the staged set): " . join(' ', sort keys %u) . "\n"; + } else { + for my $cn (@dist_list) { $u{$_} = 1 for release_claimed_arches($adir, $cn); } + print " expected arches (from each codename's Release 'Architectures:'): " + . join(' ', sort keys %u) . "\n"; + } + return [sort keys %u]; +} + +# verify_assembled_repo($manifest_href, $apt_dir, $dists_aref, $expect_arches_aref, $sig_required): +# the ONE completeness+signature gate, shared by the automatic pre-swap run (inside publish_repo) and +# the standalone --verify-repo mode. It is the IO layer: it PARSES the repository (parse_packages_index +# of each published binary-/Packages -> %present; gpg --verify of each dists//InRelease -> +# %observed signer), PARSES the manifest (-> %expected pkg pins per cell) and resolves the GPG key +# (--gpg-key-id -> expected signer), then delegates the DECISION to the pure verify_repo_arches (the +# arch set), verify_repo_packages (per codename x arch) and verify_repo_signature (per codename). +# Package problems are [/]-prefixed; the pure arch/signature problems already carry their +# unit. Any problem dies non-zero. +# +# $expect_arches_aref is the EXPECTED arch set from resolve_expect_arches -- required and non-empty. +# Every expected arch is checked (a missing index is MISSING-INDEX, an index with no native package is +# MISSING-ARCH), and any arch that published natives WITHOUT being expected is UNEXPECTED-ARCH. The +# gate never derives what it should demand from what it happens to find. sub verify_assembled_repo { - my ($man, $adir, $dists, $arches, $sig_enabled) = @_; + my ($man, $adir, $dists, $arches, $sig_required) = @_; my @all; - # $sig_enabled: whether a valid signature is REQUIRED. The post-assembly auto-run passes $gpg_sign - # -- a repo assembled WITHOUT --gpg-sign is intentionally unsigned, so don't demand a signature and - # false-fail. Standalone --verify-repo passes undef -> fall back to "a gpg key/home is configured". - $sig_enabled = ($gpg_home ne '' || $gpg_sign) unless defined $sig_enabled; - my $expected_key = $sig_enabled ? resolve_expected_key() : undef; + my @expected = sort @{ $arches || [] }; + die "FATAL: verify_assembled_repo: no expected architecture set (pass --expect-arch)\n" + unless @expected; + # $sig_required: whether a valid signature is DEMANDED. The pre-swap auto-run passes $gpg_sign -- + # a repo assembled WITHOUT --gpg-sign is intentionally unsigned, so don't demand a signature and + # false-fail. Standalone --verify-repo passes !--no-verify-signature, i.e. ON unless opted out. + my $expected_key = $sig_required ? resolve_expected_key() : undef; my $expected_is_fpr = defined($expected_key) ? 1 : 0; print " apt-dir: $adir\n"; - print " signature check: " . ($sig_enabled + print " expected arches: " . join(' ', @expected) . "\n"; + print " signature check: " . ($sig_required ? "on (expected key " . ($expected_key // $gpg_key_id) . ($expected_is_fpr ? '' : ' [UNRESOLVED -> hard fail]') . ")" - : "SKIPPED (no --gpg-home/--gpg-sign)") . "\n"; + : "OFF (" . ($no_verify_signature ? '--no-verify-signature' + : 'repo assembled without --gpg-sign') . ")") . "\n"; # STRICT: if signing is expected but --gpg-key-id does not resolve to a fingerprint (not in the # keyring), we CANNOT confirm the signer -- that is a hard failure, never a presence-only pass. - push @all, "SIGKEY: cannot resolve --gpg-key-id '$gpg_key_id' to a fingerprint (in the $gpg_home keyring?)" - if $sig_enabled && !$expected_is_fpr; + push @all, "SIGKEY: cannot resolve --gpg-key-id '$gpg_key_id' to a fingerprint (in the " + . ($gpg_home ne '' ? $gpg_home : 'ambient GNUPGHOME') . " keyring?)" + if $sig_required && !$expected_is_fpr; - my (%exp_sig, %obs_sig, %checked_arch); + my (%exp_sig, %obs_sig); for my $cn (@$dists) { - # Arches to verify for THIS codename: the caller's --arch set (the required FLOOR -- always - # checked, so an explicitly-requested arch whose index is empty/missing is still caught) - # UNIONed with any arch that actually published a non-empty binary-/Packages. This lets - # the multi-arch assemble (invoked with a single --arch amd64) still verify the ppc64el debs it - # carries, while a genuinely single-arch run (e.g. BUILD_PPC=false, amd64 only) never - # false-fails demanding an arch it did not build. - my %want = map { $_ => 1 } @{ $arches || [] }; - # Add an arch iff it published NATIVE debs (a Packages stanza with Architecture == that arch), - # NOT merely a non-empty index: every binary-/Packages carries the Architecture:all debs - # (grub2-xcat, genesis), so a non-empty ppc index does NOT imply ppc was built. Native-arch - # detection keeps a genuine single-arch run (BUILD_PPC=false) from demanding the ppc section. - for my $a (qw(amd64 ppc64el)) { + # Which arches carry NATIVE debs here (a Packages stanza with Architecture == that arch), as + # opposed to only the Architecture:all debs (grub2-xcat, genesis) that ride into EVERY + # binary- index. Collected for the EXPECTED set plus the other supported arches, so the + # pure verify_repo_arches can report both directions: expected-but-absent and present-but- + # unexpected (a stale arch left behind in the tree). + my %native; + for my $a (do { my %s = map { $_ => 1 } (@expected, qw(amd64 ppc64el)); sort keys %s }) { my $idx = "$adir/dists/$cn/main/binary-$a/Packages"; + $native{$a} = 0; next unless -f $idx; open my $ifh, '<', $idx or next; local $/; my $body = <$ifh>; close $ifh; - $want{$a} = 1 if index_has_native_arch($body, $a); + $native{$a} = index_has_native_arch($body, $a) ? 1 : 0; } + push @all, map { "[$cn] $_" } verify_repo_arches(\@expected, \%native); + # completeness: manifest (source of truth) vs the PUBLISHED index, per codename x arch. - for my $a (sort keys %want) { - $checked_arch{$a} = 1; + for my $a (@expected) { my $tgt = "$cn-$a"; my $req = $man->{$tgt}; + # A cell we are REQUIRED to serve but have no manifest section for cannot be verified at + # all -- that is a configuration error, not a reason to pass it. (It used to be silently + # skipped, which made an unlisted target a free PASS.) unless ($req && %$req) { - print " [$cn/$a] no manifest section [$tgt] -- skipping (codename does not target this arch)\n"; + push @all, "[$cn/$a] NO-MANIFEST section [$tgt] in $manifest, but $a is expected"; next; } my @names = required_pkgs([sort keys %$req], $skip_genesis, $skip_xcat_dep); @@ -711,73 +866,111 @@ sub verify_assembled_repo { } # signature IO: record the expected + observed signer for this codename (compared in bulk below). # Only when the expected key resolved to a fingerprint (else the SIGKEY hard-fail above stands). - if ($sig_enabled && $expected_is_fpr) { + if ($sig_required && $expected_is_fpr) { $exp_sig{$cn} = $expected_key; $obs_sig{$cn} = sig_observed_key($adir, $cn); } } - push @all, verify_repo_signature(\%exp_sig, \%obs_sig) if $sig_enabled && $expected_is_fpr; + push @all, verify_repo_signature(\%exp_sig, \%obs_sig) if $sig_required && $expected_is_fpr; if (@all) { print "$_\n" for @all; die "FATAL: apt repo INCOMPLETE (" . scalar(@all) . " problem(s))\n"; } print "[verify-repo] complete: all required packages present + version-pinned" - . ($sig_enabled ? " + Release signatures valid (key $expected_key)" : "") - . " for [" . join(' ', @$dists) . "] x {" . join(',', sort keys %checked_arch) . "}\n"; + . ($sig_required ? " + Release signatures valid (key $expected_key)" : "") + . " for [" . join(' ', @$dists) . "] x {" . join(',', @expected) . "}\n"; return; } # --------------------------------------------------------------------------------------------------- -# Phase: assemble + sign the apt repo (absorbed build-apt-repo.sh; promote-on-success) +# Phase: PUBLISH -- the single, locked, atomic finalization (absorbed build-apt-repo.sh). +# +# Concern #1 of the PR #63 review: the amd64 and ppc64el build runs execute CONCURRENTLY on their two +# hosts against the same --apt-dir. Two publishers there would interleave their wipes and rewrites of +# pool/, dists/, Release, InRelease and the tarball. The fix has three parts: +# +# 1. Build runs do not publish at all (see the $publish decision above): they only fill staging. +# Publishing is a separate step, run once, after every arch has staged and validated. +# 2. That step takes ONE GLOBAL publish lock -- not the per-arch build lock -- so even a mistaken +# second publisher (cron racing a manual run on the same host) serializes instead of interleaving. +# 3. It assembles into a SIDE TREE and swaps it in with rename(2) once the gate has passed. A reader +# (deploy.sh's rsync, an apt client on a served tree) therefore only ever sees the previous +# complete repo or the new complete repo -- never a half-wiped pool or an index that does not +# match its Release. A failed gate leaves the published tree untouched. +# +# The lock file lives on the shared tree; within a host flock() is authoritative, which is what +# matters, since the pipeline's finalization step always runs on one host (the amd64 Ubuntu builder). # --------------------------------------------------------------------------------------------------- -sub assemble_apt { - return if $skip_createrepo; - print_step('Assemble apt repo (promote validated staging -> published repo)'); - run("command -v apt-ftparchive >/dev/null 2>&1 || { echo 'need apt-utils' >&2; exit 1; }", always => 1) unless $dry_run; +my $PUBLISH_LOCK_FH; +my $PUBLISH_LOCK_WAIT = 1800; # seconds to wait for a concurrent publisher before giving up + +sub acquire_publish_lock { + make_path($output_root); + my $lockfile = "$output_root/.sbuild-all.publish.lock"; + open($PUBLISH_LOCK_FH, '>', $lockfile) or die "FATAL: cannot open publish lock $lockfile: $!\n"; + unless (flock($PUBLISH_LOCK_FH, LOCK_EX | LOCK_NB)) { + print " publish lock is held by another run -- waiting up to ${PUBLISH_LOCK_WAIT}s: $lockfile\n"; + local $SIG{ALRM} = sub { + die "FATAL: timed out after ${PUBLISH_LOCK_WAIT}s waiting for the publish lock $lockfile\n"; + }; + alarm($PUBLISH_LOCK_WAIT); + my $ok = flock($PUBLISH_LOCK_FH, LOCK_EX); + alarm(0); + die "FATAL: cannot take the publish lock $lockfile: $!\n" unless $ok; + } + print " publish lock acquired: $lockfile\n"; + return $lockfile; +} + +# assemble_into($dir, $expect_arches): (re)assemble every --dists codename inside $dir from the +# validated staging tree, index it per expected binary-, write + sign Release. $dir is the SIDE +# tree, never the published one. +sub assemble_into { + my ($dir, $expect) = @_; for my $cn (@dist_list) { my $ver = codename_to_version($cn); - my $pool = "$apt_dir/pool/main/$cn"; - # wipe ONLY this codename's published pool+dists, then repopulate from validated staging - # (both arches: staging//{amd64,ppc64el}/*.deb). Wiping first is what removes stale debs - # from a prior run so the published repo never carries a mixture (concern #1). - wipe_tree($pool, "$apt_dir/dists/$cn", "$apt_dir/$ver") unless $dry_run; - make_path($pool, "$apt_dir/$ver") unless $dry_run; - unless ($dry_run) { - # Collect this codename's staged debs across both arches, deduping on binary package - # NAME+ARCH: if two files resolve to the same package+arch (e.g. a native ppc genesis and - # an amd64-host cross-converted one both claiming xcat-genesis-base-ppc64el/all) only ONE - # may reach the pool. Keep the highest version and warn naming both -- a safety net that - # holds regardless of the --skip-genesis single-producer contract (concern #4). - my %best; # "name|arch" => { file => path, ver => version } - for my $deb (glob("$staging/$cn/*/*.deb")) { - my $name = deb_field($deb, 'Package'); - my $darch = deb_field($deb, 'Architecture'); - my $dver = deb_field($deb, 'Version'); - my $key = "$name|$darch"; - if (my $cur = $best{$key}) { - my $new_wins = deb_ver_gt($dver, $cur->{ver}); - my ($win, $lose) = $new_wins ? ($deb, $cur->{file}) : ($cur->{file}, $deb); - warn "WARN: duplicate binary $name/$darch in staging for $cn -- keeping " - . basename($win) . ", dropping " . basename($lose) . "\n"; - $best{$key} = { file => $deb, ver => $dver } if $new_wins; - next; - } - $best{$key} = { file => $deb, ver => $dver }; - } - for my $key (sort keys %best) { - my $deb = $best{$key}{file}; - my $b = basename($deb); - link($deb, "$pool/$b") or copy($deb, "$pool/$b"); - copy($deb, "$apt_dir/$ver/$b"); + my $pool = "$dir/pool/main/$cn"; + # wipe ONLY this codename's pool+dists in the side tree, then repopulate from validated + # staging (all arches: staging//{amd64,ppc64el}/*.deb). Wiping first is what removes stale + # debs from a prior run so the published repo never carries a mixture. + wipe_tree($pool, "$dir/dists/$cn", "$dir/$ver"); + make_path($pool, "$dir/$ver"); + # Collect this codename's staged debs across both arches, deduping on binary package + # NAME+ARCH: if two files resolve to the same package+arch (e.g. a native ppc genesis and + # an amd64-host cross-converted one both claiming xcat-genesis-base-ppc64el/all) only ONE + # may reach the pool. Keep the highest version and warn naming both -- a safety net that + # holds regardless of the --skip-genesis single-producer contract. + my %best; # "name|arch" => { file => path, ver => version } + for my $deb (glob("$staging/$cn/*/*.deb")) { + my $name = deb_field($deb, 'Package'); + my $darch = deb_field($deb, 'Architecture'); + my $dver = deb_field($deb, 'Version'); + my $key = "$name|$darch"; + if (my $cur = $best{$key}) { + my $new_wins = deb_ver_gt($dver, $cur->{ver}); + my ($win, $lose) = $new_wins ? ($deb, $cur->{file}) : ($cur->{file}, $deb); + warn "WARN: duplicate binary $name/$darch in staging for $cn -- keeping " + . basename($win) . ", dropping " . basename($lose) . "\n"; + $best{$key} = { file => $deb, ver => $dver } if $new_wins; + next; } + $best{$key} = { file => $deb, ver => $dver }; } - # Packages index per binary-: an arch's index carries that arch's debs + all Architecture:all. - for my $a (qw(amd64 ppc64el)) { - my $bindir = "$apt_dir/dists/$cn/main/binary-$a"; - make_path($bindir) unless $dry_run; - next if $dry_run; - my $all = `cd ${\ sh_quote($apt_dir)} && apt-ftparchive packages pool/main/$cn`; + for my $key (sort keys %best) { + my $deb = $best{$key}{file}; + my $b = basename($deb); + link($deb, "$pool/$b") or copy($deb, "$pool/$b"); + copy($deb, "$dir/$ver/$b"); + } + # Packages index per EXPECTED binary-: an arch's index carries that arch's debs + all + # Architecture:all. Only the expected arches get an index -- writing a binary-ppc64el index + # containing nothing but the Architecture:all debs would advertise a ppc64el repo that cannot + # actually satisfy a ppc64el client (and would then be mistaken for "ppc was published"). + my $all = `cd ${\ sh_quote($dir)} && apt-ftparchive packages pool/main/$cn`; + for my $a (@$expect) { + my $bindir = "$dir/dists/$cn/main/binary-$a"; + make_path($bindir); open my $pf, '>', "$bindir/Packages" or die "write Packages: $!\n"; for my $para (split /\n\n+/, $all) { next unless $para =~ /\S/; @@ -787,61 +980,130 @@ sub assemble_apt { close $pf; run("gzip -9 -kf -n " . sh_quote("$bindir/Packages")); } - next if $dry_run; - # Advertise ONLY the arches actually staged for this codename: an arch counts iff its - # binary-/Packages is non-empty. A single-arch run must not claim a missing arch in - # Release (apt would then error on the absent index). - my @staged_arches = grep { -s "$apt_dir/dists/$cn/main/binary-$_/Packages" } qw(amd64 ppc64el); - @staged_arches = ('amd64') unless @staged_arches; # never emit an empty Architectures line - # Release + sign + # Release advertises EXACTLY the expected arch set -- the same claim the gate then holds the + # tree to, and the claim a later standalone --verify-repo falls back on. my @rel = ('apt-ftparchive', '-o', 'APT::FTPArchive::Release::Origin=xCAT', '-o', 'APT::FTPArchive::Release::Label=xcat-dep', '-o', "APT::FTPArchive::Release::Suite=$cn", '-o', "APT::FTPArchive::Release::Codename=$cn", - '-o', 'APT::FTPArchive::Release::Architectures=' . join(' ', @staged_arches), + '-o', 'APT::FTPArchive::Release::Architectures=' . join(' ', @$expect), '-o', 'APT::FTPArchive::Release::Components=main', '-o', "APT::FTPArchive::Release::Description=xCAT dependency packages for $ver", - 'release', "$apt_dir/dists/$cn/"); - run(join(' ', map { sh_quote($_) } @rel) . " > " . sh_quote("$apt_dir/dists/$cn/Release")); + 'release', "$dir/dists/$cn/"); + run(join(' ', map { sh_quote($_) } @rel) . " > " . sh_quote("$dir/dists/$cn/Release")); my $det = strftime("%a, %d %b %Y %H:%M:%S +0000", gmtime($build_timestamp)); - run("sed -i " . sh_quote("s/^Date: .*/Date: $det/") . " " . sh_quote("$apt_dir/dists/$cn/Release"), nofail => 1); + run("sed -i " . sh_quote("s/^Date: .*/Date: $det/") . " " . sh_quote("$dir/dists/$cn/Release"), nofail => 1); if ($gpg_sign) { my $g = $gpg_home ? "GNUPGHOME=" . sh_quote($gpg_home) . " " : ''; - my $rel = "$apt_dir/dists/$cn/Release"; + my $rel = "$dir/dists/$cn/Release"; run("${g}gpg --default-key " . sh_quote($gpg_key_id) . " --batch --yes --armor --detach-sign -o " . sh_quote("$rel.gpg") . " " . sh_quote($rel)); run("${g}gpg --default-key " . sh_quote($gpg_key_id) . " --batch --yes --armor --clearsign -o " - . sh_quote("$apt_dir/dists/$cn/InRelease") . " " . sh_quote($rel)); + . sh_quote("$dir/dists/$cn/InRelease") . " " . sh_quote($rel)); } - print " assembled + " . ($gpg_sign ? 'signed' : 'UNSIGNED') . " apt tree for $cn\n"; + print " assembled + " . ($gpg_sign ? 'signed' : 'UNSIGNED') . " apt tree for $cn (" + . join(' ', @$expect) . ")\n"; } # export the signing pubkey for clients - if ($gpg_sign && !$dry_run) { + if ($gpg_sign) { my $g = $gpg_home ? "GNUPGHOME=" . sh_quote($gpg_home) . " " : ''; my $keysrc = "$repo_root/repomd.xml.key"; - if (-f $keysrc) { copy($keysrc, "$apt_dir/xcat-dep.asc"); } - else { run("${g}gpg --armor --export " . sh_quote($gpg_key_id) . " > " . sh_quote("$apt_dir/xcat-dep.asc"), nofail => 1); } - } - # Automatic post-assembly GATE on the just-published index (completeness + Release signatures). - # Runs once every codename's dists//.../Packages + signed Release are written. Suppressed with - # --no-verify-repo (iteration/debug); skipped under --dry-run (nothing was published). - unless ($no_verify_repo || $dry_run) { - print_step('Verify published apt repo (post-assembly completeness + signature gate)'); - # Require a valid signature iff we actually signed (--gpg-sign); a repo assembled without it is - # intentionally unsigned and must not false-fail. (Standalone --verify-repo omits this arg.) - verify_assembled_repo(\%MANIFEST, $apt_dir, \@dist_list, [$arch], $gpg_sign); + if (-f $keysrc) { copy($keysrc, "$dir/xcat-dep.asc"); } + else { run("${g}gpg --armor --export " . sh_quote($gpg_key_id) . " > " . sh_quote("$dir/xcat-dep.asc"), nofail => 1); } } } +# swap_into_place($tmp, $target): publish $tmp AS $target with rename(2) -- the only moment the +# published repo changes, and it changes all at once. Both paths are siblings, so both renames are +# same-filesystem and atomic. If the second rename fails the previous tree is put straight back. +sub swap_into_place { + my ($tmp, $target) = @_; + my $old = "$target.old-$run_id.$$"; + wipe_tree($old) if -d $old; + my $had_old = 0; + if (-d $target) { + rename($target, $old) or die "FATAL: cannot move the published repo aside ($target -> $old): $!\n"; + $had_old = 1; + } + unless (rename($tmp, $target)) { + my $err = $!; + rename($old, $target) if $had_old; # put the previous repo back; nothing was lost + die "FATAL: cannot swap the assembled repo into place ($tmp -> $target): $err\n"; + } + print " published atomically: $target\n"; + wipe_tree($old) if $had_old; +} + +sub publish_repo { + unless ($publish) { + print_step('Publish SKIPPED -- this run produced STAGING ONLY'); + print " staged under $staging (arch $arch)\n"; + print " the apt repo at $apt_dir is untouched: publishing is a separate, singly-locked,\n" + . " atomic step so two concurrent arch builds can never rewrite it at the same time.\n" + . " Finalize with: sbuild-all.pl --skip-build --skip-genesis --publish " + . "--expect-arch \"\" --gpg-sign ...\n"; + return; + } + print_step('Publish apt repo (locked, assembled aside, swapped in atomically)'); + my $expect = resolve_expect_arches('publish', $apt_dir); + if ($dry_run) { + print " [dry-run] would lock $output_root/.sbuild-all.publish.lock, assemble @dist_list for " + . join(' ', @$expect) . " into $apt_dir.publish-., gate it, then rename it " + . "onto $apt_dir\n"; + return; + } + run("command -v apt-ftparchive >/dev/null 2>&1 || { echo 'need apt-utils' >&2; exit 1; }", always => 1); + warn "WARN: publishing an UNSIGNED apt repo (no --gpg-sign) -- apt clients will reject it\n" + unless $gpg_sign; + acquire_publish_lock(); + + # Side-tree name carries the pid too: --run-id has second granularity, and two runs that start in + # the same second must not pick the same scratch path. + my $tmp = "$apt_dir.publish-$run_id.$$"; + wipe_tree($tmp) if -e $tmp; + make_path(dirname($apt_dir), $tmp); + # Seed the side tree from the currently published one so codenames OUTSIDE --dists survive the + # swap (a --dists noble run must not drop focal/jammy/resolute). A hardlink copy is cheap and + # safe here: assemble_into only unlinks and re-creates files, it never writes through a link. + if (-d $apt_dir) { + run("cp -al " . sh_quote("$apt_dir/.") . " " . sh_quote("$tmp/") . " 2>/dev/null || " + . "cp -a " . sh_quote("$apt_dir/.") . " " . sh_quote("$tmp/")); + } + + # Assemble, then GATE BEFORE PUBLISH: the completeness + signature check runs against the side + # tree, so an incomplete or mis-signed repo is never swapped in -- the previously published tree + # stays exactly as it was. The gate is suppressed with --no-verify-repo (iteration/debug). + # Anything that fails here takes the side tree with it and leaves $apt_dir untouched. + eval { + assemble_into($tmp, $expect); + unless ($no_verify_repo) { + print_step('Verify assembled apt repo (completeness + signature gate, before publishing)'); + # Require a valid signature iff we actually signed (--gpg-sign); a repo assembled without + # it is intentionally unsigned and must not false-fail. + verify_assembled_repo(\%MANIFEST, $tmp, \@dist_list, $expect, $gpg_sign); + } + 1; + } or do { + my $err = $@ || "unknown error\n"; + print " NOT publishing: $apt_dir keeps its previous contents\n"; + wipe_tree($tmp) if -d $tmp; + die $err; + }; + + swap_into_place($tmp, $apt_dir); + make_tarball(); +} + # --------------------------------------------------------------------------------------------------- -# Phase: tarball (optional) +# Phase: tarball (optional) -- part of PUBLISH, under the same lock. A per-arch build run must not tar +# the shared apt tree: with both arches running it would archive a tree the other arch is rewriting. # --------------------------------------------------------------------------------------------------- sub make_tarball { return if $skip_tarball; print_step('Tarball'); my $tb = "$output_root/$run_id/xcat-dep-$arch-$run_id.tar.gz"; - make_path(dirname($tb)) unless $dry_run; # the run dir may not exist yet (e.g. an assemble-only run) + make_path(dirname($tb)); # the run dir may not exist yet (e.g. a publish-only run) run("tar -C " . sh_quote(dirname($apt_dir)) . " -czf " . sh_quote($tb) . " " . sh_quote(basename($apt_dir)), nofail => 1); print " $tb\n"; } @@ -855,9 +1117,8 @@ unless ($skip_build) { } build_genesis() unless $skip_genesis; validate_manifest(); -assemble_apt(); -make_tarball(); -print_step("Completed ($arch: @dist_list)"); +publish_repo(); # no-op unless --publish (or a --skip-build finalization run); tarball is inside +print_step("Completed ($arch: @dist_list)" . ($publish ? '' : ' -- staging only, not published')); __END__ @@ -869,17 +1130,26 @@ sbuild-all.pl - build, validate, sign and assemble the xcat-dep Ubuntu/Debian ap sbuild-all.pl [options] - # build ALL supported Ubuntu versions for this host's arch, sign + assemble the apt tree: - sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \ - --xcat-source ../xcat-core --genesis-rpm \ + # STEP 1 -- per arch, on that arch's build host: build + validate into staging (does NOT publish): + sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \ + --xcat-source ../xcat-core --genesis-rpm + sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" --skip-genesis + + # STEP 2 -- ONCE, after every arch has staged: assemble, sign, gate and publish atomically: + sbuild-all.pl --skip-build --skip-genesis --publish --expect-arch "amd64 ppc64el" \ --gpg-sign --gpg-key-id xcat@megware.com --gpg-home # build ONE Ubuntu version only: sbuild-all.pl --arch amd64 --dists noble ... sbuild-all.pl --target noble-amd64 ... # equivalent single-target form - # assemble-only (re-sign/re-index from already-built staging): - sbuild-all.pl --skip-build --skip-genesis --gpg-sign --gpg-key-id --gpg-home + # single host, build AND publish in one go (add --publish explicitly): + sbuild-all.pl --arch amd64 --dists noble --genesis-rpm --publish --expect-arch amd64 \ + --gpg-sign --gpg-key-id --gpg-home + + # verify an already-published tree out of band (signatures checked by DEFAULT): + sbuild-all.pl --verify-repo --dists "focal jammy noble resolute" \ + --expect-arch "amd64 ppc64el" --gpg-key-id --gpg-home sbuild-all.pl --help # option summary sbuild-all.pl --man # this manual @@ -897,10 +1167,32 @@ packaging (never re-implemented) via its per-package C<< /sbuild.pl >> buil One host builds one architecture (C<--arch>, default C) for a set of Ubuntu codenames (C<--dists>). Each C<< - >> is a B with a section in C. Everything is built and validated into a fresh, per-arch B tree -first; the published apt repo is (re)assembled from validated staging only after the complete -expected set validates -- so a partial or failed build never reaches the repo and stale debs never -accumulate. Any missing chroot / package / artifact, or any version-pin mismatch, fails the whole -run non-zero. +first -- so a partial or failed build never reaches the repo and stale debs never accumulate. Any +missing chroot / package / artifact, or any version-pin mismatch, fails the whole run non-zero. + +=head2 Build runs stage; publishing is a separate, locked, atomic step + +An architecture build run B. The two arches build concurrently on their two hosts +against the same C<--apt-dir>, so a build run that also assembled would interleave its wipe and +rewrite of C, C, C, C and the tarball with the other arch's. + +Publishing happens only with C<--publish> -- or implicitly on a run that builds nothing +(C<--skip-build>), which B the finalization step. It takes B (not the +per-arch build lock), assembles the whole tree into a side directory, runs the completeness + +signature gate against B tree, and only then swaps it onto C<--apt-dir> with a single +C. Readers therefore see either the previous complete repo or the new complete repo, never +a half-written one, and a failed gate leaves the published tree untouched. Codenames outside +C<--dists> survive the swap (the side tree is seeded from the current published one). + +=head2 Each package builds in a clean, disposable chroot + +Every package is built in its own C session, and the session must be B (an +overlay/union mount or a snapshot). C repairs a chroot that lacks one +(C) and hard-fails if it still is not disposable. That is what makes the fail-hard +dependency handling meaningful: inside the session, C, the common build tooling and +the package's C (resolved with C, so versions/alternatives/arch +qualifiers are honoured) are all B on failure -- and because nothing survives the session, a +package whose C forgets a C cannot build green on a sibling's leftovers. =head1 PHASES @@ -909,12 +1201,14 @@ run non-zero. =item Ensure chroots Auto-initializes any missing C<< --sbuild >> chroot on first run (main + universe, -fast mirror, shared-tree bind-mount); idempotent. Skipped with C<--skip-build>. +fast mirror, shared-tree bind-mount) and ensures each one hands out B sessions +(C); idempotent. Skipped with C<--skip-build>. =item Build Runs each manifest package's C<< /sbuild.pl >> in the matching chroot into -CcodenameE/EarchE/>. +CcodenameE/EarchE/>. Dependency installation inside the chroot is fatal on +failure (see L). =item Genesis @@ -927,29 +1221,42 @@ ppc64el genesis (issue #7610) unless C<--require-ppc-genesis> gates it. Skipped Asserts every manifest-required package is present at its pinned version (zero tolerance). -=item Assemble +=item Publish (C<--publish>; locked + atomic) -Wipes+repopulates each codename's published C/C from validated staging, indexes per -CarchE> (Architecture:all packages land in every arch index) and gpg-signs -C/C. Skipped with C<--skip-createrepo>. +Takes the global publish lock, seeds a side tree from the currently published repo, wipes+repopulates +each C<--dists> codename's C/C there from validated staging, indexes per B +CarchE> (Architecture:all packages land in every expected arch index), advertises +exactly the expected arch set in C, gpg-signs C/C, runs the gate below +and -- only if it passes -- swaps the side tree onto C<--apt-dir> with C. Suppressed with +C<--skip-createrepo>; not run at all on a build run without C<--publish>. -=item Verify (published-repo gate) +=item Verify (repo gate) -After assembly, a manifest-driven gate asserts -- per codename E arch, against the B -CarchE/Packages> index apt clients actually see (not the staging pool) -- that every -manifest-required package is present at its pinned upstream version, and that each codename's -C is validly gpg-signed by the expected key (C, or detached C). Any -missing package, version mismatch, missing index, or unsigned/wrong-key signature fails the run. The -pure decisions (C for completeness, C -for the signer) and the index parsing (C) are unit-tested; this -script's IO layer parses the repository + resolves the gpg key and feeds those pure deciders. This -automatic gate is suppressed with C<--no-verify-repo>; the same check runs standalone against an -already-assembled tree via C<--verify-repo=Eapt_dirE>. +A manifest-driven gate asserts -- per codename E expected arch, against the +CarchE/Packages> index apt clients will actually see (not the staging pool) -- that +every manifest-required package is present at its pinned upstream version, that the repo serves +exactly the B, and that each codename's C is validly gpg-signed by +the expected key (C, or detached C). Any missing package, version mismatch, +missing index, missing/unexpected architecture, missing manifest section, or unsigned/wrong-key +signature fails the run. The pure decisions (C for the arch set, +C for completeness, C for the signer) and the parsing +(C, C) are unit-tested; this script's IO layer +parses the repository + resolves the gpg key and feeds those pure deciders. + +The expected architecture set is always a B, never an inference from what happens to be +present: C<--expect-arch> if given, else the staged arch set when publishing, else each codename's own +C C line when verifying standalone. An entirely missing secondary +architecture is therefore reported (C), not read as "this run did not build it". + +The gate runs automatically before the swap (suppress with C<--no-verify-repo>) and standalone against +an already-published tree via C<--verify-repo=Eapt_dirE>, where signatures are checked B (opt out with C<--no-verify-signature>). =item Tarball -A repo tarball build artifact (the deployable offline FRS dep bundle is produced by the pipeline's -C). Skipped with C<--skip-tarball>. +A repo tarball build artifact of the published tree, taken under the publish lock (the deployable +offline FRS dep bundle is produced by the pipeline's C). Skipped with +C<--skip-tarball>. =back @@ -1015,20 +1322,41 @@ the default gives 8 concurrent build streams for a 4-codename matrix (4 per host =item B<--skip-build> B<--skip-install> B<--skip-genesis> B<--skip-xcat-dep> B<--skip-createrepo> B<--skip-tarball> -Skip the corresponding phase(s). C<--skip-build --skip-genesis> gives an assemble-only run. +Skip the corresponding phase(s). C<--skip-build --skip-genesis> is the finalization run (it publishes +by default; see C<--publish>). C<--skip-createrepo> forces "do not publish" and always wins. + +=item B<--publish> + +Run the finalization phase: take the global publish lock, assemble + sign into a side tree, gate it and +swap it onto C<--apt-dir> atomically. B, so the two +arches can build concurrently without racing each other on the shared repo. Defaults to on for a run +that builds nothing (C<--skip-build>), which is the finalization step. C<--no-publish> forces it off. + +=item B<--expect-arch> C<< amd64|ppc64el >> + +The architecture set the published repo must serve, stated explicitly. Repeatable, and each value may +be a space/comma list (C<--expect-arch "amd64 ppc64el">). Used by the gate: an expected arch with no +native package is C, an unexpected arch that published natives is C, +and C advertises exactly this set. Without it the gate falls back to the staged arch set when +publishing, or to each codename's C C when verifying standalone. =item B<--verify-repo> C<< = >> -Standalone mode: verify an already-assembled apt tree at C<< >> (completeness + Release -signatures) using this script's manifest resolution (C<--manifest> or the default -C), C<--dists>, and C<--gpg-key-id>/C<--gpg-home>, then exit. Takes no run lock and -builds nothing. +Standalone mode: verify an already-published apt tree at C<< >> (architecture set + +completeness + Release signatures) using this script's manifest resolution (C<--manifest> or the +default C), C<--dists>, C<--expect-arch>, and C<--gpg-key-id>/C<--gpg-home>, then +exit. Signatures are verified B. Takes no run lock and builds nothing. =item B<--no-verify-repo> -Suppress the B post-assembly completeness+signature gate (for iteration/debug). The gate is +Suppress the B pre-swap completeness+signature gate (for iteration/debug). The gate is ON by default. +=item B<--no-verify-signature> + +Explicitly skip the Release signature check in the standalone C<--verify-repo> mode (for an +intentionally unsigned local tree). Without it, an unsigned or wrongly-signed repo fails the gate. + =item B<--dry-run> Print the planned actions without executing them. diff --git a/t/sbuild-all.t b/t/sbuild-all.t index f26471f..b53d9c1 100644 --- a/t/sbuild-all.t +++ b/t/sbuild-all.t @@ -13,12 +13,14 @@ use File::Temp qw(tempdir); use File::Path qw(make_path); use File::Basename qw(basename); use BuildUtils qw(required_pkgs version_matches read_manifest standard_options - verify_repo_packages verify_repo_signature parse_packages_index resolve_present_names + verify_repo_packages verify_repo_signature verify_repo_arches + parse_packages_index parse_release_architectures resolve_present_names index_has_native_arch control_binary_arch skip_arch_all_on codename_to_version version_to_codename known_codenames - chroot_name chroot_sources_list + chroot_name chroot_sources_list chroot_is_disposable chroot_build_script control_field genesis_deb_control - deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb); + deb_field deb_version deb_upstream_version deb_hash cross_copy_genesis_deb + build_deb_in_chroot); # Run a printing sub with STDOUT muted so its progress lines do not pollute TAP. sub quiet(&) { @@ -382,4 +384,150 @@ SKIP: { ok(!skip_arch_all_on($ctl, 'syslinux-xcat', undef), 'undef arch -> not skipped (no crash)'); } +# ---- verify_repo_arches: the expected arch SET is a claim, never "whatever is present" ------------ +# PR #63 review concern #3: with the set inferred from presence, an entirely missing secondary +# architecture read as "this run did not build it" and PASSED. These are the two directions that must +# now be reported. +{ + my @expect = qw(amd64 ppc64el); + + is_deeply([verify_repo_arches(\@expect, { amd64 => 1, ppc64el => 1 })], [], + 'every expected arch has native packages -> no problems'); + + # The false-PASS: a two-arch repo whose ppc64el side is entirely absent. + my @m = verify_repo_arches(\@expect, { amd64 => 1, ppc64el => 0 }); + is(scalar(@m), 1, 'an expected arch with no native package -> one problem'); + like($m[0], qr/^MISSING-ARCH ppc64el /, '... reported as MISSING-ARCH, not silently passed'); + + # An arch:all-only ppc index (grub2-xcat/genesis ride into every binary-) is NOT evidence + # that ppc was published -- index_has_native_arch is what feeds %native, so it stays MISSING-ARCH. + my $allonly = "Package: grub2-xcat\nVersion: 2.12-1\nArchitecture: all\n"; + my @m2 = verify_repo_arches(\@expect, + { amd64 => 1, ppc64el => (index_has_native_arch($allonly, 'ppc64el') ? 1 : 0) }); + like($m2[0], qr/^MISSING-ARCH ppc64el /, + 'an arch:all-only index does not satisfy an expected arch'); + + # The other direction: a stale arch left in a tree that is no longer built for it. + my @u = verify_repo_arches(['amd64'], { amd64 => 1, ppc64el => 1 }); + is(scalar(@u), 1, 'a native arch outside the expected set -> one problem'); + like($u[0], qr/^UNEXPECTED-ARCH ppc64el .*expected set \[amd64\]/, + '... reported as UNEXPECTED-ARCH naming the expected set'); + + # A genuine single-arch run (BUILD_PPC=false) expects only amd64 and must stay clean. + is_deeply([verify_repo_arches(['amd64'], { amd64 => 1, ppc64el => 0 })], [], + 'single-arch expectation with no ppc natives -> no problems (no false MISSING/UNEXPECTED)'); +} + +# ---- parse_release_architectures: the repo's own claim (standalone-verify fallback) --------------- +{ + my $rel = "Origin: xCAT\nLabel: xcat-dep\nSuite: noble\nCodename: noble\n" + . "Architectures: amd64 ppc64el\nComponents: main\nDate: Thu, 21 Aug 2026 00:00:00 +0000\n"; + is_deeply([parse_release_architectures($rel)], [qw(amd64 ppc64el)], + 'Architectures: line parsed in file order'); + is_deeply([parse_release_architectures("Architectures: amd64 amd64 ppc64el \n")], + [qw(amd64 ppc64el)], 'duplicates collapsed, trailing whitespace ignored'); + is_deeply([parse_release_architectures("Origin: xCAT\nComponents: main\n")], [], + 'no Architectures: field -> empty list (no claim)'); + is_deeply([parse_release_architectures(undef)], [], 'undef input -> empty list (no crash)'); +} + +# ---- chroot_is_disposable: each package must build in a THROWAWAY environment (concern #2) -------- +# A shared type=directory chroot with no union mount keeps package N's build-deps installed for +# package N+1, so a missing Build-Depends builds green. build_deb_in_chroot hard-fails on that. +{ + my $overlay = "[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\n" + . "union-type=overlay\nprofile=sbuild\n"; + ok(chroot_is_disposable($overlay), 'union-type=overlay -> disposable (what sbuild-createchroot sets)'); + ok(chroot_is_disposable("type=directory\nunion-type=aufs\n"), 'union-type=aufs -> disposable'); + ok(chroot_is_disposable("type=file\nfile=/srv/chroot/noble.tar.gz\n"), + 'type=file (tarball unpacked per session) -> disposable'); + ok(chroot_is_disposable("type=btrfs-snapshot\n"), 'snapshot chroot -> disposable'); + + ok(!chroot_is_disposable("[x]\ntype=directory\ndirectory=/srv/chroot/x\nunion-type=none\n"), + 'type=directory with union-type=none -> NOT disposable (build-deps would leak)'); + ok(!chroot_is_disposable("[x]\ntype=directory\ndirectory=/srv/chroot/x\n"), + 'type=directory with no union-type at all -> NOT disposable'); + ok(!chroot_is_disposable(''), 'empty config -> not disposable (fail closed)'); + ok(!chroot_is_disposable(undef), 'undef config -> not disposable (no crash, fail closed)'); +} + +# ---- chroot_build_script: the in-chroot build is FAIL-HARD (concern #2) -------------------------- +# Regression guard for the exact defects the review named: the common package installation ended in +# `|| true`, and a failed Build-Depends installation only warned -- so a package could build without +# its declared dependencies and ship. +{ + my $s = chroot_build_script(); + like($s, qr/^set -euo pipefail$/m, 'the in-chroot script aborts on any unchecked failure'); + + unlike($s, qr/apt-get[^\n]*\|\|\s*true/, 'no `apt-get ... || true` (installation is fatal)'); + unlike($s, qr/apt_retry[^\n]*\|\|\s*true/, 'no `apt_retry ... || true`'); + unlike($s, qr/\[warn\][^\n]*build-dep/i, 'a failed build-dep install is not downgraded to a warning'); + unlike($s, qr/^\s*BD=\$\(sed/m, 'the sed Build-Depends extraction is gone'); + + like($s, qr/mk-build-deps --install --remove/, + 'Build-Depends come from mk-build-deps (honours versions/alternatives/arch qualifiers)'); + like($s, qr/apt_retry update -q/, 'apt-get update is retried then fatal'); + like($s, qr/apt_retry install -y/, 'the common tooling install is retried then fatal'); + like($s, qr/\bequivs\b/, 'equivs is installed (mk-build-deps needs it)'); + + # the mk-build-deps dummy package must never be collected as build output + like($s, qr/!\s*-name\s+'\*-build-deps_\*'/, 'the *-build-deps dummy deb is excluded from collection'); + like($s, qr/!\s*-name\s+'\*-dbgsym_\*'/, 'debug symbols are excluded from collection'); + + # Every apt-get in the script is either the ONE inside the fatal apt_retry helper, the helper's + # own diagnostics, or the resolver mk-build-deps is told to use -- never a direct, best-effort + # `apt-get ... || true` call. + my @apt = grep { /\bapt-get\b/ && !/^\s*#/ } split /\n/, $s; + my @unaccounted = grep { !/if apt-get "\$\@"; then return 0; fi/ + && !/^\s*echo / + && !/--tool 'apt-get / } @apt; + is_deeply(\@unaccounted, [], + 'every apt-get goes through the fatal apt_retry helper (or is mk-build-deps\' --tool)') + or diag("unaccounted apt-get line(s):\n" . join("\n", @unaccounted)); +} + +# ---- build_deb_in_chroot REFUSES a non-disposable chroot (the guard, not just the predicate) ------ +# Driven with a stub `schroot` on PATH so the guard can be exercised without a real chroot: it answers +# `-l` (the chroot exists), `--config` (the configuration under test) and swallows the build itself. +{ + my $fakebin = tempdir(CLEANUP => 1); + open my $fh, '>', "$fakebin/schroot" or die "write stub schroot: $!"; + print $fh <<'STUB'; +#!/bin/bash +case "$1" in + -l) echo "chroot:noble-amd64-sbuild"; exit 0 ;; + --config) printf '%s\n' "$FAKE_SCHROOT_CONFIG"; exit 0 ;; +esac +exit 0 +STUB + close $fh; + chmod 0755, "$fakebin/schroot"; + + my $work = tempdir(CLEANUP => 1); + make_path("$work/pkg", "$work/out"); + my @args = (pkg => 'fixture', chroot => 'noble-amd64-sbuild', pkg_dir => "$work/pkg", + result_dir => "$work/out", build_timestamp => 1755000000, build => "true\n"); + + { + local $ENV{PATH} = "$fakebin:$ENV{PATH}"; + local $ENV{FAKE_SCHROOT_CONFIG} = + "[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\n"; + my $ok = eval { quiet { build_deb_in_chroot(@args) }; 1 }; + ok(!$ok, 'build_deb_in_chroot refuses to build in a non-disposable chroot'); + like($@, qr/is NOT disposable/, '... with an explicit "NOT disposable" error'); + like($@, qr/union-type=overlay/, '... telling the operator how to fix it'); + } + { + # Same stub, now reporting an overlay: the guard passes, so the failure must be the LATER + # host-side "the debs did not land" check -- proving the disposability gate is not blanket-red. + local $ENV{PATH} = "$fakebin:$ENV{PATH}"; + local $ENV{FAKE_SCHROOT_CONFIG} = + "[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\nunion-type=overlay\n"; + my $ok = eval { quiet { build_deb_in_chroot(@args) }; 1 }; + ok(!$ok, 'a disposable chroot gets past the guard (and then fails for another reason)'); + unlike($@, qr/is NOT disposable/, '... the failure is NOT the disposability guard'); + like($@, qr/no \.deb is visible/, '... it is the host-side "debs did not land" check'); + } +} + done_testing; diff --git a/t/verify-repo.t b/t/verify-repo.t new file mode 100644 index 0000000..931698b --- /dev/null +++ b/t/verify-repo.t @@ -0,0 +1,209 @@ +#!/usr/bin/perl +# Focused end-to-end tests for the PUBLISHED-REPO GATE, driving the real `sbuild-all.pl --verify-repo` +# against hand-built fixture apt trees. The pure decisions are unit-tested in t/sbuild-all.t; what is +# tested HERE is the wiring that the PR #63 review found to have false-PASS cases (concern #3): +# +# * the standalone gate must verify Release SIGNATURES BY DEFAULT -- it used to skip the check +# silently unless --gpg-home happened to be passed, so `--verify-repo ` on an unsigned tree +# reported success; +# * an entirely MISSING secondary architecture must be reported, not read as "this run did not build +# that arch". The expected arch set is a claim (--expect-arch, else the repo's own Release +# "Architectures:" line), never an inference from which binary- dirs happen to be populated. +# +# The fixtures are plain text (Packages/Release), so no dpkg/apt/gpg tooling is required. +use strict; +use warnings; +use Test::More; +use FindBin qw($RealBin); +use File::Temp qw(tempdir); +use File::Path qw(make_path remove_tree); + +my $SCRIPT = "$RealBin/../sbuild-all.pl"; +plan skip_all => "sbuild-all.pl not found at $SCRIPT" unless -f $SCRIPT; + +my $tmp = tempdir(CLEANUP => 1); + +# An empty GNUPGHOME makes the signature outcome deterministic wherever this runs: --gpg-key-id can +# never resolve to a fingerprint here, so "signatures are checked" is observable as a hard failure +# rather than depending on the developer's keyring. +my $gpghome = "$tmp/gpg-home"; +make_path($gpghome); +chmod 0700, $gpghome; + +sub write_file { + my ($path, $body) = @_; + make_path($path =~ m{^(.*)/[^/]+$} ? $1 : '.'); + open my $fh, '>', $path or die "write $path: $!"; + print $fh $body; + close $fh; +} + +sub stanza { + my ($pkg, $ver, $arch) = @_; + return "Package: $pkg\nVersion: $ver\nArchitecture: $arch\n" + . "Filename: pool/main/noble/${pkg}_${ver}_${arch}.deb\nDescription: fixture\n\n"; +} + +# The two Architecture:all packages ride into EVERY binary- index, exactly as apt-ftparchive +# emits them -- which is why a non-empty ppc index is not evidence that ppc64el was built. +sub arch_all_stanzas { + my ($genesis_arch) = @_; + return stanza('grub2-xcat', '2.12-1', 'all') + . stanza("xcat-genesis-base-$genesis_arch", '2.19.0-snap202608211200', 'all'); +} + +sub native_stanzas { + my ($a) = @_; + return stanza('ipmitool-xcat', "1.8.18-snap202608211200", $a) + . stanza('goconserver', "0.3.3-snap202608211200", $a); +} + +# make_repo(%opt): a fixture apt tree for codename 'noble'. +# arches => the arches to write a binary-/Packages for (default both) +# release_arches => what Release advertises (default: the same list) +# native => arches that get NATIVE stanzas (default: all of `arches`) +my $repo_seq = 0; +sub make_repo { + my (%o) = @_; + my @arches = @{ $o{arches} // ['amd64', 'ppc64el'] }; + my @rel = @{ $o{release_arches} // [@arches] }; + my %native = map { $_ => 1 } @{ $o{native} // [@arches] }; + my $dir = "$tmp/repo" . (++$repo_seq); + for my $a (@arches) { + my $body = ($native{$a} ? native_stanzas($a) : '') . arch_all_stanzas($a); + write_file("$dir/dists/noble/main/binary-$a/Packages", $body); + } + write_file("$dir/dists/noble/Release", + "Origin: xCAT\nLabel: xcat-dep\nSuite: noble\nCodename: noble\n" + . "Architectures: " . join(' ', @rel) . "\nComponents: main\n" + . "Date: Thu, 21 Aug 2026 12:00:00 +0000\n"); + return $dir; +} + +# The manifest is the source of truth for what each codename x arch must carry. +my $manifest = "$tmp/debs-manifest.conf"; +write_file($manifest, <<'MAN'); +[noble-amd64] +ipmitool-xcat=1.8.18 +goconserver=0.3.3 +grub2-xcat=* +xcat-genesis-base=2.* + +[noble-ppc64el] +ipmitool-xcat=1.8.18 +goconserver=0.3.3 +grub2-xcat=* +xcat-genesis-base=2.* +MAN + +# run_cmd(@argv) -> ($exit_code, $combined_output) +sub run_cmd { + my (@cmd) = @_; + my $line = join(' ', map { my $x = $_; $x =~ s/'/'"'"'/g; "'$x'" } @cmd) . ' 2>&1'; + local $ENV{GNUPGHOME} = $gpghome; + open my $ph, '-|', $line or die "cannot run: $line: $!"; + my $out = do { local $/; <$ph> }; + close $ph; + return ($? >> 8, defined $out ? $out : ''); +} + +# run_gate($repo, @extra) -> ($exit_code, $output). --arch amd64 is always pinned so the result does +# not depend on the host running the test. Note that --gpg-home is deliberately NOT passed: the +# reviewer's report was that omitting it silently disabled the signature check, so the tests below +# exercise exactly that invocation. (run_cmd still points GNUPGHOME at an empty keyring, so the +# outcome does not depend on the developer's own keys.) +sub run_gate { + my ($repo, @extra) = @_; + return run_cmd($^X, $SCRIPT, '--verify-repo', $repo, '--manifest', $manifest, + '--dists', 'noble', '--arch', 'amd64', @extra); +} + +# ---- a complete two-arch repo passes (the baseline: the gate is not simply always-red) ----------- +{ + my $repo = make_repo(); + my ($rc, $out) = run_gate($repo, '--no-verify-signature'); + is($rc, 0, 'complete two-arch repo passes the gate') or diag($out); + like($out, qr/\[verify-repo\] complete/, '... and says so'); + like($out, qr/expected arches: amd64 ppc64el/, + '... having taken the expected arch set from the repo\'s own Release claim'); +} + +# ---- THE FALSE-PASS: an entirely missing secondary architecture ----------------------------------- +# The repo advertises amd64 + ppc64el but ships no binary-ppc64el at all. Verified with --arch amd64 +# and no --expect-arch -- the exact invocation that used to report success. +{ + my $repo = make_repo(arches => ['amd64'], release_arches => ['amd64', 'ppc64el']); + my ($rc, $out) = run_gate($repo, '--no-verify-signature'); + isnt($rc, 0, 'a repo missing an advertised architecture FAILS (was a silent pass)') or diag($out); + like($out, qr/MISSING-ARCH ppc64el|\[noble\/ppc64el\] MISSING-INDEX/, + '... naming the missing architecture'); +} + +# ... and the same tree with the expected set passed EXPLICITLY, which is what the CD pipeline does. +{ + my $repo = make_repo(arches => ['amd64'], release_arches => ['amd64']); + my ($rc, $out) = run_gate($repo, '--no-verify-signature', '--expect-arch', 'amd64 ppc64el'); + isnt($rc, 0, '--expect-arch demands the arch even when Release does not advertise it') or diag($out); + like($out, qr/MISSING-ARCH ppc64el|\[noble\/ppc64el\] MISSING-INDEX/, '... naming it'); +} + +# An index that exists but carries ONLY the Architecture:all packages is not a built architecture: +# grub2-xcat and the genesis debs land in every binary-/Packages. +{ + my $repo = make_repo(arches => ['amd64', 'ppc64el'], native => ['amd64']); + my ($rc, $out) = run_gate($repo, '--no-verify-signature'); + isnt($rc, 0, 'an arch:all-only ppc index does not count as a built ppc64el') or diag($out); + like($out, qr/MISSING-ARCH ppc64el/, '... reported as MISSING-ARCH'); + like($out, qr/MISSING ipmitool-xcat|MISSING goconserver/, + '... and its native packages are reported missing'); +} + +# A genuinely single-arch repo (BUILD_PPC=false) that advertises only amd64 must NOT false-fail. +{ + my $repo = make_repo(arches => ['amd64'], release_arches => ['amd64']); + my ($rc, $out) = run_gate($repo, '--no-verify-signature'); + is($rc, 0, 'an honest single-arch repo passes (no false MISSING-ARCH)') or diag($out); +} + +# A stale architecture left in a tree that is no longer expected is also a problem. +{ + my $repo = make_repo(); + my ($rc, $out) = run_gate($repo, '--no-verify-signature', '--expect-arch', 'amd64'); + isnt($rc, 0, 'natives for an arch outside --expect-arch FAIL (stale architecture)') or diag($out); + like($out, qr/UNEXPECTED-ARCH ppc64el/, '... reported as UNEXPECTED-ARCH'); +} + +# ---- THE OTHER FALSE-PASS: standalone verification skipped signatures ---------------------------- +# Same complete tree, without --no-verify-signature AND without --gpg-home -- the invocation that used +# to report success because the signature check was silently skipped. The fixture has no +# InRelease/Release.gpg, so a gate that actually checks signatures must fail. +{ + my $repo = make_repo(); + my ($rc, $out) = run_gate($repo); + isnt($rc, 0, 'an UNSIGNED repo fails standalone verification by default') or diag($out); + like($out, qr/signature check: on/, '... the signature check is on by default'); + like($out, qr/UNSIGNED noble|SIGKEY/, '... and the problem is reported'); +} + +# ... and the opt-out is explicit, not accidental. +{ + my $repo = make_repo(); + my ($rc, $out) = run_gate($repo, '--no-verify-signature'); + is($rc, 0, '--no-verify-signature is the deliberate opt-out') or diag($out); + like($out, qr/signature check: OFF \(--no-verify-signature\)/, + '... and the log names the flag that turned it off'); +} + +# ---- a cell with no manifest section is a configuration error, not a free pass -------------------- +{ + my $thin = "$tmp/thin-manifest.conf"; + write_file($thin, "[noble-amd64]\nipmitool-xcat=1.8.18\ngoconserver=0.3.3\n" + . "grub2-xcat=*\nxcat-genesis-base=2.*\n"); + my $repo = make_repo(); + my ($rc, $out) = run_cmd($^X, $SCRIPT, '--verify-repo', $repo, '--manifest', $thin, + '--dists', 'noble', '--arch', 'amd64', '--no-verify-signature'); + isnt($rc, 0, 'an expected cell with no manifest section FAILS (was silently skipped)') or diag($out); + like($out, qr/NO-MANIFEST section \[noble-ppc64el\]/, '... naming the missing section'); +} + +done_testing;