diff --git a/BUILD.md b/BUILD.md index fdf59a9..322095b 100644 --- a/BUILD.md +++ b/BUILD.md @@ -1,12 +1,16 @@ -# Build Guide (`mockbuild-all.pl`) +# Build Guide (xcat-dep) -This guide explains how to use `mockbuild-all.pl` to build, validate, and package xCAT dependencies and optional xCAT packages into a unified EL10 repository layout. -It also documents the operational flags for controlling build, install-check, collection, and packaging behavior. +This guide explains how to build, validate, and package the **xcat-dep** dependency packages +with `mockbuild-all.pl` — the RPM build orchestrator for EL targets, driven by `mock`. + +The full xCAT **core** is NOT built here — it is built and published separately by the +xcat-core pipeline. `mockbuild-all.pl` builds only the dependency packages plus the +OS-dependent `xCAT-genesis-base` (pulled individually out of the xcat-core source tree). # Purpose -`mockbuild-all.pl` is the top-level build orchestrator for generating a **unified xCAT repository**. -It builds required dependency RPMs and, by default, xCAT RPMs, then assembles: +`mockbuild-all.pl` is the top-level build orchestrator for the **xcat-dep** RPM repository. +It builds the dependency RPMs and the OS-dependent `xCAT-genesis-base`, then assembles: - a binary RPM repo tree with repodata - an SRPM repo tree with repodata @@ -14,12 +18,14 @@ It builds required dependency RPMs and, by default, xCAT RPMs, then assembles: # Historical Context -Historically, the deployment flow used separate repositories: +The deployment flow uses two separate repositories: -- `xcat-core` for xCAT packages -- `xcat-dep` for dependency packages +- `xcat-core` for xCAT packages (built by the xcat-core pipeline) +- `xcat-dep` for dependency packages (built here) -The current flow produces a single **unified `xcat` repository** containing all required packages together. +An earlier iteration of this script also built the full xCAT core into one unified tree +(via a `--skip-xcat` toggle). That is gone: the core is always built by the xcat-core +pipeline now, and `mockbuild-all.pl` builds only xcat-dep plus `xCAT-genesis-base`. # Placeholder Conventions @@ -42,32 +48,62 @@ This guide uses the following placeholders consistently: - `/ipmitool/mockbuild.pl` - `/syslinux/mockbuild.pl` - `/goconserver/mockbuild.pl` +- `/conserver/mockbuild.pl` +- `/xnba/mockbuild.pl` - `/mockbuild-perl-packages.pl` -- `/buildrpms.pl` (unless `--skip-xcat` is set) +- `/buildrpms.pl` — only to build the OS-dependent `xCAT-genesis-base` package (unless `--skip-genesis` is set); the full xCAT core is built separately by the xcat-core pipeline, not here. Each build path uses `mock` for chroot isolation. Top-level steps are parallelized by `mockbuild-all.pl`, and perl dependency builds are also parallelized internally by `mockbuild-perl-packages.pl`. +## Per-target package manifest + +`packages-manifest.conf` (repo root) declares, per target, exactly which packages are required — +one `[]` section (matching `--target`, e.g. `[alma+epel-10-x86_64]`) of +`=` lines. For each target, `mockbuild-all.pl` builds **only** the packages +listed for it; a package absent from a target's section is not built for that target (the per-EL +perl set differs because the OS/EPEL already provides some modules). Note `conserver-xcat` is **not** +pulled in by `dnf install xCAT` (goconserver superseded it), yet it is listed in — and therefore +built for — every target, because some deployments still use it. The lists were derived empirically — on a clean MN of each +(EL, arch), `dnf install xCAT` from xcat.org latest, and the packages whose `from_repo=xcat-dep` +are exactly the required set. See the file header for details. + +Build failures are **not tolerated**: any required (manifest) package that fails to build fails +the whole run. + `mockbuild-all.pl` does more than building RPMs. In a default run it performs these stages: 1. Optional chroot cleanup (`--scrub-all-chroots`) -2. Parallel build execution -3. Optional install/smoke checks inside child builders (disabled with `--skip-install`) +2. Parallel build execution — only the target's manifest packages; any failure fails the run +3. Post-build chroot scrub — reclaims each build step's mock chroot (unless `--keep-buildroots`) 4. Binary RPM collection into `repo//` 5. Source RPM collection into `repo-src/` 6. `createrepo --update` on both repo trees 7. Tarball creation for both repo trees 8. Summary generation (`summary.txt`) +The build process does **not** install any built RPM onto the build host. Installing an EL8/EL9 +package on the (single, possibly EL10) build host corrupts the host RPM database; the real +install-and-run verification happens in the CI's separate Test phase (`cluster-test.pl` boots a +matching MN and installs xCAT + the freshly built xcat-dep there). + +# Packages notes + +- **`pyodbc`** is intentionally not built or listed in any target's manifest: modern EL provides + `python3-pyodbc` from appstream/EPEL, so xcat-dep no longer ships its own. The legacy `pyodbc/` + directory (an old `pyodbc-3.0.7` RPM spec) is kept for historical reference only. +- **`conserver-xcat`** was replaced by `goconserver` but is provided for completeness and backward + compatibility. Core packages depend on `goconserver`; to use conserver you must install + `conserver-xcat` explicitly (it is **not** pulled in as a dependency), disable the `goconserver` + service and enable the `conserver` service. + # Skip and Control Flags Use these flags to skip specific operations: -- `--skip-install` - - Skips install/smoke checks performed by child builder scripts after RPM build. -- `--skip-xcat` - - Skips `/buildrpms.pl` (xCAT package build step). +- `--skip-genesis` + - Skips the `xCAT-genesis-base` build (`/buildrpms.pl --package xCAT-genesis-base`). - `--skip-xcat-dep` - - Skips non-perl xcat-dep package builders (`elilo`, `grub2-xcat`, `ipmitool-xcat`, `syslinux-xcat`, `goconserver`). + - Skips non-perl xcat-dep package builders (`elilo`, `grub2-xcat`, `ipmitool-xcat`, `syslinux-xcat`, `goconserver`, `conserver-xcat`, `xnba-undi`). - `--skip-perl` - Skips `/mockbuild-perl-packages.pl`. - `--skip-build` @@ -83,6 +119,16 @@ Use these flags to skip specific operations: per-EL Genesis packages. - `--scrub-all-chroots` - Runs `mock -r --scrub=all` before build and collection. +- `--keep-buildroots` + - Keeps each build step's mock chroot after the build instead of scrubbing it. By default, + after the parallel build phase every step's buildroot (dep packages, the per-package perl + chroots, and `xCAT-genesis-base`) is reclaimed with + `mock -r --uniqueext --scrub=chroot --scrub=bootstrap` — a lock-safe scrub (a + chroot still held by a concurrent build is refused and skipped). Both the build chroot and its + per-uniqueext bootstrap chroot are removed (each build step gets its own bootstrap, so both + must go); the shared root cache under `/var/cache/mock` is kept so rebuilds stay fast. This + stops `/var/lib/mock` from growing unbounded across runs. Pass `--keep-buildroots` to preserve + a buildroot for debugging a failed build. - `--collect-dir ` - Adds extra artifact roots to the collection phase (repeatable). - `--dry-run` @@ -96,7 +142,17 @@ Use these flags to skip specific operations: - `mockbuild-all.pl` and package sources present under ``. - xCAT sources present under ``. -Install baseline tooling: +Install baseline tooling — let the script do it, so the list cannot drift from what it loads: + +```bash +./mockbuild-all.pl --install-deps # as root, once per build host +``` + +It installs the toolchain and the Perl modules for this host's package manager (dnf on EL, zypper +on SUSE), then **loads** each module and fails if one is still missing. That last step is the point: +a missing module surfaces otherwise as a compile-time abort inside `XCAT::BuildUtils`, in the middle +of a CD run, which is how `perl-File-Slurper` and `perl-IPC-Cmd` each took a pipeline down. The +equivalent by hand: ```bash dnf -y install perl perl-File-Slurper perl-IPC-Cmd \ @@ -104,7 +160,7 @@ dnf -y install perl perl-File-Slurper perl-IPC-Cmd \ dnf-plugins-core wget git ``` -If you will build xCAT packages (that is, you will **not** use `--skip-xcat`), install xCAT build dependencies: +If you will build the `xCAT-genesis-base` package (that is, you will **not** use `--skip-genesis`), install xCAT build dependencies: ```bash cd @@ -138,9 +194,16 @@ Equivalent derivation: mock -r ... ``` -# Build Full Unified Repository (xCAT + Dependencies) +By default (no `--target`), `mockbuild-all.pl` builds all three EL releases for the host +arch: `rh8`, `rh9`, and `rh10`. Pass `--target ` to build a single target instead; it +takes **one** value and is not repeatable — run the script once per target to build several, or +omit it to build all three. -Use this mode to build dependency packages and xCAT packages together. +# Build the Dependency Repository + +`mockbuild-all.pl` builds the xcat-dep packages (dep packages, perl packages, and the +OS-dependent `xCAT-genesis-base`). The full xCAT core is **not** built here — it is built and +published separately by the xcat-core pipeline. ```bash cd /root/xcat-dep @@ -152,8 +215,10 @@ perl ./mockbuild-all.pl \ Notes: -- Install/smoke checks run by default inside child builders. -- Add `--skip-install` to skip those checks. +- The build never installs a built RPM onto the build host (see above); install-and-run + verification is the CI Test phase's job. +- Add `--skip-genesis` to skip the `xCAT-genesis-base` build (the only step that invokes + `/buildrpms.pl`). - `` is optional; when omitted it is timestamp-based. # Add an OpenEmbedded Genesis Release @@ -221,29 +286,9 @@ The per-target repository tarballs do not contain `xcat-dep/common`. For an offline installation, mirror the common repository with its metadata before disconnecting the installation network. -# Build Unified Repository Without xCAT (`--skip-xcat`) - -Use this mode to build dependency packages only and skip invoking `/root/xcat-dep/xcat-source-code/buildrpms.pl`. - -```bash -cd /root/xcat-dep -perl ./mockbuild-all.pl \ - --repo-root /root/xcat-dep \ - --xcat-source /root/xcat-dep/xcat-source-code \ - --scrub-all-chroots \ - --skip-xcat \ - --skip-install -``` - -Important behavior: - -- `--skip-xcat` skips the xCAT build step, but collection still scans: - - `/dist//rpms` -- If that path already has xCAT RPMs, they are included in the resulting unified repo. - # Common Build Modes -Full unified repo (xCAT + dependencies, with install/smoke checks): +xcat-dep repo: ```bash cd @@ -253,7 +298,7 @@ perl ./mockbuild-all.pl \ --scrub-all-chroots ``` -Full unified repo (xCAT + dependencies, skip install/smoke checks): +Dependency repo without the `xCAT-genesis-base` build: ```bash cd @@ -261,19 +306,7 @@ perl ./mockbuild-all.pl \ --repo-root \ --xcat-source \ --scrub-all-chroots \ - --skip-install -``` - -Dependency-only repo (skip xCAT package build): - -```bash -cd -perl ./mockbuild-all.pl \ - --repo-root \ - --xcat-source \ - --scrub-all-chroots \ - --skip-xcat \ - --skip-install + --skip-genesis ``` Collection-only pass from existing build artifacts: @@ -286,6 +319,30 @@ perl ./mockbuild-all.pl \ --skip-build ``` +# Cross-arch genesis-base (`--finalize-xcat-dep`) + +`xCAT-genesis-base` is a noarch package whose *name* carries the target arch +(`xCAT-genesis-base-x86_64`, `xCAT-genesis-base-ppc64` — xCAT collapses `ppc64le` to `ppc64` +via `tarch`; there is no big-endian code in it). A management node must be able to netboot +nodes of the *other* arch, so — as in 2.17 — the `x86_64` dep repo must also ship the +`ppc64` genesis and the `ppc64le` dep repo must ship the `x86_64` genesis. + +Each arch is built on its own build host, so once both per-arch repos exist, run a final, +build-free pass that cross-copies the noarch genesis between them and re-indexes + re-signs +the affected repos: + +```bash +perl ./mockbuild-all.pl --finalize-xcat-dep \ + --x86_64-repo /x86_64> \ + --ppc64le-repo /ppc64le> \ + --gpg-sign --gpg-key-name "xCAT Signing Key" --gpg-home +``` + +- If both arches were built into one shared tree, pass the same path to both options. +- Idempotent: a repo pair already carrying the fresh foreign-arch genesis is left untouched; + any stale foreign-arch genesis is dropped before the fresh one is copied in. +- It builds nothing and holds no output lock — use it alone. + # Output Artifacts and Paths For each run: @@ -387,7 +444,8 @@ of an hour on a large host. Chroots and caches live under `/var/lib/mock` and `mockbuild-all.pl` knows `rocky-10-riscv64-xcat` as a *forcearch target* (see `%forcearch_targets` in the script): it is selected with `--target` only (the default rh8/rh9/rh10 run stays the host arch), installs the shipped config into `/etc/mock/` if -missing, and then builds: +missing, and then builds the `[rocky-10-riscv64-xcat]` section of `packages-manifest.conf` +(as every target does -- a target with no section there is fatal): | what | how | |---|---| @@ -432,7 +490,7 @@ cd perl ./mockbuild-all.pl \ --target rocky-10-riscv64-xcat \ --output \ - --skip-xcat --skip-genesis --skip-install \ + --skip-xcat --skip-genesis \ --max-parallel 8 ``` @@ -442,9 +500,8 @@ deployable repo `/xcat-dep/rh10/riscv64/` (rpms, `repodata/`, `xcat-dep. `mklocalrepo.sh`, `buildinfo.txt`). Builder failures are tolerated and listed at the end. The cross-built rpms are smoke-tested without installing them on the host: ipmitool-xcat, conserver-xcat and the XS perl modules inside the emulated chroot, goconserver by running -its binaries through the binfmt handler. `--skip-install` only skips those checks and the -host installation of the noarch perl rpms (built in the native chroot); pass it when the -build host must stay untouched. +its binaries through the binfmt handler. No build step installs an rpm on the build host +(see "Per-target package manifest"). # Validation Commands @@ -466,6 +523,47 @@ find /build-output/mockbuild-all//build-logs -type f | sort - `mock target not found` - Validate with `mock -r --print-root-path` and install the required mock config packages. +# Tests + +The reusable, side-effect-free helpers live in `MockBuildUtils.pm` (package selection under the +`--skip-*` flags, version-pin matching incl. globs, RPM-identity comparison, and the cross-arch +genesis `finalize` logic). Focused fixture tests cover them: + +```bash +prove t/ # or: perl t/mockbuild-all.t +``` + +The RPM-identity / `cross_copy_genesis` cases build tiny fixture rpms and are skipped +automatically if `rpmbuild` is unavailable. + +# Repository verification gate (EL) + +After each per-target repo is built + signed, `mockbuild-all.pl` runs a **manifest-driven gate** that +fails the build if the published repo is incomplete or mis-signed. It uses `packages-manifest.conf` as +the single source of truth and is layered so the decision logic is pure and unit-tested +(`MockBuildUtils::verify_repo_packages` / `verify_repo_signature`), separate from the disk/gpg I/O. + +- **Runs automatically** at the end of `deploy_target` (per `rh/` cell). Suppress with + `--no-verify-repo`. Verify an already-built repo out of band with `--verify-repo=` + (target derived from the `rh/` path, or pass `--target`; manifest from `/ + packages-manifest.conf`; key/home from `--gpg-key-name`/`--gpg-home`). +- **Completeness:** every package the target's manifest section requires (after `required_pkgs` + skip-filtering) must be present with a version satisfying its pin. +- **Signature:** the repo's `repodata/repomd.xml.asc` must be a *good* signature whose **primary-key + fingerprint equals the fingerprint of `--gpg-key-name`** — i.e. the repo was signed by exactly the + CLI key. Expired/revoked keys and expired signatures are rejected (not just `VALIDSIG`). If the CLI + key cannot be resolved to a fingerprint (not in the keyring) the gate fails (`SIGKEY`), never passes. + +**Semantic idiosyncrasies (intentional, and mirrored in the Ubuntu `sbuild-all.pl` gate):** + +- **What "the repo" is:** the EL gate reads the **binary rpm files** in the per-target dir (via + `rpm_version`); the Ubuntu gate reads the **published `binary-/Packages` index**. Both check + the artifact that ships; they differ only in the RHEL-vs-Debian notion of "the repository". +- **Duplicate = hard error:** if a required package appears with **two distinct versions** (a stale + artifact not cleaned before the build), the gate **dies loudly** rather than silently picking one — + identical on both EL (`rpm_version`) and Ubuntu (`parse_packages_index`). +- **Version pins** are the manifest's *upstream* version; the Debian gate strips the epoch/revision + (`deb_upstream_version`) before comparing, the EL gate compares `%{version}` directly. # Ubuntu / Debian dependency build (`sbuild-all.pl`) The EL/SUSE path above uses `mockbuild-all.pl` (rpm + mock). The Ubuntu/Debian dependency packages @@ -617,7 +715,7 @@ prints the full manual; the shared flags (`--repo-root`, `--manifest`, `--skip-build/-install/-genesis/-xcat-dep`, `--build-number`, `--gpg-sign`, `--dry-run`, …) match `mockbuild-all.pl`. -# Repository verification gate +# Repository verification gate (Ubuntu) Before an assembled repo is published, `sbuild-all.pl` runs a **manifest-driven gate** that fails the build if the repo is incomplete, serves the wrong architectures, or is mis-signed. It uses @@ -677,15 +775,6 @@ parsing/resolution), separate from the disk/gpg I/O. size/mtime and still counts). Without this, a stale checked-in artifact is republished under the current run's name — and with full-version pins it also collides with the freshly built one. -# Packages notes - -- **`pyodbc`** is intentionally not built or listed: modern Ubuntu provides `python3-pyodbc` from apt - (and EL from appstream/EPEL), so xcat-dep no longer ships its own. The legacy `pyodbc/` directory - (an old `pyodbc-3.0.7` RPM spec, no `debian/`) is kept for historical reference only. -- **`conserver-xcat`** was replaced by `goconserver` but is provided for completeness and backward - compatibility. Core packages depend on `goconserver`; to use conserver you must install - `conserver-xcat` explicitly (it is **not** pulled in as a dependency), disable the `goconserver` - service and enable the `conserver` service. # References diff --git a/MockBuildUtils.pm b/MockBuildUtils.pm new file mode 100644 index 0000000..aa9065b --- /dev/null +++ b/MockBuildUtils.pm @@ -0,0 +1,596 @@ +package MockBuildUtils; +# Reusable, unit-testable helpers factored out of mockbuild-all.pl. Kept free of that script's +# globals so t/mockbuild-all.t can exercise them directly. The two orchestration helpers that +# need signing / re-indexing (cross_copy_genesis, finalize_xcat_dep) take those as injected +# callbacks instead of reaching for gpg/createrepo state, so they stay pure and testable. +use strict; +use warnings; +use Exporter 'import'; +use File::Basename qw(basename); +use File::Copy qw(copy); +use File::Find; +use Sys::Hostname; +use Digest::MD5 qw(md5_hex); + +our @EXPORT_OK = qw( + install_deps_packages install_deps_command missing_perl_modules + sh_quote print_step + version_matches required_pkgs have_rpm read_manifest + verify_repo_packages verify_repo_signature verify_rpm_signatures + parse_evr evr_cmp evr_constraint_ok parse_pin rpmkeys_checksig_problem + rpm_version rpm_release rpm_sigmd5 rpm_is_signed restamp_release_line + cross_copy_genesis finalize_xcat_dep bump_dep_release_suffix + build_mock_uniqueext +); + +# install_deps_packages($os_id): the host packages mockbuild-all.pl needs to run at all, for the +# given /etc/os-release ID. Kept as data, beside the code that needs them, because the failure mode +# is a build host provisioned by hand: xcat-master-ub had no perl-File-Slurper and xcat-master-ppc no +# perl-IPC-Cmd, and each surfaced as a compile-time abort in the middle of a CD run. +sub install_deps_packages { + my ($os_id) = @_; + $os_id = '' unless defined $os_id; + # The perl modules are what actually break a run; the rest is the toolchain the script drives. + return qw(perl perl-File-Slurper perl-IPC-Cmd perl-Parallel-ForkManager perl-Digest-SHA + mock createrepo_c tar findutils rpm rpm-build rpm-sign rpmdevtools gnupg2 wget git) + if $os_id =~ /^(?:opensuse|sles|sled)/; + return qw(perl perl-File-Slurper perl-IPC-Cmd perl-Parallel-ForkManager perl-Digest-SHA + mock createrepo_c tar findutils rpm rpm-build rpm-sign rpmdevtools + dnf-plugins-core gnupg2 wget git); +} + +# install_deps_command($os_id): the argv that installs them, non-interactively. +sub install_deps_command { + my ($os_id) = @_; + $os_id = '' unless defined $os_id; + my @pkgs = install_deps_packages($os_id); + return ('zypper', '--non-interactive', 'install', '--no-recommends', @pkgs) + if $os_id =~ /^(?:opensuse|sles|sled)/; + return ('dnf', '-y', 'install', @pkgs); +} + +# missing_perl_modules(@modules): those that cannot be loaded, in order. The point of --install-deps +# is that the run AFTER it cannot die on a missing module, so the modules are proven by loading +# them, not by trusting the package manager's exit code. +sub missing_perl_modules { + my (@modules) = @_; + my @missing; + for my $m (@modules) { + my $file = $m; + $file =~ s{::}{/}g; + $file .= '.pm'; + eval { require $file; 1 } or push @missing, $m; + } + return @missing; +} + +# sh_quote: single-quote a string for safe use in a shell command. +sub sh_quote { + my ($s) = @_; + $s = '' if !defined $s; + $s =~ s/'/'"'"'/g; + return "'$s'"; +} + +# print_step: print a step banner. +sub print_step { + my ($msg) = @_; + print "\n== $msg ==\n"; +} + +# version_matches: does the built version $got satisfy the manifest pin $want? $want may be an +# exact version (2.19.0), a shell-style glob (2.* or 2.19.*), or '*' (any). Globs support * and +# ? and are anchored. Used so xCAT-genesis-base can pin 2.* (its Version walks with xcat-core) +# while the real xcat-dep packages stay exactly pinned. +sub version_matches { + my ($got, $want) = @_; + return 1 if !defined($want) || $want eq '*'; + return ($got eq $want) unless $want =~ /[*?]/; + my $re = quotemeta($want); + $re =~ s/\\\*/.*/g; + $re =~ s/\\\?/./g; + return $got =~ /\A$re\z/ ? 1 : 0; +} + +# required_pkgs: given a list of manifest package names and the skip flags, return the subset +# that must actually be built and validated. A package whose builder was skipped is NOT required: +# --skip-genesis drops xCAT-genesis-base, --skip-perl drops perl-*, --skip-xcat-dep drops the dep +# builders (everything that is neither genesis nor perl). Pure function (flags passed in) so both +# the version-pin check and assert_required_deps use it and it is unit-testable. +sub required_pkgs { + my ($pkgs, $skip_genesis, $skip_perl, $skip_dep) = @_; + return grep { + !($skip_genesis && $_ eq 'xCAT-genesis-base') + && !($skip_perl && /^perl-/) + && !($skip_dep && $_ ne 'xCAT-genesis-base' && $_ !~ /^perl-/) + } @$pkgs; +} + +# verify_repo_packages: the PURE completeness-decision layer of the repo gate. Given the manifest's +# %expected { pkg => version-pin } and the %present { pkg => version-found-or-undef } actually in a +# built repo, return a list of human-readable problem strings (empty list = every package present at +# its pin): +# "MISSING (manifest requires )" when %present has no (defined) version for +# "VERSION : repo has , manifest pins " when present but !version_matches(got, pin) +# Uses version_matches (same semantics as the in-line manifest pin loop), so a '*' or glob pin is +# accepted exactly as there. No file/manifest I/O here -- the disk layer builds %present and passes +# both hashes in, keeping this unit-testable in isolation. +# parse_evr($s): split an EVR string "[epoch:]version[-release]" into ($epoch, $version, $release). +# epoch defaults to '0' when absent or '(none)'; release is undef when the string carries none (so a +# release-less constraint compares version-only). Neither version nor release may contain '-', so the +# single '-' cleanly separates them. +sub parse_evr { + my ($s) = @_; + $s = '' unless defined $s; + my ($epoch, $rest); + if ($s =~ /^\s*(\d+):(.*)$/) { ($epoch, $rest) = ($1, $2); } + else { ($epoch, $rest) = ('0', $s); } + $epoch = '0' if !defined $epoch || $epoch eq '' || lc($epoch) eq '(none)'; + my ($ver, $rel) = split /-/, $rest, 2; + return ($epoch, $ver, $rel); # $rel undef when no release given +} + +# evr_cmp($got, $want, $vercmp): compare two EVRs with rpm's labelCompare semantics -- epoch first +# (numeric), then version, then release -- returning -1/0/1 (got vs want). $vercmp->($a,$b) is an +# injected rpm-native segment comparator (rpm's rpmvercmp) returning -1/0/1, so this stays pure and +# unit-testable. Release is compared only when the CONSTRAINT specifies one (rpm's EVR semantics: a +# version-only requirement ignores the built release). +sub evr_cmp { + my ($got, $want, $vercmp) = @_; + my ($ge, $gv, $gr) = parse_evr($got); + my ($we, $wv, $wr) = parse_evr($want); + return (($ge <=> $we) <=> 0) if ($ge <=> $we) != 0; # epoch: numeric + my $c = $vercmp->($gv, $wv); + return $c if $c; + return 0 unless defined $wr && $wr ne ''; # constraint release-agnostic + $gr = '' unless defined $gr; + return $vercmp->($gr, $wr); +} + +# evr_constraint_ok($got, $op, $want, $vercmp): does the observed EVR satisfy " "? +sub evr_constraint_ok { + my ($got, $op, $want, $vercmp) = @_; + my $c = evr_cmp($got, $want, $vercmp); + return $c >= 0 if $op eq '>='; + return $c > 0 if $op eq '>'; + return $c <= 0 if $op eq '<='; + return $c < 0 if $op eq '<'; + return $c == 0 if $op eq '=' || $op eq '=='; + return undef; # unknown operator +} + +# parse_pin($pin): classify a manifest version pin. +# '*' -> ('any') +# ' ' (>=,>,<=,<,=) -> ('evr', $op, $evr) full EPOCH:VERSION-RELEASE constraint +# glob or exact version -> ('version') %{VERSION}-only match (version_matches) +sub parse_pin { + my ($pin) = @_; + return ('any') if !defined($pin) || $pin eq '*'; + return ('evr', $1, $2) if $pin =~ /^\s*(>=|<=|==|=|>|<)\s*(\S+)\s*$/; + return ('version'); +} + +sub verify_repo_packages { + my ($expected, $present_ver, $present_evr, $vercmp) = @_; + $present_evr //= $present_ver; + my @problems; + for my $pkg (sort keys %$expected) { + my $pin = $expected->{$pkg}; + my $got_ver = $present_ver->{$pkg}; + if (!defined $got_ver) { + push @problems, "MISSING $pkg (manifest requires " . (defined($pin) ? $pin : '*') . ")"; + next; + } + my ($kind, $op, $want) = parse_pin($pin); + if ($kind eq 'evr') { + my $got_evr = $present_evr->{$pkg} // $got_ver; + if (!$vercmp) { + push @problems, "EVR $pkg: no EVR comparator available to check '$op $want'"; + } elsif (!evr_constraint_ok($got_evr, $op, $want, $vercmp)) { + push @problems, "EVR $pkg: repo has $got_evr, manifest requires $op $want"; + } + } elsif ($kind eq 'version') { # VERSION glob/exact (unchanged) + push @problems, "VERSION $pkg: repo has $got_ver, manifest pins $pin" + if !version_matches($got_ver, $pin); + } + # 'any' -> accept + } + return @problems; +} + +# rpmkeys_checksig_problem($name, $rc, $out): pure verdict for one `rpmkeys --checksig -v` run against +# an isolated keyring holding only the signing key. A clean rpm exits 0 and every digest/signature +# line reads OK; a tampered digest reads NOT OK; an rpm signed by another key (or unsigned) reads +# NOKEY. Return a problem string (or empty list) so the gate is testable without rpm. +sub rpmkeys_checksig_problem { + my ($name, $rc, $out) = @_; + $out = '' unless defined $out; + return () if ($rc // 0) == 0 && $out !~ /NOT OK|NOKEY|MISSING KEYS/i; + my $why = $out =~ /NOT OK/i ? 'digest/signature NOT OK' + : $out =~ /NOKEY/i ? 'NOKEY (unsigned or signed by an unaccepted key)' + : $out =~ /MISSING KEYS/i ? 'MISSING KEYS' + : "rpmkeys --checksig failed (rc=" . ($rc // '?') . ")"; + return "BADSIG rpm $name: $why"; +} + +# verify_repo_signature: the PURE signature-decision layer of the repo gate. Given %expected +# { unit => expected signing-key identity } and %observed { unit => key that ACTUALLY signed (a +# string the script extracts from gpg), or undef/'' when unsigned / verification failed }, return a +# list of problem strings (empty = every unit signed by the expected key). For EL the single unit is +# 'repomd'. This does a plain string compare only -- it invokes NO gpg: the caller runs gpg --verify, +# extracts the observed key id, and resolves --gpg-key-name to the SAME identity form before calling. +# "UNSIGNED (expected )" when %observed is absent/empty for +# "WRONGKEY : signed by , expected " when both defined but differ +sub verify_repo_signature { + my ($expected, $observed) = @_; + my @problems; + for my $unit (sort keys %$expected) { + my $exp = $expected->{$unit}; + my $obs = $observed->{$unit}; + if (!defined($obs) || $obs eq '') { + push @problems, "UNSIGNED $unit (expected " . (defined($exp) ? $exp : '') . ")"; + } elsif (defined($exp) && $obs ne $exp) { + push @problems, "WRONGKEY $unit: signed by $obs, expected $exp"; + } + } + return @problems; +} + +# verify_rpm_signatures: pure decision for the per-rpm signature gate. $rpm_sigs is an arrayref of +# [rpm_basename, observed_keyid|undef] (rpm reports the signing SUBKEY id); $accept is a hashref set +# of acceptable key ids (the signing key's primary + subkey ids, lowercased). Returns one problem per +# rpm that is unsigned or signed by a key not in the set. A signed repomd over unsigned/foreign-signed +# rpms still makes DNF reject the install, so the packages must be checked, not just the metadata. +sub verify_rpm_signatures { + my ($rpm_sigs, $accept) = @_; + my @problems; + for my $rs (@$rpm_sigs) { + my ($name, $kid) = @$rs; + if (!defined($kid) || $kid eq '') { + push @problems, "UNSIGNED rpm $name"; + } elsif (!$accept->{ lc $kid }) { + push @problems, "WRONGKEY rpm $name: signed by $kid, expected one of " + . join('/', sort keys %$accept); + } + } + return @problems; +} + +# have_rpm: is there a non-src rpm named -... under $dir? +sub have_rpm { + my ($dir, $name) = @_; + my @m = grep { !/\.src\.rpm$/ } glob("$dir/${name}-*.rpm"); + return scalar(@m) > 0; +} + +# rpm_sigmd5: the SIGMD5 of an rpm -- the digest of its header+payload, independent of the GPG +# signature. Used to compare RPM identity/content: two rpms that share a basename but differ in +# content have different SIGMD5 (a bare filename match is not enough to call them identical). +sub rpm_sigmd5 { + my ($f) = @_; + return '' unless defined $f && -f $f; + my $v = `rpm -qp --qf '%{SIGMD5}' ${\ sh_quote($f)} 2>/dev/null`; + chomp $v; + return $v; +} + +# rpm_is_signed: does the rpm carry a PGP/GPG header signature? SIGMD5 (above) is content-only and +# is identical whether or not the rpm is signed, so a cross-copied genesis that was copied but not +# yet signed (a crash between the copy and the rpmsign) still matches by SIGMD5. finalize uses this +# to treat such a rpm as NOT up to date so the copy+sign path re-runs and heals it. +sub rpm_is_signed { + my ($f) = @_; + return 0 unless defined $f && -f $f; + my $v = `rpm -qp --qf '%{SIGPGP}%{SIGGPG}' ${\ sh_quote($f)} 2>/dev/null`; + return 0 if !defined $v; + $v =~ s/\(none\)//g; # unsigned rpms report "(none)" for both tags + $v =~ s/\s+//g; + return $v ne '' ? 1 : 0; +} + +# restamp_release_line: given a spec `Release: ...` line and a CD suffix (".snap."), +# return (new_line, changed). Idempotent: a line already ending in exactly $suffix is returned +# unchanged (changed=0). A line carrying a DIFFERENT prior .snap stamp (or several, from an earlier +# corrupted run) has it stripped before the new suffix is appended, so a re-run in a reused tree +# REPLACES the stamp instead of accumulating a second one (…snap...57 -> …snap...58, never +# …snap...57.snap...58). Only the Release token is touched; a non-Release line is returned as-is. +sub restamp_release_line { + my ($line, $suffix) = @_; + return ($line, 0) unless defined $line && $line =~ /^Release:\s*\S/i; + my $qs = quotemeta($suffix); + return ($line, 0) if $line =~ /$qs\s*$/; # already carries THIS suffix + (my $new = $line) =~ s/(?:\.snap\d{12}\.\d+)+(\s*)$/$1/; # drop any prior CD stamp(s) + $new =~ s/(^Release:\s*\S+)/$1$suffix/i; + return ($new, 1); +} + +# rpm_version: %{version} of the built binary rpm named under $dir (undef if absent). +# Skips src/debug rpms and confirms the rpm's real %{name} matches (glob can over-match). +# 'xCAT-genesis-base' matches the arch-suffixed rpm name (xCAT-genesis-base-x86_64 / -ppc64). +sub rpm_version { + my ($dir, $name) = @_; + my $glob = ($name eq 'xCAT-genesis-base') + ? "$dir/xCAT-genesis-base-*.rpm" + : "$dir/${name}-*.rpm"; + my %vers; # distinct %{version}s of the matching binary rpms + for my $f (sort glob($glob)) { + next if $f =~ /\.src\.rpm$/ || $f =~ /-debug(?:info|source)-/; + my $n = `rpm -qp --qf '%{name}' ${\ sh_quote($f)} 2>/dev/null`; + my $match = ($name eq 'xCAT-genesis-base') + ? ($n =~ /^xCAT-genesis-base-/) : ($n eq $name); + next unless $match; + my $v = `rpm -qp --qf '%{version}' ${\ sh_quote($f)} 2>/dev/null`; + chomp $v; + $vers{$v} = 1 if $v ne ''; + } + return undef unless %vers; + # More than one distinct version present means a stale artifact was not cleaned before the + # build -- a version pin could then pass against the wrong rpm and both could be shipped. + # (For genesis both arches share the same Version, so a normal x86_64+ppc64 pair is one entry.) + die "Multiple versions of $name present in $dir: " . join(', ', sort keys %vers) + . " (stale artifact not cleaned before the build)\n" if keys(%vers) > 1; + my ($v) = keys %vers; + return $v; +} + +# rpm_release: %{release} of the built binary rpm named under $dir (undef if absent). Same +# name-matching as rpm_version. Used to confirm a CD --build-number/--release-suffix bump actually +# landed in the built rpm's Release (validating %{VERSION} alone can't catch a silently un-bumped NVR). +sub rpm_release { + my ($dir, $name) = @_; + my $glob = ($name eq 'xCAT-genesis-base') + ? "$dir/xCAT-genesis-base-*.rpm" + : "$dir/${name}-*.rpm"; + for my $f (sort glob($glob)) { + next if $f =~ /\.src\.rpm$/ || $f =~ /-debug(?:info|source)-/; + my $n = `rpm -qp --qf '%{name}' ${\ sh_quote($f)} 2>/dev/null`; + my $match = ($name eq 'xCAT-genesis-base') + ? ($n =~ /^xCAT-genesis-base-/) : ($n eq $name); + next unless $match; + my $r = `rpm -qp --qf '%{release}' ${\ sh_quote($f)} 2>/dev/null`; + chomp $r; + return $r if $r ne ''; + } + return undef; +} + +# read_manifest: parse packages-manifest.conf into %{ target => { package => version|'*' } }. +# INI format: [target] sections; "package=version|*" entries; blank / "#" / ";" lines ignored. +# Returns an empty hash if the file is absent (callers that build require a section per target). +sub read_manifest { + my ($path) = @_; + my %m; + return %m unless -f $path; + open my $fh, '<', $path or die "Cannot read manifest $path: $!\n"; + my $sec; + while (my $line = <$fh>) { + $line =~ s/\r?\n\z//; + $line =~ s/^\s+|\s+$//g; + next if $line eq '' || $line =~ /^[#;]/; + if ($line =~ /^\[(.+?)\]$/) { $sec = $1; $m{$sec} ||= {}; next; } + next unless defined $sec; + my ($k, $v) = split /=/, $line, 2; + $k =~ s/\s+\z//; + $v = defined($v) ? ($v =~ s/^\s+//r) : ''; + $m{$sec}{$k} = ($v ne '') ? $v : '*'; + } + close $fh; + return %m; +} + +# cross_copy_genesis: copy the noarch xCAT-genesis-base--*.rpm from $from into $to, dropping +# any stale foreign-arch genesis already in $to so the repo ends with exactly the fresh set. +# Returns the count of rpms newly copied (0 = already up to date, so the caller can skip +# re-indexing). Idempotent. $sign is an optional coderef ($rpm_path) invoked on each copied rpm +# (e.g. to re-sign it); pass undef to skip signing. Content is compared by SIGMD5, so a stale +# same-name rpm is refreshed rather than mistaken for up to date. +sub cross_copy_genesis { + my ($from, $to, $tarch, $sign) = @_; + my @src = grep { !/\.src\.rpm$/ } glob("$from/xCAT-genesis-base-$tarch-*.rpm"); + return 0 if !@src; + my %want = map { basename($_) => $_ } @src; + my @existing = grep { !/\.src\.rpm$/ } glob("$to/xCAT-genesis-base-$tarch-*.rpm"); + if (scalar(@existing) == scalar(keys %want)) { + my $up_to_date = 1; + for my $base (keys %want) { + my $dst = "$to/$base"; + my $src_sig = rpm_sigmd5($want{$base}); + # An empty SIGMD5 (unreadable rpm) means "cannot confirm identical" -> refresh rather + # than risk skipping on a false match (two '' would otherwise compare equal). + if (!-f $dst || $src_sig eq '' || $src_sig ne rpm_sigmd5($dst)) { $up_to_date = 0; last; } + # Content matches, but SIGMD5 cannot see the signature: a crash between the copy and the + # per-rpm sign leaves a same-content-but-UNSIGNED rpm. When a signer is configured, treat + # an unsigned dst as not-up-to-date so the copy+sign path re-runs and signs it. + if ($sign && !rpm_is_signed($dst)) { $up_to_date = 0; last; } + } + return 0 if $up_to_date; + } + for my $old (@existing) { + unlink $old or die "Failed to remove stale genesis $old: $!\n"; + print "[finalize] - " . basename($old) . " (stale foreign-arch, removed from $to)\n"; + } + my $copied = 0; + for my $base (sort keys %want) { + copy($want{$base}, "$to/$base") + or die "Failed to cross-copy genesis $want{$base} -> $to: $!\n"; + print "[finalize] + $base ($from -> $to)\n"; + $sign->("$to/$base") if $sign; # e.g. re-sign so the deploy gate never sees an unsigned rpm + $copied++; + } + return $copied; +} + +# finalize_xcat_dep: cross-populate the noarch xCAT-genesis-base between each matching +# /x86_64 and /ppc64le repo pair (issue #7610), then re-index the repos that changed. +# %opt: sign => coderef($rpm) applied to copied rpms (or undef); reindex => coderef($dir) run on +# a repo whose rpm set changed (or undef). Both injected so this stays free of gpg/createrepo +# state and is unit-testable. Requires each arch's own genesis rpm to be present (a pair with no +# genesis is a hard error, never a silent no-op) and fails if no repo pair is found at all. +# Architectures whose xCAT-genesis-base is cross-provisioned into every peer repo, so a management +# node can netboot nodes of any arch (issue #7610). Each entry maps the repo/subdir arch name to the +# genesis rpm's xCAT "tarch" (xCAT collapses ppc/ppc64le into tarch ppc64; x86_64 stays x86_64). This +# is the SINGLE SOURCE OF TRUTH for the cross-arch matrix -- to add an arch later (e.g. aarch64, +# riscv64) add an entry here AND wire its repo root into finalize_xcat_dep's %repo (the caller passes +# it). Discovery, the per-arch input gate, and the N-way cross-copy all iterate this list. +our @GENESIS_ARCHES = ( + { arch => 'x86_64', tarch => 'x86_64' }, + { arch => 'ppc64le', tarch => 'ppc64' }, +); + +sub finalize_xcat_dep { + my ($x86_64_repo, $ppc64le_repo, %opt) = @_; + my $sign = $opt{sign}; + my $reindex = $opt{reindex}; + print_step('Finalize xcat-dep: cross-arch genesis-base provisioning (issue #7610)'); + print "x86_64-repo: $x86_64_repo\n"; + print "ppc64le-repo: $ppc64le_repo\n"; + + # Per-arch repo root, keyed by the @GENESIS_ARCHES arch name. A new arch added to that list must + # also get its root wired here (today both roots are the same CD tree); a missing one fails loudly + # below rather than silently skipping. + my %repo = ( x86_64 => $x86_64_repo, ppc64le => $ppc64le_repo ); + + # Discover the UNION of OS dirs across ALL arch repos. Anchoring discovery on one arch let an + # built for only the OTHER arch slip through unseen -- finalize then never cross-populated + # that cell and still exited 0 (PR #62 review). Every discovered must carry every arch below. + my %os; + for my $a (@GENESIS_ARCHES) { + my $root = $repo{ $a->{arch} } + // die "FATAL: [finalize] no repo root configured for arch '$a->{arch}' (wire it in %repo)\n"; + $os{ basename($_) } = 1 for grep { -d "$_/$a->{arch}" } glob("$root/*"); + } + + my $pairs = 0; + for my $osdir (sort keys %os) { + my %adir = map { $_->{arch} => "$repo{$_->{arch}}/$osdir/$_->{arch}" } @GENESIS_ARCHES; + # Pass 1 -- every arch peer repo dir must exist: a one-arch is an incomplete input, not + # something to skip past (skipping would leave a cell without a foreign-arch genesis and still + # exit 0). Checked before the rpm pass so a missing peer is reported as such. Symmetric across + # all arches (catches an x86_64-only AND a ppc64le-only ). + for my $a (@GENESIS_ARCHES) { + die "FATAL: [finalize] $osdir: no $a->{arch} peer repo at $adir{$a->{arch}}\n" + . " (every arch must build every EL before finalize)\n" if !-d $adir{ $a->{arch} }; + } + # Pass 2 -- every arch must have produced its OWN genesis rpm before finalize cross-populates + # them; otherwise a pair with no genesis rpms would make finalize a silent no-op that still + # exits 0. xCAT collapses ppc/ppc64le into tarch=ppc64, so match on each arch's tarch. + for my $a (@GENESIS_ARCHES) { + die "FATAL: [finalize] $osdir: no $a->{arch} xCAT-genesis-base rpm (tarch $a->{tarch}) in $adir{$a->{arch}}\n" + if !grep { !/\.src\.rpm$/ } glob("$adir{$a->{arch}}/xCAT-genesis-base-$a->{tarch}-*.rpm"); + } + # N-way cross-copy: put each arch's genesis into EVERY other arch's repo dir. + my @summary; + for my $src (@GENESIS_ARCHES) { + for my $dst (@GENESIS_ARCHES) { + next if $src->{arch} eq $dst->{arch}; + my $n = cross_copy_genesis($adir{$src->{arch}}, $adir{$dst->{arch}}, $src->{tarch}, $sign); + push @summary, "$n $src->{tarch} -> $dst->{arch}"; + } + } + # Re-index+sign EVERY arch repo of this each finalize, not only when an rpm was copied + # this run: a crash after a prior run's copy+sign but before its createrepo leaves the genesis + # rpm on disk (so cross_copy_genesis now returns 0) yet ABSENT from repomd.xml -- which no + # signature gate catches. Re-indexing is cheap (tiny repos) and idempotent, and heals that + # partial state; skipped only when no signer/indexer was injected. + if ($reindex) { $reindex->($adir{$_->{arch}}) for @GENESIS_ARCHES; } + print "[finalize] $osdir: " . join(', ', @summary) . "\n"; + $pairs++; + } + die "FATAL: --finalize-xcat-dep found no repo dir under any arch root\n" + . " --x86_64-repo '$x86_64_repo'\n --ppc64le-repo '$ppc64le_repo'\n" if $pairs == 0; + print_step('Finalize complete'); +} + +# bump_dep_release_suffix: append $suffix (e.g. ".snap202607161200.57") to the Release: line of +# every xcat-dep package spec under $repo_root, so the CD build stamps a fresh, monotonic NVR. +# Idempotent: a spec already carrying this exact suffix is left alone (a re-run in the same tree +# does not double-stamp). Preserves any %{?dist}/%{?distver} macro already on the line. Returns the +# count of specs newly stamped. Dies only if NO spec under $repo_root carries a Release: line. +# Pure (takes everything as args) so t/mockbuild-all.t can exercise it directly. +sub bump_dep_release_suffix { + my ($repo_root, $suffix) = @_; + my @specs; + # Only stamp xcat-dep's OWN specs. If someone checked xcat-core out NESTED under $repo_root (the + # legacy `xcat-source-code`/`xcat-core` layout), do NOT descend into it -- rewriting a core spec + # (e.g. xCAT-genesis-base.spec's dynamic Release) would break the lockstep with genesis-scripts. + find(sub { + if (-d $_ && ($_ eq 'xcat-core' || $_ eq 'xcat-source-code')) { $File::Find::prune = 1; return; } + push @specs, $File::Find::name if /\.spec$/ && -f $_; + }, $repo_root); + my ($with_release, $bumped, $already) = (0, 0, 0); + for my $spec (sort @specs) { + open my $in, '<', $spec or die "open $spec: $!\n"; + my @lines = <$in>; + close $in; + my ($has_release, $changed) = (0, 0); + for my $line (@lines) { + # case-insensitive: some specs (e.g. Sys-Virt.spec) use a lowercase `release:` + next unless $line =~ /^Release:\s*\S/i; + $has_release = 1; + # restamp_release_line is idempotent (no-op if already carrying $suffix) and strips any + # prior .snap stamp before applying the new one, so a re-run with a different + # --build-number replaces rather than accumulates (unit-tested in t/mockbuild-all.t). + my ($new, $ch) = restamp_release_line($line, $suffix); + if ($ch) { $line = $new; $changed = 1; } + last; # only the first Release: line + } + $with_release++ if $has_release; + $already++ if $has_release && !$changed; + next unless $changed; + # atomic write (temp + rename) so a concurrent per-arch build on the shared NFS tree never + # sees a torn spec; identical suffix -> identical content, so last-writer-wins is safe. The + # temp name carries the hostname AND pid: the two arch build hosts share the NFS tree and can + # reuse the same pid, so pid alone could collide across hosts. + my $tmp = "$spec.bump." . hostname() . ".$$"; + open my $out, '>', $tmp or die "open> $tmp: $!\n"; + print {$out} @lines; + close $out; + rename $tmp, $spec or die "rename $tmp -> $spec: $!\n"; + $bumped++; + } + print "Release bump '$suffix': $bumped newly stamped, $already already stamped, of $with_release spec(s) with a Release line under $repo_root\n"; + # Only a genuine "no dep specs at all" is fatal. All-already-stamped is the expected idempotent + # case (re-run in the same tree, or the other arch bumped first) -- NOT an error. + die "FATAL: --build-number given but NO spec carried a Release: line under $repo_root (wrong tree?)\n" + if $with_release == 0; + return $bumped; +} + +# build_mock_uniqueext: a mock --uniqueext UNIQUE per (run, build-step) so concurrent mock builds +# never share a chroot root (/var/lib/mock/-). $run is the per-target run id +# (e.g. "alma+epel-8-ppc64le-"), $seq orders the step, $label names the package. +# +# The run id must NOT be blindly tail-truncated. The per-target id leads with the EL/arch token, and +# for the 7-char "ppc64le" arch the EL digit is exactly what falls off the front of a keep-the-last-24 +# truncation -- so alma+epel-{8,9,10}-ppc64le all collapse to the same run part. That is catastrophic +# for goconserver, which compiles EVERY EL in the el10 chroot (build_cfg rewritten to -10-): the +# chroot NAME is then identical across the three ELs, and the uniqueext is the ONLY thing keeping +# their roots apart, so three concurrent el8/el9/el10 ppc64le goconserver builds race in one root. +# When the id is too long, keep a readable leading token AND append a short digest of the FULL id, so +# distinct ids always yield distinct uniqueext regardless of where in the string they differ. +sub build_mock_uniqueext { + my ($run, $seq, $label) = @_; + + my $run_part = defined($run) ? $run : 'run'; + $run_part =~ s/[^A-Za-z0-9_.-]+/-/g; + $run_part =~ s/^-+|-+$//g; + $run_part = 'run' if $run_part eq ''; + if (length($run_part) > 24) { + my $digest = substr(md5_hex($run_part), 0, 8); + (my $head = substr($run_part, 0, 15)) =~ s/-+$//; + $run_part = "$head-$digest"; + } + + my $label_part = defined($label) ? $label : 'step'; + $label_part =~ s/[^A-Za-z0-9_.-]+/-/g; + $label_part =~ s/^-+|-+$//g; + $label_part = 'step' if $label_part eq ''; + $label_part = substr($label_part, 0, 20) if length($label_part) > 20; + + my $idx = defined($seq) ? int($seq) : 0; + $idx = 0 if $idx < 0; + + return sprintf("mba-%02d-%s-%s", $idx, $run_part, $label_part); +} + +1; diff --git a/conserver/mockbuild.pl b/conserver/mockbuild.pl index 61fca86..5851d7d 100755 --- a/conserver/mockbuild.pl +++ b/conserver/mockbuild.pl @@ -1,11 +1,10 @@ #!/usr/bin/perl # # mockbuild.pl - build conserver-xcat (the traditional C conserver, 8.2.1) for one -# mock target and smoke-test the resulting binaries. Mirrors the other xcat-dep +# mock target. Mirrors the other xcat-dep # builders (goconserver/mockbuild.pl, ipmitool/mockbuild.pl): stage sources + spec, -# build a SRPM, `mock --rebuild` it in the target chroot, copy the RPMs to -# --result-dir, then (unless --skip-install) install into the chroot and run -# `console -V` / `conserver -V` to confirm the binaries work. +# build a SRPM, `mock --rebuild` it in the target chroot, and copy the RPMs to +# --result-dir. # # conserver is NOT part of the default mockbuild-all.pl dep set (xCAT uses goconserver), # so this builder is standalone. Usage: @@ -29,7 +28,6 @@ my $target_arch = ''; my $mock_uniqueext = ''; my $result_dir = "$script_dir/../build-output/list-conserver/conserver"; my $log_dir = "$script_dir/../build-logs/list-conserver/conserver"; -my $skip_install = 0; my $build_timestamp; GetOptions( @@ -39,7 +37,6 @@ GetOptions( 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); @@ -118,19 +115,6 @@ for my $r (@rpms) { } print "built: " . join(', ', map { basename($_) } @rpms) . "\n"; -# ---- smoke test: install into the chroot and run the binaries --------------- -unless ($skip_install) { - print "== Smoke test (install + run) ==\n"; - run("mock -r " . sh_quote($mock_cfg) . $uniq . " --install " . sh_quote($main) - . " > " . sh_quote("$log_dir/install.log") . " 2>&1"); - # console (client) and conserver (daemon) both print their version to stderr/stdout. - my $smoke = capture("mock -r " . sh_quote($mock_cfg) . $uniq - . " --chroot -- " . sh_quote('/usr/bin/console -V 2>&1; /usr/sbin/conserver -V 2>&1') - . " 2>&1"); - print " output: $smoke\n"; - die "FATAL: smoke test did not report version $version\n" unless $smoke =~ /\Q$version\E/; - print "smoke test PASSED (console/conserver report $version)\n"; -} print "DONE: conserver-xcat $version for $mock_cfg -> $result_dir\n"; # ---- helpers ---------------------------------------------------------------- @@ -160,6 +144,6 @@ sub run_mock { } sub sh_quote { my ($s) = @_; $s =~ s/'/'\\''/g; return "'$s'"; } sub usage { - return "usage: mockbuild.pl --mock-cfg [--target-arch ARCH] [--result-dir DIR] [--work-dir DIR]\n" - . " [--log-dir DIR] [--mock-uniqueext EXT] [--skip-install]\n"; + return "usage: mockbuild.pl --mock-cfg [--target-arch ARCH] [--result-dir DIR]\n" + . " [--work-dir DIR] [--log-dir DIR] [--mock-uniqueext EXT]\n"; } diff --git a/elilo/elilo-xcat.spec b/elilo/elilo-xcat.spec index 3cbd284..397a815 100644 --- a/elilo/elilo-xcat.spec +++ b/elilo/elilo-xcat.spec @@ -19,10 +19,29 @@ Patch1: elilo-xcat.patch Patch2: elilo-big-bzimage-limit.patch Patch3: elilo-gnu-efi-strncpy-conflict.patch Source4: elilo-xcat-3.14-6.noarch.rpm +# Ship the tracked prebuilt EFI payload (SOURCE4) instead of compiling on targets where the +# gnu-efi toolchain layout does not support elilo's build: ppc64le (no x86 EFI toolchain) and +# EL8 (gnu-efi-devel places elf_x86_64_efi.lds under a path elilo's Makefile does not find). +# elilo-x64.efi is a noarch artifact, so the prebuilt is byte-identical to the compiled one. +# Use the tracked prebuilt on ppc (no x86 EFI toolchain) and on EL8 (gnu-efi lds path gap). +# Three SEPARATE %if blocks, each a single simple compare -- NOT one `A || B` expression (the +# older rpm in the EL8/EL9 mock chroot mis-evaluates `||`), and NOT %ifarch (during `rpmbuild -bs` +# the srpm's BuildRequires are frozen against %{_host_cpu}, not the target). Crucially, alma ppc +# chroots report %{_host_cpu}=powerpc64le while rocky reports ppc64le, so BOTH must be matched -- +# that mismatch is exactly why elilo pulled in the (ppc-absent) gnu-efi BuildRequires on alma ppc. +%if "%{_host_cpu}" == "ppc64le" +%global use_prebuilt 1 +%endif +%if "%{_host_cpu}" == "powerpc64le" +%global use_prebuilt 1 +%endif +%if 0%{?rhel} == 8 +%global use_prebuilt 1 +%endif BuildRequires: gcc BuildRequires: make BuildRequires: cpio -%if "%{_host_cpu}" != "ppc64le" +%if ! 0%{?use_prebuilt} BuildRequires: gnu-efi BuildRequires: gnu-efi-devel %endif @@ -48,8 +67,8 @@ sed -i 's|^GNUEFILIB[[:space:]]*=.*|GNUEFILIB = /usr/lib64|' Make.defaults sed -i 's|^EFILIB[[:space:]]*=.*|EFILIB = /usr/lib64|' Make.defaults sed -i 's|^EFICRT0[[:space:]]*=.*|EFICRT0 = /usr/lib|' Make.defaults %endif -%if "%{_host_cpu}" == "ppc64le" -# On ppc64le, reuse the prebuilt EFI payload from the tracked noarch package. +%if 0%{?use_prebuilt} +# Reuse the prebuilt EFI payload from the tracked noarch package (ppc64le / EL8). mkdir -p prebuilt rpm2cpio %{SOURCE4} | (cd prebuilt && cpio -idm --quiet) test -f prebuilt/tftpboot/xcat/elilo-x64.efi @@ -59,7 +78,7 @@ test -f prebuilt/tftpboot/xcat/elilo-x64.efi rm -rf %{buildroot} -%if "%{_host_cpu}" != "ppc64le" +%if ! 0%{?use_prebuilt} make %endif @@ -67,7 +86,7 @@ make %install mkdir -p %{buildroot}/tftpboot/xcat -%if "%{_host_cpu}" == "ppc64le" +%if 0%{?use_prebuilt} cp prebuilt/tftpboot/xcat/elilo-x64.efi %{buildroot}/tftpboot/xcat/elilo-x64.efi %else cp elilo.efi %{buildroot}/tftpboot/xcat/elilo-x64.efi diff --git a/elilo/mockbuild.pl b/elilo/mockbuild.pl index aee1619..663145d 100755 --- a/elilo/mockbuild.pl +++ b/elilo/mockbuild.pl @@ -13,32 +13,28 @@ my $repo_root = abs_path("$script_dir/.."); my $pkg_dir = "$repo_root/elilo"; my $spec_file = "$pkg_dir/elilo-xcat.spec"; -my $source_url = 'https://downloads.sourceforge.net/project/elilo/elilo/elilo-3.14/elilo-3.14-all.tar.gz'; my $source_file = ''; my $work_dir = '/tmp/elilo-xcat-mockbuild'; my $mock_cfg = ''; my $mock_uniqueext = ''; my $result_dir = "$repo_root/build-output/list3/elilo-xcat"; my $log_dir = "$repo_root/build-logs/list3/elilo-xcat"; -my $skip_install = 0; my $build_timestamp; GetOptions( - 'source-url=s' => \$source_url, 'source-file=s' => \$source_file, 'work-dir=s' => \$work_dir, 'mock-cfg=s' => \$mock_cfg, 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); die "Run as root (current uid=$>)\n" if $> != 0; die "Missing spec file: $spec_file\n" if !-f $spec_file; -for my $bin (qw(wget mock rpmbuild rpm dnf file bash grep)) { +for my $bin (qw(mock rpmbuild rpm dnf file bash grep)) { run("command -v " . sh_quote($bin) . " >/dev/null 2>&1"); } @@ -80,9 +76,7 @@ print "result_dir: $result_dir\n"; print "log_dir: $log_dir\n"; print "mock_cfg: $mock_cfg\n"; print "mock_uniqueext: " . ($mock_uniqueext ne '' ? $mock_uniqueext : '(none)') . "\n"; -print "source_url: $source_url\n"; print "source_file:$source_file\n"; -print "skip_install: $skip_install\n"; make_path($result_dir); make_path($log_dir); @@ -90,10 +84,25 @@ make_path($log_dir); print_step("Mock config check"); run("mock -r " . sh_quote($mock_cfg) . $mock_uniqueext_opt . " --print-root-path >/dev/null"); -print_step("Download upstream source"); -run("wget --spider " . sh_quote($source_url)); -run("wget -O " . sh_quote($source_path) . " " . sh_quote($source_url)); -normalize_source_archive($source_path, $version, $work_dir); +print_step("Verify tracked source archive"); +# Upstream source (documented for provenance; NOT fetched at build time -- see below): +# https://downloads.sourceforge.net/project/elilo/elilo/elilo-3.14/elilo-3.14-all.tar.gz +# Source0 (elilo--source.tar.gz) is tracked in the repo, already normalized to an elilo/ +# top-level tree, and consumed directly by mock (--sources $pkg_dir below). There is nothing to +# download: the old fetch re-derived this SAME tracked file and rewrote it IN PLACE. Because the +# checkout is on a shared (NFS) mount that BOTH arch build hosts (x86 + ppc) build against at the +# same time, that in-place rewrite raced the other host's concurrent elilo build -- it could read +# the file mid-write and get a truncated archive ("missing elilo top-level tree" failures). We now +# only READ the tracked file, so concurrent builds can never race on it. Fail loudly (do NOT +# silently re-fetch) if the checkout is missing/broken -- that is repo corruption, not a fetch miss. +die "Tracked elilo source missing: $source_path (incomplete checkout?)\n" if !-f $source_path; +my $top = capture( + "tar -tzf " . sh_quote($source_path) . + " 2>/dev/null | grep -E '^(\\./)?elilo/' | head -n1 || true" +); +die "Tracked elilo source is not normalized (no elilo/ top-level tree): $source_path\n" + if $top eq ''; +print "Using tracked normalized source archive (read-only, no fetch, no shared write): $source_path\n"; print_step("Verify spec assets"); for my $asset (@spec_assets) { @@ -195,37 +204,6 @@ for my $log (qw(build.log root.log state.log hw_info.log installed_pkgs.log)) { or die "Failed to copy $src to $log_dir: $!\n"; } -if (!$skip_install) { - print_step("Install RPM and run smoke tests"); - run("dnf -y install " . sh_quote($main_rpm)); - - my $efi_file = '/tftpboot/xcat/elilo-x64.efi'; - die "Missing installed EFI binary: $efi_file\n" if !-f $efi_file; - - my $file_log = "$log_dir/smoke-file.log"; - my $qf_log = "$log_dir/smoke-rpm-qf.log"; - my $rc_file = run_capture_rc("file $efi_file", $file_log); - my $rc_qf = run_capture_rc("rpm -qf $efi_file", $qf_log); - - die "Smoke check failed: file returned $rc_file\n" if $rc_file != 0; - die "Smoke check failed: rpm -qf returned $rc_qf\n" if $rc_qf != 0; - - my $file_out = slurp($file_log); - my $qf_out = slurp($qf_log); - - die "EFI file signature check failed:\n$file_out\n" - if $file_out !~ /(EFI application|PE32\+ executable)/i; - die "Installed file is not owned by elilo-xcat:\n$qf_out\n" - if $qf_out !~ /^elilo-xcat-/m; - - my $summary = "$log_dir/smoke-summary.txt"; - open my $sfh, '>', $summary or die "Cannot write $summary: $!\n"; - print {$sfh} "efi_file=$efi_file\n"; - print {$sfh} "rc_file=$rc_file\n"; - print {$sfh} "rc_qf=$rc_qf\n"; - close $sfh; -} - print_step("Completed"); print "Main RPM: $main_rpm\n"; print "Artifacts: $result_dir\n"; @@ -235,14 +213,12 @@ exit 0; sub usage { return <<"USAGE"; Usage: $0 [options] - --source-url URL Upstream tarball URL (default: $source_url) --source-file FILE Source filename stored in elilo/ (default: inferred from spec version) --work-dir PATH Temporary work dir (default: $work_dir) --mock-cfg NAME Mock config (default: +epel-10-) --mock-uniqueext TXT Optional mock --uniqueext suffix to isolate concurrent builds --result-dir PATH Output RPM/SRPM directory (default: $result_dir) --log-dir PATH Log directory (default: $log_dir) - --skip-install Skip dnf install + smoke tests --build-timestamp EPOCH Unix timestamp for SOURCE_DATE_EPOCH (deterministic builds) USAGE } @@ -284,49 +260,6 @@ sub parse_spec { return ($version, @assets); } -sub normalize_source_archive { - my ($archive, $version, $work_base) = @_; - - my $has_elilo = capture( - "tar -tzf " . sh_quote($archive) . - " | grep -E '^(\\./)?elilo/' | head -n1 || true" - ); - return if $has_elilo ne ''; - - my $nested = capture( - "tar -tzf " . sh_quote($archive) . - " | grep -E '^(\\./)?elilo-$version-source\\.tar\\.gz\$' | head -n1 || true" - ); - die "Downloaded archive does not contain elilo source payload: $archive\n" - if $nested eq ''; - - my $normalize_dir = "$work_base/source-normalize"; - remove_tree($normalize_dir) if -d $normalize_dir; - make_path($normalize_dir); - - run( - "tar -xzf " . sh_quote($archive) . - " -C " . sh_quote($normalize_dir) . - " " . sh_quote($nested) - ); - - my $nested_rel = $nested; - $nested_rel =~ s{^\./}{}; - my $nested_path = "$normalize_dir/$nested_rel"; - die "Failed to extract nested source archive: $nested_path\n" - if !-f $nested_path; - - copy($nested_path, $archive) - or die "Failed to normalize source archive $archive: $!\n"; - - my $recheck = capture( - "tar -tzf " . sh_quote($archive) . - " | grep -E '^(\\./)?elilo/' | head -n1 || true" - ); - die "Normalized source archive still missing elilo top-level tree: $archive\n" - if $recheck eq ''; -} - sub print_step { my ($msg) = @_; print "\n== $msg ==\n"; diff --git a/goconserver/gomod/README.md b/goconserver/gomod/README.md index ce0b7a6..6f84eb3 100644 --- a/goconserver/gomod/README.md +++ b/goconserver/gomod/README.md @@ -1,14 +1,14 @@ # Pinned `go.mod` / `go.sum` for the goconserver build -These pin the Go module graph for goconserver at the commit built by `../sbuild.pl` -(`REF=6166fe5ec1c5b3c20475e322a9f0e8e93c87e45f`). `sbuild.pl` overlays them into the freshly -cloned upstream tree and compiles with `GOFLAGS=-mod=mod`, so modules are downloaded from the Go -proxy but **pinned and integrity-checked by `go.sum`** — the build is reproducible, with **no -`go mod tidy`** at build time (which would float transitive versions from the network). +These pin the Go module graph for goconserver at the commit built by `../sbuild.pl` (Ubuntu) and +`../mockbuild.pl` (EL): `REF=6166fe5ec1c5b3c20475e322a9f0e8e93c87e45f`. Both builders overlay them +into the freshly cloned upstream tree and compile with `GOFLAGS=-mod=mod`, so modules are downloaded +from the Go proxy but **pinned and integrity-checked by `go.sum`** — the build is reproducible, with +**no `go mod tidy`** at build time (which would float transitive versions from the network). -Generated with the same pinned toolchain the build uses (`GO_PIN` in `../sbuild.pl`, currently -go 1.25.12), so `go.mod`'s `go` directive matches — do not copy the EL branch's `gomod/` (it is -generated with a different Go and pins slightly different minor versions). +The `go` directive (currently go 1.25.12) is the floor every builder must meet: the `GO_PIN` +toolchain of `../sbuild.pl` and the `golang` of the EL10 mock chroot. Regenerate with the lowest of +them, so neither build is rejected. ## Regenerate (when bumping `REF` or `GO_PIN`, or a dependency) diff --git a/goconserver/mockbuild.pl b/goconserver/mockbuild.pl old mode 100644 new mode 100755 index 5daae09..15195c1 --- a/goconserver/mockbuild.pl +++ b/goconserver/mockbuild.pl @@ -18,10 +18,13 @@ my $target_arch = ''; my $mock_uniqueext = ''; my $result_dir = "$repo_root/build-output/list5/goconserver"; my $log_dir = "$repo_root/build-logs/list5/goconserver"; -my $skip_install = 0; my $version = '0.3.3'; my $go_repo = 'https://github.com/xcat2/goconserver.git'; -my $go_ref = 'master'; +# Immutable pin: goconserver 0.3.3 is unreleased (newest tag v0.3.2) so it lives only on master. +# mockbuild-all.pl passes --go-ref with the canonical pin; this default keeps standalone runs +# reproducible too. The committed go.mod/go.sum (no vendor tree) correspond to THIS SHA. +my $go_ref = '6166fe5ec1c5b3c20475e322a9f0e8e93c87e45f'; +my $release_suffix = ''; # CD Release bump (".snap."); passed by mockbuild-all.pl my $build_timestamp; GetOptions( @@ -31,19 +34,15 @@ GetOptions( 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, 'version=s' => \$version, 'go-repo=s' => \$go_repo, 'go-ref=s' => \$go_ref, + 'release-suffix=s' => \$release_suffix, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); die "Run as root (current uid=$>)\n" if $> != 0; -for my $bin (qw(go git rpmbuild rpm)) { - run("command -v " . sh_quote($bin) . " >/dev/null 2>&1"); -} - my $arch = capture('uname -m'); if (!$mock_cfg) { my $os_id = capture(q{bash -lc 'source /etc/os-release; echo $ID'}); @@ -52,14 +51,22 @@ if (!$mock_cfg) { my ($rel) = $mock_cfg =~ /-(\d+)-/; $rel //= '10'; -# Arch of the rpm to produce: the host arch unless --mock-cfg is a forcearch (cross) config, -# e.g. rocky-10-riscv64-xcat built on x86_64 (see BUILD.md "riscv64"). goconserver is not built -# in the chroot: go cross-compiles on the host and rpmbuild --target packages the result. + +# --target-arch names the arch of the rpm to produce. It differs from the host arch only for a +# forcearch target (rocky-10-riscv64-xcat on an x86_64 host; see BUILD.md "riscv64"). $target_arch = $arch if $target_arch eq ''; my %goarch = (x86_64 => 'amd64', aarch64 => 'arm64', ppc64le => 'ppc64le', s390x => 's390x', riscv64 => 'riscv64'); my $cross = $target_arch ne $arch; die "No GOARCH known for target arch $target_arch\n" if $cross && !exists $goarch{$target_arch}; +# For the host arch the Go compile happens INSIDE the mock chroot (BuildRequires: golang), so the +# host only fetches the pinned source and drives mock. A forcearch chroot would run that compile +# under qemu, so the cross build instead cross-compiles on the host and packages the result with +# rpmbuild --target. +for my $bin (qw(git rpm), ($cross ? qw(go rpmbuild) : qw(mock))) { + run("command -v " . sh_quote($bin) . " >/dev/null 2>&1"); +} + my $SOURCE_DATE_EPOCH; $SOURCE_DATE_EPOCH = $build_timestamp if defined $build_timestamp; if (!$SOURCE_DATE_EPOCH && -f "$repo_root/Gitepoch") { @@ -78,19 +85,25 @@ unless ($SOURCE_DATE_EPOCH && $SOURCE_DATE_EPOCH =~ /^\d+$/) { $SOURCE_DATE_EPOCH = time() unless $SOURCE_DATE_EPOCH =~ /^\d+$/; $ENV{SOURCE_DATE_EPOCH} = $SOURCE_DATE_EPOCH; +# goconserver is a CGO-free static Go binary. el8/el9 chroots ship a Go too old to build 0.3.3, so +# always COMPILE in the el10 chroot for this arch (regardless of the target EL), then ship the static +# binary to every EL repo. The Release still carries the target's dist tag (4.el$rel) so each EL repo +# gets a correctly-named, byte-identical rpm. +(my $build_cfg = $mock_cfg) =~ s/-\d+-/-10-/; + print_step("Configuration"); -print "repo_root: $repo_root\n"; -print "pkg_dir: $pkg_dir\n"; -print "work_dir: $work_dir\n"; -print "result_dir: $result_dir\n"; -print "log_dir: $log_dir\n"; -print "mock_cfg: $mock_cfg\n"; -print "arch: $arch\n"; -print "target_arch:$target_arch" . ($cross ? " (GOARCH=$goarch{$target_arch}, rpmbuild --target)" : '') . "\n"; -print "version: $version\n"; -print "go_repo: $go_repo\n"; -print "go_ref: $go_ref\n"; -print "skip_install: $skip_install\n"; +print "repo_root: $repo_root\n"; +print "pkg_dir: $pkg_dir\n"; +print "work_dir: $work_dir\n"; +print "result_dir: $result_dir\n"; +print "log_dir: $log_dir\n"; +print "mock_cfg: $mock_cfg (target dist tag: el$rel)\n"; +print "build_cfg: $build_cfg (el10 -- portable static build for arch $arch)\n" if !$cross; +print "arch: $arch\n"; +print "target_arch: $target_arch" . ($cross ? " (GOARCH=$goarch{$target_arch}, rpmbuild --target)" : '') . "\n"; +print "version: $version\n"; +print "go_ref: $go_ref\n"; +print "release_suffix: " . ($release_suffix ne '' ? $release_suffix : '(none)') . "\n"; make_path($result_dir); make_path($log_dir); @@ -99,77 +112,33 @@ print_step("Stage build environment"); remove_tree($work_dir) if -d $work_dir; make_path($work_dir); -# Unique per run (nested under the run/target-scoped --work-dir) so concurrent builds -- e.g. -# parallel EL targets on one host -- don't wipe each other. (Was a shared -# /var/tmp/xcat-rpmbuild-goconserver, which collided under parallelism.) -my $rpmbuild_top = "$work_dir/rpmbuild"; -remove_tree($rpmbuild_top) if -d $rpmbuild_top; -for my $d (qw(BUILD BUILDROOT RPMS SOURCES SPECS SRPMS)) { - make_path("$rpmbuild_top/$d"); -} - +# --- Fetch the pinned goconserver source (immutable SHA -> reproducible) --- print_step("Clone goconserver source"); my $src_dir = "$work_dir/goconserver-src"; -run("git clone --depth 1 --branch " . sh_quote($go_ref) . " " . - sh_quote($go_repo) . " " . sh_quote($src_dir) . - " >" . sh_quote("$log_dir/git-clone.log") . " 2>&1"); +my $clone_log = sh_quote("$log_dir/git-clone.log"); +run("git init -q " . sh_quote($src_dir) . " >$clone_log 2>&1"); +run("git -C " . sh_quote($src_dir) . " remote add origin " . sh_quote($go_repo) . " >>$clone_log 2>&1"); +run("git -C " . sh_quote($src_dir) . " fetch --depth 1 origin " . sh_quote($go_ref) . " >>$clone_log 2>&1"); +run("git -C " . sh_quote($src_dir) . " checkout -q FETCH_HEAD >>$clone_log 2>&1"); -# etcd storage backend has broken deps with modern Go modules; -# xCAT only uses file storage, so remove etcd before building. +# etcd storage backend has broken deps with modern Go modules; xCAT only uses file storage. unlink "$src_dir/storage/etcd.go"; remove_tree("$src_dir/storage/etcd") if -d "$src_dir/storage/etcd"; +remove_tree("$src_dir/.git") if -d "$src_dir/.git"; # keep the SRPM tarball clean + reproducible -print_step("Initialize Go modules"); -$ENV{GOPATH} = "$work_dir/gopath"; -$ENV{GOCACHE} = "$work_dir/gocache"; -$ENV{GOMODCACHE} = "$work_dir/gomodcache"; -$ENV{CGO_ENABLED} = '0'; -$ENV{GOARCH} = $goarch{$target_arch} if $cross; +# --- Overlay the committed, PINNED go.mod/go.sum (no vendored tree) --- +# go.mod already replaces the archived github.com/kr/pty with creack/pty (see gomod/README.md), and +# go.sum integrity-checks every module. The in-chroot build downloads the modules from the Go proxy +# (mock networking is enabled) but is reproducible because go.sum pins them -- no `go mod tidy`, and +# no 400k-line vendor tree committed. +print_step("Overlay pinned go.mod/go.sum"); +my $gomod_dir = "$pkg_dir/gomod"; +die "pinned go.mod/go.sum missing under $gomod_dir (regenerate per gomod/README.md)\n" + unless -f "$gomod_dir/go.mod" && -f "$gomod_dir/go.sum"; +copy("$gomod_dir/go.mod", "$src_dir/go.mod") or die "copy go.mod: $!\n"; +copy("$gomod_dir/go.sum", "$src_dir/go.sum") or die "copy go.sum: $!\n"; -# The archived github.com/kr/pty sets SysProcAttr.Ctty to the parent-side fd, -# which modern Go's os/exec rejects with "Setctty set but Ctty not valid in -# child". Replace it with the API-identical maintained fork creack/pty. -run("cd " . sh_quote($src_dir) . " && " . - "go mod init github.com/xcat2/goconserver && " . - "go mod edit -replace github.com/kr/pty=github.com/creack/pty\@v1.1.21 && " . - "go mod tidy" . - " >" . sh_quote("$log_dir/go-mod.log") . " 2>&1"); - -print_step("Build goconserver binaries"); -my $go_build_dir = "$work_dir/bin"; -make_path($go_build_dir); - -my $ldflags = "-X main.Version=$version"; -# rpm's brp-strip on the host cannot strip a foreign-arch ELF, so let the Go linker do it. -$ldflags .= ' -s -w' if $cross; - -run("cd " . sh_quote($src_dir) . " && " . - "go build -trimpath -buildvcs=false -ldflags " . sh_quote($ldflags) . - " -o " . sh_quote("$go_build_dir/goconserver") . " goconserver.go" . - " >" . sh_quote("$log_dir/go-build-server.log") . " 2>&1"); - -run("cd " . sh_quote($src_dir) . " && " . - "go build -trimpath -buildvcs=false -ldflags " . sh_quote($ldflags) . - " -o " . sh_quote("$go_build_dir/congo") . " cmd/congo.go" . - " >" . sh_quote("$log_dir/go-build-client.log") . " 2>&1"); - -die "goconserver binary not built\n" if !-x "$go_build_dir/goconserver"; -die "congo binary not built\n" if !-x "$go_build_dir/congo"; - -print_step("Create source tarball"); -my $payload_dir = "$work_dir/goconserver-$version"; -make_path("$payload_dir/usr/bin"); -make_path("$payload_dir/usr/lib/systemd/system"); -make_path("$payload_dir/etc/goconserver"); - -copy("$go_build_dir/goconserver", "$payload_dir/usr/bin/goconserver") - or die "copy goconserver: $!\n"; -copy("$go_build_dir/congo", "$payload_dir/usr/bin/congo") - or die "copy congo: $!\n"; -chmod 0755, "$payload_dir/usr/bin/goconserver"; -chmod 0755, "$payload_dir/usr/bin/congo"; - -write_file("$payload_dir/usr/lib/systemd/system/goconserver.service", <<'SERVICE'); +my $service_unit = <<'SERVICE'; [Unit] Description=goconserver console server After=network.target @@ -184,54 +153,86 @@ StateDirectory=goconserver WantedBy=multi-user.target SERVICE -write_file("$payload_dir/etc/goconserver/server.conf", <<'CONF'); -[server] -host = 0.0.0.0 -port = 12430 -console_port = 12431 -log_file = /var/log/goconserver/server.log -log_timestamp = true -log_level = info +# The goconserver binary parses server.conf as YAML. Ship a VALID YAML default (the old INI-style +# [server] block is read by the YAML parser as a sequence -> `panic: cannot unmarshal !!seq` at +# startup -> systemd rate-limits the service to `failed`). On an xCAT MN, xCAT::Goconserver.pm +# overwrites this with a cert-enabled config; this default only has to PARSE and start. +my $server_conf = <<'CONF'; +global: + host: 0.0.0.0 + logfile: /var/log/goconserver/server.log +api: + port: 12429 +console: + datadir: /var/lib/goconserver/ + port: 12430 + log_timestamp: true CONF -my $tarball = "$rpmbuild_top/SOURCES/goconserver-$version.tar.gz"; -run("tar --sort=name --owner=0 --group=0 --mtime=\@$SOURCE_DATE_EPOCH" . - " -C " . sh_quote($work_dir) . " -czf " . sh_quote($tarball) . - " goconserver-$version"); +if ($cross) { + cross_build_and_package(); + exit 0; +} -print_step("Create spec and build RPM"); -# rpm refuses 'BuildArch: ' on this host, so a cross build gets its arch from -# rpmbuild --target instead. -my $buildarch_line = $cross ? '' : "BuildArch: $arch"; -my $spec_content = <<"SPEC"; + +# --- Assemble SRPM sources: the source tree (go.mod/go.sum, no vendor) + the xcat-authored unit + config --- +print_step("Assemble SRPM sources"); +my $srctop = "goconserver-$version"; +my $staged = "$work_dir/$srctop"; +remove_tree($staged) if -d $staged; +run("cp -a " . sh_quote($src_dir) . " " . sh_quote($staged)); +my $sources_dir = "$work_dir/sources"; +make_path($sources_dir); +my $tarball = "$sources_dir/goconserver-$version.tar.gz"; +run("tar --sort=name --owner=0 --group=0 --mtime=\@$SOURCE_DATE_EPOCH" . + " -C " . sh_quote($work_dir) . " -czf " . sh_quote($tarball) . " " . sh_quote($srctop)); + +write_file("$sources_dir/goconserver.service", $service_unit); +write_file("$sources_dir/server.conf", $server_conf); + +# --- Spec: the Go compile runs in %build INSIDE the chroot; modules fetched from the proxy, pinned by go.sum --- +print_step("Write spec"); +my $spec_file = "$work_dir/goconserver.spec"; +write_file($spec_file, <<"SPEC"); +# Go binaries carry no useful DWARF debugsource; the empty debuginfo subpackage otherwise fails +# packaging ("Empty %files debugsourcefiles.list"). Disable it. +%global debug_package %{nil} Name: goconserver Version: $version -Release: 3.el$rel +Release: 4.el$rel$release_suffix Summary: Console server written in Go for xCAT License: EPL-1.0 URL: https://github.com/xcat2/goconserver -$buildarch_line +BuildArch: $arch Source0: goconserver-%{version}.tar.gz +Source1: goconserver.service +Source2: server.conf + +BuildRequires: golang %description goconserver is a scalable console server written in Go. It provides console logging and management for xCAT cluster nodes. %prep -%setup -n goconserver-%{version} +%setup -q -n goconserver-%{version} + +%build +# Compile in-chroot. Modules are downloaded from the Go proxy at build time (mock networking is on) +# but PINNED + integrity-checked by the committed go.sum, so the build is reproducible without a +# vendored tree. GOTOOLCHAIN=local pins the chroot's Go (never auto-downloads a toolchain). +export GOFLAGS=-mod=mod GOTOOLCHAIN=local CGO_ENABLED=0 +export GOCACHE=%{_builddir}/.gocache GOPATH=%{_builddir}/.gopath GOMODCACHE=%{_builddir}/.gomodcache +go build -trimpath -buildvcs=false -ldflags "-X main.Version=%{version}" -o goconserver goconserver.go +go build -trimpath -buildvcs=false -ldflags "-X main.Version=%{version}" -o congo cmd/congo.go %install -mkdir -p %{buildroot}/usr/bin -mkdir -p %{buildroot}/usr/lib/systemd/system -mkdir -p %{buildroot}/etc/goconserver -mkdir -p %{buildroot}/var/log/goconserver -mkdir -p %{buildroot}/var/lib/goconserver - -install -m 755 usr/bin/goconserver %{buildroot}/usr/bin/goconserver -install -m 755 usr/bin/congo %{buildroot}/usr/bin/congo -install -m 644 usr/lib/systemd/system/goconserver.service %{buildroot}/usr/lib/systemd/system/goconserver.service -install -m 644 etc/goconserver/server.conf %{buildroot}/etc/goconserver/server.conf +install -Dm0755 goconserver %{buildroot}/usr/bin/goconserver +install -Dm0755 congo %{buildroot}/usr/bin/congo +install -Dm0644 %{SOURCE1} %{buildroot}/usr/lib/systemd/system/goconserver.service +install -Dm0644 %{SOURCE2} %{buildroot}/etc/goconserver/server.conf +mkdir -p %{buildroot}/var/log/goconserver %{buildroot}/var/lib/goconserver %files /usr/bin/goconserver @@ -242,95 +243,236 @@ install -m 644 etc/goconserver/server.conf %{buildroot}/etc/goconserver/server.c %dir /var/lib/goconserver %changelog -* Mon Jun 08 2026 xCAT EL10 build - 0.3.3-2.el10 -- Replace archived github.com/kr/pty with github.com/creack/pty to fix - "Setctty set but Ctty not valid in child" console fork failure on modern Go. +* Mon Aug 10 2026 xCAT build - $version-4.el$rel +- Build inside a mock chroot (no host build). Modules are downloaded at build time but pinned + + integrity-checked by a committed go.sum (no `go mod tidy`, no vendored tree). Compiled in the + el10 chroot for the arch and shipped to every EL repo (CGO-free static binary). +- Ship /etc/goconserver/server.conf as YAML (the format the goconserver binary parses). +- Replace archived github.com/kr/pty with github.com/creack/pty (console fork on modern Go). SPEC -my $spec_file = "$rpmbuild_top/SPECS/goconserver.spec"; -write_file($spec_file, $spec_content); +# --- Build in the el10 chroot for this arch --- +my $mock_uniqueext_opt = $mock_uniqueext ne '' ? ' --uniqueext ' . sh_quote($mock_uniqueext) : ''; +print_step("Mock config check"); +run("mock -r " . sh_quote($build_cfg) . $mock_uniqueext_opt . " --print-root-path >/dev/null"); +my $det_cfg = create_deterministic_mock_cfg($build_cfg, $SOURCE_DATE_EPOCH, $work_dir); -run( - "rpmbuild --define " . sh_quote("_topdir $rpmbuild_top") . +print_step("Build SRPM with mock"); +my $srpm_out = "$work_dir/srpm"; +make_path($srpm_out); +run("mock -r " . sh_quote($det_cfg) . $mock_uniqueext_opt . + " --buildsrpm --spec " . sh_quote($spec_file) . + " --sources " . sh_quote($sources_dir) . " --define " . sh_quote("use_source_date_epoch_as_buildtime 1") . " --define " . sh_quote("clamp_mtime_to_source_date_epoch 1") . " --define " . sh_quote("_buildhost xcat-build") . - ($cross ? " --target " . sh_quote($target_arch) : '') . - " -ba " . sh_quote($spec_file) . - " >" . sh_quote("$log_dir/rpmbuild.log") . " 2>&1" -); + " --resultdir " . sh_quote($srpm_out) . + " >" . sh_quote("$log_dir/mock-buildsrpm.log") . " 2>&1"); +my @srpms = sort glob("$srpm_out/goconserver-*.src.rpm"); +die "SRPM not generated in $srpm_out\n" if !@srpms; +my $srpm = $srpms[-1]; +print "SRPM: $srpm\n"; + +print_step("Rebuild RPM with mock (offline, in-chroot go build)"); +my $rpm_out = "$work_dir/rpm"; +make_path($rpm_out); +run("mock -r " . sh_quote($det_cfg) . $mock_uniqueext_opt . + " --rebuild " . sh_quote($srpm) . + " --define " . sh_quote("use_source_date_epoch_as_buildtime 1") . + " --define " . sh_quote("clamp_mtime_to_source_date_epoch 1") . + " --define " . sh_quote("_buildhost xcat-build") . + " --resultdir " . sh_quote($rpm_out) . + " >" . sh_quote("$log_dir/mock-rebuild.log") . " 2>&1"); print_step("Collect results"); -for my $rpm (glob("$rpmbuild_top/RPMS/*/*.rpm"), glob("$rpmbuild_top/SRPMS/*.rpm")) { +my @arch_rpms = sort grep { !/\.src\.rpm$/ } glob("$rpm_out/goconserver-*.$arch.rpm"); +die "No goconserver $arch rpm generated in $rpm_out\n" if !@arch_rpms; +for my $rpm (@arch_rpms, glob("$rpm_out/*.src.rpm")) { my $dest = "$result_dir/" . basename($rpm); copy($rpm, $dest) or die "Failed to copy $rpm to $dest: $!\n"; print "Copied: $dest\n"; } - -if (!$skip_install && $cross) { - # A cross-built rpm cannot be installed on this host. Unpack it and run the static Go - # binaries through the binfmt_misc handler (qemu-user-static) that the forcearch mock - # builds of the other deps need anyway. - print_step("Smoke test the $target_arch binaries (binfmt)"); - my @built = glob("$rpmbuild_top/RPMS/$target_arch/goconserver-*.rpm"); - die "No $target_arch RPM found\n" if !@built; - my $smoke_root = "$work_dir/smoke-root"; - remove_tree($smoke_root) if -d $smoke_root; - make_path($smoke_root); - run("cd " . sh_quote($smoke_root) . " && rpm2cpio " . sh_quote($built[0]) . " | cpio -idm --quiet" . - " >" . sh_quote("$log_dir/smoke-unpack.log") . " 2>&1"); - for my $bin (qw(goconserver congo)) { - my $path = "$smoke_root/usr/bin/$bin"; - die "Missing $path in the $target_arch rpm\n" if !-x $path; - my $rc = run_rc(sh_quote($path) . " -h >" . sh_quote("$log_dir/smoke-$bin.log") . " 2>&1"); - die "$bin -h failed (rc=$rc): running a $target_arch binary on this $arch host needs the" - . " qemu-user-static binfmt handler (or pass --skip-install)\n" if $rc > 1; - } - print "Smoke tests passed ($target_arch binaries ran through binfmt).\n"; +for my $log (qw(build.log root.log state.log)) { + my $s = "$rpm_out/$log"; + copy($s, "$log_dir/mock-$log") if -f $s; } -elsif (!$skip_install) { - print_step("Install and smoke test"); - my @built = glob("$rpmbuild_top/RPMS/$arch/goconserver-*.rpm"); - die "No arch RPM found\n" if !@built; - my $main_rpm = $built[0]; - run("dnf -y install " . sh_quote($main_rpm) . - " >" . sh_quote("$log_dir/dnf-install.log") . " 2>&1"); - - die "Missing /usr/bin/goconserver\n" if !-x '/usr/bin/goconserver'; - die "Missing /usr/bin/congo\n" if !-x '/usr/bin/congo'; - - my $rc_help = run_rc("goconserver -h >" . sh_quote("$log_dir/smoke-help.log") . " 2>&1"); - die "goconserver -h failed (rc=$rc_help)\n" if $rc_help > 1; - - my $rc_congo = run_rc("congo -h >" . sh_quote("$log_dir/smoke-congo.log") . " 2>&1"); - die "congo -h failed (rc=$rc_congo)\n" if $rc_congo > 1; - - print "Smoke tests passed.\n"; -} +# Reclaim goconserver's own build chroot. mockbuild-all's scrub keys on the TARGET cfg (el$rel), not +# the el10 build cfg used here, so scrub it ourselves to avoid leaking /var/lib/mock. Best-effort. +# Scrub BOTH the chroot and its bootstrap: the el10 build cfg is bootstrap-image based, so +# --scrub=chroot alone leaves the ~190 MiB -bootstrap- tree behind (it accumulated +# one per target per run in /var/lib/mock -- the disk leak of VersatusHPC/xcat-core#51). +system("mock -r " . sh_quote($build_cfg) . $mock_uniqueext_opt . + " --scrub=chroot --scrub=bootstrap >" . sh_quote("$log_dir/mock-scrub.log") . " 2>&1"); print_step("Completed"); print "Results in: $result_dir\n"; exit 0; + +# A forcearch mock chroot runs every command through qemu, so an in-chroot Go compile would run the +# whole toolchain emulated. For a foreign target arch the build cross-compiles on the host with the +# same pinned go.mod/go.sum and lets rpmbuild --target name the arch. See BUILD.md ("riscv64"). +sub cross_build_and_package { + my $rpmbuild_top = "$work_dir/rpmbuild"; + remove_tree($rpmbuild_top) if -d $rpmbuild_top; + make_path("$rpmbuild_top/$_") for qw(BUILD BUILDROOT RPMS SOURCES SPECS SRPMS); + + print_step("Cross-compile goconserver for $target_arch"); + local $ENV{GOPATH} = "$work_dir/gopath"; + local $ENV{GOCACHE} = "$work_dir/gocache"; + local $ENV{GOMODCACHE} = "$work_dir/gomodcache"; + local $ENV{CGO_ENABLED} = '0'; + local $ENV{GOFLAGS} = '-mod=mod'; + local $ENV{GOTOOLCHAIN} = 'local'; + local $ENV{GOARCH} = $goarch{$target_arch}; + + my $bin_dir = "$work_dir/bin"; + make_path($bin_dir); + # rpm's brp-strip cannot strip a foreign-arch ELF, so the Go linker strips instead. + my $ldflags = "-X main.Version=$version -s -w"; + for my $target (['goconserver', 'goconserver.go'], ['congo', 'cmd/congo.go']) { + my ($out, $main) = @{$target}; + run("cd " . sh_quote($src_dir) . " && go build -trimpath -buildvcs=false -ldflags " . + sh_quote($ldflags) . " -o " . sh_quote("$bin_dir/$out") . " " . sh_quote($main) . + " >" . sh_quote("$log_dir/go-build-$out.log") . " 2>&1"); + die "$out binary not built\n" if !-x "$bin_dir/$out"; + } + + print_step("Assemble SRPM sources"); + my $srctop = "goconserver-$version"; + my $payload = "$work_dir/$srctop"; + remove_tree($payload) if -d $payload; + make_path("$payload/usr/bin", "$payload/usr/lib/systemd/system", "$payload/etc/goconserver"); + for my $out (qw(goconserver congo)) { + copy("$bin_dir/$out", "$payload/usr/bin/$out") or die "copy $out: $!\n"; + chmod 0755, "$payload/usr/bin/$out"; + } + write_file("$payload/usr/lib/systemd/system/goconserver.service", $service_unit); + write_file("$payload/etc/goconserver/server.conf", $server_conf); + run("tar --sort=name --owner=0 --group=0 --mtime=\@$SOURCE_DATE_EPOCH" . + " -C " . sh_quote($work_dir) . " -czf " . sh_quote("$rpmbuild_top/SOURCES/$srctop.tar.gz") . + " " . sh_quote($srctop)); + + print_step("Write spec"); + my $spec_file = "$rpmbuild_top/SPECS/goconserver.spec"; + # The payload is already compiled, so there is no %build. rpm refuses a foreign 'BuildArch:' + # here; rpmbuild --target below sets the arch. + write_file($spec_file, <<"SPEC"); +# Go binaries carry no useful DWARF debugsource; the empty debuginfo subpackage otherwise fails +# packaging ("Empty %files debugsourcefiles.list"). Disable it. +%global debug_package %{nil} +Name: goconserver +Version: $version +Release: 4.el$rel$release_suffix +Summary: Console server written in Go for xCAT +License: EPL-1.0 +URL: https://github.com/xcat2/goconserver + +Source0: goconserver-%{version}.tar.gz + +%description +goconserver is a scalable console server written in Go. It provides +console logging and management for xCAT cluster nodes. + +%prep +%setup -q -n goconserver-%{version} + +%install +install -Dm0755 usr/bin/goconserver %{buildroot}/usr/bin/goconserver +install -Dm0755 usr/bin/congo %{buildroot}/usr/bin/congo +install -Dm0644 usr/lib/systemd/system/goconserver.service %{buildroot}/usr/lib/systemd/system/goconserver.service +install -Dm0644 etc/goconserver/server.conf %{buildroot}/etc/goconserver/server.conf +mkdir -p %{buildroot}/var/log/goconserver %{buildroot}/var/lib/goconserver + +%files +/usr/bin/goconserver +/usr/bin/congo +/usr/lib/systemd/system/goconserver.service +%config(noreplace) /etc/goconserver/server.conf +%dir /var/log/goconserver +%dir /var/lib/goconserver + +%changelog +* Mon Aug 10 2026 xCAT build - $version-4.el$rel +- Cross-compile on the build host (GOARCH=$goarch{$target_arch}) with the committed go.mod/go.sum, + and package with rpmbuild --target $target_arch. The forcearch chroot would run the Go toolchain + under qemu. +- Ship /etc/goconserver/server.conf as YAML (the format the goconserver binary parses). +- Replace archived github.com/kr/pty with github.com/creack/pty (console fork on modern Go). +SPEC + + print_step("Build RPM with rpmbuild --target $target_arch"); + run("rpmbuild --define " . sh_quote("_topdir $rpmbuild_top") . + " --define " . sh_quote("use_source_date_epoch_as_buildtime 1") . + " --define " . sh_quote("clamp_mtime_to_source_date_epoch 1") . + " --define " . sh_quote("_buildhost xcat-build") . + " --target " . sh_quote($target_arch) . + " -ba " . sh_quote($spec_file) . + " >" . sh_quote("$log_dir/rpmbuild.log") . " 2>&1"); + + print_step("Collect results"); + my @arch_rpms = sort glob("$rpmbuild_top/RPMS/$target_arch/goconserver-*.rpm"); + die "No goconserver $target_arch rpm generated in $rpmbuild_top/RPMS\n" if !@arch_rpms; + for my $rpm (@arch_rpms, glob("$rpmbuild_top/SRPMS/*.src.rpm")) { + my $dest = "$result_dir/" . basename($rpm); + copy($rpm, $dest) or die "Failed to copy $rpm to $dest: $!\n"; + print "Copied: $dest\n"; + } + + # The rpm cannot be installed on this host, and nothing else runs the cross-built binaries. + # Unpack it and run them through the binfmt_misc handler the forcearch chroot needs anyway. + print_step("Smoke test the $target_arch binaries (binfmt)"); + my $smoke_root = "$work_dir/smoke-root"; + remove_tree($smoke_root) if -d $smoke_root; + make_path($smoke_root); + run("cd " . sh_quote($smoke_root) . " && rpm2cpio " . sh_quote($arch_rpms[0]) . + " | cpio -idm --quiet >" . sh_quote("$log_dir/smoke-unpack.log") . " 2>&1"); + for my $out (qw(goconserver congo)) { + my $path = "$smoke_root/usr/bin/$out"; + die "Missing $path in the $target_arch rpm\n" if !-x $path; + # -h exits 0 or 1 depending on the subcommand parser; anything above that is a real failure. + my $rc = run_rc(sh_quote($path) . " -h >" . sh_quote("$log_dir/smoke-$out.log") . " 2>&1"); + die "$out -h failed (rc=$rc): running a $target_arch binary on this $arch host needs the" . + " qemu-user-static binfmt handler\n" if $rc > 1; + } + print "Smoke tests passed ($target_arch binaries ran through binfmt).\n"; + + print_step("Completed"); + print "Results in: $result_dir\n"; + return; +} + +sub run_rc { + my ($cmd) = @_; + print "+ $cmd\n"; + my $rc = system($cmd); + return $rc == -1 ? 255 : ($rc >> 8); +} + sub usage { return <<"USAGE"; Usage: $0 [options] -Build goconserver RPM from source. +Build the goconserver RPM: fetch the pinned source, overlay the committed go.mod/go.sum, and +compile (modules downloaded at build time but pinned by go.sum -- no vendored tree, no +`go mod tidy`). For the host arch the compile runs IN-CHROOT, in the el10 chroot (goconserver is a +CGO-free static binary; el8/el9 ship too old a Go). For a foreign --target-arch it cross-compiles on +the host, because a forcearch chroot would run the Go toolchain under qemu. Either way the rpm is +tagged with the target EL (4.el) so every EL repo gets an identical static binary. Options: --work-dir PATH Working directory (default: /tmp/goconserver-mockbuild) - --mock-cfg NAME Mock config name (auto-detected if omitted) + --mock-cfg NAME Target mock config (sets the EL dist tag; the build runs in its el10 peer) --target-arch ARCH Arch of the rpm to build (default: uname -m); another arch is cross-compiled (GOARCH) and packaged with rpmbuild --target - --mock-uniqueext STR Mock uniqueext value (for compatibility with mockbuild-all.pl) + --mock-uniqueext STR Mock uniqueext (for concurrency isolation under mockbuild-all.pl) --result-dir PATH Output directory for RPMs --log-dir PATH Output directory for logs - --skip-install Skip dnf install + smoke tests --version VER Version string (default: 0.3.3) --go-repo URL Git repo URL (default: github.com/xcat2/goconserver) - --go-ref REF Git ref to build (default: master) + --go-ref REF Git ref/SHA to build (default: the pinned commit) + --release-suffix STR Appended to Release for CD (e.g. .snap.) --build-timestamp EPOCH SOURCE_DATE_EPOCH for deterministic builds USAGE } @@ -357,13 +499,6 @@ sub run { } } -sub run_rc { - my ($cmd) = @_; - print "+ $cmd\n"; - my $rc = system($cmd); - return $rc == -1 ? 255 : ($rc >> 8); -} - sub capture { my ($cmd) = @_; my $out = `$cmd`; @@ -391,3 +526,18 @@ sub resolve_mock_cfg { } return "${os_id}+epel-${rel}-${arch}"; } + +sub create_deterministic_mock_cfg { + my ($base_cfg, $epoch, $dir) = @_; + my $cfg_path = "$dir/mock-deterministic.cfg"; + open my $fh, '>', $cfg_path or die "Cannot write $cfg_path: $!\n"; + print $fh "include('/etc/mock/${base_cfg}.cfg')\n"; + print $fh "config_opts['environment']['SOURCE_DATE_EPOCH'] = '$epoch'\n"; + print $fh "config_opts['environment']['ZERO_AR_DATE'] = '1'\n"; + # Allow network during %build so `go build` can download the (go.sum-pinned) modules -- we build + # against the proxy rather than committing a vendored tree. + print $fh "config_opts['rpmbuild_networking'] = True\n"; + print $fh "config_opts['use_host_resolv'] = True\n"; + close $fh; + return $cfg_path; +} diff --git a/grub2-xcat/mockbuild.pl b/grub2-xcat/mockbuild.pl old mode 100644 new mode 100755 index 9c978d9..8835fd4 --- a/grub2-xcat/mockbuild.pl +++ b/grub2-xcat/mockbuild.pl @@ -20,7 +20,6 @@ my $mock_cfg = ''; my $mock_uniqueext = ''; my $result_dir = "$repo_root/build-output/list3/grub2-xcat"; my $log_dir = "$repo_root/build-logs/list3/grub2-xcat"; -my $skip_install = 0; my $build_timestamp; GetOptions( @@ -30,7 +29,6 @@ GetOptions( 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); @@ -88,7 +86,6 @@ print "result_dir: $result_dir\n"; print "log_dir: $log_dir\n"; print "mock_cfg: $mock_cfg\n"; print "mock_uniqueext: " . ($mock_uniqueext ne '' ? $mock_uniqueext : '(none)') . "\n"; -print "skip_install: $skip_install\n"; make_path($result_dir); make_path($log_dir); @@ -231,68 +228,6 @@ for my $log (qw(build.log root.log state.log hw_info.log installed_pkgs.log)) { or die "Failed to copy $src to $log_dir: $!\n"; } -if (!$skip_install) { - print_step("Install RPM and run smoke tests"); - run("dnf -y install " . sh_quote($main_rpm)); - - my $core = '/tftpboot/boot/grub2/powerpc-ieee1275/core.elf'; - my $grub2ppc = '/tftpboot/boot/grub2/grub2.ppc'; - die "Missing installed core image: $core\n" if !-f $core; - die "Missing installed post script output: $grub2ppc\n" if !-f $grub2ppc; - - my $file_core_log = "$log_dir/smoke-file-core.log"; - my $file_ppc_log = "$log_dir/smoke-file-grub2ppc.log"; - my $qf_log = "$log_dir/smoke-rpm-qf.log"; - - my $rc_file_core = run_capture_rc("file $core", $file_core_log); - my $rc_file_ppc = run_capture_rc("file $grub2ppc", $file_ppc_log); - my $rc_qf = run_capture_rc("rpm -qf $core", $qf_log); - my $rc_cmp = run_capture_rc("cmp -s $core $grub2ppc", "$log_dir/smoke-cmp.log"); - - die "Smoke check failed: file core returned $rc_file_core\n" if $rc_file_core != 0; - die "Smoke check failed: file grub2.ppc returned $rc_file_ppc\n" if $rc_file_ppc != 0; - die "Smoke check failed: rpm -qf returned $rc_qf\n" if $rc_qf != 0; - die "Smoke check failed: core.elf and grub2.ppc differ (cmp rc=$rc_cmp)\n" if $rc_cmp != 0; - - my $riscv_image = '/tftpboot/boot/grub2/riscv64-efi/grubriscv64.efi'; - my $grub2riscv = '/tftpboot/boot/grub2/grub2.riscv64'; - die "Missing installed riscv64 grub2 image: $riscv_image\n" if !-f $riscv_image; - die "Missing installed post script output: $grub2riscv\n" if !-f $grub2riscv; - # The installed image must be the verified source image byte for byte, and %post must have - # copied it to grub2.riscv64. - my $rc_cmp_riscv_src = run_capture_rc("cmp -s " . sh_quote($riscv_efi) . " $riscv_image", - "$log_dir/smoke-cmp-riscv64-source.log"); - my $rc_cmp_riscv = run_capture_rc("cmp -s $riscv_image $grub2riscv", "$log_dir/smoke-cmp-riscv64.log"); - die "Smoke check failed: installed grubriscv64.efi differs from the source image (cmp rc=$rc_cmp_riscv_src)\n" - if $rc_cmp_riscv_src != 0; - die "Smoke check failed: grubriscv64.efi and grub2.riscv64 differ (cmp rc=$rc_cmp_riscv)\n" if $rc_cmp_riscv != 0; - my $riscv_type = pe_image_type($grub2riscv); - die "riscv64 grub2 image type check failed: $grub2riscv is $riscv_type\n" - if $riscv_type ne 'PE32+ RISC-V 64-bit'; - - my $core_out = slurp($file_core_log); - my $qf_out = slurp($qf_log); - - die "Core image signature check failed:\n$core_out\n" - if $core_out !~ /ELF|data/i; - die "Installed core image is not owned by grub2-xcat:\n$qf_out\n" - if $qf_out !~ /^grub2-xcat-/m; - - my $summary = "$log_dir/smoke-summary.txt"; - open my $sfh, '>', $summary or die "Cannot write $summary: $!\n"; - print {$sfh} "core=$core\n"; - print {$sfh} "grub2ppc=$grub2ppc\n"; - print {$sfh} "rc_file_core=$rc_file_core\n"; - print {$sfh} "rc_file_ppc=$rc_file_ppc\n"; - print {$sfh} "rc_qf=$rc_qf\n"; - print {$sfh} "rc_cmp=$rc_cmp\n"; - print {$sfh} "grub2riscv64=$grub2riscv\n"; - print {$sfh} "rc_cmp_riscv64_source=$rc_cmp_riscv_src\n"; - print {$sfh} "type_riscv64=$riscv_type\n"; - print {$sfh} "rc_cmp_riscv64=$rc_cmp_riscv\n"; - close $sfh; -} - print_step("Completed"); print "Main RPM: $main_rpm\n"; print "Artifacts: $result_dir\n"; @@ -308,7 +243,6 @@ Usage: $0 [options] --mock-uniqueext TXT Optional mock --uniqueext suffix to isolate concurrent builds --result-dir PATH Output RPM/SRPM directory (default: $result_dir) --log-dir PATH Log directory (default: $log_dir) - --skip-install Skip dnf install + smoke tests --build-timestamp EPOCH Unix timestamp for deterministic builds (SOURCE_DATE_EPOCH) USAGE } diff --git a/ipmitool/mockbuild.pl b/ipmitool/mockbuild.pl old mode 100644 new mode 100755 index 9a5bba8..df923ab --- a/ipmitool/mockbuild.pl +++ b/ipmitool/mockbuild.pl @@ -3,7 +3,7 @@ use strict; use warnings; use Cwd qw(abs_path); -use File::Basename qw(dirname basename); +use File::Basename qw(dirname); use File::Copy qw(copy); use File::Path qw(make_path remove_tree); use Getopt::Long qw(GetOptions); @@ -13,7 +13,6 @@ my $repo_root = abs_path("$script_dir/.."); my $pkg_dir = "$repo_root/ipmitool"; my $spec_file = "$pkg_dir/ipmitool.spec"; -my $source_url = 'https://github.com/ipmitool/ipmitool/archive/refs/tags/IPMITOOL_1_8_18.tar.gz'; my $source_file = ''; my $work_dir = '/tmp/ipmitool-xcat-mockbuild'; my $mock_cfg = ''; @@ -21,11 +20,9 @@ my $target_arch = ''; my $mock_uniqueext = ''; my $result_dir = "$repo_root/build-output/list3/ipmitool-xcat"; my $log_dir = "$repo_root/build-logs/list3/ipmitool-xcat"; -my $skip_install = 0; my $build_timestamp; GetOptions( - 'source-url=s' => \$source_url, 'source-file=s' => \$source_file, 'work-dir=s' => \$work_dir, 'mock-cfg=s' => \$mock_cfg, @@ -33,14 +30,13 @@ GetOptions( 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); die "Run as root (current uid=$>)\n" if $> != 0; die "Missing spec file: $spec_file\n" if !-f $spec_file; -for my $bin (qw(wget mock rpmbuild rpm dnf ldd bash)) { +for my $bin (qw(mock rpmbuild rpm dnf ldd bash)) { run("command -v " . sh_quote($bin) . " >/dev/null 2>&1"); } @@ -86,9 +82,7 @@ print "log_dir: $log_dir\n"; print "mock_cfg: $mock_cfg\n"; print "target_arch:$target_arch\n"; print "mock_uniqueext: " . ($mock_uniqueext ne '' ? $mock_uniqueext : '(none)') . "\n"; -print "source_url: $source_url\n"; print "source_file:$source_file\n"; -print "skip_install: $skip_install\n"; print "SOURCE_DATE_EPOCH: $SOURCE_DATE_EPOCH\n"; make_path($result_dir); @@ -97,10 +91,24 @@ make_path($log_dir); print_step("Mock config check"); run("mock -r " . sh_quote($mock_cfg) . $mock_uniqueext_opt . " --print-root-path >/dev/null"); -print_step("Download upstream source"); -run("wget --spider " . sh_quote($source_url)); -run("wget -O " . sh_quote($source_path) . " " . sh_quote($source_url)); -normalize_source_archive($source_path, $version, $work_dir); +print_step("Verify tracked source archive"); +# Upstream source (documented for provenance; NOT fetched at build time -- see below): +# https://github.com/ipmitool/ipmitool/archive/refs/tags/IPMITOOL_1_8_18.tar.gz +# The ipmitool source (ipmitool-.tar.gz) is tracked in the repo, already normalized to the +# ipmitool-/ top-level that %setup -n expects, and consumed directly by mock (--sources +# $pkg_dir below). There is nothing to download: the old fetch re-derived this SAME tracked file +# and rewrote it IN PLACE, and the checkout is shared between the two arch build hosts building at +# once -- so the in-place rewrite raced the other host's concurrent ipmitool build, which could +# read the file mid-write and get a truncated archive. We only READ it now, so concurrent builds +# can never race on it. Fail loudly (do NOT silently re-fetch) if the checkout is missing/broken. +die "Tracked ipmitool source missing: $source_path (incomplete checkout?)\n" if !-f $source_path; +my $top = capture( + "tar -tzf " . sh_quote($source_path) . + " 2>/dev/null | grep -E '^(\\./)?ipmitool-$version/' | head -n1 || true" +); +die "Tracked ipmitool source is not the expected ipmitool-$version/ tree: $source_path\n" + if $top eq ''; +print "Using tracked source archive (read-only, no fetch, no shared write): $source_path\n"; print_step("Verify spec assets"); for my $asset (@spec_assets) { @@ -201,7 +209,7 @@ for my $log (qw(build.log root.log state.log hw_info.log installed_pkgs.log)) { or die "Failed to copy $src to $log_dir: $!\n"; } -if (!$skip_install && $target_arch ne $arch) { +if ($target_arch ne $arch) { # A cross-built rpm cannot be installed on this host: install it into the (emulated) build # chroot instead and run the binary there. print_step("Install RPM into the chroot and run smoke tests"); @@ -222,59 +230,6 @@ if (!$skip_install && $target_arch ne $arch) { print {$sfh} "rc_version=$rc_version\n"; close $sfh; } -elsif (!$skip_install) { - print_step("Install RPM and run smoke tests"); - run("dnf -y install " . sh_quote($main_rpm)); - - my $bin = '/opt/xcat/bin/ipmitool-xcat'; - die "Missing installed binary: $bin\n" if !-x $bin; - - my $help_short_log = "$log_dir/smoke-help-short.log"; - my $help_long_log = "$log_dir/smoke-help-long.log"; - my $version_log = "$log_dir/smoke-version.log"; - my $open_log = "$log_dir/smoke-open-mc-info.log"; - my $ldd_log = "$log_dir/smoke-ldd.log"; - - my $rc_help_short = run_capture_rc("$bin -h", $help_short_log); - my $rc_help_long = run_capture_rc("$bin --help", $help_long_log); - my $rc_version = run_capture_rc("$bin -V", $version_log); - my $rc_open = run_capture_rc("$bin -I open mc info", $open_log); - my $rc_ldd = run_capture_rc("ldd $bin", $ldd_log); - - die "Smoke check failed: -h returned $rc_help_short\n" if $rc_help_short != 0; - die "Smoke check failed: -V returned $rc_version\n" if $rc_version != 0; - die "Smoke check failed: ldd returned $rc_ldd\n" if $rc_ldd != 0; - - my $help_short_out = slurp($help_short_log); - my $help_long_out = slurp($help_long_log); - my $version_out = slurp($version_log); - my $open_out = slurp($open_log); - my $ldd_out = slurp($ldd_log); - - die "Short help output does not contain usage text\n" - if $help_short_out !~ /usage:/i; - die "Long help output does not contain usage text\n" - if $help_long_out !~ /usage:/i; - die "Long help returned unexpected rc=$rc_help_long (expected 0 or 1)\n" - if $rc_help_long != 0 && $rc_help_long != 1; - die "Version output missing expected version string\n" - if $version_out !~ /ipmitool-xcat version \Q$version\E/i; - die "ldd output missing libcrypto dependency\n" - if $ldd_out !~ /libcrypto/; - if ($rc_open != 0 && $open_out !~ m{Could not open device|/dev/ipmi}) { - die "IPMI probe failed with unexpected output:\n$open_out\n"; - } - - my $summary = "$log_dir/smoke-summary.txt"; - open my $sfh, '>', $summary or die "Cannot write $summary: $!\n"; - print {$sfh} "binary=$bin\n"; - print {$sfh} "rc_help_short=$rc_help_short\n"; - print {$sfh} "rc_help_long=$rc_help_long\n"; - print {$sfh} "rc_version=$rc_version\n"; - print {$sfh} "rc_open=$rc_open\n"; - print {$sfh} "rc_ldd=$rc_ldd\n"; - close $sfh; -} print_step("Completed"); print "Main RPM: $main_rpm\n"; @@ -285,7 +240,6 @@ exit 0; sub usage { return <<"USAGE"; Usage: $0 [options] - --source-url URL Upstream tarball URL (default: $source_url) --source-file FILE Source filename stored in ipmitool/ (default: inferred from spec version) --work-dir PATH Temporary work dir (default: $work_dir) --mock-cfg NAME Mock config (default: +epel-10-) @@ -294,7 +248,6 @@ Usage: $0 [options] --mock-uniqueext TXT Optional mock --uniqueext suffix to isolate concurrent builds --result-dir PATH Output RPM/SRPM directory (default: $result_dir) --log-dir PATH Log directory (default: $log_dir) - --skip-install Skip dnf install + smoke tests --build-timestamp N Unix epoch for SOURCE_DATE_EPOCH (deterministic builds) USAGE } @@ -337,38 +290,6 @@ sub parse_spec { return ($version, @assets); } -sub normalize_source_archive { - my ($archive, $version, $work_base) = @_; - - my $normalize_dir = "$work_base/source-normalize"; - remove_tree($normalize_dir) if -d $normalize_dir; - make_path($normalize_dir); - - run("tar -xzf " . sh_quote($archive) . " -C " . sh_quote($normalize_dir)); - - my @entries = grep { $_ !~ m{/\.\.?$} } glob("$normalize_dir/*"); - die "Unexpected archive layout in $archive\n" if @entries != 1; - my $top_path = $entries[0]; - die "Unexpected non-directory top-level entry in $archive: $top_path\n" - if !-d $top_path; - - my $expected_top = "ipmitool-$version"; - my $actual_top = basename($top_path); - if ($actual_top ne $expected_top) { - my $new_path = "$normalize_dir/$expected_top"; - run("rm -rf " . sh_quote($new_path)); - run("mv " . sh_quote($top_path) . " " . sh_quote($new_path)); - } - - # Repack using the expected top-level directory required by the spec. - run( - "tar --sort=name --owner=0 --group=0 --mtime=\@$SOURCE_DATE_EPOCH" . - " -C " . sh_quote($normalize_dir) . - " -czf " . sh_quote($archive) . - " " . sh_quote($expected_top) - ); -} - sub print_step { my ($msg) = @_; print "\n== $msg ==\n"; diff --git a/mockbuild-all.pl b/mockbuild-all.pl old mode 100644 new mode 100755 index d6dd92b..f108ff0 --- a/mockbuild-all.pl +++ b/mockbuild-all.pl @@ -10,16 +10,23 @@ use File::Find qw(find); use File::Glob qw(bsd_glob); use File::Path qw(make_path remove_tree); use File::Temp qw(tempdir tempfile); -use FindBin; use Getopt::Long qw(GetOptions); use Parallel::ForkManager; use POSIX qw(strftime); -use lib "$FindBin::Bin/lib"; +use FindBin qw($RealBin); +use lib $RealBin, "$RealBin/lib"; +use MockBuildUtils qw(sh_quote print_step version_matches required_pkgs + install_deps_packages install_deps_command missing_perl_modules + read_manifest verify_repo_packages verify_repo_signature verify_rpm_signatures + rpm_version rpm_release rpm_sigmd5 restamp_release_line + cross_copy_genesis finalize_xcat_dep bump_dep_release_suffix + build_mock_uniqueext rpmkeys_checksig_problem); +# print_step and sh_quote come from MockBuildUtils above; XCAT::BuildUtils carries the same +# print_step, so it is deliberately NOT imported here (one definition, no redefinition warning). use XCAT::BuildUtils qw( capture_command every_step_failed hashes_equal - print_step read_lines require_command run_command @@ -30,6 +37,43 @@ use XCAT::GenesisRelease qw( verify_release_file ); +# --- Mount-namespace isolation: guard the host cgroup against mock teardown propagation ---------- +# mock mounts /sys/fs/cgroup into every build chroot. On these systemd build hosts every mount is +# `shared`, so the chroot's cgroup joins the HOST's cgroup peer group. When mock tears a chroot down +# -- its post-build --scrub, or an aborted build's cleanup -- the unmount PROPAGATES back through the +# shared peer group and unmounts the HOST's /sys/fs/cgroup, after which every later mock (and even new +# login sessions) dies with "Failed to determine whether the unified cgroups hierarchy is used: No +# medium found". This bit ppc hardest (it leaks corpse chroot mounts on abort) but x86 shares the same +# shared-cgroup exposure. Re-exec inside a private mount namespace made rslave +# (`unshare --mount --propagation slave`): the namespace still sees host mounts (slave = one-way), but +# nothing mock mounts/unmounts can propagate OUT to the host. As a bonus the namespace tears down every +# mount mock leaks when we exit, so an aborted build can no longer leave corpse mounts under +# /var/lib/mock. Best-effort: only as root (needs CAP_SYS_ADMIN) and only if `unshare` exists; +# otherwise warn loudly and continue unisolated. MOCKBUILD_ALL_MOUNTNS guards against a re-exec loop. +# Build-free modes (--verify-repo, --finalize-xcat-dep) run no mock and are documented no-root, so they +# skip the re-exec entirely -- no cgroup exposure, and no spurious non-root warning. +my $mountns_build_free = grep { /^--(?:verify-repo|finalize-xcat-dep|install-deps)(?:=|$)/ } @ARGV; +unless ($ENV{MOCKBUILD_ALL_MOUNTNS} || $mountns_build_free) { + if ($> != 0) { + warn "WARN: not root -- skipping mount-namespace isolation (host-cgroup propagation guard); " + . "run as root in CI so mock chroot teardown cannot unmount the host /sys/fs/cgroup\n"; + } elsif (system('sh', '-c', 'command -v unshare >/dev/null 2>&1') != 0) { + warn "WARN: 'unshare' not found -- skipping mount-namespace isolation; mock chroot teardown " + . "may unmount the host /sys/fs/cgroup on a shared-propagation host\n"; + } else { + $ENV{MOCKBUILD_ALL_MOUNTNS} = 1; + # Absolute path to self (resolved against cwd, which unshare preserves) so the re-exec'd perl + # finds a relative $0. A bare $PATH-only $0 isn't resolved (abs_path doesn't search $PATH), but + # a shell invocation yields a full $0 there anyway. + my $self = abs_path($0) // $0; + my @reexec = ('unshare', '--mount', '--propagation', 'slave', '--', $^X, $self, @ARGV); + exec { $reexec[0] } @reexec; + # exec only returns on failure -- fall through and run unisolated rather than abort the build. + warn "WARN: exec unshare failed ($!) -- continuing without mount-namespace isolation\n"; + delete $ENV{MOCKBUILD_ALL_MOUNTNS}; + } +} + my $script_dir = abs_path(dirname(__FILE__)); my $repo_root = abs_path($script_dir); my $xcat_src = "$repo_root/../xcat-core"; @@ -46,17 +90,26 @@ my $parallel_targets = 1; # 1 = serial (default; safe). 0/auto = all EL target my $max_parallel = 0; # 0/auto = host nproc: global cap on concurrent mock builds (all targets) my $run_id = ''; my $build_timestamp; -my $skip_install = 0; +# CD version bump: when set, every xcat-dep package spec's Release gets a +# ".snap." suffix so each pipeline run publishes a +# fresh, monotonic NVR (deploy's additive rsync is a no-op otherwise). NOT applied +# to xCAT-genesis-base (built from xcat-core, kept in lockstep with genesis-scripts). +my $build_number; +# Pinned goconserver upstream commit (xcat2/goconserver). goconserver 0.3.3 is unreleased (the +# newest tag is v0.3.2), so it exists only on master -- pin an immutable SHA instead of the moving +# branch so the build is reproducible. Bump this deliberately when uptaking a new goconserver. +my $GOCONSERVER_REF = '6166fe5ec1c5b3c20475e322a9f0e8e93c87e45f'; my $skip_build = 0; my $skip_xcat_dep = 0; my $skip_perl = 0; -my $skip_xcat = 0; +my $install_deps = 0; my $skip_genesis = 0; my $skip_createrepo = 0; my $skip_tarball = 0; my $genesis_release = ''; my $genesis_release_checksums; my $scrub_all_chroots = 0; +my $keep_buildroots = 0; # keep per-step mock chroots after build (default: --scrub=chroot each) my $dry_run = 0; my @extra_collect_dirs; my $repo_dep = ''; @@ -65,6 +118,18 @@ my $gpg_key_name = 'xCAT Signing Key'; my $gpg_home = ''; my $gpg_program = ''; my $force_unlock = 0; +# --finalize-xcat-dep: post-build cross-arch genesis provisioning (issue #7610). Takes the two +# per-arch repo roots and cross-populates the noarch xCAT-genesis-base between them. +my $finalize_xcat_dep = 0; +my $x86_64_repo = ''; +my $ppc64le_repo = ''; +# --verify-repo=: standalone, build-free completeness + signature gate over one already-built +# per-target repo (see verify_target_repo). Empty means "not in standalone verify mode". The target +# is derived from the repo path (.../rh/ -> alma+epel--) or taken from --target. +my $verify_repo = ''; +# --no-verify-repo suppresses the AUTOMATIC post-build gate deploy_target runs after each target is +# finalized+signed (for iteration/debug). Verification is ON by default. +my $no_verify_repo = 0; my @HELD_LOCKS; my $LOCK_OWNER_PID; my ($COMMON_STAGE, $COMMON_DESTINATION, $COMMON_BACKUP); @@ -82,6 +147,11 @@ GetOptions( 'gpg-key-name=s' => \$gpg_key_name, 'gpg-home=s' => \$gpg_home, 'force-unlock!' => \$force_unlock, + 'finalize-xcat-dep!' => \$finalize_xcat_dep, + 'x86_64-repo=s' => \$x86_64_repo, + 'ppc64le-repo=s' => \$ppc64le_repo, + 'verify-repo=s' => \$verify_repo, + 'no-verify-repo!' => \$no_verify_repo, 'target=s' => \$target, 'nproc=i' => \$nproc, 'parallel-builds=i' => \$parallel_builds, @@ -89,21 +159,27 @@ GetOptions( 'max-parallel=i' => \$max_parallel, 'run-id=s' => \$run_id, 'build-timestamp=i' => \$build_timestamp, - 'skip-install!' => \$skip_install, + 'build-number=i' => \$build_number, 'skip-build!' => \$skip_build, 'skip-xcat-dep!' => \$skip_xcat_dep, 'skip-perl!' => \$skip_perl, - 'skip-xcat!' => \$skip_xcat, + 'install-deps!' => \$install_deps, 'skip-genesis!' => \$skip_genesis, 'skip-createrepo!' => \$skip_createrepo, 'skip-tarball!' => \$skip_tarball, 'genesis-release=s' => \$genesis_release, 'scrub-all-chroots!' => \$scrub_all_chroots, + 'keep-buildroots!' => \$keep_buildroots, 'collect-dir=s@' => \@extra_collect_dirs, 'dry-run!' => \$dry_run, ) or die usage(); -die "Run as root (uid=$>)\n" if $> != 0; +die "Run as root (uid=$>)\n" if $> != 0 && !$finalize_xcat_dep && !$verify_repo; +# --skip-build collects a prior build's artifacts from that build's per-target tree, so it must +# know the target. Without --target the default is "all three EL targets", and each would collect +# the same artifacts and cross-publish them into every repo (foreign-EL / foreign-arch rpms). +die "--skip-build requires an explicit --target (collection is per-target)\n" + if $skip_build && $target eq ''; die "--parallel-builds must be >= 1\n" if defined($parallel_builds) && $parallel_builds < 1; @@ -126,6 +202,94 @@ if ($run_id eq '') { $run_id = strftime('%Y%m%d-%H%M%S', gmtime($SOURCE_DATE_EPOCH)); } +# --verify-repo=: a distinct, build-free completeness + signature gate over ONE already-built +# per-target repo. The value is just the repo dir; the manifest comes from the script's existing +# resolution (repo_root/packages-manifest.conf) and the gpg key/home from --gpg-key-name/--gpg-home. +# The target is derived from the repo path (.../rh/ -> alma+epel--) unless --target +# is given. Delegates the whole check to verify_target_repo (the SAME gate the auto-run uses), so it +# exits 0 when complete or dies listing every problem. Runs alone -- no build, no lock, no root. +if ($verify_repo ne '') { + require_command('rpm'); + my $rdir = abs_path($verify_repo) or die "--verify-repo repo '$verify_repo' not found\n"; + die "--verify-repo repo '$rdir' is not a directory\n" if !-d $rdir; + my $tgt = $target ne '' ? $target : derive_target_from_repo_path($rdir); + die "--verify-repo: cannot derive a target from repo path '$rdir'; pass --target\n" + if !defined($tgt) || $tgt eq ''; + # sig_required=1: a standalone verify MUST assert the repomd signature (its documented contract), + # never silently skip it when no gpg key/home is configured (that would be a false PASS on sigs). + verify_target_repo($rdir, $tgt, undef, 1); # manifest defaults to repo_root/packages-manifest.conf + exit 0; +} + +# --finalize-xcat-dep: a distinct, build-free mode. After BOTH arch build hosts have +# produced their per-EL repos (each carrying only its own xCAT-genesis-base), the x86_64 +# repo must ALSO ship the noarch xCAT-genesis-base-ppc64 (so an x86_64 MN can netboot ppc +# nodes) and the ppc64le repo must ship xCAT-genesis-base-x86_64 -- the 2.17 behaviour +# that issue #7610 regressed. This mode ONLY cross-copies the genesis-base rpm(s) between +# the two repos (dropping any stale foreign-arch genesis) and re-indexes + re-signs the +# affected repomd; it builds nothing and holds no output lock. +if ($finalize_xcat_dep) { + die "--finalize-xcat-dep requires --x86_64-repo and --ppc64le-repo\n" + if $x86_64_repo eq '' || $ppc64le_repo eq ''; + require_command('createrepo_c'); + require_command('rpm'); + require_command('rpmsign') if $gpg_sign; + require_command('gpg') if $gpg_sign; + my $x86 = abs_path($x86_64_repo) or die "--x86_64-repo '$x86_64_repo' not found\n"; + my $ppc = abs_path($ppc64le_repo) or die "--ppc64le-repo '$ppc64le_repo' not found\n"; + die "--x86_64-repo '$x86' is not a directory\n" if !-d $x86; + die "--ppc64le-repo '$ppc' is not a directory\n" if !-d $ppc; + # Inject the per-rpm gpg re-sign and the repo re-index as callbacks so the finalize logic in + # MockBuildUtils stays free of this script's gpg/createrepo state. + finalize_xcat_dep($x86, $ppc, + sign => ($gpg_sign ? sub { + my ($rpm) = @_; + local $ENV{GNUPGHOME} = $gpg_home if $gpg_home; + run_simple("rpmsign --define " . sh_quote("%_gpg_name $gpg_key_name") . " --addsign " . sh_quote($rpm)); + } : undef), + reindex => \&reindex_and_sign_repo, + ); + + # finalize just RE-INDEXED + RE-SIGNED each per-EL repo and cross-copied the foreign-arch genesis + # in -- i.e. it produced the FINAL shipped state, which the per-target gate in deploy_target (run + # earlier, pre-finalize) never saw. So run the SAME manifest completeness + signature gate here, on + # every finalized cell, so the build script verifies its own final output by default (no external + # --verify-repo needed). Suppressible with --no-verify-repo. + unless ($no_verify_repo) { + my %seen; + for my $root ($x86, $ppc) { + my @cells = (glob("$root/rh*/x86_64"), glob("$root/rh*/ppc64le")); + for my $d (sort @cells) { + next unless -d $d; + my $abs = abs_path($d); + next if $seen{$abs}++; + my $tgt = derive_target_from_repo_path($abs) + or die "FATAL: finalize verify -- cannot derive target from '$abs'\n"; + verify_target_repo($abs, $tgt); + } + } + } + exit 0; +} + +# CD version bump. Rewrite every xcat-dep package spec's Release line in this +# (freshly-checked-out, git-clean) tree so the built rpms carry a fresh, monotonic +# NVR each run. Runs BEFORE any child builder is invoked. genesis-base lives under +# xcat-core, not $repo_root, so it is untouched (stays in lockstep with the deployed +# core's genesis-scripts). +my $RELEASE_BUMP = ''; +if (defined $build_number) { + die "--build-number must be a non-negative integer\n" if $build_number < 0; + $RELEASE_BUMP = strftime('.snap%Y%m%d%H%M', gmtime($SOURCE_DATE_EPOCH)) . ".$build_number"; + # A dry run must not touch the tree. Report what would be stamped and leave the specs alone; + # $RELEASE_BUMP is still set so the rest of the (no-op) dry-run plan reflects it. + if ($dry_run) { + print "[dry-run] would stamp Release suffix '$RELEASE_BUMP' on xcat-dep specs under $repo_root (no files written)\n"; + } else { + bump_dep_release_suffix($repo_root, $RELEASE_BUMP); + } +} + # Single output base for every NFS-shared write. Two hosts build in parallel on one NFS by # passing distinct --output paths. --output-root/--repo-dep, if given, override the derived # values. Default keeps the historical layout so existing callers are unaffected. @@ -164,6 +328,28 @@ die "Could not resolve ID from /etc/os-release\n" if $os_id eq ''; die "Could not resolve major release from VERSION_ID='$version_id' in /etc/os-release\n" if !defined($rel) || $rel eq ''; +# --install-deps: make THIS host able to run the script, then exit. It has to come before the +# require_command checks below -- those are the very things it installs, and a host that lacks them +# would die here with no way to fix itself. Run once per build host, as root. +# +# The perl modules are re-checked by LOADING them afterwards rather than trusting the package +# manager: a module that is still missing is exactly the failure this mode exists to prevent, and it +# aborted CD runs mid-build twice (perl-File-Slurper on xcat-master-ub, perl-IPC-Cmd on +# xcat-master-ppc), each time as a compile-time error inside XCAT::BuildUtils. +if ($install_deps) { + die "--install-deps must run as root (uid=$>)\n" if $> != 0; + my @cmd = install_deps_command($os_id); + print_step("Install build prerequisites ($os_id)"); + print " " . join(' ', @cmd) . "\n"; + run_command(@cmd); + my @modules = qw(File::Slurper IPC::Cmd Parallel::ForkManager Digest::SHA); + my @missing = missing_perl_modules(@modules); + die "FATAL: still missing after install: " . join(', ', @missing) . "\n" if @missing; + print " perl modules present: " . join(', ', @modules) . "\n"; + print " host is ready\n"; + exit 0; +} + for my $bin (qw(perl uname createrepo_c tar find rpm)) { require_command($bin); } @@ -281,6 +467,13 @@ sub build_one_target { my $rel = $profile->{rel}; $arch = $profile->{arch}; + # Per-target required set from packages-manifest.conf: build ONLY these packages, and fail the + # run if any of them fails. A package absent from this target's section is not built for it. + my %MANIFEST = read_manifest("$repo_root/packages-manifest.conf"); + my %req = %{ $MANIFEST{$target} // {} }; + die "FATAL: no manifest section for target '$target' in packages-manifest.conf\n" + if !%req; + my $run_root = "$output_root/$run_id"; my $build_root = "$run_root/build-results"; my $log_root = "$run_root/build-logs"; @@ -290,6 +483,16 @@ my $tarball = "$output_root/mockbuild-all-$target-$run_id.tar.gz"; my $srpm_repo_dir = "$run_root/repo-src"; my $srpm_tarball = "$output_root/mockbuild-all-$target-$run_id-srpm.tar.gz"; +# Each real build must start from a clean per-target tree. run_id is derived from the deterministic +# commit timestamp, so re-runs of the same commit resolve to the SAME $run_root -- without a wipe, a +# stale rpm or a stale perl status.txt from an earlier (possibly failed) run could be reused and mask +# a failure (see mockbuild-perl-packages.pl, which reads per-package status files back). --skip-build +# deliberately KEEPS the tree (it collects a prior build's artifacts); --dry-run writes nothing. +if (!$skip_build && !$dry_run && -d $run_root) { + print "Cleaning stale per-target tree before build: $run_root\n"; + remove_tree($run_root); +} + # All dep builders run natively on every arch. xnba-undi and grub2-xcat are noarch packagings of # committed artifacts (an x86 UNDI ROM / the grub2 resource tarball) with no arch-specific build # step, so ppc builds them the same as x86 -- no cross-arch import. A forcearch target builds @@ -307,8 +510,10 @@ my %profile_builds = map { $_ => 1 } @{ $profile->{dep_builders} }; my $perl_builder = "$repo_root/mockbuild-perl-packages.pl"; +# buildrpms.pl (in xcat-core) is only needed for the OS-dependent xCAT-genesis-base +# build below; the full xCAT core is built separately by the xcat-core pipeline. die "Missing xCAT build script: $xcat_src/buildrpms.pl\n" - if !$skip_xcat && !-f "$xcat_src/buildrpms.pl"; + if !$skip_genesis && !-f "$xcat_src/buildrpms.pl"; my @active_dep_builders; for my $b (@dep_builders) { @@ -359,12 +564,11 @@ print "parallel_builds: " . (defined($parallel_builds) ? $parallel_builds : 'au print "skip_build: $skip_build\n"; print "skip_xcat_dep: $skip_xcat_dep\n"; print "skip_perl: $skip_perl\n"; -print "skip_xcat: $skip_xcat\n"; print "skip_genesis: $skip_genesis\n"; -print "skip_install: $skip_install\n"; print "skip_createrepo: $skip_createrepo\n"; print "skip_tarball: $skip_tarball\n"; print "scrub_all_chroots:$scrub_all_chroots\n"; +print "keep_buildroots: $keep_buildroots\n"; print "dry_run: $dry_run\n"; print "perl_builder: $perl_builder\n"; print "tarball: $tarball\n"; @@ -394,6 +598,7 @@ if (!$skip_build) { if (!$skip_xcat_dep) { for my $builder (@active_dep_builders) { + next unless $req{ $builder->{name} }; # manifest: build only required dep packages my $name = $builder->{name}; my $script = $builder->{script}; my $step_result = "$build_root/$name"; @@ -409,19 +614,29 @@ if (!$skip_build) { # host-local, run-scoped work dir so /tmp doesn't collide between runs '--work-dir', shell_quote("/tmp/mockbuild-all-$run_id/$name"), '--build-timestamp', $SOURCE_DATE_EPOCH, - ($skip_install ? '--skip-install' : ()), + # goconserver generates its spec at build time (from an upstream clone), so the + # in-tree spec Release bump above cannot reach it. Hand the CD suffix down so its + # NVR advances per run too, and pin the clone to an immutable commit (not the moving + # 'master') so the build is reproducible. + ($name eq 'goconserver' + ? ('--go-ref', sh_quote($GOCONSERVER_REF), + ($RELEASE_BUMP ne '' ? ('--release-suffix', sh_quote($RELEASE_BUMP)) : ())) + : ()), ); push @build_steps, { id => "xcat-dep:$name", step => "Build xcat-dep: $name", cmd => $cmd, log => "$log_root/$name/run.log", + scrub_cfg => $target, + scrub_uniqueext => $step_uniqueext, }; push @collect_roots, $step_result; } } - if (!$skip_perl) { + my @perl_pkgs = sort grep { /^perl-/ } keys %req; # manifest: perl packages required here + if (!$skip_perl && @perl_pkgs) { my $perl_result = "$build_root/perl/$arch"; my $perl_log = "$log_root/perl/$arch"; my $perl_uniqueext = build_mock_uniqueext($run_id, ++$build_step_seq, 'perl-list6'); @@ -429,19 +644,24 @@ if (!$skip_build) { # forks one mock build per perl package (~7), which -- multiplied by parallel EL targets -- # oversubscribes the host. my $cmd = join(' ', - 'perl', shell_quote($perl_builder), - '--mock-cfg', shell_quote($target), + 'perl', sh_quote($perl_builder), + '--mock-cfg', sh_quote($target), ($profile->{forcearch} - ? ('--target-arch', shell_quote($arch), '--noarch-mock-cfg', shell_quote($profile->{noarch_cfg})) + ? ('--target-arch', sh_quote($arch), '--noarch-mock-cfg', sh_quote($profile->{noarch_cfg})) : ()), ($profile->{epel} ? () : ('--epel-gap')), - '--mock-uniqueext', shell_quote($perl_uniqueext), - '--result-dir', shell_quote($perl_result), - '--log-dir', shell_quote($perl_log), - '--work-dir', shell_quote("/tmp/mockbuild-all-$run_id/perl-list6"), + '--mock-uniqueext', sh_quote($perl_uniqueext), + '--result-dir', sh_quote($perl_result), + '--log-dir', sh_quote($perl_log), + '--work-dir', sh_quote("/tmp/mockbuild-all-$run_id/perl-list6"), + '--packages', sh_quote(join(',', @perl_pkgs)), # manifest: only required perl pkgs (($max_build_workers && $max_build_workers >= 1) ? ('--jobs', $max_build_workers) : ()), '--build-timestamp', $SOURCE_DATE_EPOCH, - ($skip_install ? '--skip-install' : ()), + # CD bump: the in-tree spec Release bump above only reaches the spec-mode perl + # packages; the srpm-mode ones (HTML-Form, IO-Stty, Net-Telnet) build from a + # committed .src.rpm, so hand the suffix down for the builder to re-stamp them. + ($RELEASE_BUMP ne '' ? ('--release-suffix', sh_quote($RELEASE_BUMP)) : ()), + ($keep_buildroots ? '--keep-buildroots' : ()), ); push @build_steps, { id => 'perl', @@ -452,26 +672,9 @@ if (!$skip_build) { push @collect_roots, $perl_result; } - if (!$skip_xcat) { - # Own HOME per target (buildrpms.pl uses $HOME/rpmbuild) so parallel targets don't race. - my $xcat_home = "/tmp/mockbuild-all-$run_id/xcat-home"; - my $mktree = join(' ', map { shell_quote("$xcat_home/rpmbuild/$_") } qw(SOURCES SPECS BUILD BUILDROOT RPMS SRPMS)); - my $cmd = "mkdir -p $mktree && HOME=" . shell_quote($xcat_home) . ' ' . join(' ', - 'perl', shell_quote("$xcat_src/buildrpms.pl"), - '--target', shell_quote($target), - '--nproc', int($nproc), - '--force', - '--verbose', - '--xcat_dep_path', shell_quote($repo_root), - ); - push @build_steps, { - id => 'xcat', - step => 'Build xCAT packages', - cmd => $cmd, - cwd => $xcat_src, - log => "$log_root/xcat-build.log", - }; - } + # NOTE: this script builds ONLY xcat-dep (its dep packages, the perl packages, and + # the OS-dependent xCAT-genesis-base below). The full xCAT core is built separately + # by the xcat-core pipeline -- mockbuild-all no longer has a monolithic core-build path. # xCAT-genesis-base is OS-dependent (its initramfs bundles the build chroot's # kernel + glibc/busybox/perl), so it is built here, per target, and shipped @@ -479,16 +682,28 @@ if (!$skip_build) { # (run in the xcat-core dir) derives the same snapYYYYMMDDHHMM Release from # xcat-core's Gitepoch, so it matches xCAT-genesis-scripts (built in core) and # the exact-version dependency genesis-scripts -> genesis-base resolves. - if (!$skip_genesis) { + if (!$skip_genesis && $req{'xCAT-genesis-base'}) { # buildrpms.pl stages sources in $HOME/rpmbuild (via rpmdev-setuptree). Give each # per-target genesis build its own HOME so parallel EL targets don't race on the shared # /root/rpmbuild tree (that race is what made concurrent genesis builds fail). my $genesis_home = "/tmp/mockbuild-all-$run_id/genesis-home"; # buildrpms.pl's rpmdev-setuptree only runs during env setup, not per build, so create the # rpmbuild tree ourselves for this per-target HOME (else $HOME/rpmbuild/SOURCES is missing). - my $mktree = join(' ', map { shell_quote("$genesis_home/rpmbuild/$_") } qw(SOURCES SPECS BUILD BUILDROOT RPMS SRPMS)); - my $cmd = "mkdir -p $mktree && HOME=" . shell_quote($genesis_home) . ' ' . join(' ', - 'perl', shell_quote("$xcat_src/buildrpms.pl"), + my $mktree = join(' ', map { sh_quote("$genesis_home/rpmbuild/$_") } qw(SOURCES SPECS BUILD BUILDROOT RPMS SRPMS)); + # The genesis chroot (xCAT-genesis-base-) is SHARED across runs -- buildrpms.pl builds it + # without a per-run --mock-uniqueext. mock's post-build scrub only runs on SUCCESS, so a killed + # or failFast-interrupted prior run leaves the chroot stunted (missing /bin/sh), and mock REUSES + # the corpse on the next run -> "FileNotFoundError: '/bin/sh'". Scrub it FIRST (best-effort: a + # no-op on the first run before the config exists) so mock recreates the chroot from the cached + # root; --scrub=bootstrap goes too (the genesis bootstrap chroot is part of the leak; genesis + # carries no --mock-uniqueext), so the fresh chroot re-bootstraps from the root cache. mock takes the buildroot lock for --scrub and BLOCKS (not + # skips) if a concurrent build holds it, but within a run the scrub is sequential before the + # build and the CD topology never runs a second same-target build at once; `timeout` bounds even + # a pathological wait so a stale lock can never hang the build. + my $genesis_scrub = "{ timeout 300 mock -r " . sh_quote("xCAT-genesis-base-$target") + . " --scrub=chroot --scrub=bootstrap >/dev/null 2>&1 || true; }"; + my $cmd = "mkdir -p $mktree && $genesis_scrub && HOME=" . sh_quote($genesis_home) . ' ' . join(' ', + 'perl', sh_quote("$xcat_src/buildrpms.pl"), '--package', 'xCAT-genesis-base', '--target', shell_quote($target), '--nproc', int($nproc), @@ -502,6 +717,7 @@ if (!$skip_build) { cmd => $cmd, cwd => $xcat_src, log => "$log_root/genesis-build.log", + scrub_cfg => "xCAT-genesis-base-$target", }; } @@ -512,44 +728,65 @@ if (!$skip_build) { ($max_build_workers && $max_build_workers >= 1) ? $max_build_workers : defined($parallel_builds) ? $parallel_builds : scalar(@build_steps); - run_build_steps_parallel( - steps => \@build_steps, - max_processes => $effective_parallel_builds, - ); + # Make --max-parallel a REAL cap. The perl builder is a single step that internally forks up + # to $effective_parallel_builds mock jobs of its own, so running it concurrently with the dep + # builders pushed live mock builds to ~2x the cap. Run it in its OWN phase, after the dep + # builders (which are quick) -- each phase then runs at most $effective_parallel_builds mock + # builds, so the cap holds, at a small bounded wall-clock cost. (The perl step sets no + # scrub_cfg and scrubs its own chroots; the scrub loop below still covers the dep/genesis steps.) + my @perl_steps = grep { $_->{id} eq 'perl' } @build_steps; + my @nonperl_steps = grep { $_->{id} ne 'perl' } @build_steps; + my @failed; + push @failed, run_build_steps_parallel( + steps => \@nonperl_steps, max_processes => $effective_parallel_builds, + ) if @nonperl_steps; + push @failed, run_build_steps_parallel( + steps => \@perl_steps, max_processes => $effective_parallel_builds, + ) if @perl_steps; + + # Reclaim each build step's mock chroot now that the step copied its RPMs/logs out to + # its --result-dir (collect_rpms reads those, never /var/lib/mock). mock's own cleanup + # leaves these chroots behind -- and keeps them entirely on failure -- so /var/lib/mock + # grows ~15-17G per run until the host fills and every dnf transaction fails for lack of + # space. Scrub each via `mock --scrub=chroot --scrub=bootstrap` (never rm): it takes the + # chroot lock, so a chroot still used by a concurrent build is refused and safely skipped. + # Both the build chroot and its per-uniqueext bootstrap are removed; the root cache stays + # for fast rebuilds. Perl packages are scrubbed inside mockbuild-perl-packages.pl (it + # derives its own per-package uniqueexts). + unless ($keep_buildroots) { + for my $s (@build_steps) { + next unless defined $s->{scrub_cfg}; + (my $slug = $s->{id}) =~ s/[^\w.-]+/-/g; + scrub_buildroot($s->{scrub_cfg}, $s->{scrub_uniqueext}, "$log_root/scrub-$slug.log"); + } + } + + # Zero-tolerance: any build step that failed fails the whole run -- genesis included. + # (xcat-core #7696 is merged: buildrpms.pl now exits 0 iff it actually produced the + # genesis rpm, so there is no cosmetic non-zero exit left to tolerate. The old workaround + # -- ignore a genesis failure when a matching rpm already exists in dist/ -- is gone; a + # stale artifact from a previous build must never mask a failed genesis build.) + die "FATAL: required build step(s) failed for $target: @failed\n" if @failed; } } +# The xCAT core is built by the xcat-core pipeline, NOT here -- so we deliberately do +# NOT collect the xCAT dist tree. Only the OS-dependent xCAT-genesis-base rpm (built by +# the genesis step above) is pulled out of it, individually, further below. my $xcat_rpms_dir = "$xcat_src/dist/$target/rpms"; -my $xcat_srpms_dir = "$xcat_src/dist/$target/srpms"; - -# In monolithic mode (no --skip-xcat) the whole xCAT core built here (incl. -# genesis-base) is collected into this repo. In the split pipeline (--skip-xcat, -# core built separately) the orchestrator (cluster-test.pl) routes -# xCAT-genesis-base from the xCAT dist tree into the per-EL dep repo itself -- -# robust to this script exiting non-zero on tolerated dep-builder failures -- so -# we deliberately do NOT collect the xCAT dist tree here. -if (!$skip_xcat) { - push @collect_roots, $xcat_rpms_dir; -} if ($skip_build) { - push @collect_roots, - "$repo_root/build-output/list3/elilo-xcat", - "$repo_root/build-output/list3/grub2-xcat", - "$repo_root/build-output/list3/ipmitool-xcat", - "$repo_root/build-output/list3/syslinux-xcat", - "$repo_root/build-output/list3/xnba-undi", - "$repo_root/build-output/list5/goconserver/$arch", - "$repo_root/goconserver-build-$arch/results/rpm", - "$repo_root/build-output/list6/perl/$arch", - "$repo_root/perl-list6/$arch"; + # Collect THIS target's previously-built artifacts from its own per-target build tree -- the + # same $build_root a normal build populates (collect_rpms recurses). NOT the legacy EL-agnostic + # build-output/list* dirs: those are scoped only by $arch, so an el8 rpm left there would be + # pulled into an el9/el10 repo, and with --target omitted the same rpms would be published into + # every EL repo. (--target is now required for --skip-build, see the option check above.) + push @collect_roots, $build_root; } push @collect_roots, @extra_collect_dirs; @collect_roots = uniq(@collect_roots); -my @srpm_collect_roots = (!$skip_xcat) - ? uniq(@collect_roots, $xcat_srpms_dir) - : uniq(@collect_roots); +my @srpm_collect_roots = uniq(@collect_roots); if ($genesis_release && !$dry_run) { remove_genesis_packages($repo_dir, 0); @@ -569,13 +806,15 @@ my ($copied, $skipped_src, $missing_roots) = collect_rpms( # Assert on what this run BUILT, before the Genesis release is added: the release is # installed from a verified directory rather than built here, so counting it first would # let a run whose builders all failed reach createrepo and the deployable tree, and fail -# much later in assert_required_deps naming packages instead of the failed builds. +# much later in the repo gate (verify_target_repo), naming missing packages instead of the +# failed builds. if (!$dry_run && $copied == 0) { die "No binary RPMs were collected. Check build logs and collection roots.\n"; } # Ensure the OS-dependent xCAT-genesis-base rpm (built by the genesis step above) -# lands in the dep repo even when the full xCAT core is built elsewhere (--skip-xcat). +# lands in the dep repo -- pull it individually out of the xcat-core dist tree (the +# rest of that tree, the full xCAT core, is built + published by the xcat-core pipeline). if (!$skip_genesis && !$dry_run) { for my $g (bsd_glob("$xcat_rpms_dir/xCAT-genesis-base-*.rpm")) { next if $g =~ /\.src\.rpm$/; @@ -585,6 +824,27 @@ if (!$skip_genesis && !$dry_run) { } } +# Repo completeness -- every required package present at its pinned version (a '*' pin accepts any), +# missing packages included -- is now gated ONCE, centrally, in deploy_target via verify_target_repo +# (the single consolidated gate; it also runs under --skip-build and validates the deployed repo). +# The only per-build check kept here is the CD --build-number bump: confirm it actually LANDED in the +# built rpms' Release, since validating %{VERSION} alone can't catch a silently un-bumped NVR (which +# deploy's additive rsync would then dedup away). Every built dep + perl package carries the suffix; +# xCAT-genesis-base is intentionally NOT bumped (kept in lockstep with xcat-core's genesis-scripts). +if (!$dry_run && $RELEASE_BUMP ne '') { + my @rmiss; + for my $pkg (required_pkgs([sort keys %req], $skip_genesis, $skip_perl, $skip_xcat_dep)) { + next if $pkg eq 'xCAT-genesis-base'; + my $rel = rpm_release($repo_dir, $pkg); + next if !defined $rel; # a missing rpm is caught by the completeness gate in deploy_target + push @rmiss, "$pkg: Release '$rel' is missing the CD bump '$RELEASE_BUMP'" + if index($rel, $RELEASE_BUMP) < 0; + } + die "FATAL: --build-number bump '$RELEASE_BUMP' did not land in built rpm(s) for $target:\n " + . join("\n ", @rmiss) . "\n" if @rmiss; + print "[manifest] Release bump '$RELEASE_BUMP' present on all built dep rpms for $target\n"; +} + print_step('Collect source RPM artifacts'); print "source collection roots:\n"; print " $_\n" for @srpm_collect_roots; @@ -735,16 +995,61 @@ sub deploy_target { my $dest = "$repo_dep/rh$rel/$tarch"; print_step("Deploy $tgt -> $dest"); return if $dry_run; - make_path($dest); - for my $rpm (bsd_glob("$src/*.rpm")) { - next if $rpm =~ /\.src\.rpm$/; - my $destination = "$dest/" . basename($rpm); - publish_file($rpm, $destination); + + # Stage the cell in a sibling temp dir, sign+index+verify it THERE, then atomically swap it into + # place. This makes the deploy self-cleaning and atomic (PR #62 review #3): + # - self-cleaning: the cell is rebuilt from scratch each run, so stale snap-NVR rpms from an + # earlier build never accumulate. Previously deploy copied ADDITIVELY into an existing $dest + # and relied on the pipeline pre-wiping rh/ -- a standalone run accumulated versions. + # - atomic + verify-before-replace: a failed sign/index/verify leaves the previously-published + # cell untouched, and no reader ever sees a half-written cell. + # The staging dir is a sibling of $dest, so the rename is a same-filesystem (atomic) move. The + # cross-arch genesis (--finalize-xcat-dep) runs later and re-populates the foreign-arch genesis, + # so rebuilding this single-arch cell from $src is correct. + make_path(dirname($dest)); + my $stage = "$dest.stage.$$"; + remove_tree($stage) if -d $stage; + make_path($stage); + my $ok = eval { + for my $rpm (bsd_glob("$src/*.rpm")) { + next if $rpm =~ /\.src\.rpm$/; + publish_file($rpm, "$stage/" . basename($rpm)); + } + # --genesis-release: the release itself is published ONCE into /common, not into + # each per-EL cell, so nothing from it is kept here -- drop any stale OpenEmbedded Genesis + # rpm an earlier layout left in the collection. On the STAGE, so the published cell is + # already correct when it is swapped in. + remove_genesis_packages($stage, 0) if $genesis_release; + sign_and_index_repo($stage); + write_dep_repo_metadata($stage, $rel, $tarch); + # Automatic completeness + signature gate on the freshly signed cell -- the single + # consolidated gate (verify_target_repo, the same one --verify-repo runs). Asserts every + # manifest-required package is present at its pinned version, the repomd signature verifies, + # AND every rpm is signed by the key. Runs on the STAGE so a failure never lands in $dest. + # Suppressible with --no-verify-repo for iteration/debug. + verify_target_repo($stage, $tgt) unless $no_verify_repo; + 1; + }; + if (!$ok) { + my $err = $@; + remove_tree($stage); # leave the previously-published cell exactly as it was + die $err; } - remove_genesis_packages($dest, 0) if $genesis_release; - assert_required_deps($dest, $info->{profile}{required}); - sign_and_index_repo($dest); - write_dep_repo_metadata($dest, $rel, $tarch); + + # Atomic replace: rename cannot overwrite a populated dir, so move the old cell aside, swap the + # staged cell in, then drop the old one. On a failed final rename, restore the old cell. + my $old = "$dest.old.$$"; + remove_tree($old) if -d $old; + if (-d $dest) { + rename($dest, $old) or die "Failed to move old cell $dest aside: $!\n"; + } + unless (rename($stage, $dest)) { + my $err = $!; + rename($old, $dest) if -d $old && !-d $dest; # best-effort restore + die "Failed to swap staged cell into $dest: $err\n"; + } + remove_tree($old) if -d $old; + my $n = scalar(grep { !/\.src\.rpm$/ } bsd_glob("$dest/*.rpm")); print "Deployed rh$rel/$tarch: $n rpms\n"; } @@ -763,12 +1068,54 @@ sub publish_genesis_common_repo { verify_genesis_release_packages('rpm', $COMMON_STAGE); sign_and_index_repo($COMMON_STAGE); write_common_repo_metadata($COMMON_STAGE); + # Gate the STAGE, so an incomplete shared repo is never swapped into place. Completeness only: + # the packages were verified against the release checksums as they were copied, and the deploy + # asserts every rpm's signature, but until now nothing checked that the repository being + # published actually carries the whole architecture set the manifest says it must. + verify_common_repo($COMMON_STAGE) unless $no_verify_repo; chmod(0755, $COMMON_STAGE) or die "Cannot make $COMMON_STAGE traversable: $!\n"; replace_common_repository($COMMON_STAGE, $dest); print "Published common Genesis repository: $published rpms\n"; } +#-------------------------------------------------------------------------------- + +=head3 verify_common_repo + + Assert the shared OpenEmbedded Genesis repository carries every package the manifest's [common] + section requires, at a version satisfying its pin. [common] is not a build target: it describes + the one repository published beside the per-EL cells, which no [] section covers. + + Arguments: + $dir - the repository to check (the staging directory, before it is swapped into place) + Returns: + 1, or dies listing every problem + +=cut + +#-------------------------------------------------------------------------------- +sub verify_common_repo { + my ($dir) = @_; + my $manifest = "$repo_root/packages-manifest.conf"; + my %MAN = read_manifest($manifest); + my %req = %{ $MAN{common} // {} }; + die "FATAL: no [common] section in $manifest -- cannot verify the shared Genesis repository\n" + if !%req; + + my @names = sort keys %req; + my %present = repo_present_versions($dir, \@names); + my %present_evr = map { $_ => rpm_evr($dir, $_) } @names; + my @problems = verify_repo_packages(\%req, \%present, \%present_evr, \&rpm_vercmp_segment); + if (@problems) { + print " - $_\n" for @problems; + die "FATAL: shared Genesis repo INCOMPLETE at $dir (" . scalar(@problems) . " problem(s))\n"; + } + print "[verify-repo] common complete: " . scalar(@names) + . " packages present + EVR-satisfied in $dir\n"; + return 1; +} + sub replace_common_repository { my ($staged, $destination) = @_; my $backup = "$repo_dep/.common.previous.$$"; @@ -842,8 +1189,8 @@ sub sign_and_index_repo { local $ENV{GNUPGHOME} = $gpg_home if $gpg_home; my $repomd = "$dir/repodata/repomd.xml"; unlink "$repomd.asc" if -f "$repomd.asc"; - run_simple(qq(gpg -a --detach-sign --default-key "$gpg_key_name" ) . shell_quote($repomd)); - run_simple(qq(gpg -a --export "$gpg_key_name" > ) . shell_quote("$repomd.key")); + run_simple("gpg -a --detach-sign --default-key " . sh_quote($gpg_key_name) . ' ' . sh_quote($repomd)); + run_simple("gpg -a --export " . sh_quote($gpg_key_name) . " > " . sh_quote("$repomd.key")); } } @@ -852,6 +1199,9 @@ sub write_dep_repo_metadata { my $baseurl = "https://xcat.org/files/xcat/repos/yum/devel/xcat-dep/rh$rel/$tarch"; my $gpgcheck = $gpg_sign ? 1 : 0; my $gpgkey_line = $gpg_sign ? "gpgkey=$baseurl/repodata/repomd.xml.key" : "# gpgkey="; + # repo_gpgcheck=1 makes clients verify the DETACHED repomd.xml signature (repomd.xml.asc) against + # gpgkey before trusting the metadata -- sign_and_index_repo produces both, so enforce it. Mirrors + # gpgcheck: off when the repo is unsigned. open my $r, '>', "$dir/xcat-dep.repo" or die "Cannot write $dir/xcat-dep.repo: $!\n"; print {$r} <<"EOF"; [xcat-dep] @@ -859,6 +1209,7 @@ name=xCAT 2 dependencies (rh$rel $tarch) baseurl=$baseurl enabled=1 gpgcheck=$gpgcheck +repo_gpgcheck=$gpgcheck $gpgkey_line EOF close $r; @@ -938,11 +1289,30 @@ EOF close $b; } + + +# Re-run createrepo_c on a repo whose rpm set changed, and (under --gpg-sign) re-sign + +# re-export repomd. Does NOT re-sign the rpms (cross_copy_genesis already did the copied +# one; the rest keep their build-time signatures). +sub reindex_and_sign_repo { + my ($dir) = @_; + run_simple(createrepo_c_cmd($dir)); + if ($gpg_sign) { + local $ENV{GNUPGHOME} = $gpg_home if $gpg_home; + my $repomd = "$dir/repodata/repomd.xml"; + unlink "$repomd.asc" if -f "$repomd.asc"; + run_simple("gpg -a --detach-sign --default-key " . sh_quote($gpg_key_name) . ' ' . sh_quote($repomd)); + run_simple("gpg -a --export " . sh_quote($gpg_key_name) . " > " . sh_quote("$repomd.key")); + } +} + sub usage { return <<"USAGE"; Usage: $0 [options] -Build xcat-dep and xCAT RPMs, consolidate binary/source artifacts, run createrepo, and create tarballs. +Build xcat-dep RPMs (dep packages, perl packages, and the OS-dependent xCAT-genesis-base), +consolidate binary/source artifacts, run createrepo, and create tarballs. The full xCAT core +is built separately by the xcat-core pipeline, not here. Options: --repo-root PATH xcat-dep repository root (default: script directory) @@ -954,6 +1324,25 @@ Options: --repo-dep PATH Override the deployable output root; rh8/rh9/rh10/ and common are assembled and signed here (default: /xcat-dep) --force-unlock Remove a stale /.lock before acquiring it + --finalize-xcat-dep Post-build cross-arch genesis mode (builds nothing). Requires + --x86_64-repo and --ppc64le-repo. For each matching /x86_64 and + /ppc64le repo pair, copies the noarch xCAT-genesis-base-ppc64 + (the ppc64le genesis; xCAT names it -ppc64 via tarch, no big-endian + code) into the x86_64 repo and xCAT-genesis-base-x86_64 into the + ppc64le repo (dropping any stale foreign-arch genesis), then + re-indexes + re-signs. Restores the 2.17 cross-arch genesis + (issue #7610). Honors --gpg-sign/--gpg-key-name/--gpg-home. Use alone. + --x86_64-repo PATH (finalize) x86_64 repo root holding /x86_64 (e.g. rh9/x86_64) + --ppc64le-repo PATH (finalize) ppc64le repo root holding /ppc64le + --verify-repo PATH Standalone completeness + signature gate over the per-target repo at PATH + (builds nothing). Asserts every package packages-manifest.conf requires for + the target is present at a version satisfying its pin AND that the repomd + is signed by --gpg-key-name; exits 0 if complete, or lists each MISSING/ + VERSION/UNSIGNED/WRONGKEY problem and fails. The target is derived from the path + (.../rh/ -> alma+epel--) unless --target is given; the + manifest and gpg key/home come from the usual options. Use alone. + --no-verify-repo Suppress the AUTOMATIC post-build completeness+signature gate that runs + after each target's repo is finalized (default: verification ON) --gpg-sign Sign RPMs and repomd.xml in every published repository --gpg-key-name NAME GPG key name (default: "xCAT Signing Key") --gpg-home PATH GNUPGHOME for signing (default: system keyring) @@ -964,17 +1353,18 @@ Options: --nproc N Parallel jobs for buildrpms.pl (default: 1) --parallel-builds N Max concurrent top-level build steps within one EL target (default: auto) --parallel-targets N Concurrent EL targets (rh8/rh9/rh10). 0/auto = all at once, 1 = serial, - N = cap at N. Each target is fully output-isolated (default: auto) + N = cap at N. Each target is fully output-isolated (default: 1 = serial) --max-parallel N Global cap on concurrent mock builds across ALL targets, to avoid oversubscribing the host. Split evenly across active targets. 0/auto = host nproc (default: auto) --run-id ID Run identifier suffix (default: derived from build timestamp) --build-timestamp EPOCH Unix epoch for deterministic builds (default: Gitepoch or git log) - --skip-install Skip install/smoke tests in child builder scripts --skip-build Skip all build steps and only collect/create repo/tarballs --skip-xcat-dep Skip xcat-dep mockbuild.pl package steps --skip-perl Skip perl package build step - --skip-xcat Skip xCAT buildrpms.pl step + --install-deps Install this host's build prerequisites (package manager + the perl + modules the script loads), verify each module now loads, then exit. + Run once per build host, as root. Use alone. --skip-genesis Skip the existing per-EL Genesis image build --skip-createrepo Skip createrepo --skip-tarball Skip binary/SRPM tarball creation @@ -986,8 +1376,8 @@ Options: Notes: - Run this script as root on the build host. - ARCH is derived from: uname -m (or from the forcearch target) - - Top-level parallel queue includes xcat-dep mockbuild.pl steps, perl builder, - and ../xcat-core/buildrpms.pl. + - Top-level parallel queue includes xcat-dep mockbuild.pl steps, the perl builder, + and the xCAT-genesis-base build (../xcat-core/buildrpms.pl --package xCAT-genesis-base). - Child mockbuild scripts are invoked with per-step mock --uniqueext values to avoid lock collisions on the same mock config. - If --target is omitted, it is deduced from /etc/os-release: @@ -1039,26 +1429,53 @@ sub run_step { } } +# Scrub a single mock buildroot via mock's own --scrub (never rm). mock takes the buildroot lock for +# the scrub, so a concurrent build holding it makes mock BLOCK until release rather than corrupt a live +# chroot (this can never rm a chroot out from under a running build). Failures (already scrubbed or +# config missing) are tolerated -- a cleanup hiccup must never fail the build. Scrubs both the +# build chroot and its per-uniqueext bootstrap chroot (each build step gets its own bootstrap, so +# both must go or /var/lib/mock still leaks). The shared root cache under /var/cache/mock is kept, +# so rebuilds stay fast. $uniqueext is optional (genesis has none). +sub scrub_buildroot { + my ($cfg, $uniqueext, $log) = @_; + return if !defined $cfg || $cfg eq ''; + my $ext = (defined $uniqueext && $uniqueext ne '') + ? ' --uniqueext ' . sh_quote($uniqueext) : ''; + eval { + run_step( + step => "Scrub chroot $cfg$ext", + cmd => "mock -r " . sh_quote($cfg) . $ext . " --scrub=chroot --scrub=bootstrap", + log => $log, + ); + 1; + } or do { + warn "WARN: chroot scrub failed (tolerated) for $cfg$ext: $@"; + }; +} + sub run_build_steps_parallel { my (%args) = @_; my $steps = $args{steps} // []; my $max_processes = $args{max_processes} // 1; return if !@{$steps}; - # Individual dep-builder failures are TOLERATED (some packages are el-/arch-pinned or - # have dead upstream source URLs, e.g. elilo on el10, perl-Sys-Virt on el8, a moved grub2 - # src.rpm). We collect whatever built and assert the REQUIRED set later (assert_required_deps), - # matching the historical build behaviour. + # Returns the ids of any steps that failed; the caller (build_one_target) enforces + # zero-tolerance -- ANY failed step fails the whole run, genesis included, with no special-case. + # We build only packages required for the target (per packages-manifest.conf), so there is no + # "expected to fail on this arch/el" case left to tolerate. There is likewise no genesis + # exception: since xcat-core #7696, buildrpms.pl exits 0 iff it produced the genesis rpm, so a + # non-zero genesis exit is a real failure (the old "tolerate if the rpm is already present" + # workaround is gone -- a stale artifact must never mask a failed build). if ($dry_run || $max_processes <= 1 || @{$steps} == 1) { - my $serial_failures = 0; + my @failed; for my $step (@{$steps}) { my $ok = eval { run_step(%{$step}); 1 }; next if $ok; - $serial_failures++; - warn "WARN: build step failed (tolerated): $step->{step}\n" . ($@ // ''); + warn "ERROR: build step failed: $step->{step}\n" . ($@ // ''); + push @failed, (defined($step->{id}) && $step->{id} ne '' ? $step->{id} : $step->{step}); } - assert_build_progress(scalar(@{$steps}), $serial_failures); - return; + assert_build_progress(scalar(@{$steps}), scalar(@failed)); + return @failed; } my $workers = $max_processes; @@ -1113,41 +1530,279 @@ sub run_build_steps_parallel { push @lines, "$id (exit=$f->{exit}, signal=$f->{signal}, core_dump=$f->{core_dump})"; } - # Tolerated: warn, don't die. The REQUIRED set is asserted after collection/deploy. - warn "WARN: some build steps failed (tolerated; required deps asserted after deploy):\n " - . join("\n ", @lines) . "\n"; + warn "ERROR: build step(s) failed:\n " . join("\n ", @lines) . "\n"; } assert_build_progress(scalar(@{$steps}), scalar(keys %failed)); + my @failed_ids = sort keys %failed; + return @failed_ids; } -# Individual failures are tolerated because some packages are el- or arch-pinned. ALL of them +# The caller enforces zero tolerance per required package (verify_target_repo). ALL steps # failing is a different thing: the builder itself is unusable (no mock, a broken chroot, no # network), this invocation produced nothing, and every package the run would go on to publish -# would come from somewhere other than this build. +# would come from somewhere other than this build -- say that, instead of naming the missing +# packages later. sub assert_build_progress { my ($attempted, $failures) = @_; return unless every_step_failed($attempted, $failures); die "FATAL: every build step failed ($failures/$attempted). Check the build logs.\n"; } -# have_rpm: is there a non-src rpm named -... under $dir? -sub have_rpm { - my ($dir, $name) = @_; - my @m = grep { !/\.src\.rpm$/ } bsd_glob("$dir/${name}-*.rpm"); - return scalar(@m) > 0; + + + + + + +# repo_present_versions: thin disk layer for the repo gate. Given a built repo dir and the list of +# required package names, return %present = (name => rpm_version($dir, $name)) for each -- reusing the +# EXISTING rpm_version so genesis's arch-suffixed naming resolves exactly as the in-line manifest pin +# check does. rpm_version returns undef for an absent package, which verify_repo_packages then reports +# as MISSING. Pure disk read; the decision itself lives in verify_repo_packages. +sub repo_present_versions { + my ($dir, $names) = @_; + my %present; + $present{$_} = rpm_version($dir, $_) for @$names; + return %present; } -# assert_required_deps: the per-EL dep repo is unusable without these (the target profile's -# required set), so a MISSING one is fatal even though individual builder failures are -# tolerated above. genesis-base is required unless --skip-genesis. -sub assert_required_deps { - my ($dir, $required) = @_; - my @req = @{$required}; - push @req, 'xCAT-genesis-base' unless $skip_genesis; - my @missing = grep { !have_rpm($dir, $_) } @req; - die "FATAL: required deps missing from $dir: @missing\n" if @missing; - print "[deps] required set present in $dir: @req\n"; +# gpg_key_fingerprint: resolve a gpg key NAME (e.g. "xCAT Signing Key") to its primary-key +# fingerprint in the given keyring, so the expected and observed signing identities are compared in +# the SAME form (a fingerprint). Falls back to the name itself when it cannot be resolved. +sub gpg_key_fingerprint { + my ($keyname, $home) = @_; + my $h = ($home ne '') ? ' --homedir ' . sh_quote($home) : ''; + my $out = `gpg$h --with-colons --fingerprint --list-keys ${\ sh_quote($keyname)} 2>/dev/null` // ''; + # Collect the PRIMARY-key fingerprint of every key matching $keyname (the fpr line right after a + # 'pub' record; subkey fprs follow 'sub' and are ignored). Return undef -- not a guess -- when the + # key is absent (unresolved) or when MORE THAN ONE key matches the name (ambiguous): the caller + # then hard-fails SIGKEY rather than comparing against a possibly-wrong key. + my (@fprs, $want); + for my $line (split /\n/, $out) { + if ($line =~ /^pub:/) { $want = 1; } + elsif ($line =~ /^sub:/) { $want = 0; } + elsif ($want && $line =~ /^fpr:+([0-9A-Fa-f]+):/) { push @fprs, $1; $want = 0; } + } + my $fingerprint; + $fingerprint = $fprs[0] if @fprs == 1; + return $fingerprint; +} + +# gpg_key_ids: all acceptable key ids (lowercased) for a signing key NAME -- the primary key id AND +# every subkey id, in both 16-hex (long) and 8-hex (short) forms. rpm header signatures report the +# signing SUBKEY id, so the per-rpm gate accepts any id belonging to the key rather than one exact +# fingerprint. Returns a hashref set (empty if the key can't be listed). +sub gpg_key_ids { + my ($keyname, $home) = @_; + my $h = ($home ne '') ? ' --homedir ' . sh_quote($home) : ''; + my $out = `gpg$h --with-colons --list-keys ${\ sh_quote($keyname)} 2>/dev/null` // ''; + my %ids; + for my $line (split /\n/, $out) { + my @f = split /:/, $line; + next unless ($f[0] // '') =~ /^(?:pub|sub)$/ && defined $f[4] && $f[4] ne ''; + my $id = $f[4]; + $ids{ lc $id } = 1; + $ids{ lc substr($id, -16) } = 1 if length($id) > 16; + $ids{ lc substr($id, -8) } = 1 if length($id) > 8; + } + return \%ids; +} + +# rpm_signer_keyid: the signing key id (lowercased hex) of a built rpm's header signature, or undef +# when the rpm is not signed. Reads the RSA (or DSA) header pgpsig and pulls the "Key ID " field. +sub rpm_signer_keyid { + my ($rpm) = @_; + my $keyid; + for my $tag (qw(RSAHEADER DSAHEADER)) { + my $out = `rpm -qp --qf '%{$tag:pgpsig}' ${\ sh_quote($rpm)} 2>/dev/null` // ''; + if ($out =~ /Key ID\s+([0-9A-Fa-f]+)/i) { $keyid = lc($1); last; } + } + return $keyid; +} + +# rpm_evr: the single distinct EPOCH:VERSION-RELEASE of package $name's binary rpm(s) in $dir (epoch +# defaults to 0 when the header carries none), or undef if none match. Mirrors rpm_version's dedup: +# more than one distinct EVR means a stale artifact was not cleaned before the build (a version pin +# could then pass against the wrong rpm). genesis's x86_64 + ppc64 rpms share one EVR, so a normal +# pair is a single entry. +sub rpm_evr { + my ($dir, $name) = @_; + my $glob = ($name eq 'xCAT-genesis-base') + ? "$dir/xCAT-genesis-base-*.rpm" + : "$dir/${name}-*.rpm"; + my %evrs; + for my $f (sort glob($glob)) { + next if $f =~ /\.src\.rpm$/ || $f =~ /-debug(?:info|source)-/; + my $n = `rpm -qp --qf '%{name}' ${\ sh_quote($f)} 2>/dev/null`; + my $match = ($name eq 'xCAT-genesis-base') + ? ($n =~ /^xCAT-genesis-base-/) : ($n eq $name); + next unless $match; + my $evr = `rpm -qp --qf '%{epochnum}:%{version}-%{release}' ${\ sh_quote($f)} 2>/dev/null`; + chomp $evr; + $evrs{$evr} = 1 if $evr ne ''; + } + my $evr; + return $evr unless %evrs; + die "Multiple EVRs of $name present in $dir: " . join(', ', sort keys %evrs) + . " (stale artifact not cleaned before the build)\n" if keys(%evrs) > 1; + ($evr) = keys %evrs; + return $evr; +} + +# rpm_vercmp_segment: ONE rpmvercmp segment comparison via rpm's own lua binding, returning -1/0/1. +# Used as the injected comparator for MockBuildUtils::evr_cmp so the EVR gate uses rpm's canonical +# version algorithm (epoch/release composition is done in evr_cmp). Long-bracket the args so any +# version char (. _ ~ ^ +) passes through literally; rpm versions never contain the ]==] sequence. +sub rpm_vercmp_segment { + my ($a, $b) = @_; + $a = '' unless defined $a; + $b = '' unless defined $b; + my $out = `rpm --eval '%{lua:print(rpm.vercmp([==[$a]==],[==[$b]==]))}' 2>/dev/null`; + chomp $out; + die "FATAL: rpm.vercmp gave no result for '$a' vs '$b'\n" unless $out =~ /^-?\d+$/; + return $out <=> 0; +} + +# verify_rpms_checksig: cryptographically verify EVERY binary rpm in $dir with `rpmkeys --checksig` +# against an ISOLATED keyring holding only the signing key. This is the RPM-native integrity + origin +# check: it verifies each rpm's header/payload digests AND that the signature is by this key (NOKEY / +# NOT OK => a real failure, since the key IS imported). Returns @problems. +sub verify_rpms_checksig { + my ($dir, $keyname, $home) = @_; + my @rpms = grep { !/\.src\.rpm$/ } glob("$dir/*.rpm"); + return () unless @rpms; + require_command('rpmkeys'); + require_command('gpg'); + my $tmpdb = tempdir('rpmkeys-XXXXXXXX', TMPDIR => 1, CLEANUP => 1); + my $h = ($home ne '') ? ' --homedir ' . sh_quote($home) : ''; + my $keyfile = "$tmpdb/pubkey.asc"; + system("gpg$h --batch --yes -a --export " . sh_quote($keyname) . ' > ' . sh_quote($keyfile) . ' 2>/dev/null'); + return ("SIGKEY: cannot export public key '$keyname' for rpmkeys --checksig") if !-s $keyfile; + my $dbopt = '--dbpath ' . sh_quote($tmpdb); + system("rpmkeys $dbopt --import " . sh_quote($keyfile) . ' >/dev/null 2>&1') == 0 + or return ("SIGKEY: rpmkeys --import of '$keyname' into the temp keyring failed"); + my @problems; + for my $rpm (@rpms) { + my $out = `rpmkeys $dbopt --checksig -v ${\ sh_quote($rpm)} 2>&1`; + push @problems, rpmkeys_checksig_problem(basename($rpm), $? >> 8, $out); + } + return @problems; +} + +# repomd_observed_signer: run gpg --verify on the detached repomd signature and extract the identity +# of the key that actually signed it, as a primary-key fingerprint (the last field of the VALIDSIG +# status line). Returns '' when the .asc is absent or verification fails (both read as "unsigned"). +sub repomd_observed_signer { + my ($asc, $file, $home) = @_; + return '' unless -f $asc && -f $file; + my $h = ($home ne '') ? ' --homedir ' . sh_quote($home) : ''; + my $out = `gpg$h --status-fd=1 --verify ${\ sh_quote($asc)} ${\ sh_quote($file)} 2>/dev/null` // ''; + # An EXPIRED or REVOKED key, or an expired signature, still emits VALIDSIG -- reject those + # explicitly so a no-longer-trustworthy signature is a problem, not a pass. A fully-good signature + # emits GOODSIG; the degraded cases emit EXPKEYSIG/REVKEYSIG/EXPSIG instead. + return '' if $out =~ /^\[GNUPG:\]\s+(?:EXPKEYSIG|REVKEYSIG|EXPSIG)\b/m; + for my $line (split /\n/, $out) { + # VALIDSIG ; the trailing field is the primary fpr. + if ($line =~ /^\[GNUPG:\]\s+VALIDSIG\s+(.*\S)\s*$/) { + my @f = split ' ', $1; + return $f[-1]; + } + } + return ''; +} + +# verify_target_repo: the completeness + signature gate for ONE built per-target repo -- the single +# source of truth for "is this repo shippable?", replacing the old assert_required_deps + in-line +# version-pin loop. It does the IO (manifest parse, rpm_version, gpg --verify) and delegates every +# DECISION to the two PURE helpers: verify_repo_packages (missing/version) and verify_repo_signature +# (unsigned/wrongkey). Both problem lists are merged. Prints a one-line OK, or dies listing every +# problem. Both the automatic post-build gate (deploy_target) and the standalone --verify-repo mode +# call this, so there is exactly one gate implementation. +sub verify_target_repo { + my ($dir, $tgt, $manifest, $sig_required) = @_; + $manifest //= "$repo_root/packages-manifest.conf"; + my %MAN = read_manifest($manifest); + my %req = %{ $MAN{$tgt} // {} }; + die "FATAL: no manifest section for target '$tgt' in $manifest\n" if !%req; + # The WHOLE manifest, deliberately -- the --skip-* flags are NOT applied here. They say what + # this INVOCATION built; they never say what the verified repository may be missing. Honouring + # them let a repo with no xCAT-genesis-base pass whenever the verifying run happened to carry + # --skip-genesis (PR #62 review). A package an earlier run built is still expected to be here. + my @names = sort keys %req; + my %present = repo_present_versions($dir, \@names); + # Full EPOCH:VERSION-RELEASE per package, so a manifest EVR constraint (e.g. genesis-base + # '>= 2:2.18.0', which %{VERSION}-only matching cannot enforce -- 2.* would accept a pre-2.18 + # genesis) is checked with rpm's own version algorithm (PR #62 review). rpm_vercmp_segment is + # rpm's rpmvercmp; evr_cmp composes epoch/version/release around it. + my %present_evr = map { $_ => rpm_evr($dir, $_) } @names; + my %expected = map { $_ => $req{$_} } @names; + my @problems = verify_repo_packages(\%expected, \%present, \%present_evr, \&rpm_vercmp_segment); + + # Signature gate: the IO (gpg) lives here; the decision is the pure verify_repo_signature. The + # pipeline always signs, so a signed repo's repomd MUST be signed by --gpg-key-name. We resolve + # that key to a fingerprint and extract the fingerprint that actually signed repomd, then compare. + # Skipped with a printed note only when no gpg key/home is configured (nothing to check against). + if ($gpg_sign || $gpg_home ne '') { + require_command('gpg'); + my $repomd = "$dir/repodata/repomd.xml"; + my $asc = "$repomd.asc"; + my $exp_fpr = gpg_key_fingerprint($gpg_key_name, $gpg_home); + # STRICT: the CLI key MUST resolve to exactly one fingerprint so we can confirm it signed the + # repo. Undef => absent or ambiguous in the keyring -> we cannot verify -> hard fail, never a + # presence-only pass. + if (!defined $exp_fpr) { + push @problems, "SIGKEY: cannot resolve --gpg-key-name '$gpg_key_name' to a single fingerprint (in the $gpg_home keyring?)"; + } else { + my %exp_sig = ('repomd' => $exp_fpr); + my %obs_sig = ('repomd' => repomd_observed_signer($asc, $repomd, $gpg_home)); + push @problems, verify_repo_signature(\%exp_sig, \%obs_sig); + + # Per-rpm signature gate: a signed repomd over an unsigned or foreign-signed rpm still + # makes DNF reject that package at install time, so verify EVERY binary rpm -- not just the + # metadata -- is signed by this key (rpm reports the signing subkey id; accept any id of + # the key). Closes the "approves a repo DNF later rejects" gap (PR #62 review #4). + require_command('rpm'); + # (a) RPM-native crypto verification: rpmkeys --checksig against an isolated keyring + # holding only this key verifies every rpm's digests AND that the signature is by the key. + push @problems, verify_rpms_checksig($dir, $gpg_key_name, $gpg_home); + # (b) Explicit signer-id origin check kept alongside: assert each rpm's header signature + # key id is one of this key's ids (primary/subkey). + my $accept = gpg_key_ids($gpg_key_name, $gpg_home); + if (!%$accept) { + push @problems, "SIGKEY: cannot list key ids for '$gpg_key_name' to verify per-rpm signatures"; + } else { + my @rpm_sigs = map { [ basename($_), rpm_signer_keyid($_) ] } + grep { !/\.src\.rpm$/ } glob("$dir/*.rpm"); + push @problems, verify_rpm_signatures(\@rpm_sigs, $accept); + } + } + } elsif ($sig_required) { + # Standalone --verify-repo advertises a signature check; with no keyring we cannot resolve the + # CLI key or read the signer, so refuse rather than silently pass (which would be a false PASS). + push @problems, "SIGKEY: --verify-repo requires --gpg-key-name + --gpg-home to check the repomd signature (none configured)"; + } else { + print "[verify-repo] $tgt: no gpg key/home configured -- skipping repomd signature check\n"; + } + + if (@problems) { + print " - $_\n" for @problems; + die "FATAL: repo INCOMPLETE for $tgt at $dir (" . scalar(@problems) . " problem(s))\n"; + } + print "[verify-repo] $tgt complete: " . scalar(@names) + . " required packages present + EVR-satisfied, repomd + every rpm checksig-verified, in $dir\n"; + return 1; +} + +# derive_target_from_repo_path: map a deployed per-target repo path .../rh/ to its manifest +# target section name alma+epel--. Returns undef when the path lacks that rh/ tail, +# so the standalone --verify-repo mode can require an explicit --target instead. +sub derive_target_from_repo_path { + my ($dir) = @_; + my $tgt; + return $tgt unless defined $dir; + $tgt = "alma+epel-$1-$2" if $dir =~ m{/rh(\d+)/([^/]+)/*$}; + return $tgt; } sub reset_staging_repo { @@ -1338,42 +1993,19 @@ sub resolve_mock_cfg { 'centos-stream' => 'centos-stream', rocky => 'rocky', ); - my $candidate = "${os_id}+epel-${rel}-${arch}"; - my $rc = system("mock -r " . shell_quote($candidate) . " --print-root-path >/dev/null 2>&1"); - if ($rc == 0) { - return $candidate; - } - if (exists $short_forms{$os_id}) { - my $short = $short_forms{$os_id}; - $candidate = "${short}+epel-${rel}-${arch}"; - $rc = system("mock -r " . shell_quote($candidate) . " --print-root-path >/dev/null 2>&1"); - if ($rc == 0) { - print "Mock config resolved (short form): $candidate\n"; + # Resolve by CONFIG-FILE existence, not by running `mock --print-root-path`: the latter can fail + # transiently (bootstrap chroot setup, a concurrent mock holding a lock) and made el10 flakily + # "resolve" to the long form that has no .cfg. Checking /etc/mock/.cfg is deterministic. + for my $id ($os_id, (exists $short_forms{$os_id} ? ($short_forms{$os_id}) : ())) { + my $candidate = "${id}+epel-${rel}-${arch}"; + if (-f "/etc/mock/${candidate}.cfg") { + print "Mock config resolved: $candidate\n" if $id ne $os_id; return $candidate; } } - die "Could not find mock config for ${os_id}+epel-${rel}-${arch}\n"; -} - -sub build_mock_uniqueext { - my ($run, $seq, $label) = @_; - - my $run_part = defined($run) ? $run : 'run'; - $run_part =~ s/[^A-Za-z0-9_.-]+/-/g; - $run_part =~ s/^-+|-+$//g; - $run_part = 'run' if $run_part eq ''; - $run_part = substr($run_part, -24) if length($run_part) > 24; - - my $label_part = defined($label) ? $label : 'step'; - $label_part =~ s/[^A-Za-z0-9_.-]+/-/g; - $label_part =~ s/^-+|-+$//g; - $label_part = 'step' if $label_part eq ''; - $label_part = substr($label_part, 0, 20) if length($label_part) > 20; - - my $idx = defined($seq) ? int($seq) : 0; - $idx = 0 if $idx < 0; - - return sprintf("mba-%02d-%s-%s", $idx, $run_part, $label_part); + my $short = $short_forms{$os_id} // $os_id; + die "Could not find mock config for ${os_id}+epel-${rel}-${arch} " + . "(tried /etc/mock/${os_id}+epel-${rel}-${arch}.cfg and /etc/mock/${short}+epel-${rel}-${arch}.cfg)\n"; } sub resolve_xcat_source { @@ -1509,3 +2141,13 @@ sub uniq { my %seen; return grep { defined($_) && !$seen{$_}++ } @_; } + +sub slurp_chomp { + my ($path) = @_; + open my $fh, '<', $path or die "Cannot read $path: $!\n"; + my $line = <$fh>; + close $fh; + chomp $line if defined $line; + return $line // ''; +} + diff --git a/mockbuild-perl-packages.pl b/mockbuild-perl-packages.pl old mode 100644 new mode 100755 index 2869aa0..8d8ee4b --- a/mockbuild-perl-packages.pl +++ b/mockbuild-perl-packages.pl @@ -15,14 +15,17 @@ my $mock_cfg = ''; my $noarch_mock_cfg = ''; my $target_arch = ''; my $mock_uniqueext = ''; +my $keep_buildroots = 0; my $result_dir = ''; my $log_dir = ''; my $packages_csv = ''; my $epel_gap = 0; my $jobs = 0; -my $skip_install = 0; -my $allow_erasing = 0; my $build_timestamp; +# CD version bump: appended to the Release of the srpm-mode packages (HTML-Form, IO-Stty, +# Net-Telnet), which build from a committed .src.rpm and so are NOT covered by mockbuild-all's +# in-tree spec bump. Spec-mode packages get bumped in-tree upstream, so we leave those alone. +my $release_suffix = ''; GetOptions( 'work-dir=s' => \$work_dir, @@ -30,14 +33,14 @@ GetOptions( 'noarch-mock-cfg=s' => \$noarch_mock_cfg, 'target-arch=s' => \$target_arch, 'mock-uniqueext=s' => \$mock_uniqueext, + 'keep-buildroots!' => \$keep_buildroots, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, 'packages=s' => \$packages_csv, 'epel-gap!' => \$epel_gap, 'jobs=i' => \$jobs, - 'skip-install!' => \$skip_install, - 'allow-erasing!' => \$allow_erasing, 'build-timestamp=i' => \$build_timestamp, + 'release-suffix=s' => \$release_suffix, ) or die usage(); die "Run as root (current uid=$>)\n" if $> != 0; @@ -300,10 +303,6 @@ $jobs = 1 if $jobs < 1; if (@packages && $jobs > scalar(@packages)) { $jobs = scalar(@packages); } -if (!$skip_install && $jobs > 1) { - print "INFO: --skip-install is disabled; forcing --jobs 1 to avoid host dnf lock contention\n"; - $jobs = 1; -} make_path($result_dir); make_path($log_dir); @@ -322,8 +321,7 @@ print "mock_uniqueext: " . ($mock_uniqueext ne '' ? $mock_uniqueext : '(none)') print "epel_gap: $epel_gap\n"; print "packages: " . join(', ', @packages) . "\n"; print "jobs: $jobs\n"; -print "skip_install:$skip_install\n"; -print "allow_erasing:$allow_erasing\n"; +print "release_suffix:" . ($release_suffix ne '' ? $release_suffix : '(none)') . "\n"; print_step("Mock config check"); run("mock -r " . sh_quote($mock_cfg) . $mock_uniqueext_opt . " --print-root-path >/dev/null"); @@ -333,14 +331,17 @@ run("mock -r " . sh_quote($noarch_mock_cfg) . $mock_uniqueext_opt . " --print-ro my @failed; my @passed; my @summary_lines; +my @built_roots; # pkg + mock cfg + uniqueext of every chroot this run made print_step("Build packages"); print "parallel jobs: $jobs\n"; +my %child_rc; # pkg => child exit code; the AUTHORITATIVE pass/fail for that build my $pm = Parallel::ForkManager->new($jobs); $pm->run_on_finish( sub { my ($pid, $exit_code, $ident) = @_; my $label = defined $ident ? $ident : "pid=$pid"; + $child_rc{$ident} = $exit_code if defined $ident; # record it; do not trust status.txt alone my $state = $exit_code == 0 ? 'PASS' : "FAIL(rc=$exit_code)"; print "[$label] $state\n"; } @@ -356,6 +357,11 @@ for my $wave (build_waves(\@packages)) { my $pkg_uniqueext = package_uniqueext($mock_uniqueext, ++$idx, $pkg); my @needs = grep { $selected{$_} } @{ $cfg->{needs} // [] }; + # The noarch packages of a forcearch target build in the native chroot, so scrub the + # chroot the package was actually built in. + my $pkg_mock_cfg = $cfg->{rpm_arch} eq 'noarch' ? $noarch_mock_cfg : $mock_cfg; + push @built_roots, { pkg => $pkg, mock_cfg => $pkg_mock_cfg, uniqueext => $pkg_uniqueext }; + my $pid = $pm->start($pkg); next if $pid; my $ok = build_package( @@ -364,21 +370,57 @@ for my $wave (build_waves(\@packages)) { work_dir => $work_dir, result_dir => $result_dir, log_dir => $log_dir, - mock_cfg => ($cfg->{rpm_arch} eq 'noarch' ? $noarch_mock_cfg : $mock_cfg), + mock_cfg => $pkg_mock_cfg, mock_uniqueext => $pkg_uniqueext, arch => $target_arch, host_arch => $arch, needs => \@needs, - skip_install => $skip_install, - allow_erasing => $allow_erasing, + release_suffix => $release_suffix, ); + unless ($keep_buildroots) { + # Reclaim ONLY this package's build chroot here (it's the ~GB disk hog). --scrub=chroot + # is uniqueext-local, so it never touches a concurrent sibling. Do NOT --scrub=bootstrap + # here: despite the --uniqueext, mock's bootstrap scrub removes the CONFIG-LEVEL shared + # bootstrap cache (/var/cache/mock/-bootstrap/, keyed by config name, NOT + # uniqueext). Doing that mid-batch deletes the cache a still-starting sibling is about to + # bind-mount into its own bootstrap root -> `mount rc=32` and a spurious build failure + # (observed: perl-Sys-Virt's buildsrpm raced a faster sibling's post-build bootstrap + # scrub). The shared bootstrap is reclaimed once below, after ALL workers finish, when + # nothing can be binding it. + (my $ps = $pkg) =~ s/[^\w.-]+/-/g; + system("mock -r " . sh_quote($pkg_mock_cfg) . " --uniqueext " . sh_quote($pkg_uniqueext) + . " --scrub=chroot > " . sh_quote("$log_dir/scrub-$ps.log") . " 2>&1"); + } $pm->finish($ok ? 0 : 1); } $pm->wait_all_children; } +# Now that every worker has exited, reclaim the per-uniqueext bootstrap roots + the shared +# config-level bootstrap cache. Serialized and post-join, so no scrub can race a concurrent +# bind (that race is exactly what the per-package note above avoids). Best-effort: the first +# scrub drops /var/cache/mock/-bootstrap; each also removes its uniqueext bootstrap root. +unless ($keep_buildroots) { + for my $root (@built_roots) { + (my $ps = $root->{pkg}) =~ s/[^\w.-]+/-/g; + system("mock -r " . sh_quote($root->{mock_cfg}) . " --uniqueext " . sh_quote($root->{uniqueext}) + . " --scrub=bootstrap >> " . sh_quote("$log_dir/scrub-$ps.log") . " 2>&1"); + } +} + for my $pkg (@packages) { my $status_file = "$log_dir/$pkg/status.txt"; + # The child exit code is authoritative: a package is PASS only if its worker exited 0 AND wrote + # a PASS status this run. A missing/non-zero child result is FAIL regardless of any status.txt + # (which could be a stale PASS left in a reused log dir, or unwritten because the worker crashed). + my $rc = $child_rc{$pkg}; + if (!defined $rc || $rc != 0) { + push @failed, $pkg; + push @summary_lines, defined $rc + ? "$pkg FAIL worker exited rc=$rc" + : "$pkg FAIL no worker result recorded"; + next; + } if (!-f $status_file) { push @failed, $pkg; push @summary_lines, "$pkg FAIL missing status file ($status_file)"; @@ -434,8 +476,7 @@ sub build_package { my $arch = $args{arch}; my $host_arch = $args{host_arch} // $arch; my $needs = $args{needs} // []; - my $skip_install = $args{skip_install}; - my $allow_erasing = $args{allow_erasing}; + my $release_suffix = $args{release_suffix}; my $pkg_run_dir = "$work_dir/$pkg"; my $pkg_result = "$result_dir/$pkg"; @@ -446,6 +487,9 @@ sub build_package { make_path($pkg_run_dir); make_path($pkg_result); make_path($pkg_log); + # Clear any status/error left by an earlier run in a reused log dir BEFORE building, so a crash + # between here and the status write below can never leave a stale PASS the aggregate would trust. + unlink $status_file, "$pkg_log/error.txt"; my $det_mock_cfg = create_deterministic_mock_cfg($mock_cfg, $SOURCE_DATE_EPOCH, $pkg_run_dir); @@ -473,6 +517,35 @@ sub build_package { if ($cfg->{mode} eq 'srpm') { $srpm_path = select_srpm($cfg->{srpm_globs}); die "Could not locate source RPM for $pkg\n" if !$srpm_path; + # CD version bump: these packages build from a committed .src.rpm, so the in-tree + # spec Release bump (mockbuild-all) never reaches them. Re-stamp here: unpack the + # srpm, append the suffix to its spec's Release (KEEPING %{?dist}, exactly like the + # spec-mode packages -> e.g. 19%{?dist} -> 19%{?dist}.snap...N), and roll a fresh + # srpm. With no suffix (non-CD run), rebuild the committed srpm unchanged. + if ($release_suffix ne '') { + my $ext = "$pkg_run_dir/restamp"; + for my $d (qw(BUILD BUILDROOT RPMS SOURCES SPECS SRPMS)) { make_path("$ext/$d"); } + run("rpm -i --define " . sh_quote("_topdir $ext") . ' ' . sh_quote($srpm_path) + . " > " . sh_quote("$pkg_log/srpm-unpack.log") . " 2>&1"); + my ($espec) = sort glob("$ext/SPECS/*.spec"); + die "No spec found after unpacking srpm for $pkg\n" if !$espec; + append_release_suffix($espec, $release_suffix); + my $restamp_result = "$pkg_run_dir/restamp-srpm"; + make_path($restamp_result); + run( + "mock -r " . sh_quote($det_mock_cfg) . $mock_uniqueext_opt . + " --buildsrpm --spec " . sh_quote($espec) . + " --sources " . sh_quote("$ext/SOURCES") . + " --define " . sh_quote("use_source_date_epoch_as_buildtime 1") . + " --define " . sh_quote("clamp_mtime_to_source_date_epoch 1") . + " --define " . sh_quote("_buildhost xcat-build") . + " --resultdir " . sh_quote($restamp_result) . + " > " . sh_quote("$pkg_log/mock-restamp-buildsrpm.log") . " 2>&1" + ); + my @restamped = sort glob("$restamp_result/*.src.rpm"); + die "No re-stamped SRPM produced for $pkg in $restamp_result\n" if !@restamped; + $srpm_path = $restamped[-1]; + } } else { my $spec = $cfg->{spec}; die "Missing spec for $pkg: $spec\n" if !-f $spec; @@ -591,7 +664,7 @@ sub build_package { } } - if (!$skip_install && $cfg->{rpm_arch} eq 'native' && $arch ne $host_arch) { + if ($cfg->{rpm_arch} eq 'native' && $arch ne $host_arch) { # A cross-built XS module cannot be loaded on this host: install the rpm into the # (emulated) build chroot and import the module there. my $module = $cfg->{module}; @@ -606,14 +679,6 @@ sub build_package { "$pkg_log/smoke-perl-module.log"); die "Perl module import failed for $pkg ($module) in the $arch chroot, rc=$rc_mod\n" if $rc_mod != 0; } - elsif (!$skip_install) { - my $install_cmd = "dnf -y install "; - $install_cmd .= "--allowerasing " if $allow_erasing; - run($install_cmd . sh_quote($main_rpm)); - my $module = $cfg->{module}; - my $rc_mod = run_capture_rc("perl -M$module -e 1", "$pkg_log/smoke-perl-module.log"); - die "Perl module import failed for $pkg ($module), rc=$rc_mod\n" if $rc_mod != 0; - } $summary = "$pkg PASS main_rpm=" . basename($main_rpm); $ok = 1; @@ -688,11 +753,36 @@ Usage: $0 [options] --epel-gap Also build the perl deps of xCAT that EL takes from EPEL (for an arch without EPEL, e.g. riscv64) --build-timestamp EPOCH Unix epoch for SOURCE_DATE_EPOCH (deterministic builds) - --skip-install Skip dnf install + perl module import checks - --allow-erasing Allow dnf to erase conflicting packages during install smoke tests + --release-suffix STR CD bump appended to the Release of the srpm-mode packages that build + from a committed .src.rpm (HTML-Form, IO-Stty, Net-Telnet) USAGE } +# Append $suffix (e.g. ".snap202607161200.57") to the first Release: line of $spec, in place. +# Mirrors mockbuild-all's bump_dep_release_suffix: case-insensitive (some specs use lowercase +# `release:`), preserves any %{?dist} macro on the line, and is idempotent (a line already +# carrying this exact suffix is left as-is). +sub append_release_suffix { + my ($spec, $suffix) = @_; + my $qs = quotemeta($suffix); + open my $in, '<', $spec or die "open $spec: $!\n"; + my @lines = <$in>; + close $in; + my $changed = 0; + for my $line (@lines) { + next unless $line =~ /^Release:\s*\S/i; + last if $line =~ /$qs\s*$/; # already stamped + $line =~ s/(^Release:\s*\S+)/$1$suffix/i; + $changed = 1; + last; # only the first Release: line + } + die "No Release: line to stamp in $spec\n" if !$changed && !grep { /^Release:\s*\S/i } @lines; + return if !$changed; + open my $out, '>', $spec or die "open> $spec: $!\n"; + print {$out} @lines; + close $out; +} + sub select_srpm { my ($globs_ref) = @_; for my $g (@{$globs_ref}) { diff --git a/packages-manifest.conf b/packages-manifest.conf new file mode 100644 index 0000000..d97d07a --- /dev/null +++ b/packages-manifest.conf @@ -0,0 +1,191 @@ +# Per-target required xcat-dep package manifest. +# +# One [section] per mockbuild-all target (matches --target). Each entry is +# = +# where is the builder/package name (the dep builder name, the perl +# package name, or xCAT-genesis-base) and is one of: +# - an exact Version (e.g. 1.8.18) -- the built %{VERSION} must equal it; +# - a shell-style glob (e.g. 2.*) -- the built %{VERSION} must match it (* and ?); +# - '*' -- any version is accepted; +# - an EVR constraint (e.g. >= 2:2.18.0, >= 0.04-5, = 1.8.18) -- an operator +# (>=, >, <=, <, =) followed by an [epoch:]version[-release]. The built rpm's +# full EPOCH:VERSION-RELEASE is compared with rpm's own version algorithm +# (rpm.vercmp; epoch numeric, then version, then release; release ignored when +# the constraint omits it). Use this to enforce a minimum that a %{VERSION} +# glob cannot -- e.g. a release floor (perl-IO-Stty >= 0.04-5) or an Epoch. +# A bare version pin matches %{VERSION} only (Release carries the per-EL dist tag +# elN and the snap, so it is not pinned there). Bump a pin here when the +# corresponding in-tree source is bumped. +# +# xCAT-genesis-base is pinned as '>= 2:2.18.0' (an EVR floor, not an exact version): +# its Version is NOT owned by xcat-dep -- it is whatever xcat-core the genesis build +# compiles against (XCAT_CORE_REF), so it walks with the paired core (2.18.x, 2.19.x, +# ...). The floor still walks (accepts any 2.18+ genesis) but, unlike the old '2.*' +# glob, REJECTS a pre-2.18 genesis -- xCAT-genesis-scripts Requires xCAT-genesis-base +# >= 2:2.18.0 (Epoch 2), which a 2.17.x genesis would violate. genesis-base carries +# Epoch 2, so the epoch in the constraint is enforced too. +# +# The pins that xCAT states as release-sensitive Requires are EVR floors taken VERBATIM from +# xcat-core's own specs, so the gate rejects a build whose Version is right but whose Release is +# older than xCAT accepts: goconserver (xCAT.spec), xnba-undi, syslinux-xcat, ipmitool-xcat +# (xCAT.spec / xCATsn.spec), perl-HTTP-Async, perl-Net-HTTPS-NB and perl-IO-Stty +# (xCAT-server.spec). Where two specs disagree the STRONGER floor is used. +# +# grub2-xcat is deliberately NOT converted. xCAT-server.spec asks for +# '>= 2.02-0.76.el7.1.snap201905160255', but the grub2-xcat this repository builds -- and that both +# published channels ship -- is 1.0-2, which cannot satisfy it (1 < 2). Encoding that Requires here +# would fail every build on a discrepancy that lives in xcat-core, not in this manifest, so the pin +# stays on the Version actually produced and the discrepancy is reported upstream instead. +# +# mockbuild-all.pl reads this file and, per target, builds ONLY the listed +# packages -- a package not listed for a target is not built for it. Any listed +# package that fails to build (or builds a mismatched version) fails the whole +# run (no tolerated failures). +# +# +# Per-EL perl set: a perl module is required from xcat-dep only on the releases +# where neither the base OS nor EPEL provides it; on the other releases EPEL/OS +# supplies it (so `dnf install xCAT` pulls the OS/EPEL copy, not xcat-dep's). +# perl-Sys-Virt omitted on el8 -- provided by EPEL on AlmaLinux 8. +# perl-HTML-Form only on el8 -- provided by the OS/EPEL on el9/el10. +# perl-Crypt-SSLeay only on el10 -- provided by the OS/EPEL on el8/el9. +# perl-Net-Telnet only on el10 -- provided by the OS/EPEL on el8/el9. +# +# conserver-xcat is NOT pulled by `dnf install xCAT` (goconserver superseded it), +# but it is built for every target anyway because some users still deploy it. + +[alma+epel-8-x86_64] +conserver-xcat=8.2.1 +elilo-xcat=3.14 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 +perl-HTML-Form=6.07 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +xCAT-genesis-base=>= 2:2.18.0 + +[alma+epel-8-ppc64le] +conserver-xcat=8.2.1 +elilo-xcat=3.14 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 +perl-HTML-Form=6.07 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +xCAT-genesis-base=>= 2:2.18.0 + +[alma+epel-9-x86_64] +conserver-xcat=8.2.1 +elilo-xcat=3.14 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +perl-Sys-Virt=11.10.0 +xCAT-genesis-base=>= 2:2.18.0 + +[alma+epel-9-ppc64le] +conserver-xcat=8.2.1 +elilo-xcat=3.14 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +perl-Sys-Virt=11.10.0 +xCAT-genesis-base=>= 2:2.18.0 + +[alma+epel-10-x86_64] +conserver-xcat=8.2.1 +elilo-xcat=3.14 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 +perl-Crypt-SSLeay=0.72 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +perl-Net-Telnet=3.04 +perl-Sys-Virt=11.10.0 +xCAT-genesis-base=>= 2:2.18.0 + +[alma+epel-10-ppc64le] +conserver-xcat=8.2.1 +elilo-xcat=3.14 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 +perl-Crypt-SSLeay=0.72 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +perl-Net-Telnet=3.04 +perl-Sys-Virt=11.10.0 +xCAT-genesis-base=>= 2:2.18.0 + +# rocky-10-riscv64-xcat is the forcearch (cross-built) EL10 riscv64 target. It differs from the +# EPEL-fed EL10 sections above in two ways (see BUILD.md, "riscv64"): +# - the x86-only boot components (elilo-xcat, syslinux-xcat, xnba-undi) are not built for it; +# - riscv64 has no EPEL, so the perl deps EL10 otherwise takes from EPEL are built here too +# (perl-Crypt-Blowfish ... perl-Path-Class below). perl-Path-Class is a build dep of +# perl-Crypt-SSLeay only. +# The per-EL perl set is the EL10 one, so perl-HTML-Form is absent here as well. +# xCAT-genesis-base is deliberately not listed: the riscv64 repo is built with --skip-genesis. +[rocky-10-riscv64-xcat] +conserver-xcat=8.2.1 +goconserver=>= 0.3.3-snap202011021058 +grub2-xcat=1.0 +ipmitool-xcat=>= 1.8.18-4 +perl-Crypt-SSLeay=0.72 +perl-HTTP-Async=>= 0.30-3 +perl-IO-Stty=>= 0.04-5 +perl-Net-HTTPS-NB=>= 0.14-3 +perl-Net-Telnet=3.04 +perl-Sys-Virt=11.10.0 +perl-Crypt-Blowfish=2.14 +perl-Crypt-CBC=2.33 +perl-Crypt-Rijndael=1.13 +perl-Digest-SHA1=2.13 +perl-Expect=1.35 +perl-Mail-Sender=0.903 +perl-Net-DNS=0.80 +perl-Net-IP=1.26 +perl-Path-Class=0.37 + +# [common] is NOT a build target. It describes the SHARED repository the OpenEmbedded Genesis +# release is published into (/common), which lives beside the per-EL cells and is +# therefore invisible to every [] section above. Without it nothing asserted the published +# shared repo was COMPLETE: its packages were checked only as they were copied, against the release +# checksums, so a repository that lost one afterwards would publish quietly. +# +# The floor tracks the paired xcat-core, like xCAT-genesis-base: these are built FROM xcat-core, so +# the version walks with it (2.18.x, 2.19.x, ...). Epoch 0 -- unlike genesis-base, these packages +# carry no Epoch. Listing every architecture is the point: the release is only useful if a +# management node can serve an image for any of them. +[common] +xCAT-genesis-openembedded-x86=>= 2.18.0 +xCAT-genesis-openembedded-x86_64=>= 2.18.0 +xCAT-genesis-openembedded-ppc64=>= 2.18.0 +xCAT-genesis-openembedded-ppc64le=>= 2.18.0 +xCAT-genesis-openembedded-armv7hf=>= 2.18.0 +xCAT-genesis-openembedded-aarch64=>= 2.18.0 +xCAT-genesis-openembedded-riscv64=>= 2.18.0 diff --git a/syslinux/mockbuild.pl b/syslinux/mockbuild.pl index cc320c1..dda65f3 100755 --- a/syslinux/mockbuild.pl +++ b/syslinux/mockbuild.pl @@ -13,34 +13,28 @@ my $repo_root = abs_path("$script_dir/.."); my $pkg_dir = "$repo_root/syslinux"; my $spec_file = "$pkg_dir/syslinux-xcat.spec"; -my $source_url = 'https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.xz'; my $source_file = ''; my $work_dir = '/tmp/syslinux-xcat-mockbuild'; my $mock_cfg = ''; my $mock_uniqueext = ''; my $result_dir = "$repo_root/build-output/list3/syslinux-xcat"; my $log_dir = "$repo_root/build-logs/list3/syslinux-xcat"; -my $skip_install = 0; -my $skip_upstream_download = 0; my $build_timestamp; GetOptions( - 'source-url=s' => \$source_url, 'source-file=s' => \$source_file, 'work-dir=s' => \$work_dir, 'mock-cfg=s' => \$mock_cfg, 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, - 'skip-upstream-download!' => \$skip_upstream_download, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); die "Run as root (current uid=$>)\n" if $> != 0; die "Missing spec file: $spec_file\n" if !-f $spec_file; -for my $bin (qw(wget mock rpmbuild rpm dnf file bash grep cut)) { +for my $bin (qw(mock rpmbuild rpm dnf file bash grep)) { run("command -v " . sh_quote($bin) . " >/dev/null 2>&1"); } @@ -90,10 +84,7 @@ print "pkg_name: $pkg_name\n"; print "version: $version\n"; print "mock_cfg: $mock_cfg\n"; print "mock_uniqueext: " . ($mock_uniqueext ne '' ? $mock_uniqueext : '(none)') . "\n"; -print "source_url: $source_url\n"; print "source_file:$source_file\n"; -print "skip_install: $skip_install\n"; -print "skip_upstream_download: $skip_upstream_download\n"; make_path($result_dir); make_path($log_dir); @@ -101,21 +92,24 @@ make_path($log_dir); print_step("Mock config check"); run("mock -r " . sh_quote($mock_cfg) . $mock_uniqueext_opt . " --print-root-path >/dev/null"); -if (!$skip_upstream_download) { - print_step("Download upstream source"); - run("wget --spider " . sh_quote($source_url)); - run("wget -O " . sh_quote($source_path) . " " . sh_quote($source_url)); - - my $sha = capture("sha256sum " . sh_quote($source_path) . " | cut -d ' ' -f1"); - my $meta_file = "$log_dir/upstream-source.txt"; - open my $mfh, '>', $meta_file or die "Cannot write $meta_file: $!\n"; - print {$mfh} "url=$source_url\n"; - print {$mfh} "file=$source_path\n"; - print {$mfh} "sha256=$sha\n"; - close $mfh; - print "Downloaded source: $source_path\n"; - print "SHA256: $sha\n"; -} +print_step("Verify tracked source archive"); +# Upstream source (documented for provenance; NOT fetched at build time -- see below): +# https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.xz +# The syslinux source (syslinux-.tar.xz, Source0) is tracked in the repo, has the +# syslinux-/ top-level that %setup -n expects, and is consumed directly by mock (--sources +# $pkg_dir below). There is nothing to download: the old fetch re-downloaded this SAME tracked file +# and rewrote it IN PLACE, and the checkout is shared between the two arch build hosts building at +# once -- so the in-place rewrite raced the other host's concurrent syslinux build, which could +# read the file mid-write and get a truncated archive. We only READ it now, so concurrent builds +# can never race on it. Fail loudly (do NOT silently re-fetch) if the checkout is missing/broken. +die "Tracked syslinux source missing: $source_path (incomplete checkout?)\n" if !-f $source_path; +my $top = capture( + "tar -tf " . sh_quote($source_path) . + " 2>/dev/null | grep -E '^(\\./)?syslinux-' | head -n1 || true" +); +die "Tracked syslinux source is not a syslinux-*/ source tree: $source_path\n" + if $top eq ''; +print "Using tracked source archive (read-only, no fetch, no shared write): $source_path\n"; print_step("Verify spec assets"); for my $asset (@all_assets) { @@ -229,48 +223,6 @@ for my $log (qw(build.log root.log state.log hw_info.log installed_pkgs.log)) { or die "Failed to copy $src to $log_dir: $!\n"; } -if (!$skip_install) { - print_step("Install RPM(s) and run smoke tests"); - run("dnf -y install " . sh_quote($xcat_rpm)); - - my $pxe_file = '/opt/xcat/share/xcat/netboot/syslinux/pxelinux.0'; - die "Missing installed PXE file: $pxe_file\n" if !-f $pxe_file; - - my $file_log = "$log_dir/smoke-file.log"; - my $qf_log = "$log_dir/smoke-rpm-qf.log"; - my $rc_file = run_capture_rc("file $pxe_file", $file_log); - my $rc_qf = run_capture_rc("rpm -qf $pxe_file", $qf_log); - - die "Smoke check failed: file returned $rc_file\n" if $rc_file != 0; - die "Smoke check failed: rpm -qf returned $rc_qf\n" if $rc_qf != 0; - - my $qf_out = slurp($qf_log); - die "Installed file is not owned by syslinux-xcat:\n$qf_out\n" - if $qf_out !~ /^syslinux-xcat-/m; - - # EL10 hosts may already carry syslinux-nonlinux, which conflicts with - # directly installing the rebuilt syslinux payload. The xcat subpackage - # is the artifact we need to smoke-test on the host; repository-level - # dependency validation happens later in the full install phase. - my $syslinux_help_log = "$log_dir/smoke-syslinux-help.log"; - if (-x '/usr/bin/syslinux') { - my $rc_help = run_capture_rc("/usr/bin/syslinux --help", $syslinux_help_log); - my $help_out = slurp($syslinux_help_log); - die "syslinux --help returned unexpected rc=$rc_help\n" - if $rc_help != 0 && $rc_help != 1; - die "syslinux --help output missing expected usage text\n" - if $help_out !~ /usage|syslinux/i; - } - - my $summary = "$log_dir/smoke-summary.txt"; - open my $sfh, '>', $summary or die "Cannot write $summary: $!\n"; - print {$sfh} "pxe_file=$pxe_file\n"; - print {$sfh} "rc_file=$rc_file\n"; - print {$sfh} "rc_qf=$rc_qf\n"; - print {$sfh} "main_rpm_available=" . ($main_rpm ? 1 : 0) . "\n"; - close $sfh; -} - print_step("Completed"); print "syslinux-xcat RPM: $xcat_rpm\n"; print "Artifacts: $result_dir\n"; @@ -280,7 +232,6 @@ exit 0; sub usage { return <<"USAGE"; Usage: $0 [options] - --source-url URL Upstream tarball URL (default: $source_url) --source-file FILE Source filename stored in syslinux/ (default: inferred from spec) --work-dir PATH Temporary work dir (default: $work_dir) --mock-cfg NAME Mock config (default: +epel-10-) @@ -288,8 +239,6 @@ Usage: $0 [options] --result-dir PATH Output RPM/SRPM directory (default: $result_dir) --log-dir PATH Log directory (default: $log_dir) --build-timestamp EPOCH SOURCE_DATE_EPOCH for deterministic builds - --skip-upstream-download Skip wget download step - --skip-install Skip dnf install + smoke tests USAGE } diff --git a/t/common-repo-gate.t b/t/common-repo-gate.t new file mode 100644 index 0000000..9658e99 --- /dev/null +++ b/t/common-repo-gate.t @@ -0,0 +1,131 @@ +#!/usr/bin/perl +# The shared OpenEmbedded Genesis repository (xcat-dep/common) is published outside the per-target +# cells, so the per-target manifest sections never described it and nothing asserted it was COMPLETE +# once published. Its packages were only checked as they were copied, against the release checksums. +# +# This drives the real mockbuild-all.pl publish path and asserts on the repository it leaves behind: +# a complete release publishes and is gated against the manifest's [common] section, and a release +# missing one architecture is refused rather than published. +use strict; +use warnings; +use Test::More; +use FindBin qw($RealBin); +use File::Temp qw(tempdir); +use File::Path qw(make_path); +use File::Copy qw(copy); +use lib "$RealBin/.."; +use MockBuildUtils qw(read_manifest); + +my $SCRIPT = "$RealBin/../mockbuild-all.pl"; +my $RELEASE = '/opt/xcat-ci-shared/builds/genesis-openembedded-initial-20260825/release'; +plan skip_all => 'mockbuild-all.pl not found' unless -f $SCRIPT; +plan skip_all => 'no Genesis release fixture' unless -d "$RELEASE/rpm"; +# root, like every other test that drives mockbuild-all.pl: the script refuses to run otherwise, +# and the CI builder (XCAT_GENESIS_CI) is root. +plan skip_all => 'rpm tooling and a root Linux builder required' + unless $^O eq 'linux' + && $> == 0 + && !system('sh', '-c', 'command -v rpm >/dev/null 2>&1') + && !system('sh', '-c', 'command -v createrepo_c >/dev/null 2>&1') + && !system('sh', '-c', 'command -v rpmbuild >/dev/null 2>&1'); + +my $tmp = tempdir(CLEANUP => 1); +my $target = 'alma+epel-10-' . do { my $m = `uname -m`; chomp $m; $m }; + +# The shipped manifest must describe the shared repo, else nothing can gate it. +{ + my %m = read_manifest("$RealBin/../packages-manifest.conf"); + ok($m{common} && %{ $m{common} }, 'the shipped manifest has a [common] section'); + is(scalar(keys %{ $m{common} // {} }), 7, + '... naming every architecture the release must carry'); +} + +# fixture_rpm: a minimal noarch rpm, built once, standing in for a compiled dep. +my $FIXTURE; +sub fixture_rpm { + return $FIXTURE if $FIXTURE; + my $top = "$tmp/rpmbuild"; + make_path("$top/SPECS"); + open my $fh, '>', "$top/SPECS/fixture.spec" or die $!; + print $fh <<'SPEC'; +Name: ipmitool-xcat +Version: 1.8.18 +Release: 4 +Summary: fixture +License: EPL +BuildArch: noarch +%description +fixture package standing in for a compiled dependency +%install +mkdir -p %{buildroot}/usr/share/ipmitool-xcat +%files +/usr/share/ipmitool-xcat +SPEC + close $fh; + system('rpmbuild', '--quiet', '-bb', '--define', "_topdir $top", "$top/SPECS/fixture.spec") == 0 + or die "cannot build the fixture rpm\n"; + ($FIXTURE) = glob("$top/RPMS/noarch/ipmitool-xcat-1.8.18-4.noarch.rpm"); + die "fixture rpm not produced\n" unless $FIXTURE && -f $FIXTURE; + return $FIXTURE; +} + +# run_publish($release_dir) -> ($exit, $output, $common_dir) +sub run_publish { + my ($release, $tag) = @_; + my $out = "$tmp/$tag"; + make_path("$out/root", "$out/collect"); + # Something to collect, so the run gets past the "built nothing" guard. It must NOT be an + # OpenEmbedded package: collect_rpms drops those when --genesis-release is given (they come from + # the release, not from the build), so collecting one would leave the run with nothing. + copy(fixture_rpm(), "$out/collect/") or die $!; + open my $fh, '>', "$out/root/packages-manifest.conf" or die $!; + # the cell carries exactly the fixture dep, so the per-target gate runs for real too + print $fh "[$target]\nipmitool-xcat=1.8.18\n"; + # the shared repo's own section, copied from the shipped manifest so the test uses the real one + my %m = read_manifest("$RealBin/../packages-manifest.conf"); + print $fh "\n[common]\n"; + print $fh "$_=$m{common}{$_}\n" for sort keys %{ $m{common} // {} }; + close $fh; + my $cmd = join(' ', map { my $x = $_; $x =~ s/'/'"'"'/g; "'$x'" } + ($^X, $SCRIPT, '--repo-root', "$out/root", '--output', "$out/build", + '--repo-dep', "$out/repo", '--target', $target, '--run-id', $tag, + '--build-timestamp', '1787672536', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--skip-tarball', + '--collect-dir', "$out/collect", '--genesis-release', $release)) . ' 2>&1'; + my $log = `$cmd`; + return ($? >> 8, $log, "$out/repo/common"); +} + +# ---- a complete release publishes, and says it was gated ----------------------------------------- +{ + my ($rc, $out, $common) = run_publish($RELEASE, 'full'); + is($rc, 0, 'a complete release publishes') or diag($out); + is(scalar(grep { !/\.src\.rpm$/ } glob("$common/*.rpm")), 7, + 'the published shared repo carries every architecture'); + like($out, qr/\[verify-repo\] common complete/, 'the shared repo is gated against [common]'); +} + +# ---- an incomplete release is refused, and publishes nothing -------------------------------------- +{ + my $partial = "$tmp/partial-release"; + make_path("$partial/rpm", "$partial/srpm"); + for my $f (glob("$RELEASE/rpm/*.rpm"), glob("$RELEASE/srpm/*.rpm")) { + next if $f =~ /riscv64/; # drop one architecture + my ($sub) = $f =~ m{/(rpm|srpm)/[^/]+$}; + copy($f, "$partial/$sub/") or die $!; + } + copy("$RELEASE/release.manifest", $partial) or die $!; + # SHA256SUMS without the dropped arch, so the release itself still self-describes consistently + open my $in, '<', "$RELEASE/SHA256SUMS" or die $!; + open my $o, '>', "$partial/SHA256SUMS" or die $!; + while (<$in>) { print {$o} $_ unless /riscv64/ } + close $in; close $o; + + my ($rc, $out, $common) = run_publish($partial, 'partial'); + isnt($rc, 0, 'a release missing an architecture is refused'); + ok(!-d $common || !glob("$common/*.rpm"), + '... and nothing is published into the shared repository'); +} + +done_testing(); diff --git a/t/genesis_openembedded_consumer.t b/t/genesis_openembedded_consumer.t index 964840f..9574623 100644 --- a/t/genesis_openembedded_consumer.t +++ b/t/genesis_openembedded_consumer.t @@ -56,7 +56,7 @@ if ($ENV{XCAT_GENESIS_CI}) { } SKIP: { - skip 'RPM repository tools require a root Linux builder', 69 + skip 'RPM repository tools require a root Linux builder', 64 unless $^O eq 'linux' && $> == 0 && command_exists('rpmbuild') @@ -68,6 +68,7 @@ SKIP: { test_legacy_rpm_consumer(); test_partial_rpm_release(); test_failed_build_release(); + test_skip_build_collects_results(); test_dry_run_release(); test_rpm_repository_lock(); test_rpm_signal_cleanup(); @@ -86,6 +87,25 @@ SKIP: { done_testing(); +# mockbuild-all.pl builds and gates each target against /packages-manifest.conf, and a +# target with no section there is fatal -- so these runs, which use a synthetic target, need a +# section to exist at all. Its CONTENT is deliberately not meaningful: the dependency packages here +# are copies of one rpm, so no set of names describes them the way a real manifest describes a real +# build. The runs therefore pass --no-verify-repo and the completeness gate is covered where it can +# be tested honestly, against purpose-built rpms, in t/verify-repo-el.t. What these runs exercise is +# the Genesis release path. +# +# They pass --skip-genesis rather than the removed --skip-xcat: this script no longer builds the +# xCAT core, so xcat-core's buildrpms.pl is required only for the per-EL xCAT-genesis-base build. +sub write_target_manifest { + my ($root, $target) = @_; + make_path($root); + write_binary( + "$root/packages-manifest.conf", + "[$target]\n" . rpm_package_name(capture_command('uname', '-m')) . "=*\n", + ); +} + sub test_rpm_consumer { my $release_root = make_package_release("$tmp/rpm", 'rpm'); my $package = "xCAT-genesis-openembedded-x86_64-$version-$release.noarch.rpm"; @@ -126,7 +146,7 @@ sub test_rpm_consumer { my $dependencies = "$tmp/rpm-dependencies"; my $scratch_repo_root = "$tmp/rpm-repo-root"; make_rpm_dependencies($dependencies, "$release_root/rpm/$package"); - make_path($scratch_repo_root); + write_target_manifest($scratch_repo_root, $target); write_binary( "$dependencies/xCAT-genesis-openembedded-x86_64-$version-old.noarch.rpm", 'stale OpenEmbedded RPM', @@ -151,7 +171,8 @@ sub test_rpm_consumer { '--target', $target, '--run-id', 'consumer', '--build-timestamp', $epoch, - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--genesis-release', $release_root, '--collect-dir', $dependencies, @@ -255,7 +276,8 @@ SH '--target', $target, '--run-id', 'publication-failure', '--build-timestamp', $epoch, - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--genesis-release', $release_root, '--collect-dir', $dependencies, @@ -513,7 +535,8 @@ sub test_signed_common_rpm_repository { my $gpg_home = "$tmp/rpm-signing-key"; my $identity = 'xCAT repository test '; make_rpm_dependencies($dependencies, "$release_root/rpm/$package"); - make_path($scratch_repo_root, $gpg_home); + make_path($gpg_home); + write_target_manifest($scratch_repo_root, $target); chmod(0700, $gpg_home) or die $!; system( 'gpg', '--batch', '--homedir', $gpg_home, '--passphrase', '', @@ -537,8 +560,9 @@ sub test_signed_common_rpm_repository { '--target', $target, '--run-id', 'signed-consumer', '--build-timestamp', $epoch, - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', - '--skip-genesis', '--skip-createrepo', '--skip-tarball', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', + '--skip-createrepo', '--skip-tarball', '--collect-dir', $dependencies, '--genesis-release', $release_root, '--gpg-sign', '--gpg-key-name', $identity, '--gpg-home', $gpg_home, @@ -577,7 +601,7 @@ sub test_legacy_rpm_consumer { my $target = 'test+epel-10-' . capture_command('uname', '-m'); my $deployed = "$output/xcat-dep/rh10/" . capture_command('uname', '-m'); make_rpm_dependencies($dependencies, "$release_root/rpm/$package"); - make_path($scratch_repo_root); + write_target_manifest($scratch_repo_root, $target); write_binary("$scratch_repo_root/Gitepoch", ''); my @perl_lib; @@ -595,7 +619,8 @@ sub test_legacy_rpm_consumer { '--output', $output, '--target', $target, '--run-id', 'legacy', - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--collect-dir', $dependencies, ); @@ -647,7 +672,8 @@ sub test_partial_rpm_release { '--target', $target, '--run-id', 'partial', '--build-timestamp', $epoch, - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--genesis-release', $release_root, ); @@ -689,12 +715,10 @@ sub test_failed_build_release { my $scratch_repo_root = "$tmp/rpm-empty-root"; my $collected = "$tmp/rpm-empty-collect"; my $run_repo = "$output/mockbuild-all/$target-empty/repo/" . capture_command('uname', '-m'); - my $results = "$output/mockbuild-all/$target-empty/build-results/ipmitool-xcat"; my $stale = "$run_repo/ipmitool-xcat-0-stale.noarch.rpm"; - my $kept = "$results/ipmitool-xcat-0-earlier.noarch.rpm"; - make_path($common, $scratch_repo_root, $collected, $run_repo, $results); + make_path($common, $scratch_repo_root, $collected, $run_repo); + write_target_manifest($scratch_repo_root, $target); write_binary($stale, 'package left by an earlier run'); - write_binary($kept, 'build output an earlier run produced'); my $log = "$tmp/rpm-empty.log"; my $status = run_capture( @@ -705,7 +729,8 @@ sub test_failed_build_release { '--target', $target, '--run-id', 'empty', '--build-timestamp', $epoch, - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--genesis-release', $release_root, '--collect-dir', $collected, @@ -718,8 +743,43 @@ sub test_failed_build_release { 'a run that built nothing publishes no release package'); ok(!-e $stale, 'a package left by an earlier run is cleared from the staging repository'); - ok(-e $kept, - 'a run that skips building keeps the build results it collects from'); +} + +# --skip-build collects THIS target's previously built artifacts out of its own build-results tree, +# so a run that skips building must not destroy the tree it collects from. (Where this test came +# from the --skip-build roots were the legacy build-output/list* directories and build-results was +# only ever kept, never collected -- which is why the empty-collection case above can no longer +# leave an rpm there.) +sub test_skip_build_collects_results { + my $release_root = make_package_release("$tmp/rpm-kept", 'rpm'); + my $package = "xCAT-genesis-openembedded-x86_64-$version-$release.noarch.rpm"; + my $output = "$tmp/kept-output"; + my $target = 'test+epel-10-' . capture_command('uname', '-m'); + my $scratch_repo_root = "$tmp/rpm-kept-root"; + my $results = "$output/mockbuild-all/$target-kept/build-results/ipmitool-xcat"; + my $kept = "$results/ipmitool-xcat-1.noarch.rpm"; + my $deployed = "$output/xcat-dep/rh10/" . capture_command('uname', '-m'); + make_path($results); + copy("$release_root/rpm/$package", $kept) or die $!; + write_target_manifest($scratch_repo_root, $target); + + my $log = "$tmp/rpm-kept.log"; + my $status = run_capture( + $log, + $^X, $rpm_consumer, + '--repo-root', $scratch_repo_root, + '--output', $output, + '--target', $target, + '--run-id', 'kept', + '--build-timestamp', $epoch, + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', + '--skip-createrepo', '--skip-tarball', + ); + + is($status, 0, 'a run that skips building collects its own build results'); + ok(-e $kept, 'the build results it collects from are kept'); + ok(-e "$deployed/" . basename($kept), 'the collected package reaches the deployed repo'); } sub test_dry_run_release { @@ -731,7 +791,7 @@ sub test_dry_run_release { my $dependencies = "$tmp/rpm-dry-dependencies"; my $scratch_repo_root = "$tmp/rpm-dry-root"; make_rpm_dependencies($dependencies, "$release_root/rpm/$package"); - make_path($scratch_repo_root); + write_target_manifest($scratch_repo_root, $target); my $log = "$tmp/rpm-dry.log"; my $status = run_capture( @@ -742,7 +802,8 @@ sub test_dry_run_release { '--target', $target, '--run-id', 'dry', '--build-timestamp', $epoch, - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--genesis-release', $release_root, '--collect-dir', $dependencies, @@ -762,6 +823,9 @@ sub test_dry_run_release { sub test_rpm_repository_lock { my $output = "$tmp/rpm-lock-output"; my $repository = "$tmp/rpm-shared-repository"; + my $target = 'test+epel-10-' . capture_command('uname', '-m'); + my $scratch_repo_root = "$tmp/rpm-lock-repo-root"; + write_target_manifest($scratch_repo_root, $target); make_path("$repository/.lock"); write_binary("$repository/.lock/owner", "host=other\npid=1\nepoch=1\n"); @@ -776,11 +840,12 @@ sub test_rpm_repository_lock { my $status = run_capture( $log, $^X, $rpm_consumer, - '--repo-root', $repo_root, + '--repo-root', $scratch_repo_root, '--output', $output, '--repo-dep', $repository, - '--target', 'test+epel-10-' . capture_command('uname', '-m'), - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--target', $target, + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--dry-run', ); isnt($status, 0, 'a shared RPM repository cannot have two publishers'); @@ -795,11 +860,12 @@ sub test_rpm_repository_lock { my $forced_status = run_capture( $forced_log, $^X, $rpm_consumer, - '--repo-root', $repo_root, + '--repo-root', $scratch_repo_root, '--output', "$tmp/rpm-force-output", '--repo-dep', $repository, - '--target', 'test+epel-10-' . capture_command('uname', '-m'), - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--target', $target, + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--dry-run', '--force-unlock', ); is($forced_status, 0, '--force-unlock recovers an interrupted RPM publication'); @@ -836,7 +902,8 @@ sub test_rpm_signal_cleanup { '--output', $output, '--repo-dep', $repository, '--target', 'test+epel-10-x86_64', - '--skip-build', '--skip-xcat', '--skip-xcat-dep', '--skip-perl', + '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', + '--no-verify-repo', '--skip-createrepo', '--skip-tarball', '--dry-run', ); exit 127; diff --git a/t/mockbuild-all.t b/t/mockbuild-all.t new file mode 100644 index 0000000..7511ca9 --- /dev/null +++ b/t/mockbuild-all.t @@ -0,0 +1,514 @@ +#!/usr/bin/perl +# Focused fixture tests for the xcat-dep build helpers (MockBuildUtils.pm), covering the review +# feedback on PR #62: skip-mode package selection, version pins, RPM-identity comparison in the +# cross-arch genesis finalize, and the "require the genesis input" guard. +use strict; +use warnings; +use Test::More; +use FindBin qw($RealBin); +use lib "$RealBin/.."; +use File::Temp qw(tempdir); +use File::Path qw(make_path); +use File::Basename qw(basename); +use MockBuildUtils qw(install_deps_packages install_deps_command missing_perl_modules + required_pkgs version_matches rpm_sigmd5 rpm_version rpm_release rpm_is_signed + restamp_release_line cross_copy_genesis finalize_xcat_dep read_manifest + verify_repo_packages verify_repo_signature verify_rpm_signatures + parse_evr evr_constraint_ok parse_pin rpmkeys_checksig_problem + bump_dep_release_suffix build_mock_uniqueext); + +# Run a printing sub with STDOUT muted so its progress lines do not pollute TAP. +sub quiet(&) { + my ($code) = @_; + open(my $save, '>&', \*STDOUT) or die "dup STDOUT: $!"; + open(STDOUT, '>', '/dev/null') or die "mute STDOUT: $!"; + my @r = eval { $code->() }; + my $err = $@; + open(STDOUT, '>&', $save) or die "restore STDOUT: $!"; + die $err if $err; + return wantarray ? @r : $r[0]; +} + +# ---- required_pkgs: a skipped builder's packages are not required (clean --skip-* runs) ------- +my @all = qw(elilo-xcat ipmitool-xcat perl-IO-Stty perl-Sys-Virt xCAT-genesis-base); +is_deeply([required_pkgs(\@all, 0, 0, 0)], \@all, + 'no skips -> every package required'); +is_deeply([required_pkgs(\@all, 1, 0, 0)], [qw(elilo-xcat ipmitool-xcat perl-IO-Stty perl-Sys-Virt)], + '--skip-genesis drops xCAT-genesis-base'); +is_deeply([required_pkgs(\@all, 0, 1, 0)], [qw(elilo-xcat ipmitool-xcat xCAT-genesis-base)], + '--skip-perl drops perl-*'); +is_deeply([required_pkgs(\@all, 0, 0, 1)], [qw(perl-IO-Stty perl-Sys-Virt xCAT-genesis-base)], + '--skip-xcat-dep drops the dep builders'); +is_deeply([required_pkgs(\@all, 1, 1, 1)], [], + 'all skips -> nothing required (a clean skip run validates nothing)'); + +# ---- gate composition: a clean --skip-* run does not flag the skipped package as MISSING ------ +# (PR #62 R3.3) verify_target_repo derives its expected set with required_pkgs(...skip flags), so a +# package a skip mode intentionally did not build must NOT be reported missing by the completeness +# gate. This tests the composition (required_pkgs -> verify_repo_packages), not either half alone. +{ + my %pins = ('elilo-xcat' => '3.14', 'perl-IO-Stty' => '0.04', 'xCAT-genesis-base' => '2.*'); + my $present = { 'elilo-xcat' => '3.14', 'perl-IO-Stty' => '0.04' }; # --skip-genesis: no genesis rpm built + my @keep = required_pkgs([sort keys %pins], 1, 0, 0); # skip_genesis + my %expected = map { $_ => $pins{$_} } @keep; + is_deeply([verify_repo_packages(\%expected, $present)], [], + 'gate: --skip-genesis run with genesis absent reports no MISSING (skipped pkg not required)'); + + # Sanity: WITHOUT the skip filter that same absent genesis IS flagged -- proving the filter is load-bearing. + my @unfiltered = verify_repo_packages(\%pins, $present); + is(scalar(@unfiltered), 1, 'gate: unfiltered, the absent genesis is flagged MISSING'); + like($unfiltered[0], qr/^MISSING xCAT-genesis-base\b/, 'gate: the flag names the absent genesis'); +} + +# ---- version_matches: exact + shell-glob pins ------------------------------------------------ +ok( version_matches('2.19.0', '2.*'), '2.* matches 2.19.0'); +ok( version_matches('2.18.2', '2.*'), '2.* matches 2.18.2 (walks with xcat-core)'); +ok(!version_matches('3.0.0', '2.*'), '2.* rejects 3.0.0'); +ok(!version_matches('20.0', '2.*'), '2.* rejects 20.0 (anchored, literal dot)'); +ok( version_matches('2.19.0', '2.19.*'), '2.19.* matches 2.19.0'); +ok(!version_matches('2.20.0', '2.19.*'), '2.19.* rejects 2.20.0'); +ok( version_matches('1.8.18', '1.8.18'), 'exact pin matches'); +ok(!version_matches('1.8.19', '1.8.18'), 'exact pin rejects a different version'); +ok( version_matches('anything', '*'), "'*' matches any version"); + +# ---- read_manifest: sections + entries ------------------------------------------------------- +{ + my $dir = tempdir(CLEANUP => 1); + my $f = "$dir/m.conf"; + open my $fh, '>', $f or die; + print $fh "# comment\n[alma+epel-8-x86_64]\nelilo-xcat=3.14\nxCAT-genesis-base=2.*\n\n" + . "[alma+epel-9-x86_64]\nperl-Sys-Virt=11.10.0\n"; + close $fh; + my %m = read_manifest($f); + is($m{'alma+epel-8-x86_64'}{'elilo-xcat'}, '3.14', 'read_manifest: exact pin'); + is($m{'alma+epel-8-x86_64'}{'xCAT-genesis-base'},'2.*', 'read_manifest: glob pin'); + is($m{'alma+epel-9-x86_64'}{'perl-Sys-Virt'}, '11.10.0', 'read_manifest: second section'); + is_deeply({read_manifest("$dir/nope.conf")}, {}, 'read_manifest: missing file -> empty'); +} + +# rpm_sigmd5 on a missing/unreadable rpm returns '' (so cross_copy treats it as "not identical"). +is(rpm_sigmd5('/nonexistent/xCAT-genesis-base-ppc64-9.9.9.noarch.rpm'), '', + 'rpm_sigmd5 returns empty for a missing rpm'); + +# ---- RPM-identity comparison + cross_copy_genesis (needs rpmbuild for real rpms) -------------- +SKIP: { + skip 'rpmbuild not available', 6 if system('command -v rpmbuild >/dev/null 2>&1') != 0; + my $tmp = tempdir(CLEANUP => 1); + my $seq = 0; + my $mk = sub { # build a genesis-named rpm with a given marker payload + my ($tarch, $content, $version) = @_; + $version ||= '2.19.0'; + my $out = "$tmp/out" . (++$seq); # unique dir: same NVR would overwrite in a shared one + my $spec = "$tmp/$tarch-$seq.spec"; + open my $fh, '>', $spec or die; + print $fh <<"SPEC"; +Name: xCAT-genesis-base-$tarch +Version: $version +Release: snapTEST +Summary: test fixture +License: EPL +BuildArch: noarch +%description +test fixture +%install +mkdir -p %{buildroot}/opt/xcat/t +echo '$content' > %{buildroot}/opt/xcat/t/marker +%files +/opt/xcat/t/marker +SPEC + close $fh; + system("rpmbuild -bb --quiet --define '_topdir $tmp/rpmb$seq' --define '_rpmdir $out' " + . "'$spec' >/dev/null 2>&1") == 0 or die "rpmbuild failed for $tarch/$content"; + my ($rpm) = glob("$out/noarch/xCAT-genesis-base-$tarch-*.rpm"); + return $rpm; + }; + my $rpmA = $mk->('ppc64', 'CONTENT_A'); + my $rpmB = $mk->('ppc64', 'CONTENT_B_is_different'); # same NVR/basename, different payload + + isnt(rpm_sigmd5($rpmA), rpm_sigmd5($rpmB), + 'rpm_sigmd5 differs for same-name rpms with different content'); + + my $base = basename($rpmA); + my ($from, $to) = ("$tmp/from", "$tmp/to"); + make_path($from, $to); + system("cp '$rpmA' '$from/$base'"); # the fresh source + system("cp '$rpmB' '$to/$base'"); # a STALE dest rpm sharing the filename + + my $n = quiet { cross_copy_genesis($from, $to, 'ppc64', undef) }; + ok($n >= 1, "cross_copy refreshes a stale same-name rpm by content (copied=$n)"); + is(rpm_sigmd5("$to/$base"), rpm_sigmd5($rpmA), + 'after cross_copy the dest matches the source content'); + + my $n2 = quiet { cross_copy_genesis($from, $to, 'ppc64', undef) }; + is($n2, 0, 'cross_copy is a no-op when content is already identical (idempotent)'); + + # A signer callback is invoked for each copied rpm. + my ($from2, $to2) = ("$tmp/from2", "$tmp/to2"); + make_path($from2, $to2); + system("cp '$rpmA' '$from2/$base'"); + my @signed; + quiet { cross_copy_genesis($from2, $to2, 'ppc64', sub { push @signed, $_[0] }) }; + is_deeply(\@signed, ["$to2/$base"], 'the sign callback runs on each copied rpm'); + + # rpm_version dies when a dir holds two DIFFERENT versions of the same package (stale artifact). + my $vdir = "$tmp/vers"; make_path($vdir); + system("cp '" . $mk->('ppc64', 'x', '2.19.0') . "' '$vdir/'"); + system("cp '" . $mk->('ppc64', 'x', '2.18.0') . "' '$vdir/'"); + my $vdied = !eval { rpm_version($vdir, 'xCAT-genesis-base'); 1 }; + ok($vdied, 'rpm_version dies when a dir holds multiple distinct versions of a package'); +} + +# ---- finalize_xcat_dep: require the genesis inputs (no silent no-op) -------------------------- +{ + my $tmp = tempdir(CLEANUP => 1); + make_path("$tmp/x/rh9/x86_64", "$tmp/p/rh9/ppc64le"); # a pair exists, but NO genesis rpms + my $ok = eval { quiet { finalize_xcat_dep("$tmp/x", "$tmp/p") }; 1 }; + ok(!$ok, 'finalize dies when a repo pair has no genesis rpms (was a silent success)'); + like($@, qr/no (x86_64|ppc64) xCAT-genesis-base/, + 'finalize error names the missing genesis input'); + + my $tmp2 = tempdir(CLEANUP => 1); # no /x86_64 pair at all + make_path("$tmp2/x", "$tmp2/p"); + my $ok2 = eval { quiet { finalize_xcat_dep("$tmp2/x", "$tmp2/p") }; 1 }; + ok(!$ok2, 'finalize dies when no /x86_64 + /ppc64le pair is found'); + + # A missing ppc64le PEER repo (not just missing rpms) is fatal, not a silent skip. + my $tmp3 = tempdir(CLEANUP => 1); + make_path("$tmp3/x/rh9/x86_64"); # x86_64 OS present, but NO ppc64le peer dir at all + my $ok3 = eval { quiet { finalize_xcat_dep("$tmp3/x", "$tmp3/p") }; 1 }; + ok(!$ok3, 'finalize dies when an x86_64 OS has no ppc64le peer repo (no silent skip)'); + like($@, qr/no ppc64le peer repo/, 'finalize error names the missing peer'); + + # Symmetric (PR #62 review #2): a ppc64le-ONLY (no x86_64 sibling) must ALSO be caught -- + # the old x86_64-anchored discovery skipped it entirely and exited 0. + my $tmp4 = tempdir(CLEANUP => 1); + make_path("$tmp4/p/rh9/ppc64le"); # ppc64le OS present, but NO x86_64 peer dir at all + my $ok4 = eval { quiet { finalize_xcat_dep("$tmp4/x", "$tmp4/p") }; 1 }; + ok(!$ok4, 'finalize dies when a ppc64le OS has no x86_64 peer repo (was silently skipped)'); + like($@, qr/no x86_64 peer repo/, 'finalize error names the missing x86_64 peer'); + + # @GENESIS_ARCHES is the single source of truth for the cross-arch matrix (add arches there). + my %tarch = map { $_->{arch} => $_->{tarch} } @MockBuildUtils::GENESIS_ARCHES; + is($tarch{x86_64}, 'x86_64', 'GENESIS_ARCHES: x86_64 maps to tarch x86_64'); + is($tarch{ppc64le}, 'ppc64', 'GENESIS_ARCHES: ppc64le maps to xCAT tarch ppc64'); +} + +# ---- restamp_release_line: CD --build-number Release stamping (PR #62 review point 1) ---------- +# A fresh stamp is appended after the Release token, preserving any %{?dist} macro. +{ + my ($l, $ch) = restamp_release_line("Release: 1%{?dist}\n", '.snap202607161200.57'); + is($l, "Release: 1%{?dist}.snap202607161200.57\n", 'stamps a fresh Release, macro preserved'); + is($ch, 1, 'reports changed'); +} +# Idempotent: the exact same suffix is a no-op (concurrent per-arch build / same-tree re-run). +{ + my $line = "Release: 1%{?dist}.snap202607161200.57\n"; + my ($l, $ch) = restamp_release_line($line, '.snap202607161200.57'); + is($l, $line, 're-stamping the SAME suffix is a no-op'); + is($ch, 0, 'reports unchanged'); +} +# A DIFFERENT build-number REPLACES the prior stamp (does not accumulate) -- the double-stamp bug. +{ + my ($l, $ch) = restamp_release_line("Release: 1%{?dist}.snap202607161200.57\n", '.snap202607161200.58'); + is($l, "Release: 1%{?dist}.snap202607161200.58\n", 'a new build-number replaces the old stamp'); + is($ch, 1, 'reports changed'); + unlike($l, qr/\.snap\d{12}\.\d+\.snap/, 'never leaves two stacked .snap stamps'); +} +# Even an already-corrupted (double-stamped) line is healed back to a single stamp. +{ + my ($l) = restamp_release_line("Release: 5.snap202601010000.1.snap202601020000.2\n", '.snap202607161200.9'); + is($l, "Release: 5.snap202607161200.9\n", 'strips multiple stacked prior stamps before re-stamping'); +} +# A non-Release line is never touched. +{ + my ($l, $ch) = restamp_release_line("Version: 0.3.3\n", '.snap202607161200.57'); + is($l, "Version: 0.3.3\n", 'non-Release line untouched'); + is($ch, 0, 'reports unchanged'); +} + +# ---- rpm_is_signed: unreadable / missing -> not signed (used by the finalize idempotency fix) --- +is(rpm_is_signed(undef), 0, 'rpm_is_signed(undef) is 0'); +is(rpm_is_signed("/no/such/file.rpm"), 0, 'rpm_is_signed on a missing file is 0'); + +# ---- rpm_release: absent package -> undef (used by the --build-number bump-landed check) --------- +is(rpm_release(tempdir(CLEANUP => 1), 'nonexistent-pkg'), undef, 'rpm_release is undef when no rpm matches'); + +# ---- manifest <-> docs consistency: conserver-xcat is in EVERY target section (PR #62 point 7c) -- +# BUILD.md documents conserver-xcat as built for every target; guard that the manifest agrees so the +# doc and the manifest can never silently drift apart again. +{ + my %m = read_manifest("$RealBin/../packages-manifest.conf"); + # Not every section is a build target: [common] describes the SHARED repository the + # OpenEmbedded Genesis release is published into, which no builder produces. Target sections are + # the ones named after a mock config (+epel--, opensuse-leap--). + my @targets = grep { /^[a-z0-9.+-]+-\d+(?:\.\d+)?-[a-z0-9_]+$/ } sort keys %m; + cmp_ok(scalar(@targets), '>=', 1, 'packages-manifest.conf has at least one target section'); + ok(!grep({ $_ eq 'common' } @targets), 'the shared-repo section is not treated as a build target'); + my @missing = grep { !exists $m{$_}{'conserver-xcat'} } @targets; + is_deeply(\@missing, [], 'conserver-xcat is present in every manifest target section') + or diag("missing conserver-xcat in: @missing"); +} + +# ---- bump_dep_release_suffix: stamps xcat-dep specs, prunes nested xcat-core, idempotent -------- +# Reviewer asked for a test on this path. It walks a tree, stamps the first Release: line of every +# xcat-dep spec, prunes a nested xcat-core/ checkout, and is idempotent on a re-run. +{ + my $tmp = tempdir(CLEANUP => 1); + # (a) a top-level dep spec that MUST be stamped + open my $a, '>', "$tmp/a.spec" or die; + print $a "Name: a\nVersion: 1.0\nRelease: 5%{?dist}\n"; + close $a; + # (b) a spec NESTED under xcat-core/ that MUST be pruned (left untouched) + make_path("$tmp/xcat-core"); + open my $b, '>', "$tmp/xcat-core/b.spec" or die; + print $b "Name: b\nVersion: 1.0\nRelease: 9\n"; + close $b; + # (c) a spec with no Release: line at all (ignored, never stamped) + open my $c, '>', "$tmp/c.spec" or die; + print $c "Name: c\nVersion: 1.0\n"; + close $c; + + my $n = quiet { bump_dep_release_suffix($tmp, '.snap202601010000') }; + is($n, 1, 'bump_dep_release_suffix stamps exactly the one dep spec with a Release line'); + + my $a_after = do { open my $fh, '<', "$tmp/a.spec" or die; local $/; <$fh> }; + like($a_after, qr/^Release: 5%\{\?dist\}\.snap202601010000$/m, + 'a.spec Release now carries the CD suffix, macro preserved'); + + my $b_after = do { open my $fh, '<', "$tmp/xcat-core/b.spec" or die; local $/; <$fh> }; + is($b_after, "Name: b\nVersion: 1.0\nRelease: 9\n", + 'nested xcat-core/b.spec is pruned and left untouched'); + + # A SECOND call is idempotent: nothing newly stamped, a.spec content unchanged. + my $n2 = quiet { bump_dep_release_suffix($tmp, '.snap202601010000') }; + is($n2, 0, 'a second bump_dep_release_suffix call stamps nothing (idempotent)'); + my $a_again = do { open my $fh, '<', "$tmp/a.spec" or die; local $/; <$fh> }; + is($a_again, $a_after, 'a.spec content unchanged on the idempotent second call'); +} + +# ---- verify_repo_packages: pure repo-completeness decision (MISSING + VERSION + wildcard) --------- +# The gate's completeness layer: given manifest pins and the versions actually present in a repo, +# return the list of problems (empty = complete). No I/O -- exercised directly with plain hashes. +{ + # happy: every required package present, one exact-pinned + one wildcard -> no problems. + my @p = verify_repo_packages({ a => '1.0', b => '*' }, { a => '1.0', b => '9.9' }); + is_deeply(\@p, [], 'verify_repo_packages: all present + pins satisfied -> 0 problems'); + + # missing: present lacks 'a' entirely -> exactly one MISSING problem naming 'a'. + my @m = verify_repo_packages({ a => '1.0', b => '*' }, { b => '9.9' }); + is(scalar(@m), 1, 'verify_repo_packages: an absent package yields exactly one problem'); + like($m[0], qr/^MISSING a\b/, 'verify_repo_packages: absent package reported as MISSING '); + + # missing via explicit undef present value is treated the same as absent. + my @mu = verify_repo_packages({ a => '1.0' }, { a => undef }); + is(scalar(@mu), 1, 'verify_repo_packages: undef present version counts as MISSING'); + like($mu[0], qr/^MISSING a\b/, 'verify_repo_packages: undef present version reported as MISSING'); + + # version: present but the wrong version -> exactly one VERSION problem naming 'a'. + my @v = verify_repo_packages({ a => '1.0' }, { a => '2.0' }); + is(scalar(@v), 1, 'verify_repo_packages: a mismatched version yields exactly one problem'); + like($v[0], qr/^VERSION a\b/, 'verify_repo_packages: version mismatch reported as VERSION '); + + # wildcard: a '*' pin accepts any present version -> no problem. + my @w = verify_repo_packages({ c => '*' }, { c => '0.0.1' }); + is_deeply(\@w, [], "verify_repo_packages: '*' pin accepts any present version"); + + # combined: one MISSING and one VERSION -> two problems (sorted by package name: a before b). + my @c = verify_repo_packages({ a => '1.0', b => '2.0' }, { b => '9.9' }); + is(scalar(@c), 2, 'verify_repo_packages: one MISSING + one VERSION -> two problems'); + like($c[0], qr/^MISSING a\b/, 'verify_repo_packages: combined case reports MISSING a'); + like($c[1], qr/^VERSION b\b/, 'verify_repo_packages: combined case reports VERSION b'); +} + +# ---- verify_repo_signature: pure signature decision (match / unsigned / wrongkey) ---------------- +# Given the expected signing-key identity per unit and the key that actually signed, return the list +# of problems (empty = every unit signed by the expected key). Plain string compare -- no gpg here. +{ + # happy: repomd signed by exactly the expected key -> no problems. + my @ok = verify_repo_signature({ repomd => 'KEYFPR' }, { repomd => 'KEYFPR' }); + is_deeply(\@ok, [], 'verify_repo_signature: observed == expected -> 0 problems'); + + # unsigned: observed empty -> one UNSIGNED problem naming the unit + expected key. + my @us = verify_repo_signature({ repomd => 'KEYFPR' }, { repomd => '' }); + is(scalar(@us), 1, 'verify_repo_signature: empty observed yields exactly one problem'); + like($us[0], qr/^UNSIGNED repomd\b/, 'verify_repo_signature: empty observed reported as UNSIGNED'); + like($us[0], qr/expected KEYFPR/, 'verify_repo_signature: UNSIGNED names the expected key'); + + # unsigned via explicit undef observed is treated the same as empty. + my @uu = verify_repo_signature({ repomd => 'KEYFPR' }, { repomd => undef }); + like($uu[0], qr/^UNSIGNED repomd\b/, 'verify_repo_signature: undef observed reported as UNSIGNED'); + + # wrongkey: signed, but by a different key -> one WRONGKEY problem naming both. + my @wk = verify_repo_signature({ repomd => 'GOODFPR' }, { repomd => 'EVILFPR' }); + is(scalar(@wk), 1, 'verify_repo_signature: a mismatched signer yields exactly one problem'); + like($wk[0], qr/^WRONGKEY repomd: signed by EVILFPR, expected GOODFPR$/, + 'verify_repo_signature: mismatch reported as WRONGKEY : signed by , expected '); +} + +# ---- verify_rpm_signatures: EVERY rpm must be signed by an accepted key (PR #62 review #4) ----- +{ + my %accept = ( '4123c420cb60ad43' => 1, 'cb60ad43' => 1 ); # signing key's long + short id + + my @ok = verify_rpm_signatures( + [ ['a-1.0.rpm', 'cb60ad43'], ['b-2.0.rpm', '4123C420CB60AD43'] ], \%accept); + is_deeply(\@ok, [], 'verify_rpm_signatures: all rpms signed by an accepted key -> no problems (case-insensitive)'); + + my @uns = verify_rpm_signatures([ ['c-3.0.rpm', undef], ['d-4.0.rpm', ''] ], \%accept); + is(scalar(@uns), 2, 'verify_rpm_signatures: undef and empty key id both flagged'); + like($uns[0], qr/^UNSIGNED rpm c-3\.0\.rpm$/, 'verify_rpm_signatures: unsigned rpm reported by name'); + + my @wrong = verify_rpm_signatures([ ['e-5.0.rpm', 'deadbeef'] ], \%accept); + is(scalar(@wrong), 1, 'verify_rpm_signatures: a foreign-signed rpm yields one problem'); + like($wrong[0], qr/^WRONGKEY rpm e-5\.0\.rpm: signed by deadbeef, expected one of\b/, + 'verify_rpm_signatures: wrong key reported as WRONGKEY rpm : signed by , expected one of ...'); +} + +# ---- EVR constraints: full EPOCH:VERSION-RELEASE validation (PR #62 review) ------------------- +# rpm's own version algorithm, via its lua rpm.vercmp binding, is the injected segment comparator -- +# the same primitive mockbuild-all passes in production, so these assert the real rpm semantics. +my $vercmp = sub { + my ($a, $b) = @_; + my $o = `rpm --eval '%{lua:print(rpm.vercmp([==[$a]==],[==[$b]==]))}' 2>/dev/null`; + chomp $o; return $o <=> 0; +}; +{ + is_deeply([parse_evr('2:2.18.0-5')], ['2','2.18.0','5'], 'parse_evr: epoch:version-release'); + is_deeply([parse_evr('2.18.0')], ['0','2.18.0',undef], 'parse_evr: bare version -> epoch 0, no release'); + is_deeply([parse_evr('0.04-5.el8')], ['0','0.04','5.el8'], 'parse_evr: release kept whole'); + + is_deeply([parse_pin('>= 2:2.18.0')], ['evr','>=','2:2.18.0'], 'parse_pin: EVR operator constraint'); + is_deeply([parse_pin('2.*')], ['version'], 'parse_pin: glob stays a version pin'); + is_deeply([parse_pin('*')], ['any'], 'parse_pin: * is any'); + + # the reviewer's cases: genesis-base >= 2:2.18.0 rejects a pre-2.18 (Epoch 2) genesis... + ok( evr_constraint_ok('2:2.19.0-snap202607211907', '>=', '2:2.18.0', $vercmp), + 'EVR: 2:2.19.0 satisfies >= 2:2.18.0'); + ok(!evr_constraint_ok('2:2.17.9-snap', '>=', '2:2.18.0', $vercmp), + 'EVR: 2:2.17.9 REJECTED by >= 2:2.18.0 (2.* would have wrongly accepted it)'); + ok(!evr_constraint_ok('0:2.18.0-1', '>=', '2:2.18.0', $vercmp), + 'EVR: epoch enforced -- 0:2.18.0 rejected by >= 2:2.18.0'); + # ...and a release floor perl-IO-Stty >= 0.04-5. + ok( evr_constraint_ok('0.04-5.el8.snap202607221225.13', '>=', '0.04-5', $vercmp), + 'EVR: 0.04-5.el8.snap... satisfies release floor >= 0.04-5'); + ok(!evr_constraint_ok('0.04-4.el8', '>=', '0.04-5', $vercmp), + 'EVR: 0.04-4 REJECTED by release floor >= 0.04-5 (VERSION-only match would have passed)'); + + # end-to-end through the gate: EVR pin honored, with the got EVR supplied separately from %{VERSION}. + my @okp = verify_repo_packages( + { 'xCAT-genesis-base' => '>= 2:2.18.0' }, + { 'xCAT-genesis-base' => '2.19.0' }, + { 'xCAT-genesis-base' => '2:2.19.0-snap202607211907' }, $vercmp); + is_deeply(\@okp, [], 'gate: EVR-satisfying genesis passes'); + my @badp = verify_repo_packages( + { 'xCAT-genesis-base' => '>= 2:2.18.0' }, + { 'xCAT-genesis-base' => '2.17.0' }, + { 'xCAT-genesis-base' => '2:2.17.0-snap' }, $vercmp); + is(scalar(@badp), 1, 'gate: pre-2.18 genesis yields exactly one problem'); + like($badp[0], qr/^EVR xCAT-genesis-base: repo has 2:2\.17\.0-snap, manifest requires >= 2:2\.18\.0$/, + 'gate: EVR failure names the observed EVR and the requirement'); +} + +# ---- rpmkeys --checksig verdict (pure) ------------------------------------------------------- +{ + is_deeply([rpmkeys_checksig_problem('a.rpm', 0, + "Header V4 RSA/SHA256 Signature, key ID cb60ad43: OK\nPayload SHA256 digest: OK\n")], [], + 'checksig: all-OK rpm -> no problem'); + my @nok = rpmkeys_checksig_problem('b.rpm', 1, "Header SHA256 digest: NOT OK\n"); + like($nok[0], qr/^BADSIG rpm b\.rpm: digest\/signature NOT OK$/, 'checksig: NOT OK flagged'); + my @nokey = rpmkeys_checksig_problem('c.rpm', 1, "Header V4 RSA/SHA256 Signature, key ID deadbeef: NOKEY\n"); + like($nokey[0], qr/^BADSIG rpm c\.rpm: NOKEY/, 'checksig: NOKEY (unaccepted/unsigned) flagged'); + my @rc = rpmkeys_checksig_problem('d.rpm', 2, ""); + like($rc[0], qr/rc=2/, 'checksig: non-zero exit with no marker still flagged'); +} + +# ---- build_mock_uniqueext: distinct per target so concurrent mock roots never collide --------- +# (PR #62 review) A long (timestamp) run id must not tail-truncate away the leading EL/arch token: +# for the 7-char "ppc64le" arch that dropped the EL digit, so alma+epel-{8,9,10}-ppc64le collapsed to +# one uniqueext -- and goconserver builds all three ELs in the SAME el10 chroot, so the roots raced. +{ + my $seq = 6; my $label = 'goconserver'; + # The reproducing case: the default timestamp run id (long), folded with the per-target prefix. + my @ppc = map { build_mock_uniqueext("alma+epel-$_-ppc64le-20260821-210716", $seq, $label) } (8, 9, 10); + my %seen; $seen{$_}++ for @ppc; + is(scalar(keys %seen), 3, + 'build_mock_uniqueext: el8/el9/el10 ppc64le get DISTINCT uniqueext on a long run id (no collision)'); + like($ppc[0], qr/^mba-06-alma-epel-8-/, 'uniqueext keeps a readable leading EL/arch token'); + + # x86_64 (6-char arch) was never broken -- assert it stays distinct too. + my @x86 = map { build_mock_uniqueext("alma+epel-$_-x86_64-20260821-210716", $seq, $label) } (8, 9, 10); + my %sx; $sx{$_}++ for @x86; + is(scalar(keys %sx), 3, 'build_mock_uniqueext: el8/el9/el10 x86_64 also distinct'); + + # Short run ids (e.g. the CD "$BUILD_NUMBER") are unchanged and already distinct per target. + isnt(build_mock_uniqueext('alma+epel-8-ppc64le-104', $seq, $label), + build_mock_uniqueext('alma+epel-9-ppc64le-104', $seq, $label), + 'build_mock_uniqueext: short (build-number) run ids distinct per target'); + + # Same run id + same step -> stable (deterministic; a re-run reuses/scrubs the same root). + is(build_mock_uniqueext('alma+epel-8-ppc64le-20260821-210716', $seq, $label), + build_mock_uniqueext('alma+epel-8-ppc64le-20260821-210716', $seq, $label), + 'build_mock_uniqueext: deterministic for a given (run, seq, label)'); +} + +# ---- the shipped manifest keeps xCAT's release-sensitive floors (PR #62 review) ----------------- +# xCAT states these as ">= version-release" Requires, so a VERSION-only pin lets the gate accept an +# rpm with the right Version and a Release older than xCAT will install against. Guard the shipped +# manifest itself, in every section, so a later edit cannot quietly drop a floor back to a bare +# version. +{ + my $shipped = "$RealBin/../packages-manifest.conf"; + SKIP: { + skip 'packages-manifest.conf not found', 2 unless -f $shipped; + my %m = read_manifest($shipped); + my @evr_pinned = qw(goconserver xnba-undi syslinux-xcat ipmitool-xcat + perl-HTTP-Async perl-Net-HTTPS-NB perl-IO-Stty xCAT-genesis-base); + my @bare; + for my $tgt (sort keys %m) { + for my $pkg (@evr_pinned) { + my $pin = $m{$tgt}{$pkg}; + next unless defined $pin; # not every target lists every package + push @bare, "[$tgt] $pkg=$pin" unless $pin =~ /^\s*(?:>=|>|<=|<|=)\s*\S/; + } + } + is_deeply(\@bare, [], 'every release-sensitive package keeps an EVR floor in every section') + or diag("VERSION-only pin(s):\n " . join("\n ", @bare)); + + # grub2-xcat is the deliberate exception: xCAT-server asks for >= 2.02-0.76.el7.1.snap..., + # which the grub2-xcat this repo builds (1.0-2) cannot satisfy. Encoding that Requires would + # fail every build, so the pin tracks what is actually produced -- see the file's header. + my @grub = grep { defined } map { $m{$_}{'grub2-xcat'} } sort keys %m; + is_deeply([ grep { /^\s*>=/ } @grub ], [], + 'grub2-xcat stays a version pin (its xCAT Requires cannot be met by what is built)'); + } +} + +# ---- --install-deps: the host prerequisites (the modules are what actually break a run) ---------- +# Two CD runs died at compile time inside XCAT::BuildUtils because a builder lacked a module +# (perl-File-Slurper on one host, perl-IPC-Cmd on another), so the list must carry every module the +# script loads, and the mode must PROVE them by loading rather than trusting the package manager. +{ + my @el = install_deps_packages('almalinux'); + for my $need (qw(perl-File-Slurper perl-IPC-Cmd perl-Parallel-ForkManager mock createrepo_c)) { + ok(scalar(grep { $_ eq $need } @el), "EL prerequisites include $need"); + } + my @cmd = install_deps_command('almalinux'); + is($cmd[0], 'dnf', 'EL installs with dnf'); + ok(scalar(grep { $_ eq '-y' } @cmd), '... non-interactively'); + + my @suse = install_deps_command('opensuse-leap'); + is($suse[0], 'zypper', 'SUSE installs with zypper'); + ok(scalar(grep { $_ eq '--non-interactive' } @suse), '... non-interactively'); + is_deeply([ grep { /^perl-/ } install_deps_packages('opensuse-leap') ], + [ grep { /^perl-/ } @el ], + 'both families install the same perl modules'); + + # the probe reports what cannot be loaded, and nothing else + is_deeply([ missing_perl_modules('Digest::SHA') ], [], + 'missing_perl_modules: a loadable module is not reported'); + is_deeply([ missing_perl_modules('No::Such::Module::Here') ], ['No::Such::Module::Here'], + 'missing_perl_modules: an absent module is reported'); + is_deeply([ missing_perl_modules('Digest::SHA', 'No::Such::Module::Here') ], + ['No::Such::Module::Here'], '... and only the absent one, from a mixed list'); +} + +done_testing; diff --git a/t/verify-repo-el.t b/t/verify-repo-el.t new file mode 100644 index 0000000..3ae9bc4 --- /dev/null +++ b/t/verify-repo-el.t @@ -0,0 +1,116 @@ +#!/usr/bin/perl +# Focused end-to-end test for the PER-TARGET REPO GATE, driving the real +# `mockbuild-all.pl --verify-repo` against a hand-built fixture repo. +# +# What is tested here is the wiring the PR #62 review found: the gate filtered the manifest through +# required_pkgs() with THIS invocation's --skip-* flags, so the flags that say what an invocation +# BUILT also decided what the published repository was allowed to be missing. A repo carrying no +# xCAT-genesis-base therefore passed when the run that verified it had been given --skip-genesis. +# +# The assertions are on the REPORTED PROBLEMS, not on the exit code: a standalone --verify-repo also +# demands a repomd signature by contract, and these fixtures are unsigned, so it exits non-zero +# either way. What distinguishes a fixed gate from a broken one is whether the missing package is +# NAMED. +use strict; +use warnings; +use Test::More; +use FindBin qw($RealBin); +use File::Temp qw(tempdir); +use File::Path qw(make_path); + +my $SCRIPT = "$RealBin/../mockbuild-all.pl"; +plan skip_all => "mockbuild-all.pl not found" unless -f $SCRIPT; +plan skip_all => "rpm tooling required" + unless $^O eq 'linux' + && !system('sh', '-c', 'command -v rpmbuild >/dev/null 2>&1') + && !system('sh', '-c', 'command -v rpm >/dev/null 2>&1') + && !system('sh', '-c', 'command -v createrepo_c >/dev/null 2>&1'); + +my $tmp = tempdir(CLEANUP => 1); +my $target = 'alma+epel-10-x86_64'; + +# build_rpm($name, $version): a minimal noarch rpm, so the gate reads a REAL header name. +sub build_rpm { + my ($name, $version) = @_; + my $top = "$tmp/rpmbuild"; + make_path("$top/SPECS"); + my $spec = "$top/SPECS/$name.spec"; + open my $fh, '>', $spec or die $!; + print $fh <<"SPEC"; +Name: $name +Version: $version +Release: 1 +Summary: fixture +License: EPL +BuildArch: noarch +%description +fixture package for the repository gate test +%install +mkdir -p %{buildroot}/usr/share/$name +%files +/usr/share/$name +SPEC + close $fh; + my $rc = system('rpmbuild', '--quiet', '-bb', '--define', "_topdir $top", $spec); + die "cannot build fixture rpm $name\n" if $rc != 0; + my ($built) = glob("$top/RPMS/noarch/$name-$version-1.noarch.rpm"); + die "fixture rpm $name not produced\n" unless $built && -f $built; + return $built; +} + +# make_repo(%opt): an indexed repo carrying ipmitool-xcat, and xCAT-genesis-base unless with_genesis +# is turned off. +sub make_repo { + my (%o) = @_; + my $dir = "$tmp/repo" . ($o{tag} // ''); + make_path($dir); + system('cp', build_rpm('ipmitool-xcat', '1.8.18'), $dir) == 0 or die $!; + system('cp', build_rpm('xCAT-genesis-base-x86_64', '2.18.0'), $dir) == 0 or die $! + if $o{with_genesis}; + system('createrepo_c', '--quiet', $dir) == 0 or die "createrepo_c failed\n"; + return $dir; +} + +# The manifest is the source of truth for what the target must carry. +my $root = "$tmp/repo-root"; +make_path($root); +open my $m, '>', "$root/packages-manifest.conf" or die $!; +print $m "[$target]\nipmitool-xcat=1.8.18\nxCAT-genesis-base=2.*\n"; +close $m; + +sub run_gate { + my ($repo, @extra) = @_; + my $cmd = join(' ', map { my $x = $_; $x =~ s/'/'"'"'/g; "'$x'" } + ($^X, $SCRIPT, '--verify-repo', $repo, '--target', $target, '--repo-root', $root, @extra)) + . ' 2>&1'; + my $out = `$cmd`; + return ($? >> 8, defined $out ? $out : ''); +} + +# ---- baseline: a complete repo reports no MISSING package --------------------------------------- +{ + my $repo = make_repo(tag => '-full', with_genesis => 1); + my (undef, $out) = run_gate($repo); + unlike($out, qr/MISSING/, 'a complete repo reports no missing package') or diag($out); +} + +# ---- --skip-genesis must not excuse a repo that lacks Genesis ------------------------------------ +{ + my $repo = make_repo(tag => '-nogenesis'); + my (undef, $out) = run_gate($repo, '--skip-genesis'); + like($out, qr/xCAT-genesis-base/, + 'a repo missing Genesis is reported even when the run passed --skip-genesis') or diag($out); +} + +# ---- the same for the compiled deps --------------------------------------------------------------- +{ + my $repo = make_repo(tag => '-nodeps'); + system('rm', '-f', glob("$repo/ipmitool-xcat-*.rpm")) == 0 or die $!; + system('createrepo_c', '--quiet', '--update', $repo) == 0 or die $!; + my (undef, $out) = run_gate($repo, '--skip-xcat-dep'); + like($out, qr/ipmitool-xcat/, + 'a repo missing a compiled dep is reported even when the run passed --skip-xcat-dep') + or diag($out); +} + +done_testing(); diff --git a/xnba/mockbuild.pl b/xnba/mockbuild.pl index 08c6084..7c1395a 100755 --- a/xnba/mockbuild.pl +++ b/xnba/mockbuild.pl @@ -19,7 +19,6 @@ my $mock_cfg = ''; my $mock_uniqueext = ''; my $result_dir = "$repo_root/build-output/list3/xnba-undi"; my $log_dir = "$repo_root/build-logs/list3/xnba-undi"; -my $skip_install = 0; my $build_timestamp; GetOptions( @@ -28,7 +27,6 @@ GetOptions( 'mock-uniqueext=s' => \$mock_uniqueext, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, - 'skip-install!' => \$skip_install, 'build-timestamp=i' => \$build_timestamp, ) or die usage(); @@ -73,7 +71,6 @@ print "work_dir: $work_dir\n"; print "result_dir: $result_dir\n"; print "log_dir: $log_dir\n"; print "mock_cfg: $mock_cfg\n"; -print "skip_install: $skip_install\n"; make_path($result_dir); make_path($log_dir); @@ -176,7 +173,6 @@ Options: --mock-uniqueext STR Mock uniqueext value --result-dir PATH Output directory for RPMs --log-dir PATH Output directory for logs - --skip-install Skip install verification --build-timestamp EPOCH Unix timestamp for reproducible builds USAGE }