diff --git a/.github/workflows/genesis-openembedded.yml b/.github/workflows/genesis-openembedded.yml index 6c87e3f..d10452f 100644 --- a/.github/workflows/genesis-openembedded.yml +++ b/.github/workflows/genesis-openembedded.yml @@ -48,6 +48,7 @@ jobs: lib/XCAT/GenesisRelease.pm \ mockbuild-all.pl \ t/build_utils.t \ + t/common-repo-gate.t \ t/genesis_openembedded_release.t \ t/genesis_openembedded_consumer.t \ t/lib/XCAT/GenesisReleaseTest.pm @@ -55,6 +56,9 @@ jobs: - name: Run package tests run: | prove -v t/build_utils.t + prove -v t/build_timeout.t + prove -v t/sbuild-all.t prove -v -It/lib t/genesis_openembedded_release.t sudo -E prove -v -It/lib t/genesis_openembedded_consumer.t prove -v t/riscv64_perl_cell.t + sudo -E prove -v -It/lib t/common-repo-gate.t diff --git a/BUILD.md b/BUILD.md index 322095b..f0c380a 100644 --- a/BUILD.md +++ b/BUILD.md @@ -247,9 +247,10 @@ generation, and signing have succeeded. If a stopped publisher leaves staging or backup directories behind, rerun it with ``--force-unlock`` to recover the previous repository before starting a new publication. -Repository publication requires a release containing every supported Genesis -architecture. The packages are `noarch`, and the common repository contains -the full set of target images. +Repository publication requires all eight current architectures. Version 1 +release manifests remain readable, but they cannot replace the current +repository because they lack `s390x`. The packages are `noarch`, and each +publication contains the full set. The release checksums cover the unsigned input packages. If repository signing is enabled, `rpmsign` changes the deployed RPM bytes after collection. @@ -265,7 +266,7 @@ The APT side takes the same option, on the run that **publishes**: ```bash ./sbuild-all.pl --skip-build --skip-genesis \ - --publish --expect-arch "amd64 ppc64el" \ + --publish --expect-arch "amd64 ppc64el riscv64" \ --genesis-release /path/to/xcat-genesis-release \ --gpg-sign --gpg-key-id --gpg-home ``` @@ -453,7 +454,7 @@ missing, and then builds the `[rocky-10-riscv64-xcat]` section of `packages-mani | goconserver | cross-compiled on the host (`GOARCH=riscv64`), packaged with `rpmbuild --target riscv64` | | grub2-xcat (noarch) | built in the native, EPEL-free `rocky-10-x86_64` chroot | | perl list6 + EPEL gap (`--epel-gap`) | `mockbuild-perl-packages.pl --target-arch riscv64 --noarch-mock-cfg rocky-10-x86_64 --epel-gap`: XS modules in the riscv64 chroot, noarch modules in the native chroot | -| elilo-xcat, syslinux-xcat, xnba-undi | not built (x86 bootloaders) | +| elilo-xcat, syslinux-xcat, xnba-undi (noarch) | built in the native `rocky-10-x86_64` chroot, like grub2-xcat: a riscv64 management node serves the x86 nodes of a mixed cluster. The target is cross-built on x86_64 only, as its mock config states | There is no EPEL for riscv64, so the perl deps of xCAT that EL10 otherwise takes from EPEL are built here as well (`--epel-gap` in `mockbuild-perl-packages.pl`: perl-Crypt-Blowfish, @@ -586,9 +587,8 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of - **Fresh staging + promote-on-success.** Everything is built + validated into a per-run staging tree first; the published apt repo is (re)assembled from staging ONLY after the complete expected set validates — a partial/failed build never reaches the repo and stale debs never accumulate. -- **Build runs stage; publishing is a separate, locked, atomic step.** The two arches build - *concurrently* on their two hosts against the same `--apt-dir`, so an arch build run **never - publishes**: it fills staging and stops. Publishing happens with **`--publish`** (implied by +- **Build runs stage; publishing is a separate, locked, atomic step.** The arches build + *concurrently* against the same `--apt-dir`, so an arch build run **never publishes**: it fills staging and stops. Publishing happens with **`--publish`** (implied by `--skip-build`, i.e. the finalization run). That step takes **one global publish lock** — not the per-arch build lock — assembles the whole tree into a **side directory**, runs the repo gate against *that* tree, and only then swaps it onto `--apt-dir` with a single `rename(2)`. Readers therefore @@ -609,8 +609,10 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of arch's `Packages` index. They are listed for **ppc64el too, as required-present**, so the gate verifies the ppc repo actually carries them (a ppc MN needs them for netboot, matching the EL manifest). `build_one_codename` **skips** an `Architecture:all` package on any non-amd64 arch - (detected via `control_binary_arch`), so ppc builds only the genuinely arch-specific compiled deps - (`ipmitool-xcat`, `conserver-xcat`, `goconserver`) yet still verifies the boot components. + (detected via `control_binary_arch`), so ppc64el and riscv64 build only the genuinely + arch-specific compiled deps (`ipmitool-xcat`, `conserver-xcat`, `goconserver`) yet still verify the + boot components they need. The riscv64 sections require the same four boot components as + ppc64el: a riscv64 management node serves the x86 nodes of a mixed cluster. - **Fail-hard.** Any required chroot / package / artifact failure, or any version-pin mismatch, fails the whole run non-zero. - **Genesis keeps its maintained packaging.** A native `xcat-genesis-base` deb is INGESTED as-is when @@ -623,7 +625,7 @@ Codename ↔ version (the single supported set — `BuildUtils` is the source of ## Files - **`sbuild-all.pl`** — the orchestrator (run as **root** on the Ubuntu build host: the amd64 host for - `amd64`, the ppc host for `ppc64el`). + `amd64` and, through qemu-user, for `riscv64`; the ppc host for `ppc64el`). - **`BuildUtils.pm`** — shared, unit-tested helpers + the canonical CLI spec (mirrors `MockBuildUtils.pm`). - **`/sbuild.pl`** ×7 — per-package builders (mirror `/mockbuild.pl`); each drives its maintained `debian/` in the chroot and collects the `.deb`(s). Invoked by `sbuild-all.pl`. @@ -647,6 +649,49 @@ probe is what asserts it is usable. The per-codename sbuild chroots are separate host state — see `ci/mk-dep-chroots.sh`. +## riscv64: cross-building on an amd64 host + +There is no riscv64 Ubuntu build host in the xCAT build farm, so the riscv64 packages are +cross-built on the amd64 host. `sbuild-all.pl` bootstraps a riscv64 `schroot` with +`debootstrap --arch=riscv64`, and every command inside it -- debootstrap's second stage, +`apt-get`, `dpkg-buildpackage` -- runs through the `qemu-riscv64` binfmt handler, so each package +is compiled by the chroot's own riscv64 toolchain. + +`--install-deps` installs `qemu-user-static` and `binfmt-support` with the rest of the toolchain. +Confirm the handler before the first riscv64 run: + +```bash +cat /proc/sys/fs/binfmt_misc/qemu-riscv64 # enabled, flags: POF +``` + +The `F` flag is what makes the handler usable from a chroot: the kernel opens the interpreter when +the handler is registered, so the static QEMU binary does not have to exist under the chroot root. +Without a registered handler `sbuild-all.pl` refuses to create the chroot and names the handler and +the packages that provide it, instead of failing deep inside debootstrap. + +`archive.ubuntu.com` carries amd64 and i386 only; every other architecture is on +`ports.ubuntu.com/ubuntu-ports`. The bootstrap mirror is defaulted from `--arch`, so a riscv64 run +needs no `--mirror`. + +| what | how | +|---|---| +| ipmitool-xcat, conserver-xcat | `dpkg-buildpackage` in the emulated riscv64 chroot | +| goconserver | same chroot, compiled by the Go toolchain the chroot installs for riscv64 | +| grub2-xcat (`Architecture:all`) | built once on amd64 and assembled into the riscv64 index; listed in the riscv64 manifest sections as required-present, because a riscv64 management node needs it to netboot | +| syslinux-xcat, elilo-xcat, xnba-undi (`Architecture:all`) | built once on amd64 and assembled into the riscv64 index; required-present like grub2-xcat, because a riscv64 management node serves the x86 nodes of a mixed cluster | +| xcat-genesis-base | not built: no riscv64 section names it, and the build skips the step when the manifest does not ask for it, so `--skip-genesis` is unnecessary here | + +The riscv64 ipmitool-xcat deb is installed into the chroot that built it and +`/opt/xcat/bin/ipmitool-xcat -V` runs there before the run is called good. A cross-built binary +links against the target's loader and libraries, so the chroot is the only place it can run at all, +and without that check a deb whose binary never executes still builds green. + +Emulated builds are slow. On an 8-core amd64 host, bootstrapping the noble riscv64 chroot took +about 4 minutes and ipmitool-xcat about 12, against seconds natively, so plan a riscv64 run of the +whole dependency set in hours rather than minutes. `--build-timeout` sets the per-package +wall-clock bound. A build that deadlocks under qemu-user is killed with a stall report rather than +hanging the pipeline, which is what goconserver did. + ## Usage (per arch, as root on the matching build host) Run `sbuild-all.pl` on the build host for the arch you are building (amd64 on the x86 Ubuntu host, @@ -668,17 +713,22 @@ finalization step publishes the assembled repo atomically. # ppc64el host — arch-specific deps only (the Architecture:all boot components and both genesis # debs come from the amd64 build): ./sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" --skip-genesis + +# riscv64 — cross-built on the amd64 host (there is no riscv64 build host); arch-specific deps +# only, and the genesis step is skipped by the manifest rather than by a flag: +./sbuild-all.pl --arch riscv64 --dists "focal jammy noble resolute" ``` These runs touch **only** `staging///`; the apt tree at `--apt-dir` is left alone, so -the two hosts can run at the same time. `--dists` may be omitted entirely — with no +they can run at the same time. The build lock is per arch, so the amd64 and riscv64 runs can share +one host. `--dists` may be omitted entirely — with no `--dists`/`--target`, **all supported codenames** are built (`focal jammy noble resolute`). ### Step 2 — publish once, after every arch has staged ```bash ./sbuild-all.pl --skip-build --skip-genesis \ - --publish --expect-arch "amd64 ppc64el" \ + --publish --expect-arch "amd64 ppc64el riscv64" \ --gpg-sign --gpg-key-id xcat@example.com --gpg-home ``` diff --git a/BuildUtils.pm b/BuildUtils.pm index c557b8b..c99005f 100644 --- a/BuildUtils.pm +++ b/BuildUtils.pm @@ -15,9 +15,14 @@ package BuildUtils; use strict; use warnings; use Exporter 'import'; -use File::Basename qw(basename); +use File::Basename qw(basename dirname); +use lib dirname(__FILE__) . '/lib'; +# XCAT::BuildUtils is loaded ON DEMAND, not imported here: it pulls in File::Slurper, which the +# Ubuntu build hosts do not all carry. A compile-time import would make every caller depend on it -- +# including `sbuild-all.pl --install-deps`, the command whose whole job is to install that module on +# a host that lacks it. use File::Copy qw(copy); -use File::Path qw(make_path); +use File::Path qw(make_path remove_tree); use Digest::MD5; use MIME::Base64 qw(encode_base64); @@ -29,7 +34,9 @@ our @EXPORT_OK = qw( parse_packages_index parse_release_architectures resolve_present_names index_has_native_arch control_binary_arch skip_arch_all_on codename_to_version version_to_codename known_codenames + supported_arches is_supported_arch chroot_name chroot_sources_list chroot_is_disposable chroot_build_script + chroot_build_timeout control_field genesis_deb_control deb_field deb_version deb_hash cross_copy_genesis_deb build_deb_in_chroot @@ -48,6 +55,17 @@ my %CODENAME_TO_VERSION = ( my %VERSION_TO_CODENAME = reverse %CODENAME_TO_VERSION; sub known_codenames { return sort keys %CODENAME_TO_VERSION; } + +# The dpkg architectures xcat-dep builds. amd64 is the native one and the single producer of the +# Architecture:all packages; every other one is a secondary architecture, is served by +# ubuntu-ports rather than archive.ubuntu.com, and is built through a qemu-user chroot when the +# build host is amd64. Keep this the single source of truth: --arch, --target, --expect-arch and +# the mirror choice all derive from it. +my @ARCHES = qw(amd64 ppc64el riscv64); +my %ARCH = map { $_ => 1 } @ARCHES; + +sub supported_arches { return @ARCHES; } +sub is_supported_arch { my ($a) = @_; return defined($a) && $ARCH{$a} ? 1 : 0; } sub codename_to_version { my ($c) = @_; return $CODENAME_TO_VERSION{$c // ''}; } sub version_to_codename { my ($v) = @_; return $VERSION_TO_CODENAME{$v // ''}; } @@ -63,9 +81,11 @@ sub install_deps_packages { # NOTE: no libipc-cmd-perl -- IPC::Cmd is CORE on Debian/Ubuntu (it ships in perl-modules) and # no such package exists, so naming it fails the whole install. That it is present is asserted # by the module probe, not by installing a package. + # A foreign-architecture chroot needs the binfmt handler these two packages register. return qw(perl libfile-slurper-perl libparallel-forkmanager-perl sbuild schroot debootstrap apt-utils dpkg-dev devscripts equivs quilt fakeroot - build-essential reprepro gnupg rsync wget git); + build-essential reprepro gnupg rsync wget git + qemu-user-static binfmt-support); } # install_deps_command(): the argv that installs them, non-interactively. @@ -718,6 +738,21 @@ INNER # make_deb.sh); $build runs with CWD = the copied package dir and must leave its .deb(s) somewhere # under the build work tree. Dies on any failure -- including a chroot that is not disposable. # %args: pkg, chroot, pkg_dir, result_dir, build_timestamp, build (required); extra_tools (arrayref, optional) +# chroot_build_timeout($chroot): the wall-clock budget for one build in $chroot. The chroot is named +# --sbuild, so its arch says whether the build is native or runs through qemu-user. +# XCAT_DEP_BUILD_TIMEOUT overrides it; sbuild-all.pl --build-timeout sets that variable, because the +# per-package /sbuild.pl builders are separate processes with their own CLI. 0 disables the bound. +sub chroot_build_timeout { + my ($chroot) = @_; + return int($ENV{XCAT_DEP_BUILD_TIMEOUT}) if defined $ENV{XCAT_DEP_BUILD_TIMEOUT} + && $ENV{XCAT_DEP_BUILD_TIMEOUT} =~ /^\d+$/; + my ($target_arch) = (($chroot // '') =~ /^.+-([^-]+)-sbuild$/); + my $host_arch = `dpkg --print-architecture 2>/dev/null`; + chomp $host_arch; + require XCAT::BuildUtils; + return XCAT::BuildUtils::emulated_build_timeout($target_arch, $host_arch); +} + sub build_deb_in_chroot { my (%a) = @_; defined $a{$_} or die "build_deb_in_chroot: missing '$_'\n" @@ -734,7 +769,17 @@ sub build_deb_in_chroot { . " hide a missing Build-Depends. Add 'union-type=overlay' to its /etc/schroot/chroot.d/ entry\n" . " (or delete the chroot and let sbuild-all.pl re-create it).\n" unless chroot_is_disposable($cfg); + # Build into a private directory UNDER the result dir, and move the debs into it only after + # every check passes. The result dir is what a publish assembles from, and its gate gets no + # further than names and versions, so a deb that lands there before the smoke can be published + # whatever the smoke would have said -- including when a cancellation kills this process before + # any cleanup could run. Same filesystem, so the promotion is a rename. make_path($a{result_dir}); + my $final = $a{result_dir}; + remove_tree($_) for grep { -d $_ } glob("$final/.build-$pkg.*"); + my $stage = "$final/.build-$pkg.$$"; + remove_tree($stage) if -d $stage; + make_path($stage); my $extra = join(' ', @{ $a{extra_tools} || [] }); my $b64 = encode_base64($a{build}, ''); @@ -745,20 +790,77 @@ sub build_deb_in_chroot { my $cmd = 'schroot -c ' . sh_quote($a{chroot}) . ' -u root -d / -- bash -c ' . sh_quote($inner) . ' bash ' - . sh_quote($a{pkg_dir}) . ' ' . sh_quote($a{result_dir}) . ' ' + . sh_quote($a{pkg_dir}) . ' ' . sh_quote($stage) . ' ' . sh_quote($a{build_timestamp}) . ' ' . sh_quote($extra) . ' ' . sh_quote($b64); - print "[$pkg] building in chroot $a{chroot} -> $a{result_dir} (SOURCE_DATE_EPOCH=$a{build_timestamp})\n"; - my $rc = system('bash', '-c', $cmd); - my $ec = $rc == -1 ? -1 : ($rc >> 8); + print "[$pkg] building in chroot $a{chroot} -> $final (SOURCE_DATE_EPOCH=$a{build_timestamp})\n"; + # A build that deadlocks under qemu-user (a resolute goconserver `go build` did, with both Go + # pids in futex_wait and no CPU ticks at all) used to hang here forever, and a hung cell reads as + # "still running" rather than as a defect. Bound it, and print the process tree, each wchan and a + # CPU sample before the kill, so the failure says WHY it stopped. + my $timeout = defined $a{timeout} ? $a{timeout} : chroot_build_timeout($a{chroot}); + require XCAT::BuildUtils; + my $r = XCAT::BuildUtils::run_bounded(cmd => $cmd, timeout => $timeout, label => "[$pkg] build in $a{chroot}", + out => \*STDOUT); + die "[$pkg] build TIMED OUT after $r->{elapsed}s (budget ${timeout}s) -- see the stall report above\n" + if $r->{timed_out}; + my $ec = $r->{ec}; die "[$pkg] build failed (rc=$ec)\n" if $ec != 0; # The debs were copied from INSIDE the chroot; that only reaches the host if --result-dir is on a # bind-mounted path. Verify host-side so a mis-configured (chroot-local) result-dir fails LOUD. - my @debs = glob("$a{result_dir}/*.deb"); - die "[$pkg] build succeeded in the chroot but no .deb is visible at $a{result_dir} on the host\n" + my @debs = glob("$stage/*.deb"); + die "[$pkg] build succeeded in the chroot but no .deb is visible at $stage on the host\n" . " (is --result-dir on a path bind-mounted into the chroot, e.g. under /opt/xcat-ci-shared?)\n" unless @debs; - print "[$pkg] OK (" . scalar(@debs) . " deb(s) in $a{result_dir})\n"; - return scalar @debs; + smoke_deb_in_chroot(%a, result_dir => $stage, debs => \@debs) if $a{smoke}; + + # Everything passed: publish the debs by moving them where the assembly step looks. + my @published; + for my $deb (@debs) { + my $target = "$final/" . basename($deb); + rename($deb, $target) or die "[$pkg] could not publish $deb as $target: $!\n"; + push @published, $target; + } + remove_tree($stage); + print "[$pkg] OK (" . scalar(@published) . " deb(s) in $final)\n"; + return scalar @published; +} + +# smoke_deb_in_chroot(%a): install a produced deb into the chroot it was built in and run a command +# from it. A cross-built binary links against the TARGET's loader and libraries, neither of which +# exists on the host, so the only place it can run is the chroot -- and a deb whose binary is the +# wrong object, or cannot resolve a library, still builds green without this. +# %a{smoke}: { deb => qr/.../, run => 'shell command', expect => qr/.../ } +sub smoke_deb_in_chroot { + my (%a) = @_; + my $s = $a{smoke}; + defined $s->{$_} or die "smoke_deb_in_chroot: missing smoke '$_'\n" for qw(deb run); + my ($deb) = grep { basename($_) =~ $s->{deb} } @{ $a{debs} }; + die "[$a{pkg}] smoke: no produced deb matches $s->{deb} in $a{result_dir}\n" unless $deb; + + my $timeout = chroot_build_timeout($a{chroot}); + my $log = "$a{result_dir}/.smoke-$a{pkg}.log"; + # apt-get, not dpkg -i: it resolves the runtime dependencies the package declares, so a missing + # or wrong Depends fails here instead of on a node. + my $inner = "set -e\n" + . "apt-get -y --no-install-recommends install \"\$1\" >/dev/null 2>&1\n" + . $s->{run} . "\n"; + my $cmd = 'schroot -c ' . sh_quote($a{chroot}) . ' -u root -d / -- bash -c ' + . sh_quote($inner) . ' bash ' . sh_quote($deb) + . ' > ' . sh_quote($log) . ' 2>&1'; + print "[$a{pkg}] smoke: " . basename($deb) . " in $a{chroot}: $s->{run}\n"; + require XCAT::BuildUtils; + my $r = XCAT::BuildUtils::run_bounded(cmd => $cmd, timeout => $timeout, + label => "[$a{pkg}] smoke in $a{chroot}", out => \*STDOUT); + my $out = ''; + if (open my $lfh, '<', $log) { local $/; $out = <$lfh>; close $lfh; } + die "[$a{pkg}] smoke TIMED OUT after $r->{elapsed}s (budget ${timeout}s), log $log\n" + if $r->{timed_out}; + die "[$a{pkg}] smoke failed (rc=$r->{ec}), log $log:\n$out\n" if $r->{ec} != 0; + die "[$a{pkg}] smoke ran but its output does not match $s->{expect}, log $log:\n$out\n" + if $s->{expect} and $out !~ $s->{expect}; + unlink $log; + print "[$a{pkg}] smoke OK ($a{chroot} ran the packaged binary)\n"; + return 1; } 1; diff --git a/MockBuildUtils.pm b/MockBuildUtils.pm index aa9065b..a03c2f7 100644 --- a/MockBuildUtils.pm +++ b/MockBuildUtils.pm @@ -20,7 +20,7 @@ our @EXPORT_OK = qw( parse_evr evr_cmp evr_constraint_ok parse_pin rpmkeys_checksig_problem rpm_version rpm_release rpm_sigmd5 rpm_is_signed restamp_release_line cross_copy_genesis finalize_xcat_dep bump_dep_release_suffix - build_mock_uniqueext + build_mock_uniqueext rpm_arch rpm_in_cell ); # install_deps_packages($os_id): the host packages mockbuild-all.pl needs to run at all, for the @@ -568,6 +568,32 @@ sub bump_dep_release_suffix { # their roots apart, so three concurrent el8/el9/el10 ppc64le goconserver builds race in one root. # When the id is too long, keep a readable leading token AND append a short digest of the FULL id, so # distinct ids always yield distinct uniqueext regardless of where in the string they differ. +# rpm_arch($rpm): the architecture of an rpm. The header decides when the file can be read, so a +# renamed file does not pass for another architecture; a bare file name falls back to its suffix. +sub rpm_arch { + my ($rpm) = @_; + return unless defined $rpm; + if (-f $rpm) { + my $arch = `rpm -qp --qf '%{ARCH}' ${\ sh_quote($rpm)} 2>/dev/null`; + chomp $arch; + return $arch if $arch ne ''; + } + my ($arch) = $rpm =~ /\.([A-Za-z0-9_]+)\.rpm$/; + return $arch; +} + +# rpm_in_cell($rpm, $target_arch): whether an rpm belongs in the repository cell of $target_arch. +# A noarch builder run in another architecture's chroot (the x86 boot loaders for riscv64) can +# emit that chroot's native rpms beside the noarch one; only noarch and the cell's own +# architecture are kept. +sub rpm_in_cell { + my ($rpm, $target_arch) = @_; + my $arch = rpm_arch($rpm); + return 0 unless defined $arch && defined $target_arch; + return 1 if $arch eq 'noarch'; + return $arch eq $target_arch ? 1 : 0; +} + sub build_mock_uniqueext { my ($run, $seq, $label) = @_; diff --git a/debs-manifest.conf b/debs-manifest.conf index 4fd8c93..f4707b4 100644 --- a/debs-manifest.conf +++ b/debs-manifest.conf @@ -51,92 +51,133 @@ # sbuild-all.pl reads this file and, per target, builds ONLY the listed packages and then validates # that EVERY listed package was produced at the pinned version -- any missing or mismatched package # fails the whole run (zero tolerance; concern #4). The four supported codenames (focal jammy noble -# resolute) each get an amd64 and a ppc64el section. +# resolute) each get an amd64, a ppc64el and a riscv64 section. +# +# riscv64 lists the x86 boot loaders like ppc64el does: a riscv64 management node serves the +# x86 nodes of a mixed cluster, so its repository must carry them. It differs from ppc64el in one +# way: xcat-genesis-base is not listed, because riscv64 Genesis is the OpenEmbedded one, published +# once into the shared pool, not the legacy per-arch netboot image. # ============================ focal (ubuntu20.04) ============================ [focal-amd64] -ipmitool-xcat=1.8.18-4 +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* [focal-ppc64el] -ipmitool-xcat=1.8.18-4 +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* -# ============================ jammy (ubuntu22.04) ============================ -[jammy-amd64] -ipmitool-xcat=1.8.18-4 +[focal-riscv64] +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 +elilo-xcat=3.14-6 +xnba-undi=1.21.1-1 + +# ============================ jammy (ubuntu22.04) ============================ +[jammy-amd64] +ipmitool-xcat=1.8.18-5 +conserver-xcat=8.2.1-1 +goconserver=0.3.3-snap* +syslinux-xcat=3.86-2 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* [jammy-ppc64el] -ipmitool-xcat=1.8.18-4 +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* -# ============================ noble (ubuntu24.04) ============================ -[noble-amd64] -ipmitool-xcat=1.8.18-4 +[jammy-riscv64] +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 +elilo-xcat=3.14-6 +xnba-undi=1.21.1-1 + +# ============================ noble (ubuntu24.04) ============================ +[noble-amd64] +ipmitool-xcat=1.8.18-5 +conserver-xcat=8.2.1-1 +goconserver=0.3.3-snap* +syslinux-xcat=3.86-2 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* [noble-ppc64el] -ipmitool-xcat=1.8.18-4 +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* -# ============================ resolute (ubuntu26.04) ========================= -[resolute-amd64] -ipmitool-xcat=1.8.18-4 +[noble-riscv64] +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 +elilo-xcat=3.14-6 +xnba-undi=1.21.1-1 + +# ============================ resolute (ubuntu26.04) ========================= +[resolute-amd64] +ipmitool-xcat=1.8.18-5 +conserver-xcat=8.2.1-1 +goconserver=0.3.3-snap* +syslinux-xcat=3.86-2 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* [resolute-ppc64el] -ipmitool-xcat=1.8.18-4 +ipmitool-xcat=1.8.18-5 conserver-xcat=8.2.1-1 goconserver=0.3.3-snap* syslinux-xcat=3.86-2 -grub2-xcat=2.12-1 +grub2-xcat=2.12-2 elilo-xcat=3.14-6 xnba-undi=1.21.1-1 xcat-genesis-base=2.* +[resolute-riscv64] +ipmitool-xcat=1.8.18-5 +conserver-xcat=8.2.1-1 +goconserver=0.3.3-snap* +syslinux-xcat=3.86-2 +grub2-xcat=2.12-2 +elilo-xcat=3.14-6 +xnba-undi=1.21.1-1 + # [shared] is NOT a build target. It describes the ONE pool the OpenEmbedded Genesis release is # published into (pool/main/xcat-genesis-openembedded), which every suite indexes and which no # [-] section covers. Without it nothing asserted the published pool was COMPLETE: @@ -155,3 +196,4 @@ xcat-genesis-openembedded-ppc64le=2.* xcat-genesis-openembedded-armv7hf=2.* xcat-genesis-openembedded-aarch64=2.* xcat-genesis-openembedded-riscv64=2.* +xcat-genesis-openembedded-s390x=2.* diff --git a/genesis-openembedded/README.md b/genesis-openembedded/README.md index 33075a0..ba33982 100644 --- a/genesis-openembedded/README.md +++ b/genesis-openembedded/README.md @@ -23,7 +23,7 @@ The packaging scripts use `File::Slurper` and `IPC::Cmd`. Install The default format is `all`, which produces RPM, SRPM, and DEB packages. Use `--format rpm` or `--format deb` when only one package family is needed. The supported image architectures are `x86`, `x86_64`, `ppc64`, `ppc64le`, -`armv7hf`, `aarch64`, and `riscv64`. +`armv7hf`, `aarch64`, `riscv64`, and `s390x`. Use `--architecture` for development builds. Repository publication requires a complete release built with `--all`. @@ -68,7 +68,7 @@ perl ./mockbuild-all.pl \ perl ./sbuild-all.pl \ --genesis-release /path/to/xcat-genesis-release \ - --publish --expect-arch "amd64 ppc64el" \ + --publish --expect-arch "amd64 ppc64el riscv64" \ [other build options] ``` @@ -88,8 +88,13 @@ APT metadata is assembled in a side tree and swapped onto the published repository with a single rename, under one global publish lock, so a failed or interrupted publication leaves the previous repository exactly as it was. -Both consumers require all seven architectures and verify package identities -and checksums before publication. A management node can install an image for a +Both consumers verify package identities and checksums before publication. +Releases containing `s390x` use format version 2. Other releases keep version 1 +and remain readable by older tools. Current publishers require all eight +architectures, so a version 1 release cannot replace the shared repository. +Building `s390x` requires an xcat-core revision that reports the target. Merge +or deploy that xcat-core change first. Builds for the other architectures still +accept older xcat-core revisions. A management node can install an image for a different target architecture. Without `--genesis-release`, both builders keep their existing behavior. The @@ -113,4 +118,5 @@ Run the package tests on a Linux builder with RPM, DEB, and repository tools: prove t/build_utils.t prove -It/lib t/genesis_openembedded_release.t sudo -E prove -It/lib t/genesis_openembedded_consumer.t +sudo -E prove -It/lib t/common-repo-gate.t ``` diff --git a/genesis-openembedded/activate b/genesis-openembedded/activate index 9a027a4..88d5c20 100755 --- a/genesis-openembedded/activate +++ b/genesis-openembedded/activate @@ -39,7 +39,7 @@ genesis_run_mknb() { genesis_activation_main() { architecture=$1 case "$architecture" in - x86|x86_64|ppc64|ppc64le|armv7hf|aarch64|riscv64) ;; + x86|x86_64|ppc64|ppc64le|armv7hf|aarch64|riscv64|s390x) ;; *) echo "Invalid Genesis architecture: $architecture" >&2 return 0 diff --git a/genesis-openembedded/build b/genesis-openembedded/build index 41fa3ba..375bd6f 100755 --- a/genesis-openembedded/build +++ b/genesis-openembedded/build @@ -23,6 +23,7 @@ use XCAT::BuildUtils qw( use XCAT::GenesisRelease qw( architectures deb_package_name + minimum_release_version rpm_package_name validate_architecture validate_release @@ -77,6 +78,28 @@ if ($xcat_ref ne '') { die "xcat-core HEAD $revision does not match $xcat_ref ($expected)\n" unless $revision eq $expected; } +if ($requested{s390x}) { + my $architecture_reporter = "$xcat_source/xCAT-genesis-builder/oe/build"; + my $probe_work = tempdir('xcat-genesis-capabilities.XXXXXX', TMPDIR => 1, CLEANUP => 1); + my $supported_output; + { + local $ENV{XCAT_GENESIS_WORK_DIR} = $probe_work; + local $ENV{TMPDIR} = $probe_work; + $supported_output = eval { + capture_command($architecture_reporter, '--list-architectures'); + }; + } + my $report_error = $@; + die "xcat-core source does not report supported Genesis architectures: $report_error" + if $report_error; + if (!defined($supported_output) || !length $supported_output) { + die "xcat-core source reported no supported Genesis architectures\n"; + } + my %supported_architecture = map { $_ => 1 } split m{\s+}xms, $supported_output; + if (!$supported_architecture{s390x}) { + die "xcat-core source does not support Genesis architecture s390x\n"; + } +} my $version = read_first_line("$xcat_source/Version"); die "Invalid xCAT version: $version\n" unless $version =~ /^\d+(?:\.\d+){1,3}$/; @@ -140,10 +163,11 @@ for my $architecture (@requested_architectures) { } my @formats = $format eq 'all' ? qw(deb rpm) : ($format); +my $manifest_version = minimum_release_version(@requested_architectures); write_binary( "$staging/release.manifest", "format=xcat-genesis-packages\n" - . "version=1\n" + . "version=$manifest_version\n" . "xcat_version=$version\n" . "xcat_release=$release\n" . "xcat_revision=$revision\n" diff --git a/ipmitool/debian/changelog b/ipmitool/debian/changelog index 7805a5d..259f78c 100644 --- a/ipmitool/debian/changelog +++ b/ipmitool/debian/changelog @@ -1,3 +1,10 @@ +ipmitool-xcat (1.8.18-5) unstable; urgency=low + + * Depend on the shared libraries the binary links against. Readline and + ncurses were missing, so the package installed and then failed to run + + -- XCAT Sun, 06 Sep 2026 12:00:00 +0000 + ipmitool-xcat (1.8.18-4) unstable; urgency=low * Build with dependencies for either libssl1.0.0 or libssl1.1 diff --git a/ipmitool/debian/control b/ipmitool/debian/control index 14e57bf..5d4ce95 100644 --- a/ipmitool/debian/control +++ b/ipmitool/debian/control @@ -6,8 +6,8 @@ Build-Depends: debhelper (>= 12), libreadline-dev, libssl-dev, quilt Standards-Version: 3.6.2.1 Package: ipmitool-xcat -Architecture: i386 amd64 ia64 ppc64el -Depends: libc6 (>= 2.15), libssl3t64 | libssl3 | libssl1.1 +Architecture: i386 amd64 ia64 ppc64el riscv64 +Depends: ${shlibs:Depends}, libc6 (>= 2.15), libssl3t64 | libssl3 | libssl1.1 Suggests: openipmi Description: utility for IPMI control with kernel driver or LAN interface A utility for managing and configuring devices that support the diff --git a/ipmitool/sbuild.pl b/ipmitool/sbuild.pl index cf26a4a..266384d 100755 --- a/ipmitool/sbuild.pl +++ b/ipmitool/sbuild.pl @@ -18,9 +18,9 @@ my $pkg_dir = abs_path($RealBin); my $pkg = basename($pkg_dir); my ($codename, $arch, $chroot, $result_dir, $log_dir) = ('', '', '', '', ''); my ($build_timestamp, $build_number, $skip_install) = (undef, undef, 0); -# --log-dir / --build-number / --skip-install are accepted for CLI-compat with sbuild-all.pl (which -# passes them uniformly to every per-package builder) but are intentionally UNUSED here: sbuild-all -# does its own per-package logging and there is no deb install-smoke. They are parsed and ignored. +# --log-dir / --build-number are accepted for CLI-compat with sbuild-all.pl (which passes them +# uniformly to every per-package builder) but are intentionally UNUSED here: sbuild-all does its own +# per-package logging. --skip-install drops the post-build smoke. GetOptions( 'codename=s' => \$codename, 'arch=s' => \$arch, 'chroot=s' => \$chroot, 'result-dir=s' => \$result_dir, 'log-dir=s' => \$log_dir, @@ -34,19 +34,29 @@ $result_dir ||= "$pkg_dir/../build-output/sbuild/$codename/$arch"; $build_timestamp = time() unless defined $build_timestamp; # ---- package-specific build (absorbed from the former make_deb.sh); CWD = the copied package dir ---- -# ipmitool is a compiled C package (Architecture: amd64 ppc64el), so it is built per-codename against +# ipmitool is a compiled C package, so it is built per-codename against # that release's libc/toolchain. Extract the upstream tarball, drop in the maintained debian/, build. -my $build = <<'BUILD'; +my $version = '1.8.18'; +my $build = <<"BUILD"; set -e -VERSION=1.8.18 -tar xvfz ipmitool-$VERSION.tar.gz -cd ipmitool-$VERSION +VERSION=$version +tar xvfz ipmitool-\$VERSION.tar.gz +cd ipmitool-\$VERSION cp -rL ../debian . -HOST_ARCH=$(dpkg --print-architecture) -TARGET_ARCH=$HOST_ARCH dpkg-buildpackage -uc -us +HOST_ARCH=\$(dpkg --print-architecture) +TARGET_ARCH=\$HOST_ARCH dpkg-buildpackage -uc -us BUILD +# The chroot is the only place a cross-built binary can run: it holds the target's loader and +# libraries. Without this an unrunnable ipmitool-xcat ships as a green build. +my $smoke = { + deb => qr/^ipmitool-xcat_/, + run => '/opt/xcat/bin/ipmitool-xcat -V', + expect => qr/ipmitool-xcat version \Q$version\E/i, +}; + build_deb_in_chroot( pkg => $pkg, chroot => $chroot, pkg_dir => $pkg_dir, result_dir => $result_dir, build_timestamp => $build_timestamp, build => $build, + ($skip_install ? () : (smoke => $smoke)), ); diff --git a/lib/XCAT/BuildUtils.pm b/lib/XCAT/BuildUtils.pm index c64a518..a464318 100644 --- a/lib/XCAT/BuildUtils.pm +++ b/lib/XCAT/BuildUtils.pm @@ -10,6 +10,8 @@ use File::Find qw(find); use File::Slurper qw(read_binary write_binary); use File::Spec; use IPC::Cmd qw(can_run); +use POSIX (); +use Time::HiRes (); our @EXPORT_OK = qw( capture_command @@ -18,6 +20,10 @@ our @EXPORT_OK = qw( digest_manifest display_quote every_step_failed + forward_signals_to_workers + block_handled_signals + exit_status + restore_signal_mask hashes_equal print_step read_binary @@ -25,8 +31,11 @@ our @EXPORT_OK = qw( read_lines relative_files require_command + run_bounded run_command shell_quote + stall_report + emulated_build_timeout write_binary ); @@ -64,6 +73,277 @@ sub run_command { . join(' ', map { display_quote($_) } @command) . "\n"; } +# --------------------------------------------------------------------------------------------------- +# Bounded command execution (build steps that can run under emulation) +# --------------------------------------------------------------------------------------------------- + +# NATIVE_BUILD_TIMEOUT: the per-package wall-clock budget for a build on the host architecture. +# EMULATION_FACTOR: qemu-user under TCG runs at roughly a tenth of native speed, so a foreign-arch +# target gets ten times the budget. Both numbers come from measured runs of this tree, not a guess: +# on xcat-master-ub the slowest native (amd64) package took 3 minutes, and the slowest emulated +# (riscv64) package -- ipmitool-xcat on resolute -- took 26 minutes with four codenames building at +# once. 900s and 9000s therefore sit about five times above the worst build ever measured, which is +# the margin that keeps a busy host from producing a false failure. +our $NATIVE_BUILD_TIMEOUT = 900; +our $EMULATION_FACTOR = 10; + +# emulated_build_timeout($target_arch, $host_arch): the budget for a build of $target_arch on +# $host_arch. Equal arches are native. Any other pair is emulated through qemu-user. +sub emulated_build_timeout { + my ($target_arch, $host_arch) = @_; + return $NATIVE_BUILD_TIMEOUT + if !defined($target_arch) || !defined($host_arch) || $target_arch eq $host_arch; + return $NATIVE_BUILD_TIMEOUT * $EMULATION_FACTOR; +} + +# proc_pgid_pids($pgid): every live pid in process group $pgid, read from /proc rather than matched +# against a command line. `pgrep -af X | grep -c Y` counts its own command line, and bracketing the +# pattern still self-matches when the surrounding command carries the bare word, so this code never +# matches text at all. +sub proc_pgid_pids { + my ($pgid) = @_; + my @pids; + opendir(my $dh, '/proc') or return (); + for my $e (sort { $a <=> $b } grep { /^\d+$/ } readdir($dh)) { + my $f = _proc_stat_fields($e) or next; + push @pids, $e if defined($f->{pgrp}) && $f->{pgrp} == $pgid; + } + closedir($dh); + return @pids; +} + +# _proc_stat_fields($pid): the /proc//stat fields this module reads. The comm field is wrapped +# in parentheses and may itself contain a space or a parenthesis, so the split starts after the LAST +# ')'. +sub _proc_stat_fields { + my ($pid) = @_; + open(my $fh, q{<}, "/proc/$pid/stat") or return; + my $line = <$fh>; + close($fh); + return unless defined $line; + my $close = rindex($line, ')'); + return if $close < 0; + my $comm = substr($line, index($line, '(') + 1, $close - index($line, '(') - 1); + my @f = split(' ', substr($line, $close + 2)); + return { + comm => $comm, + state => $f[0], + ppid => $f[1], + pgrp => $f[2], + ticks => (($f[11] // 0) + ($f[12] // 0)), + }; +} + +sub _proc_read { + my ($path) = @_; + open(my $fh, '<', $path) or return ''; + local $/; + my $t = <$fh> // ''; + close($fh); + $t =~ s/\0/ /g; + $t =~ s/\s+\z//; + return $t; +} + +# _proc_socket_count($pid): how many of the pid's descriptors are sockets. A build that is waiting on +# the network holds one; the deadlock this bound exists for held none. +sub _proc_socket_count { + my ($pid) = @_; + opendir(my $dh, "/proc/$pid/fd") or return -1; + my $n = 0; + for my $fd (grep { /^\d+$/ } readdir($dh)) { + my $l = readlink("/proc/$pid/fd/$fd") // ''; + $n++ if $l =~ /^socket:/; + } + closedir($dh); + return $n; +} + +# stall_report($pgid, %opt): the evidence a hung build needs, gathered before the kill. For every pid +# in the process group it prints the parent, the state, the kernel wchan and stack, the socket count, +# and the CPU ticks the pid consumed across a sample window. The tick delta is what separates a +# deadlock from a slow build: a package that is merely slow keeps accumulating ticks, and the +# goconserver deadlock this bound exists for accumulated none over twenty seconds. +# %opt: sample (seconds, default 20), out (filehandle, default STDERR) +sub stall_report { + my ($pgid, %opt) = @_; + my $sample = defined $opt{sample} ? $opt{sample} : 20; + my $out = $opt{out} || \*STDERR; + + my @pids = proc_pgid_pids($pgid); + my %before = map { $_ => (_proc_stat_fields($_) || {})->{ticks} } @pids; + sleep($sample) if $sample > 0; + my @after_pids = proc_pgid_pids($pgid); + my %seen = map { $_ => 1 } @pids; + push @pids, grep { !$seen{$_} } @after_pids; + + my $total = 0; + print {$out} "--- stall report: process group $pgid, ${sample}s CPU sample ---\n"; + printf {$out} "%-8s %-8s %-5s %-10s %-10s %-24s %s\n", + 'PID', 'PPID', 'STAT', 'TICKS', 'DELTA', 'WCHAN', 'CMD'; + for my $pid (@pids) { + my $f = _proc_stat_fields($pid) or next; + my $delta = ($f->{ticks} // 0) - ($before{$pid} // 0); + $delta = 0 if $delta < 0; + $total += $delta; + my $cmd = _proc_read("/proc/$pid/cmdline") || "[$f->{comm}]"; + $cmd = substr($cmd, 0, 120); + printf {$out} "%-8s %-8s %-5s %-10s %-10s %-24s %s\n", + $pid, $f->{ppid}, $f->{state}, $f->{ticks}, $delta, + (_proc_read("/proc/$pid/wchan") || '?'), $cmd; + my $stack = _proc_read("/proc/$pid/stack"); + print {$out} " stack: $_\n" for grep { length } split(/\n/, $stack); + my $socks = _proc_socket_count($pid); + print {$out} " sockets: $socks\n" if $socks >= 0; + } + print {$out} "--- CPU ticks consumed by the whole group during the sample: $total\n"; + print {$out} $total == 0 + ? "--- no CPU ticks: the build is DEADLOCKED, not slow.\n" + : "--- the build still consumes CPU: it exceeded the budget rather than deadlocking.\n"; + return $total; +} + +# exit_status($status): the exit code of a waited-for child, or 128 plus the signal that killed it. +# A child killed by a signal has 0 in the high byte, so reading only that byte reports a build the +# kernel terminated as a build that succeeded. +sub exit_status { + my ($status) = @_; + return 0 unless defined $status; + return ($status & 127) ? 128 + ($status & 127) : $status >> 8; +} + +# block_handled_signals(): block INT, TERM and HUP and return the previous mask, for the window +# between forking a child and being able to signal it. A cancellation arriving in that window would +# otherwise kill the parent under a handler that does not know the child yet, and the child would +# keep running. A blocked signal stays pending and is delivered by restore_signal_mask(). +sub block_handled_signals { + my $handled = POSIX::SigSet->new(POSIX::SIGINT(), POSIX::SIGTERM(), POSIX::SIGHUP()); + my $previous = POSIX::SigSet->new(); + POSIX::sigprocmask(POSIX::SIG_BLOCK(), $handled, $previous); + return $previous; +} + +# restore_signal_mask($previous): put the mask back, delivering anything that arrived meanwhile. +sub restore_signal_mask { + my ($previous) = @_; + POSIX::sigprocmask(POSIX::SIG_SETMASK(), $previous) if $previous; + return; +} + +# forward_signals_to_workers(%a): return an INT/TERM/HUP handler that passes the signal on to the +# forked workers, waits for them, then re-raises it. An orchestrator that dies without this releases +# its locks while its workers keep building and writing into staging, and the next run races +# processes it cannot see. run_bounded covers the build inside ONE worker; this covers the workers. +# %a: pids (hashref keyed by live worker pid), reap (coderef that waits for them) +sub forward_signals_to_workers { + my (%a) = @_; + my $pids = $a{pids} or die "forward_signals_to_workers: missing 'pids'\n"; + my $reap = $a{reap} or die "forward_signals_to_workers: missing 'reap'\n"; + return sub { + my ($sig) = @_; + kill($sig, keys %{$pids}); + $reap->(); + # die by the same signal, so the caller's exit status says what happened + $SIG{$sig} = 'DEFAULT'; + kill($sig, $$); + }; +} + +# run_bounded(%a): run a shell command with a wall-clock budget. On expiry it prints a stall report +# and kills the whole process group, so a build that deadlocks fails LOUDLY instead of hanging a +# pipeline forever -- an unbounded hang reads as "still running", never as a defect. +# The command runs in its own process group: a build spawns schroot, mock, qemu and make, and only a +# group signal reaches all of them. Signalling the group also means no child survives holding the +# caller's stdout open, which would turn the timeout back into a hang one level up. +# %a: cmd (required), timeout (seconds; <=0 runs without a deadline), label, sample, out +# Returns { ec, timed_out, elapsed }. ec is 124 on a timeout, matching timeout(1). +sub run_bounded { + my (%a) = @_; + my $cmd = defined $a{cmd} ? $a{cmd} : die "run_bounded: missing 'cmd'\n"; + my $timeout = $a{timeout} || 0; + my $label = defined $a{label} ? $a{label} : 'command'; + my $out = $a{out} || \*STDERR; + my $t0 = time; + + # Block the handled signals across the fork. A cancellation landing between fork() and the + # handler below would kill this process under the inherited handler and leave the new process + # group running. A blocked signal stays pending and is delivered once the handler is in place. + # The child restores the mask before exec, or the build would inherit a blocked TERM. + my $previous = block_handled_signals(); + + my $pid = fork(); + unless (defined $pid) { + restore_signal_mask($previous); + die "run_bounded: fork failed: $!\n"; + } + if ($pid == 0) { + POSIX::setpgid(0, 0); + restore_signal_mask($previous); + exec('bash', '-c', $cmd) or POSIX::_exit(127); + } + # setpgid from BOTH sides: whichever runs first wins, so the group exists before the first signal + # whatever the scheduler does. + POSIX::setpgid($pid, $pid); + + # A signal to this process must reach the build too. Without this the orchestrator dies and + # releases its locks while schroot, mock and qemu keep writing into staging, so the next run + # races an orphan it cannot see. + my $reap_group = sub { + kill('TERM', -$pid); + for (1 .. 40) { + last if waitpid($pid, POSIX::WNOHANG()) == $pid; + Time::HiRes::sleep(0.5); + } + kill('KILL', -$pid); + waitpid($pid, 0); + }; + my $forward = sub { + my ($sig) = @_; + $reap_group->(); + # die by the same signal, so the caller's exit status says what happened + $SIG{$sig} = 'DEFAULT'; + kill($sig, $$); + }; + local $SIG{INT} = $forward; + local $SIG{TERM} = $forward; + local $SIG{HUP} = $forward; + restore_signal_mask($previous); + + # An unbounded run still forks: it is the process group, not the deadline, that lets a signal + # to the orchestrator reach the build. + my $deadline = $timeout > 0 ? $t0 + $timeout : undef; + my $timed_out = 0; + my $status; + while (1) { + my $r = waitpid($pid, POSIX::WNOHANG()); + if ($r == $pid) { $status = $?; last; } + if ($r == -1) { $status = 0; last; } + if (defined $deadline and time >= $deadline) { $timed_out = 1; last; } + Time::HiRes::sleep(0.5); + } + + if ($timed_out) { + my $el = time - $t0; + print {$out} "\nFATAL: $label exceeded its ${timeout}s budget (ran ${el}s).\n"; + eval { stall_report($pid, sample => (defined $a{sample} ? $a{sample} : 20), out => $out); 1 } + or print {$out} "stall report failed: $@"; + # Unconditional final reap: a child left unreaped keeps the group alive and the caller + # blocks on output that never ends. + $reap_group->(); + return { ec => 124, timed_out => 1, elapsed => time - $t0 }; + } + + my $ec = exit_status($status); + # The leader can die without its descendants. A SIGKILL, or the OOM killer, takes the shell but + # leaves schroot and qemu in its process group, still holding the chroot and writing into + # staging after this call reports the build finished. They are not children of this process, so + # there is nothing to wait for: signal the group and move on. + kill('TERM', -$pid); + Time::HiRes::sleep(0.2); + kill('KILL', -$pid); + return { ec => $ec, timed_out => 0, elapsed => time - $t0 }; +} + sub display_quote { my ($value) = @_; return $value if $value =~ /^[A-Za-z0-9_.,+\/:=@~-]+$/; diff --git a/lib/XCAT/GenesisRelease.pm b/lib/XCAT/GenesisRelease.pm index b07368f..b05f9a1 100644 --- a/lib/XCAT/GenesisRelease.pm +++ b/lib/XCAT/GenesisRelease.pm @@ -8,20 +8,32 @@ use XCAT::BuildUtils qw(digest_file read_lines relative_files); our @EXPORT_OK = qw( architectures + deb_package_prefix deb_package_name + minimum_release_version read_checksum_manifest read_release_manifest + rpm_package_prefix rpm_package_name validated_release_checksums validate_architecture validate_complete_release validate_export + validate_repository_packages validate_release verify_release_file ); -my @ARCHITECTURES = qw(x86 x86_64 ppc64 ppc64le armv7hf aarch64 riscv64); +my @RELEASE_V1_ARCHITECTURES = qw(x86 x86_64 ppc64 ppc64le armv7hf aarch64 riscv64); +my @RELEASE_V2_ARCHITECTURES = (@RELEASE_V1_ARCHITECTURES, 's390x'); +my @ARCHITECTURES = @RELEASE_V2_ARCHITECTURES; +my $RPM_PACKAGE_PREFIX = 'xCAT-genesis-openembedded-'; +my $DEB_PACKAGE_PREFIX = 'xcat-genesis-openembedded-'; my %ARCHITECTURE = map { $_ => 1 } @ARCHITECTURES; +my %RELEASE_ARCHITECTURES = ( + 1 => \@RELEASE_V1_ARCHITECTURES, + 2 => \@RELEASE_V2_ARCHITECTURES, +); sub architectures { return @ARCHITECTURES; @@ -34,17 +46,48 @@ sub validate_architecture { return $architecture; } +sub rpm_package_prefix { + return $RPM_PACKAGE_PREFIX; +} + sub rpm_package_name { my ($architecture) = @_; validate_architecture($architecture); - return "xCAT-genesis-openembedded-$architecture"; + return $RPM_PACKAGE_PREFIX . $architecture; +} + +sub deb_package_prefix { + return $DEB_PACKAGE_PREFIX; } sub deb_package_name { my ($architecture) = @_; validate_architecture($architecture); $architecture =~ tr/_/-/; - return "xcat-genesis-openembedded-$architecture"; + return $DEB_PACKAGE_PREFIX . $architecture; +} + +sub minimum_release_version { + my @architectures = @_; + die "Release format selection requires a Genesis architecture\n" unless @architectures; + validate_architecture($_) for @architectures; + for my $version (sort { $a <=> $b } keys %RELEASE_ARCHITECTURES) { + my %supported = map { $_ => 1 } @{ $RELEASE_ARCHITECTURES{$version} }; + return $version unless grep { !$supported{$_} } @architectures; + } + die "No release format supports the requested Genesis architectures\n"; +} + +sub validate_repository_packages { + my ($packages, $section, $prefix, @supported_names) = @_; + my %supported = map { $_ => 1 } @supported_names; + my @missing = grep { !exists $packages->{$_} } @supported_names; + my @unknown = grep { + index($_, $prefix) == 0 && !$supported{$_} + } sort keys %{$packages}; + die "FATAL: [$section] is missing supported packages: @missing\n" if @missing; + die "FATAL: [$section] has unsupported packages: @unknown\n" if @unknown; + return $packages; } sub _read_key_values { @@ -161,7 +204,7 @@ sub _validate_release { die "Unsupported Genesis package release format\n" unless $manifest->{format} eq 'xcat-genesis-packages'; die "Unsupported Genesis package release version\n" - unless $manifest->{version} eq '1'; + unless $RELEASE_ARCHITECTURES{ $manifest->{version} }; die "Invalid xCAT version in release manifest\n" unless $manifest->{xcat_version} =~ /^\d+(?:\.\d+){1,3}$/; die "Invalid xCAT release in release manifest\n" @@ -172,9 +215,13 @@ sub _validate_release { unless $manifest->{source_date_epoch} =~ /^\d+$/; my @architectures = split(/,/, $manifest->{architectures}); + my %version_architecture = map { $_ => 1 } + @{ $RELEASE_ARCHITECTURES{ $manifest->{version} } }; my %seen_arch; for my $architecture (@architectures) { validate_architecture($architecture); + die "Genesis architecture $architecture is not valid in release version $manifest->{version}\n" + unless $version_architecture{$architecture}; die "Duplicate release architecture: $architecture\n" if $seen_arch{$architecture}++; } die "Release manifest has no architectures\n" unless @architectures; @@ -223,7 +270,8 @@ sub validate_complete_release { my $manifest = validate_release($directory); my %present = map { $_ => 1 } split(/,/, $manifest->{architectures}); my @missing = grep { !$present{$_} } @ARCHITECTURES; - die "Genesis release is missing supported architectures: @missing\n" if @missing; + die "Genesis release version $manifest->{version} omits currently supported architectures: @missing\n" + if @missing; return $manifest; } diff --git a/mock-configs/rocky-10-riscv64-xcat.cfg b/mock-configs/rocky-10-riscv64-xcat.cfg index 686ea67..2c591f5 100644 --- a/mock-configs/rocky-10-riscv64-xcat.cfg +++ b/mock-configs/rocky-10-riscv64-xcat.cfg @@ -4,11 +4,13 @@ # used from the x86_64 hosts that produce the xcat-dep repos; this config includes the same # template and adds forcearch. There is no EPEL for riscv64, so nothing EPEL-only is reachable # from this chroot: mockbuild-all.pl builds the EPEL-only perl deps of xCAT for this target. +# The host is x86_64 only: the noarch boot loaders the target ships build in the native +# rocky-10-x86_64 chroot, and syslinux builds on x86 and ppc64le alone. # See BUILD.md ("riscv64") for the host prerequisites and how mockbuild-all.pl uses it. include('templates/rocky-10.tpl') config_opts['root'] = 'rocky-10-riscv64-xcat' config_opts['description'] = 'Rocky Linux 10 riscv64 (xCAT dependency build, forcearch)' config_opts['target_arch'] = 'riscv64' -config_opts['legal_host_arches'] = ('x86_64', 'riscv64') +config_opts['legal_host_arches'] = ('x86_64',) config_opts['forcearch'] = 'riscv64' diff --git a/mockbuild-all.pl b/mockbuild-all.pl index f108ff0..d46e207 100755 --- a/mockbuild-all.pl +++ b/mockbuild-all.pl @@ -15,7 +15,7 @@ use Parallel::ForkManager; use POSIX qw(strftime); use FindBin qw($RealBin); use lib $RealBin, "$RealBin/lib"; -use MockBuildUtils qw(sh_quote print_step version_matches required_pkgs +use MockBuildUtils qw(sh_quote print_step version_matches required_pkgs rpm_in_cell install_deps_packages install_deps_command missing_perl_modules read_manifest verify_repo_packages verify_repo_signature verify_rpm_signatures rpm_version rpm_release rpm_sigmd5 restamp_release_line @@ -26,13 +26,22 @@ use MockBuildUtils qw(sh_quote print_step version_matches required_pkgs use XCAT::BuildUtils qw( capture_command every_step_failed + forward_signals_to_workers + block_handled_signals + restore_signal_mask hashes_equal read_lines + emulated_build_timeout require_command + run_bounded run_command shell_quote ); use XCAT::GenesisRelease qw( + architectures + rpm_package_prefix + rpm_package_name + validate_repository_packages validated_release_checksums verify_release_file ); @@ -88,6 +97,9 @@ my $parallel_targets = 1; # 1 = serial (default; safe). 0/auto = all EL target # NOTE: parallel targets need every per-package mockbuild.pl to avoid # shared-path writes (repo tarballs, $HOME/rpmbuild); serial is safe today. my $max_parallel = 0; # 0/auto = host nproc: global cap on concurrent mock builds (all targets) +# Per-build-step wall-clock bound for the dep and perl steps. undef = derived from the target arch +# (a forcearch target is cross-built through qemu-user); an explicit 0 removes the bound. +my $build_timeout; my $run_id = ''; my $build_timestamp; # CD version bump: when set, every xcat-dep package spec's Release gets a @@ -157,6 +169,7 @@ GetOptions( 'parallel-builds=i' => \$parallel_builds, 'parallel-targets=i' => \$parallel_targets, 'max-parallel=i' => \$max_parallel, + 'build-timeout=i' => \$build_timeout, 'run-id=s' => \$run_id, 'build-timestamp=i' => \$build_timestamp, 'build-number=i' => \$build_number, @@ -364,6 +377,7 @@ if ($genesis_release ne '') { unless -d $genesis_release; my $verifier = "$script_dir/genesis-openembedded/verify-release"; die "Genesis release verifier not found: $verifier\n" unless -x $verifier; + common_repository_requirements(); # Checksum, verify, checksum again. The verifier reads the tree it validates, so a # release rewritten together with its SHA256SUMS while the verifier runs would satisfy # both the verifier and any single pass taken afterwards; comparing the pass taken @@ -387,17 +401,19 @@ my @build_targets = $target # What a target builds. The mock-core-configs targets (+epel--) build every # dep natively on the host arch. The forcearch targets shipped in mock-configs/ cross-build -# another arch that has no EPEL: the x86-only bootloaders are not built for it, the EPEL-only -# perl deps of xCAT are (mockbuild-perl-packages.pl --epel-gap), and the noarch deps are built -# in the native, EPEL-free chroot of the same release (the rpms are identical for every arch -# and an emulated build is an order of magnitude slower). See BUILD.md ("riscv64"). +# another arch that has no EPEL: the EPEL-only perl deps of xCAT are built for it +# (mockbuild-perl-packages.pl --epel-gap), and the noarch deps, the x86 boot loaders among them, +# are built in the native, EPEL-free chroot of the same release (the rpms are identical for +# every arch and an emulated build is an order of magnitude slower). See BUILD.md ("riscv64"). my %forcearch_targets = ( 'rocky-10-riscv64-xcat' => { rel => 10, arch => 'riscv64', - noarch_cfg => "rocky-10-$host_arch", - dep_builders => [qw(grub2-xcat ipmitool-xcat goconserver conserver-xcat)], - required => [qw(ipmitool-xcat grub2-xcat perl-IO-Stty perl-HTTP-Async perl-Net-HTTPS-NB)], + # x86_64 only, as the mock config admits: syslinux-xcat builds on x86 and ppc64le alone. + noarch_cfg => 'rocky-10-x86_64', + dep_builders => [qw(elilo-xcat grub2-xcat ipmitool-xcat syslinux-xcat goconserver conserver-xcat xnba-undi)], + required => [qw(ipmitool-xcat syslinux-xcat grub2-xcat xnba-undi + perl-IO-Stty perl-HTTP-Async perl-Net-HTTPS-NB)], }, ); @@ -432,12 +448,38 @@ print "parallel_targets: " . ($parallel_targets > 0 ? $parallel_targets : "auto( print "max_parallel: $cap (per-target build workers: $per_target_builds)\n"; my $tgt_pm = Parallel::ForkManager->new($tgt_workers <= 1 ? 0 : $tgt_workers); my $tgt_fail = 0; +# A signal to this process must reach the forked workers. Without forwarding, the orchestrator exits +# and releases its locks while the workers keep building and writing into the deploy tree. +my %tgt_kids; +$tgt_pm->run_on_start(sub { $tgt_kids{ $_[0] } = 1 }); $tgt_pm->run_on_finish(sub { - my ($pid, $exit) = @_; - $tgt_fail++ if $exit; + my ($pid, $exit, $ident, $signal, $core_dump) = @_; + delete $tgt_kids{$pid}; + # A worker killed by a signal exits with code 0 in this callback, so reading the code alone + # reports a target that died mid-deploy as built. + $tgt_fail++ if $exit or $signal or $core_dump; }); +my $tgt_forward = forward_signals_to_workers( + pids => \%tgt_kids, + reap => sub { $tgt_pm->wait_all_children }, +); +local $SIG{INT} = $tgt_forward; +local $SIG{TERM} = $tgt_forward; +local $SIG{HUP} = $tgt_forward; for my $tgt (@build_targets) { - $tgt_pm->start and next; + # ForkManager records the worker in run_on_start, which runs AFTER the fork, so the handler + # cannot signal a worker that arrives in between. Hold the signals across both -- but wait for + # a free slot FIRST, with them unblocked, or a cancellation would sit pending for as long as + # the pool stays full. + # max_procs 0 is ForkManager's no-fork mode, where asking for a slot is an error. + $tgt_pm->wait_for_available_procs(1) if $tgt_pm->max_procs; + my $orchestrator = $$; + my $previous = block_handled_signals(); + if ($tgt_pm->start) { restore_signal_mask($previous); next; } + # With one worker ForkManager does not fork at all, and this is still the orchestrator: only a + # real child drops the forwarder, whose copy names siblings the parent already signals. + $SIG{$_} = 'DEFAULT' for ($$ == $orchestrator ? () : qw(INT TERM HUP)); + restore_signal_mask($previous); my $rc = 0; eval { my $info = build_one_target($tgt, $run_id, $per_target_builds); @@ -497,11 +539,12 @@ if (!$skip_build && !$dry_run && -d $run_root) { # committed artifacts (an x86 UNDI ROM / the grub2 resource tarball) with no arch-specific build # step, so ppc builds them the same as x86 -- no cross-arch import. A forcearch target builds # only the builders its profile lists; the noarch ones run in the profile's native chroot. +# syslinux-xcat is noarch too, and its spec builds on x86 and ppc64le only. my @dep_builders = ( { name => 'elilo-xcat', script => "$repo_root/elilo/mockbuild.pl", noarch => 1 }, { name => 'grub2-xcat', script => "$repo_root/grub2-xcat/mockbuild.pl", noarch => 1 }, { name => 'ipmitool-xcat', script => "$repo_root/ipmitool/mockbuild.pl" }, - { name => 'syslinux-xcat', script => "$repo_root/syslinux/mockbuild.pl" }, + { name => 'syslinux-xcat', script => "$repo_root/syslinux/mockbuild.pl", noarch => 1 }, { name => 'goconserver', script => "$repo_root/goconserver/mockbuild.pl" }, { name => 'conserver-xcat', script => "$repo_root/conserver/mockbuild.pl" }, { name => 'xnba-undi', script => "$repo_root/xnba/mockbuild.pl", noarch => 1 }, @@ -596,6 +639,15 @@ if (!$skip_build) { my @build_steps; my $build_step_seq = 0; + # Bound only what can run emulated. A forcearch target (rocky-10-riscv64-xcat) cross-builds every + # dep through qemu-user, where a deadlock burns no CPU and never returns. Native mock steps keep + # their present, unbounded behaviour: no measurement of them exists here, and a bound guessed for + # a step that legitimately runs long would turn a trusted cell red for no reason. + # --build-timeout overrides both; 0 removes the bound. + my $step_timeout = defined $build_timeout ? $build_timeout + : $profile->{forcearch} ? emulated_build_timeout($arch, $host_arch) + : 0; + if (!$skip_xcat_dep) { for my $builder (@active_dep_builders) { next unless $req{ $builder->{name} }; # manifest: build only required dep packages @@ -604,9 +656,10 @@ if (!$skip_build) { my $step_result = "$build_root/$name"; my $step_log = "$log_root/$name"; my $step_uniqueext = build_mock_uniqueext($run_id, ++$build_step_seq, $name); + my $mock_cfg = $builder->{noarch} ? $profile->{noarch_cfg} : $target; my $cmd = join(' ', 'perl', shell_quote($script), - '--mock-cfg', shell_quote($builder->{noarch} ? $profile->{noarch_cfg} : $target), + '--mock-cfg', shell_quote($mock_cfg), ($profile->{forcearch} && !$builder->{noarch} ? ('--target-arch', shell_quote($arch)) : ()), '--mock-uniqueext', shell_quote($step_uniqueext), '--result-dir', shell_quote($step_result), @@ -624,11 +677,12 @@ if (!$skip_build) { : ()), ); push @build_steps, { - id => "xcat-dep:$name", - step => "Build xcat-dep: $name", - cmd => $cmd, - log => "$log_root/$name/run.log", - scrub_cfg => $target, + id => "xcat-dep:$name", + step => "Build xcat-dep: $name", + cmd => $cmd, + timeout => $step_timeout, + log => "$log_root/$name/run.log", + scrub_cfg => $mock_cfg, scrub_uniqueext => $step_uniqueext, }; push @collect_roots, $step_result; @@ -664,10 +718,13 @@ if (!$skip_build) { ($keep_buildroots ? '--keep-buildroots' : ()), ); push @build_steps, { - id => 'perl', - step => 'Build perl xcat-dep packages', - cmd => $cmd, - log => "$log_root/perl-build.log", + id => 'perl', + step => 'Build perl xcat-dep packages', + cmd => $cmd, + # The perl builder forks its own mock jobs, so under forcearch it is emulated too. Give it + # the per-package budget times the number of packages it builds serially per worker. + timeout => ($step_timeout ? $step_timeout * scalar(@perl_pkgs) : 0), + log => "$log_root/perl-build.log", }; push @collect_roots, $perl_result; } @@ -797,11 +854,13 @@ print_step('Collect RPM artifacts'); print "collection roots:\n"; print " $_\n" for @collect_roots; -my ($copied, $skipped_src, $missing_roots) = collect_rpms( +my ($copied, $skipped_src, $missing_roots, $skipped_foreign) = collect_rpms( roots => \@collect_roots, dest_dir => $repo_dir, + arch => $arch, dry_run => $dry_run, ); +print "skipped $skipped_foreign rpm(s) of another architecture\n" if $skipped_foreign; # Assert on what this run BUILT, before the Genesis release is added: the release is # installed from a verified directory rather than built here, so counting it first would @@ -1083,9 +1142,8 @@ sub publish_genesis_common_repo { =head3 verify_common_repo - Assert the shared OpenEmbedded Genesis repository carries every package the manifest's [common] - section requires, at a version satisfying its pin. [common] is not a build target: it describes - the one repository published beside the per-EL cells, which no [] section covers. + Assert the shared repository carries every package required by [common]. [common] must + describe every currently supported Genesis architecture. Arguments: $dir - the repository to check (the staging directory, before it is swapped into place) @@ -1097,16 +1155,11 @@ sub publish_genesis_common_repo { #-------------------------------------------------------------------------------- sub verify_common_repo { my ($dir) = @_; - my $manifest = "$repo_root/packages-manifest.conf"; - my %MAN = read_manifest($manifest); - my %req = %{ $MAN{common} // {} }; - die "FATAL: no [common] section in $manifest -- cannot verify the shared Genesis repository\n" - if !%req; - - my @names = sort keys %req; - my %present = repo_present_versions($dir, \@names); + my %req = %{ common_repository_requirements() }; + my @names = sort keys %req; + my %present = repo_present_versions($dir, \@names); my %present_evr = map { $_ => rpm_evr($dir, $_) } @names; - my @problems = verify_repo_packages(\%req, \%present, \%present_evr, \&rpm_vercmp_segment); + my @problems = verify_repo_packages(\%req, \%present, \%present_evr, \&rpm_vercmp_segment); if (@problems) { print " - $_\n" for @problems; die "FATAL: shared Genesis repo INCOMPLETE at $dir (" . scalar(@problems) . " problem(s))\n"; @@ -1116,6 +1169,21 @@ sub verify_common_repo { return 1; } +sub common_repository_requirements { + my $manifest = "$repo_root/packages-manifest.conf"; + my %MAN = read_manifest($manifest); + my %common = %{ $MAN{common} // {} }; + die "FATAL: no [common] section in $manifest -- cannot verify the shared Genesis repository\n" + if !%common; + + return validate_repository_packages( + \%common, + 'common', + rpm_package_prefix(), + map { rpm_package_name($_) } architectures(), + ); +} + sub replace_common_repository { my ($staged, $destination) = @_; my $backup = "$repo_dep/.common.previous.$$"; @@ -1351,6 +1419,12 @@ Options: cross-builds that arch on this host); default is the host arch across rh8, rh9 and rh10 --nproc N Parallel jobs for buildrpms.pl (default: 1) + --build-timeout SECONDS Wall-clock bound for one build step. Default: none for a native + target, and 9000 for a forcearch (qemu-user) target, which runs at + roughly a tenth of native speed. 0 removes the bound. On expiry the + run prints the step's process tree, each pid's wchan and stack, and + a 20-second CPU sample -- a deadlocked build uses no ticks -- then + kills the whole process group. --parallel-builds N Max concurrent top-level build steps within one EL target (default: auto) --parallel-targets N Concurrent EL targets (rh8/rh9/rh10). 0/auto = all at once, 1 = serial, N = cap at N. Each target is fully output-isolated (default: 1 = serial) @@ -1422,9 +1496,15 @@ sub run_step { $full_cmd .= " > " . shell_quote($log) . " 2>&1"; } - my $rc = system($full_cmd); - if ($rc != 0) { - my $exit = $rc == -1 ? 255 : ($rc >> 8); + # A forcearch step cross-builds through qemu-user, where a deadlocked build consumes no CPU and + # never exits. Bound the steps that can run emulated, and report why the build stopped. + my $timeout = $args{timeout} || 0; + my $r = run_bounded(cmd => $full_cmd, timeout => $timeout, label => $step, out => \*STDOUT); + die "Step TIMED OUT after $r->{elapsed}s (budget ${timeout}s): $step\nCommand: $cmd\n" + . " See the stall report above" . ($log ? " and $log" : '') . ".\n" + if $r->{timed_out}; + if ($r->{ec} != 0) { + my $exit = $r->{ec} == -1 ? 255 : $r->{ec}; die "Step failed (rc=$exit): $step\nCommand: $cmd\n"; } } @@ -1488,9 +1568,12 @@ sub run_build_steps_parallel { my %failed; my $pm = Parallel::ForkManager->new($workers); + my %kids; + $pm->run_on_start(sub { $kids{ $_[0] } = 1 }); $pm->run_on_finish( sub { my ($pid, $exit_code, $ident, $signal, $core_dump) = @_; + delete $kids{$pid}; return if $exit_code == 0 && $signal == 0 && !$core_dump; my $key = defined($ident) ? $ident : "pid:$pid"; $failed{$key} = { @@ -1501,13 +1584,29 @@ sub run_build_steps_parallel { } ); + # Same reason as the per-target workers: a signal here must reach the builds these workers run. + my $forward = forward_signals_to_workers( + pids => \%kids, + reap => sub { $pm->wait_all_children }, + ); + local $SIG{INT} = $forward; + local $SIG{TERM} = $forward; + local $SIG{HUP} = $forward; + for my $step (@{$steps}) { my %step_copy = %{$step}; my $ident = delete $step_copy{id}; $ident = $step_copy{step} if !defined($ident) || $ident eq ''; + # Same window as the per-target workers, and the same rule about waiting for a slot with + # the signals unblocked. + $pm->wait_for_available_procs(1) if $pm->max_procs; + my $orchestrator = $$; + my $previous = block_handled_signals(); my $pid = $pm->start($ident); - next if $pid; + if ($pid) { restore_signal_mask($previous); next; } + $SIG{$_} = 'DEFAULT' for ($$ == $orchestrator ? () : qw(INT TERM HUP)); + restore_signal_mask($previous); my $ok = eval { run_step(%step_copy); @@ -1888,11 +1987,13 @@ sub collect_rpms { my (%args) = @_; my $roots = $args{roots} // []; my $dest = $args{dest_dir} // die "collect_rpms missing dest_dir\n"; + my $cell_arch = $args{arch} // die "collect_rpms missing arch\n"; my $is_dry = $args{dry_run} ? 1 : 0; my %seen; my $copied = 0; my $skipped_src = 0; + my $skipped_foreign = 0; my $missing_roots = 0; for my $root (@{$roots}) { @@ -1920,6 +2021,10 @@ sub collect_rpms { my $base = basename($rpm); next if $genesis_release && $base =~ /^xCAT-genesis-openembedded-/; + if (!rpm_in_cell($rpm, $cell_arch)) { + $skipped_foreign++; + next; + } next if $seen{$base}++; if ($is_dry) { print "DRY-RUN copy: $rpm -> $dest/$base\n"; @@ -1932,7 +2037,7 @@ sub collect_rpms { } } - return ($copied, $skipped_src, $missing_roots); + return ($copied, $skipped_src, $missing_roots, $skipped_foreign); } sub collect_srpms { @@ -2150,4 +2255,3 @@ sub slurp_chomp { chomp $line if defined $line; return $line // ''; } - diff --git a/packages-manifest.conf b/packages-manifest.conf index de53504..5370461 100644 --- a/packages-manifest.conf +++ b/packages-manifest.conf @@ -144,19 +144,22 @@ perl-Sys-Virt=11.10.0 xCAT-genesis-base=>= 2:2.18.0 # rocky-10-riscv64-xcat is the forcearch (cross-built) EL10 riscv64 target. It differs from the -# EPEL-fed EL10 sections above in two ways (see BUILD.md, "riscv64"): -# - the x86-only boot components (elilo-xcat, syslinux-xcat, xnba-undi) are not built for it; -# - riscv64 has no EPEL, so the perl deps EL10 otherwise takes from EPEL are built here too -# (perl-Crypt-Blowfish ... perl-Path-Class below). perl-Path-Class is a build dep of -# perl-Crypt-SSLeay only. +# EPEL-fed EL10 sections above in one way (see BUILD.md, "riscv64"): riscv64 has no EPEL, so the +# perl deps EL10 otherwise takes from EPEL are built here too (perl-Crypt-Blowfish ... +# perl-Path-Class below). perl-Path-Class is a build dep of perl-Crypt-SSLeay only. The x86 boot +# loaders (elilo-xcat, syslinux-xcat, xnba-undi) are noarch and are listed like on ppc64le: a +# riscv64 management node serves the x86 nodes of a mixed cluster. # The per-EL perl set is the EL10 one plus perl-HTML-Form: EPEL supplies it on the other # architectures, nothing supplies it on riscv64, and perl-xCAT requires perl(HTML::Form). # xCAT-genesis-base is deliberately not listed: the riscv64 repo is built with --skip-genesis. [rocky-10-riscv64-xcat] conserver-xcat=8.2.1 +elilo-xcat=3.14 goconserver=>= 0.3.3-snap202011021058 grub2-xcat=1.0 ipmitool-xcat=>= 1.8.18-4 +syslinux-xcat=>= 6.03-1 +xnba-undi=>= 1.21.1-1 perl-Crypt-SSLeay=0.72 perl-HTML-Form=6.07 perl-HTTP-Async=>= 0.30-3 @@ -192,3 +195,4 @@ xCAT-genesis-openembedded-ppc64le=>= 2.18.0 xCAT-genesis-openembedded-armv7hf=>= 2.18.0 xCAT-genesis-openembedded-aarch64=>= 2.18.0 xCAT-genesis-openembedded-riscv64=>= 2.18.0 +xCAT-genesis-openembedded-s390x=>= 2.18.0 diff --git a/sbuild-all.pl b/sbuild-all.pl index c22aef1..80044e3 100755 --- a/sbuild-all.pl +++ b/sbuild-all.pl @@ -50,7 +50,8 @@ use BuildUtils qw(sh_quote print_step version_matches required_pkgs read_manifes verify_repo_packages verify_repo_signature verify_repo_arches parse_packages_index parse_release_architectures resolve_present_names index_has_native_arch control_binary_arch skip_arch_all_on - codename_to_version known_codenames chroot_name chroot_sources_list + codename_to_version known_codenames supported_arches is_supported_arch + chroot_name chroot_sources_list chroot_is_disposable control_field genesis_deb_control deb_field deb_version deb_hash cross_copy_genesis_deb); @@ -73,6 +74,9 @@ my $build_number; # arches on their two hosts in parallel, "all 4 codenames per host" gives 8 concurrent build streams # (4 per host). N caps it to N; 1 forces serial. my $parallel_targets = 0; +# Per-package wall-clock bound. undef = the arch-derived default in BuildUtils::chroot_build_timeout +# (native vs qemu-user emulated); an explicit 0 disables the bound. +my $build_timeout; my ($skip_build, $skip_install, $skip_genesis, $skip_xcat_dep) = (0,0,0,0); my $install_deps = 0; my ($skip_createrepo, $skip_tarball) = (0,0); @@ -174,6 +178,7 @@ $spec{'apt-dir=s'} = \$apt_dir; $spec{'mirror=s'} = \$mirror; $spec{'gpg-key-id=s'} = \$gpg_key_id; $spec{'parallel-targets=i'} = \$parallel_targets; +$spec{'build-timeout=i'} = \$build_timeout; # per-package wall-clock bound (0 = unbounded) $spec{'genesis-deb=s'} = \@genesis_debs; $spec{'genesis-rpm=s'} = \$genesis_rpm; $spec{'genesis-rpm-ppc=s'} = \$genesis_rpm_ppc; @@ -220,16 +225,17 @@ $xcat_src = abs_path($xcat_src) if -d $xcat_src; $manifest ||= "$repo_root/debs-manifest.conf"; $arch ||= `dpkg --print-architecture 2>/dev/null`; chomp $arch; $arch ||= 'amd64'; -die "FATAL: unsupported --arch '$arch' (amd64|ppc64el)\n" unless $arch =~ /^(amd64|ppc64el)$/; -# Arch-aware chroot bootstrap mirror: ppc64el is NOT served by archive.ubuntu.com -- it lives on -# ubuntu-ports. Only defaulted when --mirror was not given explicitly. -$mirror ||= ($arch eq 'ppc64el') ? 'http://ports.ubuntu.com/ubuntu-ports' - : 'http://br.archive.ubuntu.com/ubuntu'; +die "FATAL: unsupported --arch '$arch' (@{[join '|', supported_arches()]})\n" unless is_supported_arch($arch); +# Arch-aware chroot bootstrap mirror: only amd64 is on archive.ubuntu.com. Every secondary +# architecture -- ppc64el and riscv64 -- lives on ubuntu-ports. Only defaulted when --mirror was not +# given explicitly. +$mirror ||= ($arch ne 'amd64') ? 'http://ports.ubuntu.com/ubuntu-ports' + : 'http://br.archive.ubuntu.com/ubuntu'; # --target "-" pins a single codename (and cross-checks the arch); otherwise --dists. my @dist_list; -if ($dists =~ /-(amd64|ppc64el)$/) { - my ($cn, $a) = $dists =~ /^(.+)-(amd64|ppc64el)$/; +if ($dists =~ /-(@{[join '|', supported_arches()]})$/) { + my ($cn, $a) = $dists =~ /^(.+)-(@{[join '|', supported_arches()]})$/; die "FATAL: --target arch '$a' != host --arch '$arch'\n" if $a ne $arch; @dist_list = ($cn); } else { @@ -243,7 +249,7 @@ for my $cn (@dist_list) { @expect_arch = grep { length } map { split /[\s,]+/ } @expect_arch; for my $a (@expect_arch) { - die "FATAL: unsupported --expect-arch '$a' (amd64|ppc64el)\n" unless $a =~ /^(amd64|ppc64el)$/; + die "FATAL: unsupported --expect-arch '$a' (@{[join '|', supported_arches()]})\n" unless is_supported_arch($a); } # --------------------------------------------------------------------------------------------------- @@ -312,6 +318,7 @@ if ($genesis_release ne '') { # the verifier runs would satisfy both the verifier and any single pass taken afterwards. require XCAT::BuildUtils; require XCAT::GenesisRelease; + shared_repository_requirements(); my $before = XCAT::GenesisRelease::validated_release_checksums($genesis_release); XCAT::BuildUtils::run_command($^X, $verifier, '--complete', '--format', 'deb', $genesis_release); my $after = XCAT::GenesisRelease::validated_release_checksums($genesis_release); @@ -356,6 +363,10 @@ unless ($dry_run) { } } +# The per-package builders are separate processes with their own CLI, so the bound travels to them in +# the environment. Without it each one derives the same default from its chroot arch. +$ENV{XCAT_DEP_BUILD_TIMEOUT} = $build_timeout if defined $build_timeout; + print_step('Configuration'); print " repo-root: $repo_root\n"; print " xcat-source: $xcat_src\n"; @@ -372,6 +383,10 @@ print " publish: " . ($publish print " expect-arch: " . (@expect_arch ? "@expect_arch" : '(derive from the staged arch set)') . "\n" if $publish; print " dry-run: " . ($dry_run ? "yes" : "no") . "\n"; +print " build-timeout: " . (defined $build_timeout + ? ($build_timeout > 0 ? "${build_timeout}s (explicit)" : 'disabled') + : BuildUtils::chroot_build_timeout(chroot_name($dist_list[0], $arch)) . "s (derived: " + . ($arch eq (do { my $h = `dpkg --print-architecture 2>/dev/null`; chomp $h; $h }) ? 'native' : 'emulated') . ")") . "\n"; # --------------------------------------------------------------------------------------------------- # Helpers @@ -456,6 +471,37 @@ sub ensure_disposable_chroot { return 1; } +# A chroot for another architecture is bootstrapped and built through qemu-user: debootstrap's +# second stage and every later build run the target's own binaries. Without a registered binfmt +# handler that fails deep inside debootstrap, so it is checked here, where the message can name what +# is missing. +my %BINFMT_HANDLER = ( + riscv64 => 'qemu-riscv64', + ppc64el => 'qemu-ppc64le', +); + +sub ensure_foreign_arch_support { + my ($target) = @_; + + my $host = `dpkg --print-architecture 2>/dev/null`; + chomp $host; + return if (!$host or $target eq $host); + + my $handler = $BINFMT_HANDLER{$target} or return; + my $node = "/proc/sys/fs/binfmt_misc/$handler"; + my $enabled = 0; + if (open my $fh, '<', $node) { + local $/; + $enabled = (<$fh> // '') =~ /^enabled/m ? 1 : 0; + close $fh; + } + return if $enabled; + + die "FATAL: building $target on a $host host runs the target's binaries through qemu-user,\n" + . " but the $handler binfmt handler is not registered (looked at $node).\n" + . " Install qemu-user-static and binfmt-support, then re-run.\n"; +} + sub ensure_chroots { print_step('Ensure sbuild chroots (auto-init on first run)'); die "FATAL: chroot init requires root (uid=$>)\n" if $> != 0 && !$dry_run; @@ -473,6 +519,7 @@ sub ensure_chroots { next; } print " chroot $name: MISSING -> creating\n"; + ensure_foreign_arch_support($arch); # debootstrap may lack a script for a new codename -> fall back to the generic one. run("[ -e /usr/share/debootstrap/scripts/$cn ] || ln -sf gutsy /usr/share/debootstrap/scripts/$cn", nofail => 1); my $root = "/srv/chroot/$cn-$arch"; @@ -548,7 +595,17 @@ sub build_one_codename { '>', sh_quote($log), '2>&1', ); print " [$cn] -> $pkg ($dir/sbuild.pl)\n"; - my $ec = run($cmd, nofail => 1); + # run_bounded, not run(): it puts the builder in its own process group and forwards a + # signal to it. system() would leave the builder, its schroot session and qemu running + # after this worker died, holding the chroot and writing into staging. The wall-clock + # bound belongs to the builder itself, so this call only carries the cancellation. + print "+ $cmd\n"; + my $ec = 0; + unless ($dry_run) { + require XCAT::BuildUtils; + $ec = XCAT::BuildUtils::run_bounded(cmd => $cmd, timeout => 0, + label => "[$cn] $pkg", out => \*STDOUT)->{ec}; + } if ($ec != 0) { warn "FATAL: [$cn] $pkg build failed (rc=$ec) -- see $log\n"; return 1; } } print "== [$cn] done ==\n"; @@ -574,17 +631,42 @@ sub build_deps { my @queue = @dist_list; my (%pid2cn, %fail); my $running = 0; + require XCAT::BuildUtils; + my $forward = XCAT::BuildUtils::forward_signals_to_workers( + pids => \%pid2cn, + reap => sub { waitpid($_, 0) for keys %pid2cn }, + ); + local $SIG{INT} = $forward; + local $SIG{TERM} = $forward; + local $SIG{HUP} = $forward; while (@queue || $running) { while (@queue && $running < $max) { my $cn = shift @queue; + # Block the handled signals across the fork AND the registration below: a cancellation + # in between would reach a handler that does not know this worker yet, and the worker + # would keep building for hours while holding the per-arch lock. + my $previous = XCAT::BuildUtils::block_handled_signals(); my $pid = fork(); - die "FATAL: fork failed: $!\n" unless defined $pid; - if ($pid == 0) { exit(build_one_codename($cn)); } # child + unless (defined $pid) { + XCAT::BuildUtils::restore_signal_mask($previous); + die "FATAL: fork failed: $!\n"; + } + # The child must not inherit the parent's forwarder: its copy names sibling workers, + # which the parent already signals. + if ($pid == 0) { + $SIG{$_} = 'DEFAULT' for qw(INT TERM HUP); + XCAT::BuildUtils::restore_signal_mask($previous); + exit(build_one_codename($cn)); + } $pid2cn{$pid} = $cn; $running++; + XCAT::BuildUtils::restore_signal_mask($previous); } my $pid = wait(); if ($pid > 0) { - my $ec = $? >> 8; + # A worker the kernel killed leaves 0 in the high byte, so the shifted status alone + # would record a cancelled or OOM-killed codename as built. + require XCAT::BuildUtils; + my $ec = XCAT::BuildUtils::exit_status($?); my $cn = delete $pid2cn{$pid} // '?'; $fail{$cn} = $ec if $ec != 0; $running--; @@ -604,7 +686,8 @@ sub maintained_genesis_control { my $f = "$xcat_src/xCAT-genesis-builder/debian/control"; return undef unless -f $f; local $/; open my $fh, '<', $f or return undef; my $t = <$fh>; close $fh; - if ($a eq 'ppc64el') { $t =~ s/amd64/ppc64el/g; } + # The tree carries the amd64 control; any other arch is the same text with the arch renamed. + $t =~ s/amd64/$a/g if $a ne 'amd64'; return $t; } # convert_genesis_rpm($rpm, $pkgname, $arch, $outdir): rpm2cpio-extract the noarch genesis rpm and @@ -637,6 +720,19 @@ sub convert_genesis_rpm { run("dpkg-deb --build " . sh_quote($pkgd) . " " . sh_quote("$outdir/${pkgname}_${ver}_all.deb")); return "$outdir/${pkgname}_${ver}_all.deb"; } +# genesis_in_manifest(): whether the legacy Genesis deb belongs to this run at all. It is named +# per target in the manifest, and riscv64 does not name it: its Genesis is the OpenEmbedded package +# published once into the shared pool. Without this, a plain --arch riscv64 run reaches +# build_genesis and dies for want of a --genesis-deb it can never have, after the dep builds. +sub genesis_in_manifest { + for my $cn (@dist_list) { + my $section = "$cn-$arch"; + next unless $MANIFEST{$section}; + return 1 if grep { /^xcat-genesis-base/ } keys %{ $MANIFEST{$section} }; + } + return 0; +} + sub build_genesis { print_step('Genesis-base deb (maintained packaging preserved)'); my $gen = "$output_root/$run_id/genesis"; wipe_tree($gen) if -d $gen; make_path($gen); @@ -854,7 +950,7 @@ sub resolve_expect_arches { for my $d (glob("$staging/$cn/*")) { next unless -d $d; my $a = basename($d); - $u{$a} = 1 if $a =~ /^(amd64|ppc64el)$/; + $u{$a} = 1 if is_supported_arch($a); } } print " expected arches (from the staged set): " . join(' ', sort keys %u) . "\n"; @@ -911,7 +1007,7 @@ sub verify_assembled_repo { # pure verify_repo_arches can report both directions: expected-but-absent and present-but- # unexpected (a stale arch left behind in the tree). my %native; - for my $a (do { my %s = map { $_ => 1 } (@expected, qw(amd64 ppc64el)); sort keys %s }) { + for my $a (do { my %s = map { $_ => 1 } (@expected, supported_arches()); sort keys %s }) { my $idx = "$adir/dists/$cn/main/binary-$a/Packages"; $native{$a} = 0; next unless -f $idx; @@ -1047,16 +1143,9 @@ sub install_genesis_release_debs { return scalar(@files); } -# verify_shared_pool($pool): assert the shared Genesis pool carries every package the manifest's -# [shared] section requires, at a version satisfying its pin. [shared] is not a build target: it -# describes the one pool every suite indexes, which no [-] section covers. Run on -# the SIDE TREE, before it is swapped into place, so an incomplete pool is never published. -# Completeness only -- the release checksums cover the bytes. sub verify_shared_pool { my ($pool) = @_; - my %req = %{ $MANIFEST{shared} // {} }; - die "FATAL: no [shared] section in $manifest -- cannot verify the shared Genesis pool\n" - if !%req; + my %req = %{ shared_repository_requirements() }; my @names = sort keys %req; my %present = map { $_ => deb_version($pool, $_) } @names; my @problems = verify_repo_packages(\%req, \%present); @@ -1068,6 +1157,19 @@ sub verify_shared_pool { return 1; } +sub shared_repository_requirements { + my %shared = %{ $MANIFEST{shared} // {} }; + die "FATAL: no [shared] section in $manifest -- cannot verify the shared Genesis pool\n" + if !%shared; + return XCAT::GenesisRelease::validate_repository_packages( + \%shared, + 'shared', + XCAT::GenesisRelease::deb_package_prefix(), + map { XCAT::GenesisRelease::deb_package_name($_) } + XCAT::GenesisRelease::architectures(), + ); +} + sub assemble_into { my ($dir, $expect) = @_; if ($genesis_release ne '') { @@ -1272,7 +1374,7 @@ unless ($skip_build) { ensure_chroots(); build_deps(); } -build_genesis() unless $skip_genesis; +build_genesis() unless ($skip_genesis or !genesis_in_manifest()); validate_manifest(); publish_repo(); # no-op unless --publish (or a --skip-build finalization run); tarball is inside print_step("Completed ($arch: @dist_list)" . ($publish ? '' : ' -- staging only, not published')); @@ -1291,9 +1393,10 @@ sbuild-all.pl - build, validate, sign and assemble the xcat-dep Ubuntu/Debian ap sbuild-all.pl --arch amd64 --dists "focal jammy noble resolute" \ --xcat-source ../xcat-core --genesis-rpm sbuild-all.pl --arch ppc64el --dists "focal jammy noble resolute" --skip-genesis + sbuild-all.pl --arch riscv64 --dists "focal jammy noble resolute" # STEP 2 -- ONCE, after every arch has staged: assemble, sign, gate and publish atomically: - sbuild-all.pl --skip-build --skip-genesis --publish --expect-arch "amd64 ppc64el" \ + sbuild-all.pl --skip-build --skip-genesis --publish --expect-arch "amd64 ppc64el riscv64" \ --gpg-sign --gpg-key-id xcat@megware.com --gpg-home # build ONE Ubuntu version only: @@ -1306,7 +1409,7 @@ sbuild-all.pl - build, validate, sign and assemble the xcat-dep Ubuntu/Debian ap # verify an already-published tree out of band (signatures checked by DEFAULT): sbuild-all.pl --verify-repo --dists "focal jammy noble resolute" \ - --expect-arch "amd64 ppc64el" --gpg-key-id --gpg-home + --expect-arch "amd64 ppc64el riscv64" --gpg-key-id --gpg-home sbuild-all.pl --help # option summary sbuild-all.pl --man # this manual @@ -1423,7 +1526,7 @@ C<--skip-tarball>. =over 4 -=item B<--arch> C +=item B<--arch> C Host architecture. Default: C. @@ -1471,10 +1574,11 @@ Publish an B alongside the packages this r release is produced separately (see F); this option only verifies it and copies the verified bytes into every selected suite. -The release must be B (every supported Genesis architecture) and must carry C -packages. It is validated before any build or publish: its C is read, the shared -verifier runs, and the checksums are read again -- a release rewritten together with its checksums -while the verifier runs is rejected. +The release must carry C packages for every currently supported architecture. Version 1 +metadata remains readable, but a complete release now requires C. The release is validated +before any build or publish: its C is read, the shared verifier runs, and the checksums +are read again -- a release rewritten together with its checksums while the verifier runs is +rejected. The packages are published B, into F, and every suite's C index points at that one copy: they are C and identical everywhere, @@ -1504,6 +1608,16 @@ Per-codename build concurrency on this host. Default 0 = auto = build every requ parallel (each in its own chroot); N caps it; 1 forces serial. With the two arches on their two hosts, the default gives 8 concurrent build streams for a 4-codename matrix (4 per host). +=item B<--build-timeout> C + +Wall-clock bound for ONE package build, passed to every per-package builder in +C. C<0> removes the bound. The default is derived from the target +architecture: 900s for a native build, and ten times that for a foreign architecture, because +qemu-user under TCG runs at roughly a tenth of native speed. When the bound expires the run prints +the process tree of the build, each pid's kernel wchan and stack, its open socket count and the CPU +ticks it used over a 20-second sample, then kills the whole process group. The sample is the +evidence that separates a deadlocked build from a slow one. + =item B<--skip-build> B<--skip-install> B<--skip-genesis> B<--skip-xcat-dep> B<--skip-createrepo> B<--skip-tarball> Skip the corresponding phase(s). C<--skip-build --skip-genesis> is the finalization run (it publishes @@ -1527,10 +1641,10 @@ swap it onto C<--apt-dir> atomically. B), which is the finalization step. C<--no-publish> forces it off. -=item B<--expect-arch> C<< amd64|ppc64el >> +=item B<--expect-arch> C<< amd64|ppc64el|riscv64 >> The architecture set the published repo must serve, stated explicitly. Repeatable, and each value may -be a space/comma list (C<--expect-arch "amd64 ppc64el">). Used by the gate: an expected arch with no +be a space/comma list (C<--expect-arch "amd64 ppc64el riscv64">). Used by the gate: an expected arch with no native package is C, an unexpected arch that published natives is C, and C advertises exactly this set. Without it the gate falls back to the staged arch set when publishing, or to each codename's C C when verifying standalone. diff --git a/t/build_timeout.t b/t/build_timeout.t new file mode 100644 index 0000000..4e4c16a --- /dev/null +++ b/t/build_timeout.t @@ -0,0 +1,403 @@ +#!/usr/bin/perl +# Behaviour test for the wall-clock bound on emulated build steps. It DRIVES run_bounded with a +# command that hangs and asserts that the call returns, fails, and prints the evidence -- it never +# reads the source of the thing it tests. +# +# Every call under test runs in a forked child whose stdio is detached to a file, and the parent +# bounds that child itself. Without the bound in run_bounded the child would block forever; a child +# holding this test's stdout would block prove instead of failing it, so the failure of the code +# under test must show up as a FAILED assertion here, never as a hung suite. +use strict; +use warnings; + +use File::Temp qw(tempdir); +use FindBin; +use POSIX (); +use Test::More; + +use lib "$FindBin::Bin/../lib", "$FindBin::Bin/.."; +use XCAT::BuildUtils qw(run_bounded emulated_build_timeout block_handled_signals restore_signal_mask exit_status); +use BuildUtils qw(chroot_build_timeout); + +my $tmp = tempdir(CLEANUP => 1); +my @strays; +# waitpid on an already-gone child sets $? to -1, and Test::Builder's END reads $? as the exit +# status. Save it, or a clean run reports "exited with -1". +END { local $?; for my $p (@strays) { kill('KILL', $p); waitpid($p, 0); } } + +# drive(): run one run_bounded call in a detached child and wait at most {deadline} seconds for it. +# Returns the child's result plus whether it finished on its own -- "did not finish" is the assertion +# that catches an unbounded run_bounded, which would otherwise hang this test. +sub drive { + my (%a) = @_; + my $out = "$tmp/$a{name}.out"; + my $res = "$tmp/$a{name}.res"; + my $t0 = time; + + my $pid = fork(); + die "fork: $!" unless defined $pid; + if ($pid == 0) { + open(STDIN, '<', '/dev/null'); + open(STDOUT, '>', $out) or POSIX::_exit(90); + open(STDERR, '>&', \*STDOUT); + POSIX::setpgid(0, 0); + my $r = eval { + run_bounded(cmd => $a{cmd}, timeout => $a{timeout}, + sample => $a{sample}, label => $a{name}, out => \*STDOUT); + } || { ec => -99, timed_out => 0, elapsed => -1 }; + if (open(my $fh, '>', $res)) { print {$fh} "$r->{ec} $r->{timed_out} $r->{elapsed}\n"; close($fh); } + POSIX::_exit(0); + } + POSIX::setpgid($pid, $pid); + + my $deadline = $t0 + $a{deadline}; + my $finished = 0; + while (time < $deadline) { + if (waitpid($pid, POSIX::WNOHANG()) == $pid) { $finished = 1; last; } + select(undef, undef, undef, 0.2); + } + # Reap unconditionally: an unreaped child keeps its group alive and the next test inherits it. + unless ($finished) { kill('KILL', -$pid); waitpid($pid, 0); } + + my ($ec, $timed_out, $elapsed) = (-1, 0, -1); + if (open(my $fh, '<', $res)) { ($ec, $timed_out, $elapsed) = split(' ', <$fh> // ''); close($fh); } + my $text = ''; + if (open(my $fh, '<', $out)) { local $/; $text = <$fh> // ''; close($fh); } + return { finished => $finished, ec => $ec, timed_out => $timed_out, + elapsed => $elapsed, out => $text, wall => time - $t0 }; +} + +# --- the budget itself ----------------------------------------------------------------------- +is(emulated_build_timeout('amd64', 'amd64'), 900, 'a native build gets the native budget'); +is(emulated_build_timeout('riscv64', 'amd64'), 9000, + 'an emulated build gets ten times the native budget'); +{ + local $ENV{XCAT_DEP_BUILD_TIMEOUT} = 42; + is(chroot_build_timeout('noble-riscv64-sbuild'), 42, 'XCAT_DEP_BUILD_TIMEOUT overrides the budget'); +} +{ + local %ENV = %ENV; delete $ENV{XCAT_DEP_BUILD_TIMEOUT}; + is(chroot_build_timeout('noble-riscv64-sbuild'), 9000, + 'a riscv64 chroot on this host derives the emulated budget'); +} + +# --- a command that exits normally is not touched -------------------------------------------- +my $ok = drive(name => 'exit0', cmd => 'exit 0', timeout => 60, sample => 1, deadline => 30); +ok($ok->{finished}, 'a fast command returns'); +is($ok->{ec}, 0, 'its exit status is 0'); +is($ok->{timed_out}, 0, 'it is not reported as timed out'); + +my $bad = drive(name => 'exit3', cmd => 'exit 3', timeout => 60, sample => 1, deadline => 30); +is($bad->{ec}, 3, 'a failing command keeps its exit status'); + +# --- THE DEFECT: a build that hangs must fail, within the budget, with evidence --------------- +my $hang = drive(name => 'hang', cmd => 'exec sleep 600 >/dev/null 2>&1', + timeout => 4, sample => 2, deadline => 90); +ok($hang->{finished}, 'a hanging command does NOT hang the caller') + or BAIL_OUT('run_bounded never returned: the bound is missing, so a hung build has no failure'); +is($hang->{timed_out}, 1, 'the hang is reported as a timeout'); +is($hang->{ec}, 124, 'the timeout exit status is 124, as timeout(1) uses'); +cmp_ok($hang->{wall}, '<', 60, 'it fails soon after the budget, not later'); +like($hang->{out}, qr/exceeded its 4s budget/, 'the failure names the budget it exceeded'); +like($hang->{out}, qr/stall report: process group \d+/, 'it prints the process group'); +like($hang->{out}, qr/PID\s+PPID\s+STAT\s+TICKS\s+DELTA\s+WCHAN\s+CMD/, 'it prints the process tree'); +like($hang->{out}, qr/sockets: \d+/, 'it prints the open socket count'); +like($hang->{out}, qr/CPU ticks consumed by the whole group during the sample: 0/, + 'it samples CPU ticks and finds none'); +like($hang->{out}, qr/DEADLOCKED, not slow/, 'it says the build was deadlocked rather than slow'); + +# --- a busy build is reported differently, so the sample means something ---------------------- +my $busy = drive(name => 'busy', cmd => 'exec bash -c "while :; do :; done" >/dev/null 2>&1', + timeout => 4, sample => 2, deadline => 90); +ok($busy->{finished}, 'a spinning command is also bounded'); +is($busy->{timed_out}, 1, 'the spin is reported as a timeout'); +like($busy->{out}, qr/still consumes CPU/, 'a spinning build is NOT called deadlocked'); +unlike($busy->{out}, qr/DEADLOCKED/, 'the CPU sample distinguishes a slow build from a deadlock'); + +# --- the whole process group dies, not just the shell ---------------------------------------- +my $pidfile = "$tmp/grandchild.pid"; +my $tree = drive(name => 'tree', + cmd => "sleep 600 >/dev/null 2>&1 & echo \$! > $pidfile; wait", + timeout => 4, sample => 1, deadline => 90); +ok($tree->{finished}, 'a command with a child of its own is bounded too'); +my $gpid = 0; +if (open(my $fh, '<', $pidfile)) { chomp($gpid = <$fh> // 0); close($fh); } +ok($gpid > 0, 'the grandchild recorded its pid'); +push @strays, $gpid if $gpid > 0; +my $alive = 1; +for (1 .. 50) { $alive = (-d "/proc/$gpid") ? 1 : 0; last unless $alive; select(undef, undef, undef, 0.2); } +is($alive, 0, 'the grandchild is killed with the group, so nothing survives holding the pipe'); + +# A signal to the orchestrator must reach the build. Without forwarding, the wrapper dies and +# releases its locks while the build keeps running and writing into staging, so the next run races +# an orphan it cannot see. The child records its own pid, the wrapper is terminated, and the pid is +# probed afterwards. +{ + my $dir = tempdir( CLEANUP => 1 ); + my $pidfile = "$dir/child.pid"; + + my $wrapper = fork(); + die "fork failed: $!" unless defined $wrapper; + if ( $wrapper == 0 ) { + # long budget: the bound must NOT be what ends this run -- the signal must be + run_bounded( + cmd => "echo \$\$ > '$pidfile'; exec sleep 300", + timeout => 600, + label => 'cancellation probe', + out => \*STDERR, + ); + POSIX::_exit(0); + } + + my $child; + for ( 1 .. 100 ) { + if ( -s $pidfile ) { + open my $fh, '<', $pidfile or last; + chomp( $child = <$fh> // '' ); + close $fh; + last if $child; + } + select( undef, undef, undef, 0.1 ); + } + + SKIP: { + skip 'child never reported its pid', 2 unless $child; + ok( kill( 0, $child ), 'the build is running before the wrapper is signalled' ); + + kill 'TERM', $wrapper; + waitpid( $wrapper, 0 ); + + my $alive = 1; + for ( 1 .. 100 ) { + $alive = kill( 0, $child ); + last unless $alive; + select( undef, undef, undef, 0.1 ); + } + ok( !$alive, 'terminating the wrapper reaps the build it started' ); + kill 'KILL', $child if $alive; + } +} + +# The same must hold with no deadline. An unbounded run used to call system(), which leaves the +# build in the orchestrator's own process group: a directed signal then killed the orchestrator and +# left the build writing into staging. +{ + my $dir = tempdir( CLEANUP => 1 ); + my $pidfile = "$dir/child.pid"; + + my $wrapper = fork(); + die "fork failed: $!" unless defined $wrapper; + if ( $wrapper == 0 ) { + run_bounded( + cmd => "echo \$\$ > '$pidfile'; exec sleep 300", + timeout => 0, + label => 'unbounded cancellation probe', + out => \*STDERR, + ); + POSIX::_exit(0); + } + + my $child; + for ( 1 .. 100 ) { + if ( -s $pidfile ) { + open my $fh, '<', $pidfile or last; + chomp( $child = <$fh> // '' ); + close $fh; + last if $child; + } + select( undef, undef, undef, 0.1 ); + } + + SKIP: { + skip 'child never reported its pid', 2 unless $child; + ok( kill( 0, $child ), 'the unbounded build is running before the wrapper is signalled' ); + + kill 'TERM', $wrapper; + waitpid( $wrapper, 0 ); + + my $alive = 1; + for ( 1 .. 100 ) { + $alive = kill( 0, $child ); + last unless $alive; + select( undef, undef, undef, 0.1 ); + } + ok( !$alive, 'terminating the wrapper reaps an unbounded build too' ); + kill 'KILL', $child if $alive; + } +} + +# forward_signals_to_workers: the orchestrator forks its own workers, so run_bounded's forwarding +# never sees a signal sent to it. Two real processes stand in for a worker and the build it runs. +{ + my $dir = tempdir( CLEANUP => 1 ); + my $pidfile = "$dir/worker.pid"; + + my $wrapper = fork(); + die "fork failed: $!" unless defined $wrapper; + if ( $wrapper == 0 ) { + my %kids; + my $worker = fork(); + if ( defined $worker && $worker == 0 ) { + open my $fh, '>', $pidfile or POSIX::_exit(1); + print {$fh} "$$\n"; + close $fh; + sleep 300; + POSIX::_exit(0); + } + $kids{$worker} = 1; + my $forward = XCAT::BuildUtils::forward_signals_to_workers( + pids => \%kids, + reap => sub { waitpid( $_, 0 ) for keys %kids }, + ); + local $SIG{TERM} = $forward; + sleep 300; + POSIX::_exit(0); + } + + my $worker; + for ( 1 .. 100 ) { + if ( -s $pidfile ) { + open my $fh, '<', $pidfile or last; + chomp( $worker = <$fh> // '' ); + close $fh; + last if $worker; + } + select( undef, undef, undef, 0.1 ); + } + + SKIP: { + skip 'worker never reported its pid', 2 unless $worker; + ok( kill( 0, $worker ), 'the worker is running before the orchestrator is signalled' ); + + kill 'TERM', $wrapper; + waitpid( $wrapper, 0 ); + + my $alive = 1; + for ( 1 .. 100 ) { + $alive = kill( 0, $worker ); + last unless $alive; + select( undef, undef, undef, 0.1 ); + } + ok( !$alive, 'signalling the orchestrator reaps its forked workers' ); + kill 'KILL', $worker if $alive; + } +} + +# The build must not inherit a blocked signal mask: run_bounded blocks INT, TERM and HUP across +# the fork so a cancellation cannot land before its handler exists, and a child that kept that mask +# would ignore the very signal the forwarding relies on. +{ + my $dir = tempdir( CLEANUP => 1 ); + my $out = "$dir/mask"; + my $r = run_bounded( + cmd => "grep ^SigBlk /proc/self/status > '$out'", + timeout => 30, + label => 'signal mask probe', + out => \*STDERR, + ); + is( $r->{ec}, 0, 'the probe ran' ); + SKIP: { + skip 'no /proc/self/status on this host', 1 unless -s $out; + open my $fh, '<', $out or die $!; + my $line = <$fh>; + close $fh; + my ($mask) = $line =~ /SigBlk:\s*([0-9a-f]+)/; + my $blocked = hex( $mask // 'ffffffffffffffff' ); + # bit n-1 is signal n: INT 2, TERM 15, HUP 1 + my $handled = ( 1 << 1 ) | ( 1 << 14 ) | ( 1 << 0 ); + is( $blocked & $handled, 0, 'the build starts with INT, TERM and HUP unblocked' ); + } +} + +# The caller's own mask must be restored: run_bounded blocks INT, TERM and HUP around the fork, and +# leaving them blocked would make the orchestrator ignore a cancellation for the rest of the run. +{ + my $dir = tempdir( CLEANUP => 1 ); + my $out = "$dir/caller-mask"; + run_bounded( cmd => 'true', timeout => 30, label => 'mask restore probe', out => \*STDERR ); + system("grep ^SigBlk /proc/self/status > '$out' 2>/dev/null"); + SKIP: { + skip 'no /proc/self/status on this host', 1 unless -s $out; + open my $fh, '<', $out or die $!; + my $line = <$fh>; + close $fh; + my ($mask) = $line =~ /SigBlk:\s*([0-9a-f]+)/; + my $blocked = hex( $mask // 'ffffffffffffffff' ); + my $handled = ( 1 << 1 ) | ( 1 << 14 ) | ( 1 << 0 ); + is( $blocked & $handled, 0, 'the caller keeps INT, TERM and HUP unblocked afterwards' ); + } +} + +# block_handled_signals holds a cancellation until the caller can act on it. This is the window +# between forking a worker and being able to signal it: delivered there, the signal kills the parent +# under a handler that does not know the child, and the child keeps building. +{ + my @caught; + local $SIG{TERM} = sub { push @caught, 'TERM' }; + + my $previous = block_handled_signals(); + kill 'TERM', $$; + select( undef, undef, undef, 0.2 ); + is_deeply( \@caught, [], 'a signal sent while blocked is not delivered' ); + + restore_signal_mask($previous); + select( undef, undef, undef, 0.2 ); + is_deeply( \@caught, ['TERM'], '... and arrives once the mask is restored' ); +} + +# A child the kernel kills leaves 0 in the high byte of its wait status. Reading only that byte +# reports a cancelled or OOM-killed build as one that succeeded, which is what the orchestrators do +# with the status wait() gives them. +{ + is( exit_status(0), 0, 'a child that exited 0 reports 0' ); + is( exit_status(256), 1, 'a child that exited 1 reports 1' ); + is( exit_status(65280), 255, 'a child that exited 255 reports 255' ); + is( exit_status(15), 143, 'a child killed by TERM reports 128 plus the signal' ); + is( exit_status(9), 137, 'a child killed by KILL reports 128 plus the signal' ); + is( exit_status(139), 139, 'a segfault with a core dump still reports the signal' ); + + # The same status through the bounded path, so the helper and its caller agree. + my $r = run_bounded( cmd => 'kill -TERM $$', timeout => 30, label => 'signal exit', + out => \*STDERR ); + is( $r->{ec}, 143, 'run_bounded reports a signalled command the same way' ); + is( $r->{timed_out}, 0, '... and does not call it a timeout' ); +} + +# The leader of the build's process group can die without its descendants: a SIGKILL or the OOM +# killer takes the shell and leaves schroot and qemu behind, still holding the chroot. They are not +# children of this process, so nothing waits for them and nothing reported them. +{ + my $dir = tempdir( CLEANUP => 1 ); + my $pidfile = "$dir/descendant.pid"; + my $r = run_bounded( + cmd => "sleep 300 & echo \$! > '$pidfile'; kill -KILL \$\$", + timeout => 60, + label => 'leader killed while a descendant runs', + out => \*STDERR, + ); + is( $r->{ec}, 137, 'the leader is reported as killed by SIGKILL' ); + + my $descendant; + for ( 1 .. 30 ) { + if ( -s $pidfile ) { + open my $fh, '<', $pidfile or last; + chomp( $descendant = <$fh> // '' ); + close $fh; + last if $descendant; + } + select( undef, undef, undef, 0.1 ); + } + SKIP: { + skip 'the descendant never reported its pid', 1 unless $descendant; + my $alive = 1; + for ( 1 .. 50 ) { + $alive = kill( 0, $descendant ); + last unless $alive; + select( undef, undef, undef, 0.1 ); + } + ok( !$alive, 'the descendant does not outlive the call' ); + kill 'KILL', $descendant if $alive; + } +} + +done_testing; diff --git a/t/common-repo-gate.t b/t/common-repo-gate.t index 9658e99..eb1fcac 100644 --- a/t/common-repo-gate.t +++ b/t/common-repo-gate.t @@ -10,34 +10,85 @@ use strict; use warnings; use Test::More; use FindBin qw($RealBin); +use File::Basename qw(basename); use File::Temp qw(tempdir); use File::Path qw(make_path); use File::Copy qw(copy); use lib "$RealBin/.."; +use lib "$RealBin/../lib"; +use lib "$RealBin/lib"; use MockBuildUtils qw(read_manifest); +use XCAT::BuildUtils qw(command_exists); +use XCAT::GenesisRelease qw(architectures rpm_package_name); +use XCAT::GenesisReleaseTest qw( + build_package_release + copy_tree + write_checksums + write_release_manifest +); my $SCRIPT = "$RealBin/../mockbuild-all.pl"; -my $RELEASE = '/opt/xcat-ci-shared/builds/genesis-openembedded-initial-20260825/release'; +my $PACKAGER = "$RealBin/../genesis-openembedded/package"; plan skip_all => 'mockbuild-all.pl not found' unless -f $SCRIPT; -plan skip_all => 'no Genesis release fixture' unless -d "$RELEASE/rpm"; -# root, like every other test that drives mockbuild-all.pl: the script refuses to run otherwise, -# and the CI builder (XCAT_GENESIS_CI) is root. -plan skip_all => 'rpm tooling and a root Linux builder required' - unless $^O eq 'linux' - && $> == 0 - && !system('sh', '-c', 'command -v rpm >/dev/null 2>&1') - && !system('sh', '-c', 'command -v createrepo_c >/dev/null 2>&1') - && !system('sh', '-c', 'command -v rpmbuild >/dev/null 2>&1'); +my @missing_requirements; +push @missing_requirements, 'Linux' unless $^O eq 'linux'; +push @missing_requirements, 'root' unless $> == 0; +for my $command (qw(createrepo_c gzip rpm rpmbuild tar)) { + push @missing_requirements, $command unless command_exists($command); +} +if (command_exists('tar')) { + my $tar_version = `tar --version 2>/dev/null`; + push @missing_requirements, 'GNU tar' unless $tar_version =~ /GNU tar/; +} +if (@missing_requirements) { + my $message = 'requires ' . join(', ', @missing_requirements); + BAIL_OUT($message) if $ENV{XCAT_GENESIS_CI}; + plan skip_all => $message; +} my $tmp = tempdir(CLEANUP => 1); my $target = 'alma+epel-10-' . do { my $m = `uname -m`; chomp $m; $m }; +my @architectures = architectures(); +my $xcat_version = '2.19.0'; +my $xcat_release = 'snap202609040000'; +my $xcat_revision = 'c' x 40; +my $source_date_epoch = 1788476400; +my $RELEASE = build_package_release( + root => "$tmp/release-fixture", + format => 'rpm', + architectures => \@architectures, + packager => $PACKAGER, + version => $xcat_version, + release => $xcat_release, + revision => $xcat_revision, + epoch => $source_date_epoch, +); +my @version_1_architectures = grep { $_ ne 's390x' } @architectures; +my $VERSION_1_RELEASE = build_package_release( + root => "$tmp/version-1-release-fixture", + format => 'rpm', + architectures => \@version_1_architectures, + packager => $PACKAGER, + version => $xcat_version, + release => $xcat_release, + revision => $xcat_revision, + epoch => $source_date_epoch, +); +write_release_manifest( + $VERSION_1_RELEASE, $xcat_version, $xcat_release, $xcat_revision, + $source_date_epoch, join(',', @version_1_architectures), 'rpm', 1, +); +write_checksums($VERSION_1_RELEASE); # The shipped manifest must describe the shared repo, else nothing can gate it. { my %m = read_manifest("$RealBin/../packages-manifest.conf"); ok($m{common} && %{ $m{common} }, 'the shipped manifest has a [common] section'); - is(scalar(keys %{ $m{common} // {} }), 7, - '... naming every architecture the release must carry'); + my @expected = sort map { rpm_package_name($_) } + qw(x86 x86_64 ppc64 ppc64le armv7hf aarch64 riscv64 s390x); + my @missing = grep { !exists $m{common}{$_} } @expected; + is_deeply(\@missing, [], + 'the shared RPM manifest lists every Genesis architecture'); } # fixture_rpm: a minimal noarch rpm, built once, standing in for a compiled dep. @@ -69,9 +120,8 @@ SPEC return $FIXTURE; } -# run_publish($release_dir) -> ($exit, $output, $common_dir) sub run_publish { - my ($release, $tag) = @_; + my ($release, $tag, $mutate_common, $skip_repo_verification) = @_; my $out = "$tmp/$tag"; make_path("$out/root", "$out/collect"); # Something to collect, so the run gets past the "built nothing" guard. It must NOT be an @@ -83,49 +133,152 @@ sub run_publish { print $fh "[$target]\nipmitool-xcat=1.8.18\n"; # the shared repo's own section, copied from the shipped manifest so the test uses the real one my %m = read_manifest("$RealBin/../packages-manifest.conf"); + $mutate_common->($m{common}) if $mutate_common; print $fh "\n[common]\n"; print $fh "$_=$m{common}{$_}\n" for sort keys %{ $m{common} // {} }; close $fh; - my $cmd = join(' ', map { my $x = $_; $x =~ s/'/'"'"'/g; "'$x'" } + my @command = ($^X, $SCRIPT, '--repo-root', "$out/root", '--output', "$out/build", '--repo-dep', "$out/repo", '--target', $target, '--run-id', $tag, '--build-timestamp', '1787672536', '--skip-build', '--skip-genesis', '--skip-xcat-dep', '--skip-perl', '--skip-tarball', - '--collect-dir', "$out/collect", '--genesis-release', $release)) . ' 2>&1'; + '--collect-dir', "$out/collect", '--genesis-release', $release); + push @command, '--no-verify-repo' if $skip_repo_verification; + my $cmd = join(' ', map { my $x = $_; $x =~ s/'/'"'"'/g; "'$x'" } @command) + . ' 2>&1'; my $log = `$cmd`; return ($? >> 8, $log, "$out/repo/common"); } +{ + my $package = rpm_package_name('s390x'); + my ($rc, $out, $common) = run_publish( + $RELEASE, + 'unsatisfied-common', + sub { $_[0]->{$package} = '>= 99.0.0' }, + ); + isnt($rc, 0, 'an unsatisfied shared-repository requirement is refused'); + like($out, qr/EVR \Q$package\E: repo has .* manifest requires >= 99\.0\.0/, + 'the shared-repository gate names the unsatisfied requirement'); + ok(!-d $common || !glob("$common/*.rpm"), + 'a failed shared-repository gate publishes nothing'); +} + # ---- a complete release publishes, and says it was gated ----------------------------------------- { my ($rc, $out, $common) = run_publish($RELEASE, 'full'); is($rc, 0, 'a complete release publishes') or diag($out); - is(scalar(grep { !/\.src\.rpm$/ } glob("$common/*.rpm")), 7, + is(scalar(grep { !/\.src\.rpm$/ } glob("$common/*.rpm")), 8, 'the published shared repo carries every architecture'); - like($out, qr/\[verify-repo\] common complete/, 'the shared repo is gated against [common]'); + like($out, qr/\[verify-repo\] common complete: 8 packages present/, + 'the shared repo is gated against [common]'); +} + +{ + my ($rc, $out, $common) = run_publish($VERSION_1_RELEASE, 'version-1'); + isnt($rc, 0, 'a version 1 release cannot replace the current repository'); + like($out, qr/Genesis release version 1 omits currently supported architectures: s390x/, + 'version 1 refusal identifies the missing architecture'); + ok(!-d $common || !glob("$common/*.rpm"), + 'a version 1 release publishes nothing'); +} + +{ + my ($rc, $out, $common) = run_publish( + $RELEASE, + 'missing-non-genesis-package', + sub { $_[0]->{'xCAT-release'} = '>= 2.0.0' }, + ); + isnt($rc, 0, 'every common manifest package is verified'); + like($out, qr/MISSING xCAT-release/, + 'the common gate identifies a missing non-Genesis package'); + ok(!-d $common || !glob("$common/*.rpm"), + 'a missing non-Genesis package prevents publication'); +} + +{ + my ($rc, $out, $common) = run_publish( + $RELEASE, + 'missing-current-package', + sub { delete $_[0]->{ rpm_package_name('s390x') } }, + ); + isnt($rc, 0, 'a current release does not hide an incomplete manifest'); + like($out, qr/\[common\] is missing supported packages: .*s390x/, + 'the manifest failure identifies the missing current package'); + ok(!-d $common || !glob("$common/*.rpm"), + 'an incomplete current manifest publishes nothing'); +} + +{ + my ($rc, $out, $common) = run_publish( + $RELEASE, + 'missing-current-package-without-repo-verification', + sub { delete $_[0]->{ rpm_package_name('s390x') } }, + 1, + ); + isnt($rc, 0, 'repository verification cannot disable the common manifest gate'); + like($out, qr/\[common\] is missing supported packages: .*s390x/, + 'the mandatory manifest gate identifies the missing package'); + ok(!-d $common || !glob("$common/*.rpm"), + 'the mandatory manifest gate publishes nothing'); +} + +{ + my ($rc, $out, $common) = run_publish( + $RELEASE, + 'unknown-current-package', + sub { $_[0]->{'xCAT-genesis-openembedded-unknown'} = '>= 2.18.0' }, + ); + isnt($rc, 0, 'an unknown shared manifest package is refused'); + like($out, qr/\[common\] has unsupported packages: xCAT-genesis-openembedded-unknown/, + 'the manifest failure identifies the unknown package'); + ok(!-d $common || !glob("$common/*.rpm"), + 'an unknown shared manifest package publishes nothing'); } # ---- an incomplete release is refused, and publishes nothing -------------------------------------- { my $partial = "$tmp/partial-release"; + my $missing_architecture = 's390x'; + my $missing_package = rpm_package_name($missing_architecture); + my @partial_architectures = grep { $_ ne $missing_architecture } @architectures; make_path("$partial/rpm", "$partial/srpm"); for my $f (glob("$RELEASE/rpm/*.rpm"), glob("$RELEASE/srpm/*.rpm")) { - next if $f =~ /riscv64/; # drop one architecture + next if basename($f) =~ /^\Q$missing_package-\E/; my ($sub) = $f =~ m{/(rpm|srpm)/[^/]+$}; copy($f, "$partial/$sub/") or die $!; } - copy("$RELEASE/release.manifest", $partial) or die $!; - # SHA256SUMS without the dropped arch, so the release itself still self-describes consistently - open my $in, '<', "$RELEASE/SHA256SUMS" or die $!; - open my $o, '>', "$partial/SHA256SUMS" or die $!; - while (<$in>) { print {$o} $_ unless /riscv64/ } - close $in; close $o; + write_release_manifest( + $partial, $xcat_version, $xcat_release, $xcat_revision, $source_date_epoch, + join(',', @partial_architectures), 'rpm', + ); + write_checksums($partial); my ($rc, $out, $common) = run_publish($partial, 'partial'); isnt($rc, 0, 'a release missing an architecture is refused'); + like($out, qr/omits currently supported architectures: \Q$missing_architecture\E/, + 'the completeness gate identifies the missing architecture'); ok(!-d $common || !glob("$common/*.rpm"), '... and nothing is published into the shared repository'); } +{ + my $inconsistent = "$tmp/inconsistent-release"; + my $missing_package = rpm_package_name('s390x'); + copy_tree($RELEASE, $inconsistent); + for my $directory (qw(rpm srpm)) { + my @packages = glob("$inconsistent/$directory/$missing_package-*"); + unlink(@packages) == @packages or die "cannot remove package fixture\n"; + } + write_checksums($inconsistent); + + my ($rc, $out, $common) = run_publish($inconsistent, 'inconsistent'); + isnt($rc, 0, 'a release inconsistent with its manifest is refused'); + like($out, qr/Genesis release is missing:/, + 'the release-layout gate reports the missing package'); + ok(!-d $common || !glob("$common/*.rpm"), + 'an inconsistent release publishes nothing'); +} + done_testing(); diff --git a/t/genesis_openembedded_consumer.t b/t/genesis_openembedded_consumer.t index 9574623..11cad2d 100644 --- a/t/genesis_openembedded_consumer.t +++ b/t/genesis_openembedded_consumer.t @@ -12,8 +12,10 @@ use POSIX (); use Test::More; use Time::HiRes qw(sleep); +use lib "$FindBin::Bin/.."; use lib "$FindBin::Bin/../lib"; use lib "$FindBin::Bin/lib"; +use BuildUtils qw(read_manifest); use XCAT::BuildUtils qw( capture_command command_exists @@ -27,10 +29,10 @@ use XCAT::GenesisRelease qw( rpm_package_name ); use XCAT::GenesisReleaseTest qw( - make_export + build_package_release run_capture - write_forkmanager_stub write_checksums + write_forkmanager_stub write_release_manifest ); @@ -75,11 +77,12 @@ SKIP: { } SKIP: { - skip 'APT repository tools are not installed', 59 + skip 'APT repository tools are not installed', 63 unless $^O eq 'linux' && command_exists('dpkg-deb') && command_exists('apt-ftparchive'); test_deb_consumer(); + test_version_1_deb_consumer(); test_legacy_deb_consumer(); test_partial_deb_release(); test_publish_lock(); @@ -100,9 +103,13 @@ done_testing(); sub write_target_manifest { my ($root, $target) = @_; make_path($root); + my %manifest = read_manifest("$repo_root/packages-manifest.conf"); + my $common = join('', map { "$_=$manifest{common}{$_}\n" } + sort keys %{ $manifest{common} // {} }); write_binary( "$root/packages-manifest.conf", - "[$target]\n" . rpm_package_name(capture_command('uname', '-m')) . "=*\n", + "[$target]\n" . rpm_package_name(capture_command('uname', '-m')) + . "=*\n\n[common]\n$common", ); } @@ -297,17 +304,11 @@ SH sub run_apt_consumer { my (%args) = @_; my @dists = @{ $args{dists} // \@APT_SUITES }; + my @build_mode = $args{build} ? ('--dry-run') : ('--skip-build'); my $manifest = $args{manifest}; unless ($manifest) { $manifest = "$args{output}/manifest.conf"; - make_path($args{output}); - # plus the shipped [shared] section verbatim: publishing a release gates the shared pool - # against it, and a manifest without it is refused rather than silently ungated. - my $shipped = read_binary("$repo_root/debs-manifest.conf"); - my ($shared) = $shipped =~ /^(\[shared\]\n(?:[^\[]*))/ms; - BAIL_OUT('debs-manifest.conf has no [shared] section') unless $shared; - write_binary($manifest, - join('', map { "[$_-amd64]\nxcat-genesis-base=*\n" } @APT_SUITES) . "\n" . $shared); + write_apt_manifest($manifest); } return run_capture( $args{log}, @@ -318,13 +319,29 @@ sub run_apt_consumer { '--manifest', $manifest, '--dists', join(' ', @dists), '--arch', 'amd64', - '--skip-build', '--skip-genesis', '--skip-tarball', + @build_mode, '--skip-genesis', '--skip-tarball', '--publish', '--expect-arch', 'amd64 ppc64el', ($args{verify} ? () : ('--no-verify-repo')), @{ $args{extra} // [] }, ); } +sub write_apt_manifest { + my ($path, $mutate) = @_; + my %shipped = read_manifest("$repo_root/debs-manifest.conf"); + BAIL_OUT('debs-manifest.conf has no [shared] section') + unless exists $shipped{shared}; + my %shared = %{ $shipped{shared} }; + $mutate->(\%shared) if $mutate; + make_path((File::Basename::dirname($path))); + write_binary( + $path, + join('', map { "[$_-amd64]\nxcat-genesis-base=*\n" } @APT_SUITES) + . "\n[shared]\n" + . join('', map { "$_=$shared{$_}\n" } sort keys %shared), + ); +} + # Stage one legacy genesis-base deb for the noble suite only -- enough for the runs that publish # no Genesis release. sub stage_legacy_deb { @@ -397,7 +414,8 @@ sub test_deb_consumer { } architectures(); is_deeply([ genesis_deb_names($shared_pool) ], \@expected_packages, 'shared APT pool contains one complete Genesis release'); - like(read_binary($log), qr/\[verify-repo\] shared pool complete: 7 packages present/, + like(read_binary($log), + qr/\[verify-repo\] shared pool complete: 8 packages present/, 'the shared pool is gated against the manifest\'s [shared] section'); my @suite_packages; for my $codename (@APT_SUITES) { @@ -524,6 +542,103 @@ sub test_deb_consumer { 'pooled package still matches the verified release'); } +sub test_version_1_deb_consumer { + my @release_architectures = grep { $_ ne 's390x' } architectures(); + my $release_root = make_package_release( + "$tmp/deb-version-1", 'deb', @release_architectures, + ); + write_release_manifest( + $release_root, $version, $release, $revision, $epoch, + join(',', @release_architectures), 'deb', 1, + ); + write_checksums($release_root); + + my $apt_root = "$tmp/apt-version-1"; + my $output = "$tmp/deb-version-1-output"; + stage_apt_suites($output, "$tmp/deb-version-1-legacy"); + my $log = "$tmp/deb-version-1.log"; + my $status = run_apt_consumer( + log => $log, output => $output, apt_dir => $apt_root, + extra => [ '--genesis-release', $release_root ], + ); + my $pool = "$apt_root/pool/main/xcat-genesis-openembedded"; + + isnt($status, 0, 'APT refuses a version 1 release for the current repository'); + like(read_binary($log), + qr/Genesis release version 1 omits currently supported architectures: s390x/, + 'the version 1 refusal identifies the missing architecture'); + ok(!-d $pool, 'a version 1 release publishes no shared pool'); + + my $current_release = make_package_release( + "$tmp/deb-current-manifest", 'deb', architectures(), + ); + + my $missing_manifest = "$tmp/deb-version-1-missing.conf"; + write_apt_manifest( + $missing_manifest, + sub { delete $_[0]->{ deb_package_name('s390x') } }, + ); + my $missing_apt = "$tmp/apt-version-1-missing"; + my $missing_output = "$tmp/deb-version-1-missing-output"; + stage_apt_suites($missing_output, "$tmp/deb-version-1-missing-legacy"); + my $missing_log = "$tmp/deb-version-1-missing.log"; + my $missing_status = run_apt_consumer( + log => $missing_log, output => $missing_output, apt_dir => $missing_apt, + manifest => $missing_manifest, + build => 1, + extra => [ '--genesis-release', $current_release ], + ); + isnt($missing_status, 0, + 'a current release does not hide an incomplete shared manifest'); + like(read_binary($missing_log), qr/\[shared\] is missing supported packages: .*s390x/, + 'the shared manifest failure identifies the missing current package'); + unlike(read_binary($missing_log), qr/Ensure sbuild chroots/, + 'an incomplete shared manifest is rejected before building'); + + my $unknown_manifest = "$tmp/deb-version-1-unknown.conf"; + write_apt_manifest( + $unknown_manifest, + sub { $_[0]->{'xcat-genesis-openembedded-unknown'} = '2.*' }, + ); + my $unknown_apt = "$tmp/apt-version-1-unknown"; + my $unknown_output = "$tmp/deb-version-1-unknown-output"; + stage_apt_suites($unknown_output, "$tmp/deb-version-1-unknown-legacy"); + my $unknown_log = "$tmp/deb-version-1-unknown.log"; + my $unknown_status = run_apt_consumer( + log => $unknown_log, output => $unknown_output, apt_dir => $unknown_apt, + manifest => $unknown_manifest, + extra => [ '--genesis-release', $current_release ], + ); + isnt($unknown_status, 0, 'an unknown shared manifest package is refused'); + like(read_binary($unknown_log), + qr/\[shared\] has unsupported packages: xcat-genesis-openembedded-unknown/, + 'the shared manifest failure identifies the unknown package'); + ok(!-d "$unknown_apt/pool/main/xcat-genesis-openembedded", + 'an unknown shared manifest package publishes nothing'); + + my $non_genesis_manifest = "$tmp/deb-non-genesis-missing.conf"; + write_apt_manifest( + $non_genesis_manifest, + sub { $_[0]->{'xcat-release'} = '2.*' }, + ); + my $non_genesis_apt = "$tmp/apt-non-genesis-missing"; + my $non_genesis_output = "$tmp/deb-non-genesis-missing-output"; + stage_apt_suites($non_genesis_output, "$tmp/deb-non-genesis-missing-legacy"); + my $non_genesis_log = "$tmp/deb-non-genesis-missing.log"; + my $non_genesis_status = run_apt_consumer( + log => $non_genesis_log, + output => $non_genesis_output, + apt_dir => $non_genesis_apt, + manifest => $non_genesis_manifest, + extra => [ '--genesis-release', $current_release ], + ); + isnt($non_genesis_status, 0, 'every shared manifest package is verified'); + like(read_binary($non_genesis_log), qr/MISSING xcat-release/, + 'the shared gate identifies a missing non-Genesis package'); + ok(!-d "$non_genesis_apt/pool/main/xcat-genesis-openembedded", + 'a missing non-Genesis package prevents APT publication'); +} + sub test_signed_common_rpm_repository { my $release_root = make_package_release("$tmp/rpm-signed", 'rpm'); my $package = "xCAT-genesis-openembedded-x86_64-$version-$release.noarch.rpm"; @@ -679,7 +794,7 @@ sub test_partial_rpm_release { ); isnt($status, 0, 'RPM repository rejects a partial Genesis release'); - like(read_binary($log), qr/Genesis release is missing supported architectures/, + like(read_binary($log), qr/Genesis release version 2 omits currently supported architectures/, 'RPM partial-release failure names the missing architectures'); ok(-f $existing, 'partial release does not remove the deployed package'); } @@ -701,7 +816,7 @@ sub test_partial_deb_release { ); isnt($status, 0, 'APT repository rejects a partial Genesis release'); - like(read_binary($log), qr/Genesis release is missing supported architectures/, + like(read_binary($log), qr/Genesis release version 2 omits currently supported architectures/, 'APT partial-release failure names the missing architectures'); ok(-f $existing, 'partial DEB release does not remove the published package'); } @@ -995,6 +1110,18 @@ SH is($status, 0, 'the activation helper accepts a local-TFTP node'); is(read_binary($log), "x86_64\n", 'the activation helper runs mknb for one architecture'); + write_binary($log, ''); + $status = run_capture($output, $driver, 's390x'); + is($status, 0, 'the activation helper accepts s390x'); + is(read_binary($log), "s390x\n", 'the activation helper runs mknb for s390x'); + + write_binary($log, ''); + $status = run_capture($output, $driver, 'unsupported'); + is($status, 0, 'an unsupported architecture does not fail the package transaction'); + is(read_binary($log), '', 'an unsupported architecture does not run mknb'); + like(read_binary($output), qr/Invalid Genesis architecture: unsupported/, + 'the activation helper reports an unsupported architecture'); + write_binary($log, ''); local $ENV{XCAT_TEST_SERVICE_NODE} = 1; local $ENV{XCAT_TEST_SHAREDTFTP} = 1; @@ -1011,50 +1138,16 @@ SH sub make_package_release { my ($root, $format, @requested_architectures) = @_; @requested_architectures = architectures() unless @requested_architectures; - my $release_root = "$root/release"; - make_path($release_root); - for my $architecture (@requested_architectures) { - my $export = make_export("$root/exports/$architecture", $architecture); - my $packages = "$root/packages/$architecture"; - die "Cannot package test release for $architecture\n" - if run_capture( - "$root/package-$architecture.log", - $packager, - '--architecture', $architecture, - '--export-dir', $export, - '--output-dir', $packages, - '--version', $version, - '--release', $release, - '--revision', $revision, - '--source-date-epoch', $epoch, - '--format', $format, - ); - if ($format eq 'rpm') { - my $name = rpm_package_name($architecture); - make_path("$release_root/rpm", "$release_root/srpm"); - copy( - "$packages/rpm/$name-$version-$release.noarch.rpm", - "$release_root/rpm/$name-$version-$release.noarch.rpm", - ) or die $!; - copy( - "$packages/srpm/$name-$version-$release.src.rpm", - "$release_root/srpm/$name-$version-$release.src.rpm", - ) or die $!; - } else { - my $name = deb_package_name($architecture); - make_path("$release_root/deb"); - copy( - "$packages/deb/${name}_${version}-${release}_all.deb", - "$release_root/deb/${name}_${version}-${release}_all.deb", - ) or die $!; - } - } - write_release_manifest( - $release_root, $version, $release, $revision, $epoch, - join(',', @requested_architectures), $format, + return build_package_release( + root => $root, + format => $format, + architectures => \@requested_architectures, + packager => $packager, + version => $version, + release => $release, + revision => $revision, + epoch => $epoch, ); - write_checksums($release_root); - return $release_root; } sub genesis_rpm_names { diff --git a/t/genesis_openembedded_release.t b/t/genesis_openembedded_release.t index 365be5b..5a35d15 100644 --- a/t/genesis_openembedded_release.t +++ b/t/genesis_openembedded_release.t @@ -11,6 +11,7 @@ use Test::More; use lib "$FindBin::Bin/../lib"; use lib "$FindBin::Bin/lib"; use XCAT::BuildUtils qw( + capture_command command_exists digest_file read_binary @@ -19,6 +20,7 @@ use XCAT::BuildUtils qw( use XCAT::GenesisRelease qw( architectures deb_package_name + minimum_release_version rpm_package_name validated_release_checksums validate_architecture @@ -53,13 +55,19 @@ if ($ENV{XCAT_GENESIS_CI}) { is_deeply( [ architectures() ], - [ qw(x86 x86_64 ppc64 ppc64le armv7hf aarch64 riscv64) ], + [ qw(x86 x86_64 ppc64 ppc64le armv7hf aarch64 riscv64 s390x) ], 'supported architectures keep their exact xCAT names', ); is(rpm_package_name('ppc64le'), 'xCAT-genesis-openembedded-ppc64le', 'RPM package keeps ppc64le distinct'); is(deb_package_name('x86_64'), 'xcat-genesis-openembedded-x86-64', 'DEB package uses a legal spelling of x86_64'); +is(minimum_release_version('x86_64'), 1, + 'legacy architectures use release format version 1'); +is(minimum_release_version('x86_64', 's390x'), 2, + 's390x requires release format version 2'); +dies_like(sub { minimum_release_version() }, qr/requires a Genesis architecture/, + 'release format selection requires an architecture'); dies_like(sub { validate_architecture('ppc') }, qr/Unsupported Genesis architecture/, 'legacy ppc alias is rejected'); @@ -181,7 +189,7 @@ like( ); dies_like( sub { validate_complete_release($release_dir) }, - qr/Genesis release is missing supported architectures/, + qr/Genesis release version 2 omits currently supported architectures/, 'partial release cannot be published', ); @@ -201,6 +209,49 @@ write_release_manifest( write_checksums($complete_release); ok(validate_complete_release($complete_release), 'complete release can be published'); +my $legacy_release = "$tmp/legacy-release"; +copy_tree($complete_release, $legacy_release); +for my $directory (qw(rpm srpm deb)) { + my @s390x_packages = glob("$legacy_release/$directory/*s390x*"); + unlink(@s390x_packages) == @s390x_packages + or die "Cannot remove the s390x package fixture: $!\n"; +} +write_release_manifest( + $legacy_release, $version, $release, $revision, $epoch, + 'x86,x86_64,ppc64,ppc64le,armv7hf,aarch64,riscv64', 'deb,rpm', 1, +); +write_checksums($legacy_release); +ok(validate_release($legacy_release), 'version 1 releases remain readable'); +dies_like( + sub { validate_complete_release($legacy_release) }, + qr/Genesis release version 1 omits currently supported architectures: s390x/, + 'version 1 releases cannot replace the current repository', +); + +my $invalid_legacy_release = "$tmp/invalid-legacy-release"; +copy_tree($complete_release, $invalid_legacy_release); +write_release_manifest( + $invalid_legacy_release, $version, $release, $revision, $epoch, + join(',', architectures()), 'deb,rpm', 1, +); +write_checksums($invalid_legacy_release); +dies_like( + sub { validate_release($invalid_legacy_release) }, + qr/Genesis architecture s390x is not valid in release version 1/, + 'version 1 rejects the version 2 architecture vocabulary', +); + +my $unknown_release_version = "$tmp/unknown-release-version"; +copy_tree($complete_release, $unknown_release_version); +write_release_manifest( + $unknown_release_version, $version, $release, $revision, $epoch, + join(',', architectures()), 'deb,rpm', 3, +); +write_checksums($unknown_release_version); +dies_like(sub { validate_release($unknown_release_version) }, + qr/Unsupported Genesis package release version/, + 'unknown release manifest versions fail'); + my $deb_only_release = "$tmp/deb-only-release"; make_path("$deb_only_release/deb"); for my $architecture (architectures()) { @@ -233,11 +284,25 @@ dies_like(sub { validate_release($missing_release) }, qr/Genesis release is miss 'incomplete architecture set fails'); SKIP: { - skip 'git is not installed', 2 unless command_exists('git'); + skip 'git is not installed', 13 unless command_exists('git'); my $source = "$tmp/dirty-xcat-core"; - make_path("$source/xCAT-genesis-builder/oe"); + my $oe = "$source/xCAT-genesis-builder/oe"; + my $capability_marker = "$tmp/capability-query-ran"; + make_path($oe); write_binary("$source/Version", "$version\n"); - write_binary("$source/xCAT-genesis-builder/oe/build", "#!/bin/sh\nexit 99\n"); + write_binary( + "$source/xCAT-genesis-builder/oe/build", + "#!/bin/sh\n" + . "if [ \"\${1-}\" = --list-architectures ]; then\n" + . " [ -z \"\${XCAT_TEST_CAPABILITY_MARKER-}\" ] || : >\"\$XCAT_TEST_CAPABILITY_MARKER\"\n" + . " mkdir -p \"\${XCAT_GENESIS_WORK_DIR:?}\"\n" + . " printf '%s\\n' x86_64\n" + . " exit 0\n" + . "fi\n" + . "exit 99\n", + ); + chmod(0755, "$source/xCAT-genesis-builder/oe/build") + or die "Cannot make fixture build executable: $!"; write_binary("$source/xCAT-genesis-builder/oe/export", "#!/bin/sh\nexit 99\n"); for my $command ( [ 'git', '-C', $source, 'init', '-q' ], @@ -248,22 +313,111 @@ SKIP: { die "Cannot prepare test repository\n" if run_capture("$tmp/git-fixture.log", @{$command}); } + my $previous_revision = capture_command('git', '-C', $source, 'rev-parse', 'HEAD'); + my $commit_source = sub { + my ($path, $message) = @_; + for my $command ( + [ 'git', '-C', $source, 'add', $path ], + [ 'git', '-C', $source, '-c', 'user.name=xCAT test', + '-c', 'user.email=xcat-test@example.invalid', 'commit', '-qm', $message ], + ) { + die "Cannot update test repository\n" + if run_capture("$tmp/git-fixture.log", @{$command}); + } + }; write_binary("$source/untracked", "not part of the commit\n"); my $log = "$tmp/dirty-source.log"; - isnt( - run_capture( - $log, $builder, '--xcat-source', $source, - '--output-dir', "$tmp/dirty-output", - ), - 0, - 'release builder rejects untracked source files', - ); + { + local $ENV{XCAT_TEST_CAPABILITY_MARKER} = $capability_marker; + isnt( + run_capture( + $log, $builder, '--xcat-source', $source, + '--architecture', 's390x', + '--output-dir', "$tmp/dirty-output", + ), + 0, + 'release builder rejects untracked source files', + ); + } like(read_binary($log), qr/xcat-core checkout is not clean/, 'dirty checkout failure is explicit'); + ok(!-e $capability_marker, + 'dirty source is rejected before its architecture helper runs'); + unlink("$source/untracked") or die "Cannot clean the source fixture: $!\n"; + + write_binary("$source/revision-marker", "new revision\n"); + $commit_source->('revision-marker', 'advance source revision'); + my $ref_log = "$tmp/ref-mismatch.log"; + { + local $ENV{XCAT_TEST_CAPABILITY_MARKER} = $capability_marker; + isnt( + run_capture( + $ref_log, $builder, '--xcat-source', $source, + '--xcat-ref', $previous_revision, + '--architecture', 's390x', + '--output-dir', "$tmp/ref-mismatch-output", + ), + 0, + 'release builder rejects a mismatched xcat-core revision', + ); + } + like(read_binary($ref_log), qr/xcat-core HEAD .* does not match \Q$previous_revision\E/, + 'revision mismatch failure is explicit'); + ok(!-e $capability_marker, + 'revision mismatch is rejected before the architecture helper runs'); + + my $target_log = "$tmp/missing-target.log"; + { + local $ENV{XCAT_TEST_CAPABILITY_MARKER} = $capability_marker; + isnt( + run_capture( + $target_log, $builder, '--xcat-source', $source, + '--architecture', 's390x', + '--output-dir', "$tmp/missing-target-output", + ), + 0, + 'release builder rejects an unsupported xcat-core target', + ); + } + like(read_binary($target_log), qr/does not support Genesis architecture s390x/, + 'missing target failure identifies the required xcat-core support'); + ok(-e $capability_marker, 'the architecture helper records a successful query'); + + write_binary("$oe/build", "#!/bin/sh\nexit 23\n"); + chmod(0755, "$oe/build") or die "Cannot update fixture build executable: $!"; + $commit_source->('xCAT-genesis-builder/oe/build', 'fail capability query'); + my $failed_log = "$tmp/failed-query.log"; + isnt( + run_capture( + $failed_log, $builder, '--xcat-source', $source, + '--architecture', 's390x', + '--output-dir', "$tmp/failed-query-output", + ), + 0, + 'release builder rejects a failed architecture query', + ); + like(read_binary($failed_log), qr/does not report supported Genesis architectures/, + 'failed architecture queries are reported'); + + write_binary("$oe/build", "#!/bin/sh\nexit 0\n"); + chmod(0755, "$oe/build") or die "Cannot update fixture build executable: $!"; + $commit_source->('xCAT-genesis-builder/oe/build', 'empty capability query'); + my $empty_log = "$tmp/empty-query.log"; + isnt( + run_capture( + $empty_log, $builder, '--xcat-source', $source, + '--architecture', 's390x', + '--output-dir', "$tmp/empty-query-output", + ), + 0, + 'release builder rejects an empty architecture query', + ); + like(read_binary($empty_log), qr/reported no supported Genesis architectures/, + 'empty architecture queries are reported'); } SKIP: { - skip 'rpmbuild and rpm are not installed', 18 + skip 'rpmbuild and rpm are not installed', 19 unless command_exists('rpmbuild') && command_exists('rpm'); exercise_packager('rpm'); } @@ -276,12 +430,12 @@ SKIP: { } SKIP: { - skip 'dpkg-deb is not installed', 12 unless command_exists('dpkg-deb'); + skip 'dpkg-deb is not installed', 14 unless command_exists('dpkg-deb'); exercise_packager('deb'); } SKIP: { - skip 'git and dpkg-deb are not installed', 6 + skip 'git and dpkg-deb are not installed', 9 unless command_exists('git') && command_exists('dpkg-deb'); exercise_builder_tmpdir(); } @@ -465,6 +619,10 @@ sub exercise_builder_tmpdir { <<'BUILD', #!/bin/sh set -eu +if [ "${1-}" = --list-architectures ]; then + printf '%s\n' x86_64 s390x + exit 0 +fi expected=$XCAT_GENESIS_WORK_DIR/build/tmp [ "${TMPDIR:-}" = "$expected" ] || exit 41 mkdir -p "$TMPDIR/deploy" @@ -516,7 +674,7 @@ EXPORT $status = run_capture( $log, $builder, '--xcat-source', $source, '--output-dir', $output, '--work-dir', $persistent_work, - '--format', 'deb', + '--format', 'deb', '--architecture', 's390x', ); } is($status, 0, 'release builder isolates the OpenEmbedded tmpdir'); @@ -527,6 +685,18 @@ EXPORT is((stat($output))[2] & oct('07777'), oct('0755'), 'release directory is readable by other users'); my $built = validate_release($output); - is($built->{architectures}, 'x86_64', 'isolated build keeps the target architecture'); + is($built->{architectures}, 's390x', 'isolated build keeps the target architecture'); is($built->{formats}, 'deb', 'isolated build keeps the requested format'); + is($built->{version}, 2, 'a release containing s390x uses format version 2'); + + my $legacy_output = "$tmp/tmpdir-legacy-release"; + my $legacy_log = "$tmp/tmpdir-legacy-builder.log"; + my $legacy_status = run_capture( + $legacy_log, $builder, '--xcat-source', $source, + '--output-dir', $legacy_output, '--work-dir', $persistent_work, + '--format', 'deb', '--architecture', 'x86_64', + ); + is($legacy_status, 0, 'release builder keeps legacy targets buildable'); + my $legacy = validate_release($legacy_output); + is($legacy->{version}, 1, 'a release without s390x uses format version 1'); } diff --git a/t/lib/XCAT/GenesisReleaseTest.pm b/t/lib/XCAT/GenesisReleaseTest.pm index 7660784..7a475e0 100644 --- a/t/lib/XCAT/GenesisReleaseTest.pm +++ b/t/lib/XCAT/GenesisReleaseTest.pm @@ -13,8 +13,10 @@ use XCAT::BuildUtils qw( relative_files write_binary ); +use XCAT::GenesisRelease qw(deb_package_name rpm_package_name); our @EXPORT_OK = qw( + build_package_release copy_tree dies_like make_export @@ -24,6 +26,58 @@ our @EXPORT_OK = qw( write_release_manifest ); +sub build_package_release { + my (%args) = @_; + my $root = $args{root}; + my $format = $args{format}; + my @architectures = @{ $args{architectures} }; + my $release_root = "$root/release"; + + make_path($release_root); + for my $architecture (@architectures) { + my $export = make_export("$root/exports/$architecture", $architecture); + my $packages = "$root/packages/$architecture"; + die "Cannot package test release for $architecture\n" + if run_capture( + "$root/package-$architecture.log", + $args{packager}, + '--architecture', $architecture, + '--export-dir', $export, + '--output-dir', $packages, + '--version', $args{version}, + '--release', $args{release}, + '--revision', $args{revision}, + '--source-date-epoch', $args{epoch}, + '--format', $format, + ); + if ($format eq 'rpm') { + my $name = rpm_package_name($architecture); + make_path("$release_root/rpm", "$release_root/srpm"); + copy( + "$packages/rpm/$name-$args{version}-$args{release}.noarch.rpm", + "$release_root/rpm/$name-$args{version}-$args{release}.noarch.rpm", + ) or die $!; + copy( + "$packages/srpm/$name-$args{version}-$args{release}.src.rpm", + "$release_root/srpm/$name-$args{version}-$args{release}.src.rpm", + ) or die $!; + } else { + my $name = deb_package_name($architecture); + make_path("$release_root/deb"); + copy( + "$packages/deb/${name}_$args{version}-$args{release}_all.deb", + "$release_root/deb/${name}_$args{version}-$args{release}_all.deb", + ) or die $!; + } + } + write_release_manifest( + $release_root, $args{version}, $args{release}, $args{revision}, + $args{epoch}, join(',', @architectures), $format, + ); + write_checksums($release_root); + return $release_root; +} + sub make_export { my ($directory, $architecture) = @_; make_path($directory); @@ -45,11 +99,12 @@ sub make_export { sub write_release_manifest { my ($directory, $xcat_version, $xcat_release, $revision, $epoch, - $architectures, $formats) = @_; + $architectures, $formats, $manifest_version) = @_; + $manifest_version //= 2; write_binary( "$directory/release.manifest", "format=xcat-genesis-packages\n" - . "version=1\n" + . "version=$manifest_version\n" . "xcat_version=$xcat_version\n" . "xcat_release=$xcat_release\n" . "xcat_revision=$revision\n" diff --git a/t/mockbuild-all.t b/t/mockbuild-all.t index 7511ca9..6adbe95 100644 --- a/t/mockbuild-all.t +++ b/t/mockbuild-all.t @@ -12,6 +12,7 @@ use File::Path qw(make_path); use File::Basename qw(basename); use MockBuildUtils qw(install_deps_packages install_deps_command missing_perl_modules required_pkgs version_matches rpm_sigmd5 rpm_version rpm_release rpm_is_signed + rpm_arch rpm_in_cell restamp_release_line cross_copy_genesis finalize_xcat_dep read_manifest verify_repo_packages verify_repo_signature verify_rpm_signatures parse_evr evr_constraint_ok parse_pin rpmkeys_checksig_problem @@ -247,6 +248,60 @@ is(rpm_release(tempdir(CLEANUP => 1), 'nonexistent-pkg'), undef, 'rpm_release is my @missing = grep { !exists $m{$_}{'conserver-xcat'} } @targets; is_deeply(\@missing, [], 'conserver-xcat is present in every manifest target section') or diag("missing conserver-xcat in: @missing"); + + # The forcearch riscv64 target carries the noarch boot components the ppc64le EL10 target + # carries, at the same pins: a riscv64 MN serves the x86 nodes of a mixed cluster too. + my ($ppc) = grep { /^[a-z+]+-10-ppc64le$/ } @targets; + ok(defined $ppc, 'an EL10 ppc64le target section exists to compare against') or $ppc = ''; + for my $boot (qw(elilo-xcat grub2-xcat syslinux-xcat xnba-undi)) { + is($m{'rocky-10-riscv64-xcat'}{$boot}, $m{$ppc}{$boot}, + "$boot pinned in the riscv64 target as in the EL10 ppc64le target"); + } +} + +# ---- rpm_in_cell: only noarch and the cell's own architecture reach a target's repository ------ +# A noarch builder run in another architecture's chroot (the x86 boot loaders for riscv64 build in +# rocky-10-x86_64) can emit that chroot's native rpms beside the noarch one, as syslinux does. +{ + is(rpm_arch('syslinux-xcat-6.03-1.noarch.rpm'), 'noarch', 'rpm_arch reads noarch'); + is(rpm_arch('syslinux-extlinux-6.03-1.x86_64.rpm'), 'x86_64', 'rpm_arch reads x86_64'); + is(rpm_arch('ipmitool-xcat-1.8.18-4.el10.riscv64.rpm'), 'riscv64', 'rpm_arch reads riscv64'); + is(rpm_arch('not-an-rpm.txt'), undef, 'rpm_arch is undef for a non-rpm name'); + ok( rpm_in_cell('syslinux-xcat-6.03-1.noarch.rpm', 'riscv64'), 'noarch belongs in the riscv64 cell'); + ok( rpm_in_cell('ipmitool-xcat-1.8.18-4.el10.riscv64.rpm', 'riscv64'), 'a riscv64 rpm belongs in the riscv64 cell'); + ok(!rpm_in_cell('syslinux-extlinux-6.03-1.x86_64.rpm', 'riscv64'), 'an x86_64 rpm does not belong in the riscv64 cell'); + ok(!rpm_in_cell('syslinux-debuginfo-6.03-1.x86_64.rpm', 'riscv64'), '... nor its debuginfo'); + ok( rpm_in_cell('syslinux-extlinux-6.03-1.x86_64.rpm', 'x86_64'), 'the same rpm belongs in the x86_64 cell'); + ok(!rpm_in_cell('syslinux-xcat-6.03-1.noarch.rpm', undef), 'no target arch -> not kept (fail-safe)'); + + # With a real file the header decides, so a renamed rpm does not pass for another architecture. + SKIP: { + skip 'rpmbuild not available', 4 if system('command -v rpmbuild >/dev/null 2>&1') != 0; + my $tmp = tempdir(CLEANUP => 1); + my $build = sub { + my ($name, $buildarch) = @_; + my $spec = "$tmp/$name.spec"; + open my $fh, '>', $spec or die; + print $fh "Name: $name\nVersion: 1.0\nRelease: 1\nSummary: fixture\nLicense: EPL\n" + . ($buildarch ? "BuildArch: $buildarch\n" : '') + . "%description\nfixture\n%install\nmkdir -p %{buildroot}/opt/t\necho x > %{buildroot}/opt/t/$name\n%files\n/opt/t/$name\n"; + close $fh; + system("rpmbuild -bb --quiet --define '_topdir $tmp/rpmb-$name' --define '_rpmdir $tmp/out-$name' '$spec' >/dev/null 2>&1") == 0 + or die "rpmbuild failed for $name"; + my ($rpm) = glob("$tmp/out-$name/*/$name-*.rpm"); + return $rpm; + }; + my $host = `uname -m`; chomp $host; + my $noarch = $build->('cell-noarch', 'noarch'); + my $native = $build->('cell-native', undef); + my $disguised_native = "$tmp/cell-native-1.0-1.noarch.rpm"; + my $disguised_noarch = "$tmp/cell-noarch-1.0-1.$host.rpm"; + require File::Copy; File::Copy::copy($native, $disguised_native) or die; File::Copy::copy($noarch, $disguised_noarch) or die; + is(rpm_arch($disguised_native), $host, 'the header names the architecture of a native rpm renamed as noarch'); + ok(!rpm_in_cell($disguised_native, 'riscv64') || $host eq 'riscv64', '... so it does not enter the riscv64 cell'); + is(rpm_arch($disguised_noarch), 'noarch', 'the header names noarch for a noarch rpm renamed as native'); + ok( rpm_in_cell($disguised_noarch, 'riscv64'), '... so it enters the riscv64 cell'); + } } # ---- bump_dep_release_suffix: stamps xcat-dep specs, prunes nested xcat-core, idempotent -------- diff --git a/t/sbuild-all.t b/t/sbuild-all.t index dfd09fa..acbc304 100644 --- a/t/sbuild-all.t +++ b/t/sbuild-all.t @@ -17,6 +17,7 @@ use BuildUtils qw(install_deps_packages install_deps_command missing_perl_module verify_repo_packages verify_repo_signature verify_repo_arches parse_packages_index parse_release_architectures resolve_present_names index_has_native_arch control_binary_arch skip_arch_all_on + supported_arches is_supported_arch codename_to_version version_to_codename known_codenames chroot_name chroot_sources_list chroot_is_disposable chroot_build_script control_field genesis_deb_control @@ -327,8 +328,8 @@ SKIP: { my %m = read_manifest("$RealBin/../debs-manifest.conf"); # Not every section is a build target: [shared] describes the ONE pool the OpenEmbedded Genesis # release is published into, which no builder produces. Target sections are -. - my @targets = grep { /^[a-z]+-(?:amd64|ppc64el)$/ } sort keys %m; - cmp_ok(scalar(@targets), '>=', 8, 'manifest has all 8 codename x arch target sections'); + my @targets = grep { /^[a-z]+-(?:amd64|ppc64el|riscv64)$/ } sort keys %m; + cmp_ok(scalar(@targets), '>=', 12, 'manifest has all 12 codename x arch target sections'); ok(!grep({ $_ eq 'shared' } @targets), 'the shared-pool section is not treated as a build target'); # goconserver is a compiled dep built for EVERY target (both arches, all codenames). @@ -337,11 +338,11 @@ SKIP: { or diag("missing goconserver in: @miss_go"); # The noarch boot components (syslinux-xcat, grub2-xcat, elilo-xcat, xnba-undi) are Architecture:all - # single-producer (built ONCE on amd64) but REQUIRED-PRESENT on EVERY target incl. ppc64el, so the - # gate verifies the ppc repo actually carries them (matches the EL manifest + the 2.16 ppc dep repo; - # a ppc MN needs them for netboot). It is the BUILD PHASE -- not the manifest -- that avoids - # rebuilding them on ppc (build_one_codename skips an Architecture:all package on non-amd64; see the - # control_binary_arch test below). + # single-producer (built ONCE on amd64) but REQUIRED-PRESENT on EVERY target incl. ppc64el and + # riscv64, so the gate verifies those repos actually carry them (matches the EL manifest + the 2.16 + # ppc dep repo; a ppc or riscv64 MN serves the x86 nodes of a mixed cluster). It is the BUILD PHASE + # -- not the manifest -- that avoids rebuilding them off amd64 (build_one_codename skips an + # Architecture:all package on non-amd64; see the control_binary_arch test below). for my $t (@targets) { for my $boot (qw(syslinux-xcat grub2-xcat elilo-xcat xnba-undi)) { ok(exists $m{$t}{$boot}, "$boot required-present on $t (arch:all, verified on every arch)"); @@ -561,6 +562,21 @@ case "$1" in -l) echo "chroot:noble-amd64-sbuild"; exit 0 ;; --config) printf '%s\n' "$FAKE_SCHROOT_CONFIG"; exit 0 ;; esac +if [ -n "$FAKE_SMOKE_MARK" ]; then + case "$*" in + *"$FAKE_SMOKE_MARK"*) [ -n "$FAKE_SMOKE_OUT" ] && printf '%s\n' "$FAKE_SMOKE_OUT" + exit ${FAKE_SMOKE_RC:-0} ;; + esac +fi +# A build session leaves its debs in the result directory it was given, which is the private +# staging directory build_deb_in_chroot creates and passes among the session arguments. +if [ -n "$FAKE_BUILD_DEB" ]; then + for arg in "$@"; do + case "$arg" in + */.build-*) [ -d "$arg" ] && : > "$arg/$FAKE_BUILD_DEB" ;; + esac + done +fi exit 0 STUB close $fh; @@ -591,6 +607,62 @@ STUB unlike($@, qr/is NOT disposable/, '... the failure is NOT the disposability guard'); like($@, qr/no \.deb is visible/, '... it is the host-side "debs did not land" check'); } + + # ---- the post-build smoke ------------------------------------------------------------------- + # A cross-built binary runs only in the chroot, so the smoke is the ONLY check that the deb + # carries a runnable binary. The stub now leaves a deb behind like a real build session, so + # the build gets as far as the smoke. + { + local $ENV{PATH} = "$fakebin:$ENV{PATH}"; + local $ENV{FAKE_SCHROOT_CONFIG} = + "[noble-amd64-sbuild]\ntype=directory\ndirectory=/srv/chroot/noble-amd64\nunion-type=overlay\n"; + local $ENV{FAKE_BUILD_DEB} = 'fixture-xcat_1.8.18-4_amd64.deb'; + + my %smoke = (deb => qr/^fixture-xcat_/, run => '/opt/xcat/bin/fixture-xcat -V', + expect => qr/fixture-xcat version 1\.8\.18/); + # The stub stands in for the whole chroot session, so it must answer the smoke without + # answering the build: only the smoke session carries the command being run. + local $ENV{FAKE_SMOKE_MARK} = $smoke{run}; + + local $ENV{FAKE_SMOKE_OUT} = 'fixture-xcat version 1.8.18'; + my $ok = eval { quiet { build_deb_in_chroot(@args, smoke => \%smoke) }; 1 }; + ok($ok, 'a deb whose binary runs and reports the expected version passes the smoke') + or diag($@); + ok(!-e "$work/out/.smoke-fixture.log", '... and the smoke log is removed on success'); + + # The passing case above published its deb; clear it so the next assertion is about what + # THIS run leaves behind. + unlink glob("$work/out/*.deb"); + + local $ENV{FAKE_SMOKE_OUT} = 'fixture-xcat version 1.8.17'; + $ok = eval { quiet { build_deb_in_chroot(@args, smoke => \%smoke) }; 1 }; + ok(!$ok, 'a binary reporting another version fails the smoke'); + like($@, qr/does not match/, '... naming the expectation it missed'); + + # The publish gate checks names and versions only, so a deb the smoke rejected must never + # reach the directory a publish assembles from. + is_deeply([ glob("$work/out/*.deb") ], [], + '... and no deb reaches the result directory'); + + + local $ENV{FAKE_SMOKE_OUT} = 'fixture-xcat version 1.8.18'; + local $ENV{FAKE_SMOKE_RC} = 3; + $ok = eval { quiet { build_deb_in_chroot(@args, smoke => \%smoke) }; 1 }; + ok(!$ok, 'a binary that cannot run fails the smoke even with matching output'); + like($@, qr/smoke failed \(rc=3\)/, '... reporting the exit status'); + + delete local $ENV{FAKE_SMOKE_RC}; + $ok = eval { quiet { build_deb_in_chroot(@args, + smoke => { %smoke, deb => qr/^nosuchpkg_/ }) }; 1 }; + ok(!$ok, 'a smoke that names a deb the build never produced fails'); + like($@, qr/no produced deb matches/, '... instead of silently skipping the check'); + + # --skip-install is what sbuild-all.pl passes to drop the smoke, so the same build with no + # smoke must still succeed -- otherwise the tests above would pass for the wrong reason. + local $ENV{FAKE_SMOKE_OUT} = 'irrelevant'; + $ok = eval { quiet { build_deb_in_chroot(@args) }; 1 }; + ok($ok, 'without a smoke the same build succeeds'); + } } # ---- --install-deps: the host prerequisites (the modules are what actually break a run) ---------- @@ -602,6 +674,12 @@ STUB for my $need (qw(libfile-slurper-perl libparallel-forkmanager-perl sbuild schroot apt-utils dpkg-dev)) { ok(scalar(grep { $_ eq $need } @pkgs), "prerequisites include $need"); } + # ensure_foreign_arch_support() refuses to bootstrap a foreign chroot without the binfmt + # handler and names these two packages, so --install-deps has to be the fix it points at. + for my $need (qw(qemu-user-static binfmt-support)) { + ok(scalar(grep { $_ eq $need } @pkgs), "prerequisites include $need"); + } + my @cmd = install_deps_command(); is($cmd[0], 'apt-get', 'installs with apt-get'); ok(scalar(grep { $_ eq '-y' } @cmd), '... non-interactively'); @@ -618,4 +696,183 @@ STUB 'missing_perl_modules: an absent module is reported'); } +# ---- every compiled dep must be buildable on every architecture xcat-dep supports ------------- +# A debian/control that names architectures explicitly silently excludes the ones it omits: +# debhelper prints "No packages to build. Possible architecture mismatch: , want: ", +# builds nothing, and the build then dies at ./configure. ipmitool-xcat did exactly that on +# riscv64. The Architecture:all packages are the single-producer boot components and are excluded +# here: they are built once on amd64 and never rebuilt per arch. +{ + my $root = "$FindBin::Bin/.."; + for my $pkg (qw(ipmitool conserver goconserver)) { + my $ctl = "$root/$pkg/debian/control"; + SKIP: { + skip "$pkg has no debian/control", 1 unless -f $ctl; + open my $fh, '<', $ctl or die "read $ctl: $!"; + local $/; my $text = <$fh>; close $fh; + my @arch_lines = ($text =~ /^Architecture:\s*(.+)$/mg); + my @explicit = grep { !/^(?:any|all)$/ } map { s/^\s+|\s+$//gr } @arch_lines; + my @missing; + for my $line (@explicit) { + my %have = map { $_ => 1 } split /\s+/, $line; + push @missing, grep { !$have{$_} } grep { $_ ne 'amd64' } supported_arches(); + } + is_deeply(\@missing, [], + "$pkg/debian/control builds on every supported arch (@{[join ' ', supported_arches()]})"); + } + } +} + +# ---- every non-glob manifest pin must match the package's own debian/changelog -------------- +# debs-manifest.conf pins the exact deb version each package must produce, and the version comes +# from that package's debian/changelog. Bumping the changelog without the pin does not fail the +# build -- it fails the manifest VALIDATION, at the end, after every package has been compiled: +# FATAL: manifest validation failed: +# [noble-amd64] grub2-xcat: built 2.12-2, manifest pins 2.12-1 +# which is a whole build's worth of time to learn about a one-line edit. grub2-xcat drifted exactly +# that way when the riscv64 UEFI image was added. Globbed pins are deliberate (goconserver's +# revision is the CD stamp; xcat-genesis-base is not versioned by xcat-dep) and are skipped. +{ + my $root = "$FindBin::Bin/.."; + my %dir_of = ( + 'ipmitool-xcat' => 'ipmitool', + 'conserver-xcat' => 'conserver', + 'syslinux-xcat' => 'syslinux', + 'grub2-xcat' => 'grub2-xcat', + 'elilo-xcat' => 'elilo', + 'xnba-undi' => 'xnba', + ); + my %manifest = read_manifest("$root/debs-manifest.conf"); + my %seen; + for my $section (sort keys %manifest) { + for my $pkg (sort keys %{ $manifest{$section} }) { + my $pin = $manifest{$section}{$pkg}; + next if !defined $pin || $pin =~ /[*?]/; + my $dir = $dir_of{$pkg} or next; + my $cl = "$root/$dir/debian/changelog"; + next unless -f $cl; + open my $fh, '<', $cl or next; + my $first = <$fh>; close $fh; + my ($ver) = $first =~ /^\S+\s+\(([^)]+)\)/; + next if $seen{"$pkg=$pin=$ver"}++; + is($pin, $ver, "manifest pin $pkg=$pin matches $dir/debian/changelog"); + } + } +} + +# In publish mode with no --expect-arch, the expected set is discovered by scanning the staged +# tree. That scan admitted a hardcoded amd64|ppc64el, so a staged riscv64 tree was dropped and +# the arch never reached the code that writes its binary- index and names it in Release. +# The scan is extracted from the script and driven here, so the test tracks the shipped code. +{ + my $src = do { + open my $fh, '<', "$FindBin::Bin/../sbuild-all.pl" or die $!; + local $/; <$fh>; + }; + + my ($scan) = $src =~ /\n( my %u = \(\$arch => 1\);\n if \(\$mode eq 'publish'\) \{\n.*?\n \}\n)/ms; + BAIL_OUT('could not extract the staged-arch scan from sbuild-all.pl') unless defined $scan; + + my $staging = tempdir( CLEANUP => 1 ); + make_path("$staging/noble/$_") for qw(amd64 ppc64el riscv64 s390x); + + my ( $arch, $mode ) = ( 'amd64', 'publish' ); + my @dist_list = ('noble'); + my %u; + ## no critic (BuiltinFunctions::ProhibitStringyEval) + # the extracted text ends inside the publish branch, so the brace closes it; the branch's + # print is deliberately left out, because it would land in the middle of the TAP stream + my $got = eval "$scan }\n[sort keys %u]"; + ## use critic + die "failed to evaluate the staged-arch scan: $@" if $@; + + is_deeply( $got, [ sort( supported_arches() ) ], + 'every supported architecture staged for a codename is expected' ); + ok( ( grep { $_ eq 'riscv64' } @$got ), + 'a staged riscv64 tree reaches the expected set' ); + ok( !( grep { $_ eq 's390x' } @$got ), + 'a staged tree for an unsupported architecture is still ignored' ); +} + +# The legacy Genesis deb is named per target in the manifest, and riscv64 does not name it: its +# Genesis is the OpenEmbedded package published once into the shared pool. build_genesis ran for +# every architecture, so a plain --arch riscv64 run died for want of a --genesis-deb it can never +# have -- and only after the dependency builds had finished. +{ + my $src = do { + open my $fh, '<', "$FindBin::Bin/../sbuild-all.pl" or die $!; + local $/; <$fh>; + }; + + my ($sub) = $src =~ /\n(sub genesis_in_manifest \{\n.*?\n\})\n/ms; + BAIL_OUT('could not extract genesis_in_manifest from sbuild-all.pl') unless defined $sub; + + our %MANIFEST = ( + 'noble-amd64' => { 'xcat-genesis-base' => '2.*', 'ipmitool-xcat' => '1.8.18-4' }, + 'noble-ppc64el' => { 'xcat-genesis-base' => '2.*' }, + 'noble-riscv64' => { 'ipmitool-xcat' => '1.8.18-4', 'goconserver' => '0.3.3-snap*' }, + ); + our @dist_list = ('noble'); + our $arch; + + ## no critic (BuiltinFunctions::ProhibitStringyEval) + eval "$sub 1" or die "failed to evaluate genesis_in_manifest: $@"; + ## use critic + + $arch = 'amd64'; + ok( genesis_in_manifest(), 'amd64 names the legacy Genesis deb, so that phase runs' ); + $arch = 'ppc64el'; + ok( genesis_in_manifest(), 'ppc64el names it too' ); + $arch = 'riscv64'; + ok( !genesis_in_manifest(), 'riscv64 does not, so the legacy Genesis phase is skipped' ); + $arch = 's390x'; + ok( !genesis_in_manifest(), 'a target with no manifest section does not demand Genesis' ); +} + +# A foreign-architecture chroot is bootstrapped and built through qemu-user, so a missing binfmt +# handler has to be reported here rather than deep inside debootstrap. The check is extracted from +# the script and driven with the handler node redirected into a temporary tree. +{ + my $src = do { + open my $fh, '<', "$FindBin::Bin/../sbuild-all.pl" or die $!; + local $/; <$fh>; + }; + + my ($sub) = $src =~ /\n(sub ensure_foreign_arch_support \{\n.*?\n\})\n/ms; + BAIL_OUT('could not extract ensure_foreign_arch_support from sbuild-all.pl') unless defined $sub; + + my ($map) = $src =~ /\n(my %BINFMT_HANDLER = \(.*?\);)\n/ms; + BAIL_OUT('could not extract the binfmt handler map') unless defined $map; + + my $fake = tempdir( CLEANUP => 1 ); + ( my $driver = "$map\n$sub" ) =~ s{/proc/sys/fs/binfmt_misc}{$fake}g; + ## no critic (BuiltinFunctions::ProhibitStringyEval) + eval "$driver 1" or die "failed to evaluate ensure_foreign_arch_support: $@"; + ## use critic + + my $host = `dpkg --print-architecture 2>/dev/null`; + chomp $host; + + SKIP: { + skip 'needs dpkg to report a host architecture', 3 unless $host; + + # the host's own architecture never needs emulation + eval { ensure_foreign_arch_support($host) }; + is( $@, '', 'a native build does not ask for a binfmt handler' ); + + my $foreign = $host eq 'riscv64' ? 'ppc64el' : 'riscv64'; + my $handler = $foreign eq 'riscv64' ? 'qemu-riscv64' : 'qemu-ppc64le'; + + eval { ensure_foreign_arch_support($foreign) }; + like( $@, qr/binfmt handler is not registered/, + "a foreign $foreign build without the handler is refused" ); + + open my $fh, '>', "$fake/$handler" or die $!; + print {$fh} "enabled\ninterpreter /usr/libexec/qemu-binfmt/$handler\nflags: POF\n"; + close $fh; + eval { ensure_foreign_arch_support($foreign) }; + is( $@, '', "a foreign $foreign build with the handler registered proceeds" ); + } +} + done_testing;