PR #62 review R3.3 objected that the --skip-genesis/--skip-perl/--skip-xcat-dep
modes still validated every manifest package, so a clean skip run failed the
completeness gate. The gate was reworked to derive its expected set with
required_pkgs(...skip flags), but the fixtures tested required_pkgs and
verify_repo_packages only in isolation -- not the composition that the gate
actually performs.
Add a fixture that feeds the required_pkgs-filtered expected set into
verify_repo_packages with the skipped package absent from the repo, asserting no
MISSING is reported; a paired unfiltered case proves the skip filter is
load-bearing (the same absent package IS flagged without it).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Landing-review nits: the genesis pre-scrub comment said 'per-uniqueext bootstrap'
but the genesis chroot carries no --mock-uniqueext; and the Packages note said
'conserver' where the shipped package is 'conserver-xcat' ('dnf install conserver'
would pull the distro package, not xCAT's).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Move the pyodbc note under a dedicated 'Packages notes' section and add a note
that conserver is shipped for completeness/backward-compat only: core packages
depend on goconserver, so conserver is not pulled in as a dependency -- to use it
you must install it explicitly, then disable the goconserver service and enable
the conserver service.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Review follow-ups (all low-impact):
- The genesis pre-build scrub and scrub_buildroot comments claimed mock 'refuses
and we skip' on a held buildroot lock; mock actually BLOCKS until release. Correct
the wording and wrap the genesis scrub in 'timeout 300' so even a pathological
stale-lock wait can never hang the build (the CD topology never runs a second
same-target build concurrently, so it is belt-and-suspenders).
- Drop the inaccurate 'no re-bootstrap' note: --scrub=bootstrap intentionally forces
a re-bootstrap from the root cache (that bootstrap is the leak).
- The mount-namespace re-exec ran for every invocation, printing a spurious non-root
warning for the documented no-root, build-free modes (--verify-repo,
--finalize-xcat-dep). Skip the re-exec for those (no mock, no cgroup exposure).
- Soften the abs_path($0) comment: it resolves a relative $0 against cwd but does
not search $PATH.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The legacy pyodbc/ RPM spec is kept for reference but pyodbc is absent from
every target's manifest because modern EL ships python3-pyodbc from
appstream/EPEL. Document this so the omission (vs the 2.16/2.17 xcat-dep repos)
is not mistaken for a gap.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Two review follow-ups to the EL matrix build:
1. Standalone --verify-repo silently PASSED the signature check when no
--gpg-key-name/--gpg-home was configured, contradicting its documented
contract (it advertises a signature assertion). verify_target_repo now
takes a sig_required flag, set for the standalone mode, so a missing
signing identity is a loud SIGKEY failure instead of a false pass. The
pipeline auto-runs are unaffected (they always pass --gpg-sign).
2. The mount-namespace re-exec used a bare $0, which the re-exec'd perl
could fail to find if the script was invoked via a $PATH name from a
different cwd (exec succeeds, so the unisolated fallback never runs and
the build dies hard). Resolve $0 to an absolute path first.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock mounts /sys/fs/cgroup into every build chroot. On the systemd build
hosts every mount carries `shared` propagation, so the chroot cgroup
joins the same peer group as the host's own /sys/fs/cgroup. When mock
tears a chroot down -- its post-build --scrub, or an aborted build's
cleanup -- the cgroup unmount PROPAGATES back through the shared peer
group and unmounts the HOST's /sys/fs/cgroup. Every subsequent mock (and
even new login sessions) then fails with 'Failed to determine whether the
unified cgroups hierarchy is used: No medium found', wedging the whole
build host until an operator remounts cgroup2.
ppc64le is hit hardest because it leaks corpse chroot mounts on abort,
but x86_64 shares the identical shared-cgroup exposure and is one bad
abort away from the same failure.
Re-exec mockbuild-all.pl inside a private mount namespace made rslave
(unshare --mount --propagation slave): the namespace still sees host
mounts one-way, but nothing mock mounts or unmounts can propagate out to
the host, so a chroot teardown can no longer unmount the host cgroup. The
namespace also auto-reaps every mount mock leaks when the process exits,
so an aborted build no longer strands corpse mounts under /var/lib/mock.
Best-effort and guarded: only re-execs as root with unshare present, and
MOCKBUILD_ALL_MOUNTNS prevents a re-exec loop.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The xCAT-genesis-base build runs buildrpms.pl without a per-run
--mock-uniqueext, so its mock chroot (xCAT-genesis-base-<target>) is
shared across CD runs. mock only scrubs a chroot on a SUCCESSFUL build,
so a killed or failFast-interrupted prior run leaves the chroot stunted
(missing /bin/sh), and mock REUSES that corpse on the next run, which
then dies with FileNotFoundError: '/bin/sh' during genesis -- failing an
otherwise-healthy build until an operator manually scrubs the chroot.
Scrub the genesis chroot before building it: a lock-safe, best-effort
'mock --scrub=chroot --scrub=bootstrap' (skipped if a concurrent build
holds the lock, a no-op on the first run before the config exists). Any
corpse from an interrupted run is dropped and mock recreates the chroot
fresh from the cached root, so an interrupted build can no longer poison
the next one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Cross-arch genesis / finalize step re-indexes + re-signs each per-EL repo AFTER the
per-target deploy_target gate ran, so the auto-run never saw the final shipped state.
Run the same manifest completeness + signature gate at the end of --finalize-xcat-dep over
every finalized rh<N>/<arch> cell, so the build script verifies its own FINAL output by
default -- no external --verify-repo call needed. Suppressible with --no-verify-repo.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Re-review polish: gpg_key_fingerprint now returns the primary-key fingerprint only when
EXACTLY ONE key matches --gpg-key-name (undef if absent or ambiguous), and the gate keys
its SIGKEY hard-fail off 'undef' rather than 'looks hex' -- so a name matching multiple
keys, or an unresolvable key, can never be silently accepted.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Per review of the gate design:
- Signature: fail (SIGKEY) if --gpg-key-name cannot be resolved to a fingerprint, so the
gate always confirms the repo was signed by EXACTLY the CLI key -- never a soft pass.
(EL already dies loudly on a duplicate rpm version via rpm_version; Ubuntu now matches.)
- Document the gate + its intentional idiosyncrasies (what 'the repo' is, duplicate=hard
error, signature identity) in BUILD.md.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Review follow-up on the completeness+signature gate:
- repomd_observed_signer keyed off VALIDSIG, which gpg also emits for an EXPIRED or
REVOKED key (and an expired signature) -- so a no-longer-trustworthy signature would
PASS the gate. Reject EXPKEYSIG/REVKEYSIG/EXPSIG explicitly before accepting VALIDSIG.
- drop the now-unused have_rpm import (its only caller, assert_required_deps, was replaced
by verify_target_repo).
prove t/mockbuild-all.t: 68/68.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Adds a real gate on the BUILT per-target repo, using packages-manifest.conf as the
single source of truth, layered so the decision is pure and unit-tested:
- MockBuildUtils::verify_repo_packages(\%expected,\%present) -- pure completeness
(MISSING / VERSION vs the manifest pins); verify_repo_signature(\%expected,\%observed)
-- pure signature identity (UNSIGNED / WRONGKEY). Both unit-tested (happy+sad).
- mockbuild-all.pl does the IO and calls both from one sub, verify_target_repo:
reads the manifest, enumerates the repo via rpm_version, resolves --gpg-key-name to a
primary-key fingerprint and extracts repomd.xml.asc's actual signer (VALIDSIG), then
merges the two pure results and dies listing every problem.
- Runs AUTOMATICALLY at the end of deploy_target (after sign+index), replacing the old
assert_required_deps + inline version-pin loop with one consolidated gate; suppressible
with --no-verify-repo. Also a standalone build-free '--verify-repo=<repo>' mode (manifest
from repo_root, gpg from --gpg-key-name/--gpg-home; target derived from the rh<N>/<arch>
path or --target).
prove t/mockbuild-all.t: 68/68 (was 50). gpg round-trip smoke-tested: right key -> OK,
wrong key -> WRONGKEY, missing .asc -> UNSIGNED.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Follow-up to the @viniciusferrao review of the EL matrix build:
- Rewrite the stale run_build_steps_parallel comment that still described the
removed 'tolerate genesis failure' workaround; the code is strict
zero-tolerance (xcat-core #7696 made buildrpms.pl exit 0 iff it built the
genesis rpm), so the comment now matches.
- sh_quote the operator-supplied --gpg-key-name at every rpmsign/gpg site
(was interpolated raw into the shell).
- Derive assert_required_deps' required set from the target's
packages-manifest.conf section (the single source of truth) instead of a
second hard-coded list that could drift.
- Move bump_dep_release_suffix into MockBuildUtils (pure, arg-driven) and add a
File::Temp fixture test (stamp, xcat-core prune, no-Release skip, idempotency)
-- the paths the review asked to cover. Its temp file now carries hostname+pid
so the two arch build hosts can't collide on the shared NFS tree.
prove t/mockbuild-all.t: 50/50.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
goconserver always compiles in the el10 chroot for the arch (el8/el9 ship a Go
too old for 0.3.3), so mockbuild-all's post-build scrub -- which keys on the target
cfg (el8/el9) -- cannot reach it, and goconserver self-scrubs its el10 build chroot.
But it ran only 'mock --scrub=chroot', leaving the ~190 MiB bootstrap-image tree
'<cfg>-bootstrap-<uniqueext>' behind. One survived per target per run and piled up in
/var/lib/mock -- part of the disk leak that filled the x86 build host to 99% and
flaked a build (VersatusHPC/xcat-core#51). Add '--scrub=bootstrap' so goconserver
reclaims its whole chroot, matching mockbuild-all's scrub_buildroot.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Completes @viniciusferrao's concern #2. Previously goconserver was built on the HOST with a runtime
`go mod tidy` against a clone of mutable `master` -- non-reproducible and non-hermetic.
- Rewrite goconserver/mockbuild.pl to build the rpm INSIDE a mock chroot via an SRPM: %build compiles
in-chroot (BuildRequires: golang, GOTOOLCHAIN=local, CGO_ENABLED=0).
- Commit only the pinned module manifest goconserver/gomod/{go.mod,go.sum} (97 lines; go.mod carries
the kr/pty -> creack/pty replace). The in-chroot build downloads the modules from the Go proxy
(mock networking enabled) but is reproducible because go.sum integrity-checks every module -- no
`go mod tidy`, and no committed vendor tree.
- goconserver is a CGO-free static binary and el8/el9 chroots ship too old a Go for 0.3.3, so always
COMPILE in the el10 chroot for the arch; the Release still carries the target's dist tag (4.el<rel>),
so every EL repo gets an identical, portable static binary. Verified on the build host: statically
linked, no shared-lib deps, correct el<rel> tag while built in the el10 chroot.
Combined with the immutable-SHA pin + --release-suffix (40feffc), goconserver is now reproducible,
built in mock, and advances its NVR per CD run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Follow-up to 40feffc, addressing the nice-to-have / secondary points from @viniciusferrao's review.
1 (over-reach): the --build-number spec walk now PRUNES a nested `xcat-core`/`xcat-source-code`
checkout under $repo_root, so the legacy nested layout can no longer rewrite an xCAT-core spec
(e.g. xCAT-genesis-base.spec's dynamic Release). In the normal sibling layout nothing changes.
4 (validate %RELEASE): after the manifest %VERSION pins, when a CD --build-number bump is in effect
the run now also asserts the bump actually LANDED in each built dep/perl rpm's %RELEASE (genesis
excluded -- it is intentionally not bumped). Catches a silently un-bumped NVR that a Version-only
check misses. New MockBuildUtils::rpm_release helper.
6 (--max-parallel a real cap): the perl builder internally forks up to $effective_parallel_builds
mock jobs, so running it concurrently with the dep builders let live mock builds reach ~2x the
cap. Run the perl builder in its OWN phase, after the (quick) dep builders -- each phase then runs
at most $effective_parallel_builds mock builds, so --max-parallel holds, at a small bounded cost.
Tests: 45/45 in t/mockbuild-all.t (rpm_release added).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Reviewed by @viniciusferrao. Each numbered point below is his; the code changes verify + fix it.
1. --build-number over-reach / dry-run / double-stamp
- The bump now runs ONLY on a real build: `--dry-run --build-number N` prints what it would
stamp and writes nothing (previously it rewrote every spec on disk during a dry run).
- Re-stamping is idempotent AND replacing: a re-run in a reused tree with a different
--build-number strips the prior .snap<ts>.<n> before applying the new one, instead of
accumulating a second stamp (…snap57.snap58). Extracted the per-line logic into the
unit-testable MockBuildUtils::restamp_release_line and covered it in t/mockbuild-all.t.
(The headline "rewrites xcat-core / xCAT-genesis-base.spec" does not occur in the real layout:
xcat-core is a sibling of $repo_root, and there is no genesis spec under the dep tree. The
legacy nested xcat-source-code case remains a non-CD layout; left as a follow-up.)
3. --finalize-xcat-dep idempotency
- cross_copy_genesis compared only SIGMD5 (content), which is blind to signature + index state.
It now also treats a same-content-but-UNSIGNED destination rpm as not-up-to-date (new
rpm_is_signed helper) so a crash between copy and sign heals on re-run.
- finalize_xcat_dep now re-indexes+signs BOTH repos of a touched pair every run, not only when
an rpm was copied, so a crash after copy+sign but before createrepo (rpm on disk, absent from
repomd) also heals.
4. Stale run-state can mask a failed build
- A real build now wipes its per-target $run_root first (run_id is derived from the deterministic
commit time, so re-runs reused the same tree). --skip-build keeps the tree; --dry-run writes nothing.
- mockbuild-perl-packages.pl clears each package's stale status.txt/error.txt BEFORE building, and
the aggregate now treats the child worker's exit code as authoritative: a package is PASS only if
its worker exited 0 AND wrote a PASS this run (a stale PASS in a reused log dir no longer counts).
5. --skip-build can publish the wrong artifacts
- --skip-build now REQUIRES an explicit --target (without it, all three EL targets collected the
same EL-agnostic roots and cross-published them).
- Collection is scoped to this target's own per-target $build_root (the same tree a normal build
populates), not the legacy build-output/list3/list5/list6 dirs.
- The manifest version-pin validation (and the "no manifest section" guard) now also run under
--skip-build, so a collection-only publish is validated exactly like a fresh build.
2. goconserver bypassed the CD bump (minimal fix; hermetic rebuild deferred)
- goconserver/mockbuild.pl gains --release-suffix, appended to its generated `Release: 4.elN`,
and mockbuild-all.pl passes the CD suffix down -- so goconserver's NVR advances per run like
every other dep package (an additive publish is no longer a silent no-op on a frozen NVR).
- Pinned the clone to an immutable upstream commit instead of the moving `master` (0.3.3 is
unreleased -- newest tag is v0.3.2 -- so a SHA pin is required; clone now fetches by ref).
- The host build + `go mod tidy` hermeticity concern is a tracked follow-up, not in this change.
7. Docs
- BUILD.md: --target is a single value, not repeatable; conserver-xcat is built for every target
(not "not required"). POD: --parallel-targets default is 1 = serial, not "auto".
- Added a manifest<->docs consistency test (conserver-xcat present in every target section).
(6, --max-parallel not a true global cap, is a documented nice-to-have and is left as a follow-up.)
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The parallel perl-package builder scrubbed each package's chroot AND bootstrap
(`mock --uniqueext <pkg> --scrub=chroot --scrub=bootstrap`) as soon as that
package finished, mid-batch. The comment assumed the --uniqueext made this
sibling-safe, but mock's bootstrap scrub ignores --uniqueext and removes the
CONFIG-LEVEL shared bootstrap cache (/var/cache/mock/<cfg>-bootstrap/, keyed by
config name, not uniqueext). Under concurrency a faster sibling's post-build
scrub deleted that shared cache while a slower sibling was still setting up its
buildsrpm chroot and about to bind-mount it, so the bind failed with mount
rc=32 and the whole target failed a build that was otherwise fine.
Observed: on alma+epel-10-ppc64le, perl-Sys-Virt (the slowest, a libvirt C
binding) died at --buildsrpm binding
/var/cache/mock/alma+epel-10-ppc64le-bootstrap/yum_cache ~1s after two faster
siblings had just scrubbed that shared bootstrap; the other five perl packages
passed. Non-deterministic and load-triggered, so it surfaced under the 3-way
concurrent CD load.
Reclaim only the (uniqueext-local) build chroot per package during the batch,
and defer the shared bootstrap reclamation to a single serialized pass after
all workers join, when nothing can be binding it. Disk reclamation is preserved
(the ~GB build chroots are still freed immediately; the bootstrap roots + shared
cache are freed at batch end).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Follow-up self-review hardening on top of the PR #62 review response:
- finalize_xcat_dep now treats a missing ppc64le PEER repo (not just missing
genesis rpms) as fatal instead of silently skipping the OS -- in the CD both
arches build every EL, so a missing peer is an incomplete input that would
otherwise leave the x86_64 repo without the ppc64 genesis and still exit 0.
- cross_copy_genesis treats an empty SIGMD5 (unreadable rpm) as "cannot confirm
identical" and refreshes, rather than risking a false up-to-date match when two
unreadable rpms both return an empty digest.
- rpm_version fails when a directory holds more than one distinct version of a
package (a stale artifact not cleaned before the build) instead of silently
returning the first sorted match, which a version pin could pass against while
the stale rpm still ships. Both arches share a Version for genesis, so a normal
x86_64+ppc64 pair is a single entry.
t/mockbuild-all.t: +4 cases (30 total) -- missing-peer fatal, rpm_sigmd5 on a
missing rpm, and rpm_version multi-version failure.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Review feedback (viniciusferrao):
1. --finalize-xcat-dep no longer succeeds with no genesis rpms and no longer treats
a shared filename as up to date when the content differs.
- finalize_xcat_dep now REQUIRES each arch's own genesis rpm for every repo pair
it processes (a pair with none is a hard FATAL, not a silent exit-0 no-op).
- cross_copy_genesis compares RPM identity by SIGMD5 (header+payload digest,
independent of the GPG signature), so a stale rpm that merely shares a basename
is refreshed instead of being mistaken for up to date.
2. Remove the genesis workaround. xcat-core #7696 is merged, so buildrpms.pl now
exits 0 iff it produced the genesis rpm; the zero-tolerance check no longer
ignores a genesis failure when a matching (possibly stale) rpm already exists --
any failed build step, genesis included, fails the run.
3. Skip modes work. required_pkgs() drops the packages whose builder was skipped, and
both the version-pin check and assert_required_deps use it, so a clean
--skip-genesis / --skip-xcat-dep / --skip-perl run no longer fails validating
packages it deliberately did not build.
4. Tests. The reusable, side-effect-free helpers are factored into MockBuildUtils.pm
(cross_copy_genesis and finalize_xcat_dep take injected sign/reindex callbacks so
they carry no gpg/createrepo state) and t/mockbuild-all.t adds focused fixture
tests for all of the above: skip-mode selection, version-pin globs, SIGMD5-based
RPM-identity comparison + cross_copy refresh/idempotency, and the finalize
require-inputs guard. Run with `prove t/`.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
elilo, ipmitool and syslinux build from a tracked, in-repo source tarball and no
longer fetch at build time, so the upstream download URL was undocumented. Add
it back as a provenance comment next to the tracked-source block so it is clear
where the tarball came from and where to re-download when bumping the version.
(xnba already records https://ipxe.org/ and goconserver keeps its git repo URL
in $go_repo; grub2-xcat is repackaged from the distribution grub2 and conserver
uses a dummy spec, so neither has a single upstream download URL.)
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Manifest version pins may now be an exact Version, a shell-style glob (* and ?),
or `*`. version_matches() anchors the glob (quotemeta then *->.* , ?->.), so
2.* matches 2.18.x / 2.19.x but not 3.x or 20.x.
Use it to pin xCAT-genesis-base=2.* on every target. genesis-base's Version is
not owned by xcat-dep -- it is whatever xcat-core (XCAT_CORE_REF) the genesis
build compiles against, so it walks with the paired core (2.18.x today, 2.19.x
on master). An exact pin would fail the run whenever the dep is built against a
different core; 2.* asserts "a 2.x genesis" without coupling the manifest to a
single core release. xCAT-genesis-scripts Requires xCAT-genesis-base >= 2:2.18.0
(a minimum, Epoch 2), so any 2.x genesis-base installs against a 2.18+ core.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Replace the `*` placeholders in package-manifest.conf with the concrete
package Versions each source builds (e.g. ipmitool-xcat=1.8.18,
perl-Sys-Virt=11.10.0, xCAT-genesis-base=2.19.0). Only the Version is pinned,
not the Release (per-EL dist tag / genesis snap<timestamp>), and the Version is
identical across all targets so the same pin applies everywhere.
Enforce the pins: after collection, rpm_version() reads each required package's
built %{version} from the repo and build_one_target fails the run if it differs
from the pin (or the package is absent). `*` still accepts any version. This
turns an unnoticed source Version bump into an explicit, actionable failure
instead of a silently-shipped surprise.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
conserver-xcat is not pulled by `dnf install xCAT` (goconserver superseded
it), but some users still deploy conserver, so build+ship it on every target.
Document why the variable perl modules are scoped per EL: each is required from
xcat-dep only where neither the base OS nor EPEL provides it. In particular
perl-Sys-Virt is omitted on el8 because EPEL provides it on AlmaLinux 8 (not a
build issue); likewise perl-HTML-Form (el9/el10), perl-Crypt-SSLeay and
perl-Net-Telnet (el8/el9) come from the OS/EPEL on the releases where they are
omitted.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Until now mockbuild-all.pl built every dep package + every perl package + genesis
on every target, and TOLERATED build-step failures: a builder "expected to fail
on a given arch/el" (e.g. perl-Sys-Virt on el8) was warned and swept under the
rug, with correctness only re-checked after the fact by a hardcoded
assert_required_deps set. That hid real failures until the post-collection gate
and shipped packages a target does not need.
Replace that with an explicit, empirically-derived manifest. package-manifest.conf
has one [<target>] section per (EL, arch) listing <package>=<version|*>; each
target builds ONLY the packages listed for it. The sets were derived
authoritatively -- on a clean MN of each of the six targets, xcat.org LATEST
xcat-core + xcat-dep were configured, `dnf install xCAT` was run, and the
packages whose from_repo=xcat-dep were captured. That is exactly what xCAT pulls
from xcat-dep on that target. Results: conserver-xcat is required by no target
(goconserver supersedes it), and the variable perl modules differ per EL because
the OS/EPEL already provides the rest there.
Build failures are no longer tolerated: run_build_steps_parallel now returns the
failed step ids and build_one_target fails the whole run if any required package
failed. The one exception is xCAT-genesis-base -- xcat-core's buildrpms.pl exits
non-zero on an unrelated post-build xCAT-release-latest cp even when the genesis
rpm IS produced, so genesis is judged by rpm-produced, not exit code.
mockbuild-perl-packages.pl already honors --packages; the orchestrator now passes
each target's required perl subset so only those are built.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The EL build orchestrator never reclaimed the per-step mock buildroots it
created. Each build step makes a build chroot AND a per-uniqueext bootstrap
chroot under /var/lib/mock (dep packages, per-package perl chroots, and
xCAT-genesis-base); the child builders copy their RPMs/logs to their
--result-dir and exit without scrubbing, and mock's own cleanup leaves them
behind (and keeps them entirely on failure). So /var/lib/mock grew ~15-17G per
run, unbounded, until the build host filled to 99% and every mock dnf
transaction failed for lack of space ("Error: needs N MB more space on the /
filesystem", rc=30), cascading to rc=2/rc=255 across packages and killing
otherwise-healthy builds.
After the parallel build phase, scrub each step's buildroot with
"mock -r <chroot> --uniqueext <ext> --scrub=chroot --scrub=bootstrap" -- mock's
own lock-safe scrub: a chroot still held by a concurrent build is refused and
skipped (never rm, which would race a live build). Both the build chroot and its
per-uniqueext bootstrap are removed (each is per-uniqueext, so both leak); the
shared root cache under /var/cache/mock is kept so rebuilds stay fast. The
orchestrator scrubs the dep-package and genesis chroots (whose uniqueext it
assigns); mockbuild-perl-packages.pl scrubs each of its per-package chroots (it
derives its own per-package uniqueext). Scrubs are non-fatal -- a cleanup hiccup
never fails the build. --keep-buildroots preserves the buildroots for debugging.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The elilo, ipmitool and syslinux builders each fetched their upstream
source and rewrote the tracked source tarball in place, inside the
package source directory that both arch builds share. When the two
per-arch builds run in parallel they were racing to fetch the source:
one build truncated and rewrote the tarball while the other read it, so
the reader got a truncated archive and failed intermittently with
"missing top-level tree" errors.
The correct, normalized source is already tracked in the repository and
is what mock consumes, so the fetch is redundant as well as unsafe. Drop
the download/normalize entirely and verify the tracked source read-only
(it exists and has the expected top-level tree). With no writer, the
shared source is only ever read, so parallel per-arch builds can no
longer race on it. Also removes the now-dead --source-url /
--skip-upstream-download options, the normalize helper, and the wget
dependency check.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The goconserver binary parses /etc/goconserver/server.conf as YAML, but the
package shipped it in the old INI ("[server]\nhost = ...") format. The YAML
parser reads the leading [server] as a sequence, so the daemon panics at startup
("yaml: cannot unmarshal !!seq into common.ServerConfig"); systemd then
rate-limits the service to 'failed' before xCAT (Goconserver.pm) rewrites the
config as YAML. On a management node this leaves goconserver down, so the
provisioning test cases' makegocons cannot register a console and the case fails.
Ship a valid minimal YAML default matching the schema xCAT itself writes
(global/api/console; api port 12429, console port 12430, datadir); xCAT still
overwrites it with the cert-enabled config on the MN, and this default now only
has to parse + start. Bumps release el -> 4.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
elilo/mockbuild.pl wget'd the upstream tarball over $source_path and then
normalized it in place, rewriting the tracked source file. Parallel builds share
that same source file, so builds running concurrently raced to fetch and rewrite
it -- one build could read the file mid-rewrite and get a truncated archive, an
intermittent "Normalized source archive still missing elilo top-level tree"
failure. Because elilo is a required dep, one such flake failed the whole run.
The tracked tarball is already normalized (elilo/ top-level), so use it read-only
when normalized and only fetch upstream when it is absent or not yet normalized.
This removes the in-place rewrite (hence the race) and the flaky sourceforge
dependency, and is more reproducible.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Per review on xcat2/xcat-dep#62, narrow this PR to the EL matrix only
(rh8/rh9/rh10 x x86_64/ppc64le). The SUSE and Ubuntu work is reverted out of the
PR's net diff and will land in its own PR, so the reviewer's SUSE/Ubuntu points
(SuSE breakage, ubuntu20.04/focal in the default set, the xcat@megware.com key
default) are moot here -- those targets are no longer part of this change.
Reverted (net-zero vs master):
- SUSE target support in mockbuild-all.pl (opensuse-leap -> sles<N>).
- SUSE perl BuildRequires compat + elilo suse_version hunk.
- build-apt-repo.sh Ubuntu changes (focal + key default).
EL-relevant review fixes kept:
- Rename the finalize options to precise arch names: --x86-repo/--ppc-repo ->
--x86_64-repo/--ppc64le-repo (and matching vars/labels). "x86"/"ppc" was
ambiguous, especially since the genesis package is named -ppc64 via tarch yet
carries no big-endian code.
- Clarify the "tolerated build" comment (reviewer #2): builder failures are
tolerated only so one flaky builder cannot abort the others; correctness is
enforced by RESULT via assert_required_deps (a missing REQUIRED rpm still fails
the run), not by exit code. Toleration is load-bearing -- perl-Sys-Virt fails
on el8 by design, and genesis "fails" cosmetically while still producing its rpm.
- BUILD.md: drop all references to the removed --skip-xcat flag and the stale
"unified xCAT repository" framing (the core is built by the xcat-core
pipeline), and document --finalize-xcat-dep.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
In 2.17 every per-arch xcat-dep repo also shipped the OTHER arch's noarch
xCAT-genesis-base (the x86_64 repo carried xCAT-genesis-base-ppc64, and the
ppc64le repo carried xCAT-genesis-base-x86_64) so an MN could netboot nodes
of the other architecture. The matrix build produces per-arch repos that
each carry only their own genesis, and stale foreign-arch genesis rpms
lingered in the repo unindexed -- which breaks `go-xcat install` on EL9
because the repodata does not resolve the whole install list
(xcat2/xcat-core#7610).
Add a build-free, lock-free `--finalize-xcat-dep --x86-repo <dir>
--ppc-repo <dir>` mode. For each matching <os>/x86_64 and <os>/ppc64le repo
pair it copies the noarch xCAT-genesis-base-ppc64 into the x86_64 repo and
xCAT-genesis-base-x86_64 into the ppc64le repo, drops any stale foreign-arch
genesis first, re-signs the copied rpm + repomd under --gpg-sign, and
re-indexes with createrepo_c. Idempotent: an already up-to-date pair copies
nothing and is not re-indexed.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses the PR #62 review (viniciusferrao):
(2) Remove the monolithic xCAT-core build from mockbuild-all.pl. The full
core is built by the xcat-core pipeline; mockbuild-all now builds ONLY
xcat-dep (dep packages, perl packages, and the OS-dependent
xCAT-genesis-base). This drops the core build that required
perl-generators -- absent on openSUSE Leap -- so the normal SUSE build
no longer fails, and it matches the Ubuntu build's split design. The
--skip-xcat flag is gone (callers updated separately).
(1) Fix packages that missed the CD Release bump so each run publishes a
fresh, monotonic NVR instead of being skipped:
- Sys-Virt: bump_dep_release_suffix now matches Release case-insensitively
(Sys-Virt.spec uses a lowercase `release:`), preserving %{?dist}.
- HTML-Form / IO-Stty / Net-Telnet build from committed .src.rpm files,
which the in-tree spec bump cannot reach. mockbuild-all now passes the
suffix down via --release-suffix; mockbuild-perl-packages re-stamps
these by unpacking the srpm, appending the suffix to the spec Release
(keeping %{?dist}), and rolling a fresh srpm before rebuild. No-op
without a suffix (non-CD runs rebuild the committed srpm unchanged).
(3) The SUSE perl BuildRequires compat repo is now built under the caller's
per-package build dir instead of the shared /tmp/xcat-suse-buildreq-compat
path, so concurrent perl builds (parallel packages and parallel
arch/target invocations) never race on a shared location.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
perl-Sys-Virt / perl-Crypt-SSLeay (XS modules) BuildRequire perl-devel, absent on
openSUSE (the dev headers live in the perl package). Add perl-devel to the compat
provides so dnf builddep resolves it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
- elilo: add %if 0%{?suse_version} to use_prebuilt (openSUSE lacks the gnu-efi
linker inputs elilo compiles against, same as EL8/ppc).
- mockbuild-perl-packages: on openSUSE mock, inject a tiny local repo whose one
noarch rpm Provides perl-generators + perl-interpreter (Requires perl), so
'dnf builddep' on the Fedora perl srpms resolves those (SUSE-absent) names.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Add target_osdir() so a build target maps to its deploy subdir + family:
alma+epel-10-* -> (el, rh10), opensuse-leap-15.6-* -> (suse, sles15). Thread the
osdir + family through deploy_target + write_dep_repo_metadata (sles/devel zypper
baseurl for SUSE, yum for EL). Lets --target opensuse-leap-{15.6,16.0}-<arch>
build the dep set into sles{15,16}/<arch>.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock --print-root-path can fail transiently (bootstrap chroot setup, a concurrent
mock holding a lock), which made el10 flakily resolve to the long os_id form that
has no .cfg and then die 'Could not find mock config for almalinux+epel-10-...'.
Check /etc/mock/<cfg>.cfg existence instead -- deterministic and fast.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Adds the missing focal (20.04) codename and switches the default apt signing
identity from the legacy xcat@megware.com key to the xCAT Signing Key
(64C82A868D818E69) so xcat-dep apt InRelease is signed by the same key as
xcat-core apt.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The mock chroot's %{_host_cpu} is 'powerpc64le' on AlmaLinux ppc chroots but
'ppc64le' on Rocky, so the single ppc64le compare left use_prebuilt unset on
alma and pulled in the ppc-absent gnu-efi BuildRequires. Match both spellings.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
EL8/EL9 mock chroots run an older rpm that does not evaluate an || between a
string compare and an arithmetic test like EL10's rpm, so use_prebuilt stayed
unset on el9-ppc and the gnu-efi BuildRequires (absent on ppc) broke the build.
Use two independent %ifarch ppc64le / %if 0%{?rhel}==8 blocks instead.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
EL8's gnu-efi-devel places elf_x86_64_efi.lds where elilo's Makefile can't find
it, so elilo failed to compile on EL8 -- yet xCAT hard-requires elilo-xcat on
every arch, so the whole dep repo became uninstallable. Reuse the same tracked
prebuilt elilo-x64.efi (SOURCE4) that ppc64le already uses (elilo-x64.efi is a
noarch artifact). Also add elilo-xcat to assert_required_deps so a missing elilo
fails the build loudly instead of surfacing later as a dnf depsolve error.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Two parallel per-arch builds share the NFS source tree; the second finds every
spec already stamped with the identical .snap<epoch>.<build_number> suffix.
Treat all-already-stamped as the normal idempotent case (fail only when NO spec
has a Release: line at all), and write specs atomically (temp + rename) so a
concurrent arch never reads a torn spec.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Append .snap<epoch>.<build_number> to every xcat-dep package spec Release so
each CD run publishes a fresh, monotonic NVR (deploy's additive rsync is a
no-op on an unchanged NVR). Applied before any child builder; genesis-base
lives under xcat-core and is untouched, keeping it in lockstep with the
deployed core's genesis-scripts.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Commit the upstream source tarballs the mock builders consume so a build never
has to fetch them from the network -- offline/reproducible builds, and
resilience to upstream URL rot for these older releases:
- ipmitool/ipmitool-1.8.18.tar.gz (re-normalized to the release tarball)
- syslinux/syslinux-6.03.tar.xz
- perl-Crypt-SSLeay/Crypt-SSLeay-0.72.tar.gz
The per-package builders use the local tarball when present and only fall back
to the upstream URL if it is missing (ipmitool/syslinux mockbuild.pl; and
mockbuild-perl-packages.pl 'spec' mode for perl-Crypt-SSLeay).
Also folded in:
- mockbuild-all.pl: switch createrepo -> createrepo_c (--database,
--set-timestamp-to-revision) for deterministic, upstream-matching repo
metadata.
- perl-HTTP-Async / perl-Net-HTTPS-NB specs: reword the brp-compress comment.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Register conserver-xcat in mockbuild-all.pl's dep-builder set so the
traditional C conserver (8.2.1) is built per-EL/arch alongside goconserver.
xCAT itself requires goconserver, so conserver stays a build-on-demand
artifact, but wiring it into the full dep build keeps it produced and
signed with the rest of xcat-dep for sites that want it.
conserver/mockbuild.pl gains the --build-timestamp option that
mockbuild-all passes to every builder (SOURCE_DATE_EPOCH for deterministic
builds); without it the child invocation would abort on an unknown option.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
conserver.spec had not been built on a modern EL toolchain and no longer
compiled on EL9/EL10:
- %prep used the bare %patch / %patch1 macros, which rpm 4.18+ (EL9/EL10)
rejects with "Patch number not specified". Switched to explicit
Patch0:/Patch1: with the numbered %patch0/%patch1 macros, which apply
cleanly on EL8 through EL10.
- BuildRequires listed only openssl-devel, so on EL9/EL10's minimal mock
buildroot the toolchain was absent and %configure failed with
"C compiler cannot create executables". Added gcc, make and glibc-devel.
Also add conserver/mockbuild.pl, a standalone per-EL builder matching the
other xcat-dep builders (goconserver/ipmitool): it stages the sources and
spec, builds the SRPM, mock-rebuilds it in the target chroot, copies the
RPMs to --result-dir, and smoke-tests console/conserver in the chroot.
conserver is not in the default mockbuild-all.pl set (xCAT uses goconserver),
so this builder is run on demand. Built + smoke-tested conserver-xcat-8.2.1
for alma+epel-{8,9,10}-x86_64.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mockbuild-all.pl treated xnba-undi/grub2-xcat as x86-only @SHARED_NOARCH imported
onto ppc, so an x86 build failure rippled into a fatal "grub2-xcat missing" on
ppc; both actually build on any arch (noarch repackaging of committed artifacts)
and xCAT requires them on ppc too. Un-gate xnba-undi and drop the import so each
host builds a complete, self-sufficient dep repo. Add a single --output that
re-roots all NFS-shared output plus a fail-fast lock at <output>/.lock (owner-pid
guarded so forked build children do not delete it). Default gpg-key-name to the
real "xCAT Signing Key" and --xcat-source to ../xcat-core.
Make concurrent builds safe: give the genesis buildrpms.pl its own HOME/rpmbuild
tree (a shared /root/rpmbuild raced across parallel targets), and nest the
xnba/goconserver rpmbuild dirs under the run-scoped --work-dir (they hard-coded
/var/tmp/xcat-rpmbuild-* and wiped each other). Add --parallel-targets/--max-parallel,
defaulting to serial since the per-package scripts still share repo tarballs under
full parallelism.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
grub2-xcat/mockbuild.pl fetched a grub2 source RPM before building, and that
fetch was the only failing step (the pinned CentOS Stream URL now 404s, and the
dynamic dnf fallback is fragile on EL10). The download is dead weight: the
package is a pure noarch repackaging of the committed grub2-res.tar.gz and the
mock build never consumed the src.rpm. Remove the resolve+download machinery.
Epoch: 1 stays so 1:1.0 satisfies xCAT-server's legacy >= 2.02 pin and upgrades
over old 0:2.02 remain clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT Requires xnba-undi and grub2-xcat on every arch, but they only build on x86_64
(xnba-undi is an x86 UNDI netboot ROM; grub2-xcat wraps the distro grub2). Since both
are noarch, a ppc build now imports them from a built x86_64 dep repo via the new
--import-noarch-repo <dir>. Both are added to the required-deps assertion (with a hint
to pass --import-noarch-repo on non-x86_64 builds), so an incomplete dep repo fails at
build time instead of at xCAT install time.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst