Landing-review nits: the genesis pre-scrub comment said 'per-uniqueext bootstrap'
but the genesis chroot carries no --mock-uniqueext; and the Packages note said
'conserver' where the shipped package is 'conserver-xcat' ('dnf install conserver'
would pull the distro package, not xCAT's).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Move the pyodbc note under a dedicated 'Packages notes' section and add a note
that conserver is shipped for completeness/backward-compat only: core packages
depend on goconserver, so conserver is not pulled in as a dependency -- to use it
you must install it explicitly, then disable the goconserver service and enable
the conserver service.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The legacy pyodbc/ RPM spec is kept for reference but pyodbc is absent from
every target's manifest because modern EL ships python3-pyodbc from
appstream/EPEL. Document this so the omission (vs the 2.16/2.17 xcat-dep repos)
is not mistaken for a gap.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Per review of the gate design:
- Signature: fail (SIGKEY) if --gpg-key-name cannot be resolved to a fingerprint, so the
gate always confirms the repo was signed by EXACTLY the CLI key -- never a soft pass.
(EL already dies loudly on a duplicate rpm version via rpm_version; Ubuntu now matches.)
- Document the gate + its intentional idiosyncrasies (what 'the repo' is, duplicate=hard
error, signature identity) in BUILD.md.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Reviewed by @viniciusferrao. Each numbered point below is his; the code changes verify + fix it.
1. --build-number over-reach / dry-run / double-stamp
- The bump now runs ONLY on a real build: `--dry-run --build-number N` prints what it would
stamp and writes nothing (previously it rewrote every spec on disk during a dry run).
- Re-stamping is idempotent AND replacing: a re-run in a reused tree with a different
--build-number strips the prior .snap<ts>.<n> before applying the new one, instead of
accumulating a second stamp (…snap57.snap58). Extracted the per-line logic into the
unit-testable MockBuildUtils::restamp_release_line and covered it in t/mockbuild-all.t.
(The headline "rewrites xcat-core / xCAT-genesis-base.spec" does not occur in the real layout:
xcat-core is a sibling of $repo_root, and there is no genesis spec under the dep tree. The
legacy nested xcat-source-code case remains a non-CD layout; left as a follow-up.)
3. --finalize-xcat-dep idempotency
- cross_copy_genesis compared only SIGMD5 (content), which is blind to signature + index state.
It now also treats a same-content-but-UNSIGNED destination rpm as not-up-to-date (new
rpm_is_signed helper) so a crash between copy and sign heals on re-run.
- finalize_xcat_dep now re-indexes+signs BOTH repos of a touched pair every run, not only when
an rpm was copied, so a crash after copy+sign but before createrepo (rpm on disk, absent from
repomd) also heals.
4. Stale run-state can mask a failed build
- A real build now wipes its per-target $run_root first (run_id is derived from the deterministic
commit time, so re-runs reused the same tree). --skip-build keeps the tree; --dry-run writes nothing.
- mockbuild-perl-packages.pl clears each package's stale status.txt/error.txt BEFORE building, and
the aggregate now treats the child worker's exit code as authoritative: a package is PASS only if
its worker exited 0 AND wrote a PASS this run (a stale PASS in a reused log dir no longer counts).
5. --skip-build can publish the wrong artifacts
- --skip-build now REQUIRES an explicit --target (without it, all three EL targets collected the
same EL-agnostic roots and cross-published them).
- Collection is scoped to this target's own per-target $build_root (the same tree a normal build
populates), not the legacy build-output/list3/list5/list6 dirs.
- The manifest version-pin validation (and the "no manifest section" guard) now also run under
--skip-build, so a collection-only publish is validated exactly like a fresh build.
2. goconserver bypassed the CD bump (minimal fix; hermetic rebuild deferred)
- goconserver/mockbuild.pl gains --release-suffix, appended to its generated `Release: 4.elN`,
and mockbuild-all.pl passes the CD suffix down -- so goconserver's NVR advances per run like
every other dep package (an additive publish is no longer a silent no-op on a frozen NVR).
- Pinned the clone to an immutable upstream commit instead of the moving `master` (0.3.3 is
unreleased -- newest tag is v0.3.2 -- so a SHA pin is required; clone now fetches by ref).
- The host build + `go mod tidy` hermeticity concern is a tracked follow-up, not in this change.
7. Docs
- BUILD.md: --target is a single value, not repeatable; conserver-xcat is built for every target
(not "not required"). POD: --parallel-targets default is 1 = serial, not "auto".
- Added a manifest<->docs consistency test (conserver-xcat present in every target section).
(6, --max-parallel not a true global cap, is a documented nice-to-have and is left as a follow-up.)
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Review feedback (viniciusferrao):
1. --finalize-xcat-dep no longer succeeds with no genesis rpms and no longer treats
a shared filename as up to date when the content differs.
- finalize_xcat_dep now REQUIRES each arch's own genesis rpm for every repo pair
it processes (a pair with none is a hard FATAL, not a silent exit-0 no-op).
- cross_copy_genesis compares RPM identity by SIGMD5 (header+payload digest,
independent of the GPG signature), so a stale rpm that merely shares a basename
is refreshed instead of being mistaken for up to date.
2. Remove the genesis workaround. xcat-core #7696 is merged, so buildrpms.pl now
exits 0 iff it produced the genesis rpm; the zero-tolerance check no longer
ignores a genesis failure when a matching (possibly stale) rpm already exists --
any failed build step, genesis included, fails the run.
3. Skip modes work. required_pkgs() drops the packages whose builder was skipped, and
both the version-pin check and assert_required_deps use it, so a clean
--skip-genesis / --skip-xcat-dep / --skip-perl run no longer fails validating
packages it deliberately did not build.
4. Tests. The reusable, side-effect-free helpers are factored into MockBuildUtils.pm
(cross_copy_genesis and finalize_xcat_dep take injected sign/reindex callbacks so
they carry no gpg/createrepo state) and t/mockbuild-all.t adds focused fixture
tests for all of the above: skip-mode selection, version-pin globs, SIGMD5-based
RPM-identity comparison + cross_copy refresh/idempotency, and the finalize
require-inputs guard. Run with `prove t/`.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Until now mockbuild-all.pl built every dep package + every perl package + genesis
on every target, and TOLERATED build-step failures: a builder "expected to fail
on a given arch/el" (e.g. perl-Sys-Virt on el8) was warned and swept under the
rug, with correctness only re-checked after the fact by a hardcoded
assert_required_deps set. That hid real failures until the post-collection gate
and shipped packages a target does not need.
Replace that with an explicit, empirically-derived manifest. package-manifest.conf
has one [<target>] section per (EL, arch) listing <package>=<version|*>; each
target builds ONLY the packages listed for it. The sets were derived
authoritatively -- on a clean MN of each of the six targets, xcat.org LATEST
xcat-core + xcat-dep were configured, `dnf install xCAT` was run, and the
packages whose from_repo=xcat-dep were captured. That is exactly what xCAT pulls
from xcat-dep on that target. Results: conserver-xcat is required by no target
(goconserver supersedes it), and the variable perl modules differ per EL because
the OS/EPEL already provides the rest there.
Build failures are no longer tolerated: run_build_steps_parallel now returns the
failed step ids and build_one_target fails the whole run if any required package
failed. The one exception is xCAT-genesis-base -- xcat-core's buildrpms.pl exits
non-zero on an unrelated post-build xCAT-release-latest cp even when the genesis
rpm IS produced, so genesis is judged by rpm-produced, not exit code.
mockbuild-perl-packages.pl already honors --packages; the orchestrator now passes
each target's required perl subset so only those are built.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The EL build orchestrator never reclaimed the per-step mock buildroots it
created. Each build step makes a build chroot AND a per-uniqueext bootstrap
chroot under /var/lib/mock (dep packages, per-package perl chroots, and
xCAT-genesis-base); the child builders copy their RPMs/logs to their
--result-dir and exit without scrubbing, and mock's own cleanup leaves them
behind (and keeps them entirely on failure). So /var/lib/mock grew ~15-17G per
run, unbounded, until the build host filled to 99% and every mock dnf
transaction failed for lack of space ("Error: needs N MB more space on the /
filesystem", rc=30), cascading to rc=2/rc=255 across packages and killing
otherwise-healthy builds.
After the parallel build phase, scrub each step's buildroot with
"mock -r <chroot> --uniqueext <ext> --scrub=chroot --scrub=bootstrap" -- mock's
own lock-safe scrub: a chroot still held by a concurrent build is refused and
skipped (never rm, which would race a live build). Both the build chroot and its
per-uniqueext bootstrap are removed (each is per-uniqueext, so both leak); the
shared root cache under /var/cache/mock is kept so rebuilds stay fast. The
orchestrator scrubs the dep-package and genesis chroots (whose uniqueext it
assigns); mockbuild-perl-packages.pl scrubs each of its per-package chroots (it
derives its own per-package uniqueext). Scrubs are non-fatal -- a cleanup hiccup
never fails the build. --keep-buildroots preserves the buildroots for debugging.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Per review on xcat2/xcat-dep#62, narrow this PR to the EL matrix only
(rh8/rh9/rh10 x x86_64/ppc64le). The SUSE and Ubuntu work is reverted out of the
PR's net diff and will land in its own PR, so the reviewer's SUSE/Ubuntu points
(SuSE breakage, ubuntu20.04/focal in the default set, the xcat@megware.com key
default) are moot here -- those targets are no longer part of this change.
Reverted (net-zero vs master):
- SUSE target support in mockbuild-all.pl (opensuse-leap -> sles<N>).
- SUSE perl BuildRequires compat + elilo suse_version hunk.
- build-apt-repo.sh Ubuntu changes (focal + key default).
EL-relevant review fixes kept:
- Rename the finalize options to precise arch names: --x86-repo/--ppc-repo ->
--x86_64-repo/--ppc64le-repo (and matching vars/labels). "x86"/"ppc" was
ambiguous, especially since the genesis package is named -ppc64 via tarch yet
carries no big-endian code.
- Clarify the "tolerated build" comment (reviewer #2): builder failures are
tolerated only so one flaky builder cannot abort the others; correctness is
enforced by RESULT via assert_required_deps (a missing REQUIRED rpm still fails
the run), not by exit code. Toleration is load-bearing -- perl-Sys-Virt fails
on el8 by design, and genesis "fails" cosmetically while still producing its rpm.
- BUILD.md: drop all references to the removed --skip-xcat flag and the stale
"unified xCAT repository" framing (the core is built by the xcat-core
pipeline), and document --finalize-xcat-dep.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst