Master gained the EL10 riscv64 forcearch target (PR #66) and the Perl Ubuntu
build (PR #63). Both touch the files this branch rewrites, so the merge is
resolved per file:
mockbuild-all.pl, mockbuild-perl-packages.pl keep the forcearch target profile,
the noarch chroot and --epel-gap from master, and this branch's manifest gate,
atomic per-cell deploy and per-package chroot scrub. assert_required_deps is
dropped: verify_target_repo replaced it. The post-join bootstrap scrub now reads
the uniqueext and the config recorded when the chroot was made, because the wave
loop of master no longer numbers packages in @packages order.
packages-manifest.conf gains a [rocky-10-riscv64-xcat] section. A target with no
section is fatal, so without it the riscv64 target cannot run.
goconserver/mockbuild.pl builds in the mock chroot for the host arch and cross-
compiles on the host for a foreign --target-arch. A forcearch chroot would run
the Go toolchain under qemu. Both paths overlay the pinned go.mod/go.sum and
ship server.conf as YAML.
The host-install smoke stays removed (it corrupts the build host rpm database).
The checks that do not install on the host are kept: the chroot install of a
cross-built ipmitool-xcat and XS perl module, and the binfmt run of the cross-
built goconserver binaries.
goconserver/gomod/ takes the pin of master, whose `go` directive is the lower of
the two, so both the EL10 chroot and the Ubuntu sbuild toolchain accept it.
t/genesis_openembedded_consumer.t: the skip count of the RPM block is 64, the
number of tests it runs. Both sides carried a stale number.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #62 review raised two build-side problems:
1. The child builders installed each freshly built RPM onto the build host
("dnf -y install" + a run smoke). Because mockbuild-all builds el8/el9/el10
on one host, this installs a foreign-EL RPM into the host RPM database and
corrupts it. The real install-and-run verification already happens in the CI
Test phase (cluster-test.pl boots a matching MN and installs xCAT + the
freshly built xcat-dep there), so the host-side smoke was redundant. Remove
the install/smoke step from every builder and the perl aggregate builder, and
drop the now-dead --skip-install flag (builders, mockbuild-all.pl, and the
pipeline invocations). The perl builder's --jobs 1 throttle existed only to
avoid host dnf-lock contention during that install, so it goes too (perl
packages build in parallel again). Also drop goconserver's now-unused run_rc.
2. build_mock_uniqueext truncated the run id by keeping the LAST 24 chars, which
for the 7-char "ppc64le" arch dropped the leading EL digit -- so
alma+epel-{8,9,10}-ppc64le collapsed to the same run part. goconserver
compiles every EL in the el10 chroot (build_cfg rewritten to -10-), so the
chroot name is identical across the three ELs and the uniqueext was the only
thing keeping their mock roots apart: with parallel targets the three ppc
goconserver builds raced in one root. Keep a readable leading token AND append
a short digest of the full id so distinct ids always yield distinct uniqueext.
Moved the helper into MockBuildUtils.pm and added fixtures (distinct per EL on
a long run id, both arches, determinism).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Review follow-up for the Ubuntu sbuild matrix:
- Add a fail-fast exclusive flock over the whole run (<output-root>/.sbuild-all.lock,
file-scoped handle) so two overlapping runs can't corrupt the shared staging/apt
tree -- this is the root of the observed 'remove_tree .../staging/<cn>/<arch>:
Directory not empty' (an NFS silly-rename from a concurrent run).
- wipe_tree(): remove_tree that captures {error} and dies loud, so an ENOTEMPTY no
longer carps-and-continues leaving stale debs; used for all staging/pool/dists wipes.
- Wire the tested, hash-based cross_copy_genesis_deb into build_genesis (was a naive
glob+copy, so the unit-tested stale-dropping copier was dead code); remove the
genuinely-unused deb_snap_version/rewrite_changelog_top helpers + their subtests
(compiled deps intentionally ship their tracked changelog version).
- Dedupe assemble_apt on binary Package+Architecture (keep highest via
dpkg --compare-versions) so a double-produced genesis can't land two versions in
the pool, independent of the --skip-genesis contract.
- Derive Release Architectures from the arches actually staged (non-empty
binary-<arch>/Packages), not a hard-coded 'amd64 ppc64el'.
- goconserver: guard 'go mod init' when a go.mod exists (+ TODO to commit go.sum for
the pinned SHA). Accept-and-ignore the unused per-package --log-dir/--build-number/
--skip-install flags (documented). Remove orphaned make_deb.sh dispatchers
(build-debs-all, build.sh, ipmitool/build.sh) + update the READMEs.
perl -c clean; prove t/sbuild-all.t: 71/71.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The per-package <dep>/sbuild.pl were generic wrappers that shelled out to each
package's make_deb.sh, so a package's build brain was split across two files and
package-specific fixes (e.g. goconserver's Go toolchain) landed in make_deb.sh
instead of the builder -- inconsistent with the EL side, where <dep>/mockbuild.pl
IS the per-package builder.
Make each <dep>/sbuild.pl own its build (mirroring <dep>/mockbuild.pl), absorbing
its make_deb.sh (source prep, patches, toolchain, dpkg-buildpackage), and remove
all seven make_deb.sh. The common chroot orchestration -- ephemeral schroot session,
apt update, build-dep install, out-of-tree copy, SOURCE_DATE_EPOCH, deb collection +
host-side verification -- moves into BuildUtils::build_deb_in_chroot; each builder
supplies only its package-specific recipe (passed base64-encoded to avoid quoting
interplay through schroot).
Also make the older Ubuntu codenames buildable:
- goconserver: install a pinned modern Go (1.25.12) in the build -- focal/jammy ship
a Go too old to even auto-switch toolchains, and goconserver's pinned deps need
Go >= 1.25 (it is a static CGO-free binary, so the pinned toolchain is portable and
reproducible across codenames).
- ipmitool + goconserver: lower debian/compat 13 -> 12 and Build-Depends debhelper
(>= 12), since Ubuntu 20.04 (focal) ships debhelper 12; compat 12 also builds cleanly
on newer codenames.
Validated: all recipe shapes build green through the new path -- tarball
(ipmitool@focal), git-clone+pinned-Go (goconserver@focal), tarball+patches
(syslinux@noble), in-place (grub2-xcat), in-place+custom-rules (xnba); 80/80 unit
tests still pass.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Each <dep>/sbuild.pl copies the built .deb(s) to --result-dir from INSIDE the
schroot session. That only reaches the host when --result-dir is on a path
bind-mounted into the chroot (the shared /opt/xcat-ci-shared tree, as the CI
uses). A mis-configured result-dir (e.g. a chroot-local /tmp) would let the
build "succeed" yet leave nothing on the host -- a silent no-output. Verify
host-side after the session that the .deb(s) are actually present and fail
loud with a pointed message otherwise (consistent with the fail-hard design).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The Ubuntu/Debian dependency build shipped as three bash scripts
(build-dep-debs.sh, build-apt-repo.sh, mk-dep-chroots.sh) whose review
(PR #63) surfaced correctness problems: partial/stale output could be
published, the rpm->deb genesis conversion dropped the maintained package
semantics (Depends/Breaks/Replaces + maintainer scripts), the arch matrix
was invalid (x86-only syslinux/elilo/xnba treated as ppc64el packages, and
Architecture:all packages with no single producer), several required
failures exited zero, and the build/repo scripts disagreed on their staging
path and codename set (focal missing from the assembler).
Rewrite it as proper, unit-tested Perl mirroring the EL side
(mockbuild-all.pl / MockBuildUtils.pm / <dep>/mockbuild.pl / t/*.t /
packages-manifest.conf), sharing one CLI vocabulary:
- BuildUtils.pm: shared, testable helpers + the canonical CLI spec, plus the
Debian-specific helpers (out-of-tree changelog stamping, genesis control
preservation, deb inspection, cross-arch genesis provisioning).
- sbuild-all.pl: the orchestrator, absorbing all three shell scripts. Builds
+ validates into a fresh per-arch staging tree and only (re)assembles the
published apt repo from validated staging -- so partial/failed output never
ships and stale debs never accumulate. Auto-initializes the per-codename
sbuild chroots on first run. Fails the whole run non-zero on any missing
chroot/package/artifact or version-pin mismatch.
- <dep>/sbuild.pl x7: per-package builders that drive each package's
MAINTAINED debian/ in the matching chroot (never re-implemented), so the
converted/built packages keep their control metadata and maintainer scripts.
- debs-manifest.conf: per-[<codename>-<arch>] required set + version pins,
encoding the per-arch package sets (x86 boot components built once on amd64
as the single producer; ppc64el builds only the arch-specific compiled deps).
- t/sbuild-all.t: fixture tests for every pure helper.
- goconserver/make_deb.sh: pin the upstream SHA instead of cloning a moving
branch, so every matrix cell builds the same source (reproducible).
Codename set unified across build, assembly, chroots and docs (focal IS
supported). BUILD.md documents the new flow.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
elilo, ipmitool and syslinux build from a tracked, in-repo source tarball and no
longer fetch at build time, so the upstream download URL was undocumented. Add
it back as a provenance comment next to the tracked-source block so it is clear
where the tarball came from and where to re-download when bumping the version.
(xnba already records https://ipxe.org/ and goconserver keeps its git repo URL
in $go_repo; grub2-xcat is repackaged from the distribution grub2 and conserver
uses a dummy spec, so neither has a single upstream download URL.)
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The elilo, ipmitool and syslinux builders each fetched their upstream
source and rewrote the tracked source tarball in place, inside the
package source directory that both arch builds share. When the two
per-arch builds run in parallel they were racing to fetch the source:
one build truncated and rewrote the tarball while the other read it, so
the reader got a truncated archive and failed intermittently with
"missing top-level tree" errors.
The correct, normalized source is already tracked in the repository and
is what mock consumes, so the fetch is redundant as well as unsafe. Drop
the download/normalize entirely and verify the tracked source read-only
(it exists and has the expected top-level tree). With no writer, the
shared source is only ever read, so parallel per-arch builds can no
longer race on it. Also removes the now-dead --source-url /
--skip-upstream-download options, the normalize helper, and the wget
dependency check.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Commit the upstream source tarballs the mock builders consume so a build never
has to fetch them from the network -- offline/reproducible builds, and
resilience to upstream URL rot for these older releases:
- ipmitool/ipmitool-1.8.18.tar.gz (re-normalized to the release tarball)
- syslinux/syslinux-6.03.tar.xz
- perl-Crypt-SSLeay/Crypt-SSLeay-0.72.tar.gz
The per-package builders use the local tarball when present and only fall back
to the upstream URL if it is missing (ipmitool/syslinux mockbuild.pl; and
mockbuild-perl-packages.pl 'spec' mode for perl-Crypt-SSLeay).
Also folded in:
- mockbuild-all.pl: switch createrepo -> createrepo_c (--database,
--set-timestamp-to-revision) for deterministic, upstream-matching repo
metadata.
- perl-HTTP-Async / perl-Net-HTTPS-NB specs: reword the brp-compress comment.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The syslinux-xcat subpackage is BuildArch: noarch but deliberately bundles the
arch-dependent PXELINUX/SYSLINUX bootloader blobs under
/opt/xcat/share/xcat/netboot/syslinux. EL10 rpm does not flag these real-mode
binaries, but EL8/EL9 rpm aborts the build with "Arch dependent binaries in
noarch package", so syslinux-xcat 6.03 could not be rebuilt for rh8/rh9 and the
2.17 -> 2.18 upgrade had no syslinux-xcat >= 6.03-1 to satisfy xCAT 2.18.
Set %_binaries_in_noarch_packages_terminate_build 0 so the self-contained xcat
netboot payload builds across EL8/EL9/EL10.
Port SOURCE_DATE_EPOCH patterns from xcat-core to all xcat-dep build
scripts. RPM and Debian packages now produce identical output given
the same Gitepoch timestamp, regardless of build host or time.
EL (RPM) changes:
- mockbuild-all.pl: --build-timestamp flag, deterministic run_id,
tar --sort/--owner/--group/--mtime, createrepo --revision
- All mockbuild.pl: SOURCE_DATE_EPOCH cascade (CLI > Gitepoch > git > time),
deterministic mock config with SOURCE_DATE_EPOCH in chroot env,
RPM macros for timestamp clamping and fixed buildhost
- goconserver: -trimpath -buildvcs=false, canonical rpmbuild path
- xnba: canonical rpmbuild path
- syslinux: ZERO_AR_DATE=1 in mock env
Ubuntu (Debian) changes:
- All make_deb.sh: SOURCE_DATE_EPOCH fallback from Gitepoch
- goconserver: deterministic SNAP_TS and changelog from epoch,
-trimpath -buildvcs=false in debian/rules
- syslinux: -fdebug-prefix-map for path-independent debug info
- build-apt-repo.sh: gzip -n, Release Date: from SOURCE_DATE_EPOCH
Verified: 31/33 package artifacts produce identical SHA256 hashes
across independent builds. 2 syslinux sub-packages (devel,
debugsource) have known limitations from upstream build system.
Daniel Hilst
xq2005
hamzy
jbjohnso