PR #62 review raised two build-side problems:
1. The child builders installed each freshly built RPM onto the build host
("dnf -y install" + a run smoke). Because mockbuild-all builds el8/el9/el10
on one host, this installs a foreign-EL RPM into the host RPM database and
corrupts it. The real install-and-run verification already happens in the CI
Test phase (cluster-test.pl boots a matching MN and installs xCAT + the
freshly built xcat-dep there), so the host-side smoke was redundant. Remove
the install/smoke step from every builder and the perl aggregate builder, and
drop the now-dead --skip-install flag (builders, mockbuild-all.pl, and the
pipeline invocations). The perl builder's --jobs 1 throttle existed only to
avoid host dnf-lock contention during that install, so it goes too (perl
packages build in parallel again). Also drop goconserver's now-unused run_rc.
2. build_mock_uniqueext truncated the run id by keeping the LAST 24 chars, which
for the 7-char "ppc64le" arch dropped the leading EL digit -- so
alma+epel-{8,9,10}-ppc64le collapsed to the same run part. goconserver
compiles every EL in the el10 chroot (build_cfg rewritten to -10-), so the
chroot name is identical across the three ELs and the uniqueext was the only
thing keeping their mock roots apart: with parallel targets the three ppc
goconserver builds raced in one root. Keep a readable leading token AND append
a short digest of the full id so distinct ids always yield distinct uniqueext.
Moved the helper into MockBuildUtils.pm and added fixtures (distinct per EL on
a long run id, both arches, determinism).
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #62 review flagged doc drift. The goconserver build was moved off a
committed vendor/ tree to modules fetched from the Go proxy and pinned by a
committed go.sum, but three comments still described a vendored tree (and one
called the compile "offline", which it is not -- mock networking is on and the
proxy is reachable). Align the comments with the actual no-vendor, go.sum-pinned
build. Comment-only; no behavior change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
goconserver always compiles in the el10 chroot for the arch (el8/el9 ship a Go
too old for 0.3.3), so mockbuild-all's post-build scrub -- which keys on the target
cfg (el8/el9) -- cannot reach it, and goconserver self-scrubs its el10 build chroot.
But it ran only 'mock --scrub=chroot', leaving the ~190 MiB bootstrap-image tree
'<cfg>-bootstrap-<uniqueext>' behind. One survived per target per run and piled up in
/var/lib/mock -- part of the disk leak that filled the x86 build host to 99% and
flaked a build (VersatusHPC/xcat-core#51). Add '--scrub=bootstrap' so goconserver
reclaims its whole chroot, matching mockbuild-all's scrub_buildroot.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Completes @viniciusferrao's concern #2. Previously goconserver was built on the HOST with a runtime
`go mod tidy` against a clone of mutable `master` -- non-reproducible and non-hermetic.
- Rewrite goconserver/mockbuild.pl to build the rpm INSIDE a mock chroot via an SRPM: %build compiles
in-chroot (BuildRequires: golang, GOTOOLCHAIN=local, CGO_ENABLED=0).
- Commit only the pinned module manifest goconserver/gomod/{go.mod,go.sum} (97 lines; go.mod carries
the kr/pty -> creack/pty replace). The in-chroot build downloads the modules from the Go proxy
(mock networking enabled) but is reproducible because go.sum integrity-checks every module -- no
`go mod tidy`, and no committed vendor tree.
- goconserver is a CGO-free static binary and el8/el9 chroots ship too old a Go for 0.3.3, so always
COMPILE in the el10 chroot for the arch; the Release still carries the target's dist tag (4.el<rel>),
so every EL repo gets an identical, portable static binary. Verified on the build host: statically
linked, no shared-lib deps, correct el<rel> tag while built in the el10 chroot.
Combined with the immutable-SHA pin + --release-suffix (40feffc), goconserver is now reproducible,
built in mock, and advances its NVR per CD run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Reviewed by @viniciusferrao. Each numbered point below is his; the code changes verify + fix it.
1. --build-number over-reach / dry-run / double-stamp
- The bump now runs ONLY on a real build: `--dry-run --build-number N` prints what it would
stamp and writes nothing (previously it rewrote every spec on disk during a dry run).
- Re-stamping is idempotent AND replacing: a re-run in a reused tree with a different
--build-number strips the prior .snap<ts>.<n> before applying the new one, instead of
accumulating a second stamp (…snap57.snap58). Extracted the per-line logic into the
unit-testable MockBuildUtils::restamp_release_line and covered it in t/mockbuild-all.t.
(The headline "rewrites xcat-core / xCAT-genesis-base.spec" does not occur in the real layout:
xcat-core is a sibling of $repo_root, and there is no genesis spec under the dep tree. The
legacy nested xcat-source-code case remains a non-CD layout; left as a follow-up.)
3. --finalize-xcat-dep idempotency
- cross_copy_genesis compared only SIGMD5 (content), which is blind to signature + index state.
It now also treats a same-content-but-UNSIGNED destination rpm as not-up-to-date (new
rpm_is_signed helper) so a crash between copy and sign heals on re-run.
- finalize_xcat_dep now re-indexes+signs BOTH repos of a touched pair every run, not only when
an rpm was copied, so a crash after copy+sign but before createrepo (rpm on disk, absent from
repomd) also heals.
4. Stale run-state can mask a failed build
- A real build now wipes its per-target $run_root first (run_id is derived from the deterministic
commit time, so re-runs reused the same tree). --skip-build keeps the tree; --dry-run writes nothing.
- mockbuild-perl-packages.pl clears each package's stale status.txt/error.txt BEFORE building, and
the aggregate now treats the child worker's exit code as authoritative: a package is PASS only if
its worker exited 0 AND wrote a PASS this run (a stale PASS in a reused log dir no longer counts).
5. --skip-build can publish the wrong artifacts
- --skip-build now REQUIRES an explicit --target (without it, all three EL targets collected the
same EL-agnostic roots and cross-published them).
- Collection is scoped to this target's own per-target $build_root (the same tree a normal build
populates), not the legacy build-output/list3/list5/list6 dirs.
- The manifest version-pin validation (and the "no manifest section" guard) now also run under
--skip-build, so a collection-only publish is validated exactly like a fresh build.
2. goconserver bypassed the CD bump (minimal fix; hermetic rebuild deferred)
- goconserver/mockbuild.pl gains --release-suffix, appended to its generated `Release: 4.elN`,
and mockbuild-all.pl passes the CD suffix down -- so goconserver's NVR advances per run like
every other dep package (an additive publish is no longer a silent no-op on a frozen NVR).
- Pinned the clone to an immutable upstream commit instead of the moving `master` (0.3.3 is
unreleased -- newest tag is v0.3.2 -- so a SHA pin is required; clone now fetches by ref).
- The host build + `go mod tidy` hermeticity concern is a tracked follow-up, not in this change.
7. Docs
- BUILD.md: --target is a single value, not repeatable; conserver-xcat is built for every target
(not "not required"). POD: --parallel-targets default is 1 = serial, not "auto".
- Added a manifest<->docs consistency test (conserver-xcat present in every target section).
(6, --max-parallel not a true global cap, is a documented nice-to-have and is left as a follow-up.)
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The goconserver binary parses /etc/goconserver/server.conf as YAML, but the
package shipped it in the old INI ("[server]\nhost = ...") format. The YAML
parser reads the leading [server] as a sequence, so the daemon panics at startup
("yaml: cannot unmarshal !!seq into common.ServerConfig"); systemd then
rate-limits the service to 'failed' before xCAT (Goconserver.pm) rewrites the
config as YAML. On a management node this leaves goconserver down, so the
provisioning test cases' makegocons cannot register a console and the case fails.
Ship a valid minimal YAML default matching the schema xCAT itself writes
(global/api/console; api port 12429, console port 12430, datadir); xCAT still
overwrites it with the cert-enabled config on the MN, and this default now only
has to parse + start. Bumps release el -> 4.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mockbuild-all.pl treated xnba-undi/grub2-xcat as x86-only @SHARED_NOARCH imported
onto ppc, so an x86 build failure rippled into a fatal "grub2-xcat missing" on
ppc; both actually build on any arch (noarch repackaging of committed artifacts)
and xCAT requires them on ppc too. Un-gate xnba-undi and drop the import so each
host builds a complete, self-sufficient dep repo. Add a single --output that
re-roots all NFS-shared output plus a fail-fast lock at <output>/.lock (owner-pid
guarded so forked build children do not delete it). Default gpg-key-name to the
real "xCAT Signing Key" and --xcat-source to ../xcat-core.
Make concurrent builds safe: give the genesis buildrpms.pl its own HOME/rpmbuild
tree (a shared /root/rpmbuild raced across parallel targets), and nest the
xnba/goconserver rpmbuild dirs under the run-scoped --work-dir (they hard-coded
/var/tmp/xcat-rpmbuild-* and wiped each other). Add --parallel-targets/--max-parallel,
defaulting to serial since the per-package scripts still share repo tarballs under
full parallelism.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Commit f6f8640 added the /var/lib/goconserver datadir so goconserver can create
its nodes.json store and makegocons can register consoles, but kept Release: 2.
The fixed build then shared an identical NVR (goconserver-0.3.3-2) with the
pre-fix build, so a stale pre-fix -2 could be (and was) served to management
nodes where makegocons still failed ("open /var/lib/goconserver/nodes.json: no
such file or directory"). Bump Release to 3 so the datadir fix is a distinct,
upgradeable version that cannot collide with the pre-fix -2.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
goconserver imports the abandoned kr/pty, whose pty.Start sets the controlling
tty in a way Go >= 1.15 rejects ("fork/exec ...: Setctty set but Ctty not valid
in child"). On current toolchains this breaks the console backend: rcons
connects but the session produces no output. Replace kr/pty with the
maintained, API-compatible creack/pty fork at build time.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The goconserver deb shipped no systemd unit, so enabling or starting the
goconserver service after installation had nothing to start (makegocons failed
with "Failed to start goconserver service"). Install the provided
goconserver.service unit during packaging.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Port SOURCE_DATE_EPOCH patterns from xcat-core to all xcat-dep build
scripts. RPM and Debian packages now produce identical output given
the same Gitepoch timestamp, regardless of build host or time.
EL (RPM) changes:
- mockbuild-all.pl: --build-timestamp flag, deterministic run_id,
tar --sort/--owner/--group/--mtime, createrepo --revision
- All mockbuild.pl: SOURCE_DATE_EPOCH cascade (CLI > Gitepoch > git > time),
deterministic mock config with SOURCE_DATE_EPOCH in chroot env,
RPM macros for timestamp clamping and fixed buildhost
- goconserver: -trimpath -buildvcs=false, canonical rpmbuild path
- xnba: canonical rpmbuild path
- syslinux: ZERO_AR_DATE=1 in mock env
Ubuntu (Debian) changes:
- All make_deb.sh: SOURCE_DATE_EPOCH fallback from Gitepoch
- goconserver: deterministic SNAP_TS and changelog from epoch,
-trimpath -buildvcs=false in debian/rules
- syslinux: -fdebug-prefix-map for path-independent debug info
- build-apt-repo.sh: gzip -n, Release Date: from SOURCE_DATE_EPOCH
Verified: 31/33 package artifacts produce identical SHA256 hashes
across independent builds. 2 syslinux sub-packages (devel,
debugsource) have known limitations from upstream build system.
Create debian/ directory and make_deb.sh for building goconserver
as a .deb package. Clones upstream, removes etcd backend, builds
with CGO_ENABLED=0 for static linking, packages via dpkg-buildpackage.
Tested on Ubuntu 22.04, 24.04, and 26.04.
Auto-detection used OS ID directly (e.g. almalinux+epel-10-ppc64le)
but mock configs use short forms (alma+epel-10-ppc64le). Add
resolve_mock_cfg helper that tries the full ID first, then falls
back to known short forms.
The smoke test previously required the IPMI probe to fail, which
breaks on any build host with a BMC. Allow rc=0 when hardware is
present; only fail on unexpected error output.
Clone goconserver from upstream master (v0.3.3), initialize Go
modules, and compile both server and client binaries with
CGO_ENABLED=0 for static linking. The etcd storage backend is
removed before building because its dependencies are incompatible
with modern Go modules; xCAT only uses file-based console storage.
v0.3.3 removes RC4 and 3DES ciphers from the TLS configuration.
Daniel Hilst
Vinícius Ferrão
GONG Jie