The rebase onto master left the repository with two modules named
BuildUtils.pm: the shared build helpers at the root, package BuildUtils, and
the target architecture parser at build-utils/lib/XCAT/BuildUtils.pm, package
XCAT::BuildUtils. buildrpms.pl loaded both, one through `@INC` and one through a
path require. A reader cannot tell which module a BuildUtils reference names,
and the test sandbox staged the wrong one.
Move the shared helpers into build-utils/lib/XCAT/BuildUtils.pm as
XCAT::BuildUtils, and export targetarch_from_target beside them. Both builders
and the four tests now put build-utils/lib on `@INC` and import from the one
module. targetarch_from_target keeps its behaviour: it returns the same
architecture as before for suffixed targets, empty and undefined input, mixed
case and every architecture token.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The two builders wrote the same run-a-command-or-stop step in opposite
polarities. builddebs.pl used sh(...) == 0 or die; buildrpms.pl used
sh(...) and die, and also used the == 0 or die form for its sh_retry calls, so
both directions appeared in one file. The 'and die' spelling reads as though
the die is what happens next rather than what happens on failure, which is a
poor thing to have to re-read at every call site.
Add BuildUtils::sh_or_die and convert the fourteen plain sh() call sites to it.
The failure message now also carries the exit code, which every one of the old
spellings discarded -- a build that failed said only that a command failed, not
what it returned. The sh_retry sites keep their own form: retry is a different
operation and sh_retry is local to buildrpms.pl.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The two builders read and wrote files three different ways. buildrpms.pl used
File::Slurper, builddebs.pl opened and closed handles inline, and BuildUtils.pm
carried its own read_file/write_file pair to avoid the dependency. The result
was that the same operation was spelled differently in each place, and the
Version stamp was read by hand in both builders: buildrpms.pl chomped it ten
lines after reading it, which is how a stamp keeps a trailing newline nobody
notices until it reaches a package name.
Drop the hand-rolled pair and use File::Slurper throughout, and add
BuildUtils::read_line for the one-line Version and Release stamps -- it strips
the newline at the point of reading and reports an absent file as undef, which
is what builddebs.pl's fallback to snap_release needs.
File::Slurper is already required by buildrpms.pl and is already in the CI
package list, so this adds no new build-host prerequisite. The lock file still
uses a raw open: flock needs a real handle.
Behaviour note: Version is now read as its first line rather than the whole
file with the final newline removed. For the one-line file it is, these are the
same; a malformed multi-line Version now yields its first line instead of an
embedded newline.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both builders ship a mklocalrepo.sh beside the packages they publish, and each
wrote it its own way: builddebs.pl opened, printed and closed the file by hand
and then chmod'ed it, while buildrpms.pl used File::Slurper and a separate
chmod. Writing the text and setting the executable bit are one operation -- a
copy published without 0775 is published broken -- but nothing tied them
together, so each caller had to remember the second step.
Add BuildUtils::write_script, which writes the file and sets the mode, and use
it from both builders. The mode is now asserted in build_utils.t instead of
being left to the callers to repeat.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both builders wrote the same six fields beside a published repository --
VERSION, RELEASE, BUILD_TIME, BUILD_MACHINE, COMMIT_ID, COMMIT_ID_LONG -- each
assembling them by hand, and each shelling out to hostname to do it.
builddebs.pl also ran git rev-parse a second time to get the commit, although
it had already derived the same value into $GITINFO earlier in the file. It now
uses that.
The filenames and the time formats stay as they were. deploy.sh copies
builddebs.pl's buildinfo verbatim and cluster-test.pl parses buildrpms.pl's
buildinfo.txt, so both are a contract with consumers outside this repository;
the shared helper takes the format from its caller rather than picking one.
Confirmed byte for byte that each builder's stamp is unchanged.
Covered by tests: the field names and their order, the seven-character short
commit, and that the two callers' formats still differ.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Both builders defined their own sh() and usage(). The usage() copies differed
only in spelling, but the sh() copies disagreed about what they return:
buildrpms.pl shifted system()'s result into an exit code, builddebs.pl returned
the raw wait status. The same command therefore reported 1 from one builder and
256 from the other, and a caller comparing sh() against a particular code was
correct in only one of them. Both current call sites only test for zero, so
nothing was broken yet.
BuildUtils.pm now provides both, shifting the status as buildrpms.pl did.
pod2usage reads the POD of the running program, so each builder keeps its own
help text while sharing how it is printed and the status it exits with.
sh() echoed the command under the builder's own --verbose flag, which a shared
function cannot see. $BuildUtils::VERBOSE is set once after option parsing
instead of threading the flag through every call site.
Covered by tests: that sh() returns the exit code rather than the wait status
it is packed into, and that a verbose run still echoes the command.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl derived the git revision and SOURCE_DATE_EPOCH with its own inline
backticks while BuildUtils.pm already provided git_revision() and
source_date_epoch() for builddebs.pl. Both did the same thing, and the copies
in buildrpms.pl were the weaker ones: neither fell back to the tracked Gitinfo
and Gitepoch files, so a build from a source tarball with no git history
recorded the revision as "unknown" and stamped the current time as the build
epoch, losing reproducibility exactly where it matters most.
Verified that the helper returns the same revision and epoch as the code it
replaces on a checkout with history.
buildrpms.pl now loads BuildUtils.pm from its own directory, so the source-only
test stages the module into its sandbox alongside Version. That test copies the
builder rather than running it in the checkout, because it rewrites Gitinfo and
creates $HOME/rpmbuild before it looks at its arguments.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl already produces a source rpm for every package on every run --
buildall() is createmockconfig -> buildsources -> buildspkgs (mock --buildsrpm)
-> buildpkgs (mock --rebuild). Source-only is that sequence without the last
step, so it belongs here rather than in a parallel implementation: the spec, the
staged sources and the mock root are identical either way, and anything built
beside them can drift from what a real build does.
--source-only stops after buildspkgs. Two things downstream had to learn about
it, and both are about not publishing something untrue:
- index_repo no longer re-indexes the binary directory. Running createrepo_c
over a directory with no binaries in it would replace working metadata with
metadata for an empty repository -- a repo that resolves nothing. The srpm
index is still regenerated.
- write_repo_metadata_dir emits nothing. The .repo file and buildinfo describe
an installable binary repository, which this mode does not produce.
--source-only with --merge-core-repos is refused: one builds packages, the other
assembles per-arch trees that are already built.
buildrpms.pl cannot be loaded by a test -- it runs mkdir, git and read_text at
file scope -- so the two routines whose behaviour changed are lifted out with a
regex and eval'd into a scratch package with their collaborators stubbed, per
the code standard, with BAIL_OUT if the extraction stops matching. The CLI
contract is exercised by running the real program. Verified by mutation:
removing the index_repo guard reddens 2 of the 7 assertions.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
targetarch_from_target took the last dash-separated part of the mock
target name as the architecture. The riscv64 forcearch configuration is
named rocky-10-riscv64-xcat, so riscv64 builds looked for
xCAT-genesis-scripts-xcat srpms and failed. Take the last part that
names an architecture and only fall back to the last part when none
does; stock targets are unaffected.
Add riscv64 to the xCAT and xCATsn build paths, keep the unavailable legacy Genesis dependency conditional, collect the architecture RPMs, and document the forcearch mock target.
The parallel-build failure gate only inspected $exit_code in run_on_finish. A
ForkManager child killed by a signal -- SIGKILL, or the OOM-killer under the
concurrent build load -- is reaped with $exit_code == 0 but $exit_signal != 0
(and possibly $core_dump). Such a worker therefore was NOT recorded as a
failure, so the parent could still index and GPG-sign a repository that is
missing the package that worker was building -- the exact partial-repo hazard
the gate was added to prevent, via a path it did not cover.
Capture $exit_signal and $core_dump from the run_on_finish callback and fail
the build when any of $exit_code, $exit_signal, or $core_dump is set.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Addresses three concurrency-safety defects raised in review of the parallel
buildrpms.pl work:
1. Per-target build lock was released immediately. The flock filehandle was a
lexical (my $blk) scoped to the guard block, so it was destroyed -- and the
lock dropped -- as soon as that block exited, before any worker forked. The
"intentionally leaked" comment did not match the code. Hold the handle in a
file-scoped $BUILD_LOCK_FH so the fd (and the lock) live for the whole
process; forked children inherit the fd but their exits never release it.
2. Build failures were silently swallowed. buildspkgs()/buildpkgs() called
sh_retry() in void context, so a mock build that failed all retries returned
non-zero into the void; the child then exited 0 and the parent's run_on_finish
ignored the exit code. The parent could therefore index and GPG-sign a repo
that was missing packages and still exit 0. Now sh_retry failures die in the
child, run_on_finish records any non-zero child, and the run aborts before
update_repo and again before signing if any child failed -- never publishing
a partial core.
3. Abort cleanup unmounted unrelated builds. sweep_mock_mounts() lazy-unmounted
every bind under /var/lib/mock, which on a shared host tears out the live
chroots of concurrent, unrelated builds. Scope it to this run's own chroots
(each chroot dir plus its -bootstrap sibling), passed in from abort_builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
merge_core_repos already assembles the multi-arch core and runs the full
finalize tail (index -> sign -> final metadata -> write_release_alias, in the
required order so the xCAT-release-latest alias lands AFTER metadata and stays
out of the repo index). The preceding commit restored a separate finalize_core
sub purely to satisfy xcat_release_package.t, which grepped for
'sub finalize_core { ... $dir ... }'. That left two overlapping entry points:
--finalize-core (finalize one pre-assembled dir) and --merge-core-repos
(assemble N per-arch dirs THEN finalize) -- the former a strict subset of the
latter, with no in-tree or CI caller.
Consolidate on the single assemble+finalize path: inline the finalize tail into
merge_core_repos, drop sub finalize_core and the --finalize-core getopt/dispatch,
and update xcat_release_package.t to assert the stable alias is created after the
final metadata pass inside merge_core_repos ($out). Behaviour is unchanged (merge
already wrote the alias); only the redundant finalize_core interface is removed.
xcat_release_package.t 26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The CI buildrpms.pl port (d2d98b724) renamed the upstream sub finalize_core($dir)
to merge_core_repos($out) and inlined its index/sign/metadata/alias tail. The
xCAT-release logic (write the xCAT-release-latest stable bootstrap alias AFTER the
final metadata pass, so it stays out of the repo index) was preserved, but the rename
broke xCAT-test/unit/xcat_release_package.t, which statically pins:
sub finalize_core { ... write_repo_metadata_dir($dir); ... write_release_alias($dir); }
so 'assembled core repository creates the stable alias after final metadata' failed
(1 of 26), turning xcat_pr_test red on PR #7701.
Restore finalize_core($dir) as the shared finalize primitive (index -> sign ->
final metadata -> stable alias) and have merge_core_repos delegate to it on the
assembled multi-arch dir. Re-expose the --finalize-core CLI (single already-populated
dir) it also feeds. Behaviour is unchanged for both --merge-core-repos (CI) and
--finalize-core; only the shared code path is named again. xcat_release_package.t now
26/26; perl -c clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl built every package with mock -N (--no-clean), reusing the per-package
<pkg>-<target><ext> chroot across runs for flat disk. But a build aborted or killed
mid-flight leaves that chroot half-initialised with a corrupt rpmdb; the NEXT run
reused it and failed (cannot open Packages database .../usr/lib/sysimage/rpm),
producing an incomplete core (e.g. missing xCAT-test) that fails the deploy-time
completeness gate.
Re-init the buildroot (mock --init) right before building each package, after the
diskcache skip so it only runs when actually building. --init restores from mock's
root-cache tarball (cheap) so disk stays flat and builds stay fast; -N is kept on the
srpm/binary calls so they still reuse the freshly initialised root within the run.
This stabilises builds against corrupt state left by any previous failed/aborted build.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 3 EL CD pipelines (xcat-core-devel-cd, xcat-core-stable-cd, xcat-dep-el-cd)
overlaid a pinned $CI/buildrpms.pl at build time because the tree's buildrpms.pl
lacked the options they depend on:
- a per-target flock guard alongside --mock-uniqueext, so concurrent same-target
builds do not corrupt each other's /var/lib/mock chroot namespace;
- --native-only (build only arch-native pkgs on the secondary arch) plus
--merge-core-repos/--output-dir/--input-core-repos, replacing --finalize-core,
to assemble one signed flat multi-arch core from per-arch build outputs;
- sh_retry() to absorb transient mock/nspawn flakes;
- a single --target guard and graceful mock cancellation
(sweep_mock_mounts/abort_builds) that unmounts chroots on abort.
The xCAT-release repository package (master/2.19 only) is preserved: its
write_release_alias() is invoked from the new merge_core_repos() and per-target.
Porting them in-tree lets CI drop the $CI/buildrpms.pl pin and run the three
pipelines in parallel without the cross-job serialize lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
write_release_alias() finds the release rpm with
glob("$repodir/xCAT-release-$VERSION-$RELEASE.noarch.rpm"). That pattern has
no wildcard, and glob() returns a wildcard-free pattern verbatim even when the
file does not exist -- so @release_rpms == 1 is true and cp runs against a
nonexistent file. A partial build that does not produce xCAT-release (e.g.
buildrpms.pl --package xCAT-genesis-base, the way the xcat-dep pipeline builds
the OS-dependent genesis image) then dies with:
Can't cp('dist/alma+epel-8-x86_64/rpms/xCAT-release-2.19.0-snap202607261133.noarch.rpm',
'dist/alma+epel-8-x86_64/rpms/xCAT-release-latest.noarch.rpm'):
No such file or directory at .../buildrpms.pl line 659
Guard the alias write on the rpm actually existing (grep { -f }), and die with
the real error if the cp itself fails. A build that does not produce
xCAT-release now simply skips the alias instead of failing.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcatprobe subcommands (xcatmn, discovery, osdeploy) and hierarchy.pm load
xCAT:: perl modules but only add $XCATROOT/probe/lib/perl to @INC, not the
main $XCATROOT/lib/perl where perl-xCAT ships them. The RPM builder stopped
embedding those helpers, so /opt/xcat/probe/lib/perl/xCAT/ was absent on
installed systems and the subcommands were reported "Unsupported sub command".
Stage GlobalDef.pm, NetworkUtils.pm and ServiceNodeUtils.pm into a temporary
xCAT-probe tree and build its source tarball once, before the parallel target
builds fork, so workers only read a complete archive. Add package-payload
regression tests. Adopts the approach from upstream PR #7581.
The remaining buildrpms.pl changes are deployment-compliance refactoring
(signed, self-contained core repo output).
Closes#7579
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl stages source tarballs into $HOME/rpmbuild/SOURCES but only runs
rpmdev-setuptree in its one-time env-setup path. On a host where that never ran
(or $HOME/rpmbuild was cleaned) source staging fails with "SOURCES/...: No such
file or directory", no srpms/rpms are produced, and the run still exits 0 -- so
the deployable repo silently ends up empty. Create the rpmbuild tree up front so
a build no longer depends on prior manual setup.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl now writes xcat-core.repo, mklocalrepo.sh and buildinfo.txt into
dist/<target>/rpms (templates ported from buildcore.sh), so the built tree is
directly deployable to xcat.org and cluster-test.pl no longer needs to collect
and re-createrepo the dist output.
The full package set is built on every arch (x86_64 and ppc64le alike), so each
arch produces a complete, self-contained xcat-core repo -- no need to copy the
noarch packages from the x86_64 build into the ppc repo. --package now replaces
the default set (so --package xCAT-genesis-base builds only genesis-base for the
dep pipeline). xCATsn added to \@native_pkgs so its arch rpm is located correctly.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The single flat xcat-core repo serves el8/el9/el10 from one build, but two spec
decisions were resolved at BUILD time and so were wrong for the other ELs that
share the same repo.
1. DHCP backend. xCAT.spec and xCATsn.spec selected the dhcp provider with a
build-time %if (rhel >= 10 -> kea, else dhcpd), so an el10 build wrongly
required kea on el8/el9 (and an el8/el9 build wrongly required dhcpd on
el10). Replace it with an install-time RPM rich dependency:
Requires: (kea if (system-release >= 10) else /usr/sbin/dhcpd)
Requires: (kea-hooks if (system-release >= 10))
dnf now resolves it per client: kea on el10+ (which dropped ISC dhcp from the
distro), dhcpd on el8/el9. SLES has no "system-release" provide, so the
condition is false there and it falls back to /usr/sbin/dhcpd, preserving
prior behavior. system-release is versioned per release package
(el10=10.x, el9=9.x, el8=8.x).
2. openssl. Make the xCAT-server openssl Requires uniform across EL (non-SUSE)
instead of el10-only, so the produced package set does not depend on which EL
the build host happened to be.
Also drop xCAT-genesis-base from the default @PACKAGES set in buildrpms.pl: its
initramfs bundles the build-chroot kernel/glibc and is therefore OS- and
arch-dependent, so it cannot ship in the single flat core. It is now built per
target by the xcat-dep pipeline (mockbuild-all.pl, via
`buildrpms.pl --package xCAT-genesis-base`) and ships in xcat-dep/rh<N>. The
explicit `--package xCAT-genesis-base` build path is retained.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
This fixes a problem where the dracut image was pulling artifacts from
the build host and not the mock image, e.g. building for EL9 in EL10.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
* Motivation: Before this commit we hardcoded "rhel" in the mock chroot
names, but these depend running over RHEL host. After this commit the
distribution name is read from /etc/os-release ID variable. So it
should work on any EL clone now.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
Also add --regennginxconfig to reset nginx config, and
ensure nginx configuration is generated for all targets
not only the ones provided in the command line. This avoid
the anoying effect of overriding the repositories of previous
builds.
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
Add buildrpms.pl to build RPMs in parallel using mock
Add xCAT-buildkit to the build list
Fix build dependency in xCAT-buildkit.spec
Add fallback in /etc/init.d/xcatd for /etc/rc.d/init.d/functions
Signed-off-by: Daniel Hilst Selli <392820+dhilst@users.noreply.github.com>
Vinícius Ferrão
Daniel Hilst