Parallel::ForkManager does not fork when max_procs is 0: start()
returns 0 in the original process. start_build_worker took that 0 as a
forked worker. It closed the only writer of the gate pipe, read
end-of-file and exited 127, so both worker loops in buildrpms.pl
stopped before they built a package or updated a repository.
start_build_worker now compares $$ before and after the fork callback.
When the callback returned 0 in the same process, it closes the gate and
returns 0, and the caller runs the work under the locks it already
holds. Both loops call start_build_worker, so both are covered.
build_worker_nofork.t passes with this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
start_build_worker writes one byte to the gate pipe to release a worker
after the build lock records its group. When the worker dies between the
record and that write, the pipe has no reader and the write raises
SIGPIPE. The default disposition ends the builder at once, so its
cleanup never runs: the Debian package tree is not restored and the lock
is not released.
Ignore SIGPIPE for that one write. The write then fails with EPIPE and
start_build_worker reports the worker as not released, which is the
existing failure path. build_worker_gate_sigpipe.t passes with this
change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcat_pr_test fails in release_repository.t with "retained for surviving
build workers in process group 1". The schedule stub in that test
returns 1 from Parallel::ForkManager::start. start_build_worker()
accepted ESRCH from setpgid() and recorded group 1 on the build lock.
Group 1 is always alive, so the next buildrpms.pl run in the same lock
directory is refused.
ESRCH means the pid is not a child of this process. The gated child, if
there is one, has not run, so start_build_worker() now records nothing
and reports failure. It also no longer writes to a gate that no child
reads.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is recovered from a dead owner when no recorded process
group is alive. sh() let the command exec before the parent recorded
its group, and attach_build_workers() ignored a failed record, so a
parent killed in that window left a running command on a lock that the
next build recovers. buildrpms.pl kept its Perl builders only in
%MOCK_INFLIGHT, so between two sh() calls a builder had no record and
a killed parent freed the lock under it.
start_build_worker() forks the worker into its own process group and
holds it on a pipe. The parent records the group on every held lock
and only then writes to the pipe. A failed record, or a parent that
dies first, makes the worker exit with status 127 before it runs.
sh() and both ForkManager loops in buildrpms.pl start their workers
this way, and run_on_finish detaches a builder once it is reaped.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The one conflict is the BuildUtils import list: master gained package_skip_reason
with xcat-core/#7978 and this branch adds acquire_lock_dir. Resolved as the union
of the two, keeping the newer POSIX comment -- this branch's abort_builds() is
what needs WNOHANG.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Two conflicts, both from xcat-core/#7978 landing on master.
build-utils/lib/XCAT/BuildUtils.pm: both branches added a package_skip_reason,
#7978 with a general @PACKAGE_SKIPS table and this branch with its own
%EL_ONLY_PACKAGES and a second sub. git merges them as separate hunks because
they sit apart in the file, and Perl takes the LAST definition, so keeping both
silently deletes whichever rule came first. There is now one table and one sub.
A rule names either the target prefixes that do NOT build the package, or, with
`only`, the pattern of the targets that DO. The second form is what an EL-only
package needs: naming the families that are not EL means a family added later
tries to build it, which is how openSUSE reached xCAT-selinux and failed
xcat-core-build-opensuse #42 on a missing selinux-policy-devel.
buildrpms.pl: an empty build plan takes upstream's answer, a clean return rather
than this branch's die. That is the decision #7978 exists for -- the dep
pipeline asks for xCAT-genesis-base alone and no openSUSE target builds it -- so
reinstating the die would fail a build that has nothing to do.
selinux_el_only_scope.t is 37 of 37 and opensuse_no_legacy_genesis.t 22 of 22 on
the merged tree, and the whole unit suite passes: 301 files, 9801 tests.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcat-core-build-opensuse #42 failed on three packages whose only error was
"No matching package to install: selinux-policy-devel", and took the whole
openSUSE family of xcat-ci #146 down with it. openSUSE Leap ships no
selinux-policy-devel, so the policy module has nothing to compile against.
package_skip_reason in build-utils/lib/XCAT/BuildUtils.pm tested the target
against /\Aopeneuler-/, because openEuler was the one non-EL family buildrpms.pl
targeted when the guard was written. openSUSE was enabled afterwards and matches
no arm of that test, so the EL-only package was selected for it.
The test is now on EL rather than on the families that are not EL, so a family
added later skips an EL-only package instead of trying to build it.
selinux_el_only_scope.t gains the two openSUSE targets. Without this change its
two new assertions fail and the other 35 pass.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Resyncs build-utils/lib/XCAT/NFSLock.pm with xcat-dep, where the defect was
found and fixed.
release deleted the owner metadata and marked the guard released before it
knew the removal could finish. A build lock with a live worker group
therefore lost its metadata when the rmdir failed, and every later build
refused it for ever rather than recovering it when the previous build died.
The sweep had the same ordering and left the same wreck.
Both clear the records of completed workers first and delete the metadata
only once the directory holds nothing else. release retries while a worker
blocks the removal, so cancel_build's retention finishes the release when
the group finally goes.
Depends on xcat-dep/#93, which carries the same change to the original.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcat-dep-build-opensuse15-x86_64 fails its xCAT-genesis-base step. BuildRequires
resolve, and verify-genesis-payload then reports 566 missing zoneinfo/posix paths,
etc/redhat-release, and 45 EL paths and tool names -- net-tools-deprecated,
kmod-compat, iputils, systemd-sysvinit, timezone, nc, mkdosfs and the lvm symlinks.
The manifest is written for EL, so the package cannot be built on openSUSE.
openSUSE ships no legacy Genesis, so it does not need to be. package_skip_reason names
the pairs a target does not build and why, buildrpms.pl prints each skipped pair, and a
plan that skips everything returns 0 rather than failing: the dep pipeline asks for
xCAT-genesis-base alone, so on openSUSE its whole plan is empty and an empty plan is
the plan.
Dropping the package alone would make xCAT unresolvable there.
xCAT-genesis-scripts.spec requires xCAT-genesis-base-%{tarch} unconditionally and xCAT
requires xCAT-genesis-scripts, so zypper install xCAT would fail on a missing package
-- later than the build and looking like something else. That Requires now stands
behind %if ! 0%{?suse_version}; the Leap chroot's /usr/lib/rpm/suse/macros sets
%suse_version 1500, and no EL or openEuler target defines it.
xCAT-test/unit/opensuse_no_legacy_genesis.t covers the skip, the empty plan and the
conditional Requires.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
BuildUtils::sh() runs each build command in its own process group, so
SIGKILL on the wrapper leaves that group running. The lock records the
wrapper, so the next build proved the wrapper dead and took the lock, and
two builds then wrote one checkout and one mock target. cancel_build keeps
the lock on purpose when a group survives its cleanup, and that recovery
undid the retention.
sh() now records the group on every lock this process holds, before it
restores the signal mask, so a cancellation cannot land between the fork
and the record. It drops the record after the group is reaped.
cancel_build drops it only when kill(0, -pgid) answers ESRCH, which is the
test it already used to decide it may release; on the retention path the
record stays, and a later acquire reads it and refuses the lock.
The vendored XCAT::NFSLock carries the rule: an owner is recoverable only
when it is provably dead and every attached group is gone. detach refuses
while any process is in the group, so the retention does not depend on
which call site remembered to run.
Depends on xcat-dep/#93, which carries the same change to the original.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
BuildUtils::sh() runs each build command in its own process group, so
SIGKILL on the wrapper leaves that group running. A lock recovered from the
dead wrapper then lets a second build into the same checkout, and
cancel_build's retention is undone by the same recovery: it keeps the lock
when workers survive, and the next build reads a dead owner and takes it.
build_lock_stale_owner.t covers a wrapper and nothing below it. The name
reads wider than the check: the holder it kills has no build worker, so the
worker case was never measured. Its comment says so now.
A new check is the reviewer's case. A worker runs in its own process group,
a wrapper takes the build lock and records it, the wrapper alone is killed,
and a second build must stay blocked. Then the worker exits and the same
build must take the lock: without that half, a lock that recovers nothing
would pass.
The handshake is two one-byte reads on separate pipes, each bounded. One
read of two bytes made it a timing test, which passed on this host and
killed the wrapper before it held the lock.
The vendored XCAT::NFSLock gains attach, detach and live_workers, which
these checks drive.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Resyncs build-utils/lib/XCAT/NFSLock.pm with xcat-dep, where the defect was
found and fixed. The module reclaimed a stranded lock.borrow by path after
validating its metadata, and rmdir names a path and not the inode that was
validated, so a contender stopped between the two steps removed whatever
was at the path when it resumed. Two builds of one target could then both
hold the build lock, which is the collision the lock exists to prevent.
The borrow is numbered now, and a transition whose taker is provably dead
is ended by creating the next generation rather than by removing the one
that was validated. A process removes only a generation it created and
still holds.
acquire_lock_dir's comment said the retry budget covers reclaiming a borrow
with no metadata over two attempts. That rule is withdrawn, because a live
taker can pause for any length of time before its first write. A take-over
needs one attempt; the budget leaves two for a contended lock.
Depends on xcat-dep/#93, which carries the same change to the original.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT-selinux is in the default package set of buildrpms.pl, so every target
builds it. openEuler 20.03sp4 and 22.03sp4 carry selinux-policy 3.14.2, which
defines none of unconfined_server_stream_connectto(), files_map_etc_files() or
chronyd_manage_pid_files(). m4 leaves an undefined interface name in its
output and checkmodule stops with a syntax error, so rpmbuild fails and the
core build for that target publishes no repository at all.
SELinux support is EL only. XCAT::BuildUtils::package_skip_reason names the
packages an openEuler target does not build, and buildrpms.pl prints each pair
it skips. openEuler 24.03sp4 carries selinux-policy 40.7 and does compile the
module. It is excluded as well, so no openEuler release ships a package the
other releases cannot build.
xCAT-server and xCATsn required xCAT-selinux when selinux-policy-targeted was
installed, on openEuler too. openEuler defines %openEuler and no %rhel, so the
condition now names EL 8 and later and Fedora only. Without this an openEuler
management node cannot resolve xCAT-server.
xCAT-test/unit/selinux_el_only_scope.t fails on the parent commit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
acquire_lock_dir dies with "another build of <what> already holds
<lockdir>" whenever XCAT::NFSLock->acquire fails, whatever the failure was.
A lockdir under a directory that does not exist, or one the build cannot
write, both tell the operator to look for a build that is not running, and
both print the advice about --mock-uniqueext. The nested text said "Trying
to unlock", which is the reverse of what acquire was doing.
acquire_lock_dir now reports contention only when the retry budget ran out,
and prints the error that happened otherwise. The vendored
XCAT::NFSLock is resynced with xcat-dep: its refusal names which failure it
was, and it reclaims a lock.borrow that has no metadata and that two
attempts find unchanged, which was a permanent strand. A lock.d with no
metadata stays refused, so a directory left by a build from before this
change is not recovered by acquire; sweep_stranded removes that one, and
only under an exclusive claim on the root.
buildrpms.pl kept "use POSIX (); # EEXIST, for the directory build lock",
and the lock has not used EEXIST since it became an XCAT::NFSLock. The
import is still needed for WNOHANG in abort_builds.
Depends on xcat-dep/#93, which carries the same change to the original
copy of XCAT/NFSLock.pm.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
SIGKILL and the OOM-killer run no handler, so a build killed that way leaves its
lock directory behind. Every later build of the same target then fails at mkdir
until an operator removes the directory by hand.
/var/lock/buildrpms.openeuler-20.03sp4-x86_64.lock.d on one build host was
created on 2026-10-14, held no owner file and no live process, and still failed
that target's dep build on 2026-10-21, one second into the genesis step. Every
other openEuler job in that run reported NOT_BUILT.
buildrpms.pl held its own lock, a bare mkdir that recorded no owner, and `use
autodie` wrapped that mkdir, so the EEXIST branch below it was dead code and the
run died on autodie's raw "Can't mkdir(...): File exists". That is why it never
refused anything and never named the build holding the lock.
XCAT::BuildUtils::take_build_lock kept a second hand-rolled lock with the same
gap.
Both now take the lock through XCAT::NFSLock, whose protocol recovers a lock
whose owner is provably dead. The lock stays in /var/lock, which is local,
because what it guards is host-local -- /etc/mock/<cfg>.cfg, /var/lib/mock/<cfg>/
and its buildroot.lock -- so a lock on the shared checkout would serialise the
per-arch builds of one checkout that run on two hosts and share nothing.
Depends on the matching xcat-dep change: that copy of NFSLock carries the
recovery of a borrow whose taker died, and a DESTROY, without which a guard
never releases at scope exit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is a directory created with mkdir, which cannot release itself. A
build killed with SIGKILL or by the OOM-killer leaves it behind, and every later
build of the same target fails until an operator removes it by hand. No test
covered that case.
These checks take the lock in a child, confirm a second build is refused by name
while the child lives, kill the child with SIGKILL, and assert the lock is
recovered. One more holds the boundary that recovery must not widen what a lock
excludes: a build of a different checkout still runs beside it.
XCAT::NFSLock arrives with them, vendored at build-utils/lib/XCAT/NFSLock.pm
beside BuildUtils.pm, because xcat-core builds with no xcat-dep checkout. Its
header names the other two copies and records that the three must stay
identical. Nothing calls it yet, so these checks fail here.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT-test/unit/fips_policy_sources.t ran build-utils/sync-fips-policy as a
subprocess, so a file in the unit directory measured a program rather than a
routine, and it asserted on the diagnostics the script printed instead of on an
answer. buildrpms.pl and builddebs.pl started the same script with system() to
check the generated libraries before packaging xCAT or xCAT-genesis-scripts.
XCAT::BuildUtils gains fips_policy_artifacts, fips_policy_plan and
sync_fips_policy. fips_policy_plan reads the canonical policy and both runtime
libraries and reports what generation would write, without writing.
sync_fips_policy applies that plan in write mode and returns the stale paths,
the paths it rewrote and the diagnostics for its caller to print. The two
builders now call sync_fips_policy in their own process and print the
diagnostics before they stop, which keeps the output xCAT-test/native/
fips_build_guard.t asserts on. build-utils/sync-fips-policy keeps its
--check and --write contract and its exit status, and is now a thin caller.
The unit test calls the sub in-process and asserts on the returned hash. It
also covers a library holding two policy blocks and a library replaced by a
symbolic link, which the subprocess form did not reach. Ten mutants of the
module, one per guard, each fail the assertion that names that guard.
xCAT-test/bats/fips_policy_artifacts.bats copies XCAT::BuildUtils into its
fixture, because the script it runs now loads the module.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock_build_owner ran mock's loader as root. mock itself loads its
configuration as the user that sudo names, so with no chrootuid it
builds as SUDO_UID. The helper reported uid 0, and
prepare_mock_resultdirs then left root-owned logs in place.
As root, the loader now builds the uid manager with mock's own
setup_uid_manager() and loads the configuration through
run_in_subprocess_without_privileges(), as mock main() does.
prepare_mock_resultdirs gives mode 0755 to the parent directories that
it creates, because make_path applies the umask. Directories that
already exist keep their mode.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildrpms.pl staged each build's sources under $HOME/rpmbuild/sources.
mock now builds as the chrootuid of the target, uid 1000 for openEuler
24.03. On the Jenkins agents HOME is /root, mode 0550, so mock --sources
failed with "Permission denied" on the staging directory.
The staging directories now go under dist/sources in the checkout, next
to the mock result directories. The per-process names, the delete on
reap and the sweep of dead pids stay the same. buildrpms.pl makes the
staging tree readable by every user after it stages the sources, so the
umask of the build does not decide what mock can read.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock_build_owner parsed chrootuid and chrootgid from the text of the
rendered configuration. A value set in an included file, an assignment
with an inline comment or a double-quoted key, and an include after an
assignment all gave a uid that mock does not build as. Also, a root
build that stopped part way left root-owned 0644 logs in the result
directories, and the next build, as uid 1000, could not append to them.
mock_build_owner now asks mock's own loader, mockbuild.config.load_config,
for chrootuid and chrootgid of the chroot that buildrpms.pl passes to
mock -r. It fails with a clear message when mock is not installed.
prepare_mock_resultdirs moves to XCAT::BuildUtils. It also gives the
files already in both result directories to that owner, so a retry can
append to the old logs and replace a package of the same version. When
mock builds as the caller, nothing changes owner.
native/mock_resultdir_owner.t gave the wrong uid for four of its
configuration cases, and failed the 11 retry checks, before this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
PR #7897 made mock create its result directory as root by appending
config_opts['chrootuid'] = 0 to every per-package configuration. chrootuid
also selects the uid rpmbuild runs as, so every target now builds as root,
and the 1000 that openeuler-24.03-ppc64le declares no longer applies inside
the chroot.
Remove the assignment and give the result directory to the uid the target
declares. mock_build_owner reads chrootuid and chrootgid from the rendered
configuration, and prepare_mock_resultdirs creates dist/<target>/rpms and its
SRPMS subdirectory and chowns both before the build starts. openeuler-24.03-ppc64le
is the only target that declares a uid; elsewhere mock builds as the caller and
the directories are left alone.
buildrpms_source_only.t records the stages buildall runs, so it now expects the
new one between createmockconfig and buildsources.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
build-utils/lib/XCAT/BuildUtils.pm said the inherited chrootuid 1000 stays in
effect for the chroot. A mock configuration is Python, so the later assignment
replaces the value and nothing reads 1000 afterwards. The in-chroot build user
becomes uid 0.
buildrpms_mock_resultdir_uid.t asserted that the inherited line is still
present. The pattern is a line of the test's own fixture and mock_config_text
only appends, so the assertion passes even when the function returns the base
text unchanged. It is removed.
The comment now states the replacement and its consequence, and records why
chrootgid is left unset. Without the chrootuid assignment the file fails 2 of
10.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock refuses every package of the openeuler-24.03-ppc64le target with
"Could not create dir dist/openeuler-24.03-ppc64le/rpms/SRPMS/.
Error: [Errno 13] Permission denied: 'dist'", so the target builds nothing.
mock 6.8 reloads its uid manager from chrootuid, so chrootuid now also owns
--resultdir and the three log files mock writes beside it. buildrpms.pl passes a
relative resultdir under a source tree owned by root. xcat-dep's
openeuler-24.03-ppc64le.cfg declares chrootuid 1000, for the native dependency
prerequisites that need a non-root build identity, and mock_config_text copied the
target configuration verbatim.
mock_config_text now assigns chrootuid 0 after the inherited lines. An xcat-core
rpm build has no declared non-root identity, and every other target already builds
as root because no other configuration sets the option. The inherited declaration
stays in the file, so the chroot still sees what xcat-dep shipped.
chrootgid is left alone. mock creates the in-chroot mock group with it, and
`groupadd mock -g 0` fails because the root group already holds gid 0: pinning both
took the target from a resultdir failure to "mock --init failed" on all fourteen
packages. root creates the resultdir whatever group it carries.
buildrpms_mock_resultdir_uid.t fails 2 of 11 without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every package of xcat-core-build-openeuler24.03-ppc64le #1 died before rpmbuild
started:
ERROR: Could not create dir dist/openeuler-24.03-ppc64le/rpms/SRPMS/.
Error: [Errno 13] Permission denied: 'dist'
30 such lines, 0 srpms. buildrpms.pl passes a relative --resultdir inside a source
tree owned by root, and mock 6.8 creates that directory as chrootuid.
openeuler-24.03-ppc64le.cfg declares chrootuid 1000 for the native dependency
prerequisites, and createmockconfig copied the target configuration verbatim, so
the core build inherited a uid that cannot write there. No openEuler x86_64
configuration declares one, which is why only the ppc64le target fails.
createmockconfig had no unit test because it built its text inline and wrote it
straight to /etc/mock. The rendering moves to XCAT::BuildUtils::mock_config_text,
which takes the base text and returns the rendered text; the script keeps the two
file operations.
buildrpms_mock_resultdir_uid.t drives it directly and fails 2 of its 11 checks, both
on the uid the rendered configuration carries.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The staging directory was keyed by package, target and uniqueext. Nothing
deleted it, so every pair left a SOURCES tree under $HOME/rpmbuild, and an
archive from an earlier run stayed beside the next run's sources.
Each forked builder now stages into $HOME/rpmbuild/sources/SOURCES.<host>.<pid>.
The pid is unique among live processes on one host, and the host name keeps two
hosts apart when $HOME is on NFS. prepare_build_sources_dir creates the directory
empty. The parent deletes it when it reaps the builder, whatever the exit, and
abort_builds deletes it for each builder that exits after SIGTERM. At start, the
run deletes the directories of this host whose pid is not alive.
buildrpms_sources_per_package.t covers the name, the empty directory, the
deletion after success and failure, and the sweep of a dead pid next to a live
one and another host.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The xCATsn templates archive held xCAT/templates/..., so xCATsn installed the
templates under /opt/xcat/share/xcat/xCAT/templates. The xCAT build wrote the
same archive rooted at templates/ into the shared SOURCES directory, and the
existence check let xCATsn reuse it when xCAT wrote first.
stage_xcatsn_templates now runs tar from xCAT/, so the members start with
templates/, and it always writes the archive. The archive matches the one
the xCAT build writes.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCATsn.spec extracts templates.tar.gz in %{prefix}/share/xcat. buildrpms.pl
creates that archive from the checkout root, so its members start with
xCAT/templates and the templates install under share/xcat/xCAT/templates.
With one shared SOURCES directory the xCAT build often wrote the archive
first, rooted at templates/, and the existence check kept it.
The tar step moves unchanged into stage_xcatsn_templates in
XCAT::BuildUtils. xcatsn_templates_staged.t stages the archive and reads its
members. It fails on this commit: the members start with xCAT/, and a second
staging keeps the old archive.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every RPM target of every family failed in buildrpms.pl before it built a package:
Can't rename('.xCAT-probe-2.20.0.UNf6IG', '/xCAT-probe-2.20.0.tar.gz'):
Invalid cross-device link at ./buildrpms.pl line 386
The staging directory is set per package in the child that buildall forks. main()
staged the xCAT-probe tarball before that fork, where $SOURCES is still the empty
string, so the destination was composed as /xCAT-probe-<version>.tar.gz.
stage_xcat_probe_sources in XCAT::BuildUtils replaces prepare_xcat_probe_source_tar.
It takes the staging directory, refuses an unset or absent one, and composes the file
name itself. buildsources calls it for the xCAT-probe package, in the child, where the
directory exists.
xcat_probe_sources_staged.t runs the staging and reads the archive: the helpers are at
xCAT-probe/lib/perl/xCAT, no temporary file is left behind, and each refusal names the
guard that fired. Removing either guard turns it red. On the unfixed tree
"./buildrpms.pl --package xCAT-probe --source-only" dies at line 386; with this change
it reaches the build lock.
Seven assertions in xcat_probe_package_payload.t matched the text of buildrpms.pl
rather than running it, and four could not fail: three matched the xCAT-release
routine, which stages the same way, and one spanned the rest of the file. All seven
stayed green while every RPM target died. They are gone.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A core build failed in %prep with "File
/builddir/build/SOURCES/xCAT-test-2.20.0.tar.gz is smaller than 13 bytes". Its
src.rpm carried 0 bytes where two sibling builds of the same commit carried
1671225.
buildrpms.pl staged every tarball into one $HOME/rpmbuild/SOURCES while forking
a child per package and target, so one child's tar overwrote an archive another
child was copying with mock --sources.
build_sources_dir keys the staging directory per package and target, like the
mock chroot. buildrpms_sources_per_package.t covers it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Master merged #7835, which changed the Genesis base files under
xCAT-genesis-builder/, the directory this branch removes. The merge
keeps master's content at this branch's paths:
- GenesisBuildRoot.pm, genesis_payload_verification.t and
genesis_ubuntu_build_root.t take master's newer versions, pointed at
xCAT-genesis-base/.
- verify-genesis-payload and GenesisPayload.pm are identical on both
sides; the copies at the old path are dropped.
- buildrpms.pl keeps stage_genesis_base_sources(), which already stages
what master added inline.
- builddebs.pl and the new genesis_deb_per_codename.t now use
xCAT-genesis-base/.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Master added a workflow and several tests after this branch renamed
xCAT-genesis-builder to xCAT-genesis-base. They still name files under
xCAT-genesis-builder/, which no longer exists, so they fail or read
nothing. The OpenEmbedded workflow, six bats files and four unit tests
now use the new paths.
Two of those unit tests could reach the code only as shell.
genesis_ubuntu_build_root.t cut the package list out of
builddeb-genesis-base and evaluated it through bash.
genesis_payload_verification.t ran the bash verifier and parsed its
stderr.
XCAT::GenesisBuildRoot::required_packages() now returns the build-root
package list, and builddeb-genesis-base calls it. XCAT::GenesisPayload
holds the mandatory-command list and the payload check, and
verify-genesis-payload runs its main() with the same arguments, exit
codes and messages. stage_genesis_base_sources() stages the module
beside the script. Both modules use core Perl only.
The two tests now call these functions with chosen inputs and assert
the exact results. genesis_base_sources_staged.t expects the module in
the staged tarball.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Master fixed debuild-xcat-genesis-base and added a bats test for it. The
branch deleted that script as unused. The merge keeps the script and
moves it to xCAT-genesis-base with the other Genesis base files.
Master also stages verify-genesis-payload for the spec %install check.
stage_genesis_base_sources() now stages it too, and
genesis_base_sources_staged.t expects it in the tarball.
The workflow, builds.rst and xcat_probe_package_payload.t take the
master version.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
buildsources_genesis_base() lived in buildrpms.pl, which runs mkdir, git
and file writes at file scope and cannot be loaded. Its test therefore
extracted the sub with a regex and evaluated a copy of the source.
Move the staging into XCAT::BuildUtils as stage_genesis_base_sources(),
which takes the checkout, the tarball path and SOURCE_DATE_EPOCH.
buildsources_genesis_base() now calls it. The tarball content and the
failure on a missing xCAT-genesis-base directory do not change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Killing the command is not killing the build. sh() ran the command through /bin/sh,
and cancellation signalled that shell alone -- but dpkg-buildpackage starts workers
of its own, and those survive their shell. The lock was then released while they
were still writing debian/changelog and debian/control, which is the state the lock
exists to prevent: the next build takes the checkout and the two rewrite it
together.
The command now runs in its own process group, so cancellation can take all of it.
Both sides call setpgid, so neither depends on which runs first, and INT and TERM
are blocked across the fork so cancellation cannot land in the window before the
group exists.
Cancellation escalates from the caught signal to KILL, and then CHECKS: a shell that
has exited is not a build that has stopped, so it waits for the whole group to
disappear rather than for the leader to be reaped. If the group is still there after
that, the locks are RETAINED and the process exits non-zero. Releasing a lock while
a worker may still be writing is worse than leaving a lock behind for a person to
clear -- the first corrupts a build, the second stops one.
cancel_build ignores INT and TERM while it runs, so a second Ctrl-C cannot interrupt
the cleanup half way and release the lock early.
sh() also reports a signalled command as 128+signal instead of 0. $? >> 8 is zero
for a child killed by a signal, so a build stopped mid-way looked to its caller like
one that had succeeded.
Two cases added to builddebs_lock_cancellation.t: a build whose worker is a
grandchild, and a command killed by a signal. Verified by signalling the pid instead
of the group, which leaves the worker running and turns the first red.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is released in DESTROY, and perl does not run DESTROY when a signal
ends the process. A build stopped with SIGTERM or SIGINT therefore left its lock
directory behind, and the next build of that checkout died on
FATAL: another build of <path> already holds <dir> (held by [pid=NNNN])
naming a pid that had already exited. Nothing clears it but a person. One such
directory blocked an openSUSE target across three consecutive CI runs before anyone
looked at what the lock actually said.
buildrpms.pl has released its lock on cancellation for some time, through an END
block and an abort handler. This is the Debian builder catching up.
The order matters, and is the reason this is not simply an END block. The command in
flight is stopped BEFORE the lock is released: handing the checkout to a second build
while dpkg-buildpackage is still rewriting debian/changelog and debian/control in it
is worse than holding the lock a moment longer. The wait for that command is bounded,
so a subprocess that ignores the signal cannot hold the lock for ever either.
sh() now forks and execs rather than calling system(), because system() gives no pid
and a handler cannot stop what it cannot name. The child _exits rather than exits, so
it never runs the parent's END block and releases a lock the parent still holds.
The handler is installed by XCAT::BuildUtils::install_build_cancellation rather than
written inline in the builder, so a test can use the same wiring the builder uses. A
test that installs an equivalent handler of its own proves the helper works while
saying nothing about whether anything calls it -- the first version of this test did
exactly that, and passed with the wiring removed.
Release is idempotent: a signal handler and then DESTROY both reach it, and the
second must not remove a directory a LATER build has since taken.
builddebs_lock_cancellation.t terminates the holder, then takes the lock again, and
checks no build subprocess was orphaned. Verified by removing the wiring: assertions
9 through 12 fail, naming the leaked lock, the refused build and the stray process.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build tree can live on an NFS re-export. The kernel refuses locks on one --
"Clients are not allowed to get file locks or delegations from a reexport
server" -- so every flock() there answers errno 524, and a build that takes one
dies before it starts.
buildrpms.pl's per-target lock and BuildUtils.pm's take_build_lock, which
builddebs.pl calls for the Ubuntu core build, are both atomic mkdir claims now.
Each records its owner and names it when it refuses.
A directory is not released by a filehandle closing, which is how both locks
were freed before. buildrpms.pl releases from END, and again in abort_builds
because that handler re-raises the signal with DEFAULT and END blocks do not run
then -- a killed build would otherwise strand the lock for every later one.
BuildUtils returns a small object whose DESTROY releases it, preserving the
caller's "hold the returned value" contract.
Both releases are guarded by owning pid: both scripts fork, and the flock they
replace could not be released by a child.
builddebs_lock.t closed the returned value to prove the lock is released, which
is "Not a GLOB reference" against the new contract. It now lets the value go out
of scope. What it asserts is unchanged: a second build of the same checkout is
refused, and the next one succeeds once the first releases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Merge current master into the openEuler support branch. Keep native
kernel, DHCP, locale and timezone requirements with the upstream payload
verifier and EL10 package paths. Preserve signed openEuler installation.
Update the native fixtures to check the merged payload contract.
deb_belongs_to_dist read any trailing ~word in a package version as the
codename the deb was built for. Debian uses ~ for a prerelease and --release
takes whatever the caller gives it, so `--release 1~rc1` left every package
excluded from every suite, with --genesis nowhere in the command and the run
still reporting what it published. Only xcat-genesis-base carries a codename,
because only it carries a kernel, so only it is asked.
genesis_dists() names the releases a Genesis image can be built on. A plain
--genesis run took the release list the rest of the build uses, and focal is on
it: focal ships debhelper 12.10 and the package declares debhelper-compat
(= 13), so sbuild stopped on the build dependencies before dracut ran and the
run ended on its first release. The default plan now leaves focal out and says
so. A release named with --genesis-dist is still built as asked.
genesis_deb_per_codename.t covers both. Applying the codename rule to every
package again fails five assertions; a genesis_dists that filters nothing, or
an image that loses its own rule, fails two each.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
genesis_ubuntu_build_root.t called BAIL_OUT at four places where an
extraction stopped matching. prove stops every remaining file on a
bail-out, so one stale regex in this file hides the results of the tests
that would have run after it. die is just as loud and costs only this
file.
The branch also states one fact in four places. That dracut copies the
kernel out of the root it runs in appears in the builddebs.pl header, in
its Genesis section, in BuildUtils.pm and in the builder, each time with
the incident that produced it. The chroot stage directory is explained
twice, once at its declaration and again at its only use. Each fact now
stands where the reader meets it, without the bug report around it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddebs.pl publishes every Architecture:all deb into every release, because
every other xcat-core deb is the same file for all of them. The Genesis image is
not: it carries the kernel of the root that built it. All three images therefore
landed in all three suites, and apt serves the newest, which belongs to another
release.
deb_belongs_to_dist reads the codename back from the version and keeps an image
out of any other suite. A deb with no codename in its version is unaffected.
The jammy build also stopped on /usr/share/terminfo/l/linux and v/vt100:
ncurses-base installs terminfo under /lib, and the module asks for the
/usr/share copies, which come from ncurses-term. noble happened to have it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
dracut copies the kernel, the kernel modules and every command out of the root
it runs in. The Ubuntu Genesis deb was built once, on the build host, so every
Ubuntu release got the build host's kernel. Nothing called the builder at all:
the pipelines converted the EL rpm with alien instead.
builddebs.pl gains --genesis, --genesis-only and --genesis-dist. The Genesis
deb is now built once per codename inside that codename's <codename>-<arch>-sbuild
chroot, the chroots xcat-dep's sbuild-all.pl already creates on the Ubuntu build
host. builddeb-genesis-base takes --expect-codename and stops when the root it
woke up in is a different release, so a build on the build host cannot produce a
codename's image.
builddebs.pl reads the build log through XCAT::BuildUtils::genesis_log_errors and
fails the build on FAILED:, a package apt cannot find and four more lines that a
zero exit status hides. The extracted payload goes through verify-genesis-payload
with the command list read back from the dracut module, plus the DHCP client, the
97xcat hooks and a /lib/modules that holds this chroot's kernel and no other.
The build root gains isc-dhcp-client, ifenslave and util-linux-extra, which
supply dhclient, ifenslave and hwclock.
genesis_deb_per_codename.t, genesis_payload_verification.t and
genesis_ubuntu_build_root.t fail without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The xcat and xcatsn packages declared only amd64 and ppc64el, and the
builder built and indexed only those two architectures, so a riscv64
management node had no package to install. Both packages are now built for
riscv64, every release the repository serves declares the architecture,
and the generated mklocalrepo.sh maps a riscv64 host to its own repository
instead of amd64. Each package now carries its own architecture list:
xCAT-genesis-scripts keeps the two it has control files for, because
riscv64 Genesis ships as an OpenEmbedded package instead.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>