dnf install xCAT on an openEuler 24.03 ppc64le management node stops with
"nothing provides ipxe-xcat >= 2.0.0-1 needed by xCAT-2.20.0". xCAT and
xCATsn require ipxe-xcat by name on ppc64le, both EL ppc64le sections of
packages-manifest.conf list it, and the openEuler section does not, so
mockbuild-all.pl never builds it and the dep repository never carries it.
Declare the package in four places, because each one refuses it
separately: the manifest section, the openEuler target profile that
selects which dep builders run, the native input catalog that resolves an
owner for every manifest name, and the catalog owner allow-list.
ipxe-xcat.spec is BuildArch noarch with no BuildRequires and no compile
step, so the owner builds as uid 1000 like grub2-xcat.
elilo-xcat and conserver-xcat stay out. No core rpm requires either one:
xCAT uses goconserver for consoles, and elilo is an EFI loader for
architectures this cell does not serve.
t/openeuler.t covers both halves and fails without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every native prerequisite after procenv itself died before rpmbuild started:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/procenv'
mock's procenv plugin executes ["/usr/bin/procenv"] on the build host, not in the
chroot, so the build needed a host package that neither this repository nor
xcat-core-ci-cd declares. The plugin is pure provenance: it writes procenv.log into
the result directory and nothing reads it. mock ships it off.
native_overlay_text now leaves procenv_enable at mock's default for every overlay,
so a native build depends on nothing outside the two repositories.
The procenv node stays in the inputs: mock appends procenv to the buildroot's
preexisting_deps whenever the plugin is on, so the package is still what makes the
plugin usable if the host is ever provisioned for it.
openeuler_native_procenv_plugin.t passes with this commit and failed two checks
without it. Measured on xcat-master-ppc: run 264b installed
procenv-0.60-1.ppc64le into the docbook-style-dsssl chroot and failed on the same
line, which is what showed the chroot was never the problem.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Every openEuler ppc64le native prerequisite after procenv itself fails before
rpmbuild starts:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/procenv'
raised from mock's procenv plugin prebuild hook. The plugin runs ["/usr/bin/procenv"]
through mockbuild.util.do with no chroot prefix, so it executes the BUILD HOST's copy,
and xcat-master-ppc has none. The plugin also appends procenv to the buildroot's
preexisting_deps itself, so the chroot already gets the package: a run that installed
it, with "Installing : procenv-0.60-1.ppc64le" in root.log, failed on the same line.
mock ships procenv_enable False. The overlays turn it on, which makes every native
build depend on a host package that neither repository declares, for a procenv.log
that nothing reads.
The overlay text was built inline in native_overlay, so there was nothing a unit test
could call. native_overlay_text in XCAT::NativeInputs now renders one overlay and
reproduces today's values exactly, and native_overlay writes what it returns.
openeuler_native_procenv_plugin.t asserts the plugin is left at its default in all
three overlays, that the prerequisite repository and the bind mount survive, and that
the uid split does not move. Two of its eleven checks fail on this commit, which is
the defect.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The openeuler-24.03-ppc64le catalog pins perl-Net-DNS-1.40-1.oe2403
from openEuler 24.03 LTS GA as a publisher input. That release has
CVE-2026-64193 and CVE-2026-64194. The GA srpm carries no fix, and
load_inputs accepts only GA rpms as publisher inputs.
Build perl-Net-DNS for this target from perl-Net-DNS/Net-DNS.spec,
the 1.57 release that rocky-10-riscv64-xcat already builds. The
catalog now declares perl-Net-DNS as an owner, load_inputs allows that
owner, and the manifest pins 1.57. The spec BuildRequires all resolve
from the openEuler 24.03 LTS OS repository.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
trust_dbpath named the keyring after a hash of the staging directory and
did not create it. A directory or symlink that another user put at that
predictable name under the local root became the rpmkeys --dbpath for
the publisher key.
trust_dbpath now creates a fresh 0700 directory with tempdir under the
same root, so XCAT_DEP_TRUST_TMP still selects it. CLEANUP removes it
when the creating process exits. stage_inputs and build_native_inputs
run in that process, so the keyring stays in place for every
verify_input of the run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
openeuler-20.03sp4-x86_64 and openeuler-22.03sp4-x86_64 build nothing. Both exit 3
at "Could not find included config file:
/etc/mock/templates/openeuler-20.03-sp4.tpl", and both targets report that every
build step failed. mock-core-configs names its openEuler templates by release
alone, openeuler-20.03.tpl and openeuler-22.03.tpl, and each already sets
releasever to the newest service pack. The two configurations spelled the service
pack into the template name, which matches no file in any mock package.
openeuler-24.03sp4-x86_64.cfg already includes openeuler-24.03.tpl, which is why
that target builds.
The ppc64le target fails before any package is built, for two further reasons.
rpmkeys --dbpath cannot import the publisher key: rpm takes an fcntl transaction
lock on <dbpath>/.rpm.lock, the staging directory is NFS mounted local_lock=none,
and that lock answers errno 524 there. The keyring database moves to local
temporary storage, keyed on the staging directory so two targets of one run do not
share it. The staged source RPMs and the gpg home do not move.
A verified publisher RPM is rejected as unreadable: reject_elf_payload read the
exit status of rpm2cpio as the verdict on the payload, and on rpm 4.19 rpm2cpio
writes the whole payload and still exits 1 with nothing on stderr. The cpio walk
moves into scan_cpio_for_elf, which returns its own answer. A truncated stream
never reaches the trailer and still dies, which is the case the exit status stood
in for, and verify_input has already matched the sha256 and run rpmkeys --checksig
before this point.
Each fix has a test that fails without it. openeuler_mock_template_include.t reads
the configurations only. openeuler_native_trust_dbpath.t and
openeuler_native_cpio_scan.t run no rpmkeys and no rpm2cpio: the second builds its
cpio streams in Perl and refuses an ELF member, a truncated stream, data after the
trailer and a stream that is not cpio, each by name.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
_log set $| on the caller's selected handle for each line. The caller
owns its buffering, so the lock log only prints the line.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build log did not show when a lock was taken, waited on, taken over
or released, so the lock order of the parallel dep builds could not be
read from their logs.
XCAT::NFSLock now prints one line per event to the selected output
handle, with the UTC time in milliseconds, the host, the pid, the event,
the label and the lock path. The events are acquired, took-over, wait,
released and release-skipped. quiet => 1 turns the log off for a lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
XCAT::NFSLock recovered a dead owner's lock under a second lock,
lock.break. A process that died after it created lock.break left it in
place, and every later acquire timed out even when both processes were
dead. The lock and its record were also put in place with rename, which
replaces an empty directory, so acquire took over a lock.d that had no
owner record.
The lock now follows the NFS lock protocol at the top of NFSLock.pm. It
uses mkdir, rmdir, unlink and plain writes, and no rename. lock.d/metadata
holds the owner identity and a hash, and invalid metadata only causes a
retry. Takeover and release both hold lock.borrow, beside lock.d. Only a
process on the owner's machine takes over a lock, after it reads the
same identity twice and proves that owner dead. acquire retries R times
with a wait of T +/- jitter, and --try-unlock-timeout maps to R.
Records written by the previous format are invalid metadata and are not
taken over. A process that dies while it holds lock.borrow still blocks
the lock. The protocol assumes that this does not happen.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The dep builds lock trees on a shared NFS tree that the hypervisors
re-export. The kernel refuses flock and fcntl locks to clients of a
re-export, so every such lock fails with errno 524.
XCAT::NFSLock builds a lock as a directory under a private name, with an
owner record and the caller's metadata, and renames it into place. The
record holds the machine id, boot id, pid, process start time and a
random token. The owner removes the lock, or a process on the owner's
host that proves the owner dead: another boot, no such pid, or a pid
with another start time. That process takes a breaker lock and proves
the owner dead again before it removes the lock. Removal renames the
lock away first. Any other acquire waits up to a timeout, retrying every
3 s by default, then fails with the mv command that moves the lock away.
A lock left by mockbuild-all.pl before this module is taken when its
pid is gone on this host.
t/nfslock.t covers each rule without NFS.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Merge current master into the openEuler dependency branch. Preserve native
inputs and publisher signatures with the upstream timeout and keyring
helpers. Use the native repository path when carrying skipped builds.
Cover signed carry-over and mixed publisher/build-key verification.
run_bounded signalled the process group on a timeout and on a cancellation, but
not when the leader itself died. A SIGKILL, or the OOM killer, takes the shell
and leaves schroot and qemu in the group, holding the chroot and writing into
staging after the call reports the build finished.
Those processes are not children of this one, so there is nothing to wait for.
Signal the group on the way out.
A child killed by a signal leaves 0 in the high byte of its wait status. The
per-codename worker loop read only that byte, so a cancelled or OOM-killed
codename was counted as built, and the run could reach validation with the
staging tree that worker never finished.
One helper decodes a wait status for both the worker loop and run_bounded, and
reports 128 plus the signal for a child the kernel killed.
The forwarding handler was in place before the fork, but the parent recorded the
worker pid after it. A signal in between reached a handler that did not know the
worker, so the orchestrator died while the worker kept building for hours and
held the per-architecture lock.
The mask discipline run_bounded already used is now a pair of helpers, and the
worker loop blocks the handled signals across both the fork and the
registration. The worker resets the inherited handlers before restoring the mask.
The handlers were installed after fork(), so a signal arriving in between killed
the wrapper under its inherited handler and left the new process group running.
INT, TERM and HUP are now blocked around the fork and delivered once the handler
is in place. The child restores the mask before exec, or the build would inherit
a blocked TERM and ignore the signal the forwarding depends on.
Both orchestrators fork a worker per codename or per build step, so a signal
sent to the orchestrator never reached the builds: run_bounded's forwarding
covers the build inside one worker, not the workers themselves. The
orchestrator exited and released its locks while its workers kept building,
and the next run raced processes it could not see.
One handler now passes INT, TERM and HUP to the live workers, waits for them
and re-raises the signal, shared by the sbuild loop and both ForkManager pools.
With no deadline the helper called system(), which leaves the build in the
orchestrator's process group and installs no signal handler. A cancellation
then killed the orchestrator, which released its locks while the build kept
writing into staging, and a build killed by a signal was reported as rc=0.
Both paths now fork; a timeout of 0 removes the deadline, nothing else.
The bounded build ran in its own process group, but a signal to the
orchestrator was not forwarded to it. On cancellation the orchestrator
exited and released its locks while schroot, mock and qemu kept running and
writing into staging, so the next run raced an orphan it could not see.
INT, TERM and HUP now reap the group before the process dies by the same
signal, which keeps the exit status honest.
Every per-package build ran through a bare system() call with no wall-clock
bound. Under qemu-user a build can deadlock -- a riscv64 goconserver `go build`
held both Go pids in futex_wait for 26 minutes with no CPU ticks and no open
socket -- and the step then never returns. The pipeline does not go red; it
stops, and a stopped run reads as "still running".
XCAT::BuildUtils::run_bounded runs the command in its own process group, kills
that group when a budget expires, and prints first what the manual
investigation had to collect by hand: the process tree, each pid's kernel wchan
and stack, its open socket count, and the CPU ticks the group used across a
20-second sample. Zero ticks names a deadlock; ticks name a build that is only
slow. BuildUtils::build_deb_in_chroot bounds every Ubuntu package build, and
mockbuild-all.pl bounds the dep and perl steps of a forcearch target.
The budget is 900 seconds for a native build and ten times that for a foreign
architecture, because qemu-user under TCG runs at roughly a tenth of native
speed. Both sit about five times above the slowest build measured on
xcat-master-ub: 3 minutes native, and 26 minutes for riscv64 ipmitool-xcat on
resolute with four codenames building at once. Native mock steps stay unbounded
-- no measurement of them exists, and a guessed budget would turn a trusted
cell red. sbuild-all.pl --build-timeout and mockbuild-all.pl --build-timeout
override the default; 0 removes the bound.
t/build_timeout.t fails without this change: run_bounded never returns and the
test reports the hang instead of blocking.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Tolerating individual dep-builder failures is deliberate: some packages are el-
or arch-pinned and are expected to fail on some targets. Tolerating all of them
is not -- it means the builder is unusable, the invocation produced nothing, and
whatever the run publishes came from somewhere other than this build.
Count the failures on both the serial and the parallel path and stop when they
account for every attempted step, before collection can take an earlier run's
artifacts for this one's. The rule itself lives in BuildUtils, where it can be
exercised without a builder.
Daniel Hilst
Vinícius Ferrão