ipxe-xcat/mockbuild.pl runs as root and calls make_path on $work_dir/srpm
and $work_dir/rpm, then gives each one to mock as --resultdir. The native
openEuler overlay sets chrootuid to the catalog's build_uid and chrootgid to
the native build gid, so mock reloads to that uid and opens state.log,
build.log and root.log in those directories. A root-made directory with the
default umask is root root 0755 and mock cannot write them. grub2-xcat,
ipmitool-xcat, syslinux-xcat and goconserver create their result directories
the same way.
What keeps those directories writable is the step wrapper, not the builder:
native_owner_command gives the step the native build gid and umask 0002, so
every directory the builder creates below it is group writable. No test
measured that, so removing either one is silent.
Move native_owner_command into MockBuildUtils so a test can call it in
process, and assert the wrapper carries setpriv --regid 1000,
--clear-groups and umask 0002, that setpriv precedes the umask and the umask
precedes the builder, and that the overlay is still bound over the target
config so chrootuid stays as the catalog declared it.
This commit is the move and the test only. Five of the six assertions fail
here, because the gid and the umask arrive in the next commit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
native/openeuler-power-inputs.t gates its whole native owner section on
"from mockbuild.util import load_config". mock 6.8 keeps load_config in
mockbuild.config, so the import fails, $can_owner is false and 52 assertions skip
on the only architecture that can run them. native/fixtures/power-mock.py imports
it the same way and cannot start at all.
The fake mock also created --resultdir as the user that started it and wrote no
logs, so it could not see what mock 6.8 does: it reloads its uid manager from
chrootuid, then creates --resultdir and opens state.log, build.log and root.log in
it as that uid.
Both now use mockbuild.config, and the fake mock creates --resultdir and opens
those three logs as chrootuid. Against the unfixed tree the native owner fails:
mock: [Errno 13] Permission denied:
.../native-results/native-leaf/state.log
ERROR: Step failed (rc=1): Build native prerequisite: native-leaf
Failed 14/115 subtests
The directory is root root 755 and chrootuid is 1000, which is the catalog's
declared build_uid for that node.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT already treats aarch64 as a target: BootPolicy hands an aarch64 node
boot/grub2/grub2.aarch64, and grub2.pm aborts the node's grub configuration
when that file is missing. xcat-dep neither ships the loader nor has an
aarch64 manifest section, so mockbuild-all.pl stops at "no manifest section
for target 'alma+epel-10-aarch64'" and an aarch64 node cannot netboot.
grub2-xcat generates grub2.aarch64. riscv64 ships the distro grub2 UEFI
image unchanged, but the EL grubaa64.efi carries the distro prefix
(/EFI/<vendor>) and never finds the boot/grub2/grub.cfg-<HEXIP> that
nodeset writes. %build runs grub2-mkimage -O arm64-efi -p /boot/grub2, the
prefix grub2.ppc uses, from the noarch grub2-efi-aa64-modules, so every EL
build host produces it. efi_uga stays out of the module list: it is x86
only and grub2-mkimage fails on it for arm64-efi. The image ships as
aarch64-efi/grubaa64.efi and %post copies it to grub2.aarch64, the boot file
of DHCP client architecture 11. The openEuler x86_64 repositories carry no
grub2-efi-aa64-modules and openEuler has no aarch64 target, so that build
ships without the image. mockbuild.pl checks the payload holds an AArch64
PE image. Release 3 -> 4.
[alma+epel-10-aarch64] is a native EPEL target like x86_64 and ppc64le, not
a forcearch cross-build like riscv64, and lists what [alma+epel-10-x86_64]
lists: the EL10 service node perl set (t/servicenode_perl_deps.t), and the
noarch x86 boot loaders, because an aarch64 management node serves the x86
nodes of a mixed cluster as a ppc64le one does.
Those loaders now build natively on aarch64 the way they do on ppc64le.
elilo-xcat ships its tracked prebuilt EFI payload, as on ppc64le: elilo
cannot compile on aarch64. syslinux-xcat admits aarch64 in ExclusiveArch and
takes the ppc64le path, installing the prebuilt bootloader blobs of the
release tarball. Both noarch rpms built on aarch64 carry the same payload,
byte for byte, as the published rh10/ppc64le ones. x86_64 and ppc64le
builds are unchanged.
t/aarch64_target.t holds the manifest cell to the EL10 x86_64 one and the
loader pins to EL10 ppc64le, checks how the spec generates, ships, installs
and removes grub2.aarch64, and where grub2-mkimage and the arm64-efi
modules are installed, runs the spec's own command, checks the image is
AArch64 with a /boot/grub2 prefix, and checks that adding efi_uga fails.
aarch64 is EL10 only and is not in the --finalize-xcat-dep cross-arch
genesis copy; BUILD.md says so.
Validated: a full mockbuild-all.pl --target alma+epel-10-aarch64 on an EL10
aarch64 host builds and verifies all 22 required packages (7 minutes).
Signed-off-by: Timothy Middelkoop <tmiddelkoop@internet2.edu>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No build script, manifest or test uses gpxe/, the gPXE 0.9.7 to 1.0.1
sources and patches, or ipxe/, an iPXE snapshot from 2010. No published
package comes from them. The xNBA builds take their source from xnba/,
and ipxe-xcat repackages the iPXE release from ipxe-xcat/.
On an EL10 host, verify-xcatsn-install.sh failed for EL8 with "nothing
provides /usr/libexec/platform-python needed by nfs-utils". EL8 packages
carry file dependencies that are only in filelists.xml, and dnf 4.19 and
later do not load filelists.
The check now sets optional_metadata_types=filelists. The usage text also
says how to import the AlmaLinux 8 key, which the rpm of an EL10 host
rejects for its SHA-1 bindings.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
trust_dbpath named the keyring after a hash of the staging directory and
did not create it. A directory or symlink that another user put at that
predictable name under the local root became the rpmkeys --dbpath for
the publisher key.
trust_dbpath now creates a fresh 0700 directory with tempdir under the
same root, so XCAT_DEP_TRUST_TMP still selects it. CLEANUP removes it
when the creating process exits. stage_inputs and build_native_inputs
run in that process, so the keyring stays in place for every
verify_input of the run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
trust_dbpath answers the same name under the local root for one staging
directory, and does not create it. A directory or symlink that another
user puts at that name is then used as the publisher keyring.
The test plants a symlink at the path the first call answered. It
expects a second call to answer a fresh 0700 directory under the root,
and the XCAT_DEP_TRUST_TMP override to still pick the root.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
An air-gapped EL service node reads BaseOS, AppStream, xcat-core and
xcat-dep, and no other repository. Nothing checked that xCATsn resolves
from that set, so a dependency that only CRB or EPEL carries was found on
the service node.
verify-xcatsn-install.sh runs dnf --installroot into an empty directory
with only those four local repositories. It disables every configured
repository, refuses a URL as input, and fails when dnf loads a repository
that is not file://. Package signatures are checked against the keys given
with -k. The installroot is removed on exit.
t/verify-xcatsn-install.bats tests the argument handling, the local
repository guard and the cleanup with dnf and rpm stubbed.
package-tests runs it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The comments in packages-manifest.conf and t/servicenode_perl_deps.t no
longer match the code. They say xCAT-server requires perl-DB_File on EL10
and that an EL10 service node takes it from EPEL. xCAT-server only
recommends perl-DB_File on EL10, and a service node reads no EPEL. The
riscv64 comment says the EL10 sections take perl-HTML-Form and the other
EPEL perl packages from EPEL, but those sections now list them.
Remove the per-release package table, which repeats the set the test
holds, and the perl-DB_File notes. Name the four packages that only the
riscv64 target builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
openeuler-20.03sp4-x86_64 and openeuler-22.03sp4-x86_64 build nothing. Both exit 3
at "Could not find included config file:
/etc/mock/templates/openeuler-20.03-sp4.tpl", and both targets report that every
build step failed. mock-core-configs names its openEuler templates by release
alone, openeuler-20.03.tpl and openeuler-22.03.tpl, and each already sets
releasever to the newest service pack. The two configurations spelled the service
pack into the template name, which matches no file in any mock package.
openeuler-24.03sp4-x86_64.cfg already includes openeuler-24.03.tpl, which is why
that target builds.
The ppc64le target fails before any package is built, for two further reasons.
rpmkeys --dbpath cannot import the publisher key: rpm takes an fcntl transaction
lock on <dbpath>/.rpm.lock, the staging directory is NFS mounted local_lock=none,
and that lock answers errno 524 there. The keyring database moves to local
temporary storage, keyed on the staging directory so two targets of one run do not
share it. The staged source RPMs and the gpg home do not move.
A verified publisher RPM is rejected as unreadable: reject_elf_payload read the
exit status of rpm2cpio as the verdict on the payload, and on rpm 4.19 rpm2cpio
writes the whole payload and still exits 1 with nothing on stderr. The cpio walk
moves into scan_cpio_for_elf, which returns its own answer. A truncated stream
never reaches the trailer and still dies, which is the case the exit status stood
in for, and verify_input has already matched the sha256 and run rpmkeys --checksig
before this point.
Each fix has a test that fails without it. openeuler_mock_template_include.t reads
the configurations only. openeuler_native_trust_dbpath.t and
openeuler_native_cpio_scan.t run no rpmkeys and no rpm2cpio: the second builds its
cpio streams in Perl and refuses an ELF member, a truncated stream, data after the
trailer and a stream that is not cpio, each by name.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT-server requires perl-Net-DNS. No AlmaLinux 10 repository carries it, and
neither does the AlmaLinux 10 DVD that copycds copies, so dnf install xCATsn on
an EL10 service node reports "nothing provides perl-Net-DNS". Only EPEL has the
package, and a service node reads no EPEL.
List perl-Net-DNS=1.57 for [alma+epel-10-x86_64] and [alma+epel-10-ppc64le].
The riscv64 target already builds that version from the same spec, so the EL10
targets get the Net::DNS that carries the KEY record ddns.pm builds. A
management node keeps the EPEL copy, which is 1.57-1.el10_3 and outranks
1.57-1, so nothing changes there.
The comment above @epel_gap_order in mockbuild-perl-packages.pl said the
EPEL-fed targets keep taking these packages from EPEL. mockbuild-all.pl passes
--packages from packages-manifest.conf, so the manifest decides, not that list.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
t/servicenode_perl_deps.t left perl-Net-DNS out of the EL10 service node set.
The recorded reason was that this tree held Net-DNS 0.80, which has no KEY
record, so shipping it would replace a working copy with a broken one. The tree
now builds 1.57 and that record is present.
No AlmaLinux 10 repository carries perl-Net-DNS, and the AlmaLinux 10 DVD that
copycds copies carries neither perl-Net-DNS nor libdb. xCAT-server requires
perl-Net-DNS, so dnf install xCATsn on an EL10 service node does not resolve.
Add perl-Net-DNS to the EL10 set. The test fails for [alma+epel-10-x86_64] and
[alma+epel-10-ppc64le] until the manifest lists the package. perl-DB_File stays
out: xCAT-server requires it outside riscv64, EL10 dropped libdb, and xcat-dep
has no source to build it from.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock --init failed for every openEuler target. repo.openeuler.org is the only
source the chroots have, and it served a build host at 7 kB/s, then reset
connections mid-transfer: "Curl error (28) Operation too slow", "Curl error
(56) SSL_read: SSL_ERROR_SYSCALL", "Curl error (92) INTERNAL_ERROR". All three
attempts per package failed, so the openEuler core build could not start.
mock-configs/templates/openeuler-lts-xcat.tpl builds dnf.conf, so baseurl now
lists two mirrors before repo.openeuler.org, which stays last as the authority
of record, and timeout rises to 300. minrate stays at dnf's default: the rate
abort is the only signal that moves dnf to the next baseurl. gpgcheck stays on
and gpgkey still names repo.openeuler.org, so a mirror cannot change what is
built. Measured on one host, same gcc rpm: repo.openeuler.org 7 kB/s,
mirrors.tuna 2.8 MB/s, mirror.nju 3.9 MB/s. A cold mock --init then completed
in 2m38s.
native/mock-configs.t asserts the slowest host is not first, the canonical url
is present, every url names the same release and architecture, and minrate is
unset. It could not run before: native/fixtures/mock-configs.py copied two
templates no source ships and raised FileNotFoundError, so the test measured no
config at all.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
t/net_dns_rr_types.t compared every perl-Net-DNS pin in the manifest
with the version of perl-Net-DNS/Net-DNS.spec. openeuler-24.03-ppc64le
does not build that spec. Its input catalog pins the openEuler
perl-Net-DNS-1.40 rpm as a publisher input, so its pin of 1.40 failed
against the spec's 1.57, and package-tests is red on master.
For a target whose catalog publishes perl-Net-DNS, compare the pin with
the version in the pinned rpm name. Other targets still compare with
the spec.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The review asked for Net::DNS 1.57 with the tarball, the spec and the
manifest pin in sync. No test held either. A 1.56 tarball passed every
check, although 1.56 recurses without bound when it re-encodes a reply
with a misplaced TSIG record (rt.cpan.org #181125, fixed in 1.57). A
second Net-DNS tarball beside the spec, or a Buildnote that names
another release, also passed.
Decode such a reply with the shipped Net::DNS and re-encode it, with
Packet::encode wrapped to stop and record recursion past 20 levels.
Also check that perl-Net-DNS/ holds only the tarball the spec builds,
and that the Buildnote names that tarball.
The recursion check fails on 1.56. The two sync checks fail on a tree
that keeps Net-DNS-0.80.tar.gz and the 0.80 Buildnote.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
perl-Net-DNS built Net::DNS 1.47 without patches. That release decodes
the EDNS EXTENDED-ERROR text with a string eval (CVE-2026-64193) and
follows compression pointer chains of any depth (CVE-2026-64194).
Net::DNS 1.56 fixes both, and 1.57 also stops an unbounded recursion on
a misplaced TSIG.
Build Net::DNS 1.57. The tarball, the spec, the Buildnote and the
riscv64 manifest pin move together. The build requirements of 1.57 are
the same as those of 1.47.
t/net_dns_rr_types.t decodes a reply with a 200-pointer chain and fails
on 1.47.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
perl-Net-DNS builds Net::DNS 1.47. That release decodes a reply whose
owner name is a chain of 200 compression pointers, one recursion per
pointer (CVE-2026-64194). Net::DNS 1.56 stops the chain with "deep
compression recursion".
The test decodes such a reply with the Net::DNS from the shipped tarball
and expects that error. It fails on 1.47.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Exercise mixed repository finalization and changed publisher inputs on x86_64. Check native manifest selection and use File::Slurper in the goconserver fixture.
Exclude native cells from cross-architecture Genesis finalization. Add the missing 24.03 SP1 manifest and expose repository signing for direct validation.
dnf install xCATsn on an EL service node does not resolve. Against the repositories a service
node has -- BaseOS and AppStream from copycds, plus xcat-core and xcat-dep -- it reports
"nothing provides" for perl-IO-Tty, perl(IO::Pty), perl-Crypt-CBC, perl-Crypt-Rijndael,
perl(Expect) and perl(HTML::Form), and per release for perl-Crypt-SSLeay, perl-Net-Telnet and
perl-Digest-SHA1. The service node then has no xcatd, and every hierarchical case fails.
packages-manifest.conf assumed CRB and EPEL stand behind every EL target. That holds for a
management node and not for a service node, so those packages were built only for
rocky-10-riscv64-xcat, the one target with no EPEL. Each EL target section now lists the perl
packages BaseOS and AppStream do not carry.
mockbuild-perl-packages.pl had no table entry for perl-IO-Tty, so no target could build it. Its
sources move to IO-Tty 1.20 with a spec. In 1.12 the Makefile.PL pty probe only takes the
address of each candidate function, the current toolchain drops that reference, the build then
defines HAVE__GETPTY on a glibc that has no _getpty, and IO/Tty/Tty.so fails to load. 1.20 calls
the function in the probe.
perl-Net-DNS stays out. An EL10 service node needs it, this tree holds 0.80, and 0.80 has no KEY
resource record that xCAT's ddns.pm builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A service node reads the OS install tree copycds made, BaseOS and AppStream, plus xcat-core and
xcat-dep. It reaches no CRB and no EPEL. xCAT-server requires perl-IO-Tty, perl(IO::Pty),
perl-Crypt-CBC, perl-Crypt-Rijndael, perl(Expect) and perl(HTML::Form), which EL keeps in CRB or
in EPEL, so dnf install xCATsn on the service node reports "nothing provides" and the node ends
with no xcatd.
packages-manifest.conf builds those packages for the riscv64 target only, because riscv64 has no
EPEL. The other EL targets take them from EPEL at build time and never publish them.
mockbuild-perl-packages.pl also has no entry for perl-IO-Tty, so no target can build it.
servicenode_perl_deps.t records the set per EL release, asserts every EL target section lists it,
and asks the builder whether it can build each name. The seven section assertions and the two
perl-IO-Tty assertions fail. The assertions for the packages already listed pass, so the test
measures the manifest and not its own lookup.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
_log set $| on the caller's selected handle for each line. The caller
owns its buffering, so the lock log only prints the line.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build log did not show when a lock was taken, waited on, taken over
or released, so the lock order of the parallel dep builds could not be
read from their logs.
XCAT::NFSLock now prints one line per event to the selected output
handle, with the UTC time in milliseconds, the host, the pid, the event,
the label and the lock path. The events are acquired, took-over, wait,
released and release-skipped. quiet => 1 turns the log off for a lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The apt publish took only the run lock of the host arch, and then read
the staging of every expected arch. A ppc64el run could refill its
staging while the publish assembled from it.
Finalize ran on one host and took the cell locks of both arches. If it
died, only that host could reclaim the locks of the other arch's cells,
and the next builds of those cells waited on them and failed.
A publish now takes the run lock of every expected arch, in name order,
before the publish lock, and waits for them up to --publish-lock-wait.
--finalize-arch limits finalize to the cells of one arch: it writes,
locks, re-indexes and verifies only those, and reads the other arches.
Without the option, finalize writes both arches as before.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The apt publish reads staging/<codename>/<arch> of every expected arch,
but takes only the run lock of the host arch. A ppc64el run can refill
its staging while the publish assembles from it.
Finalize runs on one host and takes the cell locks of both arches. If it
dies, only that host can reclaim the locks of the other arch's cells,
and the builds that own those cells wait on them and fail.
These tests fail until the fix: the publish must refuse while any
expected arch holds its run lock, and finalize --finalize-arch must
write and lock only the cells of that arch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
XCAT::NFSLock recovered a dead owner's lock under a second lock,
lock.break. A process that died after it created lock.break left it in
place, and every later acquire timed out even when both processes were
dead. The lock and its record were also put in place with rename, which
replaces an empty directory, so acquire took over a lock.d that had no
owner record.
The lock now follows the NFS lock protocol at the top of NFSLock.pm. It
uses mkdir, rmdir, unlink and plain writes, and no rename. lock.d/metadata
holds the owner identity and a hash, and invalid metadata only causes a
retry. Takeover and release both hold lock.borrow, beside lock.d. Only a
process on the owner's machine takes over a lock, after it reads the
same identity twice and proves that owner dead. acquire retries R times
with a wait of T +/- jitter, and --try-unlock-timeout maps to R.
Records written by the previous format are invalid metadata and are not
taken over. A process that dies while it holds lock.borrow still blocks
the lock. The protocol assumes that this does not happen.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
payload.sha256 differs from the bare release tree in the two shims, and
matches the tree once the committed ipxe/shim shims replace them, as the
builders install them. SHA256SUMS covers the shims with the archives.
The rpm and the deb install ipxe-shimx64.efi and ipxe-shimaa64.efi over
x86_64-sb/shimx64.efi and arm64-sb/shimaa64.efi of the release tree, so
firmware that trusts only the Microsoft UEFI CA 2023 loads the shim
with Secure Boot on. The ipxe-shim.efi and snponly-shim.efi links keep
their targets, and every other file of the tree is unchanged.
payload.sha256 lists the digests of the new shims, which both builders
check. The README says how to update them.
The next commit installs the ipxe/shim 16.1 shims over the shims of the
release tree, so payload.sha256 no longer describes the bare tree. The
commit after it checks the tree with the shims in place.
The shims in the iPXE v2.0.0 release tree carry only the Microsoft UEFI
CA 2011 signature, so firmware that trusts only the UEFI CA 2023 refuses
them with Secure Boot on. ipxe-shimx64.efi and ipxe-shimaa64.efi are the
ipxe/shim ipxe-16.1 release assets that ipxe replaced on 2026-05-27 with
a build signed by both CAs. SHA256SUMS holds the digests that GitHub
publishes for them.
The per-arch runs of one dep build share --repo-dep and each took
<repo-dep>/.lock. The first run won and the others died, so the
pipeline passed --force-unlock. That option removed any lock it found:
a rerun of the same refs could take a cell from a run still writing it,
and common recovery could remove the staging tree of a live publisher.
mockbuild-all.pl now takes XCAT::NFSLock locks: <output>/.lock, one
<repo-dep>/rh<N>/.<arch>.lock per target, and .common-publish.lock while
it recovers or publishes common. --finalize-xcat-dep locks the cells it
rewrites. --try-unlock-timeout N replaces --force-unlock. A run that is
not a dry run recovers an interrupted common publication when no other
run holds the common lock.
sbuild-all.pl takes its per-arch run lock and its apt publish lock as
XCAT::NFSLock locks instead of flock, which fails on the shared tree.
createrepo_c runs without --database. SQLite needs locks, which a client
of an NFS re-export cannot take.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The dep builds lock trees on a shared NFS tree that the hypervisors
re-export. The kernel refuses flock and fcntl locks to clients of a
re-export, so every such lock fails with errno 524.
XCAT::NFSLock builds a lock as a directory under a private name, with an
owner record and the caller's metadata, and renames it into place. The
record holds the machine id, boot id, pid, process start time and a
random token. The owner removes the lock, or a process on the owner's
host that proves the owner dead: another boot, no such pid, or a pid
with another start time. That process takes a breaker lock and proves
the owner dead again before it removes the lock. Removal renames the
lock away first. Any other acquire waits up to a timeout, retrying every
3 s by default, then fails with the mv command that moves the lock away.
A lock left by mockbuild-all.pl before this module is taken when its
pid is gone on this host.
t/nfslock.t covers each rule without NFS.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Timothy Middelkoop
Vinícius Ferrão
Vinícius Ferrão