mockbuild-all.pl builds a perl package with mockbuild-perl-packages.pl when the
native catalog declares it an owner. That script runs as root and creates the
directories it gives mock as --resultdir. mock 6.8 reloads its uid manager from
chrootuid and opens state.log, build.log and root.log in --resultdir as that
uid, so each directory must be writable by it. No test reads the owner or the
mode of those directories.
The test checks the uid read from a mock configuration, then writes a state.log
in each result directory as that uid. The write runs in a forked child that
exits 2 when setuid fails, so a write made as root cannot read as a pass.
The result-directory checks need root, so the workflow runs the file under sudo.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT already treats aarch64 as a target: BootPolicy hands an aarch64 node
boot/grub2/grub2.aarch64, and grub2.pm aborts the node's grub configuration
when that file is missing. xcat-dep neither ships the loader nor has an
aarch64 manifest section, so mockbuild-all.pl stops at "no manifest section
for target 'alma+epel-10-aarch64'" and an aarch64 node cannot netboot.
grub2-xcat generates grub2.aarch64. riscv64 ships the distro grub2 UEFI
image unchanged, but the EL grubaa64.efi carries the distro prefix
(/EFI/<vendor>) and never finds the boot/grub2/grub.cfg-<HEXIP> that
nodeset writes. %build runs grub2-mkimage -O arm64-efi -p /boot/grub2, the
prefix grub2.ppc uses, from the noarch grub2-efi-aa64-modules, so every EL
build host produces it. efi_uga stays out of the module list: it is x86
only and grub2-mkimage fails on it for arm64-efi. The image ships as
aarch64-efi/grubaa64.efi and %post copies it to grub2.aarch64, the boot file
of DHCP client architecture 11. The openEuler x86_64 repositories carry no
grub2-efi-aa64-modules and openEuler has no aarch64 target, so that build
ships without the image. mockbuild.pl checks the payload holds an AArch64
PE image. Release 3 -> 4.
[alma+epel-10-aarch64] is a native EPEL target like x86_64 and ppc64le, not
a forcearch cross-build like riscv64, and lists what [alma+epel-10-x86_64]
lists: the EL10 service node perl set (t/servicenode_perl_deps.t), and the
noarch x86 boot loaders, because an aarch64 management node serves the x86
nodes of a mixed cluster as a ppc64le one does.
Those loaders now build natively on aarch64 the way they do on ppc64le.
elilo-xcat ships its tracked prebuilt EFI payload, as on ppc64le: elilo
cannot compile on aarch64. syslinux-xcat admits aarch64 in ExclusiveArch and
takes the ppc64le path, installing the prebuilt bootloader blobs of the
release tarball. Both noarch rpms built on aarch64 carry the same payload,
byte for byte, as the published rh10/ppc64le ones. x86_64 and ppc64le
builds are unchanged.
t/aarch64_target.t holds the manifest cell to the EL10 x86_64 one and the
loader pins to EL10 ppc64le, checks how the spec generates, ships, installs
and removes grub2.aarch64, and where grub2-mkimage and the arm64-efi
modules are installed, runs the spec's own command, checks the image is
AArch64 with a /boot/grub2 prefix, and checks that adding efi_uga fails.
aarch64 is EL10 only and is not in the --finalize-xcat-dep cross-arch
genesis copy; BUILD.md says so.
Validated: a full mockbuild-all.pl --target alma+epel-10-aarch64 on an EL10
aarch64 host builds and verifies all 22 required packages (7 minutes).
Signed-off-by: Timothy Middelkoop <tmiddelkoop@internet2.edu>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No build script, manifest or test uses gpxe/, the gPXE 0.9.7 to 1.0.1
sources and patches, or ipxe/, an iPXE snapshot from 2010. No published
package comes from them. The xNBA builds take their source from xnba/,
and ipxe-xcat repackages the iPXE release from ipxe-xcat/.
The openeuler-24.03-ppc64le catalog pins perl-Net-DNS-1.40-1.oe2403
from openEuler 24.03 LTS GA as a publisher input. That release has
CVE-2026-64193 and CVE-2026-64194. The GA srpm carries no fix, and
load_inputs accepts only GA rpms as publisher inputs.
Build perl-Net-DNS for this target from perl-Net-DNS/Net-DNS.spec,
the 1.57 release that rocky-10-riscv64-xcat already builds. The
catalog now declares perl-Net-DNS as an owner, load_inputs allows that
owner, and the manifest pins 1.57. The spec BuildRequires all resolve
from the openEuler 24.03 LTS OS repository.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
openeuler-24.03-ppc64le ships the openEuler GA rpm perl-Net-DNS-1.40.
That release has CVE-2026-64193 (EDNS EXTENDED-ERROR code injection) and
CVE-2026-64194 (unbounded compression-pointer chains). Net::DNS 1.56
fixes both.
The test now reads the release each target ships: the version of the
pinned publisher rpm, or else the version the spec builds. It fails
when that release is below 1.56. It fails on openeuler-24.03-ppc64le
until that target builds Net::DNS from the xcat-dep spec.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
On an EL10 host, verify-xcatsn-install.sh failed for EL8 with "nothing
provides /usr/libexec/platform-python needed by nfs-utils". EL8 packages
carry file dependencies that are only in filelists.xml, and dnf 4.19 and
later do not load filelists.
The check now sets optional_metadata_types=filelists. The usage text also
says how to import the AlmaLinux 8 key, which the rpm of an EL10 host
rejects for its SHA-1 bindings.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
trust_dbpath named the keyring after a hash of the staging directory and
did not create it. A directory or symlink that another user put at that
predictable name under the local root became the rpmkeys --dbpath for
the publisher key.
trust_dbpath now creates a fresh 0700 directory with tempdir under the
same root, so XCAT_DEP_TRUST_TMP still selects it. CLEANUP removes it
when the creating process exits. stage_inputs and build_native_inputs
run in that process, so the keyring stays in place for every
verify_input of the run.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
trust_dbpath answers the same name under the local root for one staging
directory, and does not create it. A directory or symlink that another
user puts at that name is then used as the publisher keyring.
The test plants a symlink at the path the first call answered. It
expects a second call to answer a fresh 0700 directory under the root,
and the XCAT_DEP_TRUST_TMP override to still pick the root.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
An air-gapped EL service node reads BaseOS, AppStream, xcat-core and
xcat-dep, and no other repository. Nothing checked that xCATsn resolves
from that set, so a dependency that only CRB or EPEL carries was found on
the service node.
verify-xcatsn-install.sh runs dnf --installroot into an empty directory
with only those four local repositories. It disables every configured
repository, refuses a URL as input, and fails when dnf loads a repository
that is not file://. Package signatures are checked against the keys given
with -k. The installroot is removed on exit.
t/verify-xcatsn-install.bats tests the argument handling, the local
repository guard and the cleanup with dnf and rpm stubbed.
package-tests runs it.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The comments in packages-manifest.conf and t/servicenode_perl_deps.t no
longer match the code. They say xCAT-server requires perl-DB_File on EL10
and that an EL10 service node takes it from EPEL. xCAT-server only
recommends perl-DB_File on EL10, and a service node reads no EPEL. The
riscv64 comment says the EL10 sections take perl-HTML-Form and the other
EPEL perl packages from EPEL, but those sections now list them.
Remove the per-release package table, which repeats the set the test
holds, and the perl-DB_File notes. Name the four packages that only the
riscv64 target builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
openeuler-20.03sp4-x86_64 and openeuler-22.03sp4-x86_64 build nothing. Both exit 3
at "Could not find included config file:
/etc/mock/templates/openeuler-20.03-sp4.tpl", and both targets report that every
build step failed. mock-core-configs names its openEuler templates by release
alone, openeuler-20.03.tpl and openeuler-22.03.tpl, and each already sets
releasever to the newest service pack. The two configurations spelled the service
pack into the template name, which matches no file in any mock package.
openeuler-24.03sp4-x86_64.cfg already includes openeuler-24.03.tpl, which is why
that target builds.
The ppc64le target fails before any package is built, for two further reasons.
rpmkeys --dbpath cannot import the publisher key: rpm takes an fcntl transaction
lock on <dbpath>/.rpm.lock, the staging directory is NFS mounted local_lock=none,
and that lock answers errno 524 there. The keyring database moves to local
temporary storage, keyed on the staging directory so two targets of one run do not
share it. The staged source RPMs and the gpg home do not move.
A verified publisher RPM is rejected as unreadable: reject_elf_payload read the
exit status of rpm2cpio as the verdict on the payload, and on rpm 4.19 rpm2cpio
writes the whole payload and still exits 1 with nothing on stderr. The cpio walk
moves into scan_cpio_for_elf, which returns its own answer. A truncated stream
never reaches the trailer and still dies, which is the case the exit status stood
in for, and verify_input has already matched the sha256 and run rpmkeys --checksig
before this point.
Each fix has a test that fails without it. openeuler_mock_template_include.t reads
the configurations only. openeuler_native_trust_dbpath.t and
openeuler_native_cpio_scan.t run no rpmkeys and no rpm2cpio: the second builds its
cpio streams in Perl and refuses an ELF member, a truncated stream, data after the
trailer and a stream that is not cpio, each by name.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT-server requires perl-Net-DNS. No AlmaLinux 10 repository carries it, and
neither does the AlmaLinux 10 DVD that copycds copies, so dnf install xCATsn on
an EL10 service node reports "nothing provides perl-Net-DNS". Only EPEL has the
package, and a service node reads no EPEL.
List perl-Net-DNS=1.57 for [alma+epel-10-x86_64] and [alma+epel-10-ppc64le].
The riscv64 target already builds that version from the same spec, so the EL10
targets get the Net::DNS that carries the KEY record ddns.pm builds. A
management node keeps the EPEL copy, which is 1.57-1.el10_3 and outranks
1.57-1, so nothing changes there.
The comment above @epel_gap_order in mockbuild-perl-packages.pl said the
EPEL-fed targets keep taking these packages from EPEL. mockbuild-all.pl passes
--packages from packages-manifest.conf, so the manifest decides, not that list.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
t/servicenode_perl_deps.t left perl-Net-DNS out of the EL10 service node set.
The recorded reason was that this tree held Net-DNS 0.80, which has no KEY
record, so shipping it would replace a working copy with a broken one. The tree
now builds 1.57 and that record is present.
No AlmaLinux 10 repository carries perl-Net-DNS, and the AlmaLinux 10 DVD that
copycds copies carries neither perl-Net-DNS nor libdb. xCAT-server requires
perl-Net-DNS, so dnf install xCATsn on an EL10 service node does not resolve.
Add perl-Net-DNS to the EL10 set. The test fails for [alma+epel-10-x86_64] and
[alma+epel-10-ppc64le] until the manifest lists the package. perl-DB_File stays
out: xCAT-server requires it outside riscv64, EL10 dropped libdb, and xcat-dep
has no source to build it from.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
mock --init failed for every openEuler target. repo.openeuler.org is the only
source the chroots have, and it served a build host at 7 kB/s, then reset
connections mid-transfer: "Curl error (28) Operation too slow", "Curl error
(56) SSL_read: SSL_ERROR_SYSCALL", "Curl error (92) INTERNAL_ERROR". All three
attempts per package failed, so the openEuler core build could not start.
mock-configs/templates/openeuler-lts-xcat.tpl builds dnf.conf, so baseurl now
lists two mirrors before repo.openeuler.org, which stays last as the authority
of record, and timeout rises to 300. minrate stays at dnf's default: the rate
abort is the only signal that moves dnf to the next baseurl. gpgcheck stays on
and gpgkey still names repo.openeuler.org, so a mirror cannot change what is
built. Measured on one host, same gcc rpm: repo.openeuler.org 7 kB/s,
mirrors.tuna 2.8 MB/s, mirror.nju 3.9 MB/s. A cold mock --init then completed
in 2m38s.
native/mock-configs.t asserts the slowest host is not first, the canonical url
is present, every url names the same release and architecture, and minrate is
unset. It could not run before: native/fixtures/mock-configs.py copied two
templates no source ships and raised FileNotFoundError, so the test measured no
config at all.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
t/net_dns_rr_types.t compared every perl-Net-DNS pin in the manifest
with the version of perl-Net-DNS/Net-DNS.spec. openeuler-24.03-ppc64le
does not build that spec. Its input catalog pins the openEuler
perl-Net-DNS-1.40 rpm as a publisher input, so its pin of 1.40 failed
against the spec's 1.57, and package-tests is red on master.
For a target whose catalog publishes perl-Net-DNS, compare the pin with
the version in the pinned rpm name. Other targets still compare with
the spec.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The review asked for Net::DNS 1.57 with the tarball, the spec and the
manifest pin in sync. No test held either. A 1.56 tarball passed every
check, although 1.56 recurses without bound when it re-encodes a reply
with a misplaced TSIG record (rt.cpan.org #181125, fixed in 1.57). A
second Net-DNS tarball beside the spec, or a Buildnote that names
another release, also passed.
Decode such a reply with the shipped Net::DNS and re-encode it, with
Packet::encode wrapped to stop and record recursion past 20 levels.
Also check that perl-Net-DNS/ holds only the tarball the spec builds,
and that the Buildnote names that tarball.
The recursion check fails on 1.56. The two sync checks fail on a tree
that keeps Net-DNS-0.80.tar.gz and the 0.80 Buildnote.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
perl-Net-DNS built Net::DNS 1.47 without patches. That release decodes
the EDNS EXTENDED-ERROR text with a string eval (CVE-2026-64193) and
follows compression pointer chains of any depth (CVE-2026-64194).
Net::DNS 1.56 fixes both, and 1.57 also stops an unbounded recursion on
a misplaced TSIG.
Build Net::DNS 1.57. The tarball, the spec, the Buildnote and the
riscv64 manifest pin move together. The build requirements of 1.57 are
the same as those of 1.47.
t/net_dns_rr_types.t decodes a reply with a 200-pointer chain and fails
on 1.47.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
perl-Net-DNS builds Net::DNS 1.47. That release decodes a reply whose
owner name is a chain of 200 compression pointers, one recursion per
pointer (CVE-2026-64194). Net::DNS 1.56 stops the chain with "deep
compression recursion".
The test decodes such a reply with the Net::DNS from the shipped tarball
and expects that error. It fails on 1.47.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Exercise mixed repository finalization and changed publisher inputs on x86_64. Check native manifest selection and use File::Slurper in the goconserver fixture.
Exclude native cells from cross-architecture Genesis finalization. Add the missing 24.03 SP1 manifest and expose repository signing for direct validation.
dnf install xCATsn on an EL service node does not resolve. Against the repositories a service
node has -- BaseOS and AppStream from copycds, plus xcat-core and xcat-dep -- it reports
"nothing provides" for perl-IO-Tty, perl(IO::Pty), perl-Crypt-CBC, perl-Crypt-Rijndael,
perl(Expect) and perl(HTML::Form), and per release for perl-Crypt-SSLeay, perl-Net-Telnet and
perl-Digest-SHA1. The service node then has no xcatd, and every hierarchical case fails.
packages-manifest.conf assumed CRB and EPEL stand behind every EL target. That holds for a
management node and not for a service node, so those packages were built only for
rocky-10-riscv64-xcat, the one target with no EPEL. Each EL target section now lists the perl
packages BaseOS and AppStream do not carry.
mockbuild-perl-packages.pl had no table entry for perl-IO-Tty, so no target could build it. Its
sources move to IO-Tty 1.20 with a spec. In 1.12 the Makefile.PL pty probe only takes the
address of each candidate function, the current toolchain drops that reference, the build then
defines HAVE__GETPTY on a glibc that has no _getpty, and IO/Tty/Tty.so fails to load. 1.20 calls
the function in the probe.
perl-Net-DNS stays out. An EL10 service node needs it, this tree holds 0.80, and 0.80 has no KEY
resource record that xCAT's ddns.pm builds.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A service node reads the OS install tree copycds made, BaseOS and AppStream, plus xcat-core and
xcat-dep. It reaches no CRB and no EPEL. xCAT-server requires perl-IO-Tty, perl(IO::Pty),
perl-Crypt-CBC, perl-Crypt-Rijndael, perl(Expect) and perl(HTML::Form), which EL keeps in CRB or
in EPEL, so dnf install xCATsn on the service node reports "nothing provides" and the node ends
with no xcatd.
packages-manifest.conf builds those packages for the riscv64 target only, because riscv64 has no
EPEL. The other EL targets take them from EPEL at build time and never publish them.
mockbuild-perl-packages.pl also has no entry for perl-IO-Tty, so no target can build it.
servicenode_perl_deps.t records the set per EL release, asserts every EL target section lists it,
and asks the builder whether it can build each name. The seven section assertions and the two
perl-IO-Tty assertions fail. The assertions for the packages already listed pass, so the test
measures the manifest and not its own lookup.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
_log set $| on the caller's selected handle for each line. The caller
owns its buffering, so the lock log only prints the line.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build log did not show when a lock was taken, waited on, taken over
or released, so the lock order of the parallel dep builds could not be
read from their logs.
XCAT::NFSLock now prints one line per event to the selected output
handle, with the UTC time in milliseconds, the host, the pid, the event,
the label and the lock path. The events are acquired, took-over, wait,
released and release-skipped. quiet => 1 turns the log off for a lock.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The apt publish took only the run lock of the host arch, and then read
the staging of every expected arch. A ppc64el run could refill its
staging while the publish assembled from it.
Finalize ran on one host and took the cell locks of both arches. If it
died, only that host could reclaim the locks of the other arch's cells,
and the next builds of those cells waited on them and failed.
A publish now takes the run lock of every expected arch, in name order,
before the publish lock, and waits for them up to --publish-lock-wait.
--finalize-arch limits finalize to the cells of one arch: it writes,
locks, re-indexes and verifies only those, and reads the other arches.
Without the option, finalize writes both arches as before.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The apt publish reads staging/<codename>/<arch> of every expected arch,
but takes only the run lock of the host arch. A ppc64el run can refill
its staging while the publish assembles from it.
Finalize runs on one host and takes the cell locks of both arches. If it
dies, only that host can reclaim the locks of the other arch's cells,
and the builds that own those cells wait on them and fail.
These tests fail until the fix: the publish must refuse while any
expected arch holds its run lock, and finalize --finalize-arch must
write and lock only the cells of that arch.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
XCAT::NFSLock recovered a dead owner's lock under a second lock,
lock.break. A process that died after it created lock.break left it in
place, and every later acquire timed out even when both processes were
dead. The lock and its record were also put in place with rename, which
replaces an empty directory, so acquire took over a lock.d that had no
owner record.
The lock now follows the NFS lock protocol at the top of NFSLock.pm. It
uses mkdir, rmdir, unlink and plain writes, and no rename. lock.d/metadata
holds the owner identity and a hash, and invalid metadata only causes a
retry. Takeover and release both hold lock.borrow, beside lock.d. Only a
process on the owner's machine takes over a lock, after it reads the
same identity twice and proves that owner dead. acquire retries R times
with a wait of T +/- jitter, and --try-unlock-timeout maps to R.
Records written by the previous format are invalid metadata and are not
taken over. A process that dies while it holds lock.borrow still blocks
the lock. The protocol assumes that this does not happen.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
payload.sha256 differs from the bare release tree in the two shims, and
matches the tree once the committed ipxe/shim shims replace them, as the
builders install them. SHA256SUMS covers the shims with the archives.
The rpm and the deb install ipxe-shimx64.efi and ipxe-shimaa64.efi over
x86_64-sb/shimx64.efi and arm64-sb/shimaa64.efi of the release tree, so
firmware that trusts only the Microsoft UEFI CA 2023 loads the shim
with Secure Boot on. The ipxe-shim.efi and snponly-shim.efi links keep
their targets, and every other file of the tree is unchanged.
payload.sha256 lists the digests of the new shims, which both builders
check. The README says how to update them.
The next commit installs the ipxe/shim 16.1 shims over the shims of the
release tree, so payload.sha256 no longer describes the bare tree. The
commit after it checks the tree with the shims in place.
The shims in the iPXE v2.0.0 release tree carry only the Microsoft UEFI
CA 2011 signature, so firmware that trusts only the UEFI CA 2023 refuses
them with Secure Boot on. ipxe-shimx64.efi and ipxe-shimaa64.efi are the
ipxe/shim ipxe-16.1 release assets that ipxe replaced on 2026-05-27 with
a build signed by both CAs. SHA256SUMS holds the digests that GitHub
publishes for them.
The per-arch runs of one dep build share --repo-dep and each took
<repo-dep>/.lock. The first run won and the others died, so the
pipeline passed --force-unlock. That option removed any lock it found:
a rerun of the same refs could take a cell from a run still writing it,
and common recovery could remove the staging tree of a live publisher.
mockbuild-all.pl now takes XCAT::NFSLock locks: <output>/.lock, one
<repo-dep>/rh<N>/.<arch>.lock per target, and .common-publish.lock while
it recovers or publishes common. --finalize-xcat-dep locks the cells it
rewrites. --try-unlock-timeout N replaces --force-unlock. A run that is
not a dry run recovers an interrupted common publication when no other
run holds the common lock.
sbuild-all.pl takes its per-arch run lock and its apt publish lock as
XCAT::NFSLock locks instead of flock, which fails on the shared tree.
createrepo_c runs without --database. SQLite needs locks, which a client
of an NFS re-export cannot take.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Timothy Middelkoop
Vinícius Ferrão
Vinícius Ferrão