setfiles without -F only corrects the type of a file and keeps its user,
role and range. Files of a new build tree get the SELinux user of the
process that created them, so about 1800 of them stayed unconfined_u
instead of system_u. Label the new tree with -F. Pack keeps relabeling
without it, so contexts set on purpose in a captured or edited tree stay.
The kernel package writes a boot loader entry inside the build chroot.
Its linux and initrd paths point into the scratch directory of the
server (grub2-mkrelpath sees the bind mount) and its options are the
kernel command line of the server. Diskless boot never reads it, but
after installing to disk it sits next to the entry post.sh writes on
the installed system, so grub may pick the broken one. Remove the
entries when building, post.sh writes the right one when installing.
The _dbpath macro set for fingerprinting an image is process global. It
was only removed after a successful query, so an error left it set and
later fingerprinting in the same process, which fancy_chroot does on
every command, read the wrong database.
Without the rpm Python bindings, fingerprinting an image found its
system-release provider in the image database but then ran rpm -qi
against the host, so an image whose release package the host lacks was
not recognized. Use the same rooted rpm command for both queries.
DebHandler raised SystemExit on a failed apt-get, but prep_root runs
under run_constrained, which only catches Exception and otherwise exits
0. A failed package install was therefore reported as a successful
build. Raise an Exception instead, as ElHandler already does.
EL9 and older keep the rpm database in /var/lib/rpm, EL10 moved it to
/usr/lib/sysimage/rpm. fingerprint_host_el opened the image with the
default path of the host, so on an EL10 server an EL9 image was not
recognized (and an empty database was created in it). pack then wrote
no profile.yaml and no addons link, and import refused the image. Use
/var/lib/rpm when it is a real directory in the image. In EL10 it is a
relative link, so the default of the host still applies.
The SUSE kernel builds it as a module, so dhcpcd failed with "Address
family not supported" and a diskless node with ipv4_method dhcp never
got an address.
imgutil build --selinux installs the targeted policy and keeps SELinux enforcing.
imgutil pack labels images that enable SELinux with their own policy.
Building a SUSE 16 image from SLE media failed every package with
"key ID fec28eaf09d9ea69: NOKEY". Leap publishes that key as
gpg-pubkey-*.asc, which the existing glob picks up; SLE publishes the
same key only as repodata/repomd.xml.key, so nothing was imported.
15 media carries both spellings, so this changes nothing there.
udev's kmod builtin dlopens libkmod, and dracut installs it from an
inst_libdir_file line in a module-setup.sh rather than by following
NEEDED. Which module carries that line moved: the dracut on SLE 16
media declares it only in 00systemd, which the diskless module set
never loads, so the image came up with no libkmod, udev autoloaded
nothing, and the guest reached the network scan with only loopback.
Leap's newer dracut also declares it in 95udev-rules, which base
depends on, which is why Leap was unaffected.
OpenSSH 10 splits each connection into sshd-session and that into
sshd-auth, so the initramfs sshd on 2222 could not serve a single
session. el10 added sshd-session for the same reason.
SLES 16 ships no ISC dhclient and nothing provides dhcp-client, so the
image could not be built from SLES media at all. Leap carries dhcpcd
too, so one client covers both. el10 made the same move when RHEL
dropped dhclient.
A trailing comma made args.cmd a tuple holding the argv list, so
fancy_chroot called startswith on a list and the child died before exec.
permissions.local was written but never applied, leaving ssh-keysign
0755 and hostbased auth inoperative on SUSE diskless images.
- the aarch64 osdeploy spec builds the stateful suse16 addons but its
diskless loop was never extended, so the aarch64 rpm shipped suse16
without suse16-diskless and a packed image got a dangling addons.cpio.
- imgutil's builddeb keeps its own copy of the directory list that
confluent_imgutil.spec.tmpl has, and it had learned about neither suse16
nor el10.
- gather_bootloader gained a /usr/share/efi fallback for shim on both
architectures but only for x86_64 on grub, so an aarch64 root found a
shim and then died copying grub.
Finally, rewriting repos.d file by file rather than copying the tree meant
a subdirectory or a file that is not valid UTF-8 aborted the build before
any package was installed, which also regressed SUSE 15. Pass anything
that is not a plain text repo definition through untouched and restore the
modes on the ones that are rewritten.
SuseHandler refused anything but 15.x. What 16 needed beyond widening it:
- its repo urls are written in terms of ${releasever}, which zypper
resolves from the target root's os-release, a file that does not exist
yet when the first packages go in
- its repos name a zypper service backed by a package-provided directory
the target root does not have, so zypper discarded every one of them
as an orphan
- there is no mkinitrd to work out which kernel to build for, and bare
dracut would build for the build host's running kernel
- the efi payloads moved out of /usr/lib64/efi, arping out of /usr/sbin,
nsswitch.conf and protocols under /usr/etc, and the presets enable
sshd already
- the module list predated virtio, so an image built for a KVM guest had
no network at all, and dm-crypt could not allocate a transform for the
encrypted image without the aes-xts modules
- urlmount still links libpthread, an empty stub since glibc 2.34 that
nothing else in the initramfs pulls in
Hand written rather than autofixed, since three of the four need the
surrounding code read to be sure they are equivalent:
- confetty: `powerstate == None` -> `is None`.
- nodeconfig: `setmode != True` / `!= False` -> `not setmode` / `setmode`.
Safe because setmode only ever holds None, True or False, and the two
lines above each test normalise None away first.
- pam: split two `if cond: stmt` one-liners.
- imgutil: `from shutil import copytree as copytree`, an alias that renames
nothing. Not a re-export marker, this is a script.
imageboot falls back to cp when unsquashfs is unavailable, but the build
side did not: a bare dracut_install/copy_exec aborts initramfs generation
when the binary is absent, and the capture prerequisite check refused to
capture the image at all.
Mark the initramfs copies optional and report the missing package as an
advisory rather than a hard prerequisite, so such images still build and
capture, just without the faster extraction path. Widen the EL check to
every release past el8 so future ones inherit it.
Release tags do not live on master: 3.15.2 through 3.15.6 were tagged on branch
3.15, so git describe reaches only 3.15.1 and dev builds were stamped
3.15.2.dev<n>. Besides being confusing, rpm and dpkg both rank the released
3.15.6 above that, so a dev package will not install over a released one.
Add a top-level VERSION file naming the release the branch is working toward
(4.0.0 on master) and a mkversion helper that stamps packages from it, keeping
the tag-derived value as a floor so a forgotten bump cannot go backwards.
mkversion also replaces the block copy-pasted into seven build scripts, and
makesetup no longer writes a per-package VERSION file, so the stale checked-in
confluent_common/VERSION goes with it.
confluent_imgutil does not depend on confluent_server, and the yaml
import is optional too, yet both capture and pack dereference osimage
and yaml unconditionally when writing manifest.yaml. With
confluent_osdeploy present but the server absent that raises rather
than producing a profile.
Guard the manifest on both being importable and say so, since rebase
is what the manifest exists for. The yaml fallback now binds None
instead of leaving the name undefined.
The two call sites carried the manifest write verbatim in both, so fold
them into one function rather than duplicate the guard as well.
Nothing unwound the loop device and dm-crypt mapping when the copy loop
raised, so interrupting a pack stranded both, still holding the profile's
rootimg.sfs.
Tear them down from a finally. The retry loop moves with them, so also
honour its tries counter, as unpack_image already does; spinning forever
inside a finally would hang the interrupt it is meant to clean up after.
A bounded retry loop can also give up, and the detach that follows would
then fail with EBUSY and, raising from a finally, replace the exception
that brought us here. Warn and leave both in place instead.
Both functions became coroutines solely to await one get_hashes call,
but their bodies are long stretches of blocking work: mksquashfs, the
encrypt_image copy loop, rsync, ssh and osdeploy.
From Python 3.11 on, asyncio.run installs a SIGINT handler that cancels
the main task and returns rather than raising, so an interrupt is only
noticed at the next await. Interrupting a pack during mksquashfs
surfaced as a CalledProcessError from the dying child instead of a
KeyboardInterrupt, and with a base profile, where nothing is ever
awaited, pack carried on and published the profile before exiting.
Run the loop only around the call that needs it.
capture and pack hash the whole profile directory, which by that point
holds rootimg.sfs, the kernel and the distribution initramfs. rebase
only ever looks up entries that came from the profile source directory,
so the image blobs cost gigabytes of hashing for nothing.
Pass the source directory as the filter, as generate_stock_profiles
already does. Older manifests keep working, since rebase reads their
entries with a default.
`unsquashfs` can use multiple CPU cores during image extraction, significantly reducing boot time.
For example the whole boot time from PXE to shell on a 8-core VM, from approximately 45 seconds to 20 seconds.
This PR adds `squashfs-tools` as a dependency. Since the package is smaller than 1 MB, the additional image size is justified by the performance improvement.
For backward compatibility, the existing `cp`-based extraction method is used when `unsquashfs` is unavailable, such as with images built before this change.
The extraction logic has also been moved into the common functions and is now shared between EL9, EL10, and Ubuntu.
Both untethered `squashfs` images and `confluent_multisquash` images are supported.
Images must be rebuilt to include `unsquashfs` and benefit from the faster extraction path.
EL10 diskless networking uses dhcpcd and no longer dhclient and ipcalc.
Keep the existing dhclient and ipcalc requirements for older EL capture targets.
Centralize the SELinux chcon helper and use it for downloaded
systemd units, onboot hooks, and apiclient files across EL7 through EL10.
Include chcon in captured EL initramfs images.
Without this fix the onboot services failed to start on SELinux enabled
captured image.
Allow building EL and Ubuntu diskless images for a foreign architecture (e.g.
aarch64 on an x86_64 host) by leveraging qemu-user-static. The target
architecture is detected automatically from a -s source tree (for EL),
or may be requested explicitly with the new --arch option.
When the target differs from the host, dnf/debootstrap is invoked with
--forcearch/--arch and the presence of an enabled binfmt_misc handler
with the F (fix-binary) flag is verified up front, so emulation keeps working
inside the installroot chroot and a missing setup yields an actionable
error instead of a confusing exec failure mid-build.
The image architecture is recorded in confluentimg.buildinfo so that
pack selects the initramfs addons for the image architecture rather
than the build host, and exec of a foreign-arch root performs the same
binfmt check.
Jarrod Johnson
Markus Hilger
Jarrod Johnson