2
0
mirror of https://github.com/xcat2/confluent.git synced 2026-11-19 08:52:20 +00:00
Commit Graph

100 Commits

Author SHA1 Message Date
Markus Hilger fc63269dc7 Give every deployed node its own machine-id
Captured images carried the machine-id of the source system, and images
built with imgutil carry the one generated in the build root, so every
node deployed from a profile shared it. That breaks anything keyed on it,
such as DHCP DUIDs and journal identities.

Capture now stores an empty /etc/machine-id. Diskless boot on EL and SUSE
writes a stable per-node one derived from the node name and the DMI
product UUID, as Ubuntu already does, using openssl since the initramfs
has no sha512sum. Install to disk copies that machine-id into the target
and renames the boot loader entries from the image to it. Otherwise
kernel-install in post.sh adds a second set of entries under another ID.
2026-11-04 15:06:58 +01:00
Jarrod Johnson 1f0202b68c Merge pull request #362 from Obihoernchen/fix/capture-optional-lvm
Make lvm optional for capture
2026-11-04 08:06:41 -05:00
Jarrod Johnson ae4bbe5300 Merge pull request #361 from Obihoernchen/fix/image2disk-localstorage-recapture
Fix install to disk of recaptured localstorage LVM nodes
2026-11-04 08:05:52 -05:00
Markus Hilger 244f75b0b2 Pass maxsplit to re.split as a keyword
ruff B034 flags the positional maxsplit argument.
2026-11-03 22:03:31 +01:00
Markus Hilger b2a40e1370 Skip deactivating volume groups when lvm is absent
lvm is optional in the capture initramfs now, so installimage printed
'lvm: command not found' on every install of a capture taken from a host
without lvm2. Only run it when it is there.
2026-11-03 17:18:46 +01:00
Markus Hilger 115de4c00e Check for the localstorage volume group first
The guess for volume groups named by the OS installer (<os>_<node>)
matched any device mapper name with an underscore followed later by a
hyphen. A localstorage volume for a mount point with a hyphen, such as
localstorage-_srv_my--data, was taken for one and got a broken volume
group name. Recognize volumes of the target volume group before trying
that guess.
2026-11-03 17:17:31 +01:00
Markus Hilger 04f45bf542 Number new partitions in creation order in image2disk
The plain partitions were addressed by their number on the source system,
while parted numbers them in the order they are created. A source with a
gap, such as swap between filesystems (it is not captured), made
image2disk format the wrong partition and look for swap on one that does
not exist. Count the partitions as they are created instead.

Also look up filesystem UUIDs through one helper.
2026-11-03 17:17:11 +01:00
Markus Hilger b5a58ef770 Install images without LVM to disk
image2disk.py formatted the target filesystems, copied the image and
ran fixup only in the branch that sets up LVM. For an image whose
filesystems are all plain partitions it created the partitions and
returned success without writing anything, and installimage then failed
on the missing target.

Run that part for both layouts. Plain partitions also need their swap
in fstab by UUID instead of the localstorage-swap volume, and on EL9
and later root= and resume= refer to the new root and swap by UUID as
well, since partition names are not stable with several disks.
2026-11-03 16:53:21 +01:00
Markus Hilger 3bd3ebeadb Install recaptured images of LVM installed nodes to disk
image2disk.py puts the volumes of a node it installs into the volume
group localstorage. Capturing such a node and installing the profile to
disk again failed with 'No mapping defined to create volume group
localstorage': volumes named localstorage-root were taken for volumes of
an extra volume group that needs volumegroupmap.yml.

Treat volumes of a volume group named like the target one as main volume
group volumes, and split device mapper names into volume group and
logical volume names correctly, including hyphens that device mapper
doubles inside either name.
2026-11-03 16:51:59 +01:00
Markus Hilger e021d29afc Keep image2disk partitions inside the disk
When the largest filesystem of a captured image is a plain partition,
it was sized as the disk minus the other filesystems. That ignored the
2048 sector gap before the first partition and the rounding of every
partition up to a 2048 sector boundary, so the last partition always
ran past the end of the disk. The clamp then set its end to the sector
count, one past the last sector and inside the backup GPT, and parted
refused it with 'The location ... is outside of the device'. Images
with the largest filesystem in LVM were not affected.

Reserve that overhead plus 1MiB at the end of the disk for the backup
GPT, and clamp to that last sector instead. Also round the swap
partition by its own size rather than by the previous partition's.
2026-11-03 15:45:52 +01:00
Jarrod Johnson 016db810ea Merge pull request #345 from Obihoernchen/fix/diskless-stop-on-extract-failure
Stop a diskless boot when the root filesystem cannot be extracted
2026-11-02 07:49:09 -05:00
Markus Hilger 5d97bc6a19 Stop a diskless boot when the root filesystem cannot be extracted
extract_untethered_rootimg reports failure, for example when the image
does not fit into the memory backed root, but imageboot.sh ignored it and
booted into the partly extracted root, which then hung later on. Stop
with a message saying why instead.
2026-11-01 22:16:57 +01:00
Jarrod Johnson 26f27da575 Merge pull request #343 from Obihoernchen/fix/image2disk-bls-kernel-paths
Point boot entries at the kernel on the installed boot filesystem
2026-11-01 13:32:38 -05:00
Markus Hilger 1d8d4f3077 Point boot entries at the kernel on the installed boot filesystem
image2disk rewrote root= in the boot loader entries but kept their
linux and initrd paths. Those come from where the image was built: in a
diskless image from imgutil they carry the scratch directory of the
server (grub2-mkrelpath sees its bind mount), and a /boot/ path does not
match the separate /boot filesystem image2disk creates for such images.
grub then fails with "you need to load the kernel". Make the paths
relative to the boot filesystem of the target.
2026-11-01 15:02:43 +01:00
Markus Hilger 0839299bd4 Fall back to other managers for untethered images
An untethered or uncompressed diskless boot downloaded the root image
only from the deployment server and retried it forever when that server
failed. Try the other managers that answered discovery as well, as the
tethered urlmount already does.

urlmount limits each small range request to 10 seconds, but the single
whole-image download cannot take a total limit. Give up on a connect
after 10 seconds and on a transfer that stalls for 30 seconds instead,
so a hung server moves on to the next one.
2026-11-01 12:09:06 +01:00
Jarrod Johnson 9186b6418a Merge pull request #329 from Obihoernchen/fix/installtodisk-selinux-relabel
Label disk installs of diskless images for SELinux at the end
2026-10-30 17:25:55 -04:00
Markus Hilger 5175e29a0a Remove the diskless dracut config before SELinux labeling
Removing it afterwards copied /etc/dracut.conf.d up unlabeled in
tethered mode on images built with SELinux. Skip the whiteout that the
removal leaves in the upper layer.
2026-10-30 20:27:53 +01:00
Markus Hilger e0be3ce5f7 Label disk installs of diskless images for SELinux at the end
The install runs before a policy is loaded, so the module index from
kernel-install and files of later post scripts stayed unlabeled and
systemd-modules-load failed. Relabel every local filesystem last.
2026-10-30 18:30:01 +01:00
Markus Hilger 63cdb6b2ed Label untethered diskless roots for SELinux
On an untethered root, files the initramfs writes before policy load
and sockets that unsquashfs creates without attributes have no label.
Label every inode without one, as the upper layer is in tethered mode.
2026-10-30 15:53:03 +01:00
Jarrod Johnson 58906b00cc Merge pull request #324 from Obihoernchen/fix/installtodisk-untethered
Keep the image mounted when installing a diskless image to disk
2026-10-30 08:02:14 -04:00
Markus Hilger dc909e6f0b Label overlay upper layer of diskless images for SELinux
With a tethered root (and on EL8 always), the initramfs writes the
overlay upper layer before the SELinux policy is loaded, so those files
are unlabeled_t. On a captured enforcing image sshd-keygen then cannot
search / and the node comes up without sshd.

Label exactly the upper layer paths with matchpathcon and chcon.
setfiles would recurse into the merged tree and restorecon is a no-op
before policy load. Built images have SELinux disabled and skip this.
2026-10-30 12:43:30 +01:00
Markus Hilger 8bec81fb95 Keep the image mounted when installing a diskless image to disk
With installtodisk and confluent_imagemethod=untethered (or
uncompressed) imageboot.sh first extracts the image to RAM and tears
down the image mount and the cryptimg device. installimage then hands
the empty /mnt/remote to image2disk.py, which wipes the target disk and
fails with:

  FileNotFoundError: [Errno 2] No such file or directory: '/dev/mapper/cryptimg'

Writing the image to disk never needs the RAM copy, so treat the image
method as tethered when installtodisk is set. The same profile can then
be switched between untethered and installtodisk by only adding or
removing installtodisk.
2026-10-30 09:55:43 +01:00
Jarrod Johnson 1777c80969 Add retry to image fetch 2026-10-21 08:54:51 -04:00
Jarrod Johnson fc1790ed80 Fix failing to reference converted LVM name 2026-09-11 10:23:38 -04:00
Jarrod Johnson 8fcb5aec9a Avoid exiting on tail failure
tail can fail in certain scenarios.  Switch to wait if that should occur.
2026-09-04 08:39:05 -04:00
Jarrod Johnson 76aef703ff avoid moving firmware directories if they don't exist 2026-08-07 15:42:10 -04:00
Jarrod Johnson c3cf2a402f Remove redundant copies of syncfileclient 2026-08-03 08:48:28 -04:00
Jarrod Johnson 3501f70c37 Merge pull request #247 from Obihoernchen/unsquashfs
Use multi-threaded unsquashfs to extract untethered images
2026-07-28 08:59:38 -04:00
Markus Hilger 67e84f15f8 Keep the root filesystem guard reachable when extraction fails
source_remote imageboot.sh is the last thing the diskless cmdline hook
runs, so returning early on a failed extraction ended the hook and left
dracut to time out. Falling through instead reaches the existing
/sysroot/sbin/init guard, which reports the failure and holds the node so
it stays reachable over ssh, as it did before extraction was checked.
2026-07-28 01:49:38 +02:00
Jarrod Johnson 5136b95cde Adjust to EL10 grub stub cfg
The syntax changed, make the code more adaptive to a variety of situations.
2026-07-20 17:11:19 -04:00
Markus Hilger 9101b07d54 Report stateless boot completion via new 'booted' status
Diskless profiles had the updatestatus callback in onboot.sh commented
out because no suitable status existed: 'complete' clears
deployment.pendingprofile, which the PXE responder requires to answer
the next network boot of a diskless node.

Add a 'booted' status that records the pending profile as
deployment.profile while leaving pendingprofile armed and skipping
autolock, and enable the onboot.sh callback in all diskless profiles.
nodedeploy now shows 'pending: <profile> (booted)' for a running
stateless node.
2026-07-20 15:59:30 +02:00
Jarrod Johnson 12ef3fc529 Merge pull request #245 from Obihoernchen/selinux
SELinux label diskless runtime files on EL
2026-07-16 20:10:43 -04:00
Markus Hilger b6fb58b31f Skip add_local_repositories if no imgutil build --source is set
BUILDSRC is only set if imgutil build is run with --source, otherwise
the build host repos are used. If --source is not used, there is no
distribution symlink and add_local_repositores failed with 404.
Check if BUILDSRC is set and skip add_local_repositories if this is the
case.
2026-07-17 00:06:57 +02:00
Markus Hilger cfc4490fe1 Use multi-threaded unsquashfs to extract untethered images
`unsquashfs` can use multiple CPU cores during image extraction, significantly reducing boot time.
For example the whole boot time from PXE to shell on a 8-core VM, from approximately 45 seconds to 20 seconds.

This PR adds `squashfs-tools` as a dependency. Since the package is smaller than 1 MB, the additional image size is justified by the performance improvement.

For backward compatibility, the existing `cp`-based extraction method is used when `unsquashfs` is unavailable, such as with images built before this change.

The extraction logic has also been moved into the common functions and is now shared between EL9, EL10, and Ubuntu.

Both untethered `squashfs` images and `confluent_multisquash` images are supported.

Images must be rebuilt to include `unsquashfs` and benefit from the faster extraction path.
2026-07-16 21:24:58 +02:00
Markus Hilger c3c4805f9d SELinux label diskless runtime files on EL
Centralize the SELinux chcon helper and use it for downloaded
systemd units, onboot hooks, and apiclient files across EL7 through EL10.
Include chcon in captured EL initramfs images.

Without this fix the onboot services failed to start on SELinux enabled
captured image.
2026-07-16 19:52:52 +02:00
Jarrod Johnson 0645788b0d Rework functions to be common file 2026-06-12 16:06:06 -04:00
Jarrod Johnson 1042404a38 Move getinstalldisk for all the Linux into a common place
esxi is the only one with a different version
2026-06-12 07:45:41 -04:00
Jarrod Johnson 98cbd7581a Fix diskless profiles for chrony.conf modification 2026-04-30 10:44:28 -04:00
Jarrod Johnson 779b07d2c2 Only try to use ssh_keys if it exists
EL10 changed from using ssh_keys and setgid to just
do setuid root instead.
2026-03-25 12:56:16 -04:00
Jarrod Johnson 9b00fe5521 Don't try to open a file that doesn't exist 2026-03-17 13:03:18 -04:00
Jarrod Johnson d20c5ac6eb Move handling of the loop directio straight to onboot
There were difficulties in the devfs after
boot, just let the full system handle it.
2025-11-13 15:33:04 -05:00
Jarrod Johnson 4484216198 Fix issues with the tethered memory optimizations 2025-11-13 15:24:26 -05:00
Jarrod Johnson e1efd6a9c5 Implement new 'uncompressed' image method
This allows the FS to just live, uncompressed, in cache.

This is generally a bad idea, however:

- In a hypothetically super-tuned diskless image, the lack of double-cache can offset the lack of compression
- The image will have supreme read performance
- It will have the most deterministic memory behavior
2025-11-13 14:39:53 -05:00
Jarrod Johnson 53c918042a Remove double-caching in tethered diskless
By default, the squashfs file was being cached as well as the contents after extraction.

This is superfluous pressure on the cache of the OS.

However, it does help keep the image afloat through 'confignet', so
leave it on until onboot completes, then reclaim cache and disable further caching.
2025-11-13 14:28:25 -05:00
Jarrod Johnson 20292cdfd0 Do not let diskless.conf persist into EL9 diskless images
It fouls run of kdump building the kdump image.
2025-11-07 13:22:21 -05:00
Jarrod Johnson 6c0d7ea60e Simplify end untethered el9 diskless environment
Rather than treat both as the same, since untethered has everything up front anyway, go ahead and extract the filesystem.

This makes the mount look more straightforward and makes it so deletion of files from
the image also frees ram.
2025-11-04 11:14:52 -05:00
Jarrod Johnson 6938bba2d3 Have confignet pause until connectivity restored
If we are reconfiguring network for a diskless node, wait for
things to settle back in before continuing.
2025-09-26 13:42:29 -04:00
Jarrod Johnson 5905510a32 Move tmp script execution out of /tmp
Some environments want noexec on /tmp, this will work in such environments.
2025-08-28 08:34:07 -04:00
Jarrod Johnson c26fba74e7 Fix issues with EL10 installtodisk 2025-08-26 09:52:21 -04:00
Jarrod Johnson a6a57e8590 Fix ssh operation during install in installtodisk for el9 2025-08-26 08:59:10 -04:00