2
0
mirror of https://github.com/xcat2/confluent.git synced 2026-11-19 17:02:22 +00:00
Commit Graph

68 Commits

Author SHA1 Message Date
Markus Hilger 973a340ed7 Use the root filesystem for grub when an image has no /boot
image2disk only took the boot UUID from a /boot volume. Images captured
from a system without a separate /boot left it unset, and rewriting the
EFI stub grub.cfg failed with "TypeError: sequence item 5: expected str
instance, NoneType found". Fall back to the UUID of the / filesystem,
which holds /boot in that case.
2026-11-12 15:17:36 +01:00
Markus Hilger 5473f46cf3 Do not autoactivate LVM on local disks on diskless boot
An old stateful OS on LVM was activated at diskless boot and kept the
disk busy, so repartitioning it for scratch failed. Set an empty
auto_activation_volume_list unless the image sets one. Ubuntu also
needs it in the initramfs, whose udev rules activate LVM earlier.
2026-11-09 21:35:40 +01:00
Markus Hilger fc63269dc7 Give every deployed node its own machine-id
Captured images carried the machine-id of the source system, and images
built with imgutil carry the one generated in the build root, so every
node deployed from a profile shared it. That breaks anything keyed on it,
such as DHCP DUIDs and journal identities.

Capture now stores an empty /etc/machine-id. Diskless boot on EL and SUSE
writes a stable per-node one derived from the node name and the DMI
product UUID, as Ubuntu already does, using openssl since the initramfs
has no sha512sum. Install to disk copies that machine-id into the target
and renames the boot loader entries from the image to it. Otherwise
kernel-install in post.sh adds a second set of entries under another ID.
2026-11-04 15:06:58 +01:00
Jarrod Johnson afc96a66e4 Merge pull request #366 from Obihoernchen/fix/ubuntu-install-fixes
Fix Ubuntu install to disk and installedargs
2026-11-04 08:28:59 -05:00
Jarrod Johnson 1f0202b68c Merge pull request #362 from Obihoernchen/fix/capture-optional-lvm
Make lvm optional for capture
2026-11-04 08:06:41 -05:00
Jarrod Johnson f31305ab7b Merge pull request #360 from Obihoernchen/fix/image2disk-partition-overflow
Fix install to disk of images with plain partitions
2026-11-04 08:05:06 -05:00
Markus Hilger 4b2c8c74c4 Rebuild the initramfs when installing Ubuntu images to disk
The installed system kept the initramfs of the image. Ubuntu 26.04
builds it with dracut in host-only mode, so a captured image's initramfs
waits for the source's volume group and filesystem UUIDs and the
installed node ends in emergency mode. Rebuild it in post.sh, as EL does
through kernel-install.
2026-11-03 20:23:43 +01:00
Markus Hilger b2a40e1370 Skip deactivating volume groups when lvm is absent
lvm is optional in the capture initramfs now, so installimage printed
'lvm: command not found' on every install of a capture taken from a host
without lvm2. Only run it when it is there.
2026-11-03 17:18:46 +01:00
Markus Hilger 04f45bf542 Number new partitions in creation order in image2disk
The plain partitions were addressed by their number on the source system,
while parted numbers them in the order they are created. A source with a
gap, such as swap between filesystems (it is not captured), made
image2disk format the wrong partition and look for swap on one that does
not exist. Count the partitions as they are created instead.

Also look up filesystem UUIDs through one helper.
2026-11-03 17:17:11 +01:00
Markus Hilger b5a58ef770 Install images without LVM to disk
image2disk.py formatted the target filesystems, copied the image and
ran fixup only in the branch that sets up LVM. For an image whose
filesystems are all plain partitions it created the partitions and
returned success without writing anything, and installimage then failed
on the missing target.

Run that part for both layouts. Plain partitions also need their swap
in fstab by UUID instead of the localstorage-swap volume, and on EL9
and later root= and resume= refer to the new root and swap by UUID as
well, since partition names are not stable with several disks.
2026-11-03 16:53:21 +01:00
Markus Hilger e021d29afc Keep image2disk partitions inside the disk
When the largest filesystem of a captured image is a plain partition,
it was sized as the disk minus the other filesystems. That ignored the
2048 sector gap before the first partition and the rounding of every
partition up to a 2048 sector boundary, so the last partition always
ran past the end of the disk. The clamp then set its end to the sector
count, one past the last sector and inside the backup GPT, and parted
refused it with 'The location ... is outside of the device'. Images
with the largest filesystem in LVM were not affected.

Reserve that overhead plus 1MiB at the end of the disk for the backup
GPT, and clamp to that last sector instead. Also round the swap
partition by its own size rather than by the previous partition's.
2026-11-03 15:45:52 +01:00
Markus Hilger 3debfdaee0 Add missing script directories to el10 and Ubuntu diskless profiles
Their scripts run onboot.d and the others, but the profiles did not
have them like el8 and el9. el10 also lacked the syncfiles template,
now linked to el9's like its other shared files, and Ubuntu lacked the
firstboot.custom that its firstboot.sh runs.
2026-11-02 18:12:47 +01:00
Jarrod Johnson 75402fe9ba Merge pull request #346 from Obihoernchen/fix/ubuntu-diskless-netplan-perms
Restrict the netplan file written by Ubuntu diskless boot
2026-11-02 07:49:20 -05:00
Jarrod Johnson 016db810ea Merge pull request #345 from Obihoernchen/fix/diskless-stop-on-extract-failure
Stop a diskless boot when the root filesystem cannot be extracted
2026-11-02 07:49:09 -05:00
Markus Hilger 6bdc3e17d7 Restrict the netplan file written by Ubuntu diskless boot
imageboot.sh wrote /etc/netplan/10-<iface>-confluentcfg.yaml with the
default mode 0644, and netplan warns that its configuration should not
be accessible by others. confignet already writes its files 0600, but
it leaves an identical file in place, so the boot-time one stayed 0644.
2026-11-01 22:39:31 +01:00
Markus Hilger 5d97bc6a19 Stop a diskless boot when the root filesystem cannot be extracted
extract_untethered_rootimg reports failure, for example when the image
does not fit into the memory backed root, but imageboot.sh ignored it and
booted into the partly extracted root, which then hung later on. Stop
with a message saying why instead.
2026-11-01 22:16:57 +01:00
Markus Hilger 0839299bd4 Fall back to other managers for untethered images
An untethered or uncompressed diskless boot downloaded the root image
only from the deployment server and retried it forever when that server
failed. Try the other managers that answered discovery as well, as the
tethered urlmount already does.

urlmount limits each small range request to 10 seconds, but the single
whole-image download cannot take a total limit. Give up on a connect
after 10 seconds and on a transfer that stalls for 30 seconds instead,
so a hung server moves on to the next one.
2026-11-01 12:09:06 +01:00
Markus Hilger b0561c61d2 Keep the image mounted for installtodisk on Ubuntu
Like on EL9 and EL10, an untethered or uncompressed boot unmounts the
image before installimage runs, so image2disk.py has nothing to copy.
2026-10-30 13:09:24 +01:00
Jarrod Johnson 1777c80969 Add retry to image fetch 2026-10-21 08:54:51 -04:00
Jarrod Johnson 31d1d38ac6 Make sure that ls reads the actual size without being confsued by symbolic links 2026-09-01 16:35:56 -04:00
Jarrod Johnson 2cf9162d6c Relocate onboot.d scripts in media based image 2026-08-27 16:08:59 -04:00
Jarrod Johnson b6d10e5e9b Continue to tweak the Media based image boot of a confluent diskless 2026-08-27 15:47:53 -04:00
Jarrod Johnson 1da744fcab Draft media based images
Prepare for media based diskless images
2026-08-26 16:13:34 -04:00
Jarrod Johnson 76aef703ff avoid moving firmware directories if they don't exist 2026-08-07 15:42:10 -04:00
Jarrod Johnson c3cf2a402f Remove redundant copies of syncfileclient 2026-08-03 08:48:28 -04:00
Jarrod Johnson 3501f70c37 Merge pull request #247 from Obihoernchen/unsquashfs
Use multi-threaded unsquashfs to extract untethered images
2026-07-28 08:59:38 -04:00
Markus Hilger 67e84f15f8 Keep the root filesystem guard reachable when extraction fails
source_remote imageboot.sh is the last thing the diskless cmdline hook
runs, so returning early on a failed extraction ended the hook and left
dracut to time out. Falling through instead reaches the existing
/sysroot/sbin/init guard, which reports the failure and holds the node so
it stays reachable over ssh, as it did before extraction was checked.
2026-07-28 01:49:38 +02:00
Markus Hilger 9101b07d54 Report stateless boot completion via new 'booted' status
Diskless profiles had the updatestatus callback in onboot.sh commented
out because no suitable status existed: 'complete' clears
deployment.pendingprofile, which the PXE responder requires to answer
the next network boot of a diskless node.

Add a 'booted' status that records the pending profile as
deployment.profile while leaving pendingprofile armed and skipping
autolock, and enable the onboot.sh callback in all diskless profiles.
nodedeploy now shows 'pending: <profile> (booted)' for a running
stateless node.
2026-07-20 15:59:30 +02:00
Markus Hilger cfc4490fe1 Use multi-threaded unsquashfs to extract untethered images
`unsquashfs` can use multiple CPU cores during image extraction, significantly reducing boot time.
For example the whole boot time from PXE to shell on a 8-core VM, from approximately 45 seconds to 20 seconds.

This PR adds `squashfs-tools` as a dependency. Since the package is smaller than 1 MB, the additional image size is justified by the performance improvement.

For backward compatibility, the existing `cp`-based extraction method is used when `unsquashfs` is unavailable, such as with images built before this change.

The extraction logic has also been moved into the common functions and is now shared between EL9, EL10, and Ubuntu.

Both untethered `squashfs` images and `confluent_multisquash` images are supported.

Images must be rebuilt to include `unsquashfs` and benefit from the faster extraction path.
2026-07-16 21:24:58 +02:00
Markus Hilger fa605160e0 Merge branch 'master' into ci 2026-07-14 17:04:16 +02:00
Markus Hilger 2af402b13c ruff auto fixes
Apply ruff's safe autofixes.
The changes are mechanical and behaviour-preserving. Issues fixed:

- F401: remove unused imports.
- F841: drop unused local variables and assignments, including discarded
  await/return values, unused "except ... as e" bindings, and unused
  "with ... as name" targets.
- F541: remove the f prefix from f-strings that contain no placeholders.
- E711: compare against None with "is"/"is not" instead of "=="/"!=".
- E712: test truthiness directly instead of comparing to True.
- E713: use "x not in y" instead of "not x in y".
- E714: use "is not" instead of "not ... is".
- E731: convert lambdas bound to a name into def statements.
- W291/W293: trim trailing whitespace on touched lines.
2026-07-14 05:03:58 +02:00
Markus Hilger 69a6714108 Use raw string notation to fix python compile warnings
E.g.: SyntaxError: "\d" is an invalid escape sequence. Did you mean "\\d"? A raw string is also an option.
2026-07-14 01:26:19 +02:00
Jarrod Johnson 5f34fac2bc confluent_nodename variable might not survive to imageboot
Pull it from the confluent.info file.
2026-07-09 14:13:12 -04:00
Jarrod Johnson 0645788b0d Rework functions to be common file 2026-06-12 16:06:06 -04:00
Jarrod Johnson 1042404a38 Move getinstalldisk for all the Linux into a common place
esxi is the only one with a different version
2026-06-12 07:45:41 -04:00
Jarrod Johnson ad48b718a1 Fixup aspects of Ubuntu diskless boot
For one, ensure a unique machine-id.  Broadly should be done, but critical for bonds to have unique mac
addressses in event of booting a captured image.

Fix ubuntu slow boot due to waiting forever for a network config. Have the transient network config bake into netplan for
a first pass before confignet comes along to do full configuration.

Clean up spurious error messages about grep true: and device busy on mounting overlay.
2026-06-11 15:42:03 -04:00
Jarrod Johnson 223ae16892 Fix missing mounts for ubuntu cloned install 2026-05-28 17:32:37 -04:00
Jarrod Johnson 41fe249151 Port diskless enhancements from el9 to ubuntu 2026-05-20 15:29:31 -04:00
Jarrod Johnson 48a0c21300 Refine getinstalldisk
Reduce obvious output about skipped devices.

Rule out any read-only device.

Amend minimum size to 2GB.

Among same priority devices, select the smallest target.
2025-08-01 09:00:25 -04:00
Jarrod Johnson 97e4d7c3d0 Skip sr0 if attempted 2025-07-29 15:33:05 -04:00
Jarrod Johnson b9f4051396 Export variables set in confluent functions 2025-02-03 16:40:57 -05:00
Jarrod Johnson 008c1308b4 Handle nvm subsystem without driver.
A variant of the M.2 RAID enablement kit does not manifest with nvme
driver.  Address this by allowing 'nvm' subsystype. to allow blank driver.

Also, to be on the safe side, have self.driver always be a string,
so it can be 'falsey' but still work as a string.
2024-10-26 08:16:56 -04:00
Jarrod Johnson c3e918fc5f Fix mistake in untethered support 2024-07-25 09:42:24 -04:00
Jarrod Johnson 714fefe31b Fix unethered boot for ubuntu 2024-07-24 14:41:39 -04:00
Jarrod Johnson bb04faed04 Explicitly request bash under ubuntu, which tends to use dash 2024-07-23 10:01:53 -04:00
Jarrod Johnson 33ed1a5e64 Add onboot for ubuntu diskless 2024-07-23 09:32:20 -04:00
Jarrod Johnson 1da27083cc Another cleanup of syncfileclient output 2024-04-09 15:08:56 -04:00
Jarrod Johnson 67b3c48dc9 Clean up error output on syncfileclient execution 2024-04-09 14:58:38 -04:00
Jarrod Johnson 02f301b5d0 Fix mistakes in syncfileclient change 2024-04-09 13:41:27 -04:00
Jarrod Johnson f68f9f4693 Make syncfile step robust or pause
If syncfiles fails, keep it retrying.

Also, slow down sync checking to avoid hammering the system.

Further, randomized delay to spread highly synchronized requestors.

Block attempts to do multiple concurrent syncfile runs.
2024-04-09 11:07:11 -04:00