2
0
mirror of https://github.com/xcat2/confluent.git synced 2026-11-15 21:38:04 +00:00
Commit Graph

7165 Commits

Author SHA1 Message Date
Markus Hilger bb59cc7fd7 Document nodeconsole video mode and recording 2026-11-04 23:43:12 +01:00
Jarrod Johnson d48de07097 Merge pull request #370 from Obihoernchen/fix/ignore-generated-man-pages
Ignore the man pages generated by addattribs.py
2026-11-04 16:48:43 -05:00
Jarrod Johnson aa8ee7a068 Merge pull request #372 from Obihoernchen/fix/runansible-ssh-args-b
Keep ansible’s default ssh_args and enable pipelining for plays
2026-11-04 16:39:15 -05:00
Markus Hilger 098d0a66c3 Enable ansible pipelining for plays by default
Without pipelining every task copies the module to the node over sftp,
then makes it executable, runs it and removes it, each a separate ssh
operation. With pipelining the module is fed to the interpreter over
the ssh session, so a task takes one operation and nothing is written
to the node's temporary directory.

Pipelining breaks become through sudo with requiretty, which rarely
applies here: plays connect as root and drop become unless become_user
is set. Setting ANSIBLE_PIPELINING or ANSIBLE_SSH_PIPELINING still
overrides this default.
2026-11-04 21:59:58 +01:00
Markus Hilger e668cdabf7 Keep ansible's default ssh_args when running plays
Setting ANSIBLE_SSH_ARGS replaces ansible's ssh_args instead of adding
to them, so plays ran without the default ControlMaster and
ControlPersist options. Every task then opened a new ssh connection
and authenticated again.

Start from ansible's default ssh_args when ANSIBLE_SSH_ARGS is not
set. An ANSIBLE_SSH_ARGS from the environment is still extended as
before.
2026-11-04 21:59:58 +01:00
Jarrod Johnson 55a4c9e631 Merge pull request #371 from Obihoernchen/perf/imgutil-crossarch
Emulate aarch64 as a Neoverse N1 cpu in imgutil
2026-11-04 15:02:00 -05:00
Markus Hilger 72510e704e Emulate aarch64 as a Neoverse N1 cpu in imgutil
The default 'max' cpu of qemu-user emulates SVE and pointer
authentication, which aarch64 binaries use and which is expensive to
emulate. Default QEMU_CPU to neoverse-n1 for aarch64 builds and exec,
unless it is already set. neoverse-n1 needs qemu 7.1 or later, the
documentation says how to choose another cpu.

Building an EL10 image with mlnx-ofa_kernel-dkms and doca-ofed-userspace
on an EL10 x86_64 host with qemu 10.2:

    DOCA build         Total
    aarch64, master    3270 s (54.5 min)
    aarch64, branch    2399 s (40 min), −27%
    x86_64, native     251 s (4 min)

Older qemu-user-static, such as qemu 8.2 of Ubuntu 24.04, emulates
pointer authentication much more slowly, so builds there can get up to
ten times faster.
2026-11-04 20:35:29 +01:00
Markus Hilger fb1a92d0f7 Ignore the man pages generated by addattribs.py
addattribs.py writes nodeattrib.ronn and nodegroupattrib.ronn from
their templates, so they show up as untracked files after a build.
2026-11-04 19:57:36 +01:00
Jarrod Johnson b59d8b24c0 Have NetworkManager run if not already enabled 2026-11-04 12:40:31 -05:00
Jarrod Johnson aa85c1a8bb Fix up some issues in suse leap 16 diskless boot 2026-11-04 12:33:09 -05:00
Jarrod Johnson 29b9b2b983 Amend ssh strategy
suse does not allow group readable ssh keys, so
keysign must be setuid root for host based authentication.

Also, chkstat got renamed, try to use the new name if possible.
2026-11-04 12:31:46 -05:00
Jarrod Johnson dbdb9e062e Fix 'grep true' error in suse boots 2026-11-04 11:48:45 -05:00
Jarrod Johnson 8b646b5728 Hide cursor during nodeconsole
Especially since the video is now rendered under 'text', the
cursor can be a bit weird looking
2026-11-04 11:29:07 -05:00
Jarrod Johnson 5e45bc598b Merge pull request #367 from Obihoernchen/feat/ansible-node-attributes
Pass node attributes to profile ansible plays
2026-11-04 10:06:59 -05:00
Jarrod Johnson 53a77ae8ce Merge pull request #365 from Obihoernchen/fix/capture-machine-id
Give every diskless and installed node its own machine-id
2026-11-04 10:05:50 -05:00
Jarrod Johnson c05aa3e674 Merge pull request #364 from Obihoernchen/fix/capture-kernelargs
Point out kernel arguments for captured profiles
2026-11-04 10:04:35 -05:00
Markus Hilger 9cbe3b91d8 Decode bytes attribute values outside of dicts too 2026-11-04 15:07:32 +01:00
Markus Hilger 0976c167c6 Point to profile.yaml more generally after capture
The comment in profile.yaml covers the kernel arguments, so the message
after capture only asks to review that file.
2026-11-04 15:07:22 +01:00
Markus Hilger 42907cb014 Keep the source machine-id in captured images
An empty machine-id in the image only works with deployment scripts that
set one per node. With older ones, which ship separately from imgutil,
systemd made up a new machine-id on every diskless boot and a disk install
got a second set of boot entries. The deployment now replaces whatever
machine-id the image carries, so leave it in the image.
2026-11-04 15:07:06 +01:00
Markus Hilger fc63269dc7 Give every deployed node its own machine-id
Captured images carried the machine-id of the source system, and images
built with imgutil carry the one generated in the build root, so every
node deployed from a profile shared it. That breaks anything keyed on it,
such as DHCP DUIDs and journal identities.

Capture now stores an empty /etc/machine-id. Diskless boot on EL and SUSE
writes a stable per-node one derived from the node name and the DMI
product UUID, as Ubuntu already does, using openssl since the initramfs
has no sha512sum. Install to disk copies that machine-id into the target
and renames the boot loader entries from the image to it. Otherwise
kernel-install in post.sh adds a second set of entries under another ID.
2026-11-04 15:06:58 +01:00
Jarrod Johnson 830b6892d6 Merge pull request #369 from Obihoernchen/fix/runansible-hangs
Stop hanging on ansible play processes that fail early
2026-11-04 08:38:33 -05:00
Jarrod Johnson 899542c128 Merge pull request #368 from Obihoernchen/fix/runansible-ansible-2.19
Run profile ansible plays on ansible-core 2.19 and later
2026-11-04 08:33:45 -05:00
Jarrod Johnson afc96a66e4 Merge pull request #366 from Obihoernchen/fix/ubuntu-install-fixes
Fix Ubuntu install to disk and installedargs
2026-11-04 08:28:59 -05:00
Jarrod Johnson 628a802433 Merge pull request #363 from Obihoernchen/fix/capture-exclude-tmp
Leave host specific leftovers out of captured images
2026-11-04 08:19:17 -05:00
Jarrod Johnson 1f0202b68c Merge pull request #362 from Obihoernchen/fix/capture-optional-lvm
Make lvm optional for capture
2026-11-04 08:06:41 -05:00
Jarrod Johnson ae4bbe5300 Merge pull request #361 from Obihoernchen/fix/image2disk-localstorage-recapture
Fix install to disk of recaptured localstorage LVM nodes
2026-11-04 08:05:52 -05:00
Jarrod Johnson f31305ab7b Merge pull request #360 from Obihoernchen/fix/image2disk-partition-overflow
Fix install to disk of images with plain partitions
2026-11-04 08:05:06 -05:00
Jarrod Johnson 2e62744fc1 Merge pull request #359 from Obihoernchen/fix/capture-prereqs
Require squashfs-tools and openssl for capture
2026-11-04 07:49:04 -05:00
Markus Hilger ec71d0b876 Stop hanging on ansible play processes that fail early
Drain the play process output while it runs, and stop waiting for its
feedback connection once it exits.
2026-11-04 00:23:33 +01:00
Markus Hilger 48b7dc62b5 Pass node attributes to profile ansible plays
Plays run by the deployment server get the target node's attributes in
the host variable 'confluent', nested at each dot of the name, e.g.
confluent['net']['ib0']['ipv4_address']. Secrets and unset attributes
are left out, and values are never templated.
2026-11-04 00:08:38 +01:00
Markus Hilger 1cad52284d Run profile ansible plays on ansible-core 2.19 and later
Newer ansible-core only takes a stdout callback name and no longer
templates untrusted YAML. Attach the results collector directly, load
plays trusted through the DataLoader, and report a crashed play
process as a failed task.
2026-11-03 23:58:43 +01:00
Markus Hilger 244f75b0b2 Pass maxsplit to re.split as a keyword
ruff B034 flags the positional maxsplit argument.
2026-11-03 22:03:31 +01:00
Markus Hilger 20a6d5cab9 Point out kernel arguments for captured profiles
A diskless boot of a captured profile only gets the profile's kernelargs,
not the kernel arguments of the source. Sources relying on some, such as
net.ifnames=0 for their interface names, then came up with different
interface names than the source and an install to disk of the same
image. Say so in the generated profile.yaml and after the capture, and
end the file with a newline so appended settings do not run into the
kernelargs line.
2026-11-03 21:32:15 +01:00
Markus Hilger 9830518131 Apply installedargs to the grub configuration of Ubuntu installs
post.sh wrote installedargs into /etc/default/grub but only ran
update-grub when it had also added a console, so grub.cfg kept the old
command line until some later grub update.
2026-11-03 21:16:53 +01:00
Markus Hilger 8fd41b1315 Clean up the capture initramfs config on Ubuntu sources
Capture copies an initramfs-tools hook and conf.d/confluent.conf into the
source to build the deployment initramfs, but only ever removed the EL
dracut module afterwards. With MODULES=list left in place, the source's
next initramfs update was limited to the hook's driver list, and a source
captured again carried both files into the image, where rebuilding the
initramfs on install to disk shrank the clones' initramfs the same way.
Remove them after capture and leave them out of the image.
2026-11-03 21:16:47 +01:00
Markus Hilger ed49d131ad Strip the key from installedargs in Ubuntu installs
post.sh took the whole installedargs line from profile.yaml, key
included, so every Ubuntu install from these profiles got a literal
'installedargs:' in GRUB_CMDLINE_LINUX.
2026-11-03 20:23:43 +01:00
Markus Hilger 4b2c8c74c4 Rebuild the initramfs when installing Ubuntu images to disk
The installed system kept the initramfs of the image. Ubuntu 26.04
builds it with dracut in host-only mode, so a captured image's initramfs
waits for the source's volume group and filesystem UUIDs and the
installed node ends in emergency mode. Rebuild it in post.sh, as EL does
through kernel-install.
2026-11-03 20:23:43 +01:00
Markus Hilger ebd2d548a1 Keep source network profiles without static addresses
Dropping every NetworkManager profile and ifcfg file also removed
settings that confignet does not recreate from confluent attributes,
such as bonds, DHCP profiles, MTU or the InfiniBand transport mode. Only
drop the profiles that carry static addresses of the source.
2026-11-03 19:47:49 +01:00
Markus Hilger 80d55d2739 Leave dhcpcd and NetworkManager host identity out of captures
dhcpcd, present on Ubuntu 24.04 and later and on EL10, keeps its leases,
DUID and secret in /var/lib/dhcpcd, and NetworkManager keeps a secret in
secret_key. Copied into an image, every clone would use the same DHCPv6
DUID and derive the same stable privacy IPv6 addresses. Both regenerate
them when missing.
2026-11-03 19:35:39 +01:00
Markus Hilger e8206bb968 Leave dhclient leases of Debian and Ubuntu out of captures
dhclient keeps its leases in /var/lib/dhcp there, and on EL also as
*.leases next to the *.lease files already excluded. Keep dhcpd.leases
of a DHCP server, which lives in the same directory.
2026-11-03 19:34:23 +01:00
Markus Hilger e65a4e960b Leave the source hostname out of captured images
The image kept /etc/hostname of the source host, so a diskless node ran
under that name until onboot set its own, and early journal and syslog
entries carried the wrong name. With an empty /etc/hostname systemd
keeps the node name the deployment initramfs already set. Install to
disk writes /etc/hostname itself.
2026-11-03 17:41:31 +01:00
Markus Hilger 8fd70fd49a Leave the source network profiles out of captured images
Captured images kept the NetworkManager connection profiles and ifcfg
files of the source host. A diskless boot kept them too, and confignet
only rewrites the profiles named like the interfaces it configures, so
every other interface came up with the source host's settings, such as
its static addresses. Install to disk already removes them, and capture
already masks netplan files for the same reason. Drop them from the
image, except ifcfg-lo.
2026-11-03 17:27:51 +01:00
Markus Hilger b2a40e1370 Skip deactivating volume groups when lvm is absent
lvm is optional in the capture initramfs now, so installimage printed
'lvm: command not found' on every install of a capture taken from a host
without lvm2. Only run it when it is there.
2026-11-03 17:18:46 +01:00
Markus Hilger 3cf6684322 Keep hard linked files and odd names right in capture excludes
mksquashfs matches excluded paths by inode, so excluding a hard linked
file also dropped its other links elsewhere in the filesystem. Keep such
files in the image. squashfs-tools 4.3 also splits exclude file lines on
whitespace, so pass paths with whitespace or backslashes with -e instead.
2026-11-03 17:18:35 +01:00
Markus Hilger 115de4c00e Check for the localstorage volume group first
The guess for volume groups named by the OS installer (<os>_<node>)
matched any device mapper name with an underscore followed later by a
hyphen. A localstorage volume for a mount point with a hyphen, such as
localstorage-_srv_my--data, was taken for one and got a broken volume
group name. Recognize volumes of the target volume group before trying
that guess.
2026-11-03 17:17:31 +01:00
Markus Hilger 04f45bf542 Number new partitions in creation order in image2disk
The plain partitions were addressed by their number on the source system,
while parted numbers them in the order they are created. A source with a
gap, such as swap between filesystems (it is not captured), made
image2disk format the wrong partition and look for swap on one that does
not exist. Count the partitions as they are created instead.

Also look up filesystem UUIDs through one helper.
2026-11-03 17:17:11 +01:00
Markus Hilger b5a58ef770 Install images without LVM to disk
image2disk.py formatted the target filesystems, copied the image and
ran fixup only in the branch that sets up LVM. For an image whose
filesystems are all plain partitions it created the partitions and
returned success without writing anything, and installimage then failed
on the missing target.

Run that part for both layouts. Plain partitions also need their swap
in fstab by UUID instead of the localstorage-swap volume, and on EL9
and later root= and resume= refer to the new root and swap by UUID as
well, since partition names are not stable with several disks.
2026-11-03 16:53:21 +01:00
Markus Hilger 3bd3ebeadb Install recaptured images of LVM installed nodes to disk
image2disk.py puts the volumes of a node it installs into the volume
group localstorage. Capturing such a node and installing the profile to
disk again failed with 'No mapping defined to create volume group
localstorage': volumes named localstorage-root were taken for volumes of
an extra volume group that needs volumegroupmap.yml.

Treat volumes of a volume group named like the target one as main volume
group volumes, and split device mapper names into volume group and
logical volume names correctly, including hyphens that device mapper
doubles inside either name.
2026-11-03 16:51:59 +01:00
Markus Hilger e021d29afc Keep image2disk partitions inside the disk
When the largest filesystem of a captured image is a plain partition,
it was sized as the disk minus the other filesystems. That ignored the
2048 sector gap before the first partition and the rounding of every
partition up to a 2048 sector boundary, so the last partition always
ran past the end of the disk. The clamp then set its end to the sector
count, one past the last sector and inside the backup GPT, and parted
refused it with 'The location ... is outside of the device'. Images
with the largest filesystem in LVM were not affected.

Reserve that overhead plus 1MiB at the end of the disk for the backup
GPT, and clamp to that last sector instead. Also round the swap
partition by its own size rather than by the previous partition's.
2026-11-03 15:45:52 +01:00
Markus Hilger 94d396a8cb Leave host specific leftovers out of captured images
Capture copied everything on the local filesystems, including user
files and sockets in /tmp and /var/tmp, crash dumps, logs and the
persistent journal, sssd cached credentials, package caches, the
random seed, DHCP leases and root's shell history. All of it ended up
on every node deployed from the profile.

Drop the contents of scratch, crash, journal, mail spool, sssd cache
and package cache directories, keep only the directory tree under
/var/log and /var/cache/apt since daemons expect their subdirectories
there, and drop a short list of individual files. The directories
themselves are kept, so their mode, ownership and labels survive.
mksquashfs gets the paths as an exclude file, so this applies the same
way whether or not a directory is its own mount.
2026-11-03 15:16:26 +01:00