2
0
mirror of https://github.com/xcat2/confluent.git synced 2026-11-19 08:52:20 +00:00
Commit Graph

7187 Commits

Author SHA1 Message Date
Markus Hilger 9d8cb97fb3 Add confluent_attrib helper to profile functions
Profile scripts had to call apiclient and parse /self/myattribs to read
a node attribute. confluent_attrib prints one attribute, comma joining
lists, and returns 1 when it is unset. Non-shell scripts can run it as
'bash /etc/confluent/functions confluent_attrib <name>'.
2026-11-11 23:02:02 +01:00
Markus Hilger c949359199 Return node groups from /self/myattribs
groups is stored as a plain list rather than a value dict, so the
value check skipped it and a node could not see its own groups. Return
it as a list, the same way ansible plays get it as host vars.
2026-11-11 23:02:02 +01:00
Jarrod Johnson 05ec7f9d4d Merge pull request #377 from Obihoernchen/fix/diskless-lvm-autoactivation
Do not autoactivate LVM on local disks on diskless boot
2026-11-09 15:38:42 -05:00
Jarrod Johnson c3bc8ce747 Merge pull request #375 from Obihoernchen/fix/image2disk-kernelargs
Keep interface naming arguments when installing images to disk
2026-11-09 15:37:55 -05:00
Jarrod Johnson ccc68b5c5a Merge pull request #376 from Obihoernchen/fix/installtodisk-sshd
Fix syncfiles during EL9 installtodisk
2026-11-09 15:37:08 -05:00
Markus Hilger 5473f46cf3 Do not autoactivate LVM on local disks on diskless boot
An old stateful OS on LVM was activated at diskless boot and kept the
disk busy, so repartitioning it for scratch failed. Set an empty
auto_activation_volume_list unless the image sets one. Ubuntu also
needs it in the initramfs, whose udev rules activate LVM earlier.
2026-11-09 21:35:40 +01:00
Jarrod Johnson a1e8f2b31b Merge pull request #374 from Obihoernchen/doc/imgutil-chroot-tools
Document running chroot tools through imgutil exec
2026-11-09 15:35:19 -05:00
Markus Hilger bd5c7be4da Add interface naming arguments to captured image cmdlines
A captured image keeps the /etc/kernel/cmdline of its source. If the
source did not run with the net.ifnames=/biosdevname= of the diskless
boot, the installed system renames the interface and the connection
written during the install does not match. Append them when missing.
2026-11-09 21:04:06 +01:00
Markus Hilger 9b84a4ecba Point the installtodisk no disk hint at port 22
While installimage waits for an install disk, the sshd of the image
listens on port 22. The el9 initramfs sshd on port 2222 only runs with
debugssh.
2026-11-09 21:03:07 +01:00
Jarrod Johnson 1665b82b70 Set target VERSION for master branch 2026-11-09 14:58:35 -05:00
Markus Hilger e23873888f Serve installtodisk syncfiles with the sshd of the image
After image2disk, installimage stopped the sshd of the image and moved
port 22 to the initramfs sshd, chrooted into the install target, by
sending that sshd HUP. The el9 initramfs only starts its sshd with
debugssh, and on EL 9.8 it cannot start at all without sshd-session, so
normally nothing listened on port 22. The syncfiles push of post.sh then
failed with connection refused while the node still reported completed.

Restart the sshd of the image instead, chrooted into the install target.
It is always present and matches the OpenSSH helpers of the image, so
this also works for images packed or captured before the initramfs fix.
Wait for the old sshd to exit first, so the new one can bind port 22.
2026-11-09 20:26:06 +01:00
Markus Hilger 81f375f6da Include sshd-session in the el9 diskless initramfs
EL 9.8 ships OpenSSH 9.9, whose sshd runs each connection through
/usr/libexec/openssh/sshd-session. Without it the initramfs sshd refuses
to start, so debugssh does not work on EL 9.8 images. Install it when
present, older EL9 releases do not have it.
2026-11-09 20:26:06 +01:00
Markus Hilger 0e3f732538 Keep interface naming arguments when installing images to disk
image2disk only carried console= and quiet into the installed system.
A built image deployed with net.ifnames=0 biosdevname=0 came up with
enp1s0 while the connection written during the install was for eth0,
so the network failed. Carry net.ifnames= and biosdevname= as well.

Also add the missing space before crashkernel= in the default el9
GRUB_CMDLINE_LINUX.
2026-11-09 20:17:18 +01:00
Markus Hilger 0581af6dcf Document running chroot tools through imgutil exec
A plain chroot, as used by ansible's chroot connection, has no DNS or
/dev in a built image. Describe what imgutil exec mounts, show ansible
running through it, and fix the exec example, which needs -- before
command options.
2026-11-09 20:01:40 +01:00
Jarrod Johnson 52505e6f2d Improve SSDP responder concurrency
Bulk boot could get serialized too harshly. Particularly could hiccup on name lookup delay.

Take the penalties concurrently instead.
2026-11-06 17:28:05 -05:00
Jarrod Johnson 84faa9a1c7 Properly indicate bynode or not in message state 2026-11-06 10:08:19 -05:00
Jarrod Johnson 58d4c6a5a9 Allow API caller to request raw binary images.
This can make it more straightforward as an interface, and also defer the penalty for base64 processing.
2026-11-06 09:37:58 -05:00
Jarrod Johnson b0fff4a8fe Default to json, and add binary types for screenshot to come down in a more straightforward way 2026-11-06 08:44:46 -05:00
Jarrod Johnson 791d0bea69 Merge pull request #373 from Obihoernchen/docs/nodeconsole-video
Document nodeconsole video mode and recording
2026-11-05 10:12:38 -05:00
Jarrod Johnson f569020c57 Merge pull request #351 from Obihoernchen/feat/imgutil-import
Add container image sources and --containerfile to imgutil build
2026-11-05 10:10:49 -05:00
Markus Hilger 875d87332c Add imgutil build --containerfile to start an image for offline builds
An image meant to build without network access must contain every
package build checks for, and that list differs by distribution, version
and architecture and changes between releases. build --containerfile
writes a Containerfile, or with --apptainer an Apptainer definition
file, generated from the same package lists, including -p, -a and
--selinux, so it cannot drift from what build expects. The OS is that of
the -s distribution or of the build system, as for a build. --arch names
the matching build platform and --from replaces the default base image.
-r is refused, the image installs from the repositories of its base
image.
The generated EL file installs full coreutils instead of the
coreutils-single of container bases, and the Ubuntu file cleans the apt
cache.
2026-11-05 12:46:44 +01:00
Markus Hilger 4a2be59d5d Add container image sources to imgutil build
For sites that already build images elsewhere, for example in CI from a
Dockerfile, imgutil build -s also takes a container image: a SIF file,
an OCI or Docker archive, an OCI layout directory, or a podman reference
with its transport, such as docker:// or containers-storage:. Files and
directories are recognized by their content, and any other source is a
distribution as before. The image is extracted into the scratch
directory, with podman or, for a SIF file, from its squashfs partition.
It is then completed the way build does: the confluent initramfs
configuration, the default package list with the package manager and
repositories of the image, and an initramfs for every kernel. The result
works with exec and pack like any built tree, and the build info names
the image as its source.

When the image already has every package of the list, the package
manager is not run, so an image prepared in CI builds without network
access. Missing packages are named before installing them. The
architecture comes from the image, and a requested --arch is checked
before extracting it. -r is refused because the image brings its own
repositories. With --arch, the matching variant of an image for several
architectures is used. Other than a local image, podman pulls or loads
the image into a temporary storage that is removed afterwards, so the
images and tags of the system stay as they are.

Container base images are not bootable systems, so the build also
unmasks the units they mask (systemd-remount-fs, getty.target,
systemd-logind and others) and removes policy-rc.d, reporting each
change, drops the boot entries kernel-install writes (they carry the
command line of the build system, installing to disk writes its own),
and with --selinux labels the tree with full contexts. Ubuntu images get
ubuntu-minimal, which debootstrap installs but container bases lack.

EL9, Ubuntu 24.04 and SUSE 16 or later images are supported, as long as
imgutil has a package list for the release. Older releases and
architectures other than x86_64 and aarch64 are refused.
2026-11-05 12:46:44 +01:00
Markus Hilger bb59cc7fd7 Document nodeconsole video mode and recording 2026-11-04 23:43:12 +01:00
Jarrod Johnson d48de07097 Merge pull request #370 from Obihoernchen/fix/ignore-generated-man-pages
Ignore the man pages generated by addattribs.py
2026-11-04 16:48:43 -05:00
Jarrod Johnson aa8ee7a068 Merge pull request #372 from Obihoernchen/fix/runansible-ssh-args-b
Keep ansible’s default ssh_args and enable pipelining for plays
2026-11-04 16:39:15 -05:00
Markus Hilger 098d0a66c3 Enable ansible pipelining for plays by default
Without pipelining every task copies the module to the node over sftp,
then makes it executable, runs it and removes it, each a separate ssh
operation. With pipelining the module is fed to the interpreter over
the ssh session, so a task takes one operation and nothing is written
to the node's temporary directory.

Pipelining breaks become through sudo with requiretty, which rarely
applies here: plays connect as root and drop become unless become_user
is set. Setting ANSIBLE_PIPELINING or ANSIBLE_SSH_PIPELINING still
overrides this default.
2026-11-04 21:59:58 +01:00
Markus Hilger e668cdabf7 Keep ansible's default ssh_args when running plays
Setting ANSIBLE_SSH_ARGS replaces ansible's ssh_args instead of adding
to them, so plays ran without the default ControlMaster and
ControlPersist options. Every task then opened a new ssh connection
and authenticated again.

Start from ansible's default ssh_args when ANSIBLE_SSH_ARGS is not
set. An ANSIBLE_SSH_ARGS from the environment is still extended as
before.
2026-11-04 21:59:58 +01:00
Jarrod Johnson 55a4c9e631 Merge pull request #371 from Obihoernchen/perf/imgutil-crossarch
Emulate aarch64 as a Neoverse N1 cpu in imgutil
2026-11-04 15:02:00 -05:00
Markus Hilger 72510e704e Emulate aarch64 as a Neoverse N1 cpu in imgutil
The default 'max' cpu of qemu-user emulates SVE and pointer
authentication, which aarch64 binaries use and which is expensive to
emulate. Default QEMU_CPU to neoverse-n1 for aarch64 builds and exec,
unless it is already set. neoverse-n1 needs qemu 7.1 or later, the
documentation says how to choose another cpu.

Building an EL10 image with mlnx-ofa_kernel-dkms and doca-ofed-userspace
on an EL10 x86_64 host with qemu 10.2:

    DOCA build         Total
    aarch64, master    3270 s (54.5 min)
    aarch64, branch    2399 s (40 min), −27%
    x86_64, native     251 s (4 min)

Older qemu-user-static, such as qemu 8.2 of Ubuntu 24.04, emulates
pointer authentication much more slowly, so builds there can get up to
ten times faster.
2026-11-04 20:35:29 +01:00
Markus Hilger fb1a92d0f7 Ignore the man pages generated by addattribs.py
addattribs.py writes nodeattrib.ronn and nodegroupattrib.ronn from
their templates, so they show up as untracked files after a build.
2026-11-04 19:57:36 +01:00
Jarrod Johnson b59d8b24c0 Have NetworkManager run if not already enabled 2026-11-04 12:40:31 -05:00
Jarrod Johnson aa85c1a8bb Fix up some issues in suse leap 16 diskless boot 2026-11-04 12:33:09 -05:00
Jarrod Johnson 29b9b2b983 Amend ssh strategy
suse does not allow group readable ssh keys, so
keysign must be setuid root for host based authentication.

Also, chkstat got renamed, try to use the new name if possible.
2026-11-04 12:31:46 -05:00
Jarrod Johnson dbdb9e062e Fix 'grep true' error in suse boots 2026-11-04 11:48:45 -05:00
Jarrod Johnson 8b646b5728 Hide cursor during nodeconsole
Especially since the video is now rendered under 'text', the
cursor can be a bit weird looking
2026-11-04 11:29:07 -05:00
Jarrod Johnson 5e45bc598b Merge pull request #367 from Obihoernchen/feat/ansible-node-attributes
Pass node attributes to profile ansible plays
2026-11-04 10:06:59 -05:00
Jarrod Johnson 53a77ae8ce Merge pull request #365 from Obihoernchen/fix/capture-machine-id
Give every diskless and installed node its own machine-id
2026-11-04 10:05:50 -05:00
Jarrod Johnson c05aa3e674 Merge pull request #364 from Obihoernchen/fix/capture-kernelargs
Point out kernel arguments for captured profiles
2026-11-04 10:04:35 -05:00
Markus Hilger 9cbe3b91d8 Decode bytes attribute values outside of dicts too 2026-11-04 15:07:32 +01:00
Markus Hilger 0976c167c6 Point to profile.yaml more generally after capture
The comment in profile.yaml covers the kernel arguments, so the message
after capture only asks to review that file.
2026-11-04 15:07:22 +01:00
Markus Hilger 42907cb014 Keep the source machine-id in captured images
An empty machine-id in the image only works with deployment scripts that
set one per node. With older ones, which ship separately from imgutil,
systemd made up a new machine-id on every diskless boot and a disk install
got a second set of boot entries. The deployment now replaces whatever
machine-id the image carries, so leave it in the image.
2026-11-04 15:07:06 +01:00
Markus Hilger fc63269dc7 Give every deployed node its own machine-id
Captured images carried the machine-id of the source system, and images
built with imgutil carry the one generated in the build root, so every
node deployed from a profile shared it. That breaks anything keyed on it,
such as DHCP DUIDs and journal identities.

Capture now stores an empty /etc/machine-id. Diskless boot on EL and SUSE
writes a stable per-node one derived from the node name and the DMI
product UUID, as Ubuntu already does, using openssl since the initramfs
has no sha512sum. Install to disk copies that machine-id into the target
and renames the boot loader entries from the image to it. Otherwise
kernel-install in post.sh adds a second set of entries under another ID.
2026-11-04 15:06:58 +01:00
Jarrod Johnson 830b6892d6 Merge pull request #369 from Obihoernchen/fix/runansible-hangs
Stop hanging on ansible play processes that fail early
2026-11-04 08:38:33 -05:00
Jarrod Johnson 899542c128 Merge pull request #368 from Obihoernchen/fix/runansible-ansible-2.19
Run profile ansible plays on ansible-core 2.19 and later
2026-11-04 08:33:45 -05:00
Jarrod Johnson afc96a66e4 Merge pull request #366 from Obihoernchen/fix/ubuntu-install-fixes
Fix Ubuntu install to disk and installedargs
2026-11-04 08:28:59 -05:00
Jarrod Johnson 628a802433 Merge pull request #363 from Obihoernchen/fix/capture-exclude-tmp
Leave host specific leftovers out of captured images
2026-11-04 08:19:17 -05:00
Jarrod Johnson 1f0202b68c Merge pull request #362 from Obihoernchen/fix/capture-optional-lvm
Make lvm optional for capture
2026-11-04 08:06:41 -05:00
Jarrod Johnson ae4bbe5300 Merge pull request #361 from Obihoernchen/fix/image2disk-localstorage-recapture
Fix install to disk of recaptured localstorage LVM nodes
2026-11-04 08:05:52 -05:00
Jarrod Johnson f31305ab7b Merge pull request #360 from Obihoernchen/fix/image2disk-partition-overflow
Fix install to disk of images with plain partitions
2026-11-04 08:05:06 -05:00
Jarrod Johnson 2e62744fc1 Merge pull request #359 from Obihoernchen/fix/capture-prereqs
Require squashfs-tools and openssl for capture
2026-11-04 07:49:04 -05:00