When a process releases a lock that another process waits for, the
waiting process must take the lock. A third process must not take it
while the waiting process holds it.
release_lock removes the lock file before it unlocks it. A process that
waited in acquire_lock then held a lock on a file without a name, while
a third process created a new file at the path and locked it too. Both
ran at once, for example two imports, or an import and a kit command.
acquire_lock now checks that the file that it locked is the one at the
path, and tries again when it is not. lock_dir returns the directory of
the lock files.
When no archive was published before and a rename of the publication
fails, no archive, checksum or metainfo may remain where none was, and
no directory for a previous archive may be created.
When no archive was published before, as for the first pack or a change
of format, and the metainfo rename failed after the archive rename, the
rollback looked for a previous archive that did not exist. It reported
that archive as kept in an empty directory, and left the new archive and
its checksum published. The rollback now removes the new archive and its
checksum.
Describe the .sha256 file that packimage and liteimg write, and how
nodeset has nodes check it. Describe the rootimgchecksum attributes of
the site and the osimage, the upgrade that keeps existing images
unchecked, and the nodeset warning. Describe the checksum that genimage
and nodeset write for an image packed before this release.
rmimage must hold the image lock and remove the stateless and statelite
archives with their checksums and metainfo, and the .rootimgcheck
marker. It must remove the temporary directory that this host abandoned
and a kept previous archive. It must keep the temporary directory of
another host and files that are not part of the image.
rmimage removed rootimg.* from the image directory. That missed rootimg-
statelite.gz with its checksum and metainfo, and the .rootimgcheck
marker of the initial ramdisk. A temporary directory that an interrupted
packimage, liteimg or imgimport left on this host also stayed, with a
partial archive in it.
rmimage now removes these files, the temporary directories that writers
on this host abandoned, and the .rootimg-previous-* directories that
failed publications kept. rootimg_stage and rmimage share this cleanup
through remove_abandoned_rootimg_stages.
A SLES statelite ramdisk node built with dracut must boot
initrd-stateless.gz, and nodeset must copy that file to TFTP. A statelite
ramdisk node without dracut and a statelite NFS node must still boot
initrd-statelite.gz.
A SLES statelite ramdisk image built with dracut boots
initrd-stateless.gz, but nodeset copied only initrd-statelite.gz to the
TFTP directory and checked only for that file. Nodes requested an initial
ramdisk that nodeset never copied, or one that an older nodeset left.
nodeset now chooses the initial ramdisk once, then copies, checks and
boots that file, as it does on Red Hat.
An Ubuntu statelite ramdisk node must get the imgurl of
rootimg-statelite.gz and no NFS root, and must boot the initial ramdisk
that nodeset copies to TFTP. A statelite NFS node must still mount the
image over NFS.
The statelite branch of debian.pm tested rootfstype without its $, so it
compared the string rootfstype with ramdisk and always took the NFS root
branch. Ubuntu statelite ramdisk nodes got an NFS root and no imgurl, so
they never downloaded rootimg-statelite.gz.
nodeset now tests $rootfstype, and Ubuntu statelite ramdisk nodes get
the imgurl of rootimg-statelite.gz, as on Red Hat and SUSE.
The upgrade step must set rootimgchecksum to no for the stateless and
statelite images that do not set it, also for an image without a
linuximage row. It must keep an image that sets the attribute, leave
other images alone, and set site.rootimgchecksum to yes. A later
upgrade, and an upgrade after the administrator set the site attribute,
must change no image.
The root image check is on by default, so an upgrade from a release
without it would turn it on for every existing stateless and statelite
image. Their nodes would then need a checksum and an initial ramdisk
that can check.
On an upgrade, while site.rootimgchecksum is not set, xcatconfig sets
the rootimgchecksum attribute to no for each of these images that does
not set it, and then sets site.rootimgchecksum to yes. The existing
images keep booting unchecked, and new images get the check. A later
upgrade finds the site attribute set and changes nothing. A new
installation sets site.rootimgchecksum to yes.
nodeset must ask nodes to check by default, with rootimgchecksum=yes. It
must leave the argument out when the site or the image turns the check
off, and the image value must win. For a statelite ramdisk node,
STATEMNT must stay after the image URL. nodeset must warn once for each
image whose initial ramdisk has no matching marker, and not for a marked
one. This must hold on the Red Hat, SUSE and Ubuntu plugins. A marker
must hold the checksum of its initial ramdisk, and imgimport must place
the marker that a bundle carries. With --noupdateinitrd, the copy in
TFTP must decide the warning. A failed copy must leave no boot entry and
keep the boot files in TFTP.
genimage and nodeset must write the checksum of an archive packed before
the check. They must write it also when no marker shows that the initial
ramdisk checks it, and when the check is turned off. They must keep a
published checksum. With the check off, a checksum that cannot be
written must not refuse the node. When another host holds the
publication lock, genimage must keep the initial ramdisk in TFTP.
nodeset must then report the node and keep its boot files. When the
first node of an image fails this step, the next node must copy the boot
files.
Nodes could not know whether an image turned the root image checksum on.
An initial ramdisk built before the check booted the image unchecked,
and nothing reported it. An archive packed before the check had no
checksum for a new initial ramdisk to find.
When the image or the site turns the checksum on, nodeset adds
rootimgchecksum=yes to the kernel command line of the node. This applies
to stateless and statelite ramdisk nodes. genimage writes
<initrd>.rootimgcheck next to an initial ramdisk that can check. The
marker holds the SHA-256 checksum of the initial ramdisk, so a replaced
ramdisk does not match it. imgexport and imgimport carry the marker with
the ramdisk.
When the checksum is on, nodeset warns once for each image whose initial
ramdisk has no matching marker. Its nodes may boot the image unchecked.
nodeset judges the initial ramdisk in TFTP after it copies the boot
files, because nodes boot that copy, also with --noupdateinitrd. A copy
that fails is now an error for the node, instead of a boot entry for the
previous boot files.
genimage and nodeset write a checksum for each archive of the image
directory that has none, also when the check is off. Nodes with older
boot entries still check. genimage does it before it copies a new
initial ramdisk to TFTP, and nodeset before it copies the boot files of
a node. Both hold the publication lock while they write. When they
cannot write the checksum, they keep the boot files that nodes use now
and report how to recover. With the check off, nodeset reports it as a
warning instead, and still sets up the node. nodeset marks the boot
files of an image as copied only after this step succeeds. When the
first node of an image fails it, the next node copies them.
The change covers the Red Hat, SUSE and Ubuntu nodeset plugins, which
share one helper in xCAT::SvrUtils.
The test runs each changed xcatroot in a bubblewrap sandbox, with stub
curl, wget, logger, mount and sleep. An image that matches its checksum
must be extracted without a wait, also when a later line of the checksum
file holds it. An image that matches no line must not be extracted. A
download that began before packimage replaced the image must match the
checksum file fetched before it. A damaged download, a failed checksum
download and a checksum file without a checksum must each lead to one
more download. A file that an interrupted download left must be checked
before it is used. A missing checksum file and a missing sha256sum must
keep the node retrying. Without rootimgchecksum=yes, or with
rootimgchecksum=no, the image must be extracted unchecked.
xcatroot extracted whatever curl or wget downloaded, so a damaged
download became the root file system of the node.
After the download, xcatroot fetches the .sha256 file next to the image
and compares it with sha256sum. It also fetches that file before each
download. A download passes when it matches a line of either file, so a
download that began before packimage replaced the image still passes.
The check can fail for any reason, such as a missing checksum file or a
missing sha256sum. xcatroot then removes the download, logs the reason,
waits, and downloads the image again. A file that an earlier attempt
left is checked like a new download. xcatroot checks only when the
kernel argument rootimgchecksum=yes asks for it, and logs when it does
not check. An older xCAT server never passes the argument, so a new
initial ramdisk boots from it unchecked, as before.
The change covers the stateless modules of dracut 033, 047 and 105 for
the Red Hat family, dracut 033 for SUSE, and the Ubuntu module. These
modules also download statelite ramdisk images. Their initial ramdisks
now include sha256sum.
liteimg must take the image lock and keep the previous archive while it
packs and after a failed pack. It must publish the new archive with its
checksum, also when the check is turned off. It must change nothing
while the lock is held. With --tracker, it must name the given tracker
URL and describe the new archive under its final name.
liteimg deleted rootimg-statelite.gz before it packed a statelite
ramdisk image, and wrote the new archive in place. It ignored a failed
pack, so nodes could get no image or a partial one. It took no image
lock. --tracker took no value, so the metainfo named the tracker 1, and
it described rootimg.gz instead of the new archive.
liteimg now takes the image lock and packs into the temporary directory
that packimage uses. It fails when a command of the pack fails. It
writes the SHA-256 checksum, and publishes it with the archive through
publish_rootimg. --tracker takes the tracker URL, as it does for
packimage, and the metainfo describes the new archive. Nodes keep
downloading the previous archive until the new one replaces it.
An import of a cpio, tar or squashfs root image over a published image
must take the image lock that packimage takes. It must replace the
image, its checksum, the kernel and the ramdisk. It must keep the
previous image as .ORIG, remove the previous metainfo, keep other
formats, and leave no temporary directory. With the check turned off,
the import must still write the checksum. When the image lock is held,
the import must fail and keep the published image, checksum, kernel,
ramdisk and image definition. When the publication lock of another host
is held, or the imported image or its checksum cannot be published, the
import must fail. The published image, checksum and metainfo must then
stay, with their kernel and ramdisk.
imgimport moved the published root image away, then moved the copy from
the bundle in. When the bundle was on another file system, that move was
a copy. Nodes could get no image or a partial one. The checksum that
packimage had published for the previous image also stayed.
imgimport now takes the image lock that genimage, packimage and rmimage
use. It stages the imported image and its checksum in the image
directory. It publishes both with the packimage helper, under the same
publication lock. It writes the checksum whatever the rootimgchecksum
attribute of the imported image, as packimage does. The previous image
stays as .ORIG through a hard link, and its metainfo is removed.
imgimport also places the root image before the kernel and ramdisk now.
When it cannot publish the root image, it stops, so the kernel and
ramdisk keep matching the published image. It takes the image lock
before it imports the image definition, so an import that another image
command blocks changes nothing.
Each packed format must publish the SHA-256 checksum of the new archive
next to it, readable by nodes. The checksums of other formats must stay,
for nodes that already download them. A pack that fails while it
compresses, lists or reads the files must keep the previous checksum. A
pack must publish the checksum also when the image and the site turn the
check off. An image that turns the check on overrides the site, and 'no'
and '0' both turn it off.
While the archive is replaced, the checksum file must list the new and
the previous archive, also one packed before the check. The previous
metainfo must be gone by then. Then the checksum file must list only the
new archive. A failed rename of the checksum list, the archive or the
metainfo must leave the previous archive, checksum and metainfo, also
without hard links. When the previous archive can be neither linked nor
copied, nothing must change. A copy under umask 0077 must keep the mode
of the previous archive. A checksum file that keeps the previous line
must not fail the publication. A pack whose previous metainfo cannot be
moved aside must keep the previous archive and checksum. A previous
archive that cannot be put back must be kept outside the staging
directory, with its metainfo and checksum.
A node could not tell a damaged root image download from a good one,
because packimage published nothing to check it against.
packimage now writes rootimg.<suffix>.sha256 in sha256sum format next to
the archive. It builds the checksum in the temporary directory with the
archive, and publish_rootimg publishes both under the publication lock.
A pack that fails before publication keeps the previous archive and
checksum. When packimage removes the other formats, it keeps their
checksums, so a node that already downloads one can still check it.
Until the archive is replaced, the checksum file lists the new and the
previous archive. publish_rootimg computes the line of a previous
archive that has none. The published archive then matches a line of it
at each step, also when the publisher is killed. The previous metainfo
leaves first, so it never describes the new archive. publish_rootimg
keeps a link to the previous archive, or a copy with its mode without
hard links. When a step fails, the previous archive, checksum and
metainfo stay or come back. When it can keep the previous archive in
neither way, it publishes nothing. When it cannot put the previous
archive back, it keeps it in a .rootimg-previous-* directory and names
that directory in the error.
The new site.rootimgchecksum attribute is yes by default, and the
rootimgchecksum attribute of an osimage overrides it. They tell nodeset
whether nodes check the archive. packimage publishes the checksum
whatever their value, because nodes with older boot entries can still
check.
imgimport must replace a root image the way packimage does. Move the
host-scoped temporary directory and the locked publication from
packimage to xCAT::SvrUtils, so both commands can share them. The
publication still covers the archive and its metainfo. The behavior of
packimage does not change.
The publication lock moves to a helper that imgimport also uses, and its
message will name both commands. Match the holder and the recovery step
of the message instead of the command name.
The test also made publication fail with a directory in place of the
previous metainfo. Publication will move that metainfo aside, which
works for a directory. Make rename and unlink of that path fail instead.
Remove source-pattern and helper-call counts. Check key-file creation and
replacement through the real DDNS code, with file access redirected to a
temporary fixture. Keep the signing and named configuration checks.
Run the unchanged initrd builder with fixture binaries and inspect both generated archives. Replace source extraction with checks for selected libraries and their contents.
Read site settings from an isolated database and inspect the sources.list written into the image. Stop at an injected package-metadata failure without downloading packages.
Run genimage and inspect the arguments passed to debootstrap instead of evaluating an extracted assignment. Keep the existing ppc64le pass-through explicit.
Exclude node postscripts through the native RPM and dpkg options. Verify
that upgrade configuration, captured output and cleanup still complete.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
Generate self-contained RPM and Debian upgrade scripts from the shared
source. Package transactions no longer load functions from the node-facing
postscript payload. Keep the existing command, logging and failure policies.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
The man page said that packimage overwrites the root image and that nodes
cannot use it while packimage runs. Describe the replacement instead, and
the disk space that it needs.
The test packs cpio and tar archives with gzip, xz and pigz, and squashfs
images, on the Red Hat path. A wrapper around the compressor, mksquashfs,
cpio or find records the file under the final name while the pack runs,
and then runs the real tool or fails.
It checks that nodes get the previous archive during the pack, that a
pack whose compressor, cpio or file listing fails keeps the previous
archive, metainfo and other formats, that a successful pack replaces
them, and that no temporary file is left. The next pack removes a
temporary file or directory that an interrupted pack on the same host
left, and keeps one that another host is writing.
For a StateLite tree, it checks that a pack that fails while listing the
files, compressing them or creating the temporary directory restores the
files that packimage switched, and that a retry keeps them. When a file
cannot be moved back, the original and the previous archive must stay,
and the next pack must refuse to run.
A pack must wait while another host holds .packimage-publish. When a
pack that did not finish left the lock, the next pack must fail, name the
holder, and keep the previous archive and the lock. With --tracker,
ctorrent must build the metainfo from the new archive under its final
name before the archive is replaced. A ctorrent failure must keep the
previous archive and metainfo, and so must a previous metainfo that
cannot be removed.
packimage deleted every rootimg.* file before it packed, and wrote the new
cpio or tar archive under its final name with mode 0600. Until the pack
finished, a node that booted got HTTP 403 for its image. When the pack
failed, the image was gone or left truncated, and no node could boot until
a later pack succeeded.
Build every archive in a temporary directory inside the image directory,
and rename the archive over the final name only after the pack succeeds,
as the openEuler path already did with a temporary file. With --tracker,
ctorrent builds the metainfo in the same directory, from the new archive
under its final name. The previous metainfo is removed before the archive
is renamed, and the new metainfo is renamed after it, so no failure
leaves a metainfo that describes another archive. The other rootimg.*
files are removed after the rename. The next pack on the same host
removes a temporary directory that an interrupted pack left. Image locks
are per host, so temporary directories of other hosts that share the
directory stay.
A pack now also fails when find cannot list the image or cpio cannot read
a file, as on the openEuler path. Before, only the compressor status
counted, and such a pack published an incomplete archive.
Every failure after packimage switches a StateLite tree to stateless files
now switches it back. That includes a failure to create the temporary
directory or the squashfs staging directory, which File::Temp reports
with die. packimage records each file that it moves and checks each move. When
a file cannot be moved back, the original stays in .statebackup,
packimage reports it, and the next pack refuses to run until the files
are restored. The new archive replaces the previous one only after the
StateLite files are back. Before, a failed pack left the tree switched,
and a retry overwrote the saved StateLite files.
The image lock is per host, and service nodes mount a shared image
directory without NFS locks. So a pack also holds the directory
.packimage-publish while it replaces the archive and removes the other
formats. mkdir is atomic for every host, also on NFS without locks. When
the lock is older than 60 seconds, or stays held for 60 seconds, the pack
fails, names the holder, and keeps the previous archive. packimage does
not take the lock over, because its holder can still be running.
A ctorrent failure now fails the pack, and the previous archive and
metainfo stay. Before, a failed ctorrent left a partial metainfo, and
nodeset pointed nodes at it.
The packimage tests skip their squashfs and pigz cases unless mksquashfs,
unsquashfs and pigz are installed. Install them in the unit test job so
that CI runs those cases.
Vinícius Ferrão
Daniel Hilst