A Subiquity autoinstall installs what its user-data packages list names,
and the templates could only name a fixed set, so the osimage pkglist
reached an Ubuntu node through ospkgs after the first boot. The preseed
token has no autoinstall form: the package list is YAML, one item per
line.
A list line that carries #INCLUDE_DEFAULT_PKGLIST_AUTOINSTALL# is now
replaced by one item per pkglist package at the same indentation, with
includes followed and without repeating the items the template lists
above it. The line is replaced in the include pass of subvars, so a site
template that includes the stock one is served too. A plain name and a task are
installed this way, and a comment after them ends the record. A version
pin or a target release stays with ospkgs, because the installer runs
apt-get without --allow-downgrades and a pin can require one, and so does
a name with an architecture qualifier, because a foreign architecture is
enabled by a postscript that runs later. A record that begins with
a removal or a group is left out whole, as ospkgs removes or installs it
whole, and so are a removal written with a trailing hyphen, markers and
preseed directives. A list that carries a #ENV: setting or an unreadable
include is left to ospkgs whole. So is the list of an osimage with
environvar, which mkinstall now hands over, and such an image's pkgdir
mirrors stay out of the installer's sources as well: those variables
reach apt-get only through ospkgs, and a mirror may need them. An osimage
without a pkglist loses only the token line.
The installer's apt configuration turns recommended packages off, as
ospkgs installs the list without them; curtin writes that setting into
the target, where the template removes it with the installer's sources.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
get_pkglist_tex joins the records of a pkglist with commas for the preseed
and OSPKGS consumers, so a caller that needs the records themselves cannot
recover a record that contains a comma, such as a tasksel directive. The
new reader returns the records whole, comments dropped and includes
followed in place. It reads each line as get_pkglist_tex does and resolves
every include, nested ones too, against the directory of the listed
pkglist, as get_pkglist_tex does. The comma text is unchanged.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
An osimage pkgdir can name mirrors after the install media, and ospkgs
installs from all of them, but the autoinstall apt configuration offered
the installer only the archive mirror and the otherpkgs repositories. A
package that only a pkgdir mirror carries could not be installed before
the first boot.
The mirrors now join the installer's apt sources, next to the otherpkgs
ones, in every form the configuration takes. An entry written as URL,
suite and components is an apt source line, as ospkgs writes it, and a
suite that is an exact path needs no component. A local directory that
is a flat repository is served by the management node and trusted, as an
otherpkgdir is. An entry that names an Ubuntu archive mirror the
installer already has a source for, the configured one or a default one,
carries that source's signing key, the archive keyring on the Deb822
releases and none before them, because apt rejects a second source for
the same suite whose signing key differs. For the same reason a
repository is offered once, whether pkgdir names it twice, as a directory
and as its URL, or the otherpkgdir names it too. Anything
else is no apt source for ospkgs either and is left out.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
The online autoinstall apt configuration offered every otherpkgdir entry
as a flat one-line source with trusted=yes. From 24.04 on curtin
converts a one-line source to Deb822 before it writes the file and keeps
only the type, URI, suite and components, so the unsigned repository
reached apt without the option and apt rejected it during the install.
An entry written as URL, suite and components, the form the otherpkgs
documentation gives for a mirror, was written whole as the URL, which
apt cannot parse either.
The sources are now Deb822 stanzas on those releases, which curtin
writes as they are, with Trusted: yes, and a mirror entry keeps its
suite and components as fields. The releases before 24.04 keep the
one-line form, with the same fields.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
mkinstall mapped x86_64 and x86 to their Debian names and accepted ppc64le
and ppc64el. Every other architecture, riscv64 included, was logged as
"Unknown arch" on each diskful install, although the install went on with
the name unchanged, which is right for riscv64.
Move the mapping into install_darch, which takes the Debian name from
xCAT::Utils::debian_arch and knows the architectures xCAT installs Ubuntu
on. riscv64 is one of them.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
The note shown when boot/grub2/grub2.<arch> is missing told the
administrator it comes from grub2-xcat or the EL installation media. On
Ubuntu neither is true: copycd builds the loader from the grub2 package on
the media, because the image the media carry cannot boot over the network.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
archive.ubuntu.com publishes amd64 and i386 only, so a ppc64el or riscv64
node was given an apt mirror carrying no package for it and the installer
could not fetch what the minimal live media lacks. The default is now the
ports archive for those architectures, chosen from the osimage's
architecture rather than the package directory, which is whatever path the
administrator configured. site.ubuntu_apt_mirror still overrides it.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
grub2 reads its configuration as a script, so an unquoted command separator
ends the linux command and everything after it is lost. The Ubuntu
installer seed is written as ds=nocloud-net;s=<url>, so the node booted
without the seed URL and without the arguments that followed it, including
BOOTIF. The installer then found no autoinstall configuration and waited
for someone to answer its questions. A separator that is neither escaped
nor inside a quoted span is now escaped where it stands, which grub2
removes before it hands the line to the kernel. A value the caller escaped
or quoted keeps exactly the form the caller gave it.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
riscv64 nodes have no boot loader unless one reaches /tftpboot/boot/grub2,
and nothing on an Ubuntu management node puts one there. The grub2 image
the media carry cannot serve: it holds a built-in configuration that looks
for the live filesystem, so a node that loads it drops to a grub prompt
instead of reading the configuration nodeset writes. copycd now builds a
netboot image from the grub2 package the media ship, and warns when it
cannot, because the node has no other source for one. An image already in
place is kept only when it is a whole executable image for the
architecture the firmware loads and carries the prefix this boot path
needs; one that is not is removed, so a rebuild that cannot run leaves
nodeset reporting a missing loader rather than serving an unusable one.
The media of every other architecture are untouched.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
The riscv64 live-server image keeps its kernel at casper/vmlinux, where every
other live image keeps casper/vmlinuz, so the probe found no kernel and mkinstall
reported that the install image was missing.
Add the riscv64 candidate pair and let copycd name the architecture the media
reports. Verified against Ubuntu-Server 24.04.4 riscv64, which carries
casper/vmlinux, casper/initrd and casper/install-sources.yaml.
Signed-off-by: Vinícius Ferrão <2031761+viniciusferrao@users.noreply.github.com>
getcredentials answered xcat_secure_pw only for root, so a postscript
had no way to get the password of another node account from the passwd
table.
xcat_secure_pw:<user> now returns the password field of the passwd row
key=system,username=<user> when <user> is root or a sudoer named in the
postscripts or postbootscripts of the requesting node, its osimage, or
xcatdefaults. A sudoer without a row or without a password gets the
locked field "!", so the node applies the reply as is. Any other user,
an invalid user name, or a failed hash answers with an error instead of
an empty reply. The root request reads the same row as before and keeps
the error reply for a missing row.
copycd translated the architecture the Ubuntu media reports with its own
if/elsif chain, and genimage translates the same names back for debootstrap with
another one. Neither can be reused, so a new architecture has to be added to
both.
Put both directions in xCAT::Utils and have copycd read from there. The names
and the fallback do not change: media that xCAT has no name for still leave the
architecture as the media reported it.
The probe spelled out every candidate path twice inside one nested condition,
once to test it and once to assign it, so adding an architecture meant adding
another branch of the same shape. Move the candidates into a table keyed by
architecture family and walk it in order.
Same paths, same precedence, same failure behaviour: a media tree that matches
nothing leaves the caller on the "install image not found" path as before.
The boot flip in compute.subiquity.tmpl addressed port 3002. xcatd's install
monitor listens on site.xcatiport, so a cluster that moves the port loses the
flip and every node PXE-loops back into the installer. The flip now reads
site.xcatiport and keeps 3002 as the default. TABLEBLANKOKAY, because the key is
optional and a plain TABLE lookup of an absent key fails the whole template.
The flip also counted any reply as an accepted request. It now requires the
monitor's "ready" greeting before it sends "next", and "done" afterwards, so a
different service on that port is not read as a flipped node.
subiquity_nfsroot_server in debian.pm called getipaddr without a family. A
dual-stack management node answers with its IPv6 address, and casper takes
everything after the first colon in nfsroot= as the path, so the live filesystem
never mounts. It now asks for IPv4, as dhcp.pm and mknb.pm do.
The DNS setup wrote the xcatmaster name as a nameserver when getent found no
address, which is the case the step exists to prevent. It now keeps the
resolv.conf DHCP gave the live installer.
ubuntu_subiquity_boot_flip.t, debian_subiquity_boot_params.t and
ubuntu_resolvconf_ip.t fail on the parent commit and pass here.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
`makedhcp -q <node>` on Ubuntu's ISC-limited releases answers "no DHCP reservation
found" when it cannot read dhcpd.conf. The operator reads that as a node without a
reservation. An InfiniBand node also gets an answer with no hardware address.
_query_isc_static_host in dhcp.pm read the file with an -r test and dropped a failed
open. It also matched only a "hardware ethernet" line, while _add_isc_static_host
writes "hardware infiniband" for an InfiniBand node and adds a twin declaration
between the same markers.
_read_isc_conf_lines now returns the read error, _query_isc_static_host returns it to
listnode, and listnode answers the caller with an error. The parser accepts any
hardware type and keeps the first declaration of the block. The path of dhcpd.conf and
the distribution name are package variables, so a test can drive the query and
listnode.
dhcp_isc_static_host_query.t covers the InfiniBand address, the twin declaration, the
unreadable file and the listnode answer. It fails without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT::NTP::Backend->available reported chrony as available on chronyd alone, while makentp
configured chrony only where systemctl was present too. On a host with chronyd and no systemctl
the selector returned chrony with no downgrade, makentp fell through to the ntpd path, and the
admin saw either a silent switch or "Please make sure ntpd is installed".
available now requires chronyd and systemctl for chrony, so the selector answers on the same
terms makentp acts on, and makentp branches on the name alone. choose therefore downgrades to
ntpd, or reports install, in the case it used to pass over. A commands argument injects the
command probe, in the same shape as the existing available argument.
ntp_backend_selection.t covers both commands. Six of its assertions fail without this change.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Fall back to an available DHCP backend on auto-selection. When the request is "auto"
and the backend chosen for this OS is not installed, use the other one if it is,
recording fallback_from so process_request can tell the operator which preferred
backend is missing. A backend the admin forced through site.dhcpbackend still fails
hard when absent, and "neither installed" still errors clearly. Fixes#7710.
Answer makedhcp -q from the static host block on Ubuntu's ISC-limited releases.
listnode now branches on _isc_static_host_fallback() before any omapi work and reads
the node's fixed-address and hardware ethernet straight out of dhcpd.conf, so the
query path never spawns the omshell its own write paths already avoid. A node with no
reservation is now reported rather than answered with silence.
Match the host-block markers exactly. _add_isc_static_host writes a fully determined
pair -- "#xCAT host declaration for <node> aka host <hostname> start" and the "}"
line carrying the matching end -- so both scans anchor on that whole shape through
shared _isc_host_start_re/_isc_host_end_re helpers. The previous /\Q$node\E\b.*/ also
matched at a hyphen, letting node "compute" act on "compute-01"'s block: the query
could return another node's address and the delete could remove another node's
reservation. _delete_isc_static_host also accepts an explicit line list now, so the
scan is unit testable without file-scoped state.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The module overrides two methods of XML::Simple, one for a recent version and
one for an older version, and each built its own parser and set its own
handlers. The two bodies were the same apart from spacing, so a change to one
refusal had to be repeated in the other, and a reader had to compare them to
see that they agreed.
Build the parser in one routine that both call. Behaviour does not change.
Six modules wrote passwords to their own log and diagnostic messages,
outside the daemon redaction pipeline. The z/VM plugin logged each
smcli command line through printSyslog, with the disk read, write and
multi passwords, the image password, the provision root password and
the page volume parm disk password, passed the real disk passwords to
checkSSH_Rc, which echoes the command to syslog and to the client on
failure, and logged raw directory entries whose USER and MDISK
statements carry the logon and disk passwords. The bmcconfig plugin
logged the BMC password in its attribute report, in syslog and in the
command response. The energy plugin logged the HCP password in a
verbose message, and the CIM utilities dumped the whole HTTP request,
with its basic authorization header, to the verbose callback. The PPC
configuration module logged the HMC, FSP and BPA passwords in its
verbose credential reports.
Mask the passwords in the logged text. The executed commands keep the
real values. The page volume log string is built by operand position,
so a decoy value in another operand cannot divert the mask. The
checkSSH_Rc calls receive the masked command string, as the routine
documentation asks. Add redact_directory_entry to the z/VM utilities.
The routine masks the USER, IDENTITY and IDENT logon password, the
MDISK passwords after the access mode in the range form and in the
DEVNO, V-DISK and T-DISK forms, the APPCPASS statement, and the
keyword password assignments in the short and the full spelling. The
match separators stay on one line, so a record without passwords never
masks the record below it, and one or more comment stars do not hide a
credential record from the rules. The COMMAND statement masks whole,
because it can start any CP command with an inline password. Every directory query sink logs
through it,
and the clone loops redact the query output at the source, because the
failure checker and the retained disk list reuse the text. The
directory helpers keep their raw return value for the callers and hand
a redacted copy to the failure checker. Every error branch that echoes
a fetched record after the output check does so through the redactor,
because a password can spell an error word and trip the check: the
directory fetch, the mini disk keyword fetch, and the four disk list
callers. The CIM dump masks
the authorization header. The bmcconfig report now names the password
state, set or missing, which the report needs for diagnosis.
The previous commit deleted the $effective_provmethod override on the grounds
that %image_hash never carries a provmethod. That was wrong, and the review
caught it: makescript builds %image_hash, calls getImage() on it, and then
hands the SAME hashref to getScripts(), which fills provmethod for every
osimage from the osimage table. getDisklessNet() already reads that key the
same way. The override was live, not dead.
Restore it and say what is actually true in the comment. nodetype.provmethod is
frequently an osimage name rather than 'install', and resolving it is the point
of the lookup.
Also close the gap that made the wrong deletion so easy to ship: reverting
mkinstall's subiquity branch to its pre-fix body left the whole unit suite
green. debian_mkinstall_subiquity_branch.t lifts that branch out and drives it
inside a real loop, so the `next` it performs is the one under test, with
report_node_error and the getipaddr seam stood in for. It calls
subiquity_boot_params with no injected resolver, exactly as production does.
The branch is selected out of debian.pm by what it contains rather than by
where it sits -- there are four `if (using_subiquity(...))` in that file, and an
earlier draft of this test silently matched the wrong one and ran past its
block.
Now observable, each verified by mutation: swapping $pkgdir and $httpport at
the call site reddens the nfsroot assertion; reverting the branch wholesale
fails the extraction guard rather than passing.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The daemon reads the XML of every request through this parser. A request could
declare an entity in its own document type declaration, and the parser expanded
it. An entity that refers to other entities grows on each level, so a short
request expands into a large document and consumes the memory and the time of
the daemon. A client holds a certificate before it can send a request, so this
needs an account, but the daemon should not accept the work.
Refuse the declaration itself. The option that stops the parser from expanding
an entity does not cover an entity that a request names inside an attribute, so
it leaves the same growth available through a different part of the document.
Measured on XML::Parser 2.46, a request of 204 bytes that names its entity in
an attribute still grew to 1014 bytes with that option set, which is what the
parser does without it.
No request that xCAT sends carries a document type declaration. The client
builds every request with XML::Simple, which does not write one.
The handler that refuses an external entity stays, so a parser that reaches it
by another route still refuses to read the named file.
Both parser constructors passed a list of options to XML::Parser as an array
reference:
XML::Parser->new(Style => 'Tree', [ load_ext_dtd => 0, ... ]);
XML::Parser->new takes a flat list of pairs. The reference is one value in that
list, so the constructor reads the pairs as Style => 'Tree' and then the
reference as the name of an option with no value. Every option inside the
reference is dropped. The names are also the names that XML::LibXML uses, not
the names that XML::Parser uses, so the parser would ignore them even if it
received them.
The options therefore never did anything, and they give the reader the
impression that the parser refuses an external entity because of them. The
handler on the next line is what refuses an external entity.
Remove them. Behaviour does not change.
The syncfiles deferral resolved the node's provmethod through
$image_hash{$osimgname}{provmethod} when the node names an osimage. makescript
fills %image_hash from getImage(), which stores pkglist, pkgdir, otherpkglist,
otherpkgdir and environvar -- and no provmethod. getScripts() has a separate
hash that does store one, which is where the pattern was copied from. So the
lookup was always undef, the override never fired, and the code claimed a
behaviour it did not have.
Pass $provmethod directly and say in the comment why there is nothing to
resolve it with. No behaviour changes -- the branch was inert -- so there is no
red to show first; what the deletion needs is coverage that the path it was
supposed to serve still works.
That is what the two new assertions do: an osimage-named provmethod with
nodesetstate 'install' still defers, and the same name with no nodesetstate is
not mistaken for a diskful install. nodesetstate is what carries the install
signal here, which is why the override was never load-bearing. Making the
deferral ignore nodesetstate and require provmethod eq 'install' reddens both.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Two things the review found, both in code added by this branch.
Reverting mkinstall's call site -- putting xCAT::NetworkUtils->getipaddr back
in place of subiquity_nfsroot_server, the exact regression the fix removes --
left the entire unit suite green. The helpers were covered; nothing linked them
to production. Compose the two steps in subiquity_boot_params(), which takes its
inputs and returns either a command line or the reason there isn't one, so the
composition can be driven; mkinstall keeps report_node_error and the loop's
`next`. That same revert now reddens 6 of 9 assertions.
The test stubs xCAT::NetworkUtils::getipaddr deliberately. Without it a call
site that bypassed the injected resolver died on a missing module -- a red, but
for the wrong reason. With it, bypassing the resolver returns the wrong answer,
which is what the assertions are there to catch.
The resolv.conf sandbox guard matched `/etc/` with a trailing slash, so the one
respelling its own comment names -- `etcdir=/etc; rm -f "$etcdir/resolv.conf"`
-- walked straight past it and the fragment would rm the runner's real
resolv.conf, as root in CI. `/etc\b` catches it: applying that respelling now
BAIL_OUTs instead of running.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The NTP backend selector was well covered and nothing connected it to makentp:
copying the base makentp.pm over the head one left the whole unit suite
byte-identical, so the branches that consume choose()'s answer -- abort on a
selector error, warn on a downgrade, abort when neither daemon is installed --
and the --backend argument handed to setupntp were covered by nothing.
They were unreachable from a test because they sat inside process_request,
which needs a management node. Move the decisions into ntp_backend_action() and
setupntp_command(), which take their inputs and return an answer; the caller
keeps send_msg and runcmd. No behaviour changes -- the same messages are sent
on the same conditions, and the same command is built.
Verified by mutation rather than by reading: dropping the install abort reds 3
of 15, the downgrade note 1, the --backend argument 2, the server-list split 1,
and the selector-error abort 2.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Vinícius Ferrão
Daniel Hilst