Study of the web interface showed that bulk requests are a key component.
This takes a sensor sweep from about 3 minutes to about 7 seconds in a test environment.
Become a notify type service. Reserve the ability to be normal should it come up, but with
the complications around fork, easiest
to punt on daemonize for now.
To handle amd64/arm64 profiles, have the pkglist allow for architecture specific qualifiers.
Additionally, soften failure to accomplish selinux changes.
If the networking didn't come up well, the 'functions' routines would not be able to handle.
Switch to using apiclient which is designed specifically to handle less cooperative
initial network conditions.
If SPCR comes up blank, see if there is one and exactly one serial with carrier detect
Failing that, give DMI a chance to indicate a preference, for now just SuperMicro, since they have the most
inconsistent carrier detect behavior
but almost always consider ttyS1 to be the answer.
This allows the FS to just live, uncompressed, in cache.
This is generally a bad idea, however:
- In a hypothetically super-tuned diskless image, the lack of double-cache can offset the lack of compression
- The image will have supreme read performance
- It will have the most deterministic memory behavior
By default, the squashfs file was being cached as well as the contents after extraction.
This is superfluous pressure on the cache of the OS.
However, it does help keep the image afloat through 'confignet', so
leave it on until onboot completes, then reclaim cache and disable further caching.
Modern SNMP devices may require AES.
Unfortunately, older ones may refuse AES.
For compatibility, continue to default to DES, but
allow AES to be indicated in attributes.
We use cryptography verification, but it's relatively new.
For compatibility, we fall back to fingerprint only.
This is pretty bad when inflicted on
unsuspecting users on autosign,
so skip autosign if cert validation
would break.
Rather than treat both as the same, since untethered has everything up front anyway, go ahead and extract the filesystem.
This makes the mount look more straightforward and makes it so deletion of files from
the image also frees ram.
For one, apply more rules from CA/B forum. This includes including KU and EKU extensions, marking basicConstraints critical, and
randomized serial numbers.
Also make the backdate and end date configurable, to allow
for the BMC certs to have a more palatable validity interval.
Begin expanding certutil to sign other certificates from external CSRs more easily.
Have certutil make the CA constraint critical.
Have the fingerprint based validator have a mechanism to check for properly signed certificate in lieu of exact match,
and update the stored fingerprint
on match.
Provide a means to request a custom subject when evaluating a
target.
Change redfish plugin to set that subject in the verifier.
The changes for getinstalldisk assumed functionality
in ESXi9. Target older
functional level for our purposes.
Also expand the fallback to cover cases where the disk interrogation fails.
Surprisingly frequently, the firmware stacks split right after the \x1b byte in
sending data down. Defer a dangling partial sequence until more data
comes in that should make it complete.
In the interest of interfering with terminal behavior as little as possible,
only apply the forced intensity if the background and foreground color are
identical and would make it otherwise literally impossible to read
when working as designed.
Terminals seem to expect 'bold or intensity' to imply intense color.
There are certain terminals that steadfastly refuse to do bold and intense. So implement the logic on behalf of
the remote terminal.
Commonly, UEFI setup menus request bold white text on white background. This fixes such menus to be readable by explicitly requesting intense white foreground rather than normal background. For example, the kitty terminal has no 'intense on bold feature.
For nodemedia, nodelicense, and nodefirmware, support
for expressions in filenames was
fouled when pass by
filehandle was added.
Restore this by adding all the files matching an expression.
The stock Ubuntu approach was inadequate. It would DHCP out every nic and take the fastest result, and no going back.
Now the CDC nic can frequently win that race.
First, rmmod cdc_ether, as a scenario that is completely right out.
But beyond that, let Ubuntu have one shot at multi-nic bringup. Beyond that, maintain a list of all link-up devices.
If the check should fail, then start doing one nic at a time, cycling through them.
Also, the openssl s_client timeout is painfully slow, use subshell and kill to speed up things.
When network configuration is applied, wait until we
can reach the deployment server again before exiting.
This should make us more robust against various potential delays after
changing the nature of network interfaces.
Some OS deployment mechanism may wish to convey the identity information more loosely. For those, it's convenient if the files are loose instead
of needing extraction from a VFAT image.
Provide a totally 'clortho' and 'copernicus' free behavior.
This allows some flows to skip the cpio addons to go straight to python.
Some scenarios demand the utilities (initramfs) and others are more awkward with the utilities,
so we enable both.
Recognize BFB embedded OS as a potential osdeploy target.
This is toward the end of identifying the appropriate 'addons.cpio' for setting up for a bf.cfg driven bfb install.
For now, it is disabled until companion os category exists.
Apart frem the gc_thresh indirect check, perform other checks.
For now, just highlight that tcp_sack being disabled can really
mess with BMC connections. Since the management node may have high speed and the BMC may be behind a 100MBit link, SACK
is needed to overcome the massive loss and
induce TCP to rate limit appropriately.
Some platforms can have a very slow category,
like disks. Give CLI a way to ask for the desired
categories and a chance to optimize away the uninteresting.
Samba by default needs executable bit on files for them to be executable by windows.
Only give executable bits to .exe files that are PE32, mitigating the chance the executable bit could mean anything for Linux.
It could still mean something with binfmt misc hooks, but that shouldn't be done much.
Add missing pre.d directory to let user know they can use such scripts
Preserve console directievs from kernelargs into installed system
Retry umount during image2disk, if processes have the filesystem busy.
Fix DNS behavior during post phase of installtodisk
Invoke confignet properly during firstboot to set up additional interfaces.
Have sshd run during the install from '/sysroot', for convenience
Fix some cosmetic error output for setupssh
If an XCC is booting, it may appear before it's ready to use redfish to manage user accounts. Handle this by delaying the discovery until
the service is ready.
Provide checks for nginx config and apache configuration, perhaps even concurrently.
Latch on the first match, since we are taking care of IP based SANs and subsequent server/virtualhost sections are irrelevant.
Latch onto a chain file, if indicated in the apache configuration, placing our CA in the chain.
For nginx, put our CA in the cert, since nginx
uses the 'certificate' file as the chain.
In this scenario, a cross-signed CA cert is possible.
Cache the directory list over a few seconds
to avoid excessive filesystem calls.
Also switchg to a more potent regex to avoid wasting time on timestamped files.
CSV must keep sensors grouped and consistent, so it waits until it gathers
all results to sort.
Normally, it presents the results as quickly as the API provides it.
This results in some different ordering for the "fast as API provides it"
versus "delayed to group the values together" behavior.
openEuler 24.03 seems closest to el8, so start with that baseline.
Modify el8 handling to recognize a totally missing CA bundle and assume
the original el8 location.
Have osimage recognize openEuler media by presence of openEuler release file.
- Set the log directory using the configuration manager
- Add _get_logdirectory function to retrieve the log directory path
- Update _redirectoutput to use the new log directory setting
The UUID and tls material usually comes from site, but some
OSes may depend entirely on the identity image, so
make the UUID available that way as well.
Reduce obvious output about skipped devices.
Rule out any read-only device.
Amend minimum size to 2GB.
Among same priority devices, select the smallest target.
If debugssh is not requsted, the initramfs is not needed.
Do not unshare the mount namespace, leave the kernel relevant namespace 'normal'
Remove some initramfs content to free up some memory.
Newer ansible requires deferred plugin initialization explicitly,
support either ansible norm.
Reliably mark a play as complete even if facing a very early init
problem, e.g. if ansible shebang points to a non-existant python.
Fix incorrect association of stderr to tasks, and present
stderr to the client exactly once.
For various reasons, it is likely/expected for fe80:: to fail the 'local'
check. This is fine in most contexts, except for credserver and ssdp.
ssdp already special cased fe80::, so special case fe80:: in credserver
as well.
If a user has custom
windows categories, match
them to media import.
To do this, we needed to go to pycdlib, as libarchive can't
do UDF-only isos.
For now, this has no progress indication, but does extract it similar
to most Linux distributions are done.
During the async vnc behavior, a copy of the geometry variables
are used that are out of sync.
Workaround by forcing a resize again after the async tasks conclude.
To extend beyond the OpenBmc wrapped dialect of VNC,
provide mechanism for plugins to provide
arbitrary cookie, password, url, and protocols parameters.
Implement for ProxMox.
A common issue in larger layer 2 configurations is
for the neighbor table to be undersized for the number of
nodes.
Detect this manifesting and present a message.
For one, understand 'ip' to potentially mean 'bmc' for list
to assign convenience.
Parallelize handling of csv importing to improve performance.
Only call rescan once per bulk assign
Some kernels may bundle the NLMSG_DONE in the last
useful system call, unlike the previous norm of sending it as
a single message in a terminating system call.
302 works for iPXE, but not for more limited UEFI
http client.
If we are dealing with UefiHttpBoot, check for a header
from nginx config and use X-Accel-Redirect to induce proxy side
redirect transparent to client.
Otherwise, give an error indicating the issue with the profile
name length and incompatibility with Apache capabilities.
When trying to set a node or group attribute, evaluate
length of any potential formatting specification to keep it under
8 characters.
This should prevent even temporary expansion over 10MB for an attribute
on the way to setting it.
Some profiles may want to have a fixed boot image,
and site specific content limited to the identity payload, or at
least the TLS so it could fetch the rest over https.
If an expression causes an inordinate amount of memory to be
used, then block it from continuing.
For now, we consider that an expression that expands beyond 16k. I
am unable to conceive of a use case where someone would want to
use an expression to derive more than 16k as it stands, as we don't
carry any particularly large opaque payloads right now.
The r8169 enjoys some popularity.
Also, we'd like to be able to try out diskless/cloning with
VMs, so vmxnet3 and virtio_net are handy to round that out.
For both proxmox and vmware, properly model the 'oldstate' feature.
For proxmox, further:
- Wait for power change to actually take effect
- Change reset to a cycle, to help nodesetboot actually work correctly.
This allows users to opt into disabling setting further profile changes.
Nodes may be 'unlocked' (normal), 'autolock' (will lock on next
completion), or 'locked' (unable to change the pending OS profile)
If a nic were aliased *and* the node had attributes
for ipv6 but used host resolution for ipv4 identity,
it was possible for PXE to pick the wrong way
to respond.
Instruct netutil to specifically consider only the matching family
for the PXE/HTTP boot context
If a confluent collective member is stopped, then the
HTTPS check passes. If we end up with a 503 indicating
the other end has a missing confluent, fall back to the loop
to check for other living collective members.
This provides nodeinventory (mac and -s most interestingly),
nodepower, nodesetboot (and by extension, nodedeploy -n),
and nodeconsole (console.method=vcenter).
localhost was added to ssh principals, but should not be used
as a candidate in syncfiles.
The syncfileclient should already be filtering this possibility,
but best to filter it everywhere that makes sense.
A node with private, unroutable addresses relative to
the deployment server may cause the deployment server
to select an unroutable address.
Address this with two strategies.
First, if any of the addresses appear local to the deployment server
networks, prefer those and filter out unroutable.
Secondly, if a node is purely remote, and thus all addresses routable,
then make all the addresses a candidate. However, since the
client can't possibly be using fe80::, we can replace the principal list
with just the clientip, provided it appears in the principal list.
If a node has not been asked to open any locally managed
video consoles before it was asked to open a peer managed console,
it would fail to start the needed vinz service.
Work around this by detecting that scenario and giving
the vinz subsystem a chance to fix itself.
Node attribute ntp.servers in nodeattrib ca now be used in stateless images
modified: confluent_osdeploy/el7-diskless/profiles/default/scripts/onboot.sh
modified: confluent_osdeploy/el8-diskless/profiles/default/scripts/onboot.sh
modified: confluent_osdeploy/el9-diskless/profiles/default/scripts/onboot.sh
It turns out that specifying height and width explicitly
does not guarantee that the image protocols will actually fill
the specified space. Notably iterm will honor aspect ratio
(which is good), but leave the cursor where the image would
naturally leave it (which is difficult with relative positioning).
Previously, relative positioning was used as a workaround
for the fact that save/restore or any absolute positioning may
be fouled by incurring scroll.
To make cursor save/restore work, we determine the total rows and
print newlines enough to incur scroll and then move cursor back up.
This lets us use save/restore to ignore cursor movement by the image.
When parceling out the screen real estate, avoid either the height
or the width from getting way out of proportion.
Better to let screen be unused than abuse it to distort the
aspect ratio too much.
Only for kitty graphics protocol.
Also, attempt to use pillow to convert, if available. Kitty itself
needs this, Konsole can work either way.
It currently does not preserve aspect ratio, to do that
we pretty much need to do some work with pillow.
If we specify just the height, then ratio is preserved, but it won't
honor the designed bounding box on wide screenshots. Also
Konsole won't even honor just one scaling factor.
So the better thing would be to determine the aspect ratio, which
needs pillow.
This directs CLI with image output to use a preferred protocol.
This is retroactively applied to stats.
Currently we prefer kitty, as it seems to be the most widely supported.
Though some things only support iterm, so that's an option.
And some only support sixel, but the user has to be the one to
figure out adding pysixel dependency.
The fe80:: is hopeless, try to send ipv4 just in case.
Technically speaking, the user may be using a different address
or real ipv6 and the ipv4 guess might fail, but it probably won't.
On exit, clear the terminal buffer and invalidate the session. This
avoids the web ui being very attached to a closed, dead session, and
leaking stale buffer to a reused sessionid.
For confetty, treat starting a shell session more like starting a
console session.
If an attempt to resize a dead session is attempted,
ignore failures.
RedHat makes grub redundantly handle serial output that firmware
already handles. If we detect EFI firmware and SPCR ACPI table and
connected serial port, that suggests that firmware will handle.
Ubuntu hates serial console by default, amend it so it can actually
work for serial users.
Have block devices checked for identity information
in a loop with network source search.
Block devices may be delayed for various reasons. The previous method
could be bypassed by fast block device cutting off slow device
enumeration. It also incurred a delay for the network install
case.
It turns out that when busybox invokes openssl for
IPv4, it does not pass a servername field.
In this case, start amending arguments after '-verify' instead, to catch
the verify_ip argument correctly.
The busybox wget invocation of openssl is broken.
Override by stubbing it out to let openssl pick the verify
hostname instead of wget specified one, which is incorrect.
The stock reverse proxy configuration strips the leading
'/confluent-api/' from the URL.
However, when doing a custom reverse proxy set up, one may preserve full
path without knowing which way to go.
Since '/confluent-api/' will never be used inside the api, just strip
it when detected to tolerate either of the likely proxy pass behaviors.
People have been putting the BMC interface as
a net section, to aid in their information
and confluent2hosts.
Tolerate that by dropping net entries that match the
hardwaremanagement.manager attribute.
Shell sessions are now wired up to vtbufferd
The shellserver now correctly accounts for sessions being started.
The sockapi now correctly allows the client to specify/attach
to a specific session id.
A service.cfg configuration can be applied to ignore nics for netboot
# cat /etc/confluent/service.cfg
#[http]
#bindhost = /var/run/confluent/httpapi
[netboot]
ignorenics=enp65s0f1np1,enp65s0f3np3
When the team modes were defined in attributes, it was based on the
teamd names. Since the ecosystem abandoned teamd, we went back to
bond.
However, we neglected to map all the names to the closest bond type equivalent.
Change confignet to do the mapping.
Newer versions of websocket change internal call, and we must follow.
This is a consequence of the library providing no means to customize the
TLS handling, so we have to dig in a little to get that customization.
It may happen that the first pass at nics misses
a viable network interface due to slow link up
or slow spanning tree forwarding.
Repeat the loop through the interfaces to have follow
up chances at success.
The SMMv3 doesn't respond to the correct SSDP service, add the
odd service.
Have SMMv3 use the standard redfish handler.
Augment the standard redfish handler to deal with non-error
password change required message.
Eventlet has "helpfully" stopped supporting headers with
underscores. Restore them since we want to support
backwards compatibility and do not have the option to
just ignore existing clients.
If the power state stays the same between queries, take no action to clear screen and
draw redundant data.
In the case of misreporting devices, it mitigates the impact of incorrect reporting,
while generally preserving the output behavior when accurate.
The 'team-slave/bond-slove' type is unneccesary, and
messes up with infiniband.
NetworkManager gets the idea if the 'ethernet' is a bond member without being told explicitly.
A variant of the M.2 RAID enablement kit does not manifest with nvme
driver. Address this by allowing 'nvm' subsystype. to allow blank driver.
Also, to be on the safe side, have self.driver always be a string,
so it can be 'falsey' but still work as a string.
If client requested more than one on a fresh confluent run,
then only one of the video consoles would properly wait.
Fix by wrapping the assure in a startingup check.
Try to hit likely DNS names, or at least provide a means
of manipulating /etc/hosts to induce
a good domain for the default certificate SAN fields.
Note putting the FQDN first in /etc/hosts will get the FQDN in the
certificate.
We may not know where we are going, but at least bump the minor number.
Ultimately we may not be building toward that number, or that number will be used in a different branch, but it
can at least handle more cases
In a scenario where we are doing the 'next-server' next to a real dhcp server,
but through relay, the missing gateway would be expected.
Rely on the final message about 'no address' as the clue to users
that something went wrong on the node side.
In routed deployments, the scan mechanism
will not be available. Fall back to routed
access to the deploy server as indicated
by deploycfg from install time.
Use the relay DHCP agent information as basis for
subnet comparison instead of self, when present.
Use the candidate prefix length verbatim since the
relay will offer no hint as to the 'proper' prefix
length for the target segment.
Instead of overwriting the SSH public code for the node concatenate all
found SSH keys together in one file.
Signed-off-by: Adrian Reber <areber@redhat.com>
Utilities that expected /dev/pts will now be satisfied,
as a new /dev/pts is mounted.
Further, systemd added a check in various utilities that
was fouled by the previous method of appearing to have a
root filesystem.
Before, after chroot, we would bind mount / to itself, and this
made things using /proc/mounts, /proc/self/mountinfo, df, mount,
etc happy that there is a real looking root filesystem.
However, by doing it after the chroot, systemd could statx on '..' and
get a different mnt id than /. So it had to be done prior to the
chroot. However it also had to be done before other mounts as
bind mounting over it would block the submounts.
This more closely imitates the initramfs behavior, where '/' starts life
as a 'real' filesystem before being mounted up and switched into.
This behavior was made to imitate the 'start_root.c' behavior as that
seems to be more broadly successful.
If an asynchronous handler is slow to
enroll a target while another target causes an iteration
of the snoop loop, the various modified structures
had been discarded in the interim.
Now persist the data structures iteration to iteration,
using 'clear()' to empty them rather than getting
brand new data structures each loop.
For XCC3, change to generic redfish onboarding mechanism.
Extend the generic mechanism to be more specific in some
ways that the XCC3 is pickier about. However, it's just reiteration
of what should have already have been the case.
V4 Lenovo servers will have XCC3, and will have differences
and mark an unambiguously redfish capable onboarding process.
For now identify XCC3 variants and mark them, stubbing them
to the xcc handler.
An XCC3 handler will be made basing on the generic redfishbmc handler
with accomodations for XCC specific data (e.g. DeviceDescription
attributes and the Lenovo default user/password choice).
While stock OpenBmc does not care about subprotocols,
some implementations use it as a carrier for the XSRF-TOKEN.
Since base OpenBmc ignores it, we just offer it to any implementation
just in case.
Provide mechanism for administrator to place a custom
key for potential interactive recovery into
/var/lib/confluent/private/os/<profile>/pending/luks.key
If not provided, generate a unique one for each install.
Either way, persist the key in /etc/confluent/luks.key, to
facilitate later resealing if the user wants (clevis nor systemd
prior to 256 supports unlock via TPM2, so keyfile is required
for now).
Migrating to otherwise escrowed passphrases and/or sealing to
specific TPMs will be left to operators and/or third parties.
The comment based hook is destroyed during early install process.
Use python to manipulate the autoinstall file in a more sophisticated way.
Also refactor the initramfs hook material to be standalone files.
Particularly if traversing a lot of linked configuration, the same key/cert
path may come up multiple times, check for equality
and if equal, just keep going.
Particularly if traversing a lot of linked configuration, the same key/cert
path may come up multiple times, check for equality
and if equal, just keep going.
Previously, items were randomly arranged in lists in the json dump. This meant that the JSON files were different after each export.
Now they are naturally sorted and identical.
This should make it easier to save and compare the JSON dumps in version control systems.
Start implementing a tpm2-initramfs-tool based approach.
This requires a bit of an odd transition as the PCR 7 is likely
to change between the install phase and the boot phase, so
we have to select different PCRs, but that requires
an argument to pass that crypttab does not support.
User could accidently run 'confluent' in a way that makes no sense,
block it the most accessible way.
The pid file should have blocked it, but systemd purges the directory
even on failure.
PyCA changes their minds about which bindings to include.
So make the binding ourselves since PyCA removed it in certain versions.
This is a backport of the implementation from the async port effort.
The infiniband section must be defined for the OS
to use the IB link. If it is missing then networking
does not come up during firstboot.
Fix this by having an inifiniband section including explicitly
declaring use of datagram mode. This should suffice for all
install use cases, and may be changed after firstboot starts.
Sometimes stateful install can fail if vgchange -a n is run after dd.
Use wipefs instead and fix order of both commands.
Furthermore, use the $INSALLDISK variable.
Technically, Grub never had 'linuxefi/initrdefi' commands
officially, so this is a bit weird.
However, if we see signs of GRUB older than 2.03, we will assume
that is requires the linuxefi/initrdefi commands from
the out of tree patch to support EFI the old way.
This corresponds with EL7. Other variants seem ok with
the more proper linux/initrd command names.
Create a generic redfish discovery and a MegaRAC specific
variant.
This should open the door for more generic common base redfish discovery
for vaguely compatible implementations. For now, MegaRAC only
overrides the default username and password (which is undefined
in the redfish spec).
Also, have SSDP recognize the variant, and tolerate odd nonsense
like SSDP replies coming from all manner of odd port numbers (no
way to make a sane firewall rule to capture that odd behavior,
but at application level we have a chance).
Create a generic redfish discovery and a MegaRAC specific
variant.
This should open the door for more generic common base redfish discovery
for vaguely compatible implementations. For now, MegaRAC only
overrides the default username and password (which is undefined
in the redfish spec).
Also, have SSDP recognize the variant, and tolerate odd nonsense
like SSDP replies coming from all manner of odd port numbers (no
way to make a sane firewall rule to capture that odd behavior,
but at application level we have a chance).
It was possible for proxyDHCP to look past the network designated end of packet.
Fix this by consistently using the memoryview that was trimmed to size.
Add a mechanism to close a session the right way
in tlvdata
Fix confluentdbutil/configmanager to restore/dump db to directory
Move auth to asyncio away from eventlet
Fix some issues with httpapi, enable reading body via aiohttp
Fix health from ipmi plugin
Fix user creation across a collective.
Some subprocess calls were reworked to use asyncio friendly
variants.
Also, osdeploy initialize was checked, and reworked the ssh and tls
handling.
osdeploy import was also reworked to functional with async only.
Have util retain tasks that are 'fire and forget', to avoid
garbage collection trying to delete the background tasks.
Move some utilities explicitly over to asynclient/asynctlvdata that
had previously been reworked.
Implement terminal resize in new asyncssh backend.
Since a lot of the traditional client did not need async,
make life easier by just having them in parallel for now.
The server must use the async client, but the client applications can
stick with the somewhat more straightforward synchronous client.
One issue is that there are multiple networkmanager connections,
clean this up, though this seems not to be a functional issue.
However, sometimes the lldpad usage screws up network configuration,
disable the facility by forcibly disabling fcoe sincec that is what triggers lldpad.
wq
One issue is that there are multiple networkmanager connections,
clean this up, though this seems not to be a functional issue.
However, sometimes the lldpad usage screws up network configuration,
disable the facility by forcibly disabling fcoe sincec that is what triggers lldpad.
wq
With asyncio, we must close the writer half of a pair
Also rework the get_next_msg to work better.
Still need to allow stop_following to interrupt get_next_msg
Make process_peer async, with socket connection being async,
and dependency.
Have getaddrinfo use the asyncio version.
Rework the snoop to be more effective.
Rework the scan to be less convoluted.
Offer a function in core to normalize plugin return.
A plugin might return an async generator, a traditional generator,
or might even return an awaitable wrapping a traditional generator.
Replace eventlet spawn with util spawn in discover core
Have node attribute update await the set_node_attributes appropriately
The change to tolerate either a space or no space ended up
greedily consuming whitespace. Do best possible in two cases:
For log, use the first line as a clue, and consistently pad or not pad
according to first line. It won't catch different pad strategies, or
handle first line being indented but other lines not being indented.
For the textgroup variant, allow subsequent lines to revise the
pad downward, and accept any whitespace, not just space.
If syncfiles fails, keep it retrying.
Also, slow down sync checking to avoid hammering the system.
Further, randomized delay to spread highly synchronized requestors.
Block attempts to do multiple concurrent syncfile runs.
It is theoretically possible for a client to get disconnected
right in the middle. In such a scenario, err on the side of letting
the mechanism stay armed for the sake of a retry being possible.
Restore 'as available' behavior to noderange over socket
Bring the httpapi to the point where the webui is able to start working,
notably bringing the asynchttp online with the websocket.
Fix a flaw in the async ipmi that would cause hangups.
Purge sockapi of remaining eventlet call
Extend asyncio into the credserver to finish out sockapi.
Have client and sockapi complete TLS connection including password checking
Fix confetty ability to 'create'.
Refresh getcsr and installcert to handle latest firmware.
Also add ability to have pre-existing CSR, and trust the SAN on the way through.
If this becomes more properly a feature, then would likely impose a SAN
on certs, similar to the SSH principals, rather than deferring to the CSR
to get it right.
Since we are rebasing to at least Python 3.6, and with
some extra ctypes wranging of the ssl context, we can likely
remove PyOpenSSL. Take first steps by removing it from 'sockapi'.
Have confluent executable become the 'top level' for eventlet, to allow
work on 'de-eventleting' on 'main.py'.
Rework tlvdata to deal with either a socket or a reader, writer tuple.
Using TLS with asyncio is easiest with the 'open_connection'
semantics, which force either a Protocol handler (callback based) or
dual streams. While protocol approach ends with a more socket-like
'transport', the 'protocol' half is a bit unwieldy. So reader and writer
streams instead.
Some versions start manifesting nvme devnames with 'c', which
are to be used to interact with multipath to have raw devices
backing a traditional nvme device.
The code was comparing two string constants, instead of
a variable to a constant. Correct the problem to enable
the preflight checks to work as intended.
When doing pxe and the noderange of the candidate managers fails,
try again without validation in case the user omitted collective members
from nodelist, but still used ',' to enumerate them.
Some firmware cannot tolerate a web session being
active during a rename. Make sure logout has been done, and
give a retry if needed to let the session close out after
logging out.
Some firmware cannot tolerate a web session being
active during a rename. Make sure logout has been done, and
give a retry if needed to let the session close out after
logging out.
For one, when using confluent expressions, induce {} to be an error to
trigger an error for someone trying to xargs something.
Another is to add warnings when clear does something deliberately, but
is something that might surprise a user, steering them toward
what they possibly might want to do instead.
When doing pxe and the noderange of the candidate managers fails,
try again without validation in case the user omitted collective members
from nodelist, but still used ',' to enumerate them.
Fix various callbacks when using IPv6 based deployment.
Do not attempt to restore erroneously cloned zram partitions.
Convert LVM names to new LVM names consistent with source naming scheme.
Push new kernel command line into /boot/loader and /etc/kernel/cmdline.
Reap ssh-agent to avoid stale agents lying around.
Remove nuisance warnings about virbr0 when present.
Do a full runthrough as the confluent user to ssh to a node when user
requests with '-a', marking known_hosts and automation key issues.
Reap ssh-agent to avoid stale agents lying around.
Remove nuisance warnings about virbr0 when present.
Do a full runthrough as the confluent user to ssh to a node when user
requests with '-a', marking known_hosts and automation key issues.
For relay agent options, preserve and echo back the option,
needed for certain environments.
Also, it turns out that for whatever reason on some platforms, iPXE's
proxyDHCP logic can't seem to get a reply. In this scenario,
provide the filename in the DHCP offer without waiting for proxyDHCP.
This change may be worth evaluating more broadly, but may carry risk of not working
right with unmanaged DHCP servers.
The code was comparing two string constants, instead of
a variable to a constant. Correct the problem to enable
the preflight checks to work as intended.
Add capability to fingerprint media without doing a full import (/deployment/fingerprinting/)
Add fetching the profile info as json under the /deployment/ api.
Prepare to support custom distribution name on import
online repositories may not be accesible for the cluster
nodes but were added from the content.xml. Editing this
files with initprofile.sh is impossible as they are executed
in parallel, so all repos starting with
https?://download.opensuse.org
are removed during post
Signed-off-by: Christian Goll <cgoll@suse.com>
Some versions start manifesting nvme devnames with 'c', which
are to be used to interact with multipath to have raw devices
backing a traditional nvme device.
If bind mounts are in use, it will foul the capture.
Notably, one example is if you install the firefox snap in
ubuntu, snapd creates a bind mount.
This will ignore bind mounts, and rely upon the system to
put it straight.
If the ip command shows altnames, do not
let the altnames interfere with locking
on to linktype.
Further, use show dev instead of grep to be more specific.
Whether due to the management node or node IP addresses,
check if deployment can reasonably proceed using IPv4 or IPv6,
and give a warning with some suggestions to check.
Also, add nodeinventory <node> -s as an example resolution for missing
uuid.
This opens the door for normalized common sensors
for clients that care about the semantics but
cannot keep track of inconsistent sensor names from
implementation to implementation.
Unfortunately, apache can get a bit odd over how it
reports a non-viable open socket for keepalive, which
can happen in certain windows.
Disable the keepalive feature and take some performance penalty in
browsers for the sake of more consistent return behavior and
fewer idle greenthreads doing nothing.
The yaml python default behavior is 'pure python' and is
tortuously slow.
As a test, yaml dump of a 17,000 element list took 70 seconds in default configuration.
Opting into the C functions, that time comes down to 10 seconds, a
nice and easy improvement for generic yaml.
For dumping a simple dumb list (e.g. the nodelist for ssh), a special
case yaml-looking result is done, which hits 0.4 seconds on that same
test. So this special case is added to nodelist, which can be very long
and very in demand at the same time.
This will take care of padding when
padding is consistent across a range.
However, we still have a problem with a progression like:
01
02
...
98
099
100
Where numbers in the middle start getting padding unexpectedly without a leading digit.
For one, shorten the DNS timeout, if the DNS server is completely out, give up quickly.
For another, if a host has a large number of net.X.hostnames, the sequential nature
was intolerable.
Have each network be evaluated in a greenthread concurrently to serve
the DNS latency concurrently.
Since the multi-iterator ambition is out,
ditch the expensive set wrangling step.
Now the procedure is:
-Suck nodes into groups, as possible
-Separately for groups and nodes:
-Sort the elements
-Chunk the elements based on 'non-numberical' situation matching
-analyze the iterators to apply [] to shorten the name
-Multi-iterator will cause a discontinuity, and a new ',' delimited name gets constructed
There's too many cases that can go wrong.
Note that with this lower ambition, it would be possible to
significantly streamline the implementation.
Notably, the 'find discontinuities' approach
was selected to *try* to
support multiple iterators,
but since that didn't pan out,
a more straightforward
numerical strategy can
be used from the onset.
The info is hard to put together client side, but
supremely easy server side.
Provide a nice call to
get the layout for a noderange, similar to (but better than) current
GUI code.
Now GUI can get a nice canned JSON
description of the layout.
Consult collective.manager
to decide to skip
consideration of a node, if
that node shouldn't be managed anyway.
This should avoid "cross-island" behavior for such
environments.
Setting attributes can be a touch expensive, since
there's a high risk
of this being old news,
check that discovery hasn't already set values
before trying to set them again.
When booting and infiniband comes up as the 'get started' nic,
make the entry use correct type so as to avoid
confusion later, particularly when confignet comes along to fixup
the configuration.
netplan, like others, makes the questionable choice to designate DNS
as a NIC specific setting, despite not mapping well
to a NIC.
Since we model DNS like NTP, a global, just repeat the DNS config for
every interface. This redundancy is fine in testing multiple
interfaces.
Please enter the commit message for your changes. Lines starting
with '' will be ignored, and an empty message aborts the commit.
On branch master
Your branch is up to date with 'origin/master'.
Changes to be committed:
modified: geist.py
While trying to address one
deprecationwarning, we
hit another. Check if the
new function exists and use it,
falling back to distutils if
everything fails.
ssh module was pausing input for the
entire websocket while doing the simple 'write' operation.
Change to background the actual
logon processing,
rather than blocking what should be a fairly trivial write operation.
Main difference from EL8 is different grub strategy. To cope, capture
the uuid of new /boot and manipulate the
stub grub.cfg with the right uuid.
In EL8, the efi has the main grub, EL9 changes to have /boot host the 'real' grub,
causing us to have to modify the stub grub.
For one, remove 'non-voting' members from being leaders.
Large number of leader candidates create long delays for
converging on a valid organization. Further, some treat 'non-voting'
more roughly, inducing the worst case convergence scenario of unclean
shutdown of leader.
Convergence now happens fairly quickly for collectives with large
number of non-voting members.
During initial DB transfer, the leader would be tied up unreasonably
long handling the jsonification of a large configuration. Offload to a worker
process to allow the leader to continue operation while this intensive, rare
operation occurs.
Reliably run a reassimilation procedure for the lifetime of the leader.
This allows orphaned members to be prompted to join the correct leader.
Serialize the onboarding of a connecting member, and have redundancy more gracefully
paused. This avoids excessive waiting in lock and more deterministic timing
with respect to timeout expectations by the connecting system.
When udev is populating the disk hierarchy, it can be a long time
before the 'by-label' is specifically ready.
Wait for that specific entry to come along before continuing to
check if there's an identity image.
Unfortunately, python being python, ansible may elect to install under a
different python runtime than confluent.
In such a case, hope that the ansible python can work.
Remove eventlet as a hard requirement, as
that is unlikely to be in ansible python.
This leaves msgpack, which is unavoidable, for now. But if it's in ansible python,
it's fine.
When a session is being discarded, it
may inadvertently hold onto logonwaiters.
The IPMI session may be reused, but the context is new and those
dead contexts can consume memory unreasonably.
When doing network evaluation in ambiguous contexts,
spend some time checking each nic and each candidate target ip address.
This may take longer, but options are limited in a routed context.
Get most things automated for the targeted use case.
Still need to handle partitioning and end of install question, add firstboot and add
staged and complete step.
When the time comes to transition, this will be useful.
The challenge is that existing images will likely have a 'distribution'
with fuse2 libs, so this isn't a transition to make lightly.
Definitely should be done for new distributions at least, current
distributions pending further investigation
For older Ubuntu 18, the automated installer is debian-installer. Derive from the
prototype debian work, with same limitation about more limited network bringup that
we are stuck with from Debian limitations.
If 123abc-abc123 were attempted, it would produce
a strange error
when it encounters a string compare to number.
Detect the scenario and treat it like similar situations.
Some plays expect the
inventory from /etc/ansible, detect
if that inventory *could* apply, use if it can.
ansible users generally anticipate leveraging the implicit '.' in the role
path,
chdir to meet that expectation.
'become' in the play without 'become_user' can upset ansible runtime.
Since we are already root, we will just ignore the implicit 'become', since we already
match that behavior.
Provide for applications
where only a small subset of collective
members should be
considered to count
toward whether the collective
can proceed.
Commonly, 'service' nodes may
be numerous to do work, but may all want to go offline
during a maintenance window.
confignet is special, it is designed
to work when networking
isn't right. So have it run during firstboot
in case post fouled up
the network for firstboot.
While servicing an enrollment,
there's a window for a collective
member to be 'defined' but not
yet active, meaning quorum may transiently be lost as multiple enrollments progress.
Serialize enrollments by holding the enrollment process open.
Also, there is a chance that a transient transfer error may occur during loading
of the DB. In such a case, restart
the connection rather thn aborting.
Floats are either unnecessarily long
in normal output, or too unconstrained in CSV output.
Normalize to as many digits as 'makes sense' up to 5 digits.
5 miight seem a bit much, but one common metric is kWh, which may need
that precision over short intervals.
Added 2 new function to check if the custom yaml file exists - /etc/confluent/authorize.yaml - and one to update the _allowbyrole and _deniedbyrole vars accordingly.
The open file handle as implemented
could not pass to the subprocess.
Rather than figure out how to open
and pass the filehandle,
simply let the subprocess
independently open the file
if it isn't passed.
While Eaton does not do HTTPS by default,
it can be configured to do so.
Support when available.
Mitigate downgrade attack by
stickying the cert fingerprint.
If fingerprint is present, then refuse
to even think about port 80.
In some scenarios, the 'default'
interface is overlapped by another connection, either
identical or as a superset in a bond.
Whittle down the default
interface if superseded
to mitigate duplicate interface setup.
Newer msgpack refuses the encoding argument, use raw=False instead.
Further, newer msgpack refuses to accept int as key by default.
Opt into it as the risk is hash collision due to msgpack int being used directly, and
we aren't dealing with untrusted
peer (we only talk to ourselves).
With V3 systems, we can now ask
the SMMs for the certificates
and use that for a verified
measurement, regardless of
whether the XCC is returning
the correct bay number.
For now, keep using x86_64 as
default, but allow overrides
for other architectures.
One day it may be cleaner to move all addons.cpio to
arch specific subdirs.
In certain environments, Confluent may have an IP address that
is fake, but then there is elsewhere with that same IP for real.
To mitigate this, follow up basic connectivity with proof of having
an associated certificate.
This will provide easy way for
client to get FDB data, potentially
for use in conjunction with discovery data.
For now, leave LLDP out, as that isn't currently cached
at the confluent layer.
Old confluent DB may have None in role. This is no longer
allowed. Restore such entries by coercing them to 'Administrator'
which is how old confluent treated such users.
For autocons, though it's copying from a static source, use strncpy anyway,
despite the length being hardcoded already. This makes static analysis happier.
Terminate the buff with a NULL. This is superfluous as the strcpies that preceed
are guaranteed to null terminate, or exit the program.
In clortho, free(tmps), which is a valid leak, though clortho isn't long running.
Also, explicitly return 0, which is ultimately returned by main().
Static analysis could not figure out that padneeded implies that keylen is short of
chunk size, so change the check to be expressly the scenario that static analysis
was worried about directly, rather than indirectly.
Hint to static analysis that we don't care about the time as a time value by masking the
lower 32 bit explicitly. This was already happening, but static analysis was afraid
that we wanted this as time instead of just some mutating value.
Python 2 and 3 are inconsistent
with how they treat memoryview,
but they are consistent on bytearray
treatment
Since rqv is merely a cheaply sliceable view of rq, use rq directly
for functions where the difference
between 2 and 3 would matter.
This uses a more self-evident breadcrumb to intuitively override
for users not wanting to use the confluent facility for timezone
adjustment.
There are other 'peculiar' substitutions that may prefer a breadcrumb
but they may require structure that would be tricky to implement
while also passing validation.
It's clear that the service will need
to explicitly track subscriptions
to enable rescan, for example,
and thus might as well restructure
the API around this information.
This provides a self-evident entry point from
CLI to extending the discovery to
affluent switches that support it.
This function will work with newer affluent
and Lenovo XCC2 systems.
With significant firstboot output, there was a tendency
for tail to be killed before it relayed all the content.
Change to run the firstboot in a subshell in the background,
and have tail explicitly run until that subshell naturally
exits and then tail will cleanly exit
This allows user to designate certain networks to be treated as
if they were local.
This enables the initial token grant to be allowed to a remote network.
This still requires that the api be armed (which should generally be a narrow window of
opportunity) and that the
request be privileged, it
just allows remote networks to be
elevated to be as trusted as local.
This enables a more manual approach
to indicate the deployment server.
This carries the assumption that a
normal OS autonetwork config
will get the node to the right network.
This is one step toward enabling a scenario where the target is remote and the DHCP is not going to relay, but instead the deployment feeds the DHCP a confluent URL entry point to get going.
Using this parameter precludes:
-Enhanced NIC auto selection. If the OS auto-selection fails to
identify the correct interface, the profile will need nic name baked in.
-Auto-select deployment server from several. This will mean that any
HA will require IP takeover be externally handled
This is of course on top of the manual process of
indicating confluent in kernelargs.
Add a seting to allow user to suppress all DHCP offer during
PXE/HTTP activity. This enables configurations
where users want to externally manage filename explicitly in their own dhcp configuration.
In some environments, there's a desire to manually manage DHCP configuration.
In such a case, provide a url
that can be given to the dhcp server
to allow confluent to control the profile
without updating such a DHCP service.
With this change, a node can be told to boot:
http://confluentserver/confluent-api/booturl/by-node/n123/boot.ipxe
To be redirected to the currently applicable os profile.
The default timeout is overkill in the nodediscover scenario.
Notably, we can receive replies from unreachable IP addresses,
and those will extend rescan to the full timeout. The devices should
comfortably reply within 3 seconds, making scans exit in
a timely fashion.
Various parts of confluent that go to try to use
all the interfaces will now skip bond members.
One example problem is that joining the SSDP multicast
group for SSDP would cause the kernel to IGMPv6 out
on bond members as well as the bond itself. This change
ensures that the bond interface is only used and never
bypassed.
It is likely that a client connects from fe80::, which
is explicitly omitted from ssh principals.
This time, have the client provide all currently set IP addresses
and the server will make a determination.
There remains the possibility it misconfigures a nic and tries to use that,
inducing failure. One strategy would be to filter the addresses and
only provide from the 'current' interface. Another is to just take
the hit as the node is likely going to suffer a lot from such a
misconfiguration anyway.
When a node installs, it may not have it's node mapped address up,
or may not have one at all. Try to use the ip if it would be in the
same set that produced it's ssh certificate.
There remains a gap if a system has no static addressing *and* doesn't
map nodename to IP, but we have an impasse as the situation is too fuzzy
to grant a prinicpal in an SSH cert, and without that we can't securely
attempt rsync. For now, this scenario would still fail and I will
just hope that doesn't come up.
Some profiles may have all disk support suppressed through blacklist until %pre comes
along to fix it. This avoids /dev/disk ever existing.
Wait up until 10 seconds before giving up. This gives disk subsystem a fair chance to
speed up and avoid a wait, with a fallback worst case of 10 seconds
The change to allow CIDR syntax
broke for configurations that use name for
bmc 'address'.
Fix by letting getaddrinfo have a chance to process the ip before
trying to pton it.
When remote ip is detected,
communicate by returning False
instead of None.
Use this indication to let ssdp
skip a transmit and growing
pending list in such a case.
First, for a given contiguous set of snoop activity, start ignoring a given peer during that contiguous chenk after it has been considered once.
Further, make get_hwaddr cheaper for attempts against
remote IPs.
To facilitate the above, create an efficient 'ip_is_local' to be
a relatively cheap function, with
potential to cache result in future
if it needs to be even cheaper.
When generating new key materials, most people say 'yes' and cause problems
where they cycle valid keys without
realizing the significance.
Replace prompting with an emphasized warning instead.
Permit user to opt into a rebase of a
profile, to pick up potential updates
from the confluent packaged stock
profiles for files the user has not yet
customized.
When importing an image and taking stock copy, mark the files to allow detection of stock
versus customized profile content.
This will be used by a rebase command to know when
to overwrite or when to leave a file alone.
-Have apiclient set timeout on getting credential to avoid hang
-Change dcuiweasel to start shell earlier for better debug
-Do not expire the ident token if deployment is armed continuous anyway
Some files that get gathered into genesis
have their license in a related rpm, but that related rpm does not otherwise get captured.
So pull all correlated license material
in the license capture process.
PXE and HTTP client define a behavior for using non-address portion of an offer if the
offer has none. However, other clients, like
udhcpc will be confounded by such offers.
Ensure that client specifically sets VCI to indicate
it understand such an offer before replying.
This as yet generally doesn't matter as UUID is not sent by OS installers, however
some OSes can't do do API calls over LLA, and
thus we want to start allowing OS requests with
UUID and only offer when it makes sense.
Newer xcc changes things yet again, but
we are comfortably in the firmware that can be bootstrapped
with redfish, so use that instead once we've cleared the
redfish incapable variants.
Newer xcc changes things yet again, but
we are comfortably in the firmware that can be bootstrapped
with redfish, so use that instead once we've cleared the
redfish incapable variants.
EL7's python3 support is barren and would require a lot of dependency
work.
Move EL7 back to python2 and will deprecate EL7 support when we ditch
python2
Going from python 2 to python 3, the dbm format
goes from the default to unsupported.
This allows a python3 confluentdbutil restore to handle
a python2 dump of unsupported format.
Cryptodome is not always packaged with the
explicit form, fall back to the Crypto names
hoping that the user wouldn't be trying to use
PyCrypto is this day and age.
OpenSSL does not allow access to custom start
date without the full 'ca' facility, do the
work to set up the full CA and then
backdate certificates.
This does open the way for managed CA if required.
Switch to sed rather than trying to use head. There may or
may not be a second match, so need sed logic to understand
whether it's a match or not to keep or throw out.
In python 3.9, the % is no longer included
in the first member of the address tuple.
Handle both scenarios by adding '%' in for versions
that seem to want to remove it.
Some applications may want much of the genesis
distribution, but want to have a bootstrap process
independent of confluent. Provide hooks either
through the identity image or the boot image itself.
Use the recvmsg hint to mask out the buffer rather
than zeroing the entire buffer. This is more efficient
and further improves efficiency of parsing of the packet.
Very large PXE requests can leave residual
information that small, non-pxe requests will
interact poorly with, leading to spurious pxe-client
with cloned uuid of most recent large request.
Clearing between IO normalizes the state to avoid the bleed
over.
Ubuntu 22.04 makes some changes, notably removing the
custom-installation hooks.
Change to injecting our modifications more directly to where the
custom-installation hooks used to be.
Remote discovery will require that the reporting agent
provides port forwarding facility for https access, to
allow access without route to the managing confluent server.
This will allow the info reported to indicate use of this
forwarding facility and the base handler adds ability to
generically get the certificate.
Currently, the 'n' for next result fails to find the next result
if it is currently on screen. Also, the feedback on
failed search could use improvement.
Use the websocket liveness as the keepalive, so
the reaper is not scheduled for such sockets.
Additionally, register the async thread as to be killed on logout.
At time of running pre, array is not assembled by default,
inject a scan to check for them.
VROC must not be specified by 'md' name, or yast thinks it
needs to own making it, and complains that the setup doesn't
specify members.
Technically there's room for a race condition where boot is attempted
before the profile is ready, but it's highly unlikely.
Conversely, there is a potential confusing race condition today where
restarting a deploymennt without armed api causes
it to be disarmed before the boot is attempted.
Older python will break by assuming that
: always means a format expression is coming.
Move the field value fetch to format_field, and ascertain if some of the
expression was shunted to format specification
by mistake.
This permits long haul node api key registration over a single port. It cannot validate that
the requester is privileged, but the auto-invalidation
offsets the risk of subsequent users having read access to the remote mount.
For routed deployment, we have to preshare some information.
Additionally, the API arm mechanism gets too open ended.
Add support for using a shared secret over another
channel to do HMAC of a key to authenticate peer,
which has an alternate api arming mechanism
that is hardened.
When trying to get a configuration
without a network context, it would fail.
Now, as intended, it generates network configuration without autosense in such a case.
A frequent scenario is to 'refresh' ssh configuration toward the
end of:
-changing trust nodes
-Adding a collective member
-Repairing a broken configuration
-As part of 'confluent-ifying' a node that wasn't confluent deployed
When processes may update the certificate authorities, the confluent
CA trust would be lost. Place it appropriately so that
update-ca-trust will keep it in the appropriate place.
When processes may update the certificate authorities, the confluent
CA trust would be lost. Place it appropriately so that
update-ca-trust will keep it in the appropriate place.
It has been stated that no one would want to do this
on purpose, and thus it should reject. The rationale
being that if no one wants to do this, but did it
anyway, they presumably made a mistake.
So now such attempts will be blocked.
Keep the dead logic around for now
in case a future opinion changes things back.
Make sure confluent has made /etc/confluent, and further always initialize the
encryption key, as it will almost certainly
be needed and easiest to just always
generate on first startup.
bandit erroneously flags 'input' based
on possible python2-ism. Avoid the
error by using 'getinput', making that
input or raw_input based on the python version.
The strategy of duping file descriptors
is inadequate. The copies share
identical offsets.
Fix this by reading the file once into
memory, and using BytesIO to fake a file.
This is relatively memory intensive in theory, but in practice
pyghmi library had been duping everything to memory
anyway, so it is a wash for now.
Depending on the options selected/not selected, the
/var/lib/confluent directory may have been initialized
incorrectly. Have all the potential paths begin with
ensuring /var/lib/confluent is correct, and then
use seteuid consistently to take care of the rest.
When two nodes are ambiguous, provide more
info (mac) and also a suggestion of
which of the two appears most
trunk-like if one of them looks
suspiciously like a trunk.
XCC will be retiring SLP and to prepare
for that, move over to SSDP.
This required that the snoop be wired
up to detection correctly, that slp
no longer report XCC, that
the model/serial number be moved to xcc handler, and various fixes to SSDP.
2022-01-05 11:43:04 -05:00
523 changed files with 41422 additions and 6641 deletions
All file of the Confluent-client software is distributed under the terms of an Apache-2.0 license as indicated below:
Files: *
Copyright: 2014-2019 Lenovo
License: Apache-2.0
File: client.py:
Copyright: 2014 IBM Corporation
2015-2019 Lenovo
License: Apache-2.0
File: sortutil.py,
tlvdata.py
Copyright: 2014 IBM Corporation
2015-2016 Lenovo
License: Apache-2.0
File: tlv.py
Copyright: 2014 IBM Corporation
License: Apache-2.0
License: Apache-2.0
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
You must give any other recipients of the Work or Derivative Works a copy of this License; and
You must cause any modified files to carry prominent notices stating that You changed the files; and
You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
ap.add_argument('-a', '--attrib', help='Pull ip addresses and hostnames from attribute database', action='store_true')
ap.add_argument('-i', '--ip', help='Expression to generate addresses (e.g. 172.16.1.{n1} or fd2b:246f:8a50::{n1:x})')
ap.add_argument('-n', '--name', help='Expression for name to add ({node}-compute, etc). If unspecified, "{node} {node}.{dns.domain}" will be used', action='append')
ap.add_argument('-f', '--fqdn-first', help='Put the FQDN first in the hosts entries', action='store_true')
ap.add_argument('-n', '--network', help='Initiate deployment over PXE/HTTP', action='store_true')
ap.add_argument('-p', '--prepareonly', help='Prepare only, skip any interaction with a BMC associated with this deployment action', action='store_true')
ap.add_argument('-m', '--maxnodes', help='Specifiy a maximum nodes to be deployed')
ap.add_argument('-r', '--redeploy', help='Redeploy nodes with the current or pending profile', action='store_true')
ap.add_argument('noderange', help='Set of nodes to deploy')
ap.add_argument('profile', nargs='?', help='Profile name to deploy')
args, extra = ap.parse_known_args(args)
@@ -78,36 +90,86 @@ def main(args):
if args.profile and not args.network:
sys.stderr.write('-n is a required argument currently to perform an install, optionally with -p\n')
return 1
if not args.profile and args.network:
if not args.profile and args.network and not args.redeploy:
sys.stderr.write('Both noderange and a profile name are required arguments to request a network deployment\n')
return 1
if args.clear and args.profile:
sys.stderr.write(
'The -c/--clear option should not be used with a profile, '
'it is a request to not deploy any profile, and will clear '
'whatever the current profile is without being specified\n')
sys.stderr.write('Requested noderange has nodes with locked deployment: ' + ','.join(lockednodes))
sys.stderr.write('\n')
sys.exit(1)
if args.profile:
profnames = []
for prof in c.read('/deployment/profiles/'):
profname = prof.get('item', {}).get('href', None)
if profname:
profname = profname.replace('/', '')
profnames.append(profname)
if profname == args.profile:
break
else:
sys.stderr.write('The specified profile "{}" is not an available profile\n'.format(args.profile))
if profnames:
sys.stderr.write('The following profiles are available:\n')
for profname in profnames:
sys.stderr.write(' ' + profname + '\n')
else:
sys.stderr.write('No deployment profiles available, try osdeploy import or imgutil capture\n')
sys.exit(1)
armonce(args.noderange, c)
setpending(args.noderange, args.profile, c)
setpending(args.noderange, args.profile, profilebynode, c)
else:
databynode = {}
for r in c.read('/noderange/{0}/attributes/current'.format(args.noderange)):
@@ -116,7 +178,7 @@ def main(args):
if node not in databynode:
databynode[node] = {}
for attr in dbn[node]:
if attr in ('deployment.pendingprofile', 'deployment.apiarmed', 'deployment.stagedprofile', 'deployment.profile'):
if attr in ('deployment.pendingprofile', 'deployment.apiarmed', 'deployment.stagedprofile', 'deployment.profile', 'deployment.state', 'deployment.state_detail'):
help='Use original nodename in print out even if substituted')
argparser.add_option('-s', '--substitutename',
help='Use a different name other than the nodename for ping, with {}, it is the entire name evaluated as an expression, otherwise it is used as a suffix')
# among other things, FD_SETSIZE limits. Besides, spawning too many
# processes can be unkind for the unaware on memory pressure and such...
(options, args) = argparser.parse_args()
if len(args) < 1:
argparser.print_help()
sys.exit(1)
client.check_globbing(args[0])
concurrentprocs = options.count
c = client.Command()
currprocs = 0
all = set([])
poller = select.epoll()
pipedesc = {}
pendingexecs = deque()
exitcode = 0
nodemap = {}
if options.substitutename:
subname = options.substitutename
if '{' not in subname:
subname = '{node}' + subname
for exp in c.create('/noderange/{0}/attributes/expression'.format(args[0]),
{'expression': subname}):
if 'error' in exp:
sys.stderr.write(exp['error'] + '\n')
exitcode |= exp.get('errorcode', 1)
ex = exp.get('databynode', ())
for node in ex:
nodemap[node] = ex[node]['value']
for rsp in c.read('/noderange/{0}/nodes/'.format(args[0])):
`confluent2hosts` can be used to help generate /etc/hosts entries for a
noderange. There are two general approaches.
It can be used ad-hoc, using -i and -n to specify the address and name portions respectively. This accepts the standard confluent expression syntax, allowing for things like 172.30.1.{n1} or {node}.{dns.domain} or {bmc}.
It can also read from the confluent db, using `-a`. In this mode, each net.<value>.<attribute> group is pulled together into hosts lines. ipv4_address and ipv6_address fields are associated with the corresponding hostname attributes. You can use `-f` to put the FQDN first.
## EXAMPLES
* Generate /etc/hosts entries ad-hoc using default name:
`# confluent2hosts -i 10.2.3.{n1} d9-d12`
* Generate /etc/hosts entries ad-hoc using alternative name:
**imgutil** is a utility for creating, managing, and deploying OS images for diskless boot and system cloning in a Confluent environment. It supports building images from scratch, capturing images from running systems, and packing/unpacking diskless profiles.
## COMMANDS
* `build`:
Build a new diskless image from scratch in the specified scratch directory.
* `exec`:
Start the specified scratch directory as a container and optionally run a command inside it.
* `unpack`:
Unpack a diskless image profile to a scratch directory for modification.
* `pack`:
Pack a scratch directory into a diskless profile that can be deployed.
* `capture`:
Capture an image for cloning from a running system.
## BUILD OPTIONS
* `-r`, `--addrepos` <repository>:
Repositories to add in addition to the main source. May be specified multiple times.
* `-p`, `--packagelist` <file>:
Filename of package list to replace default pkglist.
* `-a`, `--addpackagelist` <file>:
A list of additional packages to include. May be specified multiple times.
* `-s`, `--source` <directory>:
Directory to pull installation from, typically a subdirectory of `/var/lib/confluent/distributions`. By default, the repositories for the build system are used. For Ubuntu, this is not supported; the build system repositories are always used.
* `-y`, `--non-interactive`:
Avoid prompting for confirmation.
* `-v`, `--volume` <mount>:
Directory to make available in the build environment. `-v /` will cause it to be mounted in image as `/run/external/`. `-v /:/run/root` will override the target to be `/run/root`. Something like `/var/lib/repository:-` will cause it to mount to the identical path inside the image. May be specified multiple times.
* <scratchdir>:
Directory to build new diskless root in.
## EXEC OPTIONS
* `-v`, `--volume` <mount>:
Directory to make available in the build environment. `-v /` will cause it to be mounted in image as `/run/external/`. `-v /:/run/root` will override the target to be `/run/root`. May be specified multiple times.
* <scratchdir>:
Directory of an unpacked diskless root.
* <cmd>:
Optional command to run (defaults to a shell).
## UNPACK OPTIONS
* <profilename>:
The diskless OS profile to unpack.
* <scratchdir>:
Directory to extract diskless root to.
## PACK OPTIONS
* `-b`, `--baseprofile` <profile>:
Profile to copy extra info from. For example, to make a new version of an existing profile, reference the previous one as baseprofile.
* `-u`, `--unencrypted`:
Pack an unencrypted image rather than encrypting.
* <scratchdir>:
Directory containing diskless root.
* <profilename>:
The desired diskless OS profile name to pack the root into.
## CAPTURE OPTIONS
* <node>:
Node to capture image from.
* <profilename>:
Profile name for captured image.
## EXAMPLES
Build a diskless image from a distribution:
imgutil build -s alma-9.6-x86_64 /tmp/myimage
Execute a shell in an unpacked image:
imgutil exec /tmp/myimage
Execute a specific command in an image:
imgutil exec /tmp/myimage /bin/rpm -qa
Unpack an existing profile for modification:
imgutil unpack myprofile /tmp/myimage
Pack a modified image into a new profile:
imgutil pack /tmp/myimage myprofile-v2
Capture an image from a running node:
imgutil capture node01 production-image
## FILES
* `/var/lib/confluent/public/os/`:
Default location for OS profiles.
* `/var/lib/confluent/private/os/`:
Location for encrypted image keys and private data.
#installedargs: example # These arguments would be added to the installed system
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Jarrod Johnson
Jarrod Johnson
Markus Hilger
Hengli Kuang
Tinashe
Wera Grzeda
Sorin Toderica
Tinashe Kucherera
Markus Hilger
Adrian Reber
Simon Thompson
weragrzeda
Christian Goll
root
Tkucherera
erderial
Andrew Wray