sockapi sent its collective refusal without awaiting tlvdata.send. redfish
handle_sensors returned a coroutine from most branches and None from the short
ones, so the caller's await raised TypeError; it is a coroutine throughout
now. console send_payload waited for a response without awaiting the wait.
Two suppressions are pyrefly limitations rather than bugs: Session defines an
async __new__, and the keepalive registry holds coroutine functions in an
untyped dict.
Taken from fix/asyncio-port-critical, limited to what pyrefly reports.
The SMM handler still used the httplib style connect/request/getresponse that
the async webclient does not have, so nothing was ever sent. It goes through
grab_response_with_status now, with allow_redirects=False to preserve httplib
behaviour, hence the new webclient parameter. That also fixes a login passing
its headers as urlencode's second positional argument.
Delta PDU's logout was a plain function both callers awaited, so the power
paths raised TypeError. XCC's set_system_configuration was the last
synchronous implementation of a method every caller awaits.
These __main__ blocks called coroutines as if they were functions, so they did
nothing at all. Single calls go through asyncio.run; sshutil, proxmox and
vcenter needed an _selftest coroutine. Two were invisible to pyrefly because
repr() and list() count as using the result: vcenter's get_vm_serial needs an
await, and proxmox's get_vm_inventory is an async generator.
lldp called _extract_neighbor_data twice, once correctly, so the bare call is
dropped. xcc3.remote_nodecfg is not a self test: every other handler defines
it as a coroutine and selfservice.py awaits it.
lxml refuses a str carrying an encoding declaration, and the SMM declares one
when it answers /data/login, so _webconfigcreds has raised ValueError on the
first thing it does after logging in ever since the switch to lxml. stdlib
ElementTree took the same input, which is why it went unnoticed. fromstring
already means to take either shape, so encode there. Confirmed against a
DW612S.
Hand written rather than autofixed, since three of the four need the
surrounding code read to be sure they are equivalent:
- confetty: `powerstate == None` -> `is None`.
- nodeconfig: `setmode != True` / `!= False` -> `not setmode` / `setmode`.
Safe because setmode only ever holds None, True or False, and the two
lines above each test normalise None away first.
- pam: split two `if cond: stmt` one-liners.
- imgutil: `from shutil import copytree as copytree`, an alias that renames
nothing. Not a re-export marker, this is a script.
Entirely mechanical, produced by `ruff check --fix --select F401,F541,E713`
and reviewed rather than taken on faith: deleting an import is only safe if
nothing imports it for its side effects or re-exports it. None of the 19
removed names is referenced anywhere in its file, none appears in any string
literal, and none of the touched files uses eval, exec, globals() or
__import__, so there is no dynamic lookup that could reach them.
The guard evaluated the `next` builtin and discarded it, which does
nothing, so a pending node with no handler fell through to
None.NodeHandler(...). The AttributeError was caught by the enclosing
except and logged as "Unexpected error during discovery", turning a node
that should have been quietly skipped into a spurious error in the log.
load_plugins() used `plugin` as the loop variable for plugin file names,
which shadows `import confluent.plugin as plugin` for the whole function.
Nothing in the function needed the module, so this was latent rather than
broken, but the next line that does need it would have failed oddly.
The DedicatedSpareDrives payload was built as a set containing a list
containing a dict comprehension with a constant key, so it collapsed to a
single entry and then raised TypeError on the unhashable list. Build a
list of drive references, the same shape as the Drives list just above it.
Three names were defined twice in the same scope, so the first definition
was unreachable:
- lenovo OEM handler: two set_user_access methods, the second silently
replacing the first. That made the SMM privilege update dead code. The
conditions are mutually exclusive (is_fpc returns None once has_xcc is
true), so merge both into the surviving method.
- redfish plugin handle_cert_authorities and prepfish
disable_host_interface: byte identical copies, drop the redundant one.
Each of these loops rebinds the name that holds the iterable. They work
today because the iterable is evaluated once before the loop starts, but
the name is then gone, so any later use reads a loop item instead of the
collection.
- nodeinventory: `for arg in args` / `for arg in arg.split(',')`.
- confignet (common and debian copies): iname holds the comma separated
interface list and is then reused for each interface in it.
- xcc _get_agentless_firmware: adata holds the adapter query response and
is then reused for each adapter.
No behaviour change, just distinct names for distinct things.
Every one of these raises NameError if its code path is reached:
- nodeapply: run_automation accumulated into an exitcode that only existed
in run(), so any automation error crashed instead of being reported. It
now keeps and returns its own, tracked separately from the exit code of
the ssh commands: the early exit after the spawn loop tests that one,
and folding automation failures into it would exit with children already
running and their pipes abandoned. Both are reported at the real exits.
- nodeconsole: redraw() reads firstnodename, which was local to
do_screenshot(); promote it to a module global like the other drawing
state.
- nodedeploy: the redeploy path appended to a lockednodes list that did not
exist yet. The block that follows re-reads the same lock state and acts
on it, so drop the dead duplicate.
- samples/nodeattrib_from_switch.py, misc/filterpasswd: missing import sys.
- xcc3: fixuuid was never imported. xcc imports xcc3, so take a local copy
the way the smm handler does instead of creating an import cycle.
- httpapi: the async session call still passed the WSGI-era env and an
extra argument to handle_async(), which has taken only querydict since
the aiohttp port. Calling it correctly exposed that handle_async()
registers an AsyncSession before raising on the discontinued long poll
path, so every request to it would leak a session that is never reaped.
It now only creates one when there is a websocket handler to yield it to.
- messages: the InputFirmwareUpdate.filename property checked
self.filebynode[node] with no node in scope. __init__ already validates
every expanded path and nodefile() rechecks per node, so drop the checks.
- pam: drop the python2 branches referencing unicode and raw_input. The
server has been python3 only since the asyncio port.
- cooltera: the sensor-name listing referenced a nonexistent sensors dict.
The available sensors depend on the model, which is only known after
reading the device, so list them from the same status data the readings
use.
- deltapdu, eatonpdu, geist: the not-implemented response in update() used
node outside the loop, unlike retrieve() in the same files and unlike
raritan/enlogic.
- confluentdbgcli: stray self. on a module-level socket connect.
This allows a user to opt into pcrs if they understand what they are doing.
Some PCRs are sensitive to firmware updates and some are sensitive to boot loader, kernel, boot config, or initramfs. All of these are an opportunity for an unsuspecting update to remove access to the boot volume. There are update processes that can be put into place to make this work,
but it is up to the OS update process to address that, and
OS update processes are likely not to address that at this time.
reply_dhcp4 logs the insecure mode remediation hint on every DHCP
discover it refuses. A node in this state never receives a reply, so it
retries for as long as it is powered on and the same message repeats
every few seconds.
Rate limit it per hardware address the way the neighbouring boot attempt
messages already do, reusing the ignoremacs window that check_reply uses
for the missing profile hint.
The per-MAC 90 second log throttle in proxydhcp has been inert: the
`skiplogging = True` reset sat in relay_proxydhcp, where it is a dead
local, while the loop in proxydhcp only ever assigns False. Once the
first packet is handled the flag stays False for the life of the
process, so every retransmitted boot request logs again even though
ignoredisco is updated to suppress it.
Reset the flag at the top of each loop iteration instead, next to the
timestamp check it belongs to, and drop the dead assignment.
reply_dhcp4 declines to answer a PXE boot request unless
deployment.useinsecureprotocols is set to firmware or always, but
proxydhcp had no such check. A node left at the default of never was
therefore still offered a TFTP bootfile and a plain http boot.ipxe URL
whenever the request arrived on port 4011 rather than port 67, so the
attribute silently did nothing in ProxyDHCP deployments alongside an
independent DHCP server.
Apply the same gate, including the UEFI HTTP boot exemption, and log the
same remediation hint. The node attributes are now fetched once and
passed through to get_deployment_profile instead of being looked up
again there.
Requests whose architecture could not be determined are ignored rather
than falling through to the reply. opts_to_dict stops parsing before the
client architecture option whenever the message type is not a request,
and such a packet would otherwise reach the iPXE branch and be handed a
plain http boot.ipxe URL without ever passing the gate.
The rsync push carried no preservation flags, so files arrived with their
special permission bits explicitly disabled and a setuid/setgid entry could
only be honored by the permissions= chmod on the client side.
Preservation was turned on once before in e52a9ff70f ("Have syncfiles
attempt to preserve more") and rolled back the same day in c0287e93ed
("Roll back rsync ownership"), because rsync also applied the staging copy's
attributes to the parent directories it merely traversed on the way to the
synced files, clobbering the permissions of system directories such as /etc.
Naming every staged file explicitly through --files-from and adding
--no-implied-dirs confines preservation to the content actually being
synchronized, leaving traversed directories alone and creating missing ones
with default attributes.
Two details follow from the way the staging tree is built. Files are staged as
symlinks, so rsync reads their attributes through to the real file, but
directories are staged as directories and need the source attributes copied
onto them for the otherwise empty ones that have to be named explicitly.
Ownership is mapped from the account the daemon runs as to root, since that
account generally does not exist on the node and would otherwise arrive as a
meaningless numeric id.
--xattrs from that earlier attempt is deliberately left out: with --copy-links
rsync reads xattrs off the symlink rather than its referent, so it transfers
nothing here while adding a failure mode on hosts without xattr support.
get_syncresult() caught the sync task's exception, logged a repr server
side and returned 200 OK with a null body. The node then called
.get('options') on that null resulted in:
c1: 'NoneType' object has no attribute 'get'
and syncfileclient still exited 0 as if syncing had succeeded.
Return the error to the requestor as a 500 with an error payload. On
the node, unwrap the body that grab_url_with_status raises for a
non-success status, print it once and exit non-zero. Only a failure the
server deliberately reported for this sync is terminal. Anything else,
such as a dropped connection, is re-raised so the existing retry loop
handles it as before. The same case now reports
c1: Error performing syncfiles: Syncing failed due to unreadable files: /etc/dangling.conf
c1: 'syncfileclient' exited with code 1
Release tags do not live on master: 3.15.2 through 3.15.6 were tagged on branch
3.15, so git describe reaches only 3.15.1 and dev builds were stamped
3.15.2.dev<n>. Besides being confusing, rpm and dpkg both rank the released
3.15.6 above that, so a dev package will not install over a released one.
Add a top-level VERSION file naming the release the branch is working toward
(4.0.0 on master) and a mkversion helper that stamps packages from it, keeping
the tag-derived value as a floor so a forgotten bump cannot go backwards.
mkversion also replaces the block copy-pasted into seven build scripts, and
makesetup no longer writes a per-package VERSION file, so the stale checked-in
confluent_common/VERSION goes with it.
osimport polls import progress from a coroutine, so a blocking sleep
between reads stalls the whole client loop. It was the only use of time
in the script, so the import goes with it.
Both loops that read the importer's output test for a percentage first,
so an ERROR: line whose text carries a % takes the percentage branch and
float() raises instead of the error being reported. The import target
name can carry one too, and that one is user supplied. importmedia runs
as a bare task, so the exception is swallowed and the client polls a
phase that never advances.
Test for ERROR: first and treat an unparsable percentage as no
percentage. Set percent on the error path of the second loop as well,
as the first already does.
The loop that drains the importer's remaining output reads a byte at a
time but clears currline on every iteration, one level out from where
the earlier loop clears it. currline is therefore never longer than a
single byte, so the percentage and ERROR: branches can never match and
the tail of an import is silently discarded.
Clear it only once a line has been consumed, as the earlier loop does.
scan_iso walks an entire ISO with blocking libarchive reads, yielding
only once per entry, and the header-sum branch of fingerprint reads the
whole file with no yield at all. Both run in the daemon, reached from
MediaImporter.init on every fingerprint and importing request.
The scan costs about 8us per entry and is indifferent to media size,
since libarchive seeks past file data rather than reading it: measured
at 80ms for 10k entries whether the image is 0.2 GB or 8.8 GB, and at
310ms for 40k. The header-sum branch is the one that scales with size,
reading a multi-gigabyte image end to end.
Make the pair plain functions and hand them to a thread instead.
check_output unwraps a single list argument, check_call never did, so
callers passing a list hit a TypeError out of create_subprocess_exec.
Two callers do: the genisoimage run behind Windows profile imports,
where an except Exception swallows the failure and the boot.iso is
silently missing, and the nodeconfig run in discovery, which takes out
automatic node configuration on discovery outright.
The asyncio port added an await between every 2048 byte read, which
roughly doubled the cost of hashing. imgutil runs entire packed images
through this, and the server pays it on rebase and media import.
Read a megabyte per iteration instead. That still yields hundreds of
times per gigabyte, so the event loop stays responsive, and sha512 is
independent of the read size, so existing manifests remain valid.
grab_response_with_status hands back bytes, so every failure path put a
b'<status>error</status>' repr in front of the operator rather than what
the SMM said. Decode at the raise, replacing rather than failing on a
body that is not valid utf8. The bodies still reach fromstring() as
bytes, which is what lxml wants when the xml carries an encoding
declaration.
The poll loop spends its retry budget on a poll that goes unanswered but
aborted the update on the first non-200, even though an SMM restarting
its web service part way through the apply keeps answering, with
whatever its httpd has to say, before it stops answering at all. Give a
bad status the same budget as a dead connection.
Every other /data call checks the status, this one discarded the
response, so an SMM that refused the user record was reported to the
caller as a successful privilege change.
Staleness is judged by age alone, so a session the SMM ended on its own
reached the operator as a raw error body instead of being retried.
Route the /data calls through a helper that logs back in and retries
once on a 401, which is how the SMM answers once a session is gone.
A hostname or domain write does not end the session, measured on a
DW612S at firmware 1.18, so this covers what the chassis drops by
itself, not a self-inflicted loss.
The counter is there to ride out a few unanswered progress polls, but
nothing ever cleared it, so three failures spread across a long apply
exhausted it and aborted an update that was still making progress.
A firmware update posts on one session for the minutes its apply loop
runs, and an FFDC collection downloads on the session it acquired, but
wc() judges a session by its age alone, so a settings call arriving
thirty seconds in logged that session out from underneath them. Flag
the long operations the way the IMM and XCC handlers already do and
leave their session in place.
Every getter and setter logged out on the way out, which nulled the
cached client and made the session cache inert on exactly the paths it
was meant to serve: a single nodeconfig walk of ntp costs two full
logins for the read and one per server for the write, each of them a
fresh TLS handshake plus, on firmware that omits st2, two extra page
fetches to scrape the tokens.
Leave the session in place and let wc() dispose of it once it expires.
This also stops one coroutine's logout from invalidating the session
another coroutine just fetched and is about to post with.
The retry counter is there to ride out a few unanswered polls, but
exhausting it broke out of the loop with complete still unset and fell
through to the 'complete' return, so an SMM that stopped answering
part way through an apply was reported to the operator as updated.
Raise instead; a genuine finish still leaves the loop on the progress
reaching 100.
Now that the expiry comparison actually caches a client, the session it
holds is shared, so tearing it down and replacing it needs the same care
the IMM handler already takes:
Dispose of an expired session with a logout instead of dropping the
reference, otherwise every refresh leaves an authenticated session
behind on an SMM that only has a handful of slots. That logout has to
tolerate a session the SMM has already reaped, hence the except.
Guard the login itself, so two coroutines arriving at an empty or
expired cache do not both log in and orphan one of the two sessions.
Stamp the vintage once the login round trips are done rather than
before, so a slow SMM cannot hand back a client that is already expired.
The old WebConnection.request() added a
'Content-Type: application/x-www-form-urlencoded' header to any POST
carrying a body, but grab_response_with_status() only sets a content
type for dict payloads, so the SMM login and every /data form POST now
go out as text/plain. This is not a fix for an observed failure: an SMM
running FPC variant 38 was measured accepting a text/plain login exactly
as readily as a urlencoded one. It restores the header the synchronous
code always sent and that the TSM and IMM handlers still set explicitly,
rather than relying on every SMM firmware level being equally lax about
what it will parse.
Also stop hard failing on responses the synchronous code discarded on
purpose. 'set=securityrollback:1' is only understood by newer SMM2
firmware. And /data/logout answers 401 once the session is gone, as
measured on that same SMM, so raising on a non-200 there turns a
completed hostname, domain or NTP operation into a spurious error.
The SMM hostname, domain, and NTP helpers looked synchronous even though their web transport is asynchronous. Removing awaits in the Lenovo OEM handler therefore returned unresolved coroutine work instead of completed settings results.
Convert the SMM settings and logout helpers to the asynchronous web interface, validate HTTP status responses, and await each operation from the OEM handler so callers only observe completed results.
The NextScale SMM path still used the removed http.client-style interface against the asynchronous WebConnection implementation. Login, configuration, diagnostic, and firmware operations consequently called unavailable methods or left request coroutines unresolved.
Make web-client creation asynchronous, migrate the affected requests to grab_response_with_status(), and await the cached client accessor. Correct the cache expiry comparison so fresh authenticated clients are reused and stale clients are renewed.
A node imported by a merge was assigned a free index and then had it
overwritten by the index carried in the backup, which may already belong
to a node in the target database. Keep the allocated index instead; a
full restore still honors the dumped index.
The -x/--exclude option drops matching node and node group attributes
from a dump, restore, or merge, so a backup can leave out dynamic state
such as deployment.state_last_updated or data that should not travel with it.
Patterns use shell-style wildcards, and a bare namespace such as net
excludes every attribute below it. The node "groups" and "id.index"
attributes and the node group "noderange" attribute are always retained
so that a restore can still reconstruct group membership and node index
assignments.
Markus Hilger
Jarrod Johnson
Jarrod Johnson