Firmware for a supply or a fan matched no fragment and so was called
core, and nothing could ever answer for misc.
Only the collections that mean one thing are matched by url. A Storage
resource is the subsystem, so its firmware is the controller rather than
a drive, and a Processor is a cpu as readily as an accelerator, so that
one is decided by asking the processor what it is.
A member of a collective that cannot reach quorum stalls in the startup loop
until quorum returns, and there was nothing in that loop that noticed a
shutdown. That was survivable while SIGTERM raised SystemExit out of the
signal handler, since that escaped the loop from wherever it happened to be.
Having the event loop deliver the signal instead leaves the stop event set with
nobody reading it until quorum is reached, so stopping the service waits out
the systemd timeout and ends in a kill.
Check the event in the loop condition, and wait on it rather than sleeping
through it, so the answer comes in milliseconds rather than whenever quorum
returns. A service stopped at this point has served nothing yet, so it goes
straight to the same configuration flush the normal exit does.
pyghmi asks for this parameter with xraw_command and catches the completion
code for a bmc that does not have it, and folding aiohmi in renamed that call
to oldraw_command rather than raw_command, so the handler could no longer fire.
Answering the code out of the returned dictionary repaired the crash but kept
the call on the older contract, which is now the only one left in the tree.
Catch it again instead: raw_command puts the completion code on the exception
as ipmicode, and nothing here reads the payload of a reply that carries a code,
which is the one thing catching gives up.
No behaviour change, checked against the previous version over the same fake
session for a good reply, an empty one, 0x80 and 0xC9 with and without a stray
payload, four other completion codes, a timeout, a lost session and a reply
with no data at all: same return value, same exception type, text and ipmicode,
same bytes on the wire.
_read_device_sdr_lun negotiates the read size down when the bmc answers 0xCA,
but the size > 5 guard leaves a size of 5 alone, so a bmc that will not serve
5 bytes at once was asked the same question for ever. Give up once the
request cannot get any smaller, and once a header read would go under the 5
bytes the record length sits in, by falling through to the raise already
there.
The stale reservation retry could not end on its own either: it cleared the
id and left taking a new one to the top of the loop, which only reserves for
a partial read, so the very first request repeated unchanged. Take one where
the code is handled.
The labels are worked out across the whole inventory, since a platform may give
every entry the same Name, so an entry that came back empty would be asked for a
name it does not have and take the naming of the others with it.
A parameter file that is not json, or that holds something other than an
object, reached the update as a raw parser message or as a TypeError from the
handler that unpacked it.
A writable directory only says the user could have created a file there, and
/tmp lets anyone do that. If the target exists, it has to be writable by the
requesting user too, or confluent would overwrite it as root.
get_user_name documents that it answers None when reading a slot fails, but
raw_command raises before the check that would return it, so that branch has
never run and one refused slot aborted the whole user list. Answer None where
the docstring says to, and let get_users drop the blanket except it grew to
work around it.
Only a completion code counts as the bmc answering about the slot. A timeout
carries the fabricated 0xffff from the session layer and a lost session carries
no code at all, and swallowing either of those reports a list truncated at the
point of failure as a complete one.
Some will not take an image without being told which kind it is, and the
only way to find out was to attempt an update and read the error, which
writes to the bmc before it gets that far.
The sdr carries unit fields for every sensor record, and they were read into
the reading whether or not the sensor has a number for them to describe. A
discrete sensor reports which of its states are asserted and no value at all,
so a watchdog came back with units of "% ", and a discrete sensor on a full
record picks up a base unit the same way, reporting degrees celsius for a
sensor that never has a temperature. A caller that shows the unit alongside
whatever value it was handed then prints a unit with nothing to apply it to,
which is what the client was taught to skip in 8c8c32cc. The client is not
the only consumer of a reading, so answer the question in the library.
The record itself says which sensors those are, and the reading path already
works it out to decide whether to decode a number: a sensor has one only if
the numeric format says how to read it, or the format is unsigned and the
record either supports thresholds or is of reading type 1. Ask that once,
where the units are assembled, and give the sensors that fail it no unit, so
the unit and the value cannot come to different conclusions about whether the
sensor has a reading at all.
A threshold or numeric sensor reports exactly what it did before, including
the ones whose unit is a percentage, a combination of two units, or nothing
because the record names no unit. On the Lenovo XCC this was checked against
nothing changes: its 305 readable sensors decode identically, discrete ones
included, because they are all compact records naming no unit in the first
place. The sensors that change are the ones the finding came from, which
name a unit on a record that has no number to put it on.
A path a caller wants confluent to save something into was being run through
the check meant for a file confluent is asked to read. That check forks, drops
to the calling user and asks os.access for R_OK, which is false for every file
that does not exist yet, so nodesupport servicedata and save_licenses could
only be given a path that was already there. Handed a name to create, they
refused, and refused in a way no caller was looking for, so the command printed
nothing and exited zero.
The previous commit worked around it by skipping the check for a download
target, which fixed the symptom by removing the guard rather than by asking the
right question. Ask the right question instead: whether the user could have
created the file in that directory themselves. A path that is already a
directory is a destination directory, anything else names the file, which is
the same rule the code that goes on to write the file follows.
So a caller can still only make confluent write where they could have written,
and this now also catches an unwritable destination at the point the request is
made rather than several layers further in.
This bmc gives all three of its firmware entries the same Name, "Software
Inventory", and puts what they actually are in the description. The first
entry took that name, and the two after it fell back to their ids, so
nodefirmware answered with "Software Inventory", "cpld_active" and
"d1dc9e4b" for what are the host, cpld and bmc images.
Decide the labels across the collection rather than one entry at a time,
so a name the platform repeats can be recognised as no name at all. Where
that happens, use a description that does tell them apart, and the id when
even that is shared. A platform whose names are already distinct keeps
exactly the names it had.
The labels are what a caller addresses an entry by, so this also turns
inventory/firmware/all/d1dc9e4b into inventory/firmware/all/bmc_image.
Processor inventory carried a single field, the model, so a platform that
does not give one had a processor in the listing with nothing in it, and
the client, which skips empty values, showed no processor at all. This
bmc names the manufacturer, the socket and the core and thread counts, and
gives no model.
Carry those, along with the speed, serial and part number where a platform
offers them, and treat a processor as missing only when the bmc says its
state is absent, rather than whenever it does not describe a state.
The top bit of the major revision byte of Get Device ID says the device is
still initialising or taking a firmware update. It was read as part of
the revision, so immediately after a bmc reset nodefirmware reported "BMC
Version: 131.11" for what is 3.11, and settled down only once the bit
cleared.
Mask it as sdr.py already does for the same byte, so the two agree about
the same field.
Reading the identify state says plainly when a platform describes no
indicator, but writing it went ahead and patched IndicatorLED regardless.
This bmc has neither that property nor the boolean that replaced it, and
answered the write with an internal service error, which reached the user
as one and the log as a traceback.
Ask the same question the read asks. With neither property present there
is nothing to write, so say so in the same words instead of finding out
from the bmc.
This bmc advertises a Bios resource on its system and answers 404 for it.
Confluent followed the link and passed the bmc's complaint on as an
unexpected error, so a nodeconfig read printed every bmc setting and then
ended with "The requested resource of type named 'Bios' was not found",
and the system half of the configuration was a 500 saying the same.
There is already a good answer for a system that offers no bios settings,
and a link that is advertised and not served is the same thing as far as a
caller is concerned, so give it the same one. The result is checked once
and remembered, including the negative, so this costs one request on the
first ask and nothing after.
Redfish identifies an account by a string, and an implementation is free
to use the account name, which this one does. The handler converted the
last element of the path to an integer, so every per user read, update and
delete answered "invalid literal for int() with base 10: 'root'" as an
unexpected error, with a traceback to match. Confluent offered the id
itself, listing the account as "root", and then could not accept it back.
Take the element as given. Everything below already compares ids as
strings, and the input parsing already keeps a non numeric uid, so only
this conversion stood in the way. nodebmcpassword goes through exactly
this path, reading users/all for the id and then writing to that account,
so it could not work at all on such a bmc.
ipmi users really are numbered slots, so the conversion is right there and
stays, but say so when it fails rather than letting a ValueError surface
as an unexpected error.
A caller asking for fans or energy got nothing from any bmc that serves
the Sensors collection. Those sensors were filed under their redfish
reading type, Rotational for a fan, while the categories are named after
the ipmi sensor types the rest of the code uses, so nothing matched.
Power appeared to work only by coincidence, Power and Current happening to
be spelled the same in both vocabularies.
Translate the reading type as the sensor is mapped, so a sensor means the
same thing whether it came from the Sensors collection, from the older
Thermal and Power documents, or from ipmi. On the bmc this was found on,
fans go from nothing to the 24 tachometers, and temperature and power
already agreed with what the same hardware reports over ipmi.
The fan controls stay out, and cannot be brought in. Their reading type
is Percent, which is also what a battery state of health reports, and this
bmc fills in no PhysicalContext to tell them apart, so there is nothing to
classify them by that would not also drag in unrelated percentages.
The loop decoded the first three bytes of a name and never consumed them,
so any sensor or fru name a bmc encodes as 6 bit packed ascii spins at
full speed, appending the same four characters until the process runs out
of memory. Measured at about 10 MB a second, so a bmc using an encoding
the spec gives its own worked example of costs a pinned core and, before
long, the daemon.
Consume each group, and decode a trailing group of one or two bytes rather
than dropping it, since those carry a character each and the name would
otherwise come back short.
The arithmetic was already right, it was only never reached a second time.
Verified by encoding names per the packing and reading them back: exact for
every length except those leaving three characters in a three byte group,
where the byte count cannot say whether three or four were meant and a
trailing space is unavoidable.
A bmc may keep its sensor data records on the sensor device instead of in
a repository, and this one does, so it had no sensors, no health and only
a partial inventory over ipmi.
The records themselves are identical, version 0x51 and the same types, so
everything that decodes them is reused as is. Only the fetching differs:
a command of its own, a reservation of its own, and records held per lun
rather than in one place. The luns to ask, and a change indicator to
cache on, come from Get Device SDR Info.
The fetch is written out rather than shared with the repository one. The
loops are alike, but nothing available here has a repository to test
against, and the price of factoring them together is that a mistake would
land on every bmc that works today rather than only on those that do not
work at all.
Records are cached in memory and on disk exactly as repository records
are, keyed on the change indicator, and a device that offers no such
indicator is read afresh each time rather than cached wrongly.
Names are stripped of the nulls that pad a fixed width field. This bmc
pads every name out to sixteen bytes, and a name carrying them cannot be
matched by a caller asking for a sensor by name.
On the bmc this was written for: 163 sensors and 9 frus, against the 163
the device reports it has. The 54 temperatures and 36 fans it now reads
match what the same machine reports over redfish, to within the precision
each side gives.
The oem lookup answers whether it found a handler for the vendor, and that
answer was being stored as whether the lookup had been done at all. On
anything the map does not name, which is every bmc that is not a Lenovo,
the flag stayed false and each oem_init issued another Get Device ID and
built another handler.
Almost everything goes through oem_init, so this is a round trip added to
almost every operation. Where those calls are close together it is far
worse than that: reading the sensor data records asks for the event
constants once per record, so a run of 172 records fired 176 Get Device ID
commands back to back, which was enough to make the bmc stop answering and
the read fail with a timeout. The same sequence now takes 3 commands.
Settling for the generic handler is an answer. The device id cannot
change within a session, so asking again buys nothing, and the handler it
throws away each time is the one holding the sensor names it had cached.
A bmc that keeps its sensor data records on the sensor device rather than
in a repository was answered with a bare NotImplementedError. With no
text of its own it reached the user as "Unexpected Error:
NotImplementedError" and was logged with a traceback, for a capability the
library had simply never implemented rather than anything having gone
wrong.
Say so instead, in all three places that give up: the two branches for a
bmc without an sdr repository, and the version check that only understands
records of version 0x51, which now names the version it was given.
This makes the inventory usable on such a bmc as a side effect. The
system fru is gathered before the records are, and an unsupported
operation is tolerated where an unexpected error was not, so nodeinventory
answers with the board, chassis and product data instead of one line of
error.
The redfish event log was taken from every log service the manager
advertises, whatever those turned out to be. On a bmc that keeps its
systemd journal there, nodeeventlog answered with a thousand lines of
kernel probe failures and daemon chatter, and the log the user asked for
was never read at all, because this implementation keeps it under the
system. Clearing was worse: of the services it did find, the ones with a
clear action were the dumps, so a clear destroyed diagnostic data, left
the event log untouched, and reported success.
Judge a log service before reading it. A service whose id or name says
journal, dump, post code, host logger or crash is not an event log, and
both reading and clearing skip it, so a clear can no longer take out
something that was never asked for.
If that leaves the manager with no event log at all, look under the
system, where such an implementation keeps it. Only then: a bmc that has
one under the manager is served exactly as before, from the same requests,
so this cannot change what an implementation that already worked reports.
The list of services was also being extended in place, and it belongs to
whatever the url cache is holding, so an extra log added by an oem handler
accumulated on every call within the cache window.
Reading the alert destinations of a bmc that has none reported "Unknown
code 0x80 encountered", which is the fallback text for a completion code
the library has no name for. 0x80 on this parameter is not a failure, it
is the platform saying it does not have alert destinations, and the
redfish side of the same resource has said so in words for a while.
The lan parameter fetch already knew how to tell those apart, so build the
alert reads on it rather than on a raw command that raises on any non-zero
code, and raise UnsupportedFunctionality with something to read. Both the
count and an individual destination are covered, so a platform that offers
one and not the other says the same thing instead of failing differently.
Splitting the completion code handling out of the parameter fetch is what
makes that reuse possible; the interpretation of the payload, and every
answer it gives, is unchanged.
The oem hook for the destination count was passing its byte through ord(),
which raises TypeError on the bytearray it is given. No handler in tree
implements the hook, so it had never been called; hand it the integer.
A bmc that does not implement a lan configuration parameter says so in the
completion code and answers with no data at all. The helper reached
straight into the payload, so such a parameter raised IndexError, and with
it went the whole of nodeconfig over ipmi: the bmc group, the plain, the
detailed, the extended and the advanced reads all ended in "bytearray
index out of range". It also took the attribute enumeration with it, so
the client then rejected names it should have accepted.
The guard that was there caught an exception carrying the completion code,
but oldraw_command reports the code in its response rather than raising on
it, so nothing was ever caught. Read the code from the response instead:
parameter not supported and parameter out of range mean the platform does
not have it, and anything else is a real failure that should say what it
was rather than be mistaken for absence.
The address configuration method was looked up in a table with no regard
for whether it had been read at all, so a bmc that does not report it
would have traded the IndexError for a KeyError. Answer None when it is
absent, as the address above it already does, and name the value when it
is present but unfamiliar.
A console whose bmc had gone away reported "Unexpected error - None", and
the api answered 504 with an error of None. The redfish plugin took the
text for an unreachable target from the strerror of the socket error it
caught, guarded by a hasattr that is always true: every OSError has a
strerror attribute, and it is None on most of the ones a bmc going away
produces, TimeoutError and gaierror among them. Ask for the text the same
way as everywhere else instead, which also keeps the errno on the errors
that do carry one.
The same applies to an unreachable target raised with no message at all,
so use the same helper there, on both transports.
Underneath that, give the node error messages a default to fall back on
rather than carrying whatever they were handed. Each subclass already had
one, in an __init__ that an explicit None went straight past; making it a
class attribute the base class applies means it holds however the message
was built, and removes five copies of the same constructor.
Also repair an affluent handler that put a closing parenthesis in the
wrong place, passing its error text to Queue.put_nowait as a second
argument. Any OSError there other than "no route to host" raised
TypeError from inside the except clause instead of reporting anything.
The receive loop treated only WSMsgType.CLOSE as the end of a session, but
aiohttp reports a peer that has gone away as CLOSED, and it does so
immediately and for every subsequent call. Everything that was not CLOSE
fell to an else branch that printed a line and went round again, so a
console whose bmc restarted became a full speed loop writing one line per
iteration: measured at 2.7 million iterations a second, and observed
filling 15 GB of log in a quarter of an hour while the daemon stopped
answering requests.
Treat every message that is not data as the end of the session, clear the
connected flag and report the disconnect once. A session that ended any
other way than a clean close is recorded in the trace log, unbuffered so
that it survives a daemon that does not, rather than printed.
Both websocket console plugins carried the same loop. While here, give
the openbmc one the parts tsmsol already had: text frames are data rather
than a surprise, and the client session is closed when the upgrade fails
and when the console does, instead of being leaked.
nodefirmware <node> disks reported the bmc version, and so did adapters and
misc. The generic handler takes a category and ignores it, and the ipmi plugin
does not filter either, so every category answered with the one entry ipmi can
report.
Apply the rule R13 established for a redfish inventory that does not categorise
itself: the bmc's own firmware is system firmware, so it answers for core and
for nothing else. Filtering in the handler that produces the entry leaves an oem
handler that does categorise its own firmware free to answer for more.
There is no ipmi command for a bmc hostname, so get_hostname falls back to the
DCMI management controller identifier. A bmc that does not implement the DCMI
group at all rejects that with "invalid command", which was handed to the caller
as if the request had been bad: nodeconfig <node> bmc read seven fields
correctly and then reported "Error: Invalid command", and the api answered 500
Unexpected error.
Route every DCMI request through one helper that turns "invalid command" and
"command disabled or unavailable" into UnsupportedFunctionality, so the
identifier, the asset tag and the hostname all report a platform that cannot do
this rather than a fault. Where the caller asked about a hostname, say that
rather than naming DCMI.
The whole group read still ends in one error line, because an operation a
platform cannot perform and one that failed are the same message to the client.
That is worth separating, but not here.
The unhandled tails of handle_request, handle_configuration and handle_alerts
raised a bare Exception('Not implemented'), so asking for a resource the
transport has no code for was reported as an unexpected error and logged with a
traceback. management_controller/location over ipmi is one such resource: R5
implemented it for redfish only.
Raise UnsupportedFunctionality naming the resource instead, which the plugins
already treat as its own case rather than a fault, and give decode_alert over
redfish the same treatment. Any resource added to the tree without an
implementation on one transport now reports that plainly.
nodereseat printed "Error: " and nothing else against a bmc that refused the
credentials. The redfish plugin reports it properly, but the message it emits
re-raises TargetEndpointBadCredentials with no arguments when a single node is
addressed, and the enclosure plugin renders that with str(e), which is empty.
Both hardwaremanagement plugins already had a helper for exactly this, one copy
each. Keep one in confluent.exceptions instead, teach it to fall back to the
description a confluent exception carries by class before falling back to the
exception name, and use it in the enclosure plugin too.
get_error_body had the mirror image of the same bug, joining the class
description and the message unconditionally and so answering "Bad Credentials -"
with a separator and nothing after it. The apierrorstr property beside it
already gets this right, so use it.
The exit callback opened the pid file unguarded, so when it was already gone the
atexit handler raised FileNotFoundError and python reported an exception ignored
in an atexit callback. The removal of the debug socket immediately above is
guarded, so this was an oversight rather than an intent. Verified by stopping
the service with the pid file deleted first.
terminate() called sys.exit(0) from a signal handler while the asyncio loop was
running. The SystemExit escaped run_forever, and closing the loop afterwards
then failed with "Cannot close a running event loop", so every clean stop wrote
a cascade of tracebacks and left a pending task behind.
Ask the loop to stop instead: deliver the signals through add_signal_handler,
which is the signal safe route, and set an event the main coroutine waits on so
that it returns and asyncio can unwind itself. Flush configuration on the way
out, as the client requested shutdown has always done.
That client requested shutdown went the same way, calling sys.exit from inside a
request coroutine, and it is the route the systemd unit uses to stop the
service. It now asks for the same orderly stop through a hook the running
service registers, keeping the old behaviour when nothing is registered.
Measured on all three routes, with redfish and ipmi sessions to a bmc in flight:
under a third of a second and no tracebacks, where before each one wrote a
cascade.
The web session helper sent X-CSRF-Token. MegaRAC checks X-CSRFTOKEN, so the
login succeeded and then every request answered Invalid Authentication, which is
why this helper has never worked. Confirmed both ways against a bmc: the same
request answers 401 with the old spelling and 200 with the new one.
IndicatorLED is deprecated in redfish in favour of the boolean
LocationIndicatorActive, so a platform that only implements the newer property
reported no identify state and could not be told to light up. Read either one,
preferring the older where both appear, and write the boolean when that is what
the resource offers. The boolean has no way to express blinking, so a request to
blink lights it steadily.
The led resource shares the same reader, so it gains this as well.
Asking for the mac addresses of a node whose inventory does not describe any,
which is every node reached over ipmi, printed absolutely nothing and exited
successfully, leaving no way to tell an empty answer from a broken command. Name
what was asked for instead. The exit code stays successful, since an inventory
that does not mention something is a valid answer rather than a failure.
get_ntp_enabled returns None to mean the platform cannot tell us, and that went
to the caller as the literal text "None", which says nothing at all. Report it as
unsupported instead, which the tooling already renders plainly.
A discrete sensor reports no value, and the unit was appended regardless, so a
watchdog came out as "Watchdog:% " and an event log sensor as "SEL:". The unit
belongs to a reading, so only print it when there is one. On the platform this
was seen on the units field is itself meaningless for such a sensor, carrying a
percent sign and a trailing space from the sdr.
Markus Hilger