Building a card health check¶
Create the shell script ha-healthcheck.sh, which determines whether the LPU cards in the host are usable, and connect it to a systemd timer and a monitoring agent so that it runs periodically. The script reads only the files that the driver exposes in sysfs, so it needs neither root privileges nor a device handle, and it neither resets cards nor sends jobs to them. It prints the result as one line per card, and the caller judges by the exit code.
Items to check¶
| Item | File read | Normal | When not normal |
|---|---|---|---|
| Driver loaded | /sys/module/ha/version |
Driver version (for example, 2.6.0) |
If the file is missing, the driver is not loaded |
| Number of cards | /sys/class/ha/haN |
As many as the number of cards installed | There is a card that the driver has not probed |
| Card state | /sys/class/ha/haN/health |
ok |
See the table below |
| Firmware ready | /sys/class/ha/haN/fw_version |
Version in major.minor.patch format |
If it is unknown, the firmware is not ready or the driver could not read the version |
health is a one-word verdict that the driver issues. The script assigns a verdict for each value as follows.
health |
Meaning | Verdict |
|---|---|---|
ok |
There is no FATAL and no failed reset, and all error counters are 0 | OK |
degraded |
A timeout or firmware error has occurred at least once since the driver probed the card. At least one of devcmd_timeouts, devcmd_fw_errors, and hdma_timeouts is not 0 |
WARN (FAIL if -s is given) |
faulted |
The card is FATAL or the last reset (an SDK reset or the automatic reset) has failed | FAIL |
unknown |
The driver cannot read the state. It does not appear once the probe of the card has finished | FAIL |
The reasons for setting the verdict criteria this way are as follows.
degradedis not treated as a failure by default. Error counters do not decrease, sodegradedremains until the driver probes the card again (hot reset, module reload, reboot), and an SDK reset does not clear it. A case in which the firmware rejects an invalid request sent by an application is also counted indevcmd_fw_errors. So ifdegradedwere treated as a failure, the health check would keep failing from then on even if an error had occurred only once. Judge whether the problem is continuing by whether the counters are increasing (Alert criteria). When you check right after installation or right after a hot reset, when the counters start at 0, pass-sso thatdegradedis also treated as a failure.fw_versionis checked as well.healthdoes not take firmware readiness into account in its verdict, so a card whose firmware is not ready can also beok.- The number of cards is checked separately. A card that the driver has not probed does not appear under
/sys/class/ha/, so looking only at thehealthof the cards that are present cannot reveal a missing card.
What the health check does not see
health does not immediately show a stall in progress. Even if a card stalls, health does not change until the response wait ceiling passes and a timeout is recorded. A wait that an application requested to be without a ceiling is not recorded as a timeout, so health stays unchanged for as long as that wait is stalled (you can tell from Devcmd waiting in the kernel log). Temperature, power, and the telemetry polling state are also not part of the verdict. Monitor these separately by following Hang detection and the watchdog and Monitoring integration.
Why only sysfs is read¶
- No device handle or root privileges are needed. Because the script does not open
/dev/haN, it does not compete with workloads for the card and can run while a workload is using the card. The telemetry nodes and/sys/module/ha/versioncan be read by all users. - Values can be read even while the card is stalled.
healthand the error counters are computed from state that the driver keeps in host memory, so they respond even while compute work is stalled.fw_versionis a value the driver has already read, and only while the value isunknowndoes a read check again whether the firmware is ready (Collection interval and cost). - It neither resets the card nor sends jobs. It only reads files. On a card whose
fw_versionisunknown, the driver checks again on read whether the firmware is ready and, if it is, fills in the value. This is the same operation that the driver's poller performs periodically.
Script¶
Save the following as ha-healthcheck.sh. It uses only POSIX sh syntax, so it also runs under a /bin/sh that is not bash (for example, dash).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | |
| Option, environment variable | Meaning |
|---|---|
-n N |
The number of cards that must be present. The range is 1–8 and the default is 1. If fewer than N cards are found, it ends with exit code 2 |
-s |
Also gives a card that is degraded the verdict FAIL |
SYSFS |
The sysfs path. The default is /sys; specify it in an environment where the host's /sys is mounted at a different path (for example, a container) |
For -n, write the number of cards installed in the host as a fixed value. You can check the number of cards visible on the PCI bus with lspci -D -d 20fb: | wc -l, but do not compute this value and use it on every run. If a card disappears from the PCI bus, that number decreases with it and the missing card is overlooked. The driver probes up to 8 cards on one host.
| Exit code | Meaning |
|---|---|
0 |
The driver is loaded, there are N or more cards, and no card is FAIL. There may be cards that are WARN |
1 |
There is a card that is FAIL |
2 |
The driver is not loaded or there are fewer than N cards. Takes precedence over 1 |
3 |
An option is invalid |
Running it and checking the result¶
Install the script and run it as a regular user. This is an example for a host with 2 cards.
1 2 | |
1 2 3 4 | |
If the last line is host OK and the exit code is 0, everything is normal. Read the output as follows.
- A card line consists of the card name, the verdict (
OK,WARN,FAIL), and the PCI address (bdf) and the values read, inname=valueformat. The error counters are not used for the verdict; they are included for the record. If you keep them from every run, you can later find when a counter increased. - The
hostline shows the overall verdict, the driver version, the number of cards found, and the number of cards expected. The overall verdict isFAILif the exit code is not 0,WARNif it is 0 and there is a card that isWARN, andOKotherwise. If the driver is not loaded, it showsdriver=missing. - A file that cannot be read is shown as
missing, and a card whosehealthorfw_versionismissingisFAIL. This is the case when the driver is in the middle of probing or removing the card, or failed to create the telemetry nodes (telemetry sysfs unavailablein the kernel log). - The card number
haNis not the identity of a card (Card numbers). Usebdfwhen referring to a card in alerts and records. bdfis the same asBus-Idof hyperaccel-smi. Ifhealthisok,degraded, orfaulted, the same value appears in uppercase in theHealthcolumn.
Test the failure path without touching a card. If you give SYSFS a nonexistent path, the result is the same as when the driver is not loaded.
1 | |
1 2 | |
If a line that is not normal appears, see the following.
| Output | Exit code | Where to look |
|---|---|---|
haN WARN ... health=degraded |
0 |
health is degraded |
haN FAIL ... health=faulted |
1 |
health is faulted |
haN FAIL ... fw_version=unknown |
1 |
fw_version is unknown |
haN FAIL ... health=missing |
1 |
Run it again after a while. If it continues, see Telemetry is unknown |
host FAIL ... cards=1 expected=2 |
2 |
/dev/haN is missing |
host FAIL driver=missing |
2 |
Module is not loaded |
Before you reset a card that is FAIL, first save the diagnostic information with Collecting diagnostic information for a problem, and then follow Recovering an unresponsive card. A reset erases the state that shows the cause.
While the module is being reloaded or ha-hotreset is making the driver probe a card again, the card directory can briefly disappear or its nodes may not have been created yet, so exit code 1 or 2 can occur. Stop the timer below or hold alerts while you do this kind of work.
Running periodically with a systemd timer¶
Create the service unit /etc/systemd/system/ha-healthcheck.service. For -n, write the number of cards in this host.
1 2 3 4 5 6 7 8 | |
Create the timer unit /etc/systemd/system/ha-healthcheck.timer. It runs every 5 minutes, starting 5 minutes after boot.
1 2 3 4 5 6 7 8 9 | |
1 2 | |
The script's output is recorded in the journal on every run, and if the exit code is not 0, systemd records the service as failed (failed). View the result of the last run with systemctl status ha-healthcheck.service.
1 2 | |
To send a failure as a notification, specify a unit that sends notifications with OnFailure= in the [Unit] section of the service unit. During maintenance, stop the timer with sudo systemctl stop ha-healthcheck.timer, and restart it with sudo systemctl start ha-healthcheck.timer when you are done.
Connecting to a monitoring agent¶
If you have a monitoring agent that runs a command and judges the state by its exit code, register the same command as a check.
- Treat only exit code
0as normal.1is a card state problem,2is a problem with driver loading or card probing, and3is a problem with the check configuration. - In an agent that interprets exit codes by the Nagios plugin convention (0 normal, 1 warning, 2 critical, 3 unknown), a card that is
FAIL(1) is shown only as a warning. Set the agent's rules so that1is also treated as critical. - If you include the
WARNandFAILlines of the output in the alert body, which card failed and why is delivered along with the alert. - If the agent runs in a container and the host's
/sysis mounted at a different path (for example,/host/sys), pass that path withSYSFS=/host/sys.
When you only need to check that cards are visible, the exit code of hyperaccel-smi is also enough. It ends with 2 if no card is visible and 0 if there is at least one. It does not look at card state, so it returns 0 even if a card is FAULTED, and if the environment variable HYPERACCEL_VISIBLE_DEVICES is set, it counts only the cards that its value allows (all cards for all, 0 cards for none or an empty value). ha-healthcheck.sh sees all cards on the host regardless of this variable.
1 | |
See also¶
- Telemetry —
healthverdict, meaning ofunknown, alert criteria - sysfs nodes — Definitions of the nodes the script reads
- Troubleshooting — When
WARNorFAILappears - FATAL and recovery
- Hang detection and the watchdog — Stalls in progress that the health check does not see
- hyperaccel-smi
- Collecting telemetry with Prometheus
- Collecting diagnostic information for a problem — What to do before a reset when
FAILappears - Recovering an unresponsive card