Skip to content

Building a card health check

Create the shell script ha-healthcheck.sh, which determines whether the LPU cards in the host are usable, and connect it to a systemd timer and a monitoring agent so that it runs periodically. The script reads only the files that the driver exposes in sysfs, so it needs neither root privileges nor a device handle, and it neither resets cards nor sends jobs to them. It prints the result as one line per card, and the caller judges by the exit code.


Items to check

Item File read Normal When not normal
Driver loaded /sys/module/ha/version Driver version (for example, 2.6.0) If the file is missing, the driver is not loaded
Number of cards /sys/class/ha/haN As many as the number of cards installed There is a card that the driver has not probed
Card state /sys/class/ha/haN/health ok See the table below
Firmware ready /sys/class/ha/haN/fw_version Version in major.minor.patch format If it is unknown, the firmware is not ready or the driver could not read the version

health is a one-word verdict that the driver issues. The script assigns a verdict for each value as follows.

health Meaning Verdict
ok There is no FATAL and no failed reset, and all error counters are 0 OK
degraded A timeout or firmware error has occurred at least once since the driver probed the card. At least one of devcmd_timeouts, devcmd_fw_errors, and hdma_timeouts is not 0 WARN (FAIL if -s is given)
faulted The card is FATAL or the last reset (an SDK reset or the automatic reset) has failed FAIL
unknown The driver cannot read the state. It does not appear once the probe of the card has finished FAIL

The reasons for setting the verdict criteria this way are as follows.

  • degraded is not treated as a failure by default. Error counters do not decrease, so degraded remains until the driver probes the card again (hot reset, module reload, reboot), and an SDK reset does not clear it. A case in which the firmware rejects an invalid request sent by an application is also counted in devcmd_fw_errors. So if degraded were treated as a failure, the health check would keep failing from then on even if an error had occurred only once. Judge whether the problem is continuing by whether the counters are increasing (Alert criteria). When you check right after installation or right after a hot reset, when the counters start at 0, pass -s so that degraded is also treated as a failure.
  • fw_version is checked as well. health does not take firmware readiness into account in its verdict, so a card whose firmware is not ready can also be ok.
  • The number of cards is checked separately. A card that the driver has not probed does not appear under /sys/class/ha/, so looking only at the health of the cards that are present cannot reveal a missing card.

What the health check does not see

health does not immediately show a stall in progress. Even if a card stalls, health does not change until the response wait ceiling passes and a timeout is recorded. A wait that an application requested to be without a ceiling is not recorded as a timeout, so health stays unchanged for as long as that wait is stalled (you can tell from Devcmd waiting in the kernel log). Temperature, power, and the telemetry polling state are also not part of the verdict. Monitor these separately by following Hang detection and the watchdog and Monitoring integration.


Why only sysfs is read

  • No device handle or root privileges are needed. Because the script does not open /dev/haN, it does not compete with workloads for the card and can run while a workload is using the card. The telemetry nodes and /sys/module/ha/version can be read by all users.
  • Values can be read even while the card is stalled. health and the error counters are computed from state that the driver keeps in host memory, so they respond even while compute work is stalled. fw_version is a value the driver has already read, and only while the value is unknown does a read check again whether the firmware is ready (Collection interval and cost).
  • It neither resets the card nor sends jobs. It only reads files. On a card whose fw_version is unknown, the driver checks again on read whether the firmware is ready and, if it is, fills in the value. This is the same operation that the driver's poller performs periodically.

Script

Save the following as ha-healthcheck.sh. It uses only POSIX sh syntax, so it also runs under a /bin/sh that is not bash (for example, dash).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
#!/bin/sh
# ha-healthcheck.sh: Reads the state of HyperAccel LPU cards from sysfs and issues a verdict.
# Usage: ha-healthcheck.sh [-n card count (1-8, default 1)] [-s (degraded also FAIL)]
# Exit codes: 0 normal, 1 a card is FAIL, 2 driver missing or too few cards, 3 invalid option
# Environment variable SYSFS: sysfs path (default /sys)

usage() {
  echo "usage: ${0##*/} [-n COUNT] [-s]" >&2
  exit 3
}

expect=1
on_degraded=WARN
while getopts n:s opt; do
  case $opt in
    n) expect=$OPTARG ;;
    s) on_degraded=FAIL ;;
    *) usage ;;
  esac
done
shift $((OPTIND - 1))
[ $# -eq 0 ] || usage
case $expect in
  [1-8]) ;;
  *) usage ;;
esac

SYSFS=${SYSFS:-/sys}

# If a file cannot be read, prints missing.
rd() {
  cat "$1" 2>/dev/null || echo missing
}

ver=$(rd "$SYSFS/module/ha/version")
set -- "$SYSFS"/class/ha/ha[0-9]*
# If there is no card at all, the pattern is left as is, so remove it.
[ -e "$1" ] || shift

rc=0
warn=0
for d in "$@"; do
  health=$(rd "$d/health")
  fw=$(rd "$d/fw_version")
  bdf=$(cd -P "$d/device" 2>/dev/null && pwd -P) || bdf=unknown
  case $health in
    ok) res=OK ;;
    degraded) res=$on_degraded ;;
    *) res=FAIL ;;
  esac
  # health does not look at firmware readiness, so fw_version is checked as well.
  case $fw in
    [0-9]*.[0-9]*.[0-9]*) ;;
    *) res=FAIL ;;
  esac
  case $res in
    WARN) warn=1 ;;
    FAIL) rc=1 ;;
  esac
  printf '%s %s bdf=%s health=%s fw_version=%s devcmd_timeouts=%s devcmd_fw_errors=%s hdma_timeouts=%s\n' \
    "${d##*/}" "$res" "${bdf##*/}" "$health" "$fw" \
    "$(rd "$d/devcmd_timeouts")" "$(rd "$d/devcmd_fw_errors")" "$(rd "$d/hdma_timeouts")"
done

if [ "$ver" = missing ] || [ $# -lt "$expect" ]; then
  rc=2
fi
res=OK
[ "$warn" -eq 0 ] || res=WARN
[ "$rc" -eq 0 ] || res=FAIL
printf 'host %s driver=%s cards=%s expected=%s\n' "$res" "$ver" "$#" "$expect"
exit "$rc"
Option, environment variable Meaning
-n N The number of cards that must be present. The range is 1–8 and the default is 1. If fewer than N cards are found, it ends with exit code 2
-s Also gives a card that is degraded the verdict FAIL
SYSFS The sysfs path. The default is /sys; specify it in an environment where the host's /sys is mounted at a different path (for example, a container)

For -n, write the number of cards installed in the host as a fixed value. You can check the number of cards visible on the PCI bus with lspci -D -d 20fb: | wc -l, but do not compute this value and use it on every run. If a card disappears from the PCI bus, that number decreases with it and the missing card is overlooked. The driver probes up to 8 cards on one host.

Exit code Meaning
0 The driver is loaded, there are N or more cards, and no card is FAIL. There may be cards that are WARN
1 There is a card that is FAIL
2 The driver is not loaded or there are fewer than N cards. Takes precedence over 1
3 An option is invalid

Running it and checking the result

Install the script and run it as a regular user. This is an example for a host with 2 cards.

1
2
sudo install -m 0755 ha-healthcheck.sh /usr/local/bin/ha-healthcheck.sh
ha-healthcheck.sh -n 2; echo "exit=$?"
1
2
3
4
ha0 OK bdf=0000:17:00.0 health=ok fw_version=1.0.5 devcmd_timeouts=0 devcmd_fw_errors=0 hdma_timeouts=0
ha1 OK bdf=0000:65:00.0 health=ok fw_version=1.0.5 devcmd_timeouts=0 devcmd_fw_errors=0 hdma_timeouts=0
host OK driver=2.6.0 cards=2 expected=2
exit=0

If the last line is host OK and the exit code is 0, everything is normal. Read the output as follows.

  • A card line consists of the card name, the verdict (OK, WARN, FAIL), and the PCI address (bdf) and the values read, in name=value format. The error counters are not used for the verdict; they are included for the record. If you keep them from every run, you can later find when a counter increased.
  • The host line shows the overall verdict, the driver version, the number of cards found, and the number of cards expected. The overall verdict is FAIL if the exit code is not 0, WARN if it is 0 and there is a card that is WARN, and OK otherwise. If the driver is not loaded, it shows driver=missing.
  • A file that cannot be read is shown as missing, and a card whose health or fw_version is missing is FAIL. This is the case when the driver is in the middle of probing or removing the card, or failed to create the telemetry nodes (telemetry sysfs unavailable in the kernel log).
  • The card number haN is not the identity of a card (Card numbers). Use bdf when referring to a card in alerts and records.
  • bdf is the same as Bus-Id of hyperaccel-smi. If health is ok, degraded, or faulted, the same value appears in uppercase in the Health column.

Test the failure path without touching a card. If you give SYSFS a nonexistent path, the result is the same as when the driver is not loaded.

1
SYSFS=/nonexistent ha-healthcheck.sh; echo "exit=$?"
1
2
host FAIL driver=missing cards=0 expected=1
exit=2

If a line that is not normal appears, see the following.

Output Exit code Where to look
haN WARN ... health=degraded 0 health is degraded
haN FAIL ... health=faulted 1 health is faulted
haN FAIL ... fw_version=unknown 1 fw_version is unknown
haN FAIL ... health=missing 1 Run it again after a while. If it continues, see Telemetry is unknown
host FAIL ... cards=1 expected=2 2 /dev/haN is missing
host FAIL driver=missing 2 Module is not loaded

Before you reset a card that is FAIL, first save the diagnostic information with Collecting diagnostic information for a problem, and then follow Recovering an unresponsive card. A reset erases the state that shows the cause.

While the module is being reloaded or ha-hotreset is making the driver probe a card again, the card directory can briefly disappear or its nodes may not have been created yet, so exit code 1 or 2 can occur. Stop the timer below or hold alerts while you do this kind of work.


Running periodically with a systemd timer

Create the service unit /etc/systemd/system/ha-healthcheck.service. For -n, write the number of cards in this host.

1
2
3
4
5
6
7
8
[Unit]
Description=HyperAccel LPU health check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ha-healthcheck.sh -n 2
# Because it only reads, it runs as a temporary non-root user.
DynamicUser=yes

Create the timer unit /etc/systemd/system/ha-healthcheck.timer. It runs every 5 minutes, starting 5 minutes after boot.

1
2
3
4
5
6
7
8
9
[Unit]
Description=Run HyperAccel LPU health check every 5 minutes

[Timer]
OnBootSec=5min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target
1
2
sudo systemctl daemon-reload
sudo systemctl enable --now ha-healthcheck.timer

The script's output is recorded in the journal on every run, and if the exit code is not 0, systemd records the service as failed (failed). View the result of the last run with systemctl status ha-healthcheck.service.

1
2
sudo journalctl -u ha-healthcheck.service --since today
sudo journalctl -u ha-healthcheck.service --since today | grep -E ' (WARN|FAIL) '

To send a failure as a notification, specify a unit that sends notifications with OnFailure= in the [Unit] section of the service unit. During maintenance, stop the timer with sudo systemctl stop ha-healthcheck.timer, and restart it with sudo systemctl start ha-healthcheck.timer when you are done.


Connecting to a monitoring agent

If you have a monitoring agent that runs a command and judges the state by its exit code, register the same command as a check.

  • Treat only exit code 0 as normal. 1 is a card state problem, 2 is a problem with driver loading or card probing, and 3 is a problem with the check configuration.
  • In an agent that interprets exit codes by the Nagios plugin convention (0 normal, 1 warning, 2 critical, 3 unknown), a card that is FAIL (1) is shown only as a warning. Set the agent's rules so that 1 is also treated as critical.
  • If you include the WARN and FAIL lines of the output in the alert body, which card failed and why is delivered along with the alert.
  • If the agent runs in a container and the host's /sys is mounted at a different path (for example, /host/sys), pass that path with SYSFS=/host/sys.

When you only need to check that cards are visible, the exit code of hyperaccel-smi is also enough. It ends with 2 if no card is visible and 0 if there is at least one. It does not look at card state, so it returns 0 even if a card is FAULTED, and if the environment variable HYPERACCEL_VISIBLE_DEVICES is set, it counts only the cards that its value allows (all cards for all, 0 cards for none or an empty value). ha-healthcheck.sh sees all cards on the host regardless of this variable.

1
hyperaccel-smi > /dev/null; echo $?

See also