hyperaccel-smi¶
hyperaccel-smi reads the telemetry that the driver exposes under /sys/class/ha/haN/ and shows the state of several cards on one screen. It does not open device nodes and does not need root privileges, so you can run it next to a workload for monitoring or health checks. This page defines the options, output formats, and exit codes. The meaning of each value is described in sysfs nodes.
Installation and privileges¶
| Path | /usr/bin/hyperaccel-smi |
| Distribution | Included in the driver deb package (hyperdriver-dkms) |
| Requirements | Python 3. It uses only the standard library, so no additional packages are needed. The deb package requires python3 as a dependency |
| Privileges | Run it as a regular user. root is not required |
| Tool version | 1.0.0 |
The tool reads only the read-only nodes under /sys/class/ha/haN/ and the /sys/class/ha/haN/device link. All users can read these nodes. Because it does not open /dev/haN, it does not compete with workloads for the device, and it keeps responding even while the device is stalled in the middle of a computation.
The tool version and the driver version are different
hyperaccel-smi -V prints the tool's own version. The Driver field in the default output only shows the module name ha. To check the loaded driver version, run:
1 | |
Synopsis¶
1 2 3 | |
Without options, the tool prints the default output once.
Options¶
| Option | Meaning |
|---|---|
-i N, --id N |
Shows only the card numbered N (haN) |
-q, --query |
Produces the detailed output, which lists the values item by item for each card |
--json |
Produces JSON output. If it is combined with -q, --json takes precedence |
-l SEC, --loop SEC |
Reads and prints again every SEC seconds. Fractional values are accepted. See loop mode |
--root PATH |
The sysfs class path in which to look for cards. The default is the value of the environment variable HYPERACCEL_SYSFS_ROOT, and if that is not set either, /sys/class/ha |
-V, --version |
Prints the tool version in the form hyperaccel-smi 1.0.0 |
-h, --help |
Prints the help text |
Environment variables¶
| Variable | Meaning |
|---|---|
HYPERACCEL_SYSFS_ROOT |
The default of --root. Use it when a container mounts the host's sysfs at a different path |
HYPERACCEL_VISIBLE_DEVICES |
A comma-separated list of the card numbers to show (example: 0,2). If it is unset or all, all cards are shown; if it is none or empty, no card is shown. Non-numeric items are ignored, so if there is no numeric item at all, no card is shown. all and none are case-insensitive |
HYPERACCEL_VISIBLE_DEVICES sets only the display scope
This variable only selects the cards that hyperaccel-smi prints. It does not change device access permissions. Choose the devices to expose to a container with the methods described in Container support.
How cards are found¶
- Under the class path (
--root), the tool finds entries namedha<number>and sorts them by number. This number is the same as the N in/dev/haN. - It filters them with
-iandHYPERACCEL_VISIBLE_DEVICES. If both are given, only the cards that satisfy both conditions remain. - For each card, the PCI address that the
devicelink points to is used as theBus-Id.
To identify the same card consistently, use the Bus-Id
The haN number can change when you reload the module or when the card configuration changes. The PCI address follows the slot in which the card is installed. Use the Bus-Id value wherever you specify a card, such as in inventories, alert rules, and ha-fw-update -d. Also use the PCI address to tell apart cards whose serial is unknown.
Default output¶
Without options, the tool prints three blocks: a card table, per-IP utilization, and a stream summary.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | |
Card table¶
The tool prints two lines for each card.
| Column | Node read | Display |
|---|---|---|
LPU |
The N in the card name haN |
Integer |
Name |
None | Always HyperAccel LPU |
Serial |
serial |
If longer than 16 characters, … is appended after the first 15 characters. See the full value with -q or --json |
Bus-Id |
device link |
PCI address (example: 0000:17:00.0) |
Health |
health |
See the Health column below |
Temp |
temp |
Integer in Celsius (example: 52C) |
Power |
power_mw |
In W, one decimal place |
Util(core) |
The core line of util |
%, one decimal place |
Memory-Usage (DRAM) |
dram_used, dram_size |
<used> / <total> GiB, two decimal places |
Fw-Err / Timeouts |
devcmd_fw_errors, devcmd_timeouts |
Two cumulative counters |
The two header lines are as follows.
- First line: the tool version and the time of output.
Firmware Versionon the second line: the value of the first card, taking the printed cards in number order, whosefw_versioncan be read. Firmware can differ from card to card, so check per-card versions with-qor--json. If there is no card to print, it isN/A.Driveron the second line: alwaysha.
Power is not the board input power but the sum of voltage × current across the monitored power rails. DMA timeouts (hdma_timeouts) are not in this table; they appear in -q and --json.
Health column¶
The value of the health node is shown converted to uppercase.
| Display | Node value |
|---|---|
OK |
ok |
DEGRADED |
degraded |
FAULTED |
faulted |
If the health node cannot be read or its value is unknown, the tool determines the state from the three RAS counters (devcmd_fw_errors, devcmd_timeouts, hdma_timeouts) instead. If none of the three can be read, the result is UNKNOWN; if any counter that was read is greater than 0, WARN; otherwise OK.
For the meaning of each value and the action to take, see Telemetry. DEGRADED does not mean that something is failing now; it means that one of the cumulative counters is greater than 0. Counters do not decrease, and they restart from 0 only when the driver probes the card again (for example, when you reload the module, or when you have ha-hotreset make the driver probe the card again). Resetting with the SDK's reset function leaves the counters unchanged. Therefore, a card that has become DEGRADED does not return to OK until then.
Per-IP utilization¶
From the util node, the tool shows the five IPs core, mpu, vpu, cdma, and cgdma as percentages with one decimal place. If util is unknown, the card's line shows unknown (firmware not sampled / unprovisioned). The other IPs that the util node reports are included in --json.
Stream summary¶
For each card, the tool shows the number of active streams (stream_count) and the number of in-flight tasks (inflight). It does not break them down by process.
Detailed output¶
With -q, the tool lists the values of each card together with item names. The following is card 1 of the example above, printed with -q -i 1.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | |
| Item | Node read | Display |
|---|---|---|
Bus-Id |
device link |
PCI address |
Serial |
serial |
The node value as is |
Firmware Version |
fw_version |
The node value as is |
HW Version (PCIe rev) |
hw_version |
The node value as is |
Health |
health |
Same as the Health column |
Temperature › Peak, Average |
temp, temp_avg |
Integer in Celsius |
Temperature › per-block lines |
temp_blocks |
Lists the lines of the node in order, including the final peak and avg lines. Omitted if the node is unknown |
Power › Total (rail V*I) |
power_mw |
In W, one decimal place |
Power › per-rail lines |
rails |
<rail> : <voltage> mV <current> mA. Omitted if the node is none or unknown. The read-error markers that the node appends after a value (V_ERR, I_ERR) are not printed, so if a rail value looks suspicious, check the rails node directly |
Memory |
dram_used, dram_size, sram_used, sram_size |
Integer in B below 1 KiB; otherwise two decimal places in the most suitable unit among KiB, MiB, and GiB |
Utilization (%) |
util |
The same five IPs as the default output. (unknown - not sampled) if unknown |
Streams, Inflight |
stream_count, inflight |
Integer |
Devcmd FW Errors, Devcmd Timeouts, HDMA Timeouts |
devcmd_fw_errors, devcmd_timeouts, hdma_timeouts |
Cumulative counters |
Last Hang |
rc, cmd_idx, and opcode of last_hang |
The node values as is. none if the node cannot be read |
Hang Verdict |
verdict of last_hang |
The node value as is |
Batch Progress |
batch_seq, exec_index, and resq_tail of last_hang |
The node values as is |
dma-buf Export › TW Window Size |
tw_window_size |
The same units as Memory |
dma-buf Export › Windows |
First line of dmabuf_exports |
<used> / <max> (pins <pins> / <pin_max>). N/A if the node is unknown |
dma-buf Export › per-window lines |
Window lines of dmabuf_exports |
region <number> ch<channel> : <da> <size> <state> pid <PID> (<comm>). <size> uses the same units as Memory, and owner hidden appears if the node's pid is - |
For the meaning of Last Hang, Hang Verdict, and Batch Progress, see Hang detection and the watchdog and sysfs nodes. dram_free and sram_free do not appear in -q; they are included in --json.
JSON output¶
With --json, the tool prints an array that holds one object per card, in number order. If there is no card to print, the array is empty ([]). The following is card 1 of the example above, printed with -i 1 --json.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | |
Fields¶
| Field | Type | Description |
|---|---|---|
index |
Integer | Card number (the N in haN) |
sysfs |
String | The card's sysfs path |
bdf |
String | PCI address |
fw_version, hw_version, serial |
String or null |
The value of the node with the same name, as is. unknown is also included as a string |
dram_size, dram_used, dram_free, sram_size, sram_used, sram_free |
Integer or null |
Bytes |
temp, temp_avg |
Integer or null |
Celsius |
temp_blocks |
Object or null |
{"<block>": <celsius>}. All lines of the temp_blocks node, including peak and avg |
power_mw |
Integer or null |
mW |
rails |
Array or null |
[{"name": "<rail>", "mv": <mV>, "ma": <mA>}]. An empty array if the node is none. The read-error markers (V_ERR, I_ERR) are not included |
util |
Object or null |
{"<IP>": <percent>}. The values are floating-point numbers, and all IPs that the util node reports are included |
inflight, stream_count |
Integer or null |
The number of in-flight tasks, the number of active streams |
devcmd_timeouts, devcmd_fw_errors, hdma_timeouts |
Integer or null |
Cumulative counters |
last_hang |
Object or null |
The table below |
tw_window_size |
Integer or null |
Bytes |
dmabuf_exports |
Object or null |
{"used", "max", "pins", "pin_max", "windows"}. windows holds one {"region", "ch", "da", "size", "state", "pid", "comm"} per window; da is a string, size is an integer in bytes, and pid and comm are null when they are - in the node |
health_node |
String or null |
The value of the health node, as is (lowercase) |
health |
String | The same value as in the Health column (uppercase) |
The last_hang object holds the fields of the last_hang node.
| Field | Type | Description |
|---|---|---|
rc, opcode |
String | The node value as is (0x… or -) |
cmd_idx |
Integer or null |
null if the node value is - |
batch_seq, exec_index, resq_tail, verdict |
String | The node value as is |
When a monitoring system collects the data, use health (or health_node) and the cumulative counters. Numeric fields can be null, so a parser must handle null as a valid value.
Loop mode¶
With -l SEC, the tool reads all values again and prints them every SEC seconds.
- A value shorter than 0.5 seconds is treated as 0.5 seconds.
- When printing to a terminal, the tool clears the screen and redraws it each time. With
--json, it does not clear the screen and instead prints one JSON array per cycle, one after another. - It does not end even if there is no card; it keeps running.
- Press Ctrl+C to end it. The exit code is
130.
In a collector, call the tool once per run instead of using loop mode
If you use -l and --json together, the output as a whole is not a single JSON document. For periodic collection, it is easier to handle if the collector runs hyperaccel-smi --json once per interval that the collector defines.
Independently of the screen refresh interval, the driver fetches the temperature, power, and utilization values from the firmware and updates them once per telemetry polling interval. Reading again at a shorter interval returns the same values. That interval is set by the module parameter ha_sysmon_poll_ms.
N/A, unknown, -¶
| Display | Meaning |
|---|---|
N/A |
The node of a numeric value is unknown, or the node cannot be read. The Serial column of the table is also N/A if serial is unknown |
unknown |
A value that the driver reports because it cannot trust the actual value. String values (Serial and Firmware Version in -q, Firmware Version in the header, and the string fields in JSON) show the node value as is, so unknown appears unchanged |
- |
A value left blank in last_hang because it has never been recorded or the driver cannot determine it. It is not 0 |
null |
Corresponds to N/A in JSON |
For the conditions under which unknown appears (during a reset, FATAL, no firmware response, and so on), see Telemetry.
Exit codes¶
| Code | Meaning |
|---|---|
0 |
At least one card was printed. -V and -h also end with 0 |
2 |
There is no card to print. This happens when the driver is not loaded, there is no card, or all cards are filtered out by -i or HYPERACCEL_VISIBLE_DEVICES. The tool also ends with 2 when a command-line option is invalid; in that case, the usage is printed to standard error |
130 |
Interrupted with Ctrl+C |
When there is no card, the default output shows a line that starts with No HyperAccel LPU found under in place of the card table.
Exit code 0 does not mean the cards are healthy
The exit code only tells you whether cards are visible. It is 0 even if a card is FAULTED. Judge the state by the health value.
Usage examples¶
View the overall state once:
1 | |
View all values of a single card:
1 | |
Watch the state, refreshing every 2 seconds:
1 | |
Extract only the fields you need:
1 2 3 4 | |
Use it as a health check. It ends with 1 if no card is visible or if any card is FAULTED:
1 2 3 4 | |
This example checks only whether any card is visible, and health. A health check that also verifies driver loading, the number of cards, and each card's fw_version, and that runs periodically, is in Building a card health check; how to collect data with Prometheus is in Collecting telemetry with Prometheus.
When you run the tool inside a container, the container must be able to read the host's /sys/class/ha. If you mounted it at a different path, specify that path:
1 | |
See also¶
- sysfs nodes — definitions of the nodes the tool reads, and of the nodes it does not show (
peer_map,peer_route,peer_sram,abi_prefetch_stats,fw_update_progress, hwmon) - Telemetry — how values are updated, the meaning of
unknown, and thehealthdetermination - Hang detection and the watchdog — how to read
Last HangandHang Verdict - FATAL and recovery — actions for a
FAULTEDcard - Container support
- Parameter reference
- Verifying the installation