Skip to content

hyperaccel-smi

hyperaccel-smi reads the telemetry that the driver exposes under /sys/class/ha/haN/ and shows the state of several cards on one screen. It does not open device nodes and does not need root privileges, so you can run it next to a workload for monitoring or health checks. This page defines the options, output formats, and exit codes. The meaning of each value is described in sysfs nodes.


Installation and privileges

Path /usr/bin/hyperaccel-smi
Distribution Included in the driver deb package (hyperdriver-dkms)
Requirements Python 3. It uses only the standard library, so no additional packages are needed. The deb package requires python3 as a dependency
Privileges Run it as a regular user. root is not required
Tool version 1.0.0

The tool reads only the read-only nodes under /sys/class/ha/haN/ and the /sys/class/ha/haN/device link. All users can read these nodes. Because it does not open /dev/haN, it does not compete with workloads for the device, and it keeps responding even while the device is stalled in the middle of a computation.

The tool version and the driver version are different

hyperaccel-smi -V prints the tool's own version. The Driver field in the default output only shows the module name ha. To check the loaded driver version, run:

1
cat /sys/module/ha/version

Synopsis

1
2
3
hyperaccel-smi [-i N] [-q] [--json] [-l SEC] [--root PATH]
hyperaccel-smi -V
hyperaccel-smi -h

Without options, the tool prints the default output once.


Options

Option Meaning
-i N, --id N Shows only the card numbered N (haN)
-q, --query Produces the detailed output, which lists the values item by item for each card
--json Produces JSON output. If it is combined with -q, --json takes precedence
-l SEC, --loop SEC Reads and prints again every SEC seconds. Fractional values are accepted. See loop mode
--root PATH The sysfs class path in which to look for cards. The default is the value of the environment variable HYPERACCEL_SYSFS_ROOT, and if that is not set either, /sys/class/ha
-V, --version Prints the tool version in the form hyperaccel-smi 1.0.0
-h, --help Prints the help text

Environment variables

Variable Meaning
HYPERACCEL_SYSFS_ROOT The default of --root. Use it when a container mounts the host's sysfs at a different path
HYPERACCEL_VISIBLE_DEVICES A comma-separated list of the card numbers to show (example: 0,2). If it is unset or all, all cards are shown; if it is none or empty, no card is shown. Non-numeric items are ignored, so if there is no numeric item at all, no card is shown. all and none are case-insensitive

HYPERACCEL_VISIBLE_DEVICES sets only the display scope

This variable only selects the cards that hyperaccel-smi prints. It does not change device access permissions. Choose the devices to expose to a container with the methods described in Container support.


How cards are found

  1. Under the class path (--root), the tool finds entries named ha<number> and sorts them by number. This number is the same as the N in /dev/haN.
  2. It filters them with -i and HYPERACCEL_VISIBLE_DEVICES. If both are given, only the cards that satisfy both conditions remain.
  3. For each card, the PCI address that the device link points to is used as the Bus-Id.

To identify the same card consistently, use the Bus-Id

The haN number can change when you reload the module or when the card configuration changes. The PCI address follows the slot in which the card is installed. Use the Bus-Id value wherever you specify a card, such as in inventories, alert rules, and ha-fw-update -d. Also use the PCI address to tell apart cards whose serial is unknown.


Default output

Without options, the tool prints three blocks: a card table, per-IP utilization, and a stream summary.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
+-----------------------------------------------------------------------------------------+
| HYPERACCEL-SMI 1.0.0      Tue Oct 06 12:02:50 2026                                      |
| Firmware Version: 1.0.5        Driver: ha                                               |
+-------------------------------------------+-----------------------+---------------------+
| LPU  Name                 Serial          | Bus-Id                | Health              |
| Temp   Power        Util(core)            | Memory-Usage (DRAM)   | Fw-Err / Timeouts   |
+===========================================+=======================+=====================+
|   0  HyperAccel LPU   HA-1a2b3c4d-5e6…    | 0000:17:00.0          | OK                  |
| 52C    38.2 W      63.5 %                 | 24.00 / 128.00 GiB    | 0 / 0               |
+-------------------------------------------+-----------------------+---------------------+
|   1  HyperAccel LPU   HA-5f0e9d8c-7b6…    | 0000:65:00.0          | FAULTED             |
| N/A    N/A         N/A                    | 8.00 / 128.00 GiB     | 0 / 1               |
+-------------------------------------------+-----------------------+---------------------+
+-----------------------------------------------------------------------------------------+
| Per-IP Utilization (%)                                                                  |
| LPU  core   mpu    vpu    cdma   cgdma                                                  |
+-----------------------------------------------------------------------------------------+
|   0  63.5   41.2   18.0   22.4   12.7                                                   |
|   1  unknown (firmware not sampled / unprovisioned)                                     |
+-----------------------------------------------------------------------------------------+
+-----------------------------------------------------------------------------------------+
| Streams / Inflight                                                                      |
| LPU  Streams  Inflight                                                                  |
+-----------------------------------------------------------------------------------------+
|   0  3        2                                                                         |
|   1  0        0                                                                         |
+-----------------------------------------------------------------------------------------+

Card table

The tool prints two lines for each card.

Column Node read Display
LPU The N in the card name haN Integer
Name None Always HyperAccel LPU
Serial serial If longer than 16 characters, … is appended after the first 15 characters. See the full value with -q or --json
Bus-Id device link PCI address (example: 0000:17:00.0)
Health health See the Health column below
Temp temp Integer in Celsius (example: 52C)
Power power_mw In W, one decimal place
Util(core) The core line of util %, one decimal place
Memory-Usage (DRAM) dram_used, dram_size <used> / <total> GiB, two decimal places
Fw-Err / Timeouts devcmd_fw_errors, devcmd_timeouts Two cumulative counters

The two header lines are as follows.

  • First line: the tool version and the time of output.
  • Firmware Version on the second line: the value of the first card, taking the printed cards in number order, whose fw_version can be read. Firmware can differ from card to card, so check per-card versions with -q or --json. If there is no card to print, it is N/A.
  • Driver on the second line: always ha.

Power is not the board input power but the sum of voltage × current across the monitored power rails. DMA timeouts (hdma_timeouts) are not in this table; they appear in -q and --json.

Health column

The value of the health node is shown converted to uppercase.

Display Node value
OK ok
DEGRADED degraded
FAULTED faulted

If the health node cannot be read or its value is unknown, the tool determines the state from the three RAS counters (devcmd_fw_errors, devcmd_timeouts, hdma_timeouts) instead. If none of the three can be read, the result is UNKNOWN; if any counter that was read is greater than 0, WARN; otherwise OK.

For the meaning of each value and the action to take, see Telemetry. DEGRADED does not mean that something is failing now; it means that one of the cumulative counters is greater than 0. Counters do not decrease, and they restart from 0 only when the driver probes the card again (for example, when you reload the module, or when you have ha-hotreset make the driver probe the card again). Resetting with the SDK's reset function leaves the counters unchanged. Therefore, a card that has become DEGRADED does not return to OK until then.

Per-IP utilization

From the util node, the tool shows the five IPs core, mpu, vpu, cdma, and cgdma as percentages with one decimal place. If util is unknown, the card's line shows unknown (firmware not sampled / unprovisioned). The other IPs that the util node reports are included in --json.

Stream summary

For each card, the tool shows the number of active streams (stream_count) and the number of in-flight tasks (inflight). It does not break them down by process.


Detailed output

With -q, the tool lists the values of each card together with item names. The following is card 1 of the example above, printed with -q -i 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
==== hyperaccel-smi 1.0.0 ====

LPU 1:
    Bus-Id                    : 0000:65:00.0
    Serial                    : HA-5f0e9d8c-7b6a-5948-3726-150400000000
    Firmware Version          : 1.0.5
    HW Version (PCIe rev)     : 0x00
    Health                    : FAULTED
    Temperature
        Peak                  : N/A
        Average               : N/A
    Power
        Total (rail V*I)      : N/A
    Memory
        DRAM Used             : 8.00 GiB
        DRAM Total            : 128.00 GiB
        SRAM Used             : 256.00 KiB
        SRAM Total            : 4.00 MiB
    Utilization (%)
        (unknown - not sampled)
    RAS / Health
        Streams               : 0
        Inflight              : 0
        Devcmd FW Errors      : 0
        Devcmd Timeouts       : 1
        HDMA Timeouts         : 0
        Last Hang             : rc=- cmd_idx=- opcode=-
        Hang Verdict          : not-picked-up
        Batch Progress        : batch_seq=41->41 exec_index=none resq_tail=0
    dma-buf Export
        TW Window Size        : 64.00 MiB
        Windows               : 0 / 8  (pins 0 / 12)
Item Node read Display
Bus-Id device link PCI address
Serial serial The node value as is
Firmware Version fw_version The node value as is
HW Version (PCIe rev) hw_version The node value as is
Health health Same as the Health column
Temperature › Peak, Average temp, temp_avg Integer in Celsius
Temperature › per-block lines temp_blocks Lists the lines of the node in order, including the final peak and avg lines. Omitted if the node is unknown
Power › Total (rail V*I) power_mw In W, one decimal place
Power › per-rail lines rails <rail> : <voltage> mV <current> mA. Omitted if the node is none or unknown. The read-error markers that the node appends after a value (V_ERR, I_ERR) are not printed, so if a rail value looks suspicious, check the rails node directly
Memory dram_used, dram_size, sram_used, sram_size Integer in B below 1 KiB; otherwise two decimal places in the most suitable unit among KiB, MiB, and GiB
Utilization (%) util The same five IPs as the default output. (unknown - not sampled) if unknown
Streams, Inflight stream_count, inflight Integer
Devcmd FW Errors, Devcmd Timeouts, HDMA Timeouts devcmd_fw_errors, devcmd_timeouts, hdma_timeouts Cumulative counters
Last Hang rc, cmd_idx, and opcode of last_hang The node values as is. none if the node cannot be read
Hang Verdict verdict of last_hang The node value as is
Batch Progress batch_seq, exec_index, and resq_tail of last_hang The node values as is
dma-buf Export › TW Window Size tw_window_size The same units as Memory
dma-buf Export › Windows First line of dmabuf_exports <used> / <max> (pins <pins> / <pin_max>). N/A if the node is unknown
dma-buf Export › per-window lines Window lines of dmabuf_exports region <number> ch<channel> : <da> <size> <state> pid <PID> (<comm>). <size> uses the same units as Memory, and owner hidden appears if the node's pid is -

For the meaning of Last Hang, Hang Verdict, and Batch Progress, see Hang detection and the watchdog and sysfs nodes. dram_free and sram_free do not appear in -q; they are included in --json.


JSON output

With --json, the tool prints an array that holds one object per card, in number order. If there is no card to print, the array is empty ([]). The following is card 1 of the example above, printed with -i 1 --json.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
[
  {
    "index": 1,
    "sysfs": "/sys/class/ha/ha1",
    "bdf": "0000:65:00.0",
    "fw_version": "1.0.5",
    "hw_version": "0x00",
    "serial": "HA-5f0e9d8c-7b6a-5948-3726-150400000000",
    "dram_size": 137438953472,
    "dram_used": 8589934592,
    "dram_free": 128849018880,
    "sram_size": 4194304,
    "sram_used": 262144,
    "sram_free": 3915776,
    "temp": null,
    "temp_avg": null,
    "temp_blocks": null,
    "power_mw": null,
    "rails": null,
    "util": null,
    "inflight": 0,
    "stream_count": 0,
    "devcmd_timeouts": 1,
    "devcmd_fw_errors": 0,
    "hdma_timeouts": 0,
    "last_hang": {
      "rc": "-",
      "cmd_idx": null,
      "opcode": "-",
      "batch_seq": "41->41",
      "exec_index": "none",
      "resq_tail": "0",
      "verdict": "not-picked-up"
    },
    "tw_window_size": 67108864,
    "dmabuf_exports": {
      "used": 0,
      "max": 8,
      "pins": 0,
      "pin_max": 12,
      "windows": []
    },
    "health_node": "faulted",
    "health": "FAULTED"
  }
]

Fields

Field Type Description
index Integer Card number (the N in haN)
sysfs String The card's sysfs path
bdf String PCI address
fw_version, hw_version, serial String or null The value of the node with the same name, as is. unknown is also included as a string
dram_size, dram_used, dram_free, sram_size, sram_used, sram_free Integer or null Bytes
temp, temp_avg Integer or null Celsius
temp_blocks Object or null {"<block>": <celsius>}. All lines of the temp_blocks node, including peak and avg
power_mw Integer or null mW
rails Array or null [{"name": "<rail>", "mv": <mV>, "ma": <mA>}]. An empty array if the node is none. The read-error markers (V_ERR, I_ERR) are not included
util Object or null {"<IP>": <percent>}. The values are floating-point numbers, and all IPs that the util node reports are included
inflight, stream_count Integer or null The number of in-flight tasks, the number of active streams
devcmd_timeouts, devcmd_fw_errors, hdma_timeouts Integer or null Cumulative counters
last_hang Object or null The table below
tw_window_size Integer or null Bytes
dmabuf_exports Object or null {"used", "max", "pins", "pin_max", "windows"}. windows holds one {"region", "ch", "da", "size", "state", "pid", "comm"} per window; da is a string, size is an integer in bytes, and pid and comm are null when they are - in the node
health_node String or null The value of the health node, as is (lowercase)
health String The same value as in the Health column (uppercase)

The last_hang object holds the fields of the last_hang node.

Field Type Description
rc, opcode String The node value as is (0x… or -)
cmd_idx Integer or null null if the node value is -
batch_seq, exec_index, resq_tail, verdict String The node value as is

When a monitoring system collects the data, use health (or health_node) and the cumulative counters. Numeric fields can be null, so a parser must handle null as a valid value.


Loop mode

With -l SEC, the tool reads all values again and prints them every SEC seconds.

  • A value shorter than 0.5 seconds is treated as 0.5 seconds.
  • When printing to a terminal, the tool clears the screen and redraws it each time. With --json, it does not clear the screen and instead prints one JSON array per cycle, one after another.
  • It does not end even if there is no card; it keeps running.
  • Press Ctrl+C to end it. The exit code is 130.

In a collector, call the tool once per run instead of using loop mode

If you use -l and --json together, the output as a whole is not a single JSON document. For periodic collection, it is easier to handle if the collector runs hyperaccel-smi --json once per interval that the collector defines.

Independently of the screen refresh interval, the driver fetches the temperature, power, and utilization values from the firmware and updates them once per telemetry polling interval. Reading again at a shorter interval returns the same values. That interval is set by the module parameter ha_sysmon_poll_ms.


N/A, unknown, -

Display Meaning
N/A The node of a numeric value is unknown, or the node cannot be read. The Serial column of the table is also N/A if serial is unknown
unknown A value that the driver reports because it cannot trust the actual value. String values (Serial and Firmware Version in -q, Firmware Version in the header, and the string fields in JSON) show the node value as is, so unknown appears unchanged
- A value left blank in last_hang because it has never been recorded or the driver cannot determine it. It is not 0
null Corresponds to N/A in JSON

For the conditions under which unknown appears (during a reset, FATAL, no firmware response, and so on), see Telemetry.


Exit codes

Code Meaning
0 At least one card was printed. -V and -h also end with 0
2 There is no card to print. This happens when the driver is not loaded, there is no card, or all cards are filtered out by -i or HYPERACCEL_VISIBLE_DEVICES. The tool also ends with 2 when a command-line option is invalid; in that case, the usage is printed to standard error
130 Interrupted with Ctrl+C

When there is no card, the default output shows a line that starts with No HyperAccel LPU found under in place of the card table.

Exit code 0 does not mean the cards are healthy

The exit code only tells you whether cards are visible. It is 0 even if a card is FAULTED. Judge the state by the health value.


Usage examples

View the overall state once:

1
hyperaccel-smi

View all values of a single card:

1
hyperaccel-smi -q -i 0

Watch the state, refreshing every 2 seconds:

1
hyperaccel-smi -l 2

Extract only the fields you need:

1
2
3
4
hyperaccel-smi --json | python3 -c '
import json, sys
for d in json.load(sys.stdin):
    print(d["index"], d["bdf"], d["health"], d["temp"], d["power_mw"])'

Use it as a health check. It ends with 1 if no card is visible or if any card is FAULTED:

1
2
3
4
hyperaccel-smi --json | python3 -c '
import json, sys
d = json.load(sys.stdin)
sys.exit(0 if d and all(x["health"] != "FAULTED" for x in d) else 1)'

This example checks only whether any card is visible, and health. A health check that also verifies driver loading, the number of cards, and each card's fw_version, and that runs periodically, is in Building a card health check; how to collect data with Prometheus is in Collecting telemetry with Prometheus.

When you run the tool inside a container, the container must be able to read the host's /sys/class/ha. If you mounted it at a different path, specify that path:

1
HYPERACCEL_SYSFS_ROOT=/host/sys/class/ha hyperaccel-smi --json

See also