Skip to content

Telemetry

ha.ko exposes the state of each device as read-only files under /sys/class/ha/haN/. The firmware version and chip serial number, temperature, power, and utilization, memory usage, job counts and error counters, and the health verdict (health) are all there, and temperature, voltage, current, and power are also exposed through Linux hwmon. This page explains where the values come from and when they are refreshed, what unknown means, how health is determined, and how to connect to a monitoring system. The output format of each node is in sysfs nodes.


Design principles

Telemetry follows the principles below so that it can be read even at the moment a device runs into a problem.

Principle What it means in operation
Does not open the device node It only reads files and never opens /dev/haN. A monitoring agent does not compete with the workload for the device, and the values can be read even when there is no workload
No privileges required All telemetry nodes are 0444. A regular user can read them with cat
Separate from the compute path Temperature, power, and utilization are fetched through a separate monitoring channel of the firmware, and most of the other values are calculated from state that the driver keeps in host memory. They do not go through the command queue and locks that compute work uses, so they respond even while compute work is stopped
Read-only Telemetry nodes can only be read. bypass_range_check in the same directory is not telemetry but a debug setting
fail-closed If a value cannot be trusted, unknown is reported. A value past its validity period is not reported again, and gaps are not filled with 0
1
2
3
cat /sys/class/ha/ha0/health
cat /sys/class/ha/ha0/temp
grep -H . /sys/class/ha/ha0/{fw_version,serial,health,temp,power_mw,dram_used,inflight}

Collectors must handle unknown

unknown is not an error but a normal output. A parser that expects only numbers fails when the firmware is not yet ready or while the device is being reset, that is, at the moments when status information is needed most. In the same situations, hwmon files return a read error (No data available) instead of unknown.


Telemetry node types and refresh methods

Telemetry nodes fall into five types, depending on how the value is obtained. The Refresh column in sysfs nodes follows this classification.

Type Nodes Where the value comes from Refresh
Fixed hw_version, dram_size, sram_size, tw_window_size, help PCI configuration space, the driver's memory configuration, and the help text built into the driver Never changes
Cache fw_version, serial Values the firmware exposes Read once when the firmware is ready. Read again when the device is reset.
Monitoring temp, temp_avg, temp_blocks, power_mw, rails, util, hwmon Firmware monitoring channel Every ha_sysmon_poll_ms (200 ms by default)
On read dram_used, dram_free, sram_used, sram_free, inflight, stream_count, devcmd_timeouts, devcmd_fw_errors, hdma_timeouts, abi_prefetch_stats, last_hang, health, peer_map, peer_route, peer_sram, dmabuf_exports State kept by the driver Calculated on every read
Device fw_update_progress The device Read directly from the device on every read

Cache: firmware version and chip serial number

fw_version and serial are values that the firmware exposes in device registers.

  • If the firmware is not yet ready when the driver probes the device, both values start as unknown. This is so that a half-written value from a firmware that is still booting is not reported as a version. The driver checks again whether the firmware is ready each time one of these nodes is read and each time the poller runs. When the firmware is ready, the driver reads and fills in the values, and logs firmware booted after probe gave up; CSR-derived telemetry refreshed to the kernel log. If the firmware is still not ready, the driver logs telemetry stays "unknown": device is not ready (the output rate is limited).
  • When the device is reset, the driver checks firmware readiness at the end of the reset and reads the values again. If it finds that the firmware has not returned to the ready state, it changes the values to unknown.
  • The stored values are the values from the time they were read. A version shown in fw_version does not mean that the firmware is responding now. Judge the current state by health and the counters.
  • fw_version does not change immediately when you update the firmware. It changes only after the card boots with the new firmware and the driver reads the value again. See Firmware update.

Periodic monitoring: temperature, power, utilization

Every ha_sysmon_poll_ms (200 ms by default, 5 times per second), the driver requests temperature, power, and utilization in turn from the firmware's monitoring channel and stores the results in a cache. sysfs nodes and hwmon read this cache. In the kernel log, the component that makes these requests, together with the channel, is called sys_mon.

  • Reading a node does not access the device. If you read more often than the polling interval, you receive the same sample again.
  • A sample is valid for 5 seconds. If a request fails, the cache is left as it is, so the values do not drop out as long as the next sample arrives within 5 seconds. When the last good sample is older than 5 seconds, the value becomes unknown.
  • If a power sample contains a physically impossible voltage or current, that sample is discarded and the previous sample is kept.
  • A single request waits up to 50 ms for a response. It does not use the same command queue as compute work, so periodic monitoring continues even when compute work has stopped. Monitoring is handled on a path separate from compute. If temperature, power, and utilization keep updating while a kernel run is not answering, monitoring is still working. These values become unknown only when monitoring or the connection to the device has stopped responding on its own.
  • ha_sysmon_poll_ms can be changed at run time and takes effect starting with the next scheduled poll. A value outside 50–2500 ms is clamped to that range, and 0 is not off but the fastest interval (50 ms). The longer the interval, the fewer failed requests it takes to exceed the 5-second validity period and turn the value unknown. See Parameter reference.

Firmware clock control

When a compute-group temperature or the power crosses a threshold, the firmware halves the core clock of each compute group. This is the only thermal response in this firmware. The firmware does not reset the card when the temperature rises further.

Clock is halved Clock is restored
Hottest compute-group temperature 90°C 80°C
Sum of rail power 350 W 300 W

Crossing either threshold halves the clock. The clock returns to its original rate only after both inputs are back under the restore values. If the temperature cannot be read, the firmware does not restore the clock. If the power cannot be read for a while, the firmware keeps the current clock decision for five tries and then judges from temperature alone.

This state does not appear in sysfs or hyperaccel-smi. temp can rise and throughput can drop while health stays ok, and this is not a hang.

The value to compare with the threshold is not the temp shown on screen. Clock control uses the hottest compute-group sensor, not the hottest of all block sensors, and the power it uses is the same rail sum as power_mw. temp is approximate, so do not compare it directly with the limits in the datasheet. 90°C and 350 W are the values the firmware applies to that sensor and to the rail sum.

Values calculated on read

Memory usage, job counts, error counters, last_hang, health, and device-to-device path information are calculated on every read from state that the driver keeps in host memory. They do not access the device, so they are always current values, and they respond even while compute work is stopped.

The error counters (devcmd_timeouts, devcmd_fw_errors, hdma_timeouts), abi_prefetch_stats, and last_hang are records that have accumulated since the driver probed the device. They start counting over only when the driver probes the device again (module reload, unbind followed by bind, ha-hotreset, host reboot).

Values read directly from the device

Each time the fw_update_progress node is read, the driver reads the firmware update progress directly from the device. It does not go through the command queue, so it can be read while an update is in progress. Do not include it in continuous collection; read it only while you are watching a firmware update. Do not read it on an unresponsive device (a device whose health is faulted or whose monitoring collection has stopped).


Conditions under which unknown appears

unknown means that there is no value, or that the value cannot be trusted at the moment. The conditions differ for each telemetry node.

Node Condition When it clears
fw_version, serial The firmware is not yet ready (it is still booting, or booting failed) When the firmware becomes ready, the driver detects it and fills in the value
fw_version The firmware version register reads as 0, or with all bits set to 1 When the driver checks again and reads a valid value
serial The firmware does not expose the chip serial number, or no number is recorded on the chip It does not change for that combination of card and firmware
All monitoring nodes The driver has just probed the device, so there is no first sample yet When the first sample arrives
All monitoring nodes The last good sample is older than 5 seconds (causes below) When samples arrive again
util The firmware has not yet finished its first measurement interval When the first interval ends
util The firmware's measurement has not progressed for more than 5 seconds When measurement progresses again
util A value above 100% was received for at least one IP When the next valid sample arrives
power_mw No rail is being measured (rails is none in this case) -
fw_update_progress The firmware is not ready, does not support progress reporting, or the value could not be read from the device -
peer_sram The LPU is being reset, is not ready, is FATAL, or is not responding and monitoring has stopped When the state returns and the path is calculated again
peer_route The PCIe path could not be determined It is determined again when the driver probes the device again

The last sample of a monitoring node becomes older than 5 seconds for the following reasons.

  • The device is being reset. Polling is stopped so that the device is not read during a reset.
  • The device is FATAL (health verdict). To avoid continuing to read a stopped device, polling requests are not sent until FATAL is cleared.
  • The monitoring component has stopped. See Monitoring suspension below.
  • The firmware has not opened the monitoring channel, or the version of the monitoring format that the firmware sends is lower than the minimum version the driver accepts.
  • The firmware does not respond to requests, or sends invalid responses.
  • The driver could not start monitoring. sys_mon telemetry unavailable is logged to the kernel log, and in this case no hwmon device is created either.

For health, the memory nodes, abi_prefetch_stats, peer_map, and peer_route, the entire node becomes unknown only when the driver cannot read its internal state, and this does not happen after device probing has completed.

Monitoring suspension

If all three requests (temperature, power, utilization) fail in a single monitoring round ha_sysmon_fail_limit times in a row (15 by default), the driver stops monitoring by itself. With the default settings (15 times, 200 ms interval), it takes about 3 seconds or more from the moment responses stop until monitoring stops, and increasing ha_sysmon_poll_ms delays it by that much.

Continuing to read an unresponsive device can cause a PCIe Completion Timeout. Some servers in which platform firmware handles PCIe errors first (firmware-first) treat this as an unrecoverable error and can reset the entire host. Monitoring is stopped so that the host is not lost because of a single unresponsive card.

When monitoring stops, an error like the following is logged to the kernel log (the numbers vary from case to case).

1
sys_mon: no response for 15 consecutive polls (3011 ms) - suspending telemetry polling. Reading a wedged device can raise a PCIe Completion Timeout that firmware-first platforms escalate to a system reset. Reset the device to resume.

After monitoring stops, the following applies.

  • Monitoring nodes and hwmon values become unknown (a read error for hwmon) 5 seconds after the last good sample.
  • For a card whose monitoring channel has been opened at least once, the driver does not poll that card until it is reset, and does not re-check fw_version and serial even if they are unknown. The peer_sram lines related to that card also become unknown.
  • Monitoring does not restart by itself. When you reset the device and it responds again, the driver restarts polling and logs sys_mon: resuming telemetry polling. When the driver probes the device again (for example with ha-hotreset), monitoring also starts over from the beginning.
  • A card whose monitoring channel has never been opened (a card that boots late) is an exception. If you read fw_version or serial after the firmware is ready, the driver confirms readiness and also restarts polling.
  • On a card whose firmware does not open the monitoring channel, all requests fail as well, so monitoring stops in the same way and the same log is written.

ha_sysmon_fail_limit can be changed at run time, and with 0 monitoring never stops. However, this means that an unresponsive device keeps being read, so do not set it to 0 on firmware-first platforms. See Parameter reference.


health verdict

health is a one-word verdict that the driver derives from its internal state. In a health check, look at this value before interpreting the counters directly. The verdicts are applied in order from the top of the table.

Value Condition When it clears
faulted The device is in the FATAL state, or the last reset (an SDK reset or the automatic reset) failed FATAL clears when a reset requested through the SDK, or an automatic reset, finishes. A reset failure clears when a later reset succeeds, or when the driver confirms that the device is ready again. Both clear when the driver probes the device again
degraded At least one of devcmd_timeouts, devcmd_fw_errors, hdma_timeouts is not 0 When the driver probes the device again
ok None of the conditions above applies -
unknown The driver cannot read the state It does not occur after device probing has completed

If, during work, the device does not respond within the driver's ceiling time (ha_devcmd_ceiling_ms, 12 seconds by default), or the firmware reports a device-level error rather than the failure of a single command, the driver marks that device FATAL. If watchdog preemption (ha_wd_preempt) is enabled, the device also becomes FATAL when the watchdog cuts off a stalled wait (when it cuts off a wait that has no ceiling, only that request fails and the device does not become FATAL). FATAL is a state of the device, so restarting the process does not clear it, and work on that device is rejected until it is reset. For the causes and the recovery procedure, see FATAL and recovery.

Keep the following in mind when you read the verdict.

  • degraded does not mean that something is failing right now. The counters only increase after the device is probed, so even a single past error leaves the verdict at degraded until the driver probes the device again. Read degraded as "check the counters", and judge whether the problem is continuing by whether the counters are increasing.
  • A device that became FATAL because of a timeout or a firmware error is shown as degraded even after recovery. This is because the cause remains in the counters.
  • Application errors also produce degraded. devcmd_fw_errors also counts cases where the firmware responded with an error to an invalid request sent by an application. Use last_hang and the kernel log to tell whether it is a hardware fault.
  • If a reset fails, health stays faulted. This holds for a reset requested through the SDK and for an automatic reset alike. The driver still accepts work again, but it does not attempt an automatic reset on that device again until a reset succeeds or the card becomes ready again.
  • health does not immediately show a stall that is in progress. Even if the device stops, it does not change until the ceiling time passes and a timeout is recorded. A wait that an application requested to be without a ceiling is not recorded as a timeout, so health keeps not changing while that wait is stalled. To detect a stall in progress, see Hang detection and the watchdog.
  • health does not factor firmware readiness or the state of monitoring nodes such as temperature and power into the verdict. A card whose firmware is not ready can also be ok, so also check that fw_version is not unknown to know whether the device is usable. Monitor overheating and telemetry gaps separately.

hwmon integration

The driver also exposes the temperature, voltage, current, and power collected by the poller as a Linux hwmon device. Tools that follow the hwmon convention (such as sensors from lm-sensors and the hwmon collector of Prometheus node_exporter) read these values without any additional configuration.

  • Each device has one hwmon device, and its name is ha. It is located at /sys/class/ha/haN/device/hwmon/hwmonM/ and is the same device as /sys/class/hwmon/hwmonM. M is a number assigned by the kernel and is unrelated to N.
  • Values are read from the same cache as the sysfs poller nodes.
  • Units follow the hwmon convention. Temperature is in millidegrees (m°C), voltage in mV, current in mA, and power in µW. Voltage and current use the same units as rails, while temperature and power differ from the sysfs nodes (°C, mW).
  • When a value cannot be trusted, a read error (ENODATA) is returned instead of unknown. The conditions are the same as for the poller nodes.
  • The kernel must support hwmon (CONFIG_HWMON). On a kernel without support, no hwmon device is created, and if registration fails on a kernel with support, hwmon register failed is logged to the kernel log. In either case, the sysfs nodes keep working as before.
1
2
hw=$(echo /sys/class/ha/ha0/device/hwmon/hwmon*)
cat "$hw/name" "$hw/temp1_label" "$hw/temp1_input"
1
2
3
ha
peak
47000

For the list of channels and labels, see sysfs nodes.


Monitoring integration

Collection interval and cost

  • Reading most nodes does not access the device. Monitoring nodes read the cache, and the others read the driver's host-side state. The exceptions are fw_update_progress (always read from the device) and fw_version and serial while fw_version is unknown (firmware readiness is checked again on each read).
  • Poller nodes change every 200 ms by default, so reading more often than that yields no new values. You can use a typical monitoring collection interval (several seconds to several tens of seconds) as it is.
  • Exclude fw_update_progress, help, and bypass_range_check from continuous collection.
  • When a device is detached from the driver (module removal, unbind, while ha-hotreset is running), /sys/class/ha/haN/ also disappears. Collectors must handle the case in which the directory disappears.

Alert criteria

Signal Meaning Recommended response
health is faulted The device is FATAL or the last reset failed Alert immediately. Follow FATAL and recovery
An error counter increases An operation exceeded the ceiling time, or a firmware error occurred Alert on the increase. Check last_hang and the kernel log
health is degraded At least one error has occurred since the device was probed Use it for status display only. Judge whether the problem is continuing by counter increases
A monitoring node stays unknown Causes include a reset, FATAL, monitoring suspension, or firmware not ready (Conditions under which unknown appears) Check health, fw_version, and the kernel log
inflight is not 0 but the workload is not making progress Work may have stopped Follow Hang detection and the watchdog

Health check script

A health check that reads only sysfs does not open the device node, so it can run alongside the workload, and it keeps working while the workload is stopped. An example script that checks whether the driver is loaded, the number of cards, and each card's health and fw_version and reports the result through its exit code, and how to run it periodically, are in Building a card health check.

hyperaccel-smi

hyperaccel-smi is a command that reads the sysfs nodes and shows them as a table. It needs neither root privileges nor a device node, and it can print JSON with the --json option. For detailed usage, see hyperaccel-smi.

Using Prometheus

Temperature, voltage, current, and power are collected from the hwmon device by the hwmon collector of node_exporter. Values that are not in hwmon, such as health and the error counters, can be exported with the textfile collector. Example scripts and alert rule examples are in Collecting telemetry with Prometheus.

Counters restart from 0 when the driver probes the device again, so base alert rules on the increase (such as increase()) rather than on the value itself.


Telemetry during a reset and during FATAL

Node During FATAL While a reset is in progress After the reset finishes
Monitoring nodes, hwmon No monitoring requests are sent, so unknown 5 seconds after the last sample Monitoring is stopped, so unknown 5 seconds after the last sample Filled in again soon after the device responds
fw_version, serial The stored values The stored values Read again if the driver confirms firmware readiness at the end of the reset; unknown if it confirms that the firmware is not ready
Memory usage Allocations remain as they are The reset reclaims all allocations, so dram_used and sram_used become 0 Only what is newly allocated
Error counters, last_hang Still readable Still readable Counters keep their pre-reset values, and timeouts or firmware errors that occur during the reset are added. last_hang is also not cleared, but batch_seq, exec_index, resq_tail, and verdict are replaced by the record of that reset if the firmware does not respond during the reset, and rc, cmd_idx, and opcode are replaced if the firmware returns an error for the reset command
health faulted faulted until the reset finishes, if the device was FATAL If the reset succeeded, FATAL is cleared, so ok or degraded depending on the counters. If it failed, faulted
peer_sram The entries for this device, and the entries on other devices that point to this device, are unknown Same as during FATAL Values return once the path is calculated again
dmabuf_exports Still readable Still readable The window reservations remain, and windows that were live become disabled
  • "Reset" in the table means a reset requested through the SDK and an automatic reset. A host-initiated reset (such as PCIe error recovery) also stops polling and reads the values again when it finishes, but it does not clear the FATAL mark.
  • If a reset fails, health remains faulted.
  • For the reset procedure and how to check the result, see FATAL and recovery.

See also