Telemetry¶
ha.ko exposes the state of each device as read-only files under /sys/class/ha/haN/. The firmware version and chip serial number, temperature, power, and utilization, memory usage, job counts and error counters, and the health verdict (health) are all there, and temperature, voltage, current, and power are also exposed through Linux hwmon. This page explains where the values come from and when they are refreshed, what unknown means, how health is determined, and how to connect to a monitoring system. The output format of each node is in sysfs nodes.
Design principles¶
Telemetry follows the principles below so that it can be read even at the moment a device runs into a problem.
| Principle | What it means in operation |
|---|---|
| Does not open the device node | It only reads files and never opens /dev/haN. A monitoring agent does not compete with the workload for the device, and the values can be read even when there is no workload |
| No privileges required | All telemetry nodes are 0444. A regular user can read them with cat |
| Separate from the compute path | Temperature, power, and utilization are fetched through a separate monitoring channel of the firmware, and most of the other values are calculated from state that the driver keeps in host memory. They do not go through the command queue and locks that compute work uses, so they respond even while compute work is stopped |
| Read-only | Telemetry nodes can only be read. bypass_range_check in the same directory is not telemetry but a debug setting |
| fail-closed | If a value cannot be trusted, unknown is reported. A value past its validity period is not reported again, and gaps are not filled with 0 |
1 2 3 | |
Collectors must handle unknown
unknown is not an error but a normal output. A parser that expects only numbers fails when the firmware is not yet ready or while the device is being reset, that is, at the moments when status information is needed most. In the same situations, hwmon files return a read error (No data available) instead of unknown.
Telemetry node types and refresh methods¶
Telemetry nodes fall into five types, depending on how the value is obtained. The Refresh column in sysfs nodes follows this classification.
| Type | Nodes | Where the value comes from | Refresh |
|---|---|---|---|
| Fixed | hw_version, dram_size, sram_size, tw_window_size, help |
PCI configuration space, the driver's memory configuration, and the help text built into the driver | Never changes |
| Cache | fw_version, serial |
Values the firmware exposes | Read once when the firmware is ready. Read again when the device is reset. |
| Monitoring | temp, temp_avg, temp_blocks, power_mw, rails, util, hwmon |
Firmware monitoring channel | Every ha_sysmon_poll_ms (200 ms by default) |
| On read | dram_used, dram_free, sram_used, sram_free, inflight, stream_count, devcmd_timeouts, devcmd_fw_errors, hdma_timeouts, abi_prefetch_stats, last_hang, health, peer_map, peer_route, peer_sram, dmabuf_exports |
State kept by the driver | Calculated on every read |
| Device | fw_update_progress |
The device | Read directly from the device on every read |
Cache: firmware version and chip serial number¶
fw_version and serial are values that the firmware exposes in device registers.
- If the firmware is not yet ready when the driver probes the device, both values start as
unknown. This is so that a half-written value from a firmware that is still booting is not reported as a version. The driver checks again whether the firmware is ready each time one of these nodes is read and each time the poller runs. When the firmware is ready, the driver reads and fills in the values, and logsfirmware booted after probe gave up; CSR-derived telemetry refreshedto the kernel log. If the firmware is still not ready, the driver logstelemetry stays "unknown": device is not ready(the output rate is limited). - When the device is reset, the driver checks firmware readiness at the end of the reset and reads the values again. If it finds that the firmware has not returned to the ready state, it changes the values to
unknown. - The stored values are the values from the time they were read. A version shown in
fw_versiondoes not mean that the firmware is responding now. Judge the current state byhealthand the counters. fw_versiondoes not change immediately when you update the firmware. It changes only after the card boots with the new firmware and the driver reads the value again. See Firmware update.
Periodic monitoring: temperature, power, utilization¶
Every ha_sysmon_poll_ms (200 ms by default, 5 times per second), the driver requests temperature, power, and utilization in turn from the firmware's monitoring channel and stores the results in a cache. sysfs nodes and hwmon read this cache. In the kernel log, the component that makes these requests, together with the channel, is called sys_mon.
- Reading a node does not access the device. If you read more often than the polling interval, you receive the same sample again.
- A sample is valid for 5 seconds. If a request fails, the cache is left as it is, so the values do not drop out as long as the next sample arrives within 5 seconds. When the last good sample is older than 5 seconds, the value becomes
unknown. - If a power sample contains a physically impossible voltage or current, that sample is discarded and the previous sample is kept.
- A single request waits up to 50 ms for a response. It does not use the same command queue as compute work, so periodic monitoring continues even when compute work has stopped. Monitoring is handled on a path separate from compute. If temperature, power, and utilization keep updating while a kernel run is not answering, monitoring is still working. These values become
unknownonly when monitoring or the connection to the device has stopped responding on its own. ha_sysmon_poll_mscan be changed at run time and takes effect starting with the next scheduled poll. A value outside 50–2500 ms is clamped to that range, and0is not off but the fastest interval (50 ms). The longer the interval, the fewer failed requests it takes to exceed the 5-second validity period and turn the valueunknown. See Parameter reference.
Firmware clock control¶
When a compute-group temperature or the power crosses a threshold, the firmware halves the core clock of each compute group. This is the only thermal response in this firmware. The firmware does not reset the card when the temperature rises further.
| Clock is halved | Clock is restored | |
|---|---|---|
| Hottest compute-group temperature | 90°C | 80°C |
| Sum of rail power | 350 W | 300 W |
Crossing either threshold halves the clock. The clock returns to its original rate only after both inputs are back under the restore values. If the temperature cannot be read, the firmware does not restore the clock. If the power cannot be read for a while, the firmware keeps the current clock decision for five tries and then judges from temperature alone.
This state does not appear in sysfs or hyperaccel-smi. temp can rise and throughput can drop while health stays ok, and this is not a hang.
The value to compare with the threshold is not the temp shown on screen. Clock control uses the hottest compute-group sensor, not the hottest of all block sensors, and the power it uses is the same rail sum as power_mw. temp is approximate, so do not compare it directly with the limits in the datasheet. 90°C and 350 W are the values the firmware applies to that sensor and to the rail sum.
Values calculated on read¶
Memory usage, job counts, error counters, last_hang, health, and device-to-device path information are calculated on every read from state that the driver keeps in host memory. They do not access the device, so they are always current values, and they respond even while compute work is stopped.
The error counters (devcmd_timeouts, devcmd_fw_errors, hdma_timeouts), abi_prefetch_stats, and last_hang are records that have accumulated since the driver probed the device. They start counting over only when the driver probes the device again (module reload, unbind followed by bind, ha-hotreset, host reboot).
Values read directly from the device¶
Each time the fw_update_progress node is read, the driver reads the firmware update progress directly from the device. It does not go through the command queue, so it can be read while an update is in progress. Do not include it in continuous collection; read it only while you are watching a firmware update. Do not read it on an unresponsive device (a device whose health is faulted or whose monitoring collection has stopped).
Conditions under which unknown appears¶
unknown means that there is no value, or that the value cannot be trusted at the moment. The conditions differ for each telemetry node.
| Node | Condition | When it clears |
|---|---|---|
fw_version, serial |
The firmware is not yet ready (it is still booting, or booting failed) | When the firmware becomes ready, the driver detects it and fills in the value |
fw_version |
The firmware version register reads as 0, or with all bits set to 1 | When the driver checks again and reads a valid value |
serial |
The firmware does not expose the chip serial number, or no number is recorded on the chip | It does not change for that combination of card and firmware |
| All monitoring nodes | The driver has just probed the device, so there is no first sample yet | When the first sample arrives |
| All monitoring nodes | The last good sample is older than 5 seconds (causes below) | When samples arrive again |
util |
The firmware has not yet finished its first measurement interval | When the first interval ends |
util |
The firmware's measurement has not progressed for more than 5 seconds | When measurement progresses again |
util |
A value above 100% was received for at least one IP | When the next valid sample arrives |
power_mw |
No rail is being measured (rails is none in this case) |
- |
fw_update_progress |
The firmware is not ready, does not support progress reporting, or the value could not be read from the device | - |
peer_sram |
The LPU is being reset, is not ready, is FATAL, or is not responding and monitoring has stopped | When the state returns and the path is calculated again |
peer_route |
The PCIe path could not be determined | It is determined again when the driver probes the device again |
The last sample of a monitoring node becomes older than 5 seconds for the following reasons.
- The device is being reset. Polling is stopped so that the device is not read during a reset.
- The device is FATAL (health verdict). To avoid continuing to read a stopped device, polling requests are not sent until FATAL is cleared.
- The monitoring component has stopped. See Monitoring suspension below.
- The firmware has not opened the monitoring channel, or the version of the monitoring format that the firmware sends is lower than the minimum version the driver accepts.
- The firmware does not respond to requests, or sends invalid responses.
- The driver could not start monitoring.
sys_mon telemetry unavailableis logged to the kernel log, and in this case no hwmon device is created either.
For health, the memory nodes, abi_prefetch_stats, peer_map, and peer_route, the entire node becomes unknown only when the driver cannot read its internal state, and this does not happen after device probing has completed.
Monitoring suspension¶
If all three requests (temperature, power, utilization) fail in a single monitoring round ha_sysmon_fail_limit times in a row (15 by default), the driver stops monitoring by itself. With the default settings (15 times, 200 ms interval), it takes about 3 seconds or more from the moment responses stop until monitoring stops, and increasing ha_sysmon_poll_ms delays it by that much.
Continuing to read an unresponsive device can cause a PCIe Completion Timeout. Some servers in which platform firmware handles PCIe errors first (firmware-first) treat this as an unrecoverable error and can reset the entire host. Monitoring is stopped so that the host is not lost because of a single unresponsive card.
When monitoring stops, an error like the following is logged to the kernel log (the numbers vary from case to case).
1 | |
After monitoring stops, the following applies.
- Monitoring nodes and hwmon values become
unknown(a read error for hwmon) 5 seconds after the last good sample. - For a card whose monitoring channel has been opened at least once, the driver does not poll that card until it is reset, and does not re-check
fw_versionandserialeven if they areunknown. Thepeer_sramlines related to that card also becomeunknown. - Monitoring does not restart by itself. When you reset the device and it responds again, the driver restarts polling and logs
sys_mon: resuming telemetry polling. When the driver probes the device again (for example with ha-hotreset), monitoring also starts over from the beginning. - A card whose monitoring channel has never been opened (a card that boots late) is an exception. If you read
fw_versionorserialafter the firmware is ready, the driver confirms readiness and also restarts polling. - On a card whose firmware does not open the monitoring channel, all requests fail as well, so monitoring stops in the same way and the same log is written.
ha_sysmon_fail_limit can be changed at run time, and with 0 monitoring never stops. However, this means that an unresponsive device keeps being read, so do not set it to 0 on firmware-first platforms. See Parameter reference.
health verdict¶
health is a one-word verdict that the driver derives from its internal state. In a health check, look at this value before interpreting the counters directly. The verdicts are applied in order from the top of the table.
| Value | Condition | When it clears |
|---|---|---|
faulted |
The device is in the FATAL state, or the last reset (an SDK reset or the automatic reset) failed | FATAL clears when a reset requested through the SDK, or an automatic reset, finishes. A reset failure clears when a later reset succeeds, or when the driver confirms that the device is ready again. Both clear when the driver probes the device again |
degraded |
At least one of devcmd_timeouts, devcmd_fw_errors, hdma_timeouts is not 0 |
When the driver probes the device again |
ok |
None of the conditions above applies | - |
unknown |
The driver cannot read the state | It does not occur after device probing has completed |
If, during work, the device does not respond within the driver's ceiling time (ha_devcmd_ceiling_ms, 12 seconds by default), or the firmware reports a device-level error rather than the failure of a single command, the driver marks that device FATAL. If watchdog preemption (ha_wd_preempt) is enabled, the device also becomes FATAL when the watchdog cuts off a stalled wait (when it cuts off a wait that has no ceiling, only that request fails and the device does not become FATAL). FATAL is a state of the device, so restarting the process does not clear it, and work on that device is rejected until it is reset. For the causes and the recovery procedure, see FATAL and recovery.
Keep the following in mind when you read the verdict.
degradeddoes not mean that something is failing right now. The counters only increase after the device is probed, so even a single past error leaves the verdict atdegradeduntil the driver probes the device again. Readdegradedas "check the counters", and judge whether the problem is continuing by whether the counters are increasing.- A device that became FATAL because of a timeout or a firmware error is shown as
degradedeven after recovery. This is because the cause remains in the counters. - Application errors also produce
degraded.devcmd_fw_errorsalso counts cases where the firmware responded with an error to an invalid request sent by an application. Uselast_hangand the kernel log to tell whether it is a hardware fault. - If a reset fails,
healthstaysfaulted. This holds for a reset requested through the SDK and for an automatic reset alike. The driver still accepts work again, but it does not attempt an automatic reset on that device again until a reset succeeds or the card becomes ready again. healthdoes not immediately show a stall that is in progress. Even if the device stops, it does not change until the ceiling time passes and a timeout is recorded. A wait that an application requested to be without a ceiling is not recorded as a timeout, sohealthkeeps not changing while that wait is stalled. To detect a stall in progress, see Hang detection and the watchdog.healthdoes not factor firmware readiness or the state of monitoring nodes such as temperature and power into the verdict. A card whose firmware is not ready can also beok, so also check thatfw_versionis notunknownto know whether the device is usable. Monitor overheating and telemetry gaps separately.
hwmon integration¶
The driver also exposes the temperature, voltage, current, and power collected by the poller as a Linux hwmon device. Tools that follow the hwmon convention (such as sensors from lm-sensors and the hwmon collector of Prometheus node_exporter) read these values without any additional configuration.
- Each device has one hwmon device, and its
nameisha. It is located at/sys/class/ha/haN/device/hwmon/hwmonM/and is the same device as/sys/class/hwmon/hwmonM.Mis a number assigned by the kernel and is unrelated toN. - Values are read from the same cache as the sysfs poller nodes.
- Units follow the hwmon convention. Temperature is in millidegrees (m°C), voltage in mV, current in mA, and power in µW. Voltage and current use the same units as
rails, while temperature and power differ from the sysfs nodes (°C, mW). - When a value cannot be trusted, a read error (
ENODATA) is returned instead ofunknown. The conditions are the same as for the poller nodes. - The kernel must support hwmon (
CONFIG_HWMON). On a kernel without support, no hwmon device is created, and if registration fails on a kernel with support,hwmon register failedis logged to the kernel log. In either case, the sysfs nodes keep working as before.
1 2 | |
1 2 3 | |
For the list of channels and labels, see sysfs nodes.
Monitoring integration¶
Collection interval and cost¶
- Reading most nodes does not access the device. Monitoring nodes read the cache, and the others read the driver's host-side state. The exceptions are
fw_update_progress(always read from the device) andfw_versionandserialwhilefw_versionisunknown(firmware readiness is checked again on each read). - Poller nodes change every 200 ms by default, so reading more often than that yields no new values. You can use a typical monitoring collection interval (several seconds to several tens of seconds) as it is.
- Exclude
fw_update_progress,help, andbypass_range_checkfrom continuous collection. - When a device is detached from the driver (module removal, unbind, while ha-hotreset is running),
/sys/class/ha/haN/also disappears. Collectors must handle the case in which the directory disappears.
Alert criteria¶
| Signal | Meaning | Recommended response |
|---|---|---|
health is faulted |
The device is FATAL or the last reset failed | Alert immediately. Follow FATAL and recovery |
| An error counter increases | An operation exceeded the ceiling time, or a firmware error occurred | Alert on the increase. Check last_hang and the kernel log |
health is degraded |
At least one error has occurred since the device was probed | Use it for status display only. Judge whether the problem is continuing by counter increases |
A monitoring node stays unknown |
Causes include a reset, FATAL, monitoring suspension, or firmware not ready (Conditions under which unknown appears) | Check health, fw_version, and the kernel log |
inflight is not 0 but the workload is not making progress |
Work may have stopped | Follow Hang detection and the watchdog |
Health check script¶
A health check that reads only sysfs does not open the device node, so it can run alongside the workload, and it keeps working while the workload is stopped. An example script that checks whether the driver is loaded, the number of cards, and each card's health and fw_version and reports the result through its exit code, and how to run it periodically, are in Building a card health check.
hyperaccel-smi¶
hyperaccel-smi is a command that reads the sysfs nodes and shows them as a table. It needs neither root privileges nor a device node, and it can print JSON with the --json option. For detailed usage, see hyperaccel-smi.
Using Prometheus¶
Temperature, voltage, current, and power are collected from the hwmon device by the hwmon collector of node_exporter. Values that are not in hwmon, such as health and the error counters, can be exported with the textfile collector. Example scripts and alert rule examples are in Collecting telemetry with Prometheus.
Counters restart from 0 when the driver probes the device again, so base alert rules on the increase (such as increase()) rather than on the value itself.
Telemetry during a reset and during FATAL¶
| Node | During FATAL | While a reset is in progress | After the reset finishes |
|---|---|---|---|
| Monitoring nodes, hwmon | No monitoring requests are sent, so unknown 5 seconds after the last sample |
Monitoring is stopped, so unknown 5 seconds after the last sample |
Filled in again soon after the device responds |
fw_version, serial |
The stored values | The stored values | Read again if the driver confirms firmware readiness at the end of the reset; unknown if it confirms that the firmware is not ready |
| Memory usage | Allocations remain as they are | The reset reclaims all allocations, so dram_used and sram_used become 0 |
Only what is newly allocated |
Error counters, last_hang |
Still readable | Still readable | Counters keep their pre-reset values, and timeouts or firmware errors that occur during the reset are added. last_hang is also not cleared, but batch_seq, exec_index, resq_tail, and verdict are replaced by the record of that reset if the firmware does not respond during the reset, and rc, cmd_idx, and opcode are replaced if the firmware returns an error for the reset command |
health |
faulted |
faulted until the reset finishes, if the device was FATAL |
If the reset succeeded, FATAL is cleared, so ok or degraded depending on the counters. If it failed, faulted |
peer_sram |
The entries for this device, and the entries on other devices that point to this device, are unknown |
Same as during FATAL | Values return once the path is calculated again |
dmabuf_exports |
Still readable | Still readable | The window reservations remain, and windows that were live become disabled |
- "Reset" in the table means a reset requested through the SDK and an automatic reset. A host-initiated reset (such as PCIe error recovery) also stops polling and reads the values again when it finishes, but it does not clear the FATAL mark.
- If a reset fails,
healthremainsfaulted. - For the reset procedure and how to check the result, see FATAL and recovery.
See also¶
- sysfs nodes — output format of each node and the hwmon channels
- Hang detection and the watchdog — detecting a stall in progress and interpreting
last_hang - FATAL and recovery — recovering a
faulteddevice - Multiple devices — interpreting
peer_map,peer_route, andpeer_sram - hyperaccel-smi
- Parameter reference —
ha_sysmon_poll_ms,ha_sysmon_fail_limit - Kernel log messages