Skip to content

Operations guide

This page describes the operational features that ha.ko provides and the tools that are installed with the driver. Feature pages cover how a feature works and what the operator should do, and tool pages cover the command options, output, and exit codes. The values that change how a feature behaves are in Kernel module parameters.


Features

Feature What it does Related tools and nodes
Telemetry Reads temperature, power, utilization, memory usage, and device state. Integrates with monitoring tools through hwmon hyperaccel-smi, health, hwmon
Hang detection and the watchdog Uses the watchdog to find devices that exceed the response wait ceiling or have stopped, and leaves a diagnostic record last_hang, devcmd_timeouts
FATAL and recovery Marks a device that cannot finish its work as FATAL to block new work from running, and recovers it with a reset ha-hotreset, ha_auto_reset
Firmware update The firmware running on the card rewrites the boot flash with a new image ha-fw-update, fw_update_progress
Multiple devices Covers the mapping between card numbers and PCI addresses, the data paths between cards, and the trust boundary peer_map, peer_route, peer_sram

Tools

Tool What it does Required privileges
hyperaccel-smi Shows card state as a table, a detailed view, or JSON Regular user
ha-hotreset Performs a PCIe hot reset of a single card root
ha-fw-update Writes new firmware to the card's boot flash and boots it root
ha-led Blinks the board LED of a single card so that you can find that card in the chassis root
ha-wipe Resets a card and fills its DRAM with zeros before the card goes to the next user root

Reference

  • sysfs nodes: the format and unit of each node, and the conditions under which a value becomes unknown
  • Kernel log messages: the meaning of each message the driver logs and what to do about it

When a problem occurs

If you are starting from a symptom, see Troubleshooting. Step-by-step procedures you can follow are in Collecting diagnostic information for a problem and Recovering an unresponsive card.


See also