Operations guide¶
This page describes the operational features that ha.ko provides and the tools that are installed with the driver. Feature pages cover how a feature works and what the operator should do, and tool pages cover the command options, output, and exit codes. The values that change how a feature behaves are in Kernel module parameters.
Features¶
| Feature | What it does | Related tools and nodes |
|---|---|---|
| Telemetry | Reads temperature, power, utilization, memory usage, and device state. Integrates with monitoring tools through hwmon | hyperaccel-smi, health, hwmon |
| Hang detection and the watchdog | Uses the watchdog to find devices that exceed the response wait ceiling or have stopped, and leaves a diagnostic record | last_hang, devcmd_timeouts |
| FATAL and recovery | Marks a device that cannot finish its work as FATAL to block new work from running, and recovers it with a reset | ha-hotreset, ha_auto_reset |
| Firmware update | The firmware running on the card rewrites the boot flash with a new image | ha-fw-update, fw_update_progress |
| Multiple devices | Covers the mapping between card numbers and PCI addresses, the data paths between cards, and the trust boundary | peer_map, peer_route, peer_sram |
Tools¶
| Tool | What it does | Required privileges |
|---|---|---|
| hyperaccel-smi | Shows card state as a table, a detailed view, or JSON | Regular user |
| ha-hotreset | Performs a PCIe hot reset of a single card | root |
| ha-fw-update | Writes new firmware to the card's boot flash and boots it | root |
| ha-led | Blinks the board LED of a single card so that you can find that card in the chassis | root |
| ha-wipe | Resets a card and fills its DRAM with zeros before the card goes to the next user | root |
Reference¶
- sysfs nodes: the format and unit of each node, and the conditions under which a value becomes
unknown - Kernel log messages: the meaning of each message the driver logs and what to do about it
When a problem occurs¶
If you are starting from a symptom, see Troubleshooting. Step-by-step procedures you can follow are in Collecting diagnostic information for a problem and Recovering an unresponsive card.