FATAL and recovery¶
When a device falls into a state where it cannot finish jobs, the driver marks it FATAL and accepts no new jobs until it is recovered by a reset. This page covers how FATAL appears to the operator; when and in what order to use the recovery methods (the SDK device reset, automatic reset, ha-hotreset, and a power cycle); the PCIe configurations in which an SBR (Secondary Bus Reset) can be issued; resets issued by the host and PCIe errors; and what happens when a card is detached from the driver. The tool's options are in ha-hotreset, and the definitions of the module parameters are in Parameter reference.
The FATAL state¶
FATAL is a state of the device itself, not of a process or of a device node that a process opened. It means that the driver has judged that no more jobs can be entrusted to that device, and it is not cleared until the device is reset. Other devices on the same host are not affected.
When a device becomes FATAL¶
If one of the following occurs while an application job is running, the driver changes the device to FATAL.
- The firmware did not respond within the driver's response wait ceiling (
ha_devcmd_ceiling_ms, default 12000 ms). - The firmware reported an error that requires a reset.
- The DMA engine of the card did not respond (
ll: no response,ring write: no response), or did not stop after a transfer timeout or abort (HDMA engine at …; quarantining …). The messages are in Data transfer (DMA). - Watchdog preemption (
ha_wd_preempt=1) is on, and the watchdog cut off a wait of an application job. This does not apply when a wait that was requested to be without a ceiling is cut off (see Hang detection and the watchdog)
What FATAL looks like¶
| Where | What you see |
|---|---|
/sys/class/ha/haN/health |
faulted |
hyperaccel-smi |
The Health column is FAULTED |
| Application | SDK calls fail with an error saying the device cannot be recovered. This stays the same until a reset, even if you reopen the device or restart the process |
| Waiting jobs | The moment the device changes to FATAL, all jobs waiting on that device are cancelled |
| Monitoring telemetry nodes | sysfs nodes that are refreshed periodically, such as temperature, power, and utilization, change to unknown within a few seconds after FATAL and stay unknown until a reset |
dmesg |
The cause of FATAL is left. If the firmware did not respond, the Devcmd no response after <ms> ms (device hang) line and the hang (ceiling timeout): ... and hang dump (ceiling timeout): ... lines after it are left; if the firmware reported an error, the Firmware error: resq[...] line is left; if the watchdog cut off a wait, the watchdog: ... wait stalled line is left; if a DMA engine did not respond or did not stop, the ll: no response … or HDMA engine at … line is left |
The meanings of the messages are in Hang detection and the watchdog and Kernel log messages, and a script that collects diagnostic information is in Collecting diagnostic information for a problem.
Recovery order¶
- Save the diagnostic information first. A reset changes the state that explains the failure. If the firmware does not respond during a reset, that record can overwrite
last_hang, andha-hotresetand a module reload restart the driver's RAS counters from 0 and resetlast_hang. A script that saves it is in Collecting diagnostic information for a problem. - Stop the application jobs that use that device.
- Reset the device. The application requests an SDK device reset, or, if automatic reset is on, the driver resets by itself.
- Check the result. Look at
healthanddmesg. See Checking the reset result. - If the reset does not recover the device, hot-reset the card with
ha-hotreset, and if that does not recover it either, power-cycle the host.
A step-by-step procedure from checking the symptoms through resetting one card with ha-hotreset and confirming the recovery is in Recovering an unresponsive card.
Recovery methods¶
In the table below, a handle means the device node (/dev/haN) that an application opened.
| Method | Run by | Scope | Open handles | What it does |
|---|---|---|---|---|
| SDK device reset | Application | One device | Proceeds even if handles are open. Only the handle that requested the reset remains usable | Re-initializes the firmware first (stage 1) and, if that fails, automatically issues an SBR (stage 2) |
| Automatic reset | Driver | One device | Proceeds even if handles are open. All handles become invalid | Same as the SDK device reset |
ha-hotreset |
Root on the host | One card | All must be closed | Removes the card from the driver, issues an SBR, and has the driver probe it again |
| Power cycle | Operator | The whole host | All are disconnected | Restarts the host and all cards |
Module reload and unbind are not recovery methods
If you reload the module, or unbind the driver and bind it again, the driver state is created anew but the card is not reset. A card whose firmware has stopped stays as it was, so sending jobs can make it FATAL again.
SDK device reset¶
An application resets a device with the SDK's device reset function (Reset API). It can be requested even in the FATAL state. The driver resets in the following order.
- Stops accepting new jobs and ends all jobs in progress or waiting with an error.
- Reclaims the device state of all processes that use that device. The targets are device memory allocations, memory mappings, inter-process memory sharing handles, and profiling sessions.
- Asks the firmware to re-initialize.
- If re-initialization fails or does not finish within the response wait ceiling, issues an SBR at the parent PCIe port. An SBR is issued only in configurations in which an SBR can be issued.
- Checks that the firmware reports ready, and reopens job intake. Job intake is reopened even if the reset fails.
The reset request does not return until this process finishes, and it can take tens of seconds if it goes as far as an SBR. Other cards on the same host are not reset.
The re-initialization in step 3 is not a boot that reads the flash again. The firmware that is already running prepares the cores again, clears the commands and execution state loaded on the cores, and then checks whether computation stops within about 50 ms. The image stored in flash does not change. The firmware does not erase device memory. Releasing allocations is the driver's work in step 2.
If an engine does not stop within that time, or if profiling does not finish, the firmware returns not ready (rc=0x1) instead of success and the driver goes on to an SBR. The reset also goes on to an SBR when command processing has stopped, so re-initialization never runs and the response wait ceiling expires. That is different from the firmware returning a timeout as rc=0x7. See Response wait ceiling.
Even when re-initialization fails, the firmware returns to a state in which it can accept commands again. That means the device can answer a command, not that the device is healthy. After a reset, a visible fw_version is not enough: also check dmesg for whether an SBR was issued or refused, as described in Checking the reset result.
The application after a reset¶
- Device memory and mappings created before the reset are invalid, so they must be created anew. Accessing a device memory mapping created before the reset can cause SIGBUS.
- Every handle that was opened before the reset finished becomes invalid. Later SDK calls fail, so that process must close and reopen the device. Usually the application is restarted.
Request resets from only one place per device
A reset can be requested even with an invalidated handle, but that reset invalidates the process's handles again. If several applications each reset whenever they receive an error, they keep cutting each other off in turn. Designate one process per device to be responsible for resets, and have the others report the error and reopen the device.
Checking the reset result¶
- If the reset succeeds,
healthleavesfaulted. The error that led to FATAL remains in the RAS counters (devcmd_timeouts,devcmd_fw_errors,hdma_timeouts), so it can showdegradedrather thanok. These counters are not cleared by an SDK device reset that did not go on to an SBR, and they start counting again from 0 when the driver probes the card again (ha-hotreset, unbind then rebind, module reload). - If the firmware re-initialization failed and the reset went on to an SBR,
issuing SBR via parent <BDF>is left indmesg. - If the reset fails,
healthstaysfaulted. The driver reopens job intake, but no further automatic reset is attempted on that card.
If the reset fails¶
When a reset fails, the last line the reset procedure leaves in dmesg is device reset failed (<err>, firmware re-init <err>): .... The end of that line tells you what to do next: whether to reset again, run ha-hotreset --bdf <BDF> on the host, or power-cycle the host, and what to do if that does not work either. The lines before it show which stage failed.
An automatic reset goes through the same procedure, so the same messages are left, followed by one more line, auto-TDR: reset failed (<err>); .... However, the last row of the table is left only by an SDK reset, and in the same case an automatic reset leaves auto-TDR: skipped, ... (Automatic reset).
dmesg |
State | What to do |
|---|---|---|
A line that contains leaves Surprise Down unmasked; refusing SBR |
The firmware re-initialization failed, and the driver issued no SBR because the parent port does not mask Surprise Down. The card stays stopped | Run sudo ha-hotreset --bdf <BDF> on the host. ha-hotreset masks the path and resets the card |
Any other line that contains refusing SBR, or No parent bridge, SBR not supported |
The firmware re-initialization failed, and no SBR was issued because this PCIe configuration does not allow one. The card stays stopped | ha-hotreset also refuses because of the same condition. Power-cycle the host |
device did not answer after SBR |
The card does not respond after the SBR | Run sudo ha-hotreset --bdf <BDF> on the host. If that does not recover it, power-cycle the host |
device came back but firmware did not report ready |
The card came back but the firmware did not report ready. The driver accepts no jobs until the firmware is ready | Read the sysfs fw_version again after a while. When the firmware is ready, a version appears instead of unknown. If it stays unknown, run ha-hotreset, and if that does not help either, power-cycle |
removed during SBR or removed after SBR while the reset was still running |
The operating system removed the card during the reset. A typical case is a hot-plug slot that treats the link drop of the SBR as a card removal | When the card is probed again and /dev/haN reappears, the application reopens the device. If it does not reappear, power-cycle |
reset refused: last fw_update unanswered |
The reset was not performed because there is a firmware update that received no response | Follow Firmware update |
Automatic reset¶
With the module parameter ha_auto_reset=1, the driver performs the same reset as the SDK device reset by itself when it detects FATAL. The default is 0 (off), it applies to all cards, and it can be changed at run time. To keep it on from the next load onward, see Turning on automatic recovery.
1 | |
- Automatic reset starts the moment the device changes to FATAL. A device that was already FATAL before you turned the parameter on is not reset automatically, so reset it yourself.
- When the reset starts, a kernel log such as
auto-TDR: the device went FATAL; resetting itis left indmesg. - If the reset fails,
auto-TDR: reset failed (<err>); auto-TDR will not run again on this card until a reset succeeds or the card reports ready againis left, and no further automatic reset is attempted on that device. This is to avoid repeating a failed reset. The same holds when a reset requested through the SDK fails.healthisfaultedat this point. For what to do next, follow If the reset fails. - A halted automatic reset resumes in the following cases.
- When a reset succeeds by another method
- When the driver probes the card again (
ha-hotreset, unbind then rebind, module reload) - When the driver confirms that firmware that had not reported ready after a reset reported ready later - If a firmware update is in progress or there is an update that received no response, the automatic reset is skipped and
auto-TDR: skipped, ...is left. The device stays FATAL. - If the operating system removes the card during a reset, it is not treated as a failure. Whether the recovery succeeds is decided by whether the card is probed again.
Turn on ha_auto_reset in an environment where applications do not request resets themselves and are operated to restart when an error occurs. If there is a separate supervisor process responsible for resets, you can leave it off.
Using it with watchdog preemption
With ha_wd_preempt=1, the watchdog cuts off a stalled wait, and if that wait belonged to an application job, the device becomes FATAL. If automatic reset is off, that device stays FATAL until someone requests a reset, so turn on both parameters together. See Hang detection and the watchdog.
ha-hotreset¶
ha-hotreset is a per-card hot reset tool that you run as root on the host. It removes the card from the driver, issues an SBR at the parent bridge, and rescans the bus so that the driver probes the card again. The ha module stays loaded and other cards are left untouched. A card that received the SBR boots again from the firmware in the flash.
Use it in the following cases.
- When an SDK device reset or an automatic reset did not recover the device
- When resetting a card while no application is using the device
- When booting the new firmware after a firmware update (Firmware update)
Before resetting, ha-hotreset checks the following. If any one of them is not satisfied, it refuses without touching anything.
- No process may have the card's
/dev/haNopen. If one does, it prints the PID and refuses. You can override this with--force, but that process's later SDK calls fail, so that process must close and reopen the device. Accessing a device memory mapping in the meantime can cause SIGBUS. - The parent port must be in a configuration in which an SBR can be issued, or it must be possible to issue the SBR by masking the path. This cannot be overridden with
--force. - There must be no device other than that card under the parent bridge, because the SBR would reset that device too. This cannot be overridden with
--force.
1 2 3 4 5 | |
Hot reset OK: <BDF>means that the card was enumerated on PCI again and, if the card had a/dev/haNbefore the reset, that node reappeared, or, if it had none, that thehadriver bound the card. Check the firmware state withhyperaccel-smiorhealth.- The card usually reappears with the same
/dev/haNnumber. Identify the card by its PCI address (the Bus-Id column ofhyperaccel-smi) rather than by the number. - The driver probes the card anew, so the RAS counters and
last_hangare reset, and if the card is healthy,healthisok.
Options, pre-flight checks, the order of operations, and exit codes are in ha-hotreset.
Power cycle¶
If the methods above do not recover the device, turn the host power off and on again. This applies in the following cases.
- The firmware re-initialization failed and the card is shown as
NOT resettablebyha-hotreset --list - The card does not respond, or is not probed again, even after
ha-hotreset
Do not turn the power off while a firmware update is in progress. See Firmware update.
Configurations in which an SBR can be issued¶
An SBR briefly drops the link between the parent port and the card. If the parent port reports this link drop as a Surprise Down error, then on servers where platform firmware handles PCIe errors, the error can lead to a restart of the entire host. So the driver issues an SBR only when one of the following holds.
- The card's parent port is a root port.
- The Surprise Down error (bit 5) is masked in the parent port's AER Uncorrectable Error Mask.
If the parent port's PCIe or AER settings cannot be read, or there is no parent bridge, no SBR is issued. The driver does not change this mask. A reset that completes with the firmware re-initialization alone proceeds regardless of these conditions.
The driver does not change the mask, but ha-hotreset blocks error reporting along the path from the parent port up to the root port itself during the reset (path masking), so it can reset a card that meets neither condition. It restores the masking when the card responds again. It refuses if the registers on the path cannot be read or the masking is not applied. See Path masking.
Check the verdict for each card in advance.
1 | |
A card shown as NOT resettable cannot be reset with ha-hotreset, so it needs a power cycle if the firmware re-initialization fails. A card shown as the path is masked for the reset does not meet the two conditions above, so the SDK device reset and the automatic reset cannot move on to an SBR when the firmware re-initialization fails. Reset such a card with ha-hotreset. You can also see the parent port's mask in the UEMsk: line of the output of sudo lspci -vvv -s <parent BDF> as SDES+ (masked) or SDES- (not masked).
A host that uses path masking for the first time
A masked error is not reported, so it does not reach the platform firmware. However, on a server whose platform firmware detects the link drop some other way, such as by reading the ports' error status registers directly, masking may not prevent a host restart. On a host where you use it for the first time, check the path with sudo ha-hotreset --check-mask --bdf <BDF>, and reset a single card first at a time when no jobs are running. To have the tool refuse instead of masking, give it --no-switch-link-drop.
Resets issued by the host and PCIe errors¶
A device can also be reset from outside the driver. Resets issued through the kernel's PCI reset function and recovery from PCIe errors (AER, DPC) are such cases. The driver receives a notification, stops jobs and reclaims the device state in the same way as an SDK device reset, and when the reset ends, it checks that the firmware is ready and reopens job intake.
- When the driver reclaims the device state, all handles that were open become invalid. There is no process that requested the reset, so there is no exception. Applications must reopen the device.
- For a kernel PCI reset,
reset_prepare: quiescing for a host-initiated resetandreset_done: reopening the gate (boot ok)are left indmesg. If the firmware boot could not be confirmed after the reset,boot unconfirmedis left instead ofboot ok. - For PCIe error recovery,
PCIe error detected (state=<N>): recoverableandPCIe error_resume: recovery completeare left. - If
healthisfaultedafter the recovery or the device does not work normally, reset the device with the SDK orha-hotreset. - If the reset stage is not reached within 30 seconds after PCIe error recovery starts, the driver reopens job intake without re-initializing the device. In this case,
PCIe recovery never reached slot_reset within 30000 msis left indmesg. If the device does not work normally afterward, reset it with the SDK. - If the firmware does not come back during the recovery,
slot_reset: firmware did not come back (<err>)is left. In this case, follow the Recovery order.
Do not reset the card with the kernel's PCI reset
Do not use the kernel's PCI reset function directly, for example by writing to /sys/bus/pci/devices/<BDF>/reset. The kernel chooses the reset method by itself and tries FLR (Function Level Reset) first. FLR is not supported on this card, and the bus reset the kernel issues next does not check the SBR conditions above. Reset a device with the SDK, and reset a card from the host with ha-hotreset.
AER reporting and ha_aer_enable¶
The module parameter ha_aer_enable (default 0) decides whether the driver turns on PCIe AER error reporting for the card. It can be set only when the module is loaded.
- Why the default is off: The kernel's recovery action for a fatal error reported through AER is an SBR at the parent bridge. This SBR does not go through the SBR conditions above, so in a configuration where the parent port does not mask Surprise Down, it can restart the host.
- With
ha_aer_enable=1,AER reporting enabled: the kernel's fatal remedy is a parent-bridge SBRis left indmesgeach time the driver probes the card. - On kernel 6.4 or later and RHEL 9.4 or later, the driver does not change the error reporting settings. On these kernels, the kernel and platform configuration decide error reporting, so errors of the card can be reported even with
ha_aer_enable=0. - Even with
ha_aer_enable=0, the driver can receive PCIe error recovery notifications. This is because, when DPC operates, when the parent port detects an error, or when a fatal error occurs on another device under the same bridge, the kernel runs the recovery procedure on all devices under it.
When a card is detached from the driver¶
Unbind, hot unplug, and ha-hotreset remove one card from the driver, and rmmod ha removes all cards.
- The handles of processes that opened that card remain, but later SDK calls fail. Do not access the device memory mappings of that card. Accessing them can cause SIGBUS. Closing a handle always succeeds.
- Jobs in progress are aborted, and the removal finishes after the calls that were in progress have exited. While waiting,
remove: <N> fd op(s) still inside the gateis left indmesgevery 200 ms. A firmware update in progress is waited for until the update finishes or the wait time runs out. - A device-to-device copy that another card was performing targeting this card's memory fails. See Multiple devices.
- When the card is probed again (rebind, rescan), open the device anew. The same card usually reappears with the same number.
1 2 3 4 5 | |
Unbind proceeds even if handles are open. rmmod ha (modprobe -r ha) fails with a module-in-use error if even one /dev/haN is open, so end the processes that use the devices first. The procedure for replacing the module is in Upgrading and removal.
Resets during a firmware update¶
- A device with a firmware update in progress refuses an SDK device reset and skips automatic reset. This is because resetting while the flash is being written can make the card boot from a partially written flash and fail to come up.
- If the firmware's response to the last update was not received, the same refusal continues until a response is received. If you request a reset in this state,
reset refused: last fw_update unansweredis left indmesg. - The driver cannot refuse a reset issued by the kernel. A reset issued through the kernel's PCI reset function waits until an update in progress finishes, but if there is an update that received no response, it proceeds anyway and
reset_prepare: last fw_update unanswered - resetting now may leave the boot flash half-writtenis left indmesg. - Do not use
ha-hotreset --forceon a card being updated. The update tool keeps the device open to block a plainha-hotreset. - On a card whose last
ha-fw-updaterun ended with exit code 4 or ended without reporting an exit code, the flash may be partially written. After the tool ends, no process has the card open, so even a plainha-hotresetis not refused and proceeds with the reset. Do not restart such a card withha-hotresetor a power cycle; follow When no response is received. Use the options that remain when the firmware never responds (module reload,ha-hotreset, power cycle) only after consulting HyperAccel. - To boot the new firmware, use
ha-hotreset, not an SDK device reset. An SDK device reset first re-initializes the running firmware, so it is not a way to load a new image.
The update procedure and the actions to take when no response is received are in Firmware update.
See also¶
- Hang detection and the watchdog — reading hang records and the records a reset erases
- Collecting diagnostic information for a problem — a script that saves diagnostic information before a reset
- Recovering an unresponsive card — the procedure for recovering one card with
ha-hotreset - Telemetry — the
healthverdict and the meaning ofunknown - Firmware update — why resets are refused during an update, and booting the new firmware
- Multiple devices — device-to-device copies when a card is detached
- Troubleshooting
- ha-hotreset — options, pre-flight checks, order of operations, exit codes
- Parameter reference —
ha_auto_reset,ha_aer_enable,ha_wd_preempt,ha_devcmd_ceiling_ms - sysfs —
health,last_hang, the RAS counters - Kernel log messages