Skip to content

Recovering an unresponsive card

When a single card does not respond, this tutorial walks you through the steps in order, from checking the symptoms to hot resetting only that card and confirming that it has come back. The reset uses ha-hotreset, run as root on the host. ha-hotreset removes the card from the driver, issues an SBR (Secondary Bus Reset) on the parent bridge, and has the driver probe the card again; it does not touch the ha module or the other cards. The background for each step is in FATAL and recovery, and the tool's options and output are in ha-hotreset.

You need root privileges on the host and ha-hotreset from the driver package (Install location); ha-hotreset uses setpci and lspci from pciutils. In the example, the unresponsive card is ha1 and its PCI address is 0000:65:00.0. Replace the card number (ha1) and PCI address used in the commands with your actual values before running them.

Consider an SDK device reset first

If the application can recover the card with the SDK's device reset, use that method first. An SDK device reset proceeds even when handles are open. This tutorial is the procedure for when that reset did not recover the card, or when you stop the applications that use the card and reset the card from the host. For the overall order, see Recovery order.


1. Check the card state

  • Application: SDK calls fail with an error saying the device cannot be recovered. Restarting the process produces the same error until the card is reset. Before the card becomes FATAL, the jobs may only appear to be stalled without finishing.
  • Operator: Check with hyperaccel-smi, sysfs health, and dmesg. All three can be read even while the card is stalled.
1
hyperaccel-smi

The header of the card table and the two lines for that card look as follows.

1
2
3
4
5
| LPU  Name                 Serial          | Bus-Id                | Health              |
| Temp   Power        Util(core)            | Memory-Usage (DRAM)   | Fw-Err / Timeouts   |
+===========================================+=======================+=====================+
|   1  HyperAccel LPU   HA-5f0e9d8c-7b6…    | 0000:65:00.0          | FAULTED             |
| N/A    N/A         N/A                    | 8.00 / 128.00 GiB     | 0 / 1               |
  • If Health is FAULTED, the driver has marked that card FATAL or the last reset has failed. A card that is FATAL accepts no new jobs until it is reset. cat /sys/class/ha/ha1/health also prints faulted.
  • Temperature, power, and utilization become N/A within a few seconds after the card becomes FATAL. This is because the driver does not send polling requests to a FATAL card.
  • Bus-Id is the PCI address of that card, and Fw-Err / Timeouts is the cumulative number of firmware errors and response wait ceiling expirations.

Look for the cause in the kernel log.

1
sudo dmesg -T | grep -E ' ha [0-9a-f]+:' | grep -E 'watchdog|Devcmd|hang|Firmware error|sys_mon|auto-TDR|device reset failed|fw_update'
Line Meaning
Devcmd no response after <ms> ms (device hang): <verdict>, followed by hang (ceiling timeout): … and hang dump (ceiling timeout): … The firmware did not respond within the response wait ceiling. If an application job was in progress, the card becomes FATAL
Firmware error: resq[<n>] rc=0x<rc> … The firmware reported that a command failed. Depending on rc, only that command fails or the card becomes FATAL (Firmware errors)
watchdog: … wait stalled … A wait exceeded the watchdog threshold. This line alone does not mean FATAL. If ha_wd_preempt=1, the watchdog cuts off that wait, and if it was the wait of an application job, the card becomes FATAL. Cutting off a wait without a ceiling does not make the card FATAL
Devcmd waiting <ms> ms with no timeout requested … A request that waits without a ceiling is not getting a response. It is not FATAL, so health does not change
sys_mon: no response for <n> consecutive polls … The card did not respond to temperature, power, and utilization requests, so the driver stopped polling. This line alone is not FATAL, so health does not change (Monitoring suspension)
auto-TDR: the device went FATAL; resetting it The automatic reset (ha_auto_reset=1) ran. If it fails, auto-TDR: reset failed (<err>); … is logged and health stays faulted
fw_update: no answer from firmware …, hang (fw_update timeout): …, auto-TDR: skipped, a fw_update is in flight or unanswered … A firmware update of this card is in progress, or no response from the firmware to the update has been received. The flash may be in the middle of being written, so do not reset the card. If an update is in progress, wait until it finishes; if no response was received, follow Resets and jobs are refused after an update
  • If health is not faulted but jobs are stalled, it may be a wait without a ceiling. Use the method in Waits without a ceiling to tell whether it is slow or stalled, and if you conclude it is stalled, go to the next step.
  • If health is degraded and jobs are running normally, no recovery is needed. See health is degraded.
  • If a reset failed, the device reset failed (...) line it left last tells you what to do next. If the table in If the reset fails directs you to a power cycle, power-cycle the host instead of following this procedure.

2. Save the diagnostic information

Before you reset the card, follow Collecting diagnostic information for a problem to save the diagnostic information as an archive. ha-hotreset erases the records that explain the failure.

  • The driver probes the card again, so the RAS counters (devcmd_timeouts, devcmd_fw_errors, hdma_timeouts) and last_hang start over from the beginning.
  • The card boots again from the firmware in flash, so the stalled state itself does not remain.
  • The messages logged while the card is removed and probed again can push old records out of the kernel log buffer.

3. Check the card's PCI address

ha-hotreset identifies a card by its PCI address (BDF). The Bus-Id in the hyperaccel-smi output in step 1 is that value, and you can also find it through the /sys/class/ha/ha1/device link, which points to the PCI device of that card.

1
basename "$(readlink -f /sys/class/ha/ha1/device)"
1
0000:65:00.0

Next, check whether this card is in a configuration that can be hot reset. This command changes nothing.

1
sudo ha-hotreset --list
1
2
0000:17:00.0  (parent <parent BDF>, parent is a root port)
0000:65:00.0  (parent <parent BDF>, parent is a root port)

If the line ends with the path is masked for the reset, the card's parent port does not mask Surprise Down, so the path is masked during the reset (step 5). If NOT resettable is appended to the end of that card's line, that card cannot be hot reset on this host. If an SDK device reset does not recover it either, you must power-cycle the host; a power cycle restarts all the cards in the host, so set the work schedule first. The meaning of the verdict is in Configurations in which an SBR can be issued. --list shows only the verdict for the parent port; open handles and other devices under the same bridge are checked at reset time.

Before you power-cycle

A power cycle restarts all the cards in the host. If pgrep -af ha-fw-update shows a process, or if sudo dmesg shows fw_update: no answer from firmware for a card and no fw_update: late completion from firmware for the same card after it, do not turn off the power; follow When no response is received first.


4. Stop the processes that use the card

ha-hotreset does not reset the card if any process has that card's /dev/ha1 open. After the card is removed, SDK calls from that process fail, and accessing a device memory mapping can raise SIGBUS. The following command reads only /proc, with no additional packages, and prints the PID and command line of each process that has opened the node or mapped device memory.

1
2
3
4
5
6
7
# Run as root (for example: sudo -i)
dev=/dev/ha1
for p in /proc/[0-9]*; do
  if readlink "$p"/fd/* | grep -qx "$dev" || grep -q " $dev\$" "$p/maps"; then
    echo "${p#/proc/} $(tr '\0' ' ' < "$p/cmdline")"
  fi
done 2>/dev/null
  • If lsof or fuser is installed, you can also check with sudo lsof /dev/ha1 or sudo fuser -v /dev/ha1.
  • End the processes in the list. If the application has a defined way to shut down, use it; otherwise end the process with kill <PID>.
  • Processes running inside containers also appear with their host PIDs. Stop that container.
  • Also stop jobs that run on other cards and exchange data with this card. See When one card is reset or removed.
  • Run the command again and check that nothing is printed.

Do not reset a card that is updating its firmware

If ha-fw-update is in the list, that card's firmware is being updated. Do not end that process; wait until the update finishes. If you reset a card while the flash is being written, the card may fail to boot. Do not reset the card when dmesg shows fw_update: no answer from firmware, either; follow Resets and jobs are refused after an update.


5. Reset the card

1
sudo ha-hotreset --bdf 0000:65:00.0

Before touching the card, ha-hotreset checks that the parent port is in a configuration in which an SBR can be issued, that this card is the only device under the parent bridge, and that no process has /dev/ha1 open. If any check fails, it changes nothing, prints Pre-flight failed - nothing was touched., and ends with exit code 3 (an open handle) or 4 (a configuration in which an SBR cannot be issued). If the parent port does not mask Surprise Down, the tool masks the path during the reset and restores it when the card responds again (Path masking).

When it is refused

The reason for the refusal appears before Pre-flight failed. Below is the first line for each reason.

Output Meaning What to do
0000:65:00.0: /dev/ha1 still open by PID(s) <PID list> A process that has the node open remains. The next line shows the card's inflight and streams values for reference Go back to step 4, end that process, and run the command again
0000:65:00.0: parent is a <port type> whose Surprise Down mask cannot be read (no AER capability) - refusing SBR, 0000:65:00.0: parent has no PCIe capability - refusing SBR, or 0000:65:00.0: cannot read <register> on <port> - the path cannot be masked, refusing SBR Whether the parent port reports the link drop caused by an SBR as an error cannot be verified, or it does and the path cannot be masked. If an SBR is issued in this configuration, the platform may restart the entire host. As the guidance printed after it says, --force cannot bypass this check either That card cannot be hot reset on this host. Power-cycle the host
0000:65:00.0: parent bridge <parent BDF> has more than one device below it: Another device is under the same parent bridge, so the SBR also resets that device. The following lines list the devices under that bridge The card cannot be reset individually. Power-cycle the host

--force

  • --force bypasses only the open-handle check. For a process that has the handle open, subsequent SDK calls fail, and accessing a device memory mapping can raise SIGBUS. Use it only in an emergency when the process cannot be ended, and never on a card whose firmware is being updated.

Reset output

If the checks pass, ha-hotreset removes the card from PCI, issues an SBR on the parent bridge, and rescans the bus so that the driver probes the card again. On success, the following lines appear in order, and the card's AER register values and status lines appear between them.

1
2
3
4
5
6
7
8
9
=== 0000:65:00.0 (parent <parent BDF>, /dev/ha1) ===
remove 0000:65:00.0
BRIDGE_CONTROL: <original value> -> <value with SBR on> (SBR=1) -> <value with SBR off> (SBR=0)
Waiting for PCIe link to retrain...
PCIe link is active.
Waiting for EP to be enumerated...
Endpoint fully enumerated.
Waiting for /dev/ha1...
Hot reset OK: 0000:65:00.0
  • For a card whose path is masked, more lines appear: a line that starts with masking <N> ports: …, a line that checks whether the card responds before the masking is restored, and AER status lines from before and after the restore.
  • The checks do not read the card itself, so they pass even if the card does not respond. An unresponsive card is exactly what you want to reset.
  • Do not interrupt it before it finishes. In particular, if you end it with kill -9 (SIGKILL), the cleanup step does not run, and the card can be left without being enumerated on PCI again.
  • Hot reset OK means the card has been enumerated on PCI again and /dev/ha1 has reappeared. You check the firmware state in the next step.

6. Verify the recovery

Check in the kernel log that the driver removed the card and probed it again.

1
sudo dmesg -T | grep -E 'ha 0000:65:00.0: (Removing device|Remove complete|Probing device|firmware ready|firmware boot sentinel|firmware never reported ready|Probe successful)'

After the time of the reset, the following lines should appear in order. Lines left from the initial probe also appear before them. If the firmware was not ready during the probe, a firmware boot sentinel not seen within <ms> ms or firmware never reported ready warning is logged instead of the firmware ready after line, and Probe successful still appears. In that case, follow the fw_version check below.

1
2
3
4
5
[<time>] ha 0000:65:00.0: Removing device
[<time>] ha 0000:65:00.0: Remove complete
[<time>] ha 0000:65:00.0: Probing device
[<time>] ha 0000:65:00.0: firmware ready after <ms> ms
[<time>] ha 0000:65:00.0: Probe successful: /dev/ha1 created

Read the card state and the firmware version.

1
grep -H . /sys/class/ha/ha1/health /sys/class/ha/ha1/fw_version
1
2
/sys/class/ha/ha1/health:ok
/sys/class/ha/ha1/fw_version:1.0.5
  • health should be ok. The driver probes the card anew and the RAS counters start again from 0, so if the card is normal it is not degraded either.
  • The firmware is ready only if fw_version is not unknown. health does not take firmware readiness into account in its verdict, so check both. If it is unknown right after the reset, read it again after a few seconds. If the firmware becomes ready late, the driver fills in the value and logs firmware booted after probe gave up; CSR-derived telemetry refreshed.
  • If fw_version differs from before the reset, the card booted the firmware that was written to flash with the reset deferred. See Resetting later.
  • In hyperaccel-smi, check that Health is OK for the card whose Bus-Id is 0000:65:00.0. If temperature and power were visible for the card before the reset, those values should also appear again.

The driver reassigns the previous number to a card that returns at the same PCI address, so the card normally returns as the same ha1 as before the reset. The number can differ under the rules in Card numbers, so identify the card by its PCI address.

Finally, restart the applications you stopped in step 4. The applications open the device anew and start their work from the beginning. The hot reset recreated the nodes on the host, so also restart the containers that used that card.


If recovery fails

Symptom What to do
ha-hotreset prints did not reappear - check dmesg for probe failure or did not re-enumerate and ends with exit code 1 The card was not enumerated on PCI again (did not re-enumerate), or the driver failed to probe the card again (did not reappear). Narrow down the cause with /dev/haN is missing, and if the card is not visible on the PCI bus either, power-cycle the host
ha-hotreset prints the card was reset but the ha driver did not bind it and ends with exit code 1 The reset was done, but the driver did not bind the card. Match the card's line in the kernel log against Errors that stop probe. If the firmware booted late, bind the card again a little later with echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind
fw_version stays unknown even after Hot reset OK Follow fw_version is unknown
health becomes faulted again soon after jobs are restarted The same problem is recurring. Save the diagnostic information again and contact the support team

Other recovery methods are as follows.

  • SDK device reset: A reset that an application requests through the SDK. It proceeds even when handles are open, and if the firmware reinitialization fails, it issues an SBR under the same SBR conditions. See SDK device reset.
  • Automatic reset: If ha_auto_reset=1, the driver resets the card by itself the moment it becomes FATAL; the default is off. How to turn it on is described in Configuring and using parameters and ha_auto_reset, and its behavior in Automatic reset.
  • Power cycle: If the methods above do not recover the card, or the card is in a configuration that cannot be hot reset, turn the host's power off and on. Do not turn it off during a firmware update. See Power cycle.
  • Requesting support: Contact HyperAccel support with the diagnostic archive created in step 2 attached. What to include is described in Requesting support.

See also