Recovering an unresponsive card¶
When a single card does not respond, this tutorial walks you through the steps in order, from checking the symptoms to hot resetting only that card and confirming that it has come back. The reset uses ha-hotreset, run as root on the host. ha-hotreset removes the card from the driver, issues an SBR (Secondary Bus Reset) on the parent bridge, and has the driver probe the card again; it does not touch the ha module or the other cards. The background for each step is in FATAL and recovery, and the tool's options and output are in ha-hotreset.
You need root privileges on the host and ha-hotreset from the driver package (Install location); ha-hotreset uses setpci and lspci from pciutils. In the example, the unresponsive card is ha1 and its PCI address is 0000:65:00.0. Replace the card number (ha1) and PCI address used in the commands with your actual values before running them.
Consider an SDK device reset first
If the application can recover the card with the SDK's device reset, use that method first. An SDK device reset proceeds even when handles are open. This tutorial is the procedure for when that reset did not recover the card, or when you stop the applications that use the card and reset the card from the host. For the overall order, see Recovery order.
1. Check the card state¶
- Application: SDK calls fail with an error saying the device cannot be recovered. Restarting the process produces the same error until the card is reset. Before the card becomes FATAL, the jobs may only appear to be stalled without finishing.
- Operator: Check with
hyperaccel-smi, sysfshealth, anddmesg. All three can be read even while the card is stalled.
1 | |
The header of the card table and the two lines for that card look as follows.
1 2 3 4 5 | |
- If
HealthisFAULTED, the driver has marked that card FATAL or the last reset has failed. A card that is FATAL accepts no new jobs until it is reset.cat /sys/class/ha/ha1/healthalso printsfaulted. - Temperature, power, and utilization become
N/Awithin a few seconds after the card becomes FATAL. This is because the driver does not send polling requests to a FATAL card. Bus-Idis the PCI address of that card, andFw-Err / Timeoutsis the cumulative number of firmware errors and response wait ceiling expirations.
Look for the cause in the kernel log.
1 | |
| Line | Meaning |
|---|---|
Devcmd no response after <ms> ms (device hang): <verdict>, followed by hang (ceiling timeout): … and hang dump (ceiling timeout): … |
The firmware did not respond within the response wait ceiling. If an application job was in progress, the card becomes FATAL |
Firmware error: resq[<n>] rc=0x<rc> … |
The firmware reported that a command failed. Depending on rc, only that command fails or the card becomes FATAL (Firmware errors) |
watchdog: … wait stalled … |
A wait exceeded the watchdog threshold. This line alone does not mean FATAL. If ha_wd_preempt=1, the watchdog cuts off that wait, and if it was the wait of an application job, the card becomes FATAL. Cutting off a wait without a ceiling does not make the card FATAL |
Devcmd waiting <ms> ms with no timeout requested … |
A request that waits without a ceiling is not getting a response. It is not FATAL, so health does not change |
sys_mon: no response for <n> consecutive polls … |
The card did not respond to temperature, power, and utilization requests, so the driver stopped polling. This line alone is not FATAL, so health does not change (Monitoring suspension) |
auto-TDR: the device went FATAL; resetting it |
The automatic reset (ha_auto_reset=1) ran. If it fails, auto-TDR: reset failed (<err>); … is logged and health stays faulted |
fw_update: no answer from firmware …, hang (fw_update timeout): …, auto-TDR: skipped, a fw_update is in flight or unanswered … |
A firmware update of this card is in progress, or no response from the firmware to the update has been received. The flash may be in the middle of being written, so do not reset the card. If an update is in progress, wait until it finishes; if no response was received, follow Resets and jobs are refused after an update |
- If
healthis notfaultedbut jobs are stalled, it may be a wait without a ceiling. Use the method in Waits without a ceiling to tell whether it is slow or stalled, and if you conclude it is stalled, go to the next step. - If
healthisdegradedand jobs are running normally, no recovery is needed. Seehealthisdegraded. - If a reset failed, the
device reset failed (...)line it left last tells you what to do next. If the table in If the reset fails directs you to a power cycle, power-cycle the host instead of following this procedure.
2. Save the diagnostic information¶
Before you reset the card, follow Collecting diagnostic information for a problem to save the diagnostic information as an archive. ha-hotreset erases the records that explain the failure.
- The driver probes the card again, so the RAS counters (
devcmd_timeouts,devcmd_fw_errors,hdma_timeouts) andlast_hangstart over from the beginning. - The card boots again from the firmware in flash, so the stalled state itself does not remain.
- The messages logged while the card is removed and probed again can push old records out of the kernel log buffer.
3. Check the card's PCI address¶
ha-hotreset identifies a card by its PCI address (BDF). The Bus-Id in the hyperaccel-smi output in step 1 is that value, and you can also find it through the /sys/class/ha/ha1/device link, which points to the PCI device of that card.
1 | |
1 | |
Next, check whether this card is in a configuration that can be hot reset. This command changes nothing.
1 | |
1 2 | |
If the line ends with the path is masked for the reset, the card's parent port does not mask Surprise Down, so the path is masked during the reset (step 5). If NOT resettable is appended to the end of that card's line, that card cannot be hot reset on this host. If an SDK device reset does not recover it either, you must power-cycle the host; a power cycle restarts all the cards in the host, so set the work schedule first. The meaning of the verdict is in Configurations in which an SBR can be issued. --list shows only the verdict for the parent port; open handles and other devices under the same bridge are checked at reset time.
Before you power-cycle
A power cycle restarts all the cards in the host. If pgrep -af ha-fw-update shows a process, or if sudo dmesg shows fw_update: no answer from firmware for a card and no fw_update: late completion from firmware for the same card after it, do not turn off the power; follow When no response is received first.
4. Stop the processes that use the card¶
ha-hotreset does not reset the card if any process has that card's /dev/ha1 open. After the card is removed, SDK calls from that process fail, and accessing a device memory mapping can raise SIGBUS. The following command reads only /proc, with no additional packages, and prints the PID and command line of each process that has opened the node or mapped device memory.
1 2 3 4 5 6 7 | |
- If
lsoforfuseris installed, you can also check withsudo lsof /dev/ha1orsudo fuser -v /dev/ha1. - End the processes in the list. If the application has a defined way to shut down, use it; otherwise end the process with
kill <PID>. - Processes running inside containers also appear with their host PIDs. Stop that container.
- Also stop jobs that run on other cards and exchange data with this card. See When one card is reset or removed.
- Run the command again and check that nothing is printed.
Do not reset a card that is updating its firmware
If ha-fw-update is in the list, that card's firmware is being updated. Do not end that process; wait until the update finishes. If you reset a card while the flash is being written, the card may fail to boot. Do not reset the card when dmesg shows fw_update: no answer from firmware, either; follow Resets and jobs are refused after an update.
5. Reset the card¶
1 | |
Before touching the card, ha-hotreset checks that the parent port is in a configuration in which an SBR can be issued, that this card is the only device under the parent bridge, and that no process has /dev/ha1 open. If any check fails, it changes nothing, prints Pre-flight failed - nothing was touched., and ends with exit code 3 (an open handle) or 4 (a configuration in which an SBR cannot be issued). If the parent port does not mask Surprise Down, the tool masks the path during the reset and restores it when the card responds again (Path masking).
When it is refused¶
The reason for the refusal appears before Pre-flight failed. Below is the first line for each reason.
| Output | Meaning | What to do |
|---|---|---|
0000:65:00.0: /dev/ha1 still open by PID(s) <PID list> |
A process that has the node open remains. The next line shows the card's inflight and streams values for reference |
Go back to step 4, end that process, and run the command again |
0000:65:00.0: parent is a <port type> whose Surprise Down mask cannot be read (no AER capability) - refusing SBR, 0000:65:00.0: parent has no PCIe capability - refusing SBR, or 0000:65:00.0: cannot read <register> on <port> - the path cannot be masked, refusing SBR |
Whether the parent port reports the link drop caused by an SBR as an error cannot be verified, or it does and the path cannot be masked. If an SBR is issued in this configuration, the platform may restart the entire host. As the guidance printed after it says, --force cannot bypass this check either |
That card cannot be hot reset on this host. Power-cycle the host |
0000:65:00.0: parent bridge <parent BDF> has more than one device below it: |
Another device is under the same parent bridge, so the SBR also resets that device. The following lines list the devices under that bridge | The card cannot be reset individually. Power-cycle the host |
--force
--forcebypasses only the open-handle check. For a process that has the handle open, subsequent SDK calls fail, and accessing a device memory mapping can raise SIGBUS. Use it only in an emergency when the process cannot be ended, and never on a card whose firmware is being updated.
Reset output¶
If the checks pass, ha-hotreset removes the card from PCI, issues an SBR on the parent bridge, and rescans the bus so that the driver probes the card again. On success, the following lines appear in order, and the card's AER register values and status lines appear between them.
1 2 3 4 5 6 7 8 9 | |
- For a card whose path is masked, more lines appear: a line that starts with
masking <N> ports: …, a line that checks whether the card responds before the masking is restored, and AER status lines from before and after the restore. - The checks do not read the card itself, so they pass even if the card does not respond. An unresponsive card is exactly what you want to reset.
- Do not interrupt it before it finishes. In particular, if you end it with
kill -9(SIGKILL), the cleanup step does not run, and the card can be left without being enumerated on PCI again. Hot reset OKmeans the card has been enumerated on PCI again and/dev/ha1has reappeared. You check the firmware state in the next step.
6. Verify the recovery¶
Check in the kernel log that the driver removed the card and probed it again.
1 | |
After the time of the reset, the following lines should appear in order. Lines left from the initial probe also appear before them. If the firmware was not ready during the probe, a firmware boot sentinel not seen within <ms> ms or firmware never reported ready warning is logged instead of the firmware ready after line, and Probe successful still appears. In that case, follow the fw_version check below.
1 2 3 4 5 | |
Read the card state and the firmware version.
1 | |
1 2 | |
healthshould beok. The driver probes the card anew and the RAS counters start again from 0, so if the card is normal it is notdegradedeither.- The firmware is ready only if
fw_versionis notunknown.healthdoes not take firmware readiness into account in its verdict, so check both. If it isunknownright after the reset, read it again after a few seconds. If the firmware becomes ready late, the driver fills in the value and logsfirmware booted after probe gave up; CSR-derived telemetry refreshed. - If
fw_versiondiffers from before the reset, the card booted the firmware that was written to flash with the reset deferred. See Resetting later. - In
hyperaccel-smi, check thatHealthisOKfor the card whoseBus-Idis0000:65:00.0. If temperature and power were visible for the card before the reset, those values should also appear again.
The driver reassigns the previous number to a card that returns at the same PCI address, so the card normally returns as the same ha1 as before the reset. The number can differ under the rules in Card numbers, so identify the card by its PCI address.
Finally, restart the applications you stopped in step 4. The applications open the device anew and start their work from the beginning. The hot reset recreated the nodes on the host, so also restart the containers that used that card.
If recovery fails¶
| Symptom | What to do |
|---|---|
ha-hotreset prints did not reappear - check dmesg for probe failure or did not re-enumerate and ends with exit code 1 |
The card was not enumerated on PCI again (did not re-enumerate), or the driver failed to probe the card again (did not reappear). Narrow down the cause with /dev/haN is missing, and if the card is not visible on the PCI bus either, power-cycle the host |
ha-hotreset prints the card was reset but the ha driver did not bind it and ends with exit code 1 |
The reset was done, but the driver did not bind the card. Match the card's line in the kernel log against Errors that stop probe. If the firmware booted late, bind the card again a little later with echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind |
fw_version stays unknown even after Hot reset OK |
Follow fw_version is unknown |
health becomes faulted again soon after jobs are restarted |
The same problem is recurring. Save the diagnostic information again and contact the support team |
Other recovery methods are as follows.
- SDK device reset: A reset that an application requests through the SDK. It proceeds even when handles are open, and if the firmware reinitialization fails, it issues an SBR under the same SBR conditions. See SDK device reset.
- Automatic reset: If
ha_auto_reset=1, the driver resets the card by itself the moment it becomes FATAL; the default is off. How to turn it on is described in Configuring and using parameters andha_auto_reset, and its behavior in Automatic reset. - Power cycle: If the methods above do not recover the card, or the card is in a configuration that cannot be hot reset, turn the host's power off and on. Do not turn it off during a firmware update. See Power cycle.
- Requesting support: Contact HyperAccel support with the diagnostic archive created in step 2 attached. What to include is described in Requesting support.
See also¶
- FATAL and recovery — Recovery methods and order, configurations in which an SBR can be issued
- ha-hotreset — Options, pre-flight checks, output, exit codes
- Collecting diagnostic information for a problem — Information to save before a reset
- Hang detection and the watchdog — How to identify a stalled wait and how to read hang records
- Telemetry —
healthverdict and the meaning ofunknown - hyperaccel-smi
- Troubleshooting
- Kernel log messages