Skip to content

ha-hotreset

ha-hotreset is a host command that performs a PCIe hot reset on a single HyperAccel card. It removes the card from the driver, issues an SBR (Secondary Bus Reset) on the parent bridge, and then rescans the bridge so that the driver probes the card again. It leaves the ha module loaded and does not touch cards that you did not request. This page defines the install location, options, pre-flight checks, sequence of operations, output, and exit codes. For when to use which recovery method, see FATAL and recovery.


Install location

ha-hotreset is installed to /usr/bin/ha-hotreset by the driver deb package hyperdriver-dkms.

Synopsis

1
2
3
4
5
ha-hotreset [--bdf BDF] [--force] [--no-switch-link-drop]
ha-hotreset --all [--force] [--no-switch-link-drop]
ha-hotreset --check-mask[=SECONDS] [--bdf BDF | --all]
ha-hotreset --list [--no-switch-link-drop]
ha-hotreset --help

Requirements

  • Run it as root on the host. Every operation except --help and --list prints Run as root and ends if you are not root. --list runs without root, but it also reads the PCIe configuration of the parent port, so run it as root if possible.
  • setpci and lspci from pciutils are required. If they are missing, the tool prints setpci not found (install pciutils) or lspci not found (install pciutils) and ends.
  • A reset and --check-mask also need flock from util-linux. If it is missing, the tool prints flock not found (install util-linux) and ends. See Host lock.
  • No process may have the device node of the card to be reset open. See Pre-flight checks.
  • ha-hotreset neither unloads nor loads the ha module. It is the loaded driver that probes the card again.

Options

Option Description
--bdf BDF, --bdf=BDF PCI address of the card to reset (for example, 0000:1a:00.0). 1a:00.0 without the domain is accepted only if that address is unique across all PCI domains. If the host has only one card, you can omit this option
--all Resets every card with vendor ID 20fb that lspci finds. It starts only if the pre-flight checks pass for all cards. It cannot be combined with --bdf
--force Proceeds even if the device node is open. It does not skip the other pre-flight checks
--list For each card that lspci finds, prints the parent bridge and the SBR verdict, then ends. It changes nothing. It cannot be combined with --check-mask
--no-switch-link-drop Refuses a card whose parent port does not mask Surprise Down, instead of resetting it with path masking. With --list, marks such a card NOT resettable. See Path masking
--allow-switch-link-drop Accepted for compatibility. Path masking is the default, so it has no effect
--check-mask[=SECONDS] Only checks whether the path masking holds on this host. It does not remove the card or drop the link. Attach SECONDS (default 60) with =. See Mask check
-h, --help Prints the usage and ends

After a reset with --force

Subsequent SDK calls from a process that had kept a handle open fail. That process must close and reopen the device, and accessing the device memory mapping in the meantime can cause SIGBUS. Do not use --force on a card whose firmware is being updated. The update tool keeps the device open to block an ordinary reset, and resetting while the flash is being written can leave the card unable to boot.

Target cards

  • A card specified with --bdf is targeted regardless of the lspci list. The address is looked up under /sys/bus/pci/devices.
  • When an address without a domain resolves to a single card, the tool prints Resolved --bdf <input> -> <BDF>. If the same address exists in several PCI domains, it prints the candidates and refuses.
  • If both --bdf and --all are omitted and lspci finds exactly one card, the tool selects that card and prints Only one card present, selecting <BDF>. If there are two or more, it prints the list and refuses. If there is none, it prints No PCIe EP with vendor 20fb found - pass --bdf explicitly.
  • --all prints Targets (<count>): <BDF list> at the start.

Pre-flight checks

Every mode except --list and --check-mask checks all target cards before touching any card. If any check fails, the tool prints Pre-flight failed - nothing was touched. and ends with the exit code in the table below. If the cards fail for different reasons, 4 takes precedence over 3, and 3 over 1.

Check Output on failure (first line) Skipped by --force Exit code
The PCI node of the card exists <BDF>: missing sysfs node /sys/bus/pci/devices/<BDF> No 1
The vendor ID is 20fb <BDF>: not a 20fb device (vendor=<value>) No 1
The configuration allows an SBR <BDF>: <verdict> - refusing SBR No 4
If the path must be masked, the registers to mask can be read on every port of the path <BDF>: cannot map the path to a root port - refusing SBR, <BDF>: cannot read <register> on <port> - the path cannot be masked, refusing SBR No 4
This card is the only device below the parent bridge <BDF>: parent bridge <parent BDF> has more than one device below it: No 4
No process has the device node open <BDF>: /dev/haN still open by PID(s) <PID list> Yes 3
  • A card whose vendor ID reads as 0xffff, or cannot be read, is not refused; the tool only prints an already unresponsive warning. This is because an unresponsive card is exactly what needs to be reset.
  • The SBR verdict is the same as for --list. See List output.
  • Even after the pre-flight checks pass, the path masking may fail to apply right before the reset. That card is then left untouched, but with --all the tool ends with the cards before it already reset. On a host where you use it for the first time, check first with Mask check.
  • When counting the devices below the parent bridge, the PCI functions of the same card are counted as one.
  • Open handles are detected by looking in /proc/<PID>/fd for processes that have the /dev/haN of that card open. When an open handle is found, the tool also prints, for reference, the sysfs inflight and stream_count values of that card on a (inflight=<n> streams=<n> - informational only; …) line. The tool refuses if a handle is open even when no work is in progress. A card that is not bound to the driver and so has no /dev/haN skips this check.
  • The tool does not check the firmware update state. After the update tool ends, no process has the card open, so the tool also resets a card whose last ha-fw-update run ended with exit code 4 or ended without reporting an exit code. Such a card may fail to come up because it boots from a partially written flash, so first follow the guidance in When no response is received: wait for the firmware to respond and finish the update. If the firmware never responds, the options that remain (reloading the module, this tool, and a power cycle) risk booting from a partially written flash, so use them only after consulting HyperAccel.

Sequence of operations

For each card, the tool proceeds in the following order. With --all, it resets the cards one at a time, and if one card fails, it stops there and does not reset the remaining cards. A card that was reset but not bound by the ha driver is the exception: the tool resets the remaining cards, then prints the list of unbound cards and ends with exit code 1.

  1. If the parent port is not a root port and does not mask Surprise Down, masks the path from the parent port up to the root port (Path masking). If this fails, the tool ends without touching that card: if no card has been reset yet, it prints Nothing was touched. and ends with exit code 4; if --all has already reset earlier cards, it ends with exit code 1. When the masking succeeds, the tool prints the setpci commands that restore it by hand.
  2. Reads and saves the AER registers of the card (Uncorrectable Error Mask, Correctable Error Mask, Uncorrectable Error Severity).
  3. Prints remove <BDF> and removes the card from PCI (/sys/bus/pci/devices/<BDF>/remove). The driver takes the card down.
  4. Sets and then clears the Secondary Bus Reset bit in the Bridge Control register of the parent bridge.
  5. Waits up to 30 seconds for the parent port to report that the link is active. Even if it does not report, the tool moves on to the next step.
  6. Rescans the parent bridge and waits until the card is enumerated again (up to 100 times, at 0.1-second intervals). If the PCI node of the card does not reappear, the tool fails.
  7. Compares the AER registers of the card with the values from step 2, restores them if they changed, and prints the AER status of the card.
  8. For a card that had /dev/haN before the reset, waits up to about 12 seconds for the driver to probe the card again and the same node to appear. If it does not appear, prints did not reappear - check dmesg for probe failure and fails. For a card that had no /dev/haN before the reset, waits the same time for the ha driver to bind the card. If it does not, the tool finishes step 9, prints the card was reset but the ha driver did not bind it with the command that binds it again, and ends with exit code 1. The tool skips this check if the ha module is not loaded, if the kernel's automatic probing is off, or if the card is kept for another driver (it was bound to another driver before the reset, or driver_override is set).
  9. If step 1 masked the path, checks that the card responds and then restores the masking on the path.
  10. Prints Hot reset OK: <BDF>.

When interrupted or failing

  • If the tool fails, or is interrupted by SIGINT, SIGTERM, or SIGHUP, before it removes the card, it only restores the path masking and ends.
  • If it fails or is interrupted after removing the card, it clears the Secondary Bus Reset bit of the parent bridge, carries out steps 5 to 9 as a successful reset does, and ends. It restores the masking only if the card was enumerated again and responds; otherwise it deliberately leaves the masking in place (Path masking, step 4).
  • After a signal, the tool finishes the cleanup and ends with exit code 129 (SIGHUP), 130 (SIGINT), or 143 (SIGTERM). Signals that arrive during the cleanup are ignored.
  • SIGKILL skips this cleanup, so the card can remain unenumerated or the masking can remain in place. Do not use it. If the path was masked, you can restore it with the setpci commands printed in step 1.

Hot reset OK means that the card was enumerated on PCI again and, if the card had a /dev/haN before the reset, that the node reappeared, or, if it had none, that the ha driver bound the card. Check the firmware state with hyperaccel-smi or the sysfs health.

Path masking

If the parent port is not a root port and does not mask Surprise Down, the parent port reports a Surprise Down error when the SBR drops the link. On a server where platform firmware handles PCIe errors, that error can lead to a restart of the whole host. So while it resets such a card, ha-hotreset stops the ports on the path from the parent port up to the root port from reporting errors (path masking). This is the default; with --no-switch-link-drop, the tool refuses such a card instead of masking. A card whose parent port is a root port or already masks Surprise Down is not masked.

A masked error is not reported, so it does not reach the platform firmware. However, on a server whose platform firmware detects the link drop some other way, such as by reading the ports' error status registers directly, masking may not prevent a host restart. On a host where you use it for the first time, check first with Mask check.

  1. For each port on the path from the parent port to the root port, sets the masking bits of the AER Uncorrectable Error Mask and Correctable Error Mask, and at the root port, clears the three System Error reporting bits of Root Control. Proceeding downward from the root port, reads back all the written values to verify them. If even one port cannot be verified, restores all the ports that were changed and does not issue the SBR.
  2. Keeps the masking in place until the card is enumerated again and the driver probes it.
  3. Waits up to 5 seconds for the sysfs temp node of the card (util if there is no temp) to report a value other than unknown. If neither exists, prints no sysmon node under <path> - falling back to a plain wait and waits 5 seconds. If the card responds, clears the AER status bits of the path and of the card, restores the masking, and reads back to verify. If the restored values are wrong, prints the setpci commands that restore them manually.
  4. If the card is not enumerated again, its device node does not come back, or it does not respond, the tool deliberately leaves the masking in place (the mask is LEFT IN PLACE on purpose), prints the setpci commands that restore it, and ends in failure. Removing the masking above a card that does not respond can let that card's errors reach the platform. While the masking remains, real link errors on that path are not reported either, so after you deal with the card problem, restore it with the printed commands.

Host lock

The ports that are masked (switch upstream ports, root ports) are shared by every card below them. If two runs overlapped, one restoring the masking could leave the other's reset unprotected. So resets and --check-mask run one at a time on a host (/run/ha-hotreset.lock). If another ha-hotreset is running, the tool prints another ha-hotreset is running on this host - waiting up to 600s and waits up to 600 seconds. If the other run does not finish in that time, it prints the other ha-hotreset did not finish in 600s - nothing was touched and ends with exit code 3. --list only reads, so it does not wait.

Mask check

--check-mask[=SECONDS] checks whether path masking can work on this host. It does not remove the card or drop the link, and it does not run the pre-flight checks, so you can run it on a card that is in use. It requires root privileges, and you specify the target with --bdf, the same way as for a reset.

  1. Prints the current settings of each port on the path (before:).
  2. Masks the path in the same way as in Path masking and reads back to verify.
  3. Waits SECONDS seconds and then checks that the masking is still in place.
  4. Prints the verdict, clears the AER status bits of the path ports, and restores the masking.
Verdict Meaning
VERDICT: the path could NOT be masked A port did not accept the masking. On this host, a card whose parent port does not mask Surprise Down cannot be reset
VERDICT: the platform CLEARED the mask on: <port> The platform reverted the masking while the tool was waiting. On this host, a card whose parent port does not mask Surprise Down cannot be reset
VERDICT: the whole path masks and holds. The whole path is masked and the masking holds. This means that writing the masking works without problems; it is not a guarantee that a reset is safe
  • While the tool waits, errors detected by the ports on the path are not reported, and the root port does not escalate the error messages it receives to System Error.
  • If the verdict for every target is the whole path masks and holds, the exit code is 0; otherwise it is 1.

Output

Progress goes to standard output (stdout), and warnings and reasons for refusal go to standard error (stderr).

List output

--list prints one line per card.

1
2
<card BDF>  (parent <parent BDF>, <verdict>)
<card BDF>  (parent <parent BDF>, <verdict> - NOT resettable)
Verdict Reset
parent is a root port Possible
parent is a <port type>, Surprise Down Error masked there Possible
parent is a <port type> with Surprise Down Error unmasked; the path is masked for the reset Possible. The reset is done with path masking
parent is a <port type> with Surprise Down Error unmasked Shown when --no-switch-link-drop is given. Not possible (NOT resettable)
parent is a <port type> whose Surprise Down mask cannot be read (no AER capability) Not possible (NOT resettable)
parent has no PCIe capability Not possible (NOT resettable)

<port type> is one of switch upstream port, switch downstream port, PCIe-to-PCI bridge, PCI-to-PCIe bridge, and port type <N>.

1
2
0000:1a:00.0  (parent 0000:19:00.0, parent is a root port)
0000:5e:00.0  (parent 0000:5d:00.0, parent is a switch downstream port with Surprise Down Error unmasked; the path is masked for the reset)

--list shows only the SBR verdict. Open handles, other devices below the parent bridge, and whether the registers on the path can be read are checked in the pre-flight checks when you reset.

Reset output

Resetting a single card prints the following lines in order. In between, the AER register values and status of the card and the path are also printed.

1
2
3
4
5
6
7
8
9
=== <BDF> (parent <parent BDF>, /dev/haN) ===
remove <BDF>
BRIDGE_CONTROL: <original value> -> <value with SBR on> (SBR=1) -> <value with SBR off> (SBR=0)
Waiting for PCIe link to retrain...
PCIe link is active.
Waiting for EP to be enumerated...
Endpoint fully enumerated.
Waiting for /dev/haN...
Hot reset OK: <BDF>

For a card whose path is masked, the tool also prints masking <number of ports> ports: <port list> and the setpci commands that restore the masking by hand before remove, and Checking the card answers before unmasking (up to 5s)... and device answering (<node>=<value>) - safe to unmask before Hot reset OK.

Without the remove <BDF> line, the card was not touched.

Exit codes

Code Meaning
0 Success. All target cards were reset, --list or --help was printed, or all --check-mask verdicts passed
1 Failure. You are not root, pciutils or flock is missing, no card was found, the card is not a 20fb device, a reset step failed, the ha driver did not bind the card after the reset, or a --check-mask verdict did not pass. A reset that failed partway without a signal also ends here
2 Usage error. This covers an unknown argument, options that cannot be combined, a non-numeric --check-mask value, a --bdf value that is missing or malformed, a --bdf value without a domain that matches no device or more than one device, and several cards being present without --bdf or --all
3 Refused, and nothing was touched. A device node is open (--force overrides this), or another ha-hotreset run did not finish within 600 seconds (Host lock). Run it again later
4 Refused, and nothing was touched. An SBR is not safe at this card's PCIe position: the parent's Surprise Down mask cannot be read, the path cannot be masked or the masking is not applied, the parent leaves it unmasked under --no-switch-link-drop, or another device is below the parent bridge
129, 130, 143 Interrupted by SIGHUP, SIGINT, or SIGTERM during a reset or mask check. The tool runs the cleanup step and then ends

When --all is refused for several reasons, 4 comes before 3, and 3 before 1. Judge failure by a non-zero code, not by 1.

After the reset

  • A card that receives an SBR boots again from the firmware in the flash. This is why this tool is used to apply the new firmware after a firmware update.
  • Because the driver probes the card afresh, the RAS counters and last_hang that the driver collected are cleared.
  • The card usually reappears with the same /dev/haN number as before the reset.
  • Other cards are not reset. However, device-to-device copies from other cards that targeted the memory of this card fail.

See also

  • FATAL and recovery — when to use which recovery method, and the configurations in which an SBR can be issued
  • Firmware update — booting with the new firmware after an update
  • ha-fw-update — resetting the card with this tool after an update
  • hyperaccel-smi — checking the PCI address (Bus-Id) and state of a card
  • sysfs nodes — health, temp, inflight, stream_count
  • Kernel log messages — dmesg output when a card is removed and probed again
  • Recovering an unresponsive card — the procedure for recovering a single card with this tool, from identifying the symptom to checking after the reset