ha-hotreset¶
ha-hotreset is a host command that performs a PCIe hot reset on a single HyperAccel card. It removes the card from the driver, issues an SBR (Secondary Bus Reset) on the parent bridge, and then rescans the bridge so that the driver probes the card again. It leaves the ha module loaded and does not touch cards that you did not request. This page defines the install location, options, pre-flight checks, sequence of operations, output, and exit codes. For when to use which recovery method, see FATAL and recovery.
Install location¶
ha-hotreset is installed to /usr/bin/ha-hotreset by the driver deb package hyperdriver-dkms.
Synopsis¶
1 2 3 4 5 | |
Requirements¶
- Run it as root on the host. Every operation except
--helpand--listprintsRun as rootand ends if you are not root.--listruns without root, but it also reads the PCIe configuration of the parent port, so run it as root if possible. setpciandlspcifrompciutilsare required. If they are missing, the tool printssetpci not found (install pciutils)orlspci not found (install pciutils)and ends.- A reset and
--check-maskalso needflockfromutil-linux. If it is missing, the tool printsflock not found (install util-linux)and ends. See Host lock. - No process may have the device node of the card to be reset open. See Pre-flight checks.
ha-hotresetneither unloads nor loads thehamodule. It is the loaded driver that probes the card again.
Options¶
| Option | Description |
|---|---|
--bdf BDF, --bdf=BDF |
PCI address of the card to reset (for example, 0000:1a:00.0). 1a:00.0 without the domain is accepted only if that address is unique across all PCI domains. If the host has only one card, you can omit this option |
--all |
Resets every card with vendor ID 20fb that lspci finds. It starts only if the pre-flight checks pass for all cards. It cannot be combined with --bdf |
--force |
Proceeds even if the device node is open. It does not skip the other pre-flight checks |
--list |
For each card that lspci finds, prints the parent bridge and the SBR verdict, then ends. It changes nothing. It cannot be combined with --check-mask |
--no-switch-link-drop |
Refuses a card whose parent port does not mask Surprise Down, instead of resetting it with path masking. With --list, marks such a card NOT resettable. See Path masking |
--allow-switch-link-drop |
Accepted for compatibility. Path masking is the default, so it has no effect |
--check-mask[=SECONDS] |
Only checks whether the path masking holds on this host. It does not remove the card or drop the link. Attach SECONDS (default 60) with =. See Mask check |
-h, --help |
Prints the usage and ends |
After a reset with --force
Subsequent SDK calls from a process that had kept a handle open fail. That process must close and reopen the device, and accessing the device memory mapping in the meantime can cause SIGBUS. Do not use --force on a card whose firmware is being updated. The update tool keeps the device open to block an ordinary reset, and resetting while the flash is being written can leave the card unable to boot.
Target cards¶
- A card specified with
--bdfis targeted regardless of thelspcilist. The address is looked up under/sys/bus/pci/devices. - When an address without a domain resolves to a single card, the tool prints
Resolved --bdf <input> -> <BDF>. If the same address exists in several PCI domains, it prints the candidates and refuses. - If both
--bdfand--allare omitted andlspcifinds exactly one card, the tool selects that card and printsOnly one card present, selecting <BDF>. If there are two or more, it prints the list and refuses. If there is none, it printsNo PCIe EP with vendor 20fb found - pass --bdf explicitly. --allprintsTargets (<count>): <BDF list>at the start.
Pre-flight checks¶
Every mode except --list and --check-mask checks all target cards before touching any card. If any check fails, the tool prints Pre-flight failed - nothing was touched. and ends with the exit code in the table below. If the cards fail for different reasons, 4 takes precedence over 3, and 3 over 1.
| Check | Output on failure (first line) | Skipped by --force |
Exit code |
|---|---|---|---|
| The PCI node of the card exists | <BDF>: missing sysfs node /sys/bus/pci/devices/<BDF> |
No | 1 |
The vendor ID is 20fb |
<BDF>: not a 20fb device (vendor=<value>) |
No | 1 |
| The configuration allows an SBR | <BDF>: <verdict> - refusing SBR |
No | 4 |
| If the path must be masked, the registers to mask can be read on every port of the path | <BDF>: cannot map the path to a root port - refusing SBR, <BDF>: cannot read <register> on <port> - the path cannot be masked, refusing SBR |
No | 4 |
| This card is the only device below the parent bridge | <BDF>: parent bridge <parent BDF> has more than one device below it: |
No | 4 |
| No process has the device node open | <BDF>: /dev/haN still open by PID(s) <PID list> |
Yes | 3 |
- A card whose vendor ID reads as
0xffff, or cannot be read, is not refused; the tool only prints analready unresponsivewarning. This is because an unresponsive card is exactly what needs to be reset. - The SBR verdict is the same as for
--list. See List output. - Even after the pre-flight checks pass, the path masking may fail to apply right before the reset. That card is then left untouched, but with
--allthe tool ends with the cards before it already reset. On a host where you use it for the first time, check first with Mask check. - When counting the devices below the parent bridge, the PCI functions of the same card are counted as one.
- Open handles are detected by looking in
/proc/<PID>/fdfor processes that have the/dev/haNof that card open. When an open handle is found, the tool also prints, for reference, the sysfsinflightandstream_countvalues of that card on a(inflight=<n> streams=<n> - informational only; …)line. The tool refuses if a handle is open even when no work is in progress. A card that is not bound to the driver and so has no/dev/haNskips this check. - The tool does not check the firmware update state. After the update tool ends, no process has the card open, so the tool also resets a card whose last
ha-fw-updaterun ended with exit code 4 or ended without reporting an exit code. Such a card may fail to come up because it boots from a partially written flash, so first follow the guidance in When no response is received: wait for the firmware to respond and finish the update. If the firmware never responds, the options that remain (reloading the module, this tool, and a power cycle) risk booting from a partially written flash, so use them only after consulting HyperAccel.
Sequence of operations¶
For each card, the tool proceeds in the following order. With --all, it resets the cards one at a time, and if one card fails, it stops there and does not reset the remaining cards. A card that was reset but not bound by the ha driver is the exception: the tool resets the remaining cards, then prints the list of unbound cards and ends with exit code 1.
- If the parent port is not a root port and does not mask Surprise Down, masks the path from the parent port up to the root port (Path masking). If this fails, the tool ends without touching that card: if no card has been reset yet, it prints
Nothing was touched.and ends with exit code4; if--allhas already reset earlier cards, it ends with exit code1. When the masking succeeds, the tool prints thesetpcicommands that restore it by hand. - Reads and saves the AER registers of the card (Uncorrectable Error Mask, Correctable Error Mask, Uncorrectable Error Severity).
- Prints
remove <BDF>and removes the card from PCI (/sys/bus/pci/devices/<BDF>/remove). The driver takes the card down. - Sets and then clears the Secondary Bus Reset bit in the Bridge Control register of the parent bridge.
- Waits up to 30 seconds for the parent port to report that the link is active. Even if it does not report, the tool moves on to the next step.
- Rescans the parent bridge and waits until the card is enumerated again (up to 100 times, at 0.1-second intervals). If the PCI node of the card does not reappear, the tool fails.
- Compares the AER registers of the card with the values from step 2, restores them if they changed, and prints the AER status of the card.
- For a card that had
/dev/haNbefore the reset, waits up to about 12 seconds for the driver to probe the card again and the same node to appear. If it does not appear, printsdid not reappear - check dmesg for probe failureand fails. For a card that had no/dev/haNbefore the reset, waits the same time for thehadriver to bind the card. If it does not, the tool finishes step 9, printsthe card was reset but the ha driver did not bind itwith the command that binds it again, and ends with exit code1. The tool skips this check if thehamodule is not loaded, if the kernel's automatic probing is off, or if the card is kept for another driver (it was bound to another driver before the reset, ordriver_overrideis set). - If step 1 masked the path, checks that the card responds and then restores the masking on the path.
- Prints
Hot reset OK: <BDF>.
When interrupted or failing¶
- If the tool fails, or is interrupted by SIGINT, SIGTERM, or SIGHUP, before it removes the card, it only restores the path masking and ends.
- If it fails or is interrupted after removing the card, it clears the Secondary Bus Reset bit of the parent bridge, carries out steps 5 to 9 as a successful reset does, and ends. It restores the masking only if the card was enumerated again and responds; otherwise it deliberately leaves the masking in place (Path masking, step 4).
- After a signal, the tool finishes the cleanup and ends with exit code
129(SIGHUP),130(SIGINT), or143(SIGTERM). Signals that arrive during the cleanup are ignored. - SIGKILL skips this cleanup, so the card can remain unenumerated or the masking can remain in place. Do not use it. If the path was masked, you can restore it with the
setpcicommands printed in step 1.
Hot reset OK means that the card was enumerated on PCI again and, if the card had a /dev/haN before the reset, that the node reappeared, or, if it had none, that the ha driver bound the card. Check the firmware state with hyperaccel-smi or the sysfs health.
Path masking¶
If the parent port is not a root port and does not mask Surprise Down, the parent port reports a Surprise Down error when the SBR drops the link. On a server where platform firmware handles PCIe errors, that error can lead to a restart of the whole host. So while it resets such a card, ha-hotreset stops the ports on the path from the parent port up to the root port from reporting errors (path masking). This is the default; with --no-switch-link-drop, the tool refuses such a card instead of masking. A card whose parent port is a root port or already masks Surprise Down is not masked.
A masked error is not reported, so it does not reach the platform firmware. However, on a server whose platform firmware detects the link drop some other way, such as by reading the ports' error status registers directly, masking may not prevent a host restart. On a host where you use it for the first time, check first with Mask check.
- For each port on the path from the parent port to the root port, sets the masking bits of the AER Uncorrectable Error Mask and Correctable Error Mask, and at the root port, clears the three System Error reporting bits of Root Control. Proceeding downward from the root port, reads back all the written values to verify them. If even one port cannot be verified, restores all the ports that were changed and does not issue the SBR.
- Keeps the masking in place until the card is enumerated again and the driver probes it.
- Waits up to 5 seconds for the sysfs
tempnode of the card (utilif there is notemp) to report a value other thanunknown. If neither exists, printsno sysmon node under <path> - falling back to a plain waitand waits 5 seconds. If the card responds, clears the AER status bits of the path and of the card, restores the masking, and reads back to verify. If the restored values are wrong, prints thesetpcicommands that restore them manually. - If the card is not enumerated again, its device node does not come back, or it does not respond, the tool deliberately leaves the masking in place (
the mask is LEFT IN PLACE on purpose), prints thesetpcicommands that restore it, and ends in failure. Removing the masking above a card that does not respond can let that card's errors reach the platform. While the masking remains, real link errors on that path are not reported either, so after you deal with the card problem, restore it with the printed commands.
Host lock¶
The ports that are masked (switch upstream ports, root ports) are shared by every card below them. If two runs overlapped, one restoring the masking could leave the other's reset unprotected. So resets and --check-mask run one at a time on a host (/run/ha-hotreset.lock). If another ha-hotreset is running, the tool prints another ha-hotreset is running on this host - waiting up to 600s and waits up to 600 seconds. If the other run does not finish in that time, it prints the other ha-hotreset did not finish in 600s - nothing was touched and ends with exit code 3. --list only reads, so it does not wait.
Mask check¶
--check-mask[=SECONDS] checks whether path masking can work on this host. It does not remove the card or drop the link, and it does not run the pre-flight checks, so you can run it on a card that is in use. It requires root privileges, and you specify the target with --bdf, the same way as for a reset.
- Prints the current settings of each port on the path (
before:). - Masks the path in the same way as in Path masking and reads back to verify.
- Waits
SECONDSseconds and then checks that the masking is still in place. - Prints the verdict, clears the AER status bits of the path ports, and restores the masking.
| Verdict | Meaning |
|---|---|
VERDICT: the path could NOT be masked |
A port did not accept the masking. On this host, a card whose parent port does not mask Surprise Down cannot be reset |
VERDICT: the platform CLEARED the mask on: <port> |
The platform reverted the masking while the tool was waiting. On this host, a card whose parent port does not mask Surprise Down cannot be reset |
VERDICT: the whole path masks and holds. |
The whole path is masked and the masking holds. This means that writing the masking works without problems; it is not a guarantee that a reset is safe |
- While the tool waits, errors detected by the ports on the path are not reported, and the root port does not escalate the error messages it receives to System Error.
- If the verdict for every target is
the whole path masks and holds, the exit code is0; otherwise it is1.
Output¶
Progress goes to standard output (stdout), and warnings and reasons for refusal go to standard error (stderr).
List output¶
--list prints one line per card.
1 2 | |
| Verdict | Reset |
|---|---|
parent is a root port |
Possible |
parent is a <port type>, Surprise Down Error masked there |
Possible |
parent is a <port type> with Surprise Down Error unmasked; the path is masked for the reset |
Possible. The reset is done with path masking |
parent is a <port type> with Surprise Down Error unmasked |
Shown when --no-switch-link-drop is given. Not possible (NOT resettable) |
parent is a <port type> whose Surprise Down mask cannot be read (no AER capability) |
Not possible (NOT resettable) |
parent has no PCIe capability |
Not possible (NOT resettable) |
<port type> is one of switch upstream port, switch downstream port, PCIe-to-PCI bridge, PCI-to-PCIe bridge, and port type <N>.
1 2 | |
--list shows only the SBR verdict. Open handles, other devices below the parent bridge, and whether the registers on the path can be read are checked in the pre-flight checks when you reset.
Reset output¶
Resetting a single card prints the following lines in order. In between, the AER register values and status of the card and the path are also printed.
1 2 3 4 5 6 7 8 9 | |
For a card whose path is masked, the tool also prints masking <number of ports> ports: <port list> and the setpci commands that restore the masking by hand before remove, and Checking the card answers before unmasking (up to 5s)... and device answering (<node>=<value>) - safe to unmask before Hot reset OK.
Without the remove <BDF> line, the card was not touched.
Exit codes¶
| Code | Meaning |
|---|---|
0 |
Success. All target cards were reset, --list or --help was printed, or all --check-mask verdicts passed |
1 |
Failure. You are not root, pciutils or flock is missing, no card was found, the card is not a 20fb device, a reset step failed, the ha driver did not bind the card after the reset, or a --check-mask verdict did not pass. A reset that failed partway without a signal also ends here |
2 |
Usage error. This covers an unknown argument, options that cannot be combined, a non-numeric --check-mask value, a --bdf value that is missing or malformed, a --bdf value without a domain that matches no device or more than one device, and several cards being present without --bdf or --all |
3 |
Refused, and nothing was touched. A device node is open (--force overrides this), or another ha-hotreset run did not finish within 600 seconds (Host lock). Run it again later |
4 |
Refused, and nothing was touched. An SBR is not safe at this card's PCIe position: the parent's Surprise Down mask cannot be read, the path cannot be masked or the masking is not applied, the parent leaves it unmasked under --no-switch-link-drop, or another device is below the parent bridge |
129, 130, 143 |
Interrupted by SIGHUP, SIGINT, or SIGTERM during a reset or mask check. The tool runs the cleanup step and then ends |
When --all is refused for several reasons, 4 comes before 3, and 3 before 1. Judge failure by a non-zero code, not by 1.
After the reset¶
- A card that receives an SBR boots again from the firmware in the flash. This is why this tool is used to apply the new firmware after a firmware update.
- Because the driver probes the card afresh, the RAS counters and
last_hangthat the driver collected are cleared. - The card usually reappears with the same
/dev/haNnumber as before the reset. - Other cards are not reset. However, device-to-device copies from other cards that targeted the memory of this card fail.
See also¶
- FATAL and recovery — when to use which recovery method, and the configurations in which an SBR can be issued
- Firmware update — booting with the new firmware after an update
- ha-fw-update — resetting the card with this tool after an update
- hyperaccel-smi — checking the PCI address (Bus-Id) and state of a card
- sysfs nodes —
health,temp,inflight,stream_count - Kernel log messages —
dmesgoutput when a card is removed and probed again - Recovering an unresponsive card — the procedure for recovering a single card with this tool, from identifying the symptom to checking after the reset