ha-wipe¶
ha-wipe is a host command that clears what the previous user left on a card before the card goes to the next user. It resets a single card with ha-hotreset, fills all allocatable DRAM with zeros, and reads some blocks back to check. This page defines the install location, options, what is and is not cleared, the sequence of operations, output, and exit codes.
Always run it before you assign a card to another user. When an application closes its last handle, the driver reclaims only the resources of that handle; it does not reinitialize the device. So the data the previous user left in DRAM and the device-wide settings stay in place.
Included starting with release 2.6.0
ha-wipe is not present in releases earlier than 2.6.0. This tool is included in the driver package starting with 2.6.0.
Install location¶
ha-wipe is installed to /usr/bin/ha-wipe by the driver deb package hyperdriver-dkms.
Synopsis¶
1 2 | |
Requirements¶
- Run it as root on the host. If you are not root, it prints
ha-wipe: root requiredand ends with exit code1. - Run it in the host's PID namespace. In a container with its own PID namespace, it cannot see the host's processes, so it cannot find other processes that have the card open.
- The
hamodule must be loaded. Leave the module parameterha_hdma_min_bytesat its default0. If it is above 2 GiB, the tool does not start and ends with exit code1. - The driver must be version 2.6.0 or later. Check the loaded version with
cat /sys/module/ha/version. - There must be enough host memory for a buffer of zeros (256 MiB by default).
ha-hotresetis required, together with thesetpci,lspci(pciutils), andflock(util-linux) it uses. If any is missing, the tool ends with exit code1.- No process may have the card open, because the reset removes every allocation on the card. There is no override such as
--force.
Options¶
| Option | Description |
|---|---|
-d BDF, --bdf BDF |
PCI address of the card to wipe, in the format of hyperaccel-smi's Bus-Id (0000:01:00.0). 01:00.0 without the domain is accepted only if that address is unique across all PCI domains. Required |
--hotreset PATH |
Reset script to use. If omitted, the tool looks for ha-hotreset.sh in its own directory and then ha-hotreset on PATH |
--boot-timeout SEC |
Seconds to wait after the reset for the card to accept commands. The default is 60 |
-h, --help |
Prints the usage and ends |
What is and is not cleared¶
| Target | Result | How |
|---|---|---|
| Device settings (address translation settings, peripheral settings, and so on) | Back to their values right after boot | The SBR (Secondary Bus Reset) reboots the card |
| DRAM | Zero | DRAM keeps its contents across an SBR, and newly allocated memory is not zeroed. So the tool allocates all allocatable DRAM and fills it with zeros |
| SRAM | Not cleared | What the previous user left stays in place |
If the configuration does not allow an SBR, the tool ends with exit code 3. It does not fall back to a firmware re-initialization (a reset that re-initializes only the firmware, without an SBR), because that leaves the address translation and peripheral settings in place. For the configurations that allow an SBR, see FATAL and recovery.
Sequence of operations¶
- Checks before touching the card. Checks root privileges, the PCI address, the reset script and the commands it needs, and whether another
ha-wipeis running on the same card. It also sets up the host buffer of zeros at this point. If any of this fails, the tool ends with exit code1or2without touching the card. [1/4] Reset— Runsha-hotreset --bdf <BDF>and passes its output through unchanged. If a process has the card open or the configuration does not allow an SBR,ha-hotresetrefuses, andha-wipeends without having touched the card. For a card whose parent port does not mask Surprise Down, the path is masked during the reset, so on a host where you use it for the first time, read Path masking first.[2/4] Allocate— Waits up to--boot-timeoutseconds for the card to be probed again and the firmware to accept commands, then allocates all allocatable DRAM.[3/4] Zero— Fills all the allocated DRAM with zeros using the DMA engines.[4/4] Read back— Does not read all of DRAM back. For each channel, reads the start and end of every copy plus some blocks chosen at random, and checks that they are all zero.
Checking for other processes¶
Once the reset finishes, the device node is back, so another process can open the card until ha-wipe ends. ha-wipe checks for other processes that have the card open and for other allocations (sysfs dram_used, sram_used) right after the card is probed again, right after its allocation, and right before it ends. If it finds either, or cannot check, it ends with exit code 8.
It cannot find a process that opens and closes the card between two checks. Until ha-wipe ends, keep every user and container away from the card.
Time taken¶
On a card linked at PCIe Gen5 x8, filling 128 GiB of DRAM with zeros took 7 to 11 seconds. The reset and the firmware boot take additional time.
Signals¶
On SIGINT, SIGTERM, or SIGHUP, the tool stops before it moves on to the next step, or, while zeroing or reading back, before it moves on to the next channel. The exit code depends on the step it stopped in.
| Stopped during | Exit code |
|---|---|
| Before the reset | 2 |
| The reset, or the wait for the card after the reset | 4. 2 if ha-hotreset stopped before it removed the card from PCI |
| Allocation | 5 |
| Zeroing | 6 |
| Read-back | 7 |
- On a signal during the reset, the tool sends SIGTERM to
ha-hotreset, waits untilha-hotresetfinishes its cleanup and has the card probed again, and then ends. - A signal that the caller set to be ignored, as
nohupor a background job does, is ignored byha-hotresettoo. - Even if you end
ha-wipewith SIGKILL,ha-hotresetleaves the card untouched if it had not removed the card yet, and has the card probed again before it ends if it had. Either way the DRAM may not be zero, so runha-wipeagain.
Output¶
Progress goes to standard output (stdout). The reason for a failure goes to standard error (stderr) as one line that starts with ha-wipe:. The output of ha-hotreset is shown on standard output, with its standard output and standard error merged.
1 2 3 4 5 6 7 8 9 10 11 12 | |
If the tool fails after the reset has begun, ; the card was reset but its DRAM is not known to be zero is appended to the reason.
Exit codes¶
The exit codes of ha-wipe are numbered differently from those of ha-hotreset. A refusal by ha-hotreset with 3 (open node, host lock wait ran out) comes out as 2, and one with 4 (SBR not safe) as 3.
| Code | Meaning | What to do |
|---|---|---|
0 |
The card was reset, its DRAM was filled with zeros, and every block read back was zero | You can assign the card |
1 |
Nothing was touched. One of: a usage error; not root; the ha module not loaded; a required command missing; no card at that address; not enough host memory for the zero buffer; ha_hdma_min_bytes above 2 GiB; ha-hotreset failed before touching the card |
Fix what the message says and run it again |
2 |
Nothing was touched. Another process has the card open, another ha-wipe is running on the same card, another ha-hotreset run did not finish within 600 seconds (Host lock), or ha-wipe or ha-hotreset was stopped by a signal before the card was touched |
Run it again later |
3 |
The SBR was refused, and nothing was touched. The parent port's Surprise Down mask cannot be read, the path cannot be masked, or another device is below the parent bridge | This tool cannot wipe the card on this host. The ha-hotreset output just above gives the reason |
4 |
The reset failed or was interrupted. The reset, the re-probe of the card, or the firmware boot did not complete | Do not assign the card. Check the card's state with hyperaccel-smi or the sysfs health, then run it again |
5 |
The DRAM allocation failed, or a signal stopped the tool during it | Do not assign the card; run it again |
6 |
Zeroing failed, or a signal stopped the tool during it | Do not assign the card; run it again |
7 |
Data read back was not zero or could not be read, or a signal stopped the tool during the read-back | Do not assign the card; run it again |
8 |
After the reset and before the DRAM was zero, another process opened the card or allocated memory on it. That process may have read the previous user's data | Stop that process before you run it again. If the message gives no PID, find it with sudo fuser -v /dev/haN or sudo lsof /dev/haN |
With exit codes 4 to 8, the reset has begun but the DRAM is not known to be zero. Do not assign the card; run the tool again until it ends with 0. If the same code repeats without a signal, collect dmesg and contact HyperAccel.
Main messages¶
| Message | Code | Meaning |
|---|---|---|
another ha-wipe is running on <BDF> |
2 |
Another ha-wipe is running on the same card |
<BDF> is in use, or another ha-hotreset held the host lock (see above); nothing was touched |
2 |
A process has the card open, or another ha-hotreset run did not finish. The ha-hotreset output just above gives the reason |
an SBR is not safe for <BDF> on this host; nothing was touched |
3 |
An SBR cannot be issued for this card on this host. The ha-hotreset output just above gives the reason |
stopped by <signal>; nothing was touched |
2 |
A signal stopped the tool before the reset |
stopped by <signal> before ha-hotreset touched <BDF>; nothing was touched |
2 |
During the reset, the tool received a signal or ha-hotreset was ended by one, before the card was touched |
the firmware on haN is not ready within <seconds> s |
4 |
The firmware did not accept commands within --boot-timeout |
/dev/haN was opened by pid(s) <PID list> after the reset |
8 |
Another process opened the card after the reset |
another process allocated on haN after the reset (...) |
8 |
Another process allocated memory on the card after the reset |
DRAM channel <N> at 0x<address> is not zero |
7 |
A block that was read back holds non-zero data |
See also¶
- ha-hotreset — the reset that
ha-wipeuses, path masking, and the host lock - FATAL and recovery — what an SBR does and the configurations in which it can be issued
- hyperaccel-smi — checking the PCI address (
Bus-Id) of a card - sysfs nodes —
dram_used,sram_used,health - Parameter reference —
ha_hdma_min_bytes