Skip to content

ha-wipe

ha-wipe is a host command that clears what the previous user left on a card before the card goes to the next user. It resets a single card with ha-hotreset, fills all allocatable DRAM with zeros, and reads some blocks back to check. This page defines the install location, options, what is and is not cleared, the sequence of operations, output, and exit codes.

Always run it before you assign a card to another user. When an application closes its last handle, the driver reclaims only the resources of that handle; it does not reinitialize the device. So the data the previous user left in DRAM and the device-wide settings stay in place.

Included starting with release 2.6.0

ha-wipe is not present in releases earlier than 2.6.0. This tool is included in the driver package starting with 2.6.0.


Install location

ha-wipe is installed to /usr/bin/ha-wipe by the driver deb package hyperdriver-dkms.

Synopsis

1
2
ha-wipe -d BDF [--hotreset PATH] [--boot-timeout SEC]
ha-wipe --help

Requirements

  • Run it as root on the host. If you are not root, it prints ha-wipe: root required and ends with exit code 1.
  • Run it in the host's PID namespace. In a container with its own PID namespace, it cannot see the host's processes, so it cannot find other processes that have the card open.
  • The ha module must be loaded. Leave the module parameter ha_hdma_min_bytes at its default 0. If it is above 2 GiB, the tool does not start and ends with exit code 1.
  • The driver must be version 2.6.0 or later. Check the loaded version with cat /sys/module/ha/version.
  • There must be enough host memory for a buffer of zeros (256 MiB by default).
  • ha-hotreset is required, together with the setpci, lspci (pciutils), and flock (util-linux) it uses. If any is missing, the tool ends with exit code 1.
  • No process may have the card open, because the reset removes every allocation on the card. There is no override such as --force.

Options

Option Description
-d BDF, --bdf BDF PCI address of the card to wipe, in the format of hyperaccel-smi's Bus-Id (0000:01:00.0). 01:00.0 without the domain is accepted only if that address is unique across all PCI domains. Required
--hotreset PATH Reset script to use. If omitted, the tool looks for ha-hotreset.sh in its own directory and then ha-hotreset on PATH
--boot-timeout SEC Seconds to wait after the reset for the card to accept commands. The default is 60
-h, --help Prints the usage and ends

What is and is not cleared

Target Result How
Device settings (address translation settings, peripheral settings, and so on) Back to their values right after boot The SBR (Secondary Bus Reset) reboots the card
DRAM Zero DRAM keeps its contents across an SBR, and newly allocated memory is not zeroed. So the tool allocates all allocatable DRAM and fills it with zeros
SRAM Not cleared What the previous user left stays in place

If the configuration does not allow an SBR, the tool ends with exit code 3. It does not fall back to a firmware re-initialization (a reset that re-initializes only the firmware, without an SBR), because that leaves the address translation and peripheral settings in place. For the configurations that allow an SBR, see FATAL and recovery.

Sequence of operations

  1. Checks before touching the card. Checks root privileges, the PCI address, the reset script and the commands it needs, and whether another ha-wipe is running on the same card. It also sets up the host buffer of zeros at this point. If any of this fails, the tool ends with exit code 1 or 2 without touching the card.
  2. [1/4] Reset — Runs ha-hotreset --bdf <BDF> and passes its output through unchanged. If a process has the card open or the configuration does not allow an SBR, ha-hotreset refuses, and ha-wipe ends without having touched the card. For a card whose parent port does not mask Surprise Down, the path is masked during the reset, so on a host where you use it for the first time, read Path masking first.
  3. [2/4] Allocate — Waits up to --boot-timeout seconds for the card to be probed again and the firmware to accept commands, then allocates all allocatable DRAM.
  4. [3/4] Zero — Fills all the allocated DRAM with zeros using the DMA engines.
  5. [4/4] Read back — Does not read all of DRAM back. For each channel, reads the start and end of every copy plus some blocks chosen at random, and checks that they are all zero.

Checking for other processes

Once the reset finishes, the device node is back, so another process can open the card until ha-wipe ends. ha-wipe checks for other processes that have the card open and for other allocations (sysfs dram_used, sram_used) right after the card is probed again, right after its allocation, and right before it ends. If it finds either, or cannot check, it ends with exit code 8.

It cannot find a process that opens and closes the card between two checks. Until ha-wipe ends, keep every user and container away from the card.

Time taken

On a card linked at PCIe Gen5 x8, filling 128 GiB of DRAM with zeros took 7 to 11 seconds. The reset and the firmware boot take additional time.

Signals

On SIGINT, SIGTERM, or SIGHUP, the tool stops before it moves on to the next step, or, while zeroing or reading back, before it moves on to the next channel. The exit code depends on the step it stopped in.

Stopped during Exit code
Before the reset 2
The reset, or the wait for the card after the reset 4. 2 if ha-hotreset stopped before it removed the card from PCI
Allocation 5
Zeroing 6
Read-back 7
  • On a signal during the reset, the tool sends SIGTERM to ha-hotreset, waits until ha-hotreset finishes its cleanup and has the card probed again, and then ends.
  • A signal that the caller set to be ignored, as nohup or a background job does, is ignored by ha-hotreset too.
  • Even if you end ha-wipe with SIGKILL, ha-hotreset leaves the card untouched if it had not removed the card yet, and has the card probed again before it ends if it had. Either way the DRAM may not be zero, so run ha-wipe again.

Output

Progress goes to standard output (stdout). The reason for a failure goes to standard error (stderr) as one line that starts with ha-wipe:. The output of ha-hotreset is shown on standard output, with its standard output and standard error merged.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
[1/4] Reset (SBR via ha-hotreset)
=== 0000:01:00.0 (parent 0000:00:01.1, /dev/ha0) ===
...
remove 0000:01:00.0
...
Hot reset OK: 0000:01:00.0
[2/4] Allocate
      ha0: DRAM <size> B
[3/4] Zero
      done in <seconds> s
[4/4] Read back
ha0 0000:01:00.0: reset and DRAM zeroed

If the tool fails after the reset has begun, ; the card was reset but its DRAM is not known to be zero is appended to the reason.

Exit codes

The exit codes of ha-wipe are numbered differently from those of ha-hotreset. A refusal by ha-hotreset with 3 (open node, host lock wait ran out) comes out as 2, and one with 4 (SBR not safe) as 3.

Code Meaning What to do
0 The card was reset, its DRAM was filled with zeros, and every block read back was zero You can assign the card
1 Nothing was touched. One of: a usage error; not root; the ha module not loaded; a required command missing; no card at that address; not enough host memory for the zero buffer; ha_hdma_min_bytes above 2 GiB; ha-hotreset failed before touching the card Fix what the message says and run it again
2 Nothing was touched. Another process has the card open, another ha-wipe is running on the same card, another ha-hotreset run did not finish within 600 seconds (Host lock), or ha-wipe or ha-hotreset was stopped by a signal before the card was touched Run it again later
3 The SBR was refused, and nothing was touched. The parent port's Surprise Down mask cannot be read, the path cannot be masked, or another device is below the parent bridge This tool cannot wipe the card on this host. The ha-hotreset output just above gives the reason
4 The reset failed or was interrupted. The reset, the re-probe of the card, or the firmware boot did not complete Do not assign the card. Check the card's state with hyperaccel-smi or the sysfs health, then run it again
5 The DRAM allocation failed, or a signal stopped the tool during it Do not assign the card; run it again
6 Zeroing failed, or a signal stopped the tool during it Do not assign the card; run it again
7 Data read back was not zero or could not be read, or a signal stopped the tool during the read-back Do not assign the card; run it again
8 After the reset and before the DRAM was zero, another process opened the card or allocated memory on it. That process may have read the previous user's data Stop that process before you run it again. If the message gives no PID, find it with sudo fuser -v /dev/haN or sudo lsof /dev/haN

With exit codes 4 to 8, the reset has begun but the DRAM is not known to be zero. Do not assign the card; run the tool again until it ends with 0. If the same code repeats without a signal, collect dmesg and contact HyperAccel.

Main messages

Message Code Meaning
another ha-wipe is running on <BDF> 2 Another ha-wipe is running on the same card
<BDF> is in use, or another ha-hotreset held the host lock (see above); nothing was touched 2 A process has the card open, or another ha-hotreset run did not finish. The ha-hotreset output just above gives the reason
an SBR is not safe for <BDF> on this host; nothing was touched 3 An SBR cannot be issued for this card on this host. The ha-hotreset output just above gives the reason
stopped by <signal>; nothing was touched 2 A signal stopped the tool before the reset
stopped by <signal> before ha-hotreset touched <BDF>; nothing was touched 2 During the reset, the tool received a signal or ha-hotreset was ended by one, before the card was touched
the firmware on haN is not ready within <seconds> s 4 The firmware did not accept commands within --boot-timeout
/dev/haN was opened by pid(s) <PID list> after the reset 8 Another process opened the card after the reset
another process allocated on haN after the reset (...) 8 Another process allocated memory on the card after the reset
DRAM channel <N> at 0x<address> is not zero 7 A block that was read back holds non-zero data

See also