Parameter reference¶
This page defines the module parameters that ha.ko accepts. For each parameter, it lists the type, permissions, default, valid range, the meaning of 0, and when a changed value takes effect. How to change values and commonly used settings are in Configuring and using parameters, and why each feature behaves as it does is explained on the feature pages under Operations guide.
At a glance¶
| Parameter | Type | Permissions | Default | Description |
|---|---|---|---|---|
ha_tw_wide_access |
bool |
0644 |
1 (Y) |
Write device memory with 32-byte AVX stores |
ha_hdma_completion |
string |
0444 |
auto |
How the driver receives DMA engine completion. auto, irq (interrupt), poll (polling) |
ha_auto_reset |
bool |
0644 |
0 (N) |
The driver automatically resets a device that has become FATAL |
ha_aer_enable |
bool |
0444 |
0 (N) |
Use PCIe AER error reporting |
ha_ready_timeout_ms |
uint |
0644 |
3000 |
How long (ms) to wait for the firmware to report ready after a reset |
ha_devcmd_ceiling_ms |
uint |
0644 |
12000 |
Ceiling (ms) on waiting for a device command response |
ha_wd_hang_ms |
uint |
0644 |
8000 |
Threshold (ms) at which the watchdog treats a command wait as a stall |
ha_wd_hdma_hang_ms |
uint |
0644 |
4000 |
Threshold (ms) at which the watchdog treats a DMA wait as a stall |
ha_wd_poll_ms |
uint |
0644 |
500 |
Watchdog check interval (ms) |
ha_wd_preempt |
bool |
0644 |
0 (N) |
Cut off a wait that has been judged to be a stall |
ha_sysmon_poll_ms |
uint |
0644 |
200 |
Telemetry polling interval (ms) |
ha_sysmon_fail_limit |
uint |
0644 |
15 |
Number of consecutive failures before telemetry polling of a non-responding device is stopped |
ha_tw_pin_wait_ms |
uint |
0644 |
12000 |
Ceiling (ms) on how long a device-to-device copy waits for the other card's window |
ha_host_mem_max_mb |
uint |
0644 |
1024 |
Cap (MiB) on the total host memory allocated for cards to read and write directly |
ha_abi_prefetch |
bool |
0644 |
1 (Y) |
Prefetch the copy of the device kernel argument block (ABI) so that it overlaps the execution of the preceding device kernel |
ha_hdma_min_bytes |
uint |
0644 |
0 |
Smallest copy between host and device that goes to the DMA engines (bytes) |
ha_hdma_cpu_latency_us |
int |
0444 |
0 |
CPU wake-up latency requested while the DMA engines copy (µs) |
The Permissions column means the following.
0644: root can change it at run time in/sys/module/ha/parameters/. When the change takes effect differs from parameter to parameter and is described in each section.0444: can be set only when the module is loaded. To change it, you must reload the module.
Every parameter applies to the module as a whole. You cannot give different values to different cards.
Device configuration¶
ha_tw_wide_access¶
| Type | bool |
| Permissions | 0644 |
| Default | 1 (Y) |
| Takes effect | From the next write to device memory |
When turned on, the driver writes device memory, such as the rings of the DMA engines, with 32-byte AVX stores where the CPU supports them. When turned off, it uses 64-bit stores.
Some hypervisors trap BAR accesses and emulate the instruction. Such a hypervisor cannot decode AVX stores, so the VM stops at the first ring write. A passthrough VM maps the BAR directly into the guest, so it does not need this setting.
When to change it: Use the default. Set it to 0 only in a VM whose hypervisor traps and emulates BAR accesses, and put it in the option file so that it applies from the first write. For the option file, see Configuring and using parameters.
ha_hdma_completion¶
| Type | string |
| Permissions | 0444 (load time only) |
| Default | auto |
| Value | auto, irq, poll |
| Other values | The module fails to load and ha: `<value>' invalid for parameter `ha_hdma_completion' is logged |
This parameter sets how the driver notices that a DMA engine has finished a transfer.
auto: The driver takes a per-engine MSI-X interrupt only when it can confirm bare metal. On x86 it treats the host as bare metal when the CPU has nohypervisorflag, and on arm64 when the kernel runs at the hypervisor privilege level (EL2). If neither applies (a VM guest, or an arm64 server whose kernel runs at EL1), it behaves likepoll.irq: The driver always takes MSI-X interrupts.poll: The driver checks at short intervals for a completion mark that the engine leaves in host memory. A VM guest uses this mode because the engine's interrupt does not reach the guest. Copy results are the same, but because the driver keeps waking up to check while it waits, it uses more CPU than interrupts.
At load time, the kernel log shows the chosen mode for each card, for example HDMA completion: MSI-X (ha_hdma_completion=auto). /sys/module/ha/parameters/ha_hdma_completion shows only the configured value (such as auto), so check the mode actually chosen with this line.
1 | |
When to change it: Normally use auto. Change it only when auto chooses wrongly.
- If every DMA copy in a VM fails and logs
ll: engine stopped but its completion MSI-X never arrived, setpollinside the VM guest. This happens in an x86 guest whose hypervisor hides thehypervisorflag, or in an arm64 guest set up under nested virtualization to run its own hypervisor. - If an arm64 bare-metal server shows
HDMA completion: polling (ha_hdma_completion=auto), its kernel runs at EL1. Interrupts reach the driver on bare metal, so withirqit receives completion by interrupt and does not spend CPU on polling. Usingirqin a VM guest makes every DMA copy fail.
For example, write options ha ha_hdma_completion=poll in /etc/modprobe.d/ha.conf and reload the module. See Setting values in an option file.
Recovery¶
ha_auto_reset¶
| Type | bool |
| Permissions | 0644 |
| Default | 0 (N) |
| Takes effect | The value at the moment the device becomes FATAL is used |
When turned on, the driver resets a device by itself when the device enters the FATAL state (health is faulted). It first re-initializes the firmware, and if that fails, it proceeds to SBR (Secondary Bus Reset) only when the parent port is a root port or the Surprise Down error is masked on the parent port. When the reset starts, dmesg shows a line that starts with auto-TDR:.
- When off (default): A FATAL device stays FATAL until someone resets it. An application resets it with the SDK's reset function, or an operator resets it with ha-hotreset.
- Scope of reclamation: Like a manual reset, an automatic reset reclaims all tasks and allocations that were in progress on that device. Applications that were using the device must reopen it.
- If it fails: If an automatic reset or a reset requested through the SDK fails, automatic reset is not tried again.
healthstaysfaulted. After a later reset succeeds or the device returns to the ready state, automatic reset works again. - During a firmware update: Automatic reset is not performed while an update is in progress or while its result has not been confirmed. This is to avoid corrupting the flash that is being written. A device skipped in this way stays FATAL. If
healthis stillfaultedafter the update ends, reset it by hand. However, do not reset it ifha-fw-updateended with exit code 4, ended without reporting an exit code, orfw_update: no answer from firmwarewas logged. In that case the flash may have been only partly written, and you cannot rerun the update while the device is FATAL, so contact HyperAccel. See When no response is received. - Devices that are already FATAL: Reset by hand any device that became FATAL before you turned this parameter on.
When to change it: Turn it on for systems that must recover from a hang without human intervention. When you turn on ha_wd_preempt, turn this on together with it. For the effect and sequence of the reset, see FATAL and recovery.
ha_aer_enable¶
| Type | bool |
| Permissions | 0444 (load time only) |
| Default | 0 (N) |
When turned on, a warning AER reporting enabled: the kernel's fatal remedy is a parent-bridge SBR is logged in dmesg every time a card is attached to the driver, and on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions) PCIe AER (Advanced Error Reporting) error reporting is turned on for this device. On kernel 6.4 and later and on RHEL 9.4 and later, the driver does not change the device's error reporting settings, and whether errors are reported follows the kernel's and the platform's AER settings.
To recover from a fatal error reported through AER, the kernel issues an SBR on the parent bridge. This SBR does not go through the conditions that apply when the driver issues one itself (the parent is a root port, or Surprise Down is masked). Therefore, on a path where Surprise Down is not masked, the platform may treat this link loss as an unrecoverable error and reboot the host. This is why the default is 0.
Even when it is off, the kernel's PCIe error recovery can run
When the parent port detects an error, when DPC (Downstream Port Containment) is triggered, or when another device under the same bridge has a fatal error, the kernel runs the recovery procedure for this device regardless of this parameter.
When to change it: Turn it on only if you have confirmed that a parent-bridge SBR is safe on that host (Surprise Down is masked on the ports along the path) and you want to use the kernel's AER recovery. For details, see FATAL and recovery.
ha_ready_timeout_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 3000 |
| Valid range | 0~14400000. A value greater than 14400000 is treated as 14400000 |
Meaning of 0 |
The driver checks the firmware once and does not wait |
| Takes effect | From the next reset. A reset that is already waiting uses the value from when it started waiting |
How long the driver waits, after a reset, for the firmware to report that it is ready. It applies to a reset requested through the SDK, an automatic reset, and a reset that the kernel issues.
- If the time runs out: The kernel log shows
device came back but firmware did not report ready (<err>); reset reported as failed, orfirmware did not report ready after host reset (<err>); ...for a reset that the kernel issued. Application requests are rejected until the firmware is ready. - Side effect: Module unload waits for an automatic reset that is in progress, so a larger value can also delay the unload.
When to change it: Raise it when these messages appear but fw_version reports a version shortly afterward, which means that the firmware needs longer than this to become ready. Otherwise, use the default.
Hang detection and the watchdog¶
The driver waits for responses to commands sent to the device and for DMA transfers to complete. The parameters below set the ceilings on those waits and the watchdog, which runs separately from the waits and finds waits that do not finish for a long time. For how it works and the diagnostic procedure, see Hang detection and the watchdog.
ha_devcmd_ceiling_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 12000 |
| Valid range | Lower bound 100. A smaller value is treated as 100 |
Meaning of 0 |
None. It is treated as the lower bound 100. The wait does not become unlimited |
| Takes effect | From waits that start after the change |
This is the maximum time the driver waits for the response to a batch of commands sent to the device firmware. If there is no response within this time, the driver judges the device to be non-responding (hang). It increments devcmd_timeouts by 1, records the progress at that point and the verdict in last_hang, and logs a Devcmd no response after <N> ms (device hang): ... line and a diagnostic dump in dmesg. The task that was waiting ends with an error. If this determination is reached while waiting for a task submitted by an application, the device enters the FATAL state (health is faulted), and if ha_auto_reset is on, an automatic reset follows.
- Long-running commands: For commands that take a long time, such as filling device memory (memset) or a firmware update, the driver adds extra time, with some margin over the expected duration, equally to this ceiling and to
ha_wd_hang_ms. You do not need to raise this value because of such tasks. - Reset: When resetting through an automatic reset or the SDK's reset function, the driver also waits up to this ceiling for the response to the firmware re-initialization. If there is no response within that time, it proceeds to SBR when the conditions are met. For the conditions, see
ha_auto_reset. - Waits requested with no time limit: These do not end at this ceiling. Instead, once this time has passed, a
Devcmd waiting <N> ms with no timeout requested ...warning starts to be logged in dmesg and repeats every 10 seconds.
When to change it: Raise it when one command of a normal workload takes longer than the default ceiling and is judged to be a hang. By the amount you raise it, the point at which a non-responding device is judged to be a hang is also delayed. Lowering it is not recommended. The firmware also reports command failure through its own timeout, and if you set this value shorter than that, the driver judges a hang before the firmware reports the cause. When you change it, keep it greater than ha_wd_hang_ms + ha_wd_poll_ms.
ha_wd_hang_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 8000 |
Meaning of 0 |
Turns off command wait monitoring |
| Takes effect | From the next watchdog check. The new threshold is also used for waits already in progress |
This is the threshold at which the watchdog judges a wait for a device command response to be a stall. The watchdog runs for each card separately from the waiting task, and when a wait exceeds this time, it logs watchdog: devcmd wait stalled (devcmd=<N> ms hdma=<N> ms) - no completion within the threshold in dmesg. It is logged only once per stall. Its purpose is to observe a stall before the wait ceiling (ha_devcmd_ceiling_ms) is reached, and also for waits requested with no time limit.
With the default values, it only reports and leaves the wait as it is. To cut off the wait, turn on ha_wd_preempt. The extra time that the driver adds for long-running commands is added equally to this threshold.
If this value + ha_wd_poll_ms is greater than or equal to ha_devcmd_ceiling_ms, this monitoring cannot fire. The driver does not change the values and logs a warning (Relationships between parameters).
ha_wd_hdma_hang_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 4000 |
Meaning of 0 |
Turns off DMA wait monitoring |
| Takes effect | From the next watchdog check |
This is the threshold at which the watchdog judges a wait for a DMA transfer to complete to be a stall. Host–device and device–device data copies are handled by the DMA engines of the card. When it makes that determination, it logs a watchdog: hdma wait stalled ... line in the same format as for ha_wd_hang_ms.
The ceiling on waiting for one DMA transfer to complete is 5 seconds, and it cannot be changed with a parameter. If the transfer does not finish within 5 seconds, it fails and hdma_timeouts increases. Therefore, if this value + ha_wd_poll_ms is 5000 or more, this monitoring cannot fire, and the driver logs a warning.
ha_wd_poll_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 500 |
| Valid range | Lower bound 50. A smaller value is treated as 50 |
Meaning of 0 |
Does not turn off the watchdog. It is treated as 50 |
| Takes effect | The previous interval applies until the first check after the change, and the new interval is used from then on |
This is the interval at which the watchdog checks the wait time of each card. To turn off the watchdog, set the two thresholds (ha_wd_hang_ms, ha_wd_hdma_hang_ms) to 0, not this value. Each threshold + this value must be less than the corresponding wait ceiling.
ha_wd_preempt¶
| Type | bool |
| Permissions | 0644 |
| Default | 0 (N) |
| Takes effect | From the next stall determination |
When turned on, the watchdog does not stop at reporting when it judges a stall but cuts off that wait. It cuts off only the side newly judged to be a stall (the command wait or the DMA wait). A wait that is cut off ends with the same error as when the device does not respond, and if it was executing an application task, the device becomes FATAL and stays that way until it is reset. When a wait requested with no time limit is cut off, only that request fails and the device does not become FATAL. For the behavior in each case, see Hang detection and the watchdog.
- Use it together with
ha_auto_reset: Ifha_auto_resetis off, a device that becomes FATAL waits for a manual reset. In that case, dmesg shows a warning that starts withwatchdog: preempting without ha_auto_reset. - It also cuts off waits with no time limit: With the default values, a wait requested with no time limit continues until it finishes. When this is on, such a wait is also cut off at the threshold. The watchdog cannot distinguish a long-running normal task from a non-responding device by time alone.
- It does not cut off waits during a firmware update.
When to change it: Turn it on together with ha_auto_reset=1 on systems where you want to clean up a non-responding device sooner than the wait ceiling and recover automatically.
Telemetry¶
ha_sysmon_poll_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 200 |
| Valid range | 50~2500. A value outside the range is treated as the nearest end value |
Meaning of 0 |
Does not turn off polling. It is treated as the fastest interval (50) |
| Takes effect | From the next poll. A poll that is already scheduled uses the previous interval |
This is the interval at which the driver fetches temperature, power, rail voltage and current, and utilization from the firmware. The sysfs values temp, temp_avg, temp_blocks, power_mw, rails, and util, and the hwmon values, are refreshed at this interval, and reading more often than the interval returns the same sample. If the last sample is older than 5 seconds, these nodes become unknown. The upper bound of 2500 ms is set so that the nodes do not change to unknown even if one poll is late.
When to change it: Raise it to reduce telemetry round trips, and lower it to see changes in finer detail. If you raise it, the decision to stop polling a non-responding device (ha_sysmon_fail_limit) is delayed by the same amount. For how values are updated, see Telemetry.
ha_sysmon_fail_limit¶
| Type | uint |
| Permissions | 0644 |
| Default | 15 |
Meaning of 0 |
Polling is not stopped even when there is no response |
| Takes effect | From the next poll |
If telemetry polling fails this many times in a row, polling of that card stops. A poll counts as a failure if it receives no value at all from the firmware. This is because continuing to read a non-responding device causes a PCIe Completion Timeout, and a platform that handles errors in a firmware-first manner can reset the entire server because of it.
- When polling stops, dmesg shows
sys_mon: no response for <N> consecutive polls (<M> ms) - suspending telemetry polling. ..., and the telemetry nodes fetched from the firmware becomeunknown. - A failure is counted once per polling interval, so it takes roughly
ha_sysmon_fail_limit×ha_sysmon_poll_ms(about 3 seconds with the default values), plus the response wait time of each poll, before polling stops. - When the device is reset, polling restarts and
sys_mon: resuming telemetry pollingis logged. Polling that has already stopped does not restart even if you change this parameter.
When to change it: Use the default. Do not use 0 on firmware-first platforms.
Multiple devices¶
ha_tw_pin_wait_ms¶
| Type | uint (ms) |
| Permissions | 0644 |
| Default | 12000 |
| Valid range | 0~14400000. A value greater than 14400000 is treated as 14400000 |
Meaning of 0 |
If no window is left, it fails immediately without waiting |
| Takes effect | From waits that start after the change. A request that is already waiting uses the value from when it started waiting |
A device-to-device copy (D2D) occupies a translation window of the other card to access that card's memory. There is a limit on the number of windows that can be occupied at the same time on one card, so when the limit is reached, a new copy waits for a window to be released. This parameter is the ceiling on that wait.
When the ceiling passes, that copy fails and a warning TW pin: <n>/<limit> slots pinned for <N>ms, giving up (...) is logged in dmesg. If the same warning repeats in a short time, only some of the warnings are logged. It is a transient failure that you can retry once a window is released.
When to change it: Raise it when device-to-device copies concentrate on one card and fail after waiting. To get a failure immediately without waiting, set it to 0. For device-to-device copies, see Multiple devices.
ha_host_mem_max_mb¶
| Type | uint (MiB) |
| Permissions | 0644 |
| Default | 1024 |
Meaning of 0 |
No cap |
| Takes effect | From the next allocation. Memory that is already allocated is left as is |
An application can get host memory from the driver and let a card read and write that memory directly. The kernel does not reclaim or swap this memory. It is allocated on the card's NUMA node, or below 4 GiB (ZONE_DMA32) when that node's memory lies above the address range the card can reach. This parameter is the cap on the total of this memory held by all cards and processes.
- An allocation that would take the total past the cap fails. Memory that is already allocated is not affected.
- Memory that a process has let go of is not freed while a card that used it is running work, is being reset, is FATAL, or failed its last reset, and it counts toward the total until it is freed.
- The memory counts toward the memory cgroup usage of the process that allocated it.
- While a process holds this memory, the
hamodule cannot be unloaded, even if that process has closed/dev/haN.modprobe -r hafails because the module is in use.
When to change it: Use the default. Raise it when several jobs on one host use this memory at the same time and their allocations hit the cap. Any user can open the device nodes, so with 0 an ordinary user can use up the host's memory below 4 GiB.
Performance¶
ha_abi_prefetch¶
| Type | bool |
| Permissions | 0644 |
| Default | 1 (Y) |
| Takes effect | From the next determination. When turned on, it also applies to tasks already in the queue |
This is a performance optimization that copies the device kernel argument block (ABI) that the next task will use from host memory to device memory in advance, while the preceding device kernel in the same stream is running. Its purpose is to reduce the total time by overlapping the copy with device kernel execution.
- The result is the same as running in order. If a preceding task can still read or write the destination of that copy, the copy is not prefetched and runs in the original order. A copy that cannot be prefetched is not an error.
- With
0, all copies run in submission order. This is the behavior from before this optimization existed. - See the effect in the sysfs
abi_prefetch_stats.issuedis the number of copies started ahead of time, andconsumedis the number of times the prefetched result was actually used. For the remaining fields, see sysfs nodes.
When to change it: Turn it off when you want to compare behavior or performance without prefetching.
ha_hdma_min_bytes¶
| Type | uint |
| Permissions | 0644 |
| Default | 0 |
| Takes effect | From the next copy to start |
The size that decides whether a copy between host and device (H2D, D2H, file load) goes to the DMA engines or is done by the CPU (PIO).
- With
0, copies of every size go to the DMA engines. - With
N, copies smaller thanNbytes are done by PIO. PIO is much slower than the DMA engines for large copies.
When to change it: Normally keep the default 0. Above 2 GiB, ha-wipe does not start.
ha_hdma_cpu_latency_us¶
| Type | int |
| Permissions | 0444 |
| Default | 0 |
| Takes effect | When the module is loaded |
A wake-up latency request that the driver places on every CPU while the DMA engines copy and for 10 ms after the last copy ends. It keeps the thread that waits for a copy from waking late out of a deep C-state and leaving the engine idle. Copy results and ordering do not change; only throughput and CPU power during copies change.
- With
0, every C-state (CPU power-saving state) is blocked, C1 included. While copies run, idle CPUs spin in a wait loop instead of saving power. - With
N, only C-states whose wake-up time (exit latency) isNµs or less are allowed. - With
-1, no request is placed. A value below-1is changed to0, with a warning in the kernel log. - When no copy runs, there is no constraint, so a
/dev/cpu_dma_latencyvalue set from user space applies as it is. Where both apply, the stricter one is used. - At load, the kernel log shows
ha: HDMA copies hold a <N> us CPU wake-up latency request while they run, or with-1,ha: HDMA copies hold no CPU wake-up latency request (ha_hdma_cpu_latency_us=-1).
When to change it: Set it to N or -1 when you need to lower CPU power during copies. Copy throughput may drop.
See also¶
- Hang detection and the watchdog — how the wait ceilings and the watchdog parameters are used
- FATAL and recovery —
ha_auto_reset,ha_aer_enable, and the SBR conditions - Telemetry —
ha_sysmon_poll_ms,ha_sysmon_fail_limit, andunknown - Multiple devices —
ha_tw_pin_wait_ms - sysfs nodes
- Kernel log messages
- hyperaccel-smi