Skip to content

Parameter reference

This page defines the module parameters that ha.ko accepts. For each parameter, it lists the type, permissions, default, valid range, the meaning of 0, and when a changed value takes effect. How to change values and commonly used settings are in Configuring and using parameters, and why each feature behaves as it does is explained on the feature pages under Operations guide.


At a glance

Parameter Type Permissions Default Description
ha_tw_wide_access bool 0644 1 (Y) Write device memory with 32-byte AVX stores
ha_hdma_completion string 0444 auto How the driver receives DMA engine completion. auto, irq (interrupt), poll (polling)
ha_auto_reset bool 0644 0 (N) The driver automatically resets a device that has become FATAL
ha_aer_enable bool 0444 0 (N) Use PCIe AER error reporting
ha_ready_timeout_ms uint 0644 3000 How long (ms) to wait for the firmware to report ready after a reset
ha_devcmd_ceiling_ms uint 0644 12000 Ceiling (ms) on waiting for a device command response
ha_wd_hang_ms uint 0644 8000 Threshold (ms) at which the watchdog treats a command wait as a stall
ha_wd_hdma_hang_ms uint 0644 4000 Threshold (ms) at which the watchdog treats a DMA wait as a stall
ha_wd_poll_ms uint 0644 500 Watchdog check interval (ms)
ha_wd_preempt bool 0644 0 (N) Cut off a wait that has been judged to be a stall
ha_sysmon_poll_ms uint 0644 200 Telemetry polling interval (ms)
ha_sysmon_fail_limit uint 0644 15 Number of consecutive failures before telemetry polling of a non-responding device is stopped
ha_tw_pin_wait_ms uint 0644 12000 Ceiling (ms) on how long a device-to-device copy waits for the other card's window
ha_host_mem_max_mb uint 0644 1024 Cap (MiB) on the total host memory allocated for cards to read and write directly
ha_abi_prefetch bool 0644 1 (Y) Prefetch the copy of the device kernel argument block (ABI) so that it overlaps the execution of the preceding device kernel
ha_hdma_min_bytes uint 0644 0 Smallest copy between host and device that goes to the DMA engines (bytes)
ha_hdma_cpu_latency_us int 0444 0 CPU wake-up latency requested while the DMA engines copy (µs)

The Permissions column means the following.

  • 0644: root can change it at run time in /sys/module/ha/parameters/. When the change takes effect differs from parameter to parameter and is described in each section.
  • 0444: can be set only when the module is loaded. To change it, you must reload the module.

Every parameter applies to the module as a whole. You cannot give different values to different cards.


Device configuration

ha_tw_wide_access

Type bool
Permissions 0644
Default 1 (Y)
Takes effect From the next write to device memory

When turned on, the driver writes device memory, such as the rings of the DMA engines, with 32-byte AVX stores where the CPU supports them. When turned off, it uses 64-bit stores.

Some hypervisors trap BAR accesses and emulate the instruction. Such a hypervisor cannot decode AVX stores, so the VM stops at the first ring write. A passthrough VM maps the BAR directly into the guest, so it does not need this setting.

When to change it: Use the default. Set it to 0 only in a VM whose hypervisor traps and emulates BAR accesses, and put it in the option file so that it applies from the first write. For the option file, see Configuring and using parameters.

ha_hdma_completion

Type string
Permissions 0444 (load time only)
Default auto
Value auto, irq, poll
Other values The module fails to load and ha: `<value>' invalid for parameter `ha_hdma_completion' is logged

This parameter sets how the driver notices that a DMA engine has finished a transfer.

  • auto: The driver takes a per-engine MSI-X interrupt only when it can confirm bare metal. On x86 it treats the host as bare metal when the CPU has no hypervisor flag, and on arm64 when the kernel runs at the hypervisor privilege level (EL2). If neither applies (a VM guest, or an arm64 server whose kernel runs at EL1), it behaves like poll.
  • irq: The driver always takes MSI-X interrupts.
  • poll: The driver checks at short intervals for a completion mark that the engine leaves in host memory. A VM guest uses this mode because the engine's interrupt does not reach the guest. Copy results are the same, but because the driver keeps waking up to check while it waits, it uses more CPU than interrupts.

At load time, the kernel log shows the chosen mode for each card, for example HDMA completion: MSI-X (ha_hdma_completion=auto). /sys/module/ha/parameters/ha_hdma_completion shows only the configured value (such as auto), so check the mode actually chosen with this line.

1
sudo dmesg | grep 'HDMA completion'

When to change it: Normally use auto. Change it only when auto chooses wrongly.

  • If every DMA copy in a VM fails and logs ll: engine stopped but its completion MSI-X never arrived, set poll inside the VM guest. This happens in an x86 guest whose hypervisor hides the hypervisor flag, or in an arm64 guest set up under nested virtualization to run its own hypervisor.
  • If an arm64 bare-metal server shows HDMA completion: polling (ha_hdma_completion=auto), its kernel runs at EL1. Interrupts reach the driver on bare metal, so with irq it receives completion by interrupt and does not spend CPU on polling. Using irq in a VM guest makes every DMA copy fail.

For example, write options ha ha_hdma_completion=poll in /etc/modprobe.d/ha.conf and reload the module. See Setting values in an option file.


Recovery

ha_auto_reset

Type bool
Permissions 0644
Default 0 (N)
Takes effect The value at the moment the device becomes FATAL is used

When turned on, the driver resets a device by itself when the device enters the FATAL state (health is faulted). It first re-initializes the firmware, and if that fails, it proceeds to SBR (Secondary Bus Reset) only when the parent port is a root port or the Surprise Down error is masked on the parent port. When the reset starts, dmesg shows a line that starts with auto-TDR:.

  • When off (default): A FATAL device stays FATAL until someone resets it. An application resets it with the SDK's reset function, or an operator resets it with ha-hotreset.
  • Scope of reclamation: Like a manual reset, an automatic reset reclaims all tasks and allocations that were in progress on that device. Applications that were using the device must reopen it.
  • If it fails: If an automatic reset or a reset requested through the SDK fails, automatic reset is not tried again. health stays faulted. After a later reset succeeds or the device returns to the ready state, automatic reset works again.
  • During a firmware update: Automatic reset is not performed while an update is in progress or while its result has not been confirmed. This is to avoid corrupting the flash that is being written. A device skipped in this way stays FATAL. If health is still faulted after the update ends, reset it by hand. However, do not reset it if ha-fw-update ended with exit code 4, ended without reporting an exit code, or fw_update: no answer from firmware was logged. In that case the flash may have been only partly written, and you cannot rerun the update while the device is FATAL, so contact HyperAccel. See When no response is received.
  • Devices that are already FATAL: Reset by hand any device that became FATAL before you turned this parameter on.

When to change it: Turn it on for systems that must recover from a hang without human intervention. When you turn on ha_wd_preempt, turn this on together with it. For the effect and sequence of the reset, see FATAL and recovery.

ha_aer_enable

Type bool
Permissions 0444 (load time only)
Default 0 (N)

When turned on, a warning AER reporting enabled: the kernel's fatal remedy is a parent-bridge SBR is logged in dmesg every time a card is attached to the driver, and on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions) PCIe AER (Advanced Error Reporting) error reporting is turned on for this device. On kernel 6.4 and later and on RHEL 9.4 and later, the driver does not change the device's error reporting settings, and whether errors are reported follows the kernel's and the platform's AER settings.

To recover from a fatal error reported through AER, the kernel issues an SBR on the parent bridge. This SBR does not go through the conditions that apply when the driver issues one itself (the parent is a root port, or Surprise Down is masked). Therefore, on a path where Surprise Down is not masked, the platform may treat this link loss as an unrecoverable error and reboot the host. This is why the default is 0.

Even when it is off, the kernel's PCIe error recovery can run

When the parent port detects an error, when DPC (Downstream Port Containment) is triggered, or when another device under the same bridge has a fatal error, the kernel runs the recovery procedure for this device regardless of this parameter.

When to change it: Turn it on only if you have confirmed that a parent-bridge SBR is safe on that host (Surprise Down is masked on the ports along the path) and you want to use the kernel's AER recovery. For details, see FATAL and recovery.

ha_ready_timeout_ms

Type uint (ms)
Permissions 0644
Default 3000
Valid range 0~14400000. A value greater than 14400000 is treated as 14400000
Meaning of 0 The driver checks the firmware once and does not wait
Takes effect From the next reset. A reset that is already waiting uses the value from when it started waiting

How long the driver waits, after a reset, for the firmware to report that it is ready. It applies to a reset requested through the SDK, an automatic reset, and a reset that the kernel issues.

  • If the time runs out: The kernel log shows device came back but firmware did not report ready (<err>); reset reported as failed, or firmware did not report ready after host reset (<err>); ... for a reset that the kernel issued. Application requests are rejected until the firmware is ready.
  • Side effect: Module unload waits for an automatic reset that is in progress, so a larger value can also delay the unload.

When to change it: Raise it when these messages appear but fw_version reports a version shortly afterward, which means that the firmware needs longer than this to become ready. Otherwise, use the default.


Hang detection and the watchdog

The driver waits for responses to commands sent to the device and for DMA transfers to complete. The parameters below set the ceilings on those waits and the watchdog, which runs separately from the waits and finds waits that do not finish for a long time. For how it works and the diagnostic procedure, see Hang detection and the watchdog.

ha_devcmd_ceiling_ms

Type uint (ms)
Permissions 0644
Default 12000
Valid range Lower bound 100. A smaller value is treated as 100
Meaning of 0 None. It is treated as the lower bound 100. The wait does not become unlimited
Takes effect From waits that start after the change

This is the maximum time the driver waits for the response to a batch of commands sent to the device firmware. If there is no response within this time, the driver judges the device to be non-responding (hang). It increments devcmd_timeouts by 1, records the progress at that point and the verdict in last_hang, and logs a Devcmd no response after <N> ms (device hang): ... line and a diagnostic dump in dmesg. The task that was waiting ends with an error. If this determination is reached while waiting for a task submitted by an application, the device enters the FATAL state (health is faulted), and if ha_auto_reset is on, an automatic reset follows.

  • Long-running commands: For commands that take a long time, such as filling device memory (memset) or a firmware update, the driver adds extra time, with some margin over the expected duration, equally to this ceiling and to ha_wd_hang_ms. You do not need to raise this value because of such tasks.
  • Reset: When resetting through an automatic reset or the SDK's reset function, the driver also waits up to this ceiling for the response to the firmware re-initialization. If there is no response within that time, it proceeds to SBR when the conditions are met. For the conditions, see ha_auto_reset.
  • Waits requested with no time limit: These do not end at this ceiling. Instead, once this time has passed, a Devcmd waiting <N> ms with no timeout requested ... warning starts to be logged in dmesg and repeats every 10 seconds.

When to change it: Raise it when one command of a normal workload takes longer than the default ceiling and is judged to be a hang. By the amount you raise it, the point at which a non-responding device is judged to be a hang is also delayed. Lowering it is not recommended. The firmware also reports command failure through its own timeout, and if you set this value shorter than that, the driver judges a hang before the firmware reports the cause. When you change it, keep it greater than ha_wd_hang_ms + ha_wd_poll_ms.

ha_wd_hang_ms

Type uint (ms)
Permissions 0644
Default 8000
Meaning of 0 Turns off command wait monitoring
Takes effect From the next watchdog check. The new threshold is also used for waits already in progress

This is the threshold at which the watchdog judges a wait for a device command response to be a stall. The watchdog runs for each card separately from the waiting task, and when a wait exceeds this time, it logs watchdog: devcmd wait stalled (devcmd=<N> ms hdma=<N> ms) - no completion within the threshold in dmesg. It is logged only once per stall. Its purpose is to observe a stall before the wait ceiling (ha_devcmd_ceiling_ms) is reached, and also for waits requested with no time limit.

With the default values, it only reports and leaves the wait as it is. To cut off the wait, turn on ha_wd_preempt. The extra time that the driver adds for long-running commands is added equally to this threshold.

If this value + ha_wd_poll_ms is greater than or equal to ha_devcmd_ceiling_ms, this monitoring cannot fire. The driver does not change the values and logs a warning (Relationships between parameters).

ha_wd_hdma_hang_ms

Type uint (ms)
Permissions 0644
Default 4000
Meaning of 0 Turns off DMA wait monitoring
Takes effect From the next watchdog check

This is the threshold at which the watchdog judges a wait for a DMA transfer to complete to be a stall. Host–device and device–device data copies are handled by the DMA engines of the card. When it makes that determination, it logs a watchdog: hdma wait stalled ... line in the same format as for ha_wd_hang_ms.

The ceiling on waiting for one DMA transfer to complete is 5 seconds, and it cannot be changed with a parameter. If the transfer does not finish within 5 seconds, it fails and hdma_timeouts increases. Therefore, if this value + ha_wd_poll_ms is 5000 or more, this monitoring cannot fire, and the driver logs a warning.

ha_wd_poll_ms

Type uint (ms)
Permissions 0644
Default 500
Valid range Lower bound 50. A smaller value is treated as 50
Meaning of 0 Does not turn off the watchdog. It is treated as 50
Takes effect The previous interval applies until the first check after the change, and the new interval is used from then on

This is the interval at which the watchdog checks the wait time of each card. To turn off the watchdog, set the two thresholds (ha_wd_hang_ms, ha_wd_hdma_hang_ms) to 0, not this value. Each threshold + this value must be less than the corresponding wait ceiling.

ha_wd_preempt

Type bool
Permissions 0644
Default 0 (N)
Takes effect From the next stall determination

When turned on, the watchdog does not stop at reporting when it judges a stall but cuts off that wait. It cuts off only the side newly judged to be a stall (the command wait or the DMA wait). A wait that is cut off ends with the same error as when the device does not respond, and if it was executing an application task, the device becomes FATAL and stays that way until it is reset. When a wait requested with no time limit is cut off, only that request fails and the device does not become FATAL. For the behavior in each case, see Hang detection and the watchdog.

  • Use it together with ha_auto_reset: If ha_auto_reset is off, a device that becomes FATAL waits for a manual reset. In that case, dmesg shows a warning that starts with watchdog: preempting without ha_auto_reset.
  • It also cuts off waits with no time limit: With the default values, a wait requested with no time limit continues until it finishes. When this is on, such a wait is also cut off at the threshold. The watchdog cannot distinguish a long-running normal task from a non-responding device by time alone.
  • It does not cut off waits during a firmware update.

When to change it: Turn it on together with ha_auto_reset=1 on systems where you want to clean up a non-responding device sooner than the wait ceiling and recover automatically.


Telemetry

ha_sysmon_poll_ms

Type uint (ms)
Permissions 0644
Default 200
Valid range 50~2500. A value outside the range is treated as the nearest end value
Meaning of 0 Does not turn off polling. It is treated as the fastest interval (50)
Takes effect From the next poll. A poll that is already scheduled uses the previous interval

This is the interval at which the driver fetches temperature, power, rail voltage and current, and utilization from the firmware. The sysfs values temp, temp_avg, temp_blocks, power_mw, rails, and util, and the hwmon values, are refreshed at this interval, and reading more often than the interval returns the same sample. If the last sample is older than 5 seconds, these nodes become unknown. The upper bound of 2500 ms is set so that the nodes do not change to unknown even if one poll is late.

When to change it: Raise it to reduce telemetry round trips, and lower it to see changes in finer detail. If you raise it, the decision to stop polling a non-responding device (ha_sysmon_fail_limit) is delayed by the same amount. For how values are updated, see Telemetry.

ha_sysmon_fail_limit

Type uint
Permissions 0644
Default 15
Meaning of 0 Polling is not stopped even when there is no response
Takes effect From the next poll

If telemetry polling fails this many times in a row, polling of that card stops. A poll counts as a failure if it receives no value at all from the firmware. This is because continuing to read a non-responding device causes a PCIe Completion Timeout, and a platform that handles errors in a firmware-first manner can reset the entire server because of it.

  • When polling stops, dmesg shows sys_mon: no response for <N> consecutive polls (<M> ms) - suspending telemetry polling. ..., and the telemetry nodes fetched from the firmware become unknown.
  • A failure is counted once per polling interval, so it takes roughly ha_sysmon_fail_limit × ha_sysmon_poll_ms (about 3 seconds with the default values), plus the response wait time of each poll, before polling stops.
  • When the device is reset, polling restarts and sys_mon: resuming telemetry polling is logged. Polling that has already stopped does not restart even if you change this parameter.

When to change it: Use the default. Do not use 0 on firmware-first platforms.


Multiple devices

ha_tw_pin_wait_ms

Type uint (ms)
Permissions 0644
Default 12000
Valid range 0~14400000. A value greater than 14400000 is treated as 14400000
Meaning of 0 If no window is left, it fails immediately without waiting
Takes effect From waits that start after the change. A request that is already waiting uses the value from when it started waiting

A device-to-device copy (D2D) occupies a translation window of the other card to access that card's memory. There is a limit on the number of windows that can be occupied at the same time on one card, so when the limit is reached, a new copy waits for a window to be released. This parameter is the ceiling on that wait.

When the ceiling passes, that copy fails and a warning TW pin: <n>/<limit> slots pinned for <N>ms, giving up (...) is logged in dmesg. If the same warning repeats in a short time, only some of the warnings are logged. It is a transient failure that you can retry once a window is released.

When to change it: Raise it when device-to-device copies concentrate on one card and fail after waiting. To get a failure immediately without waiting, set it to 0. For device-to-device copies, see Multiple devices.

ha_host_mem_max_mb

Type uint (MiB)
Permissions 0644
Default 1024
Meaning of 0 No cap
Takes effect From the next allocation. Memory that is already allocated is left as is

An application can get host memory from the driver and let a card read and write that memory directly. The kernel does not reclaim or swap this memory. It is allocated on the card's NUMA node, or below 4 GiB (ZONE_DMA32) when that node's memory lies above the address range the card can reach. This parameter is the cap on the total of this memory held by all cards and processes.

  • An allocation that would take the total past the cap fails. Memory that is already allocated is not affected.
  • Memory that a process has let go of is not freed while a card that used it is running work, is being reset, is FATAL, or failed its last reset, and it counts toward the total until it is freed.
  • The memory counts toward the memory cgroup usage of the process that allocated it.
  • While a process holds this memory, the ha module cannot be unloaded, even if that process has closed /dev/haN. modprobe -r ha fails because the module is in use.

When to change it: Use the default. Raise it when several jobs on one host use this memory at the same time and their allocations hit the cap. Any user can open the device nodes, so with 0 an ordinary user can use up the host's memory below 4 GiB.


Performance

ha_abi_prefetch

Type bool
Permissions 0644
Default 1 (Y)
Takes effect From the next determination. When turned on, it also applies to tasks already in the queue

This is a performance optimization that copies the device kernel argument block (ABI) that the next task will use from host memory to device memory in advance, while the preceding device kernel in the same stream is running. Its purpose is to reduce the total time by overlapping the copy with device kernel execution.

  • The result is the same as running in order. If a preceding task can still read or write the destination of that copy, the copy is not prefetched and runs in the original order. A copy that cannot be prefetched is not an error.
  • With 0, all copies run in submission order. This is the behavior from before this optimization existed.
  • See the effect in the sysfs abi_prefetch_stats. issued is the number of copies started ahead of time, and consumed is the number of times the prefetched result was actually used. For the remaining fields, see sysfs nodes.

When to change it: Turn it off when you want to compare behavior or performance without prefetching.

ha_hdma_min_bytes

Type uint
Permissions 0644
Default 0
Takes effect From the next copy to start

The size that decides whether a copy between host and device (H2D, D2H, file load) goes to the DMA engines or is done by the CPU (PIO).

  • With 0, copies of every size go to the DMA engines.
  • With N, copies smaller than N bytes are done by PIO. PIO is much slower than the DMA engines for large copies.

When to change it: Normally keep the default 0. Above 2 GiB, ha-wipe does not start.

ha_hdma_cpu_latency_us

Type int
Permissions 0444
Default 0
Takes effect When the module is loaded

A wake-up latency request that the driver places on every CPU while the DMA engines copy and for 10 ms after the last copy ends. It keeps the thread that waits for a copy from waking late out of a deep C-state and leaving the engine idle. Copy results and ordering do not change; only throughput and CPU power during copies change.

  • With 0, every C-state (CPU power-saving state) is blocked, C1 included. While copies run, idle CPUs spin in a wait loop instead of saving power.
  • With N, only C-states whose wake-up time (exit latency) is N µs or less are allowed.
  • With -1, no request is placed. A value below -1 is changed to 0, with a warning in the kernel log.
  • When no copy runs, there is no constraint, so a /dev/cpu_dma_latency value set from user space applies as it is. Where both apply, the stricter one is used.
  • At load, the kernel log shows ha: HDMA copies hold a <N> us CPU wake-up latency request while they run, or with -1, ha: HDMA copies hold no CPU wake-up latency request (ha_hdma_cpu_latency_us=-1).

When to change it: Set it to N or -1 when you need to lower CPU power during copies. Copy throughput may drop.


See also