Skip to content

Configuring and using parameters

ha.ko uses module parameters to control the behavior of recovery, hang detection, telemetry, device-to-device paths, and performance optimization. This page covers how to check the current values and change them, when a changed value takes effect, and commonly used settings. The type, default, valid range, and meaning of 0 for each parameter are in the Parameter reference.

These are parameters of the ha module

These values are parameters of the ha module. You normally set them in an option file in /etc/modprobe.d/, but a value written on the kernel command line as ha.<name>=<value> is also used when the module is loaded. Every parameter applies identically to all cards in the system, and you cannot give different values to different cards.


Categories

Category Parameters When they are usually changed
Recovery ha_auto_reset, ha_aer_enable, ha_ready_timeout_ms When you want to recover a FATAL device without human intervention, or to turn on PCIe AER error reporting for the card on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions), or when the firmware needs longer to become ready after a reset
Hang detection and the watchdog ha_devcmd_ceiling_ms, ha_wd_hang_ms, ha_wd_hdma_hang_ms, ha_wd_poll_ms, ha_wd_preempt When a normal task is judged to be a hang, or when you want to cut off a stalled wait sooner than the ceiling
Telemetry ha_sysmon_poll_ms, ha_sysmon_fail_limit When you want to change the refresh interval of sensor values, or the criterion for stopping monitoring of a card that does not respond
Device configuration and multiple devices ha_tw_wide_access, ha_hdma_completion, ha_tw_pin_wait_ms, ha_host_mem_max_mb When a hypervisor traps BAR accesses, when every DMA copy in a VM fails or an arm64 bare-metal server polls for DMA completion, when device-to-device copies concentrate on one card, or when allocations of host memory that cards read and write directly hit the cap
Performance ha_abi_prefetch, ha_hdma_min_bytes, ha_hdma_cpu_latency_us When you want to compare behavior or performance without the optimization, to handle small copies between host and device with a CPU copy instead of a DMA engine, or to reduce CPU power while copies run

Most systems run with the default values. The defaults and permissions are in At a glance.


Checking current values

Parameter values are in /sys/module/ha/parameters/<name>, and regular users can read them too. A bool parameter reads as Y (1) or N (0).

1
cat /sys/module/ha/parameters/ha_auto_reset

View the operational parameters all at once:

1
2
3
4
5
cd /sys/module/ha/parameters && grep -H . ha_tw_wide_access ha_hdma_completion \
  ha_auto_reset ha_aer_enable ha_ready_timeout_ms \
  ha_devcmd_ceiling_ms ha_wd_hang_ms ha_wd_hdma_hang_ms ha_wd_poll_ms ha_wd_preempt \
  ha_sysmon_poll_ms ha_sysmon_fail_limit ha_tw_pin_wait_ms ha_host_mem_max_mb \
  ha_abi_prefetch ha_hdma_min_bytes ha_hdma_cpu_latency_us

This directory also contains parameter files that are not described in this document. They are values used for development and testing, so do not change them without guidance from HyperAccel.

The file shows the value you wrote, as is

If you write an out-of-range value, the file keeps the value you wrote as is, and the driver adjusts it into the valid range when it uses the value. For example, if you write 0 to ha_sysmon_poll_ms, the file shows 0, but the actual interval is 50 ms.


How to change values

Method Takes effect After a reboot Parameters you can change
Option file The next time the module is loaded Kept All
Kernel command line When the module is loaded after you edit the command line and reboot Kept as long as it remains on the command line All
modprobe command line That load only Lost All
Writing at run time Depends on the parameter Lost Those with permissions 0644

Of the three places from which values are read at load time in the table above (the option file, the kernel command line, and the modprobe command line), write a value in only one. To see which value is in use, look in /sys/module/ha/parameters/, as described in Checking current values.

Setting values in an option file

Create an option file under /etc/modprobe.d/. The file name only needs to end in .conf. You can write several parameters on one line.

1
2
# /etc/modprobe.d/ha.conf
options ha ha_auto_reset=1 ha_wd_preempt=1

This file takes effect the next time the module is loaded. Reboot, or reload the module:

1
2
sudo modprobe -r ha
sudo modprobe ha

Distributions that build the initial RAM disk with dracut

On distributions where dracut builds a host-specific initial RAM disk (initramfs), such as the RHEL family and Fedora, the ha module and the /etc/modprobe.d/ option files at that time may be included in the initial RAM disk when it is rebuilt (for example, on a kernel update). The module is then loaded at boot with the options contained in the initial RAM disk, so to apply an edited option file with a reboot, rebuild the initial RAM disk with sudo dracut -f --regenerate-all. After rebooting, check the values in /sys/module/ha/parameters/.

/etc/modules-load.d/ha.conf, which the installation process creates, is a setting that loads the module automatically at boot; it is not a file for writing parameters. The installation and removal procedures neither create nor delete option files in /etc/modprobe.d/, so your settings remain unchanged when you upgrade the driver.

If you write a value that does not match the type, or a value the parameter does not accept, in an option file or on the modprobe command line, the module fails to load and dmesg shows a ha: `<value>' invalid for parameter `<name>' line. A parameter whose name is wrong is ignored with only the warning ha: unknown parameter '<name>' ignored. After you edit an option file, reload the module and check the values in /sys/module/ha/parameters/.

Reloading the module takes down all cards

modprobe -r ha detaches every card in the system from the driver. If even one process has a device open, the unload fails (Module ha is in use). Do this only after you stop all workloads.

A parameter with permissions 0644 can be changed at run time without reloading. If you only want to restore the state of a single card, do not reload the module; use ha-hotreset.

Writing values on the kernel command line

If you write ha.<parameter>=<value> on the boot loader's kernel command line, that value is used as a module option when the module is loaded (both the automatic load at boot and modprobe ha). Use this on systems where editing the option file is difficult.

1
ha.ha_auto_reset=1 ha.ha_sysmon_poll_ms=500
  • Ubuntu and Debian: Add it to GRUB_CMDLINE_LINUX in /etc/default/grub, run sudo update-grub, and reboot.
  • RHEL family and Fedora: Add it with a command such as sudo grubby --update-kernel=ALL --args="ha.ha_auto_reset=1", and reboot.
  • Check the command line of the running kernel with cat /proc/cmdline.
  • Values on the kernel command line are validated in the same way as in an option file. If a value does not match its type, the module is not loaded, and because the command line changes only on reboot, modprobe ha also fails for the same reason for the rest of that boot. Fix the command line and reboot again.
  • Kernel command-line options without the ha. prefix also affect driver behavior. Depending on the IOMMU setting (for example, iommu=pt), the addressing scheme of the data path between cards changes. The supported configurations are in Supported environments, and the differences between configurations are in IOMMU modes and device-to-device paths. If you turn off MSI with pci=nomsi, the driver cannot set up the card's interrupts and fails to probe the card (Errors that stop probe).

Specifying values for a single load

A value written on the modprobe command line applies to that load only.

1
sudo modprobe ha ha_auto_reset=1 ha_sysmon_poll_ms=500

If the module is already loaded, modprobe does nothing, and the values on its command line are not applied. Because the installation process configures the module to load at boot, first unload the module with sudo modprobe -r ha.

Changing values at run time

root can change a parameter with permissions 0644 directly. For a bool, write 1/0 or Y/N.

1
2
echo 1 | sudo tee /sys/module/ha/parameters/ha_auto_reset
echo 1000 | sudo tee /sys/module/ha/parameters/ha_sysmon_poll_ms
  • A value changed at run time is lost when you reload the module. To keep using it, also write it in an option file.
  • ha_hdma_completion, ha_aer_enable, and ha_hdma_cpu_latency_us, which have permissions 0444, cannot be changed at run time. Writes are denied.
  • A value that does not match the type (a negative number or a string for a uint) is rejected with Invalid argument, and the value does not change.

When a changed value starts to be used differs from parameter to parameter. See "Takes effect" in the table in each section of the Parameter reference.


Common settings

Most of the examples below are in the format you write in /etc/modprobe.d/ha.conf. After you edit the file, you must reload the module or reboot for it to take effect. A parameter with permissions 0644 can also be changed immediately, as in Changing values at run time.

Turning on automatic recovery

Have the driver reset a device that has become FATAL, without anyone resetting it by hand.

1
2
# /etc/modprobe.d/ha.conf
options ha ha_auto_reset=1
  • To turn it on right now, run echo 1 | sudo tee /sys/module/ha/parameters/ha_auto_reset. The driver uses the value at the moment the device becomes FATAL, so reset by hand any device that is already FATAL before you turn it on.
  • When an automatic reset starts, dmesg shows a line that starts with auto-TDR:. Applications that were using the reset device must reopen it.
  • A device on which a firmware update is in progress, or for which the response to the update has not been confirmed, is not reset automatically.

For the reset sequence and the behavior on failure, see Automatic reset.

Cutting off stalled waits early and recovering

By default, the watchdog only reports a stalled wait in the kernel log. To clean it up sooner than the response wait ceiling (12 seconds by default) and recover automatically, turn on ha_wd_preempt together with ha_auto_reset.

1
options ha ha_auto_reset=1 ha_wd_preempt=1
  • A device command wait is cut off at about the threshold (ha_wd_hang_ms, 8 seconds by default) rather than at the ceiling (12 seconds by default), and a DMA transfer is cut off at about the DMA wait threshold (ha_wd_hdma_hang_ms, 4 seconds by default). A DMA transfer cut off this way makes the device FATAL. With the default values, when a DMA transfer does not finish within 5 seconds, only that task fails, and the device does not become FATAL.
  • The watchdog looks only at elapsed time, so it also cuts off a normal task that exceeds the threshold. Do not turn it on for a system that runs tasks that wait for a long time with no upper limit.
  • If you turn it on without ha_auto_reset, a device that becomes FATAL waits for a manual reset, and dmesg shows a warning that starts with watchdog: preempting without ha_auto_reset. When you turn it on at run time, turn on ha_auto_reset first.
  • When you change a threshold, observe the relationships between parameters.

For the behavior in each case, see Cutting off hung waits.

Preventing long-running normal tasks from being judged a hang

The driver sends device commands in batches, and the response wait ceiling applies to the total time spent waiting for the response to one batch. If one batch of a normal workload takes longer than the ceiling (12 seconds by default) to process, so that Devcmd no response after 12000 ms (device hang) is logged and the device becomes FATAL, raise the ceiling well above the time that task takes, and raise the watchdog threshold along with it.

1
options ha ha_devcmd_ceiling_ms=30000 ha_wd_hang_ms=25000
  • Both values can also be changed at run time. The ceiling applies from waits that start after the change, and the threshold from the next watchdog check.
  • If you leave the threshold unchanged, a watchdog warning is logged for every such task. On a system with ha_wd_preempt=1, the watchdog cuts off that wait at the threshold (8 seconds by default) and the device becomes FATAL, so raising only the ceiling has no effect.
  • If the device became FATAL because the firmware itself reported a timeout (rc=0x7 on the Firmware error line), raising these two values changes nothing. In that case, contact the support team.
  • By the amount you raise them, the point at which a device that is actually not responding is judged to be a hang is delayed, and the time spent waiting for the response to the firmware re-initialization during a reset also grows longer.
  • The driver allows extra time for filling device memory and for firmware updates, so you do not need to raise the values because of those operations.
  • The two values must satisfy the relationships between parameters (example: 25000 + 500 < 30000).

Changing the telemetry refresh interval

This is the interval at which temperature, power, and utilization are fetched from the firmware. The default is 200 ms.

1
options ha ha_sysmon_poll_ms=1000
  • The valid range is 50–2500 ms. A value outside the range is treated as the nearest end value, and 0 means the shortest interval (50 ms).
  • If you lengthen the interval, the decision to stop polling a device that does not respond is also delayed. It takes roughly ha_sysmon_fail_limit × the interval.
  • If the last sample is older than 5 seconds, the value becomes unknown. The longer you set the interval, the more easily a value becomes unknown when polling fails once or twice.
  • You can also change it at run time. A poll that is already scheduled uses the previous interval, and the new interval is used from the poll after that.

Turning on PCIe AER error reporting for the card

With ha_aer_enable=1, on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions) the driver turns on PCIe AER error reporting for the card. On kernel 6.4 and later and on RHEL 9.4 and later, this parameter changes nothing except for logging one warning line. On those kernels, the kernel and platform configuration determine error reporting, so even with ha_aer_enable=0 the card's errors can be reported and the kernel's recovery procedure can proceed.

1
options ha ha_aer_enable=1
  • It can be set only at load time (0444). To turn it on, write it in an option file and reload the module. The driver does not turn off error reporting once it has been turned on, so to turn it off again on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions), change the value to 0 in the option file or remove it, and then reboot the host. If you do not want to reboot, reload the module and then reset each card with ha-hotreset. Reloading the module alone leaves error reporting on the card turned on.
  • To recover from a fatal error reported through AER, the kernel issues an SBR (Secondary Bus Reset, a reset of the entire bus below the bridge) on the parent bridge. If the Surprise Down error (an error indicating that the link went down without warning) is not masked on the ports along the path, the platform can reboot the host, so turn it on only if you have confirmed that SBR is safe on that host.

For details, see AER reporting and ha_aer_enable and Configurations where SBR can be issued.

Turning off input copy prefetching

ha_abi_prefetch (on by default) is a performance optimization that copies in advance the argument block that the next device kernel will use, while the preceding device kernel in the same stream is running. Turn it off when you compare behavior or performance without this optimization. You can change it at run time.

1
echo 0 | sudo tee /sys/module/ha/parameters/ha_abi_prefetch
  • When it is off, all copies run in submission order. Even when it is on, the result is the same as running in order.
  • The number of times copies were prefetched is shown in the sysfs abi_prefetch_stats. To turn it on again, write 1.

Relationships between parameters

Relationship Condition to satisfy If violated
Command wait monitoring ha_wd_hang_ms + ha_wd_poll_ms < ha_devcmd_ceiling_ms Command wait monitoring cannot fire
DMA wait monitoring ha_wd_hdma_hang_ms + ha_wd_poll_ms < 5000 DMA wait monitoring cannot fire
Cutting off waits ha_auto_reset=1 if ha_wd_preempt=1 A device that becomes FATAL stays as it is until a manual reset
Telemetry suspension Roughly ha_sysmon_fail_limit × ha_sysmon_poll_ms or more until the determination Lengthening the interval delays the determination too

The defaults satisfy the first two conditions (8000 + 500 < 12000, 4000 + 500 < 5000). Monitoring whose threshold is 0 is turned off and is therefore not checked.

If a combination violates a watchdog condition, the driver does not change the values and logs the following warning in dmesg at the next watchdog check. It logs the warning only once for the same combination of values, and logs it again if the condition is still violated after the values change.

1
watchdog: devcmd arm cannot fire with this configuration (hang=<ms> ms hdma_hang=<ms> ms poll=<ms> ms devcmd_ceiling=<ms> ms hdma_timeout=5000 ms); lower the threshold or the poll interval

In the devcmd position, the monitoring that cannot fire appears as one of devcmd, hdma, or devcmd+hdma.


See also