Configuring and using parameters¶
ha.ko uses module parameters to control the behavior of recovery, hang detection, telemetry, device-to-device paths, and performance optimization. This page covers how to check the current values and change them, when a changed value takes effect, and commonly used settings. The type, default, valid range, and meaning of 0 for each parameter are in the Parameter reference.
These are parameters of the ha module
These values are parameters of the ha module. You normally set them in an option file in /etc/modprobe.d/, but a value written on the kernel command line as ha.<name>=<value> is also used when the module is loaded. Every parameter applies identically to all cards in the system, and you cannot give different values to different cards.
Categories¶
| Category | Parameters | When they are usually changed |
|---|---|---|
| Recovery | ha_auto_reset, ha_aer_enable, ha_ready_timeout_ms |
When you want to recover a FATAL device without human intervention, or to turn on PCIe AER error reporting for the card on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions), or when the firmware needs longer to become ready after a reset |
| Hang detection and the watchdog | ha_devcmd_ceiling_ms, ha_wd_hang_ms, ha_wd_hdma_hang_ms, ha_wd_poll_ms, ha_wd_preempt |
When a normal task is judged to be a hang, or when you want to cut off a stalled wait sooner than the ceiling |
| Telemetry | ha_sysmon_poll_ms, ha_sysmon_fail_limit |
When you want to change the refresh interval of sensor values, or the criterion for stopping monitoring of a card that does not respond |
| Device configuration and multiple devices | ha_tw_wide_access, ha_hdma_completion, ha_tw_pin_wait_ms, ha_host_mem_max_mb |
When a hypervisor traps BAR accesses, when every DMA copy in a VM fails or an arm64 bare-metal server polls for DMA completion, when device-to-device copies concentrate on one card, or when allocations of host memory that cards read and write directly hit the cap |
| Performance | ha_abi_prefetch, ha_hdma_min_bytes, ha_hdma_cpu_latency_us |
When you want to compare behavior or performance without the optimization, to handle small copies between host and device with a CPU copy instead of a DMA engine, or to reduce CPU power while copies run |
Most systems run with the default values. The defaults and permissions are in At a glance.
Checking current values¶
Parameter values are in /sys/module/ha/parameters/<name>, and regular users can read them too. A bool parameter reads as Y (1) or N (0).
1 | |
View the operational parameters all at once:
1 2 3 4 5 | |
This directory also contains parameter files that are not described in this document. They are values used for development and testing, so do not change them without guidance from HyperAccel.
The file shows the value you wrote, as is
If you write an out-of-range value, the file keeps the value you wrote as is, and the driver adjusts it into the valid range when it uses the value. For example, if you write 0 to ha_sysmon_poll_ms, the file shows 0, but the actual interval is 50 ms.
How to change values¶
| Method | Takes effect | After a reboot | Parameters you can change |
|---|---|---|---|
| Option file | The next time the module is loaded | Kept | All |
| Kernel command line | When the module is loaded after you edit the command line and reboot | Kept as long as it remains on the command line | All |
modprobe command line |
That load only | Lost | All |
| Writing at run time | Depends on the parameter | Lost | Those with permissions 0644 |
Of the three places from which values are read at load time in the table above (the option file, the kernel command line, and the modprobe command line), write a value in only one. To see which value is in use, look in /sys/module/ha/parameters/, as described in Checking current values.
Setting values in an option file¶
Create an option file under /etc/modprobe.d/. The file name only needs to end in .conf. You can write several parameters on one line.
1 2 | |
This file takes effect the next time the module is loaded. Reboot, or reload the module:
1 2 | |
Distributions that build the initial RAM disk with dracut
On distributions where dracut builds a host-specific initial RAM disk (initramfs), such as the RHEL family and Fedora, the ha module and the /etc/modprobe.d/ option files at that time may be included in the initial RAM disk when it is rebuilt (for example, on a kernel update). The module is then loaded at boot with the options contained in the initial RAM disk, so to apply an edited option file with a reboot, rebuild the initial RAM disk with sudo dracut -f --regenerate-all. After rebooting, check the values in /sys/module/ha/parameters/.
/etc/modules-load.d/ha.conf, which the installation process creates, is a setting that loads the module automatically at boot; it is not a file for writing parameters. The installation and removal procedures neither create nor delete option files in /etc/modprobe.d/, so your settings remain unchanged when you upgrade the driver.
If you write a value that does not match the type, or a value the parameter does not accept, in an option file or on the modprobe command line, the module fails to load and dmesg shows a ha: `<value>' invalid for parameter `<name>' line. A parameter whose name is wrong is ignored with only the warning ha: unknown parameter '<name>' ignored. After you edit an option file, reload the module and check the values in /sys/module/ha/parameters/.
Reloading the module takes down all cards
modprobe -r ha detaches every card in the system from the driver. If even one process has a device open, the unload fails (Module ha is in use). Do this only after you stop all workloads.
A parameter with permissions 0644 can be changed at run time without reloading. If you only want to restore the state of a single card, do not reload the module; use ha-hotreset.
Writing values on the kernel command line¶
If you write ha.<parameter>=<value> on the boot loader's kernel command line, that value is used as a module option when the module is loaded (both the automatic load at boot and modprobe ha). Use this on systems where editing the option file is difficult.
1 | |
- Ubuntu and Debian: Add it to
GRUB_CMDLINE_LINUXin/etc/default/grub, runsudo update-grub, and reboot. - RHEL family and Fedora: Add it with a command such as
sudo grubby --update-kernel=ALL --args="ha.ha_auto_reset=1", and reboot. - Check the command line of the running kernel with
cat /proc/cmdline. - Values on the kernel command line are validated in the same way as in an option file. If a value does not match its type, the module is not loaded, and because the command line changes only on reboot,
modprobe haalso fails for the same reason for the rest of that boot. Fix the command line and reboot again. - Kernel command-line options without the
ha.prefix also affect driver behavior. Depending on the IOMMU setting (for example,iommu=pt), the addressing scheme of the data path between cards changes. The supported configurations are in Supported environments, and the differences between configurations are in IOMMU modes and device-to-device paths. If you turn off MSI withpci=nomsi, the driver cannot set up the card's interrupts and fails to probe the card (Errors that stop probe).
Specifying values for a single load¶
A value written on the modprobe command line applies to that load only.
1 | |
If the module is already loaded, modprobe does nothing, and the values on its command line are not applied. Because the installation process configures the module to load at boot, first unload the module with sudo modprobe -r ha.
Changing values at run time¶
root can change a parameter with permissions 0644 directly. For a bool, write 1/0 or Y/N.
1 2 | |
- A value changed at run time is lost when you reload the module. To keep using it, also write it in an option file.
ha_hdma_completion,ha_aer_enable, andha_hdma_cpu_latency_us, which have permissions0444, cannot be changed at run time. Writes are denied.- A value that does not match the type (a negative number or a string for a
uint) is rejected withInvalid argument, and the value does not change.
When a changed value starts to be used differs from parameter to parameter. See "Takes effect" in the table in each section of the Parameter reference.
Common settings¶
Most of the examples below are in the format you write in /etc/modprobe.d/ha.conf. After you edit the file, you must reload the module or reboot for it to take effect. A parameter with permissions 0644 can also be changed immediately, as in Changing values at run time.
Turning on automatic recovery¶
Have the driver reset a device that has become FATAL, without anyone resetting it by hand.
1 2 | |
- To turn it on right now, run
echo 1 | sudo tee /sys/module/ha/parameters/ha_auto_reset. The driver uses the value at the moment the device becomes FATAL, so reset by hand any device that is already FATAL before you turn it on. - When an automatic reset starts, dmesg shows a line that starts with
auto-TDR:. Applications that were using the reset device must reopen it. - A device on which a firmware update is in progress, or for which the response to the update has not been confirmed, is not reset automatically.
For the reset sequence and the behavior on failure, see Automatic reset.
Cutting off stalled waits early and recovering¶
By default, the watchdog only reports a stalled wait in the kernel log. To clean it up sooner than the response wait ceiling (12 seconds by default) and recover automatically, turn on ha_wd_preempt together with ha_auto_reset.
1 | |
- A device command wait is cut off at about the threshold (
ha_wd_hang_ms, 8 seconds by default) rather than at the ceiling (12 seconds by default), and a DMA transfer is cut off at about the DMA wait threshold (ha_wd_hdma_hang_ms, 4 seconds by default). A DMA transfer cut off this way makes the device FATAL. With the default values, when a DMA transfer does not finish within 5 seconds, only that task fails, and the device does not become FATAL. - The watchdog looks only at elapsed time, so it also cuts off a normal task that exceeds the threshold. Do not turn it on for a system that runs tasks that wait for a long time with no upper limit.
- If you turn it on without
ha_auto_reset, a device that becomes FATAL waits for a manual reset, and dmesg shows a warning that starts withwatchdog: preempting without ha_auto_reset. When you turn it on at run time, turn onha_auto_resetfirst. - When you change a threshold, observe the relationships between parameters.
For the behavior in each case, see Cutting off hung waits.
Preventing long-running normal tasks from being judged a hang¶
The driver sends device commands in batches, and the response wait ceiling applies to the total time spent waiting for the response to one batch. If one batch of a normal workload takes longer than the ceiling (12 seconds by default) to process, so that Devcmd no response after 12000 ms (device hang) is logged and the device becomes FATAL, raise the ceiling well above the time that task takes, and raise the watchdog threshold along with it.
1 | |
- Both values can also be changed at run time. The ceiling applies from waits that start after the change, and the threshold from the next watchdog check.
- If you leave the threshold unchanged, a watchdog warning is logged for every such task. On a system with
ha_wd_preempt=1, the watchdog cuts off that wait at the threshold (8 seconds by default) and the device becomes FATAL, so raising only the ceiling has no effect. - If the device became FATAL because the firmware itself reported a timeout (
rc=0x7on theFirmware errorline), raising these two values changes nothing. In that case, contact the support team. - By the amount you raise them, the point at which a device that is actually not responding is judged to be a hang is delayed, and the time spent waiting for the response to the firmware re-initialization during a reset also grows longer.
- The driver allows extra time for filling device memory and for firmware updates, so you do not need to raise the values because of those operations.
- The two values must satisfy the relationships between parameters (example: 25000 + 500 < 30000).
Changing the telemetry refresh interval¶
This is the interval at which temperature, power, and utilization are fetched from the firmware. The default is 200 ms.
1 | |
- The valid range is 50–2500 ms. A value outside the range is treated as the nearest end value, and
0means the shortest interval (50 ms). - If you lengthen the interval, the decision to stop polling a device that does not respond is also delayed. It takes roughly
ha_sysmon_fail_limit× the interval. - If the last sample is older than 5 seconds, the value becomes
unknown. The longer you set the interval, the more easily a value becomesunknownwhen polling fails once or twice. - You can also change it at run time. A poll that is already scheduled uses the previous interval, and the new interval is used from the poll after that.
Turning on PCIe AER error reporting for the card¶
With ha_aer_enable=1, on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions) the driver turns on PCIe AER error reporting for the card. On kernel 6.4 and later and on RHEL 9.4 and later, this parameter changes nothing except for logging one warning line. On those kernels, the kernel and platform configuration determine error reporting, so even with ha_aer_enable=0 the card's errors can be reported and the kernel's recovery procedure can proceed.
1 | |
- It can be set only at load time (
0444). To turn it on, write it in an option file and reload the module. The driver does not turn off error reporting once it has been turned on, so to turn it off again on kernels earlier than 6.4 (earlier than 9.4 on RHEL-family distributions), change the value to0in the option file or remove it, and then reboot the host. If you do not want to reboot, reload the module and then reset each card withha-hotreset. Reloading the module alone leaves error reporting on the card turned on. - To recover from a fatal error reported through AER, the kernel issues an SBR (Secondary Bus Reset, a reset of the entire bus below the bridge) on the parent bridge. If the Surprise Down error (an error indicating that the link went down without warning) is not masked on the ports along the path, the platform can reboot the host, so turn it on only if you have confirmed that SBR is safe on that host.
For details, see AER reporting and ha_aer_enable and Configurations where SBR can be issued.
Turning off input copy prefetching¶
ha_abi_prefetch (on by default) is a performance optimization that copies in advance the argument block that the next device kernel will use, while the preceding device kernel in the same stream is running. Turn it off when you compare behavior or performance without this optimization. You can change it at run time.
1 | |
- When it is off, all copies run in submission order. Even when it is on, the result is the same as running in order.
- The number of times copies were prefetched is shown in the sysfs
abi_prefetch_stats. To turn it on again, write1.
Relationships between parameters¶
| Relationship | Condition to satisfy | If violated |
|---|---|---|
| Command wait monitoring | ha_wd_hang_ms + ha_wd_poll_ms < ha_devcmd_ceiling_ms |
Command wait monitoring cannot fire |
| DMA wait monitoring | ha_wd_hdma_hang_ms + ha_wd_poll_ms < 5000 |
DMA wait monitoring cannot fire |
| Cutting off waits | ha_auto_reset=1 if ha_wd_preempt=1 |
A device that becomes FATAL stays as it is until a manual reset |
| Telemetry suspension | Roughly ha_sysmon_fail_limit × ha_sysmon_poll_ms or more until the determination |
Lengthening the interval delays the determination too |
The defaults satisfy the first two conditions (8000 + 500 < 12000, 4000 + 500 < 5000). Monitoring whose threshold is 0 is turned off and is therefore not checked.
If a combination violates a watchdog condition, the driver does not change the values and logs the following warning in dmesg at the next watchdog check. It logs the warning only once for the same combination of values, and logs it again if the condition is still violated after the values change.
1 | |
In the devcmd position, the monitoring that cannot fire appears as one of devcmd, hdma, or devcmd+hdma.
See also¶
- Parameter reference — type, default, valid range, and when the value takes effect for each parameter
- FATAL and recovery —
ha_auto_reset,ha_aer_enable - Hang detection and the watchdog — wait ceilings and the watchdog parameters
- Telemetry —
ha_sysmon_poll_ms,ha_sysmon_fail_limit - Kernel log messages — warnings for invalid values and contradictory combinations