Skip to content

Kernel log messages

This page collects the messages that ha.ko writes to the kernel log (dmesg), organized by situation. Each message lists its meaning and what the operator should do, and detailed procedures link to the relevant feature pages. If you are starting from a symptom, see Troubleshooting first.


Message format

Every driver message described on this page starts with one of two prefixes.

Prefix Example Used for
ha <BDF>: ha 0000:01:00.0: Probing device Messages about a single card. <BDF> is the card's PCI address
ha: ha: Initializing module Messages about the module as a whole, and some card messages that are logged without a <BDF>
  • Find the mapping between <BDF> and /dev/haN in the Probe successful: /dev/ha<N> created line, the Bus-Id column of hyperaccel-smi, or the /sys/class/ha/haN/device link.
  • Messages that the kernel logs about this card (such as PCIe error reports) can also appear with the same ha <BDF>: prefix. Messages about the upstream PCIe port that the card is attached to appear with that port's prefix (for example, pcieport <BDF>:).
  • In the logs described on this page, <...> is a value that changes from case to case. For long messages, only the first part is shown, with … added at the end. When searching, use the first part.
  • A negative number in parentheses is a Linux kernel error number. Common values are -110 (timeout), -5 (I/O error), -11 (not ready yet, try again), -19 (no such device), -95 (not supported, for example a configuration in which an SBR cannot be issued), -121 (remote I/O error), and -4 (interrupted by a signal).
  • The Level column of each table is the name used with dmesg --level. err is an error, warn is a warning, info is informational, and crit is a severe error.
  • If the same message repeats within a short time, only some of the occurrences are printed, and the kernel reports the number it omitted in a callbacks suppressed line.

Kernel warning traces

If the call trace in a block that starts with WARNING: contains [ha], the driver's internal check has been triggered. Attach the complete log from before and after it to the support request.


Viewing logs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Driver messages only, with human-readable timestamps
sudo dmesg -T | grep -E '(^|\] )ha[ :]'

# In real time while reproducing the problem
sudo dmesg -wT | grep -E '(^|\] )ha[ :]'

# Warnings and errors only
sudo dmesg -T --level=err,warn | grep -E '(^|\] )ha[ :]'

# Kernel log of the previous boot (if the systemd journal is stored persistently)
sudo journalctl -k -b -1 | grep -E ' ha[ :]'

Messages from the upstream PCIe port have a different prefix and are not matched by the filter above. Attach the full dmesg output, without the filter, to a support request.

Save the log before you reset

A hot reset or a module reload clears the sysfs error counters and last_hang, and the messages that a reset leaves can push earlier records out of the kernel log buffer. Before you reset a card, save the full dmesg output and the sysfs values. A script that saves them all at once is in Collecting diagnostic information for a problem.


Enabling detailed logs

The driver leaves the detailed per-command log (debug level) off by default. You can turn it on with the kernel's dynamic debug, which requires the kernel configuration option CONFIG_DYNAMIC_DEBUG=y.

1
2
3
4
5
# Check the kernel configuration (most distributions)
grep -E '^CONFIG_DYNAMIC_DEBUG=' /boot/config-$(uname -r)

# After loading the module, check whether there are items you can enable (if greater than 0, you can enable them)
sudo grep -c '\[ha\]' /sys/kernel/debug/dynamic_debug/control
1
2
3
4
5
6
7
8
# Enable on a loaded module
echo 'module ha +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

# Disable
echo 'module ha -p' | sudo tee /sys/kernel/debug/dynamic_debug/control

# Enable from the time the module is loaded (while the module is not loaded)
sudo modprobe ha dyndbg=+p
  • To use /sys/kernel/debug/dynamic_debug/control, debugfs must be mounted at /sys/kernel/debug. If debugfs is not available, check whether /proc/dynamic_debug/control exists; if it does, replace the path in the commands above with that file. The format and commands are the same.
  • The enabled log is at debug level, so it does not appear on the console; read it with dmesg.

Enable it only while you reproduce the problem

A log line is written for every command, so the kernel log buffer fills up quickly. If you leave it on, the hang dumps you need can be pushed out. Turn it off when you finish reproducing the problem.


Loading and unloading the module

Message Level Meaning Action
ha: Initializing module info Module loading has started. Probe messages follow for each card —
ha: more than 8 cards; the extra cards take any free number, so card numbers can change across reloads warn More than 8 LPUs are installed. Only the 8 with the lowest PCI addresses get a reserved number, so after a reload the card numbers and the cards that get nodes can change. See Card numbers Keep 8 or fewer LPUs on one host
ha: Module initialized info Module loading is complete —
ha: Failed to create class, ha: Failed to allocate chrdev region, ha: Failed to register PCI driver err The module could not be loaded Contact the support team and include the kernel messages from before and after it
ha: Exiting module, ha: Module exited info Module unloading has started, and has finished. Between the two lines, a removal message is logged for each card —

When the module is first loaded, the kernel logs ha: loading out-of-tree module taints kernel.; this is the kernel's generic message for a module built outside the distribution. If you load an unsigned module on a kernel that does not enforce signature checking, ha: module verification failed: signature and/or required key missing - tainting kernel is also logged.


Device node probe messages

Normal probe

Each card logs messages in the following order. The values depend on the configuration, and when there are several cards, device-to-device path messages are added in between.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
ha <BDF>: Probing device
ha <BDF>: Mapped userbar BAR (hw_index=0) at virt=0x<address>, phys=0x<address>, bus=0x<address>, len=<bytes>
ha <BDF>:   - memory mapped
...
ha <BDF>: wait_boot: expecting lo=0x<value> hi=0x<value>
ha <BDF>: TW: pool BAR2 inbound base 0x<address> (bus 0x<address>), published by firmware
ha <BDF>: TW manager initialized (Pool: TW 1-15)
ha <BDF>: Mmap manager initialized
ha <BDF>: user mappings are <mode>
ha <BDF>: Device command manager initialized
ha <BDF>: HDMA completion: <mode> (ha_hdma_completion=<value>)
ha <BDF>: Device initialized successfully
ha <BDF>: HDMA ha_hdma_wr0 MSI-X: addr=0x<value>_<value> data=0x<value>
...
ha <BDF>: firmware ready after <ms> ms
ha <BDF>: ABI protocol v<n> (driver v<n>, min v<n>)
ha <BDF>: AEXP: bootloader default programmed after probe
ha <BDF>: Probe successful: /dev/ha<N> created
ha <BDF>: Config: vendor=0x20fb, device=0x<value>, class=0x<value>
ha <BDF>: Config: sub_vendor=0x<value>, sub_device=0x<value>

The HDMA completion line shows how the driver notices that a DMA engine has finished a transfer, and the ha_hdma_completion setting in effect.

  • If <mode> is MSI-X, each engine signals with an interrupt, and one HDMA … MSI-X line is logged for each DMA engine. In the name, ha_hdma_rd<n> is a read-direction engine and ha_hdma_wr<n> is a write-direction engine.
  • If <mode> is polling, the driver checks at short intervals for a completion mark that the engine leaves in host memory. No HDMA … MSI-X lines are logged.

With the default (auto), the driver uses MSI-X only when it can confirm bare metal, and polling otherwise, for example in a VM guest. If ha_hdma_completion is set to irq or poll, the driver uses that method.

If Probe successful appears, the node has been created. Even for a card whose firmware is not ready, probe runs to completion and creates the node, as long as the driver received the card-side BAR2 address. In that case, the warnings in Firmware readiness and compatibility are logged as well.

Errors that stop probe

If one of the messages below appears, probe of that card fails and no node is left. The kernel also logs ha: probe of <BDF> failed with error <err>.

Message Level Meaning Action
Failed to enable PCI device err The kernel could not enable the card Check the preceding PCI messages (such as resource allocation failures)
Failed to request <name> BAR (hw_index=<n>) err The card's BAR address region could not be claimed. Either another driver is using that region, or the BIOS could not assign an address Check the Region lines of sudo lspci -vv -d 20fb:. If no address was assigned, check the BIOS PCIe address space settings (64-bit MMIO, Above 4G Decoding, and so on)
BAR<n> length is 0 err No address was assigned to the BAR Same as above
Failed to map <name> BAR (hw_index=<n>) err The BAR could not be mapped into the kernel address space Check the preceding kernel messages
firmware booted but did not publish the BAR2 inbound base within <ms> ms; not binding the card. Firmware before 1.0.1 never publishes it and has to be reflashed; with newer firmware, bind the card again through /sys/bus/pci/drivers/ha/bind err The firmware booted but did not report the card-side BAR2 address. Whatever the server configuration, the driver does not bind the card without this address. The kernel does not log a probe of … failed line Bind the card again with echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind. If the same line appears again, reboot the firmware with sudo ha-hotreset --bdf <BDF> and check again. If it still appears, contact support
firmware has not booted, so the BAR2 inbound base is unknown; not binding the card. Once the firmware boots, bind the card again through /sys/bus/pci/drivers/ha/bind err Neither the firmware boot signal nor the card-side BAR2 address arrived while the driver waited, so it did not bind the card. A firmware boot sentinel not seen warning comes just before. The kernel does not log a probe of … failed line The driver does not bind the card by itself even if the firmware boots later. Wait a while, then bind it again with echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind. ha-hotreset boots the firmware again from the start, so use it only when the bind keeps being refused. If nothing changes, power-cycle the host, and if it recurs on the same card, consider replacing the card
Failed to name device, Failed to publish char device err The /dev/haN character device could not be created Contact the support team and include the preceding messages
Failed to initialize device (one of Failed to initialize memory manager, Failed to initialize TW manager, Failed to fix the SRAM translation window (<err>), Failed to initialize mmap manager, Failed to initialize devcmd manager, and Failed to initialize HDMA engine is logged before it), Failed to initialize activity manager, Failed to initialize scheduler, Failed to initialize scheduler resources, Failed to create sysfs attribute err Internal driver initialization failed. In most cases the kernel ran out of memory Check the memory state and reload the module. If it repeats, contact the support team
Failed to setup MSI-X err The card's interrupts (MSI-X) could not be set up Check that MSI is not disabled on the kernel command line (pci=nomsi)
ABI protocol mismatch: firmware v<n>, driver requires >= v<n>; refusing probe err The card firmware's protocol is older than the minimum version this driver requires The firmware must be upgraded to a version that matches this driver. Because the card is not registered with the driver, ha-fw-update cannot be used, so contact the support team. Supported environments
ABI protocol handshake: CSR unmapped err The firmware information region (CSR) of the firmware that was confirmed ready could not be accessed Hot reset the card or power-cycle the host, then check again

If there are more than 8 cards, from the 9th card onward only Probing device is logged, with no other messages after it, and no node is created. The driver handles up to 8 cards per host.

Warnings during probe

These are cases where probe completes but some functions are limited.

Message Level Meaning Action
AEXP: geometry is not the one the bootloader default assumes (…); leaving AEXP alone info The memory configuration of this card differs from the default configuration the driver knows, so the driver does not touch the card's address expander (AEXP) settings If it appears on a card where you did not expect it, report it to the support team

Probe warnings about the telemetry poller, the telemetry nodes, and hwmon registration are in Telemetry; warnings about the AER configuration are in Host-initiated resets and PCIe errors.


Firmware readiness and compatibility messages

The driver waits for the firmware in two stages, at probe time and after a reset. It first waits for the firmware boot signal (3 seconds at probe, 10 seconds after an SBR or a reset issued by the kernel), and then waits for the firmware to report that it is ready (3 seconds). A reset that reinitializes the firmware does not reboot the firmware, so it waits only for the ready report. Even if the firmware is not ready within the time limit, the node remains as long as the driver received the card-side BAR2 address, but until the firmware is ready, the telemetry read from the firmware information region (fw_version, serial) is unknown and application requests are rejected. If the firmware becomes ready later, the driver accepts it automatically. A card whose card-side BAR2 address did not arrive gets no node (Errors that stop probe).

Message Level Meaning Action
wait_boot: expecting lo=0x<value> hi=0x<value> info Waiting for the boot signal has started. These are the values it waits for. The progress lines while waiting (wait_boot: elapsed=…) are debug level and do not appear in the default kernel log —
Firmware boot timeout after <ms> ms err No boot signal arrived within the time limit See the messages that follow
firmware boot sentinel not seen within <ms> ms (<err>); CSR-derived telemetry will report "unknown" warn There was no boot signal at probe time. The firmware has not booted, or is still booting If a firmware has not booted, so the BAR2 inbound base is unknown line follows, the card was not bound; follow that line. Otherwise, read fw_version again after a short while. If it stays unknown, follow fw_version is unknown
firmware ready after <ms> ms info The firmware reported that it is ready —
firmware has no boot-ready flag (needs ABI >= v<n>); falling back to a fixed <ms> ms settle info This is an older firmware that has no ready indicator. The driver waits a fixed time and then treats the firmware as ready Check the firmware version in Supported environments
firmware not ready <ms> ms after boot sentinel err The boot signal was seen, but the firmware did not report that it was ready within the time limit See the messages that follow
firmware never reported ready (<err>); CSR-derived telemetry will report "unknown" and commands are refused … warn Readiness could not be confirmed at probe time. Either the firmware is still booting, or it failed its self-test during boot and did not report that it was ready Same as the firmware boot sentinel not seen line above
firmware boot not confirmed; skipping ABI protocol check (CSR may be mid-publication) warn Firmware readiness could not be confirmed, so the protocol check was skipped. It appears together with one of the two warnings above —
firmware v<x.y.z> predates the ABI protocol handshake (needs >= v<x.y.z>); skipping protocol check warn This is an older firmware that does not support the protocol check Check the firmware version in Supported environments
ABI protocol v<n> (driver v<n>, min v<n>) info The firmware's protocol version is compatible with the driver —
post-reset ABI protocol check FAILED -- device stays bound but its wire ABI is incompatible; unbind it err The firmware that booted again after the reset is not compatible with this driver. The card stays registered, but normal operation cannot be expected Do not use the card. As the message says, unbind the card from the driver and contact the support team

ABI protocol mismatch, which stops probe, is covered in Errors that stop probe. The command that unbinds a card from the driver is as follows.

1
echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/unbind

Telemetry messages

Message Level Meaning Action
firmware version CSR reads 0x<value>; reporting "unknown" warn The firmware version was read, but the value was not valid (0x00000000 or 0xffffffff). fw_version is unknown Reset the card and check again
telemetry stays "unknown": device is not ready (<err>) -- <reason> warn The firmware information region cannot be read yet, so fw_version and serial are unknown. If <reason> is CSR window will not bind, a window to access the information region could not be created; if it is not serviceable -- no boot sentinel, still booting, or the CSR reads back garbage, there is no boot signal, the firmware is still booting, or the value read is not valid Check the messages in Firmware readiness and compatibility
firmware booted after probe gave up; CSR-derived telemetry refreshed info A firmware whose readiness could not be confirmed at probe or reset became ready later, and fw_version and serial were read —
device latched ready but its CSR window has since gone dead; telemetry stays "unknown" warn The firmware information region of a card whose readiness was confirmed can no longer be read Check health and the error messages from the same time, and reset the card if necessary
sys_mon: no response for <n> consecutive polls (<ms> ms) - suspending telemetry polling. … err Temperature, power, and utilization polling received no response for consecutive polls (ha_sysmon_fail_limit times, 15 by default), so polling was stopped. This is because continuing to read an unresponsive card can cause some platforms to reset the entire host. The nodes obtained by polling become unknown Reset the card. If the reset succeeds, polling restarts
sys_mon: resuming telemetry polling info Polling that had been stopped has restarted —
sys_mon telemetry unavailable: <err> warn The telemetry poller could not be started at probe time. The temperature, power, and utilization nodes are unknown. Computation is not affected Check the memory state, then reload the module or hot reset the card
telemetry sysfs unavailable (<err>) warn The telemetry nodes under /sys/class/ha/haN/ could not be created. /dev/haN works normally Report it to the support team along with the kernel version
hwmon register failed (<err>) warn The card could not be registered as a hwmon device. This card does not appear in /sys/class/hwmon. sysfs telemetry is unaffected —

When the device is FATAL, the driver stops polling without logging anything, and the temperature, power, and utilization nodes become unknown after 5 seconds. health is faulted at this point. For the unknown conditions of each node, see sysfs and Telemetry.


Watchdog and hang messages

The driver waits for the response to a firmware command for up to ha_devcmd_ceiling_ms (12 seconds by default). For a long-running command, this ceiling is extended by the time the command needs. Separately, the watchdog checks the waits every ha_wd_poll_ms (500 ms by default) and gives advance notice when a firmware command wait exceeds ha_wd_hang_ms (8 seconds by default) or a DMA transfer wait exceeds ha_wd_hdma_hang_ms (4 seconds by default). How this works is described in Hang detection and the watchdog.

Watchdog

Message Level Meaning Action
watchdog: <watch> wait stalled (devcmd=<ms> ms hdma=<ms> ms) - no completion within the threshold warn A wait exceeded the watchdog threshold. <watch> is the side that newly crossed the threshold: devcmd (firmware command response), hdma (DMA transfer), or devcmd+hdma if both did. The values in parentheses are the wait times at that point. It can also appear for work that legitimately takes a long time, so this line alone does not mean a hang. It is logged only once per stall Check whether a response wait ceiling expiry message follows, and whether inflight is decreasing
watchdog: preempting without ha_auto_reset - the device will stay FATAL until … warn Because ha_wd_preempt=1, the watchdog cut off a stalled wait, but ha_auto_reset is off. If the device became FATAL because of the wait that was cut off, it stays that way until it is reset Reset the card. To recover automatically, use it together with ha_auto_reset=1. FATAL and recovery

Warnings about configurations in which the watchdog cannot fire are in Configuration warnings.

Response wait ceiling expiry and the hang dump

Message Level Meaning Action
Devcmd no response after <ms> ms (device hang): <verdict> err The firmware did not respond within the response wait ceiling (<ms>). devcmd_timeouts increases by 1, and two hang dump lines follow immediately. If the ceiling expires while waiting for application work, the device usually becomes FATAL (health is faulted). If it expires during a reset, the driver moves on to the next reset step See Reading the hang dump below and follow FATAL and recovery
Devcmd waiting <ms> ms with no timeout requested … warn An application made a request that waits without a ceiling, and the driver has been waiting <ms> ms for its response. It is logged first when the wait exceeds ha_devcmd_ceiling_ms, and then again every 10 seconds. The name and ID of the waiting thread are included Check on the application side whether this is work that legitimately takes a long time. Each time this line is printed, the progress is updated in sysfs last_hang. Read last_hang twice, at least 20 seconds apart; if batch_seq and exec_index stay the same, a single command is taking a long time or has stopped. If you read at a shorter interval, you may see the same values even while work is progressing. If you judge that it has stopped, save the diagnostic information, then end that process and reset the card
hang (<reason>): …, hang dump (<reason>): … warn Diagnostic block for a command that is not responding See Reading the hang dump below

Reading the hang dump

Two lines appear at the moment the driver determines that there is no response.

1
2
ha <BDF>: hang (<reason>): fw <command> (op 0x<op>), batch slot <n> (last staged, not necessarily the stuck one)<details>; opened by pid <pid> (<process>) -- <verdict sentence>
ha <BDF>: hang dump (<reason>): stream=0x<tag> corr=<n> opcode=0x<op> cmd_idx=<n> payload=[<words>] staged=<n> flushed=<n> mbox_head=<n> db=<n> hdma_wr=<value>/<value> hdma_rd=<value>/<value>
  • The first line is a human-readable summary, and the second line holds the raw values. Paste both lines into the support request as they are.
  • <reason> is one of ceiling timeout (the response wait ceiling expired), watchdog (watchdog verdict), and fw_update timeout (the firmware update time limit expired). A watchdog dump appears only for a wait without a ceiling, or when ha_wd_preempt=1.
  • fw <command> is the last command sent to the firmware. As the wording says, it may not be the command that stopped.
  • <details> holds items such as the address and size of the command when it is a kernel execution command.
  • opened by pid is the process that opened the card. If a handle was handed over and is used by another process, it can differ from the process that actually submitted the work. This part is attached only to commands whose owner can be determined.
  • stream=0x<tag> is a tag that masks the application's stream with a value that is different on every boot. Use it only to tell whether it is the same stream within one boot.
  • If (hdma unreadable: link down?) is appended to the end of the second line, the card's registers could not be read. The PCIe link is down or the card is not responding.

Verdict

The verdict is an estimate of where the firmware stopped, based on the progress the firmware reported when the ceiling expired. The token appears in the Devcmd … line and in verdict= of sysfs last_hang, and the sentence appears in the first hang line.

Token Sentence in the first hang line Meaning
not-picked-up firmware never picked up the batch The firmware did not pick up the command batch the driver sent
picked-up-no-dispatch firmware took the batch but dispatched no command The firmware picked up the command batch but did not start any command
stuck-in-command firmware started a command and never finished it The firmware started the command at the exec_index position in last_hang and did not finish it
completed-irq-lost every command completed; only the interrupt was lost, so a reset is NOT needed Every command finished, and only the completion interrupt was not delivered. The card itself has not stopped
rebooted the device rebooted underneath us The card rebooted while the driver was waiting
unsupported this firmware does not publish enough progress to judge The firmware does not report enough progress information, or the values read cannot be trusted, so no verdict can be reached. The progress fields of last_hang are -
none no progress data No progress information was recorded

For the watchdog reason, the first hang line ends with progress not sampled here, so no verdict instead of a verdict.

FATAL is not cleared even when the verdict is completed-irq-lost

The verdict is for diagnosis. Once the response wait ceiling passes, the driver handles the case the same way regardless of the verdict. If the device has become FATAL (health is faulted), you must reset it before you can use it again, even if the verdict is completed-irq-lost. If this verdict repeats, report it to the support team along with the dump.

Firmware errors

1
2
3
ha <BDF>: Firmware error: resq[<n>] rc=0x<rc> cmd_idx=<n> opcode=0x<op>
ha <BDF>:   resq dump (sent=<n>):
ha <BDF>:   resq[<i>] = 0x<value>

The firmware reported that a command failed (level err). After the first line, 12 lines of raw values from the firmware response area follow. devcmd_fw_errors increases by 1, and the same values are recorded in rc, cmd_idx, and opcode of sysfs last_hang. A rate limit shared by all cards applies, so if the message repeats many times only some occurrences are printed, but the counter counts every one of them.

The outcome depends on rc. The table below applies when the error came from application work.

rc Meaning Result
0x2 Unknown command Only that command fails
0x3 Invalid argument Only that command fails
0x4 Out of memory on the firmware side Only that command fails
0x5 Buffer overflow Only that command fails
0x8 Core error limited to that command (command decoding error, division by 0, and so on) Only that command fails
0xa~0xd The result of a firmware update. See Firmware update Only that command fails
0x1 The device is not ready The device becomes FATAL
0x7 The firmware ended only that command with a timeout. The device can still answer commands The device becomes FATAL. This differs from a wait that expires with no response at all, and a soft reset can still reach the firmware. See Response wait ceiling
0x9 Fatal core error (multi-bit ECC error, address range exceeded, and so on) The device becomes FATAL
Any other value Unknown result The device becomes FATAL

"Only that command fails" means that the card can keep being used, so look for the cause in the application log. If the device became FATAL, follow FATAL and recovery.

Data transfer (DMA)

Copies between the host and a card, and between cards, are handled by the DMA engines of the card. There are several engines per direction. In the messages, <direction> is read for an engine that reads over PCIe and writes into card memory (copies from the host to the card, file loads, and copies pulled from another card) and write for an engine that reads card memory and writes over PCIe (copies from the card to the host and copies pushed to another card), and <ch> is the channel number of the engine.

Message Level Meaning Action
ll: timeout (n=<n> channel=<ch> <direction>) err A DMA transfer failed because it did not finish within 5 seconds. hdma_timeouts increases by 1. If the driver can stop the engine, only the work that the transfer belongs to fails and the device does not become FATAL. If it cannot stop the engine, a HDMA engine at … line follows immediately and the device becomes FATAL. If the engine had already stopped, the ll: engine stopped … line below is logged instead of this one Run the failed work again. If it repeats on the same card, save the diagnostic information and contact the support team. Also check the PCIe error messages from the same time and the link speed and width (current_link_speed, current_link_width)
ll: engine stopped but its completion <mode> never arrived (status=<n> channel=<ch> <direction>) err The driver did not receive the completion signal of a DMA transfer within 5 seconds, but the engine had already stopped. The completion signal that the engine sent did not reach the driver. hdma_timeouts increases by 1, and only the work that the transfer belongs to fails. <mode> is MSI-X when completion comes by interrupt and write when the driver polls; this write is unrelated to the <direction> at the end of the line. If <mode> is MSI-X and every copy logs this line, the driver has usually mistaken a VM guest for bare metal (an x86 guest whose CPU hides the hypervisor flag, or an arm64 guest set up under nested virtualization to run its own hypervisor). The same happens when ha_hdma_completion=irq is given in a VM If <mode> is MSI-X and every copy logs this line, reload the driver inside the VM guest with ha_hdma_completion=poll. If the line appeared only once, treat it like ll: timeout. If <mode> is write, or it repeats after the change, save the diagnostic information and contact the support team
ll: n=<n> channel=<ch> <direction> ret=<n> status=<n> int_status=0x<value> error=<n> err The DMA engine reported an error Same as above
ll: ABORTED error=<n> err The DMA transfer was aborted. Only the work that the transfer belongs to fails Same as above
ll: no response (channel=<ch> <direction>), ring write: no response (channel=<ch> <direction>) err The card does not respond to the status register of the DMA engine (every bit of the value read is 1). The link may be down, or the card may have stopped or been removed. The transfer fails and the device becomes FATAL Follow FATAL and recovery. Also check the PCIe error messages from the same time
HDMA engine at 0x<offset> <state>; quarantining its pages until the engines stop (reset or remove) crit <state> is not responding (the engine does not respond) or still running after force stop (the engine is still running even after being force-stopped). The engine may still be able to write to host memory, so the driver does not release the host pages that the transfer was using but quarantines them, and blocks new DMA transfers until all engines stop. The device becomes FATAL Reset the card. health is faulted
HDMA engines stopped, accepting transfers again info The engines that could not be stopped have all been confirmed stopped, and DMA transfers are accepted again —
HDMA: freed <n> quarantined batch(es) of the <direction> engine on channel <ch> info The quarantined host pages were released after the engine stopped —
HDMA: device not responding at reset, engines not stopped warn The card did not respond during the reset, so it could not be confirmed that the engines stopped. The driver checks again after the reset finishes After the reset finishes, check whether a device reset failed line was logged. Reset progress and result
HDMA: no free <direction> engine after 10 s warn No <direction> engine became free for the copy for 10 seconds. It repeats every 10 seconds while waiting Check the hang and DMA messages from the same time, and inflight
HDMA: <direction> engine <ch>: pipeline stage not finished after <n> s, still waiting warn The preparation stage of a copy (pinning and mapping host pages) or its cleanup stage (unmapping) has not finished for <n> seconds, and the driver keeps waiting. It repeats every 10 seconds. It appears mainly when securing the pages of the host buffer takes a long time Check the memory state of the host
HDMA: bounce-buffered DMA, using <n>-page batches info Logged once when the card is registered. On this host, the card's DMA goes through a bounce buffer (swiotlb) and the size that can be mapped at a time is limited, so copies are split into smaller units. Copy throughput can be lower Check the IOMMU and DMA settings to see whether this is the intended configuration
HDMA: batches of <n>-<n> pages, <n> slots, <n> hdma_pre_wk, <n> channel(s) x 2 engines of 2 x <n> LL entries info A summary of the DMA processing configuration, logged once when the card is registered —
ha: stream 0x<value> waits for a prefetch copy warn The wait for the input copy that the driver started ahead of time (ha_abi_prefetch) to finish has lasted more than 10 seconds. It repeats every 10 seconds while waiting. 0x<value> is a value that identifies the application's stream Check the hang and DMA messages from the same time, and health

Reset messages

A card is reset either by an application through the SDK's reset function, or by the driver's automatic reset (ha_auto_reset=1). A reset first reinitializes the firmware, and if that step fails, it issues an SBR (Secondary Bus Reset) on the parent PCIe port. An SBR is issued only in a configuration where the parent is a root port or the parent port has masked Surprise Down errors. On success, firmware ready after <ms> ms and AEXP: bootloader default programmed after device reset are usually logged. ha-hotreset removes the card and has it probed again, so removal and probe messages are logged. How a reset behaves is described in FATAL and recovery.

Automatic reset

Message Level Meaning Action
auto-TDR: the device went FATAL; resetting it warn The device became FATAL, and the driver starts an automatic reset Check the result messages that follow
auto-TDR: reset failed (<err>); auto-TDR will not run again on this card until a reset succeeds or the card reports ready again err The automatic reset failed. Automatic reset for this card is suspended until one reset succeeds or the card becomes ready again, and health is faulted Do what the device reset failed line just before it says
auto-TDR: this instance was removed during the reset (<err>); re-probe decides whether the card comes back warn The card was removed from the driver during the reset. For example, the SBR took the link down and the operating system removed the card. If the card is detected again, it is probed anew Confirm recovery with a Probe successful message for the same <BDF>

Messages about skipping the automatic reset because of a firmware update are in Firmware update.

Reset progress and result

Message Level Meaning Action
issuing SBR via parent <bridge> warn Recovery by reinitializing the firmware did not work, so an SBR is issued on the parent port. The card reboots —
parent <bridge> leaves Surprise Down unmasked; refusing SBR (would reboot the host) warn The parent port does not mask Surprise Down errors, so the SBR was refused. In this configuration, taking the link down without masking can make the platform reset the entire host Run sudo ha-hotreset --bdf <BDF> on the host. ha-hotreset masks the path and resets the card. ha-hotreset
parent <bridge> has no AER capability, so the Surprise Down mask cannot be checked; refusing SBR warn The parent port has no AER capability, so it cannot be checked whether the error is masked, and the SBR was refused ha-hotreset cannot reset this card either. Power-cycle the host to recover. sudo ha-hotreset --list lets you check in advance, for each card, whether the configuration allows an SBR
parent <bridge>: cannot tell whether Surprise Down is masked (no PCIe capability, or the mask is unreadable); refusing SBR warn The masking state could not be read, so the SBR was refused Same as above
No parent bridge, SBR not supported err The card has no parent bridge, so an SBR cannot be issued Same as above
device did not answer after SBR (parent link <state>) err The card did not respond within 10 seconds after the SBR. <state> is the link state of the parent port (up, down, unknown) Run sudo ha-hotreset --bdf <BDF> on the host, and if that does not recover it, power-cycle the host
removed during SBR (<cause>); this instance is gone - re-probe decides whether the card comes back warn The operating system removed the card during the SBR. <cause> is link lost (the link went down) or driver unbind. If the card is detected again, it is probed anew Confirm recovery with the Probe successful message. Applications must reopen the node
removed after SBR while the reset was still running (<err>); this instance is gone - … warn The card was removed while the reset was being completed after the SBR Same as above
device came back but firmware did not report ready (<err>); reset reported as failed warn The card came back, but the firmware did not report that it was ready within ha_ready_timeout_ms (3 seconds by default), so the reset was reported as failed. Application requests are rejected until the firmware is ready Check fw_version after a short while. If it stays unknown, a hot reset or a host power cycle is required
AEXP: bootloader default programmed after <event> info After a probe or a reset, the card's address expander (AEXP) was set again to the bootloader default. <event> is one of probe, device reset, and host reset —
AEXP: bootloader default not programmed after <event> (<err>); state unknown warn The setting above could not be written. It is written again at the next reset Check the other error messages from the same time
ha: quiesce drain stalled <s>s in <process>(pid <pid>): inflight=<n> fileio=<n> leases=<n> warn A reset or removal has been waiting more than 10 seconds for in-progress work to finish. It repeats every 10 seconds. <process> is the one that is waiting. It shows which of inflight (work submitted to the card), fileio (read and write calls), and leases (other firmware commands) has not finished If it continues, save the diagnostic information and contact the support team
device reset failed (<err>, firmware re-init <err>): <advice> err The reset did not recover the card. This is the last line the reset procedure leaves (an automatic reset adds one more, auto-TDR: reset failed (<err>); ...), and health stays faulted. Depending on the case, <advice> says whether to reset again, run ha-hotreset --bdf <BDF> on the host, or power-cycle the host, and what to do if that does not work either Follow <advice>. The lines before it show which stage failed

If the firmware protocol does not match after a reset, post-reset ABI protocol check FAILED is logged. See Firmware readiness and compatibility.


Host-initiated resets and PCIe error messages

A card is also reset from outside the driver: when the kernel resets the card (sysfs reset, a bus reset for another device under the same bridge, and so on) and when it recovers from a PCIe error (AER, DPC). The driver receives a notification, stops and cleans up its work, and all open handles to that card become invalid. Applications must reopen the node.

Message Level Meaning Action
reset_prepare: quiescing for a host-initiated reset info The kernel is about to reset the card, so the driver stops and cleans up its work If the reset was unexpected, find out what triggered it. Do not reset this card through sysfs reset
reset_done: reopening the gate (boot <result>) info The reset has finished, and the driver reopens work submission. <result> is ok or unconfirmed If it is unconfirmed, see the line below
reset_done: firmware boot not confirmed (<err>) warn There was no firmware boot signal within 10 seconds after the reset Hot reset the card or power-cycle the host
firmware did not report ready after host reset (<err>); CSR caches dropped, AEXP state unknown warn The firmware did not report that it was ready within ha_ready_timeout_ms (3 seconds by default) after the reset. fw_version and serial are unknown, and application requests are rejected until the firmware is ready Check fw_version after a short while. If it stays unknown, a hot reset or a host power cycle is required
PCIe error detected (state=<n>): <kind> err PCIe error recovery has started. The driver stops its work and waits for the kernel's link reset. <kind> is recoverable or permanent failure, and state is 1 (normal), 2 (I/O blocked), or 3 (permanent failure) Check the kind of error in the kernel AER message immediately before it
slot_reset: re-enable failed err The card could not be re-enabled after the link reset, so error recovery ended in failure Power-cycle the host
slot_reset: firmware did not come back (<err>) err There was no firmware boot signal within 10 seconds after the link reset, so error recovery ended in failure Hot reset the card or power-cycle the host
PCIe error_resume: recovery complete info PCIe error recovery is complete Check that the application has reopened the node
PCIe recovery never reached slot_reset within <ms> ms; reopening the submit gate. The device was not re-initialised … err Error recovery did not reach the link reset step within 30 seconds, so the driver reopened work submission. The card was not reinitialized If the card does not work normally, reset the card
AER reporting enabled: the kernel's fatal remedy is a parent-bridge SBR warn The module was loaded with ha_aer_enable=1. For fatal PCIe errors, the kernel recovers with a parent-bridge SBR, and this reset does not go through the driver's Surprise Down check Check that this is the intended setting. Parameter reference
AER reporting unavailable; error handlers stay idle info ha_aer_enable=1 is set, but the kernel could not enable AER reporting for this card (for example, in a configuration where platform firmware manages AER). It appears only on some kernel versions —

Even when ha_aer_enable is off, error recovery messages can appear because of DPC, a fatal error on another device, and so on. On kernel 6.4 or later and RHEL 9.4 or later, they can also appear because of the card's own errors.


Firmware update messages

These are the messages logged while and after ha-fw-update performs an update. The procedure and safety rules are in Firmware update.

Message Level Meaning Action
fw_update: pa=0x<address> size=<bytes> scope=<n> flags=0x<value> crc32=0x<value> budget=<ms> ms info The firmware update command is sent. budget is the time, calculated from the image size, that is added to the response wait ceiling —
ha: fw_update: discarded <n> stale staged command(s) left by a failed batch warn Commands left behind by an earlier failed job were discarded before the update —
fw_update: nothing sent (CSR window unavailable) warn The update command could not be sent. The flash is unchanged Run the update again
fw_update: nothing sent (staging cleared by an abort) warn The pending command was cleared by a reset or similar, so it could not be sent. The flash is unchanged Run the update again
fw_update: device vanished during the write (all-ones) err The card stopped responding during the write. The flash state is unknown Do not power off or reboot the host; follow Firmware update
fw_update: no response after <ms> ms: <verdict> err The firmware did not report the update result within the time limit. A fw_update timeout hang dump follows. The flash state is unknown Same as above
fw_update: CSR window unavailable after completion err The firmware reported completion, but the driver could not read the result, so it treated the flash state as unknown. A fw_update: no answer from firmware line follows Do not reboot the host or reset the card; contact the support team
fw_update: no answer from firmware (<err>); resets stay refused until it answers or the update is retried err The firmware's response could not be confirmed after the command was sent. The firmware may still be writing the flash, so resets and application work on this card are rejected until a response is confirmed. The step in which ha-fw-update uploads the image to card memory is also rejected, so running it again in this state ends with upload failed: Device or resource busy (exit code 2), and the flash is not changed Do not reboot the host or reset the card. When the firmware responds and fw_update: late completion from firmware; resets re-enabled is logged, run ha-fw-update again with the same image. See Firmware update
fw_update: firmware answered; resets re-enabled warn The firmware answered an update command that was sent again while the response was unconfirmed, so the rejection was lifted. In this state ha-fw-update has its upload step rejected, so the rejection is usually lifted by the late completion line below —
fw_update: late completion from firmware; resets re-enabled warn The rejection was lifted by a completion response that arrived late. This line does not carry the result of the write Run the update again with the same image and finish it through the reset and verification
reset refused: last fw_update unanswered (flash may still be written); retry ha-fw-update once the firmware answers, else contact HyperAccel support warn The reset request was rejected because there is an update whose response was not confirmed After the firmware responds and fw_update: late completion from firmware; resets re-enabled is logged, run ha-fw-update again. If the firmware never responds, contact the support team. Reloading the module, ha-hotreset, and power cycling risk booting from a partially written flash, so use them only after consulting the support team
auto-TDR: skipped, a fw_update is in flight or unanswered (flash may be mid-write) warn The automatic reset was skipped because an update is in progress or there is an update whose response was not confirmed. The device remains FATAL If an update is in progress, wait until it finishes. If ha-fw-update ended with exit code 4, ended without reporting an exit code, or fw_update: no answer from firmware was logged, do not reset the card, even after … resets re-enabled is logged. That line does not carry the result of the write, and while the device is FATAL, running the update again is rejected with device allocation failed: State not recoverable, so contact the support team. In other cases, if health is still faulted after the update finishes, reset the card
auto-TDR: skipped, the fw_update it queued behind ended unanswered warn The automatic reset was skipped for the same reason Same as above
reset_prepare: last fw_update unanswered - resetting now may leave the boot flash half-written err The kernel is resetting the card even though there is an update whose response was not confirmed. The driver cannot block a reset that the kernel issues If the card does not boot after the reset, contact the support team

The result code that the firmware returns for the update command (rc=0xa~0xd) is also logged in a Firmware error line. The response to a result code is decided from the exit code of ha-fw-update. See ha-fw-update.


Configuration warning messages

Message Level Meaning Action
watchdog: <watch> arm cannot fire with this configuration (hang=<ms> ms hdma_hang=<ms> ms poll=<ms> ms devcmd_ceiling=<ms> ms hdma_timeout=<ms> ms); lower the threshold or the poll interval warn The sum of the watchdog threshold and the polling interval is equal to or greater than the ceiling for that wait (ha_devcmd_ceiling_ms for devcmd, 5000 ms for hdma), so that watch cannot report a wait stall. The driver does not change the values; it only reports this. It is printed only once for a given set of values, and printed again if the values change Lower ha_wd_hang_ms, ha_wd_hdma_hang_ms, and ha_wd_poll_ms, or raise ha_devcmd_ceiling_ms. Relationships between parameters
ha: invalid ha_hdma_cpu_latency_us=<value> (-1 or >= 0), using 0 warn ha_hdma_cpu_latency_us was given a value below -1, so the module was loaded with 0 Correct the value in the option file. Parameter reference
watchdog: preempting without ha_auto_reset - … warn See Watchdog —
AER reporting enabled: … warn See Host-initiated resets and PCIe errors —
ha: `<value>' invalid for parameter `<name>' err A value of the wrong type, or a value the parameter does not accept, was written in the options file, on the kernel command line, or on the modprobe command line, so module loading failed Correct the value, then reload the module. If the value was written on the kernel command line, correct the command line and reboot. How to change
ha: unknown parameter '<name>' ignored warn The module was loaded, and the parameter with the wrong name was ignored Correct the name. Parameter reference

For the other parameters that have a defined range (ha_devcmd_ceiling_ms, ha_wd_poll_ms, ha_tw_pin_wait_ms, ha_sysmon_poll_ms), a value outside the range is clamped to the nearest end of the range, and no separate message is logged. The ranges are in Parameter reference.


Device-to-device path messages

These are the messages logged when there are several cards. The meaning of the paths is explained in Multiple devices, and the sysfs nodes that show the same information (peer_map, peer_route, peer_sram) are in sysfs.

Message Level Meaning Action
P2P peer[<n>]: <path> info The verdict for the PCIe path from this card to /dev/ha<n>. direct is a path that stays under a common upstream bridge without going through the host; host-mem is a path that goes through the host (root complex); unknown means that the configuration of a bridge on the path could not be read. It is the same value as peer_route —
dma_map_resource failed for peer[<n>] warn The BAR of /dev/ha<n> could not be mapped into this card's DMA address space. Device-to-device copies between this card and /dev/ha<n> cannot be performed (peer_map is unmapped) Check the IOMMU configuration
ACS state unreadable; P2P route stays unknown warn The ACS settings of this bridge could not be read, so the path could not be determined. The prefix of this message is the bridge, not the card —
outbound: <BDF> <path> base 0x<address> info The path and address by which the device kernel running on this card accesses the SRAM of another LPU (<BDF>). <path> is bus or iova. It is the same information as peer_sram —
outbound: <BDF> remap:<k> base 0x<address> info Same as above; a path created by the driver reconfiguring outbound window <k> —
outbound: region <k> -> 0x<address> (<BDF>), outbound: adopted region <k> (<BDF>) info Outbound window <k> was set up for the path above, or the window left by a previously loaded driver is reused —
outbound: <BDF> none:<reason> info The driver does not tell the device kernel running on this card the SRAM address of that LPU. <reason> is the same as the none: reason of peer_sram See peer_sram in sysfs
outbound ATU[<k>]: enable timeout, outbound ATU[<k>]: readback … does not match warn The outbound window could not be set up. It can appear while the card is being reset or when it does not respond. The path to that LPU remains unknown in peer_sram Check peer_sram and health. If the path stays unknown, reset the card
outbound: no memory, kernels on this card get no windows warn Memory ran short, so the outbound windows for this card could not be created Check the memory state, then hot reset the card or reload the module
ha: outbound: more than 32 LPUs, ignoring <BDF> warn More than 32 LPUs are installed, so that card was left out of the path calculation —
TW pin: <n>/<max> slots pinned for <ms>ms, giving up (da=0x<address>) warn The translation window needed for a device-to-device copy could not be obtained for ha_tw_pin_wait_ms (12 seconds by default), so the copy failed. The card in the prefix is the one that is short of windows, that is, the peer card of the copy Reduce the number of simultaneous copies directed at one card

Device memory access and application request messages

Message Level Meaning Action
ATU[<k>]: enable timeout warn Setting up the window the host uses to access device memory timed out. It can appear when the card does not respond Check health and the error messages from the same time
TW: no free pool window (<n> pinned, <n> held by kernel access, <n> fixed, <n> in pool); <operation> failed err There is no free window in the translation window pool, so a new access could not be handled. The parentheses show how the windows of the pool are being used: pinned is windows pinned by device-to-device copies, exports, and so on; held by kernel access is windows that the driver is holding for its own accesses; fixed is windows fixed for SRAM access; and in pool is the total number of windows in the pool. <operation> is the operation that failed (bind, pin, hold, or SRAM reserve). An application that was using mapped device memory can receive an error (SIGBUS) Reduce the device-to-device copies and exports in progress at the same time
TW: write to da 0x<address> did not reach the device err After the host wrote to device memory, the value read back did not match the value written. That request ends with an I/O error. If the same message repeats within a short time, only some are logged. It can appear when the card is not responding or there is a problem with the link Check health and the PCIe error messages from the same time

When an application sends an invalid request, the driver may reject it and log the reason. Examples are an alignment error in a mapping size or offset, an unallocated address, or freeing memory that the application does not own. These messages are unrelated to the card state. If they repeat, pass them to the application developer together with the application log.


dma-buf export messages

Messages logged when an application requests a dma-buf export so that a peer device such as an RDMA NIC can access the DRAM of the card directly. What an export is and how to see the reserved windows are in sysfs nodes. <device> in the messages is the name (PCI address) of the peer device.

Message Level Meaning Action
ha: dma-buf attach by <device> refused: device gone or reset info The peer device's attempt to attach to the export was refused because the card was reset or removed after the export. A new attachment cannot be made to an export that a reset has passed over The application must export again and register the new export with the peer device
ha: dma-buf attach by <device> refused: importer does not take peer-to-peer addresses info Refused because the peer device and its driver stated that they do not accept addresses for direct access between PCIe devices (peer-to-peer). This is a cause of memory registration failing Check whether that device and its driver support peer-to-peer
ha: dma-buf map #<n> by <device> refused: device gone or reset info For the same reason as above, the address request of an already attached peer device was refused Same as above
ha: dma-buf map #<n> by <device>: dma_map_resource failed for <address> warn The BAR address of the card could not be mapped as a DMA address of the peer device Check the IOMMU configuration
TW export: region <n> readback does not match target 0x<address> warn The export window could not be set up, so that export failed. It can appear when the card is being reset or is not responding Check health and the error messages from the same time
TW export: region <n> still enabled after <when> warn The driver turned off an export window (<when> is boot or reset), but the window was still read as enabled If it repeats, save the diagnostic information and contact the support team
TW teardown with <n> dma-buf export window(s) reserved; a later bind may reuse their regions under a registered MR warn When the card was removed, the reservations of <n> export windows remained. If the peer device has not released its memory registration (MR), those windows can be used for other DRAM ranges after this card is recognized again, and the peer device can access ranges that were not intended Before removing the card, hot-resetting it, or unloading the module, end the applications that use dma-buf and release the memory registrations

Device removal messages

These are the messages logged when a card is removed from the driver (module unload, unbind, hot reset, hot unplug).

Message Level Meaning Action
Removing device info Card removal has started —
remove: <n> fd op(s) still inside the gate warn Waiting for <n> calls in progress on open handles to finish. It repeats every 0.2 seconds while waiting If it continues, check the processes that are using that card
Mmap manager cleanup, Device cleanup complete info Cleanup steps —
ha: abi prefetch issued=<n> consumed=<n> info The cumulative count of input copies started ahead of time on this card, with the same values as issued and consumed in sysfs abi_prefetch_stats. If there were copies started ahead of time, one line is logged at removal —
Remove complete info Card removal is complete —

If in-progress work does not finish for a long time, ha: quiesce drain stalled … is logged as well. See Reset progress and result.


See also