Kernel log messages¶
This page collects the messages that ha.ko writes to the kernel log (dmesg), organized by situation. Each message lists its meaning and what the operator should do, and detailed procedures link to the relevant feature pages. If you are starting from a symptom, see Troubleshooting first.
Message format¶
Every driver message described on this page starts with one of two prefixes.
| Prefix | Example | Used for |
|---|---|---|
ha <BDF>: |
ha 0000:01:00.0: Probing device |
Messages about a single card. <BDF> is the card's PCI address |
ha: |
ha: Initializing module |
Messages about the module as a whole, and some card messages that are logged without a <BDF> |
- Find the mapping between
<BDF>and/dev/haNin theProbe successful: /dev/ha<N> createdline, theBus-Idcolumn ofhyperaccel-smi, or the/sys/class/ha/haN/devicelink. - Messages that the kernel logs about this card (such as PCIe error reports) can also appear with the same
ha <BDF>:prefix. Messages about the upstream PCIe port that the card is attached to appear with that port's prefix (for example,pcieport <BDF>:). - In the logs described on this page,
<...>is a value that changes from case to case. For long messages, only the first part is shown, with…added at the end. When searching, use the first part. - A negative number in parentheses is a Linux kernel error number. Common values are
-110(timeout),-5(I/O error),-11(not ready yet, try again),-19(no such device),-95(not supported, for example a configuration in which an SBR cannot be issued),-121(remote I/O error), and-4(interrupted by a signal). - The Level column of each table is the name used with
dmesg --level.erris an error,warnis a warning,infois informational, andcritis a severe error. - If the same message repeats within a short time, only some of the occurrences are printed, and the kernel reports the number it omitted in a
callbacks suppressedline.
Kernel warning traces
If the call trace in a block that starts with WARNING: contains [ha], the driver's internal check has been triggered. Attach the complete log from before and after it to the support request.
Viewing logs¶
1 2 3 4 5 6 7 8 9 10 11 | |
Messages from the upstream PCIe port have a different prefix and are not matched by the filter above. Attach the full dmesg output, without the filter, to a support request.
Save the log before you reset
A hot reset or a module reload clears the sysfs error counters and last_hang, and the messages that a reset leaves can push earlier records out of the kernel log buffer. Before you reset a card, save the full dmesg output and the sysfs values. A script that saves them all at once is in Collecting diagnostic information for a problem.
Enabling detailed logs¶
The driver leaves the detailed per-command log (debug level) off by default. You can turn it on with the kernel's dynamic debug, which requires the kernel configuration option CONFIG_DYNAMIC_DEBUG=y.
1 2 3 4 5 | |
1 2 3 4 5 6 7 8 | |
- To use
/sys/kernel/debug/dynamic_debug/control, debugfs must be mounted at/sys/kernel/debug. If debugfs is not available, check whether/proc/dynamic_debug/controlexists; if it does, replace the path in the commands above with that file. The format and commands are the same. - The enabled log is at
debuglevel, so it does not appear on the console; read it withdmesg.
Enable it only while you reproduce the problem
A log line is written for every command, so the kernel log buffer fills up quickly. If you leave it on, the hang dumps you need can be pushed out. Turn it off when you finish reproducing the problem.
Loading and unloading the module¶
| Message | Level | Meaning | Action |
|---|---|---|---|
ha: Initializing module |
info | Module loading has started. Probe messages follow for each card | — |
ha: more than 8 cards; the extra cards take any free number, so card numbers can change across reloads |
warn | More than 8 LPUs are installed. Only the 8 with the lowest PCI addresses get a reserved number, so after a reload the card numbers and the cards that get nodes can change. See Card numbers | Keep 8 or fewer LPUs on one host |
ha: Module initialized |
info | Module loading is complete | — |
ha: Failed to create class, ha: Failed to allocate chrdev region, ha: Failed to register PCI driver |
err | The module could not be loaded | Contact the support team and include the kernel messages from before and after it |
ha: Exiting module, ha: Module exited |
info | Module unloading has started, and has finished. Between the two lines, a removal message is logged for each card | — |
When the module is first loaded, the kernel logs ha: loading out-of-tree module taints kernel.; this is the kernel's generic message for a module built outside the distribution. If you load an unsigned module on a kernel that does not enforce signature checking, ha: module verification failed: signature and/or required key missing - tainting kernel is also logged.
Device node probe messages¶
Normal probe¶
Each card logs messages in the following order. The values depend on the configuration, and when there are several cards, device-to-device path messages are added in between.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | |
The HDMA completion line shows how the driver notices that a DMA engine has finished a transfer, and the ha_hdma_completion setting in effect.
- If
<mode>isMSI-X, each engine signals with an interrupt, and oneHDMA … MSI-Xline is logged for each DMA engine. In the name,ha_hdma_rd<n>is a read-direction engine andha_hdma_wr<n>is a write-direction engine. - If
<mode>ispolling, the driver checks at short intervals for a completion mark that the engine leaves in host memory. NoHDMA … MSI-Xlines are logged.
With the default (auto), the driver uses MSI-X only when it can confirm bare metal, and polling otherwise, for example in a VM guest. If ha_hdma_completion is set to irq or poll, the driver uses that method.
If Probe successful appears, the node has been created. Even for a card whose firmware is not ready, probe runs to completion and creates the node, as long as the driver received the card-side BAR2 address. In that case, the warnings in Firmware readiness and compatibility are logged as well.
Errors that stop probe¶
If one of the messages below appears, probe of that card fails and no node is left. The kernel also logs ha: probe of <BDF> failed with error <err>.
| Message | Level | Meaning | Action |
|---|---|---|---|
Failed to enable PCI device |
err | The kernel could not enable the card | Check the preceding PCI messages (such as resource allocation failures) |
Failed to request <name> BAR (hw_index=<n>) |
err | The card's BAR address region could not be claimed. Either another driver is using that region, or the BIOS could not assign an address | Check the Region lines of sudo lspci -vv -d 20fb:. If no address was assigned, check the BIOS PCIe address space settings (64-bit MMIO, Above 4G Decoding, and so on) |
BAR<n> length is 0 |
err | No address was assigned to the BAR | Same as above |
Failed to map <name> BAR (hw_index=<n>) |
err | The BAR could not be mapped into the kernel address space | Check the preceding kernel messages |
firmware booted but did not publish the BAR2 inbound base within <ms> ms; not binding the card. Firmware before 1.0.1 never publishes it and has to be reflashed; with newer firmware, bind the card again through /sys/bus/pci/drivers/ha/bind |
err | The firmware booted but did not report the card-side BAR2 address. Whatever the server configuration, the driver does not bind the card without this address. The kernel does not log a probe of … failed line |
Bind the card again with echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind. If the same line appears again, reboot the firmware with sudo ha-hotreset --bdf <BDF> and check again. If it still appears, contact support |
firmware has not booted, so the BAR2 inbound base is unknown; not binding the card. Once the firmware boots, bind the card again through /sys/bus/pci/drivers/ha/bind |
err | Neither the firmware boot signal nor the card-side BAR2 address arrived while the driver waited, so it did not bind the card. A firmware boot sentinel not seen warning comes just before. The kernel does not log a probe of … failed line |
The driver does not bind the card by itself even if the firmware boots later. Wait a while, then bind it again with echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind. ha-hotreset boots the firmware again from the start, so use it only when the bind keeps being refused. If nothing changes, power-cycle the host, and if it recurs on the same card, consider replacing the card |
Failed to name device, Failed to publish char device |
err | The /dev/haN character device could not be created |
Contact the support team and include the preceding messages |
Failed to initialize device (one of Failed to initialize memory manager, Failed to initialize TW manager, Failed to fix the SRAM translation window (<err>), Failed to initialize mmap manager, Failed to initialize devcmd manager, and Failed to initialize HDMA engine is logged before it), Failed to initialize activity manager, Failed to initialize scheduler, Failed to initialize scheduler resources, Failed to create sysfs attribute |
err | Internal driver initialization failed. In most cases the kernel ran out of memory | Check the memory state and reload the module. If it repeats, contact the support team |
Failed to setup MSI-X |
err | The card's interrupts (MSI-X) could not be set up | Check that MSI is not disabled on the kernel command line (pci=nomsi) |
ABI protocol mismatch: firmware v<n>, driver requires >= v<n>; refusing probe |
err | The card firmware's protocol is older than the minimum version this driver requires | The firmware must be upgraded to a version that matches this driver. Because the card is not registered with the driver, ha-fw-update cannot be used, so contact the support team. Supported environments |
ABI protocol handshake: CSR unmapped |
err | The firmware information region (CSR) of the firmware that was confirmed ready could not be accessed | Hot reset the card or power-cycle the host, then check again |
If there are more than 8 cards, from the 9th card onward only Probing device is logged, with no other messages after it, and no node is created. The driver handles up to 8 cards per host.
Warnings during probe¶
These are cases where probe completes but some functions are limited.
| Message | Level | Meaning | Action |
|---|---|---|---|
AEXP: geometry is not the one the bootloader default assumes (…); leaving AEXP alone |
info | The memory configuration of this card differs from the default configuration the driver knows, so the driver does not touch the card's address expander (AEXP) settings | If it appears on a card where you did not expect it, report it to the support team |
Probe warnings about the telemetry poller, the telemetry nodes, and hwmon registration are in Telemetry; warnings about the AER configuration are in Host-initiated resets and PCIe errors.
Firmware readiness and compatibility messages¶
The driver waits for the firmware in two stages, at probe time and after a reset. It first waits for the firmware boot signal (3 seconds at probe, 10 seconds after an SBR or a reset issued by the kernel), and then waits for the firmware to report that it is ready (3 seconds). A reset that reinitializes the firmware does not reboot the firmware, so it waits only for the ready report. Even if the firmware is not ready within the time limit, the node remains as long as the driver received the card-side BAR2 address, but until the firmware is ready, the telemetry read from the firmware information region (fw_version, serial) is unknown and application requests are rejected. If the firmware becomes ready later, the driver accepts it automatically. A card whose card-side BAR2 address did not arrive gets no node (Errors that stop probe).
| Message | Level | Meaning | Action |
|---|---|---|---|
wait_boot: expecting lo=0x<value> hi=0x<value> |
info | Waiting for the boot signal has started. These are the values it waits for. The progress lines while waiting (wait_boot: elapsed=…) are debug level and do not appear in the default kernel log |
— |
Firmware boot timeout after <ms> ms |
err | No boot signal arrived within the time limit | See the messages that follow |
firmware boot sentinel not seen within <ms> ms (<err>); CSR-derived telemetry will report "unknown" |
warn | There was no boot signal at probe time. The firmware has not booted, or is still booting | If a firmware has not booted, so the BAR2 inbound base is unknown line follows, the card was not bound; follow that line. Otherwise, read fw_version again after a short while. If it stays unknown, follow fw_version is unknown |
firmware ready after <ms> ms |
info | The firmware reported that it is ready | — |
firmware has no boot-ready flag (needs ABI >= v<n>); falling back to a fixed <ms> ms settle |
info | This is an older firmware that has no ready indicator. The driver waits a fixed time and then treats the firmware as ready | Check the firmware version in Supported environments |
firmware not ready <ms> ms after boot sentinel |
err | The boot signal was seen, but the firmware did not report that it was ready within the time limit | See the messages that follow |
firmware never reported ready (<err>); CSR-derived telemetry will report "unknown" and commands are refused … |
warn | Readiness could not be confirmed at probe time. Either the firmware is still booting, or it failed its self-test during boot and did not report that it was ready | Same as the firmware boot sentinel not seen line above |
firmware boot not confirmed; skipping ABI protocol check (CSR may be mid-publication) |
warn | Firmware readiness could not be confirmed, so the protocol check was skipped. It appears together with one of the two warnings above | — |
firmware v<x.y.z> predates the ABI protocol handshake (needs >= v<x.y.z>); skipping protocol check |
warn | This is an older firmware that does not support the protocol check | Check the firmware version in Supported environments |
ABI protocol v<n> (driver v<n>, min v<n>) |
info | The firmware's protocol version is compatible with the driver | — |
post-reset ABI protocol check FAILED -- device stays bound but its wire ABI is incompatible; unbind it |
err | The firmware that booted again after the reset is not compatible with this driver. The card stays registered, but normal operation cannot be expected | Do not use the card. As the message says, unbind the card from the driver and contact the support team |
ABI protocol mismatch, which stops probe, is covered in Errors that stop probe. The command that unbinds a card from the driver is as follows.
1 | |
Telemetry messages¶
| Message | Level | Meaning | Action |
|---|---|---|---|
firmware version CSR reads 0x<value>; reporting "unknown" |
warn | The firmware version was read, but the value was not valid (0x00000000 or 0xffffffff). fw_version is unknown |
Reset the card and check again |
telemetry stays "unknown": device is not ready (<err>) -- <reason> |
warn | The firmware information region cannot be read yet, so fw_version and serial are unknown. If <reason> is CSR window will not bind, a window to access the information region could not be created; if it is not serviceable -- no boot sentinel, still booting, or the CSR reads back garbage, there is no boot signal, the firmware is still booting, or the value read is not valid |
Check the messages in Firmware readiness and compatibility |
firmware booted after probe gave up; CSR-derived telemetry refreshed |
info | A firmware whose readiness could not be confirmed at probe or reset became ready later, and fw_version and serial were read |
— |
device latched ready but its CSR window has since gone dead; telemetry stays "unknown" |
warn | The firmware information region of a card whose readiness was confirmed can no longer be read | Check health and the error messages from the same time, and reset the card if necessary |
sys_mon: no response for <n> consecutive polls (<ms> ms) - suspending telemetry polling. … |
err | Temperature, power, and utilization polling received no response for consecutive polls (ha_sysmon_fail_limit times, 15 by default), so polling was stopped. This is because continuing to read an unresponsive card can cause some platforms to reset the entire host. The nodes obtained by polling become unknown |
Reset the card. If the reset succeeds, polling restarts |
sys_mon: resuming telemetry polling |
info | Polling that had been stopped has restarted | — |
sys_mon telemetry unavailable: <err> |
warn | The telemetry poller could not be started at probe time. The temperature, power, and utilization nodes are unknown. Computation is not affected |
Check the memory state, then reload the module or hot reset the card |
telemetry sysfs unavailable (<err>) |
warn | The telemetry nodes under /sys/class/ha/haN/ could not be created. /dev/haN works normally |
Report it to the support team along with the kernel version |
hwmon register failed (<err>) |
warn | The card could not be registered as a hwmon device. This card does not appear in /sys/class/hwmon. sysfs telemetry is unaffected |
— |
When the device is FATAL, the driver stops polling without logging anything, and the temperature, power, and utilization nodes become unknown after 5 seconds. health is faulted at this point. For the unknown conditions of each node, see sysfs and Telemetry.
Watchdog and hang messages¶
The driver waits for the response to a firmware command for up to ha_devcmd_ceiling_ms (12 seconds by default). For a long-running command, this ceiling is extended by the time the command needs. Separately, the watchdog checks the waits every ha_wd_poll_ms (500 ms by default) and gives advance notice when a firmware command wait exceeds ha_wd_hang_ms (8 seconds by default) or a DMA transfer wait exceeds ha_wd_hdma_hang_ms (4 seconds by default). How this works is described in Hang detection and the watchdog.
Watchdog¶
| Message | Level | Meaning | Action |
|---|---|---|---|
watchdog: <watch> wait stalled (devcmd=<ms> ms hdma=<ms> ms) - no completion within the threshold |
warn | A wait exceeded the watchdog threshold. <watch> is the side that newly crossed the threshold: devcmd (firmware command response), hdma (DMA transfer), or devcmd+hdma if both did. The values in parentheses are the wait times at that point. It can also appear for work that legitimately takes a long time, so this line alone does not mean a hang. It is logged only once per stall |
Check whether a response wait ceiling expiry message follows, and whether inflight is decreasing |
watchdog: preempting without ha_auto_reset - the device will stay FATAL until … |
warn | Because ha_wd_preempt=1, the watchdog cut off a stalled wait, but ha_auto_reset is off. If the device became FATAL because of the wait that was cut off, it stays that way until it is reset |
Reset the card. To recover automatically, use it together with ha_auto_reset=1. FATAL and recovery |
Warnings about configurations in which the watchdog cannot fire are in Configuration warnings.
Response wait ceiling expiry and the hang dump¶
| Message | Level | Meaning | Action |
|---|---|---|---|
Devcmd no response after <ms> ms (device hang): <verdict> |
err | The firmware did not respond within the response wait ceiling (<ms>). devcmd_timeouts increases by 1, and two hang dump lines follow immediately. If the ceiling expires while waiting for application work, the device usually becomes FATAL (health is faulted). If it expires during a reset, the driver moves on to the next reset step |
See Reading the hang dump below and follow FATAL and recovery |
Devcmd waiting <ms> ms with no timeout requested … |
warn | An application made a request that waits without a ceiling, and the driver has been waiting <ms> ms for its response. It is logged first when the wait exceeds ha_devcmd_ceiling_ms, and then again every 10 seconds. The name and ID of the waiting thread are included |
Check on the application side whether this is work that legitimately takes a long time. Each time this line is printed, the progress is updated in sysfs last_hang. Read last_hang twice, at least 20 seconds apart; if batch_seq and exec_index stay the same, a single command is taking a long time or has stopped. If you read at a shorter interval, you may see the same values even while work is progressing. If you judge that it has stopped, save the diagnostic information, then end that process and reset the card |
hang (<reason>): …, hang dump (<reason>): … |
warn | Diagnostic block for a command that is not responding | See Reading the hang dump below |
Reading the hang dump¶
Two lines appear at the moment the driver determines that there is no response.
1 2 | |
- The first line is a human-readable summary, and the second line holds the raw values. Paste both lines into the support request as they are.
<reason>is one ofceiling timeout(the response wait ceiling expired),watchdog(watchdog verdict), andfw_update timeout(the firmware update time limit expired). Awatchdogdump appears only for a wait without a ceiling, or whenha_wd_preempt=1.fw <command>is the last command sent to the firmware. As the wording says, it may not be the command that stopped.<details>holds items such as the address and size of the command when it is a kernel execution command.opened by pidis the process that opened the card. If a handle was handed over and is used by another process, it can differ from the process that actually submitted the work. This part is attached only to commands whose owner can be determined.stream=0x<tag>is a tag that masks the application's stream with a value that is different on every boot. Use it only to tell whether it is the same stream within one boot.- If
(hdma unreadable: link down?)is appended to the end of the second line, the card's registers could not be read. The PCIe link is down or the card is not responding.
Verdict¶
The verdict is an estimate of where the firmware stopped, based on the progress the firmware reported when the ceiling expired. The token appears in the Devcmd … line and in verdict= of sysfs last_hang, and the sentence appears in the first hang line.
| Token | Sentence in the first hang line | Meaning |
|---|---|---|
not-picked-up |
firmware never picked up the batch |
The firmware did not pick up the command batch the driver sent |
picked-up-no-dispatch |
firmware took the batch but dispatched no command |
The firmware picked up the command batch but did not start any command |
stuck-in-command |
firmware started a command and never finished it |
The firmware started the command at the exec_index position in last_hang and did not finish it |
completed-irq-lost |
every command completed; only the interrupt was lost, so a reset is NOT needed |
Every command finished, and only the completion interrupt was not delivered. The card itself has not stopped |
rebooted |
the device rebooted underneath us |
The card rebooted while the driver was waiting |
unsupported |
this firmware does not publish enough progress to judge |
The firmware does not report enough progress information, or the values read cannot be trusted, so no verdict can be reached. The progress fields of last_hang are - |
none |
no progress data |
No progress information was recorded |
For the watchdog reason, the first hang line ends with progress not sampled here, so no verdict instead of a verdict.
FATAL is not cleared even when the verdict is completed-irq-lost
The verdict is for diagnosis. Once the response wait ceiling passes, the driver handles the case the same way regardless of the verdict. If the device has become FATAL (health is faulted), you must reset it before you can use it again, even if the verdict is completed-irq-lost. If this verdict repeats, report it to the support team along with the dump.
Firmware errors¶
1 2 3 | |
The firmware reported that a command failed (level err). After the first line, 12 lines of raw values from the firmware response area follow. devcmd_fw_errors increases by 1, and the same values are recorded in rc, cmd_idx, and opcode of sysfs last_hang. A rate limit shared by all cards applies, so if the message repeats many times only some occurrences are printed, but the counter counts every one of them.
The outcome depends on rc. The table below applies when the error came from application work.
rc |
Meaning | Result |
|---|---|---|
0x2 |
Unknown command | Only that command fails |
0x3 |
Invalid argument | Only that command fails |
0x4 |
Out of memory on the firmware side | Only that command fails |
0x5 |
Buffer overflow | Only that command fails |
0x8 |
Core error limited to that command (command decoding error, division by 0, and so on) | Only that command fails |
0xa~0xd |
The result of a firmware update. See Firmware update | Only that command fails |
0x1 |
The device is not ready | The device becomes FATAL |
0x7 |
The firmware ended only that command with a timeout. The device can still answer commands | The device becomes FATAL. This differs from a wait that expires with no response at all, and a soft reset can still reach the firmware. See Response wait ceiling |
0x9 |
Fatal core error (multi-bit ECC error, address range exceeded, and so on) | The device becomes FATAL |
| Any other value | Unknown result | The device becomes FATAL |
"Only that command fails" means that the card can keep being used, so look for the cause in the application log. If the device became FATAL, follow FATAL and recovery.
Data transfer (DMA)¶
Copies between the host and a card, and between cards, are handled by the DMA engines of the card. There are several engines per direction. In the messages, <direction> is read for an engine that reads over PCIe and writes into card memory (copies from the host to the card, file loads, and copies pulled from another card) and write for an engine that reads card memory and writes over PCIe (copies from the card to the host and copies pushed to another card), and <ch> is the channel number of the engine.
| Message | Level | Meaning | Action |
|---|---|---|---|
ll: timeout (n=<n> channel=<ch> <direction>) |
err | A DMA transfer failed because it did not finish within 5 seconds. hdma_timeouts increases by 1. If the driver can stop the engine, only the work that the transfer belongs to fails and the device does not become FATAL. If it cannot stop the engine, a HDMA engine at … line follows immediately and the device becomes FATAL. If the engine had already stopped, the ll: engine stopped … line below is logged instead of this one |
Run the failed work again. If it repeats on the same card, save the diagnostic information and contact the support team. Also check the PCIe error messages from the same time and the link speed and width (current_link_speed, current_link_width) |
ll: engine stopped but its completion <mode> never arrived (status=<n> channel=<ch> <direction>) |
err | The driver did not receive the completion signal of a DMA transfer within 5 seconds, but the engine had already stopped. The completion signal that the engine sent did not reach the driver. hdma_timeouts increases by 1, and only the work that the transfer belongs to fails. <mode> is MSI-X when completion comes by interrupt and write when the driver polls; this write is unrelated to the <direction> at the end of the line. If <mode> is MSI-X and every copy logs this line, the driver has usually mistaken a VM guest for bare metal (an x86 guest whose CPU hides the hypervisor flag, or an arm64 guest set up under nested virtualization to run its own hypervisor). The same happens when ha_hdma_completion=irq is given in a VM |
If <mode> is MSI-X and every copy logs this line, reload the driver inside the VM guest with ha_hdma_completion=poll. If the line appeared only once, treat it like ll: timeout. If <mode> is write, or it repeats after the change, save the diagnostic information and contact the support team |
ll: n=<n> channel=<ch> <direction> ret=<n> status=<n> int_status=0x<value> error=<n> |
err | The DMA engine reported an error | Same as above |
ll: ABORTED error=<n> |
err | The DMA transfer was aborted. Only the work that the transfer belongs to fails | Same as above |
ll: no response (channel=<ch> <direction>), ring write: no response (channel=<ch> <direction>) |
err | The card does not respond to the status register of the DMA engine (every bit of the value read is 1). The link may be down, or the card may have stopped or been removed. The transfer fails and the device becomes FATAL | Follow FATAL and recovery. Also check the PCIe error messages from the same time |
HDMA engine at 0x<offset> <state>; quarantining its pages until the engines stop (reset or remove) |
crit | <state> is not responding (the engine does not respond) or still running after force stop (the engine is still running even after being force-stopped). The engine may still be able to write to host memory, so the driver does not release the host pages that the transfer was using but quarantines them, and blocks new DMA transfers until all engines stop. The device becomes FATAL |
Reset the card. health is faulted |
HDMA engines stopped, accepting transfers again |
info | The engines that could not be stopped have all been confirmed stopped, and DMA transfers are accepted again | — |
HDMA: freed <n> quarantined batch(es) of the <direction> engine on channel <ch> |
info | The quarantined host pages were released after the engine stopped | — |
HDMA: device not responding at reset, engines not stopped |
warn | The card did not respond during the reset, so it could not be confirmed that the engines stopped. The driver checks again after the reset finishes | After the reset finishes, check whether a device reset failed line was logged. Reset progress and result |
HDMA: no free <direction> engine after 10 s |
warn | No <direction> engine became free for the copy for 10 seconds. It repeats every 10 seconds while waiting |
Check the hang and DMA messages from the same time, and inflight |
HDMA: <direction> engine <ch>: pipeline stage not finished after <n> s, still waiting |
warn | The preparation stage of a copy (pinning and mapping host pages) or its cleanup stage (unmapping) has not finished for <n> seconds, and the driver keeps waiting. It repeats every 10 seconds. It appears mainly when securing the pages of the host buffer takes a long time |
Check the memory state of the host |
HDMA: bounce-buffered DMA, using <n>-page batches |
info | Logged once when the card is registered. On this host, the card's DMA goes through a bounce buffer (swiotlb) and the size that can be mapped at a time is limited, so copies are split into smaller units. Copy throughput can be lower | Check the IOMMU and DMA settings to see whether this is the intended configuration |
HDMA: batches of <n>-<n> pages, <n> slots, <n> hdma_pre_wk, <n> channel(s) x 2 engines of 2 x <n> LL entries |
info | A summary of the DMA processing configuration, logged once when the card is registered | — |
ha: stream 0x<value> waits for a prefetch copy |
warn | The wait for the input copy that the driver started ahead of time (ha_abi_prefetch) to finish has lasted more than 10 seconds. It repeats every 10 seconds while waiting. 0x<value> is a value that identifies the application's stream |
Check the hang and DMA messages from the same time, and health |
Reset messages¶
A card is reset either by an application through the SDK's reset function, or by the driver's automatic reset (ha_auto_reset=1). A reset first reinitializes the firmware, and if that step fails, it issues an SBR (Secondary Bus Reset) on the parent PCIe port. An SBR is issued only in a configuration where the parent is a root port or the parent port has masked Surprise Down errors. On success, firmware ready after <ms> ms and AEXP: bootloader default programmed after device reset are usually logged. ha-hotreset removes the card and has it probed again, so removal and probe messages are logged. How a reset behaves is described in FATAL and recovery.
Automatic reset¶
| Message | Level | Meaning | Action |
|---|---|---|---|
auto-TDR: the device went FATAL; resetting it |
warn | The device became FATAL, and the driver starts an automatic reset | Check the result messages that follow |
auto-TDR: reset failed (<err>); auto-TDR will not run again on this card until a reset succeeds or the card reports ready again |
err | The automatic reset failed. Automatic reset for this card is suspended until one reset succeeds or the card becomes ready again, and health is faulted |
Do what the device reset failed line just before it says |
auto-TDR: this instance was removed during the reset (<err>); re-probe decides whether the card comes back |
warn | The card was removed from the driver during the reset. For example, the SBR took the link down and the operating system removed the card. If the card is detected again, it is probed anew | Confirm recovery with a Probe successful message for the same <BDF> |
Messages about skipping the automatic reset because of a firmware update are in Firmware update.
Reset progress and result¶
| Message | Level | Meaning | Action |
|---|---|---|---|
issuing SBR via parent <bridge> |
warn | Recovery by reinitializing the firmware did not work, so an SBR is issued on the parent port. The card reboots | — |
parent <bridge> leaves Surprise Down unmasked; refusing SBR (would reboot the host) |
warn | The parent port does not mask Surprise Down errors, so the SBR was refused. In this configuration, taking the link down without masking can make the platform reset the entire host | Run sudo ha-hotreset --bdf <BDF> on the host. ha-hotreset masks the path and resets the card. ha-hotreset |
parent <bridge> has no AER capability, so the Surprise Down mask cannot be checked; refusing SBR |
warn | The parent port has no AER capability, so it cannot be checked whether the error is masked, and the SBR was refused | ha-hotreset cannot reset this card either. Power-cycle the host to recover. sudo ha-hotreset --list lets you check in advance, for each card, whether the configuration allows an SBR |
parent <bridge>: cannot tell whether Surprise Down is masked (no PCIe capability, or the mask is unreadable); refusing SBR |
warn | The masking state could not be read, so the SBR was refused | Same as above |
No parent bridge, SBR not supported |
err | The card has no parent bridge, so an SBR cannot be issued | Same as above |
device did not answer after SBR (parent link <state>) |
err | The card did not respond within 10 seconds after the SBR. <state> is the link state of the parent port (up, down, unknown) |
Run sudo ha-hotreset --bdf <BDF> on the host, and if that does not recover it, power-cycle the host |
removed during SBR (<cause>); this instance is gone - re-probe decides whether the card comes back |
warn | The operating system removed the card during the SBR. <cause> is link lost (the link went down) or driver unbind. If the card is detected again, it is probed anew |
Confirm recovery with the Probe successful message. Applications must reopen the node |
removed after SBR while the reset was still running (<err>); this instance is gone - … |
warn | The card was removed while the reset was being completed after the SBR | Same as above |
device came back but firmware did not report ready (<err>); reset reported as failed |
warn | The card came back, but the firmware did not report that it was ready within ha_ready_timeout_ms (3 seconds by default), so the reset was reported as failed. Application requests are rejected until the firmware is ready |
Check fw_version after a short while. If it stays unknown, a hot reset or a host power cycle is required |
AEXP: bootloader default programmed after <event> |
info | After a probe or a reset, the card's address expander (AEXP) was set again to the bootloader default. <event> is one of probe, device reset, and host reset |
— |
AEXP: bootloader default not programmed after <event> (<err>); state unknown |
warn | The setting above could not be written. It is written again at the next reset | Check the other error messages from the same time |
ha: quiesce drain stalled <s>s in <process>(pid <pid>): inflight=<n> fileio=<n> leases=<n> |
warn | A reset or removal has been waiting more than 10 seconds for in-progress work to finish. It repeats every 10 seconds. <process> is the one that is waiting. It shows which of inflight (work submitted to the card), fileio (read and write calls), and leases (other firmware commands) has not finished |
If it continues, save the diagnostic information and contact the support team |
device reset failed (<err>, firmware re-init <err>): <advice> |
err | The reset did not recover the card. This is the last line the reset procedure leaves (an automatic reset adds one more, auto-TDR: reset failed (<err>); ...), and health stays faulted. Depending on the case, <advice> says whether to reset again, run ha-hotreset --bdf <BDF> on the host, or power-cycle the host, and what to do if that does not work either |
Follow <advice>. The lines before it show which stage failed |
If the firmware protocol does not match after a reset, post-reset ABI protocol check FAILED is logged. See Firmware readiness and compatibility.
Host-initiated resets and PCIe error messages¶
A card is also reset from outside the driver: when the kernel resets the card (sysfs reset, a bus reset for another device under the same bridge, and so on) and when it recovers from a PCIe error (AER, DPC). The driver receives a notification, stops and cleans up its work, and all open handles to that card become invalid. Applications must reopen the node.
| Message | Level | Meaning | Action |
|---|---|---|---|
reset_prepare: quiescing for a host-initiated reset |
info | The kernel is about to reset the card, so the driver stops and cleans up its work | If the reset was unexpected, find out what triggered it. Do not reset this card through sysfs reset |
reset_done: reopening the gate (boot <result>) |
info | The reset has finished, and the driver reopens work submission. <result> is ok or unconfirmed |
If it is unconfirmed, see the line below |
reset_done: firmware boot not confirmed (<err>) |
warn | There was no firmware boot signal within 10 seconds after the reset | Hot reset the card or power-cycle the host |
firmware did not report ready after host reset (<err>); CSR caches dropped, AEXP state unknown |
warn | The firmware did not report that it was ready within ha_ready_timeout_ms (3 seconds by default) after the reset. fw_version and serial are unknown, and application requests are rejected until the firmware is ready |
Check fw_version after a short while. If it stays unknown, a hot reset or a host power cycle is required |
PCIe error detected (state=<n>): <kind> |
err | PCIe error recovery has started. The driver stops its work and waits for the kernel's link reset. <kind> is recoverable or permanent failure, and state is 1 (normal), 2 (I/O blocked), or 3 (permanent failure) |
Check the kind of error in the kernel AER message immediately before it |
slot_reset: re-enable failed |
err | The card could not be re-enabled after the link reset, so error recovery ended in failure | Power-cycle the host |
slot_reset: firmware did not come back (<err>) |
err | There was no firmware boot signal within 10 seconds after the link reset, so error recovery ended in failure | Hot reset the card or power-cycle the host |
PCIe error_resume: recovery complete |
info | PCIe error recovery is complete | Check that the application has reopened the node |
PCIe recovery never reached slot_reset within <ms> ms; reopening the submit gate. The device was not re-initialised … |
err | Error recovery did not reach the link reset step within 30 seconds, so the driver reopened work submission. The card was not reinitialized | If the card does not work normally, reset the card |
AER reporting enabled: the kernel's fatal remedy is a parent-bridge SBR |
warn | The module was loaded with ha_aer_enable=1. For fatal PCIe errors, the kernel recovers with a parent-bridge SBR, and this reset does not go through the driver's Surprise Down check |
Check that this is the intended setting. Parameter reference |
AER reporting unavailable; error handlers stay idle |
info | ha_aer_enable=1 is set, but the kernel could not enable AER reporting for this card (for example, in a configuration where platform firmware manages AER). It appears only on some kernel versions |
— |
Even when ha_aer_enable is off, error recovery messages can appear because of DPC, a fatal error on another device, and so on. On kernel 6.4 or later and RHEL 9.4 or later, they can also appear because of the card's own errors.
Firmware update messages¶
These are the messages logged while and after ha-fw-update performs an update. The procedure and safety rules are in Firmware update.
| Message | Level | Meaning | Action |
|---|---|---|---|
fw_update: pa=0x<address> size=<bytes> scope=<n> flags=0x<value> crc32=0x<value> budget=<ms> ms |
info | The firmware update command is sent. budget is the time, calculated from the image size, that is added to the response wait ceiling |
— |
ha: fw_update: discarded <n> stale staged command(s) left by a failed batch |
warn | Commands left behind by an earlier failed job were discarded before the update | — |
fw_update: nothing sent (CSR window unavailable) |
warn | The update command could not be sent. The flash is unchanged | Run the update again |
fw_update: nothing sent (staging cleared by an abort) |
warn | The pending command was cleared by a reset or similar, so it could not be sent. The flash is unchanged | Run the update again |
fw_update: device vanished during the write (all-ones) |
err | The card stopped responding during the write. The flash state is unknown | Do not power off or reboot the host; follow Firmware update |
fw_update: no response after <ms> ms: <verdict> |
err | The firmware did not report the update result within the time limit. A fw_update timeout hang dump follows. The flash state is unknown |
Same as above |
fw_update: CSR window unavailable after completion |
err | The firmware reported completion, but the driver could not read the result, so it treated the flash state as unknown. A fw_update: no answer from firmware line follows |
Do not reboot the host or reset the card; contact the support team |
fw_update: no answer from firmware (<err>); resets stay refused until it answers or the update is retried |
err | The firmware's response could not be confirmed after the command was sent. The firmware may still be writing the flash, so resets and application work on this card are rejected until a response is confirmed. The step in which ha-fw-update uploads the image to card memory is also rejected, so running it again in this state ends with upload failed: Device or resource busy (exit code 2), and the flash is not changed |
Do not reboot the host or reset the card. When the firmware responds and fw_update: late completion from firmware; resets re-enabled is logged, run ha-fw-update again with the same image. See Firmware update |
fw_update: firmware answered; resets re-enabled |
warn | The firmware answered an update command that was sent again while the response was unconfirmed, so the rejection was lifted. In this state ha-fw-update has its upload step rejected, so the rejection is usually lifted by the late completion line below |
— |
fw_update: late completion from firmware; resets re-enabled |
warn | The rejection was lifted by a completion response that arrived late. This line does not carry the result of the write | Run the update again with the same image and finish it through the reset and verification |
reset refused: last fw_update unanswered (flash may still be written); retry ha-fw-update once the firmware answers, else contact HyperAccel support |
warn | The reset request was rejected because there is an update whose response was not confirmed | After the firmware responds and fw_update: late completion from firmware; resets re-enabled is logged, run ha-fw-update again. If the firmware never responds, contact the support team. Reloading the module, ha-hotreset, and power cycling risk booting from a partially written flash, so use them only after consulting the support team |
auto-TDR: skipped, a fw_update is in flight or unanswered (flash may be mid-write) |
warn | The automatic reset was skipped because an update is in progress or there is an update whose response was not confirmed. The device remains FATAL | If an update is in progress, wait until it finishes. If ha-fw-update ended with exit code 4, ended without reporting an exit code, or fw_update: no answer from firmware was logged, do not reset the card, even after … resets re-enabled is logged. That line does not carry the result of the write, and while the device is FATAL, running the update again is rejected with device allocation failed: State not recoverable, so contact the support team. In other cases, if health is still faulted after the update finishes, reset the card |
auto-TDR: skipped, the fw_update it queued behind ended unanswered |
warn | The automatic reset was skipped for the same reason | Same as above |
reset_prepare: last fw_update unanswered - resetting now may leave the boot flash half-written |
err | The kernel is resetting the card even though there is an update whose response was not confirmed. The driver cannot block a reset that the kernel issues | If the card does not boot after the reset, contact the support team |
The result code that the firmware returns for the update command (rc=0xa~0xd) is also logged in a Firmware error line. The response to a result code is decided from the exit code of ha-fw-update. See ha-fw-update.
Configuration warning messages¶
| Message | Level | Meaning | Action |
|---|---|---|---|
watchdog: <watch> arm cannot fire with this configuration (hang=<ms> ms hdma_hang=<ms> ms poll=<ms> ms devcmd_ceiling=<ms> ms hdma_timeout=<ms> ms); lower the threshold or the poll interval |
warn | The sum of the watchdog threshold and the polling interval is equal to or greater than the ceiling for that wait (ha_devcmd_ceiling_ms for devcmd, 5000 ms for hdma), so that watch cannot report a wait stall. The driver does not change the values; it only reports this. It is printed only once for a given set of values, and printed again if the values change |
Lower ha_wd_hang_ms, ha_wd_hdma_hang_ms, and ha_wd_poll_ms, or raise ha_devcmd_ceiling_ms. Relationships between parameters |
ha: invalid ha_hdma_cpu_latency_us=<value> (-1 or >= 0), using 0 |
warn | ha_hdma_cpu_latency_us was given a value below -1, so the module was loaded with 0 |
Correct the value in the option file. Parameter reference |
watchdog: preempting without ha_auto_reset - … |
warn | See Watchdog | — |
AER reporting enabled: … |
warn | See Host-initiated resets and PCIe errors | — |
ha: `<value>' invalid for parameter `<name>' |
err | A value of the wrong type, or a value the parameter does not accept, was written in the options file, on the kernel command line, or on the modprobe command line, so module loading failed |
Correct the value, then reload the module. If the value was written on the kernel command line, correct the command line and reboot. How to change |
ha: unknown parameter '<name>' ignored |
warn | The module was loaded, and the parameter with the wrong name was ignored | Correct the name. Parameter reference |
For the other parameters that have a defined range (ha_devcmd_ceiling_ms, ha_wd_poll_ms, ha_tw_pin_wait_ms, ha_sysmon_poll_ms), a value outside the range is clamped to the nearest end of the range, and no separate message is logged. The ranges are in Parameter reference.
Device-to-device path messages¶
These are the messages logged when there are several cards. The meaning of the paths is explained in Multiple devices, and the sysfs nodes that show the same information (peer_map, peer_route, peer_sram) are in sysfs.
| Message | Level | Meaning | Action |
|---|---|---|---|
P2P peer[<n>]: <path> |
info | The verdict for the PCIe path from this card to /dev/ha<n>. direct is a path that stays under a common upstream bridge without going through the host; host-mem is a path that goes through the host (root complex); unknown means that the configuration of a bridge on the path could not be read. It is the same value as peer_route |
— |
dma_map_resource failed for peer[<n>] |
warn | The BAR of /dev/ha<n> could not be mapped into this card's DMA address space. Device-to-device copies between this card and /dev/ha<n> cannot be performed (peer_map is unmapped) |
Check the IOMMU configuration |
ACS state unreadable; P2P route stays unknown |
warn | The ACS settings of this bridge could not be read, so the path could not be determined. The prefix of this message is the bridge, not the card | — |
outbound: <BDF> <path> base 0x<address> |
info | The path and address by which the device kernel running on this card accesses the SRAM of another LPU (<BDF>). <path> is bus or iova. It is the same information as peer_sram |
— |
outbound: <BDF> remap:<k> base 0x<address> |
info | Same as above; a path created by the driver reconfiguring outbound window <k> |
— |
outbound: region <k> -> 0x<address> (<BDF>), outbound: adopted region <k> (<BDF>) |
info | Outbound window <k> was set up for the path above, or the window left by a previously loaded driver is reused |
— |
outbound: <BDF> none:<reason> |
info | The driver does not tell the device kernel running on this card the SRAM address of that LPU. <reason> is the same as the none: reason of peer_sram |
See peer_sram in sysfs |
outbound ATU[<k>]: enable timeout, outbound ATU[<k>]: readback … does not match |
warn | The outbound window could not be set up. It can appear while the card is being reset or when it does not respond. The path to that LPU remains unknown in peer_sram |
Check peer_sram and health. If the path stays unknown, reset the card |
outbound: no memory, kernels on this card get no windows |
warn | Memory ran short, so the outbound windows for this card could not be created | Check the memory state, then hot reset the card or reload the module |
ha: outbound: more than 32 LPUs, ignoring <BDF> |
warn | More than 32 LPUs are installed, so that card was left out of the path calculation | — |
TW pin: <n>/<max> slots pinned for <ms>ms, giving up (da=0x<address>) |
warn | The translation window needed for a device-to-device copy could not be obtained for ha_tw_pin_wait_ms (12 seconds by default), so the copy failed. The card in the prefix is the one that is short of windows, that is, the peer card of the copy |
Reduce the number of simultaneous copies directed at one card |
Device memory access and application request messages¶
| Message | Level | Meaning | Action |
|---|---|---|---|
ATU[<k>]: enable timeout |
warn | Setting up the window the host uses to access device memory timed out. It can appear when the card does not respond | Check health and the error messages from the same time |
TW: no free pool window (<n> pinned, <n> held by kernel access, <n> fixed, <n> in pool); <operation> failed |
err | There is no free window in the translation window pool, so a new access could not be handled. The parentheses show how the windows of the pool are being used: pinned is windows pinned by device-to-device copies, exports, and so on; held by kernel access is windows that the driver is holding for its own accesses; fixed is windows fixed for SRAM access; and in pool is the total number of windows in the pool. <operation> is the operation that failed (bind, pin, hold, or SRAM reserve). An application that was using mapped device memory can receive an error (SIGBUS) |
Reduce the device-to-device copies and exports in progress at the same time |
TW: write to da 0x<address> did not reach the device |
err | After the host wrote to device memory, the value read back did not match the value written. That request ends with an I/O error. If the same message repeats within a short time, only some are logged. It can appear when the card is not responding or there is a problem with the link | Check health and the PCIe error messages from the same time |
When an application sends an invalid request, the driver may reject it and log the reason. Examples are an alignment error in a mapping size or offset, an unallocated address, or freeing memory that the application does not own. These messages are unrelated to the card state. If they repeat, pass them to the application developer together with the application log.
dma-buf export messages¶
Messages logged when an application requests a dma-buf export so that a peer device such as an RDMA NIC can access the DRAM of the card directly. What an export is and how to see the reserved windows are in sysfs nodes. <device> in the messages is the name (PCI address) of the peer device.
| Message | Level | Meaning | Action |
|---|---|---|---|
ha: dma-buf attach by <device> refused: device gone or reset |
info | The peer device's attempt to attach to the export was refused because the card was reset or removed after the export. A new attachment cannot be made to an export that a reset has passed over | The application must export again and register the new export with the peer device |
ha: dma-buf attach by <device> refused: importer does not take peer-to-peer addresses |
info | Refused because the peer device and its driver stated that they do not accept addresses for direct access between PCIe devices (peer-to-peer). This is a cause of memory registration failing | Check whether that device and its driver support peer-to-peer |
ha: dma-buf map #<n> by <device> refused: device gone or reset |
info | For the same reason as above, the address request of an already attached peer device was refused | Same as above |
ha: dma-buf map #<n> by <device>: dma_map_resource failed for <address> |
warn | The BAR address of the card could not be mapped as a DMA address of the peer device | Check the IOMMU configuration |
TW export: region <n> readback does not match target 0x<address> |
warn | The export window could not be set up, so that export failed. It can appear when the card is being reset or is not responding | Check health and the error messages from the same time |
TW export: region <n> still enabled after <when> |
warn | The driver turned off an export window (<when> is boot or reset), but the window was still read as enabled |
If it repeats, save the diagnostic information and contact the support team |
TW teardown with <n> dma-buf export window(s) reserved; a later bind may reuse their regions under a registered MR |
warn | When the card was removed, the reservations of <n> export windows remained. If the peer device has not released its memory registration (MR), those windows can be used for other DRAM ranges after this card is recognized again, and the peer device can access ranges that were not intended |
Before removing the card, hot-resetting it, or unloading the module, end the applications that use dma-buf and release the memory registrations |
Device removal messages¶
These are the messages logged when a card is removed from the driver (module unload, unbind, hot reset, hot unplug).
| Message | Level | Meaning | Action |
|---|---|---|---|
Removing device |
info | Card removal has started | — |
remove: <n> fd op(s) still inside the gate |
warn | Waiting for <n> calls in progress on open handles to finish. It repeats every 0.2 seconds while waiting |
If it continues, check the processes that are using that card |
Mmap manager cleanup, Device cleanup complete |
info | Cleanup steps | — |
ha: abi prefetch issued=<n> consumed=<n> |
info | The cumulative count of input copies started ahead of time on this card, with the same values as issued and consumed in sysfs abi_prefetch_stats. If there were copies started ahead of time, one line is logged at removal |
— |
Remove complete |
info | Card removal is complete | — |
If in-progress work does not finish for a long time, ha: quiesce drain stalled … is logged as well. See Reset progress and result.
See also¶
- Troubleshooting — from symptom to cause and resolution
- Hang detection and the watchdog
- FATAL and recovery
- Firmware update
- Telemetry
- Multiple devices
- Parameter reference
- sysfs