Skip to content

Hang detection and the watchdog

This page explains how ha.ko notices that a device is not responding to a command and what it records. The driver judges a non-response against the response wait ceiling, separately observes waits that grow long with the watchdog, and records the results in the RAS counters, the sysfs last_hang, and the hang dump in the kernel log. This page covers when each feature operates, how to read the records, and what to collect when a hang is suspected. Parameters are defined in Parameter reference, how to configure them is in Configuring and using parameters, and the procedure for reviving a stopped device is in FATAL and recovery.


Waits and ceilings

The driver waits for the device in two places. One is the response to a command sent to the firmware (devcmd in kernel log messages and sysfs node names), and the other is the completion of a transfer that moves data between the host and a device, or between two devices. The driver bundles several device commands and sends them at once, and this bundle is called a batch.

Each of the two waits has its own ceiling, and the watchdog watches both waits from the outside with a threshold shorter than the ceiling.

Wait for a device command's response Wait for a DMA transfer to complete
Ceiling ha_devcmd_ceiling_ms (default 12000 ms) 5 seconds per transfer. It cannot be changed with a parameter
When the ceiling is exceeded The command is judged as not responding. If an application job was running, the device becomes FATAL The job that includes that transfer fails. The device does not become FATAL
Records left devcmd_timeouts, last_hang, the hang dump in the kernel log hdma_timeouts
Watchdog threshold ha_wd_hang_ms (default 8000 ms) ha_wd_hdma_hang_ms (default 4000 ms)

The watchdog only reports waits that crossed the threshold in the kernel log, and with the default settings it does not intervene in waits. What ends a wait and decides a failure is whether the response wait ceiling was exceeded.


Response wait ceiling

After sending a command to the firmware, the driver waits for the response. The response wait ceiling is the maximum time allowed for waiting for a response, and it applies not only to the commands of application jobs but to all commands the driver sends, such as the firmware re-initialization during a reset. If a command does not finish for a long time, the firmware reports a timeout on its own, and the wait ceiling is that time plus some margin. So if the firmware is alive, a failure result that contains the cause arrives first, and the ceiling expires only when even the firmware does not respond.

The host watchdog (ha_wd_*) and the time limit the firmware sets for each command are separate. The firmware allows a longer limit for commands that take longer, and it waits at least 8 seconds when that limit would be shorter. The driver's default ceiling is 12 seconds, chosen so that the firmware reports the timeout first.

The two outcomes are recovered differently.

  • If the firmware returns rc=0x7, the device can still answer commands, and only that command ended in a timeout. When an application job is running, the driver marks the device FATAL, so an SDK device reset can still reach the firmware.
  • If the ceiling expires, the firmware did not send a response. If command processing has stopped, re-initialization does not run either, and the reset goes on to an SBR.

If you set ha_devcmd_ceiling_ms shorter than the firmware's time limit, the driver marks the device FATAL for no response before rc=0x7 arrives. This is the same constraint as the warning in Changing the ceiling.

When the ceiling expires

The driver judges the command as not responding and leaves the following records.

  • The kernel log gets the error Devcmd no response after <N> ms (device hang): <verdict> and two lines of the hang dump. <N> is the ceiling that was applied (in ms). How to read them is described in Hang dump.
  • devcmd_timeouts increases by 1, and the batch progress information and the verdict are recorded in last_hang.

If an application job was running, the driver switches the device to FATAL.

  • The kernel or job that was running fails, and other waiting kernels or whole jobs are cancelled as well. The application receives the error through the error code returned by the runtime.
  • Until a reset, the device accepts no new jobs. This is true even if you restart the process.
  • health becomes faulted. Telemetry that is read periodically, such as temperature, power, and utilization, soon changes to unknown and stays that way until a reset.
  • If ha_auto_reset=1, the driver attempts a device reset by itself. With the default 0, the device stays FATAL until it is reset. See FATAL and recovery.

The same records can also be left during a reset

If the firmware does not respond to the re-initialization command during a reset, the ceiling expires, a ceiling timeout hang dump is left, and devcmd_timeouts increases. The driver then moves on to the next recovery step. A hang dump you see right after a reset may be a record of the reset, not of the original hang.

Long-running commands

Large memory fills and firmware updates can legitimately take longer than the default ceiling. When the driver sends such a command, it extends the ceiling by the command's expected duration plus a margin, and it extends the watchdog thresholds by the same amount. So these operations are not judged to be hangs when they take a long time normally. In exchange, if the device really stops during such an operation, the verdict is also delayed by the extended amount.

Changing the ceiling

ha_devcmd_ceiling_ms can be changed at run time, and the new value applies from the waits that start after the change. A value smaller than 100 is treated as 100 ms, and 0 is also 100 ms, not an infinite wait. In this case, /sys/module/ha/parameters/ha_devcmd_ceiling_ms shows the value you wrote unchanged.

Do not lower the ceiling below the default

The ceiling must be longer than the firmware's own timeout judgment. If it becomes shorter, the driver judges the command as not responding before the firmware reports the cause, so the device can become FATAL even for a command that would have completed normally. Lowering the ceiling can also keep the watchdog's arms from firing (Thresholds and the polling interval).

Raising the ceiling delays the verdict by the same amount when the device has really stopped.


Watchdog

One watchdog runs per device, and every ha_wd_poll_ms (default 500 ms) it checks how long the waits in progress have lasted. The waiting side reaches a verdict only when its own ceiling runs out, but the watchdog looks from outside the wait, so it reports a wait that is getting long before the ceiling. The watchdog is also the first to report a wait that has no ceiling.

Records left by the watchdog

When a wait crosses the threshold, the watchdog leaves one warning line in the kernel log.

1
ha 0000:3b:00.0: watchdog: devcmd wait stalled (devcmd=8312 ms hdma=0 ms) - no completion within the threshold
  • The word after watchdog: is the arm that crossed its threshold this time: devcmd, hdma, or devcmd+hdma if both crossed together.
  • The parentheses show how long the two waits had lasted at that moment. 0 means there is no wait of that kind.
  • Only one warning is issued per wait. If another wait crosses the threshold after that wait has ended, it is reported again. The two arms are counted separately.
  • The watchdog makes no judgment while the device is being reset or is FATAL.
  • If the wait reported by the devcmd arm is a wait without a ceiling, or if ha_wd_preempt=1, the warning is followed by a hang dump with the watchdog tag. For a wait that has a ceiling, a dump with a verdict is left when the ceiling expires, so the watchdog does not leave a dump. The hdma arm does not leave a dump.

The time added by Long-running commands is added to the threshold of the devcmd arm. The threshold of the hdma arm applies to the wait for a single transfer handed to the DMA engine, not to the entire copy operation.

The watchdog only warns and gives no verdict

The watchdog looks only at elapsed time. It cannot distinguish a wait that legitimately takes long from a device that is not responding, so the warning text says not "hang" but "no completion within the threshold". Read a warning as an observation, and base the verdict on the records that follow. With the default (ha_wd_preempt=0), read it as follows.

  • If a devcmd warning is not followed by the ceiling-expiry records (Devcmd no response and the ceiling timeout hang dump), that wait ended before the ceiling.
  • If hdma_timeouts did not increase after an hdma warning, that transfer finished within 5 seconds.
  • A wait without a ceiling produces a warning when it crosses the threshold even if it is normal operation. See Waits without a ceiling.

Thresholds and the polling interval

The watchdog checks the state of the waits once per polling interval, so it can take up to one more interval to notice a wait that crossed the threshold. If the waiting side's ceiling ends in the meantime, the watchdog never sees that wait. So for each arm to fire, the following must hold.

Arm Condition Default
devcmd ha_wd_hang_ms + ha_wd_poll_ms < ha_devcmd_ceiling_ms 8000 + 500 < 12000
hdma ha_wd_hdma_hang_ms + ha_wd_poll_ms < 5000 4000 + 500 < 5000

The time added by long-running commands is added equally to the threshold and the ceiling, so this relationship holds as is.

If an arm condition is broken, the driver does not change the values and reports it in the kernel log. For example, if you raise ha_wd_poll_ms to 1000, the hdma arm cannot fire.

1
ha 0000:3b:00.0: watchdog: hdma arm cannot fire with this configuration (hang=8000 ms hdma_hang=4000 ms poll=1000 ms devcmd_ceiling=12000 ms hdma_timeout=5000 ms); lower the threshold or the poll interval

This warning appears only once for the same configuration. If you change one of the two thresholds, the polling interval, or the response wait ceiling and the condition is still broken, it appears again. Lower the threshold or the polling interval to satisfy the condition. Watchdog parameters can be changed at run time and apply from the next poll.

The meaning of 0 differs from parameter to parameter.

Parameter If you use 0
ha_wd_hang_ms The devcmd arm is turned off
ha_wd_hdma_hang_ms The hdma arm is turned off
ha_wd_poll_ms It is not turned off. A value smaller than 50 is treated as 50 ms

The only way to turn off an arm is to set its threshold, ha_wd_hang_ms or ha_wd_hdma_hang_ms, to 0. The arms keep running every interval, so if you set a threshold again, the arm comes back from the next poll. If both thresholds are 0, the watchdog judges nothing.

Waits without a ceiling

The response wait ceiling does not apply to a device command that the application asked to wait for without a ceiling. Even if the device does not respond, the driver does not end the wait by itself; such a wait ends when a response arrives, when the waiting process receives a termination signal, or when the device is reset or removed. If ha_wd_preempt=1, the watchdog cuts the wait off at the threshold (Cutting off a stalled wait).

The records left by such a wait without a ceiling are as follows.

  • When the watchdog threshold (default 8 seconds) is exceeded, a watchdog warning and a hang dump with the watchdog tag are left. This dump does not read progress information, so it ends with progress not sampled here, so no verdict instead of a verdict.
  • When the wait exceeds the same time as the response wait ceiling (default 12 seconds), a warning that starts with Devcmd waiting <N> ms with no timeout requested appears, and it repeats every 10 seconds after that. The name and ID of the waiting thread (name[ID]) and the verdict value are appended at the end of the line. Each time the warning is issued, the progress information is read anew and recorded in last_hang.

This wait does not increase the devcmd_timeouts count and does not make the device FATAL. It does not show up in health alone, so check the kernel log.

Telling a long operation from a stopped device

Read last_hang twice at least 20 seconds apart. If batch_seq or exec_index has changed, the device is making progress. If they have not changed, a single command is taking a long time or has stopped. Judge by comparing with how long that command normally takes, and if you conclude that it has stopped, save the diagnostic information and then follow FATAL and recovery.

Cutting off a stalled wait

If you enable ha_wd_preempt (default 0) by setting it to 1, the watchdog does not stop at a warning but cuts off a wait that crossed the threshold by itself. A wait that is cut off ends in the same failure as a non-responding device, so if an application job was running, the device becomes FATAL. If the wait that was cut off is a device command wait without a ceiling, only that request fails and the device does not become FATAL.

Situation ha_wd_preempt=0 (default) ha_wd_preempt=1
A device command wait crosses the threshold Only a warning is left. When the ceiling expires, the command is judged as not responding The wait is cut off on the spot
A device command wait without a ceiling crosses the threshold Only a warning is left and the wait continues The wait is cut off on the spot. That request fails, but the device does not become FATAL
A data transfer crosses the threshold Only a warning is left. After 5 seconds, only the job that includes that transfer fails and the device does not become FATAL The transfer is cut off on the spot and handled as a failure at the same level as a non-responding device

Enabling ha_wd_preempt lets a non-response end sooner. A device command ends around the threshold (default 8 seconds) instead of the ceiling (default 12 seconds), and waits without a ceiling end too. In exchange, a data transfer that crosses the threshold is also cut off and the device becomes FATAL. With the default (0), only that job fails, and only after 5 seconds have passed; the device does not become FATAL.

A wait cut off by the watchdog with ha_wd_preempt=1 is not a ceiling expiry, so it does not increase devcmd_timeouts or hdma_timeouts, and it does not change the progress fields or the verdict of last_hang. The records left are the watchdog warning and the watchdog hang dump in the kernel log (for the devcmd arm), and faulted in health if the device became FATAL. If an automatic reset succeeds, faulted is cleared too, so on systems with ha_wd_preempt=1, monitor the kernel log as well. When a wait without a ceiling is cut off, the device does not become FATAL, so no automatic reset happens either. In this case, if the device appears to have stopped, reset it yourself following FATAL and recovery.

If you enable ha_wd_preempt while ha_auto_reset=0, a device that became FATAL stays that way until someone resets it. In this configuration, when the watchdog cuts off a wait, it leaves a warning that starts with watchdog: preempting without ha_auto_reset. If possible, use ha_wd_preempt together with ha_auto_reset=1. To enable them at run time, enable automatic reset first.

1
2
echo 1 | sudo tee /sys/module/ha/parameters/ha_auto_reset
echo 1 | sudo tee /sys/module/ha/parameters/ha_wd_preempt

To apply them every time the module is loaded, see Setting values in an option file. While a firmware update is in progress or the response to an update is unconfirmed, the watchdog does not cut off waits. This is to protect the boot flash that is being written.

Waits that legitimately take long are cut off too

The watchdog looks only at elapsed time, so it also cuts off normal operations that cross the threshold. Do not enable ha_wd_preempt on systems that run jobs that wait a long time without a ceiling.


Reading the records after a wait ceiling expires (hang)

RAS counters

Node When it increases
devcmd_timeouts When the response wait ceiling of a device command expires. It also increases when the response deadline of a firmware update passes
devcmd_fw_errors When the firmware returns an error result for a command. It counts not only the firmware's own timeouts but also per-command failures such as invalid arguments
hdma_timeouts When a single data transfer does not finish within 5 seconds, or its completion signal does not arrive within 5 seconds
  • The three counters never decrease. A device reset does not bring them to 0; they start counting again from 0 only when the driver probes the device again (module reload, ha-hotreset, reboot, and so on).
  • If any one of the three is not 0, health is degraded. If the device is FATAL or the last reset failed, faulted takes precedence. So degraded does not mean the device is broken now; it means that an error has occurred at least once since the driver probed the device.
  • In monitoring, watch for increases, not the value itself. If the value is greater than the one you last read, it is a new event. See Telemetry.
  • Watchdog warnings, warnings for waits without a ceiling, and waits cut off by ha_wd_preempt=1 do not increase these counters.

last_hang

1
cat /sys/class/ha/ha0/last_hang
1
rc=- cmd_idx=- opcode=- batch_seq=1041->1042 exec_index=0 resq_tail=0 verdict=stuck-in-command

One line holds two records of different natures.

  • The first three fields (rc, cmd_idx, opcode) describe the last command for which the firmware returned an error result. Despite the name, it records every firmware error result, not only hangs.
  • The last four fields (batch_seq, exec_index, resq_tail, verdict) are the batch progress information the firmware exposed at the time of the last hang observation, and its verdict. They are recorded when a response wait ceiling expires, when the response deadline of a firmware update expires, and at the periodic warnings of a wait without a ceiling; watchdog warnings do not change these values.

A response wait ceiling expiry is a case where the firmware did not respond at all, so it does not fill in the first three fields. They stay -, as in the example above, or point to an earlier, different error, so read a non-responding hang from verdict and the progress fields.

Field Value Meaning when -
rc Firmware error result code (hex). See Firmware error result There has been no firmware error yet (devcmd_fw_errors is 0)
cmd_idx Position of the failed command within the batch Same as above
opcode Firmware command code of the failed command (hex) Same as above
batch_seq <value just before the command was sent>-><value at observation time>. A number the firmware increments when it picks up a command batch There has been no hang observation yet, or no verdict can be made (verdict is none or unsupported)
exec_index Position within the batch of the command the firmware was executing at observation time. none if no command was executing Same as above
resq_tail The amount (in 32-bit words) the firmware had written to the response area at observation time. The firmware resets this value to 0 when it picks up a batch, so if verdict is not-picked-up, it is the value from the previous batch Same as above
verdict Batch progress verdict. See Batch progress verdict Not used

The same values also appear in the RAS / Health section of hyperaccel-smi -q as Last Hang, Hang Verdict, and Batch Progress.

Batch progress verdict

When a hang is observed, the driver reads the progress information exposed by the firmware once and judges where in the batch the firmware stopped. The verdict value appears in verdict of last_hang and at the end of the error line in the kernel log, and a sentence with the same meaning appears at the end of the human-readable line of the hang dump. For example, stuck-in-command means that the firmware started the command at position exec_index but did not finish it. The full list of verdict values and sentences is in Kernel log messages.

The verdict is diagnostic information

The driver does not change its behavior according to the verdict. When the response wait ceiling expires, the same records are left whatever the verdict is, and if an application job was running, the device becomes FATAL and must be reset before it can be used again. The same applies to completed-irq-lost. The sentence for this verdict (a reset is NOT needed) means the device did not stop, and tells you that the cause is in the delivery of the completion interrupt, not in the device. If the same verdict repeats, report it to the support team with the diagnostic information attached.

Firmware error result

When the firmware returns an error result for a command, the following line is left in the kernel log, followed by a response area dump that starts with resq dump. The same values are recorded in the first three fields of last_hang, and devcmd_fw_errors increases by 1.

1
ha 0000:3b:00.0: Firmware error: resq[0] rc=0x7 cmd_idx=0 opcode=0x2

If many errors occur in a short time, only some of them are left in the kernel log, but devcmd_fw_errors counts every one.

Depending on rc, a result either fails only that command and lets you keep using the device, or makes the device FATAL if an application job is running. 0x7 means the firmware gave up on that command at its time limit and wrote a response, and 0x9 is a fatal core error. Both make the device FATAL when an application job is running. The difference between 0x7 and an expired response wait ceiling is in Response wait ceiling. The meaning and consequence of each rc value are in Kernel log messages, and for firmware update results (0xa to 0xd), see fw_rc values of ha-fw-update.

The firmware keeps running the rest of a batch after an earlier command in that batch fails. Commands from the same batch may have run after the first Firmware error, so do not read that single line as the point where the device stopped.

Hang dump

Two lines are left in the kernel log when the response wait ceiling expires (ceiling timeout), when the response deadline of a firmware update passes (fw_update timeout), and when the watchdog reports a device command wait and that wait is a wait without a ceiling or ha_wd_preempt=1 is set (watchdog). The tag in parentheses is the reason the record was left. The first line is a human-readable summary, and the second line is the raw data to attach as is to a support request. Each line is self-contained, so you can read them even if other log lines come in between.

This is an example for a response wait ceiling expiry. The ha at the start of the line is the driver name, and 0000:3b:00.0 is the PCI address of that device.

1
2
3
ha 0000:3b:00.0: Devcmd no response after 12000 ms (device hang): stuck-in-command
ha 0000:3b:00.0: hang (ceiling timeout): fw LAUNCH_TCTRL_INST (op 0x02), batch slot 0 (last staged, not necessarily the stuck one); kernel@0x0000008090004000 size=36 cores=0x1 buf_sel=0; opened by pid 4182 (python3) -- firmware started a command and never finished it
ha 0000:3b:00.0: hang dump (ceiling timeout): stream=0x3f9a21c4 corr=0 opcode=0x02 cmd_idx=0 payload=[00000002 00000000 90004000 00000080 00000001 00000024 00000000 00000000] staged=0 flushed=1 mbox_head=1 db=1 hdma_wr=00000000/00000000 hdma_rd=00000000/00000000

Interpret the human-readable line (the 2nd line of the example, hang (...)) as follows.

Part Meaning
fw <name> (op 0x<code>) The name and code of the last firmware command sent. A command whose name is not known appears as unknown
batch slot <n> (last staged, ...) The position of that command within the batch. It is the last command sent, which may not be the command that stopped. Look at exec_index of last_hang for the position where it stopped
kernel@<address> size=<n> cores=<mask> buf_sel=<n> For a kernel launch command (LAUNCH_TCTRL_INST), the kernel's address, size, the mask of the cores used for execution, and the kernel binary buffer number
abi@<address> size=<n> For a LAUNCH_IDMA command, the address and size of the data that the command moves
opened by pid <PID> (<name>) The process that submitted the command to the device. If a process that inherited the file descriptor or was handed it submitted the job, this PID can differ from the actual submitter. It does not appear for commands that are not a specific process's job, such as a reset
-- <sentence> The sentence of the Batch progress verdict. A line with the watchdog tag does not read progress information, so it ends with progress not sampled here, so no verdict

Interpret the fields of the raw line (the 3rd line of the example, hang dump (...)) as follows.

Field Meaning
stream A tag that identifies the stream that issued the command. 0 is a command that does not belong to a specific stream
corr The ID that the submitter attached to the command if it used the profiler. 0 if there is none
opcode, cmd_idx Same as op and batch slot of the human-readable line
payload The first 8 words (hex) of that command's body
staged, flushed The number of commands queued to be sent, and the number of commands last sent to the device
mbox_head, db These are the values the driver wrote to the device when it sent the command, read back. If mbox_head differs from flushed or is 4294967295 (all bits are 1), the driver's write did not reach the device, which is a problem with the PCIe link or the access path, not a non-response of the device
hdma_wr, hdma_rd The status/interrupt status register values of the DMA engine's write (device→host) channel and read (host→device) channel. If all four are ffffffff, (hdma unreadable: link down?) is appended to the end of the line. This means the link is down or the device is not responding to PCIe reads

The full list of kernel log messages is in Kernel log messages.


When a hang is suspected

What to check first

The sysfs nodes below do not access the device, so you can read them without root privileges even while the device is stopped.

  1. Check whether health is faulted. If it is faulted, the device is FATAL and accepts no jobs until it is reset, or the last reset has failed.
  2. Check whether inflight stays at a value other than 0 without decreasing. It can be a sign that jobs are not progressing.
  3. Check whether devcmd_timeouts, devcmd_fw_errors, and hdma_timeouts have increased since the values you last saw.
  4. Use verdict of last_hang to see where the firmware stopped.
  5. Look in the kernel log for watchdog warnings, ceiling expiries, hang dumps, and firmware error records.
1
2
3
d=/sys/class/ha/ha0
grep . $d/health $d/inflight $d/devcmd_timeouts $d/devcmd_fw_errors $d/hdma_timeouts $d/last_hang
sudo dmesg -T | grep -E ' ha [0-9a-f]+:' | grep -E 'watchdog|Devcmd|hang|Firmware error'

The kernel log refers to a device by its PCI address. Check the mapping to the node number with Bus-Id in hyperaccel-smi -q or with readlink -f /sys/class/ha/ha0/device.

Collecting diagnostic information

When you request support, send the driver version, the hang-related parameters, the sysfs nodes of each device, the output of hyperaccel-smi -q, and the entire kernel log together. An example script that collects all of these at once is in Collecting diagnostic information for a problem.

Attach the collected results as they are. Do not pick out only the lines that seem relevant. The cause is often in lines that come before the hang dump.

Save before resetting

A reset erases the state that explains the failure. Save the diagnostic information before resetting.

  • A reset ends all stuck jobs and re-initializes the device. The stuck state itself is gone, and only the records remain.
  • If the firmware does not respond during a reset, the progress fields and the verdict of last_hang are replaced by the record of that reset; if the firmware returns an error for a reset command, the first three fields are replaced.
  • A recovery in which the driver probes the device again, such as ha-hotreset or a module reload, returns the counters and last_hang to their initial state.
  • The kernel log is a ring buffer, so older lines are pushed out first as time passes.

If ha_auto_reset=1, the driver resets by itself right after FATAL, so there is no chance to save the pre-reset state. In this case, diagnose from the kernel log and from the counters and last_hang that remain after the reset.


See also