Hang detection and the watchdog¶
This page explains how ha.ko notices that a device is not responding to a command and what it records. The driver judges a non-response against the response wait ceiling, separately observes waits that grow long with the watchdog, and records the results in the RAS counters, the sysfs last_hang, and the hang dump in the kernel log. This page covers when each feature operates, how to read the records, and what to collect when a hang is suspected. Parameters are defined in Parameter reference, how to configure them is in Configuring and using parameters, and the procedure for reviving a stopped device is in FATAL and recovery.
Waits and ceilings¶
The driver waits for the device in two places. One is the response to a command sent to the firmware (devcmd in kernel log messages and sysfs node names), and the other is the completion of a transfer that moves data between the host and a device, or between two devices. The driver bundles several device commands and sends them at once, and this bundle is called a batch.
Each of the two waits has its own ceiling, and the watchdog watches both waits from the outside with a threshold shorter than the ceiling.
| Wait for a device command's response | Wait for a DMA transfer to complete | |
|---|---|---|
| Ceiling | ha_devcmd_ceiling_ms (default 12000 ms) |
5 seconds per transfer. It cannot be changed with a parameter |
| When the ceiling is exceeded | The command is judged as not responding. If an application job was running, the device becomes FATAL | The job that includes that transfer fails. The device does not become FATAL |
| Records left | devcmd_timeouts, last_hang, the hang dump in the kernel log |
hdma_timeouts |
| Watchdog threshold | ha_wd_hang_ms (default 8000 ms) |
ha_wd_hdma_hang_ms (default 4000 ms) |
The watchdog only reports waits that crossed the threshold in the kernel log, and with the default settings it does not intervene in waits. What ends a wait and decides a failure is whether the response wait ceiling was exceeded.
Response wait ceiling¶
After sending a command to the firmware, the driver waits for the response. The response wait ceiling is the maximum time allowed for waiting for a response, and it applies not only to the commands of application jobs but to all commands the driver sends, such as the firmware re-initialization during a reset. If a command does not finish for a long time, the firmware reports a timeout on its own, and the wait ceiling is that time plus some margin. So if the firmware is alive, a failure result that contains the cause arrives first, and the ceiling expires only when even the firmware does not respond.
The host watchdog (ha_wd_*) and the time limit the firmware sets for each command are separate. The firmware allows a longer limit for commands that take longer, and it waits at least 8 seconds when that limit would be shorter. The driver's default ceiling is 12 seconds, chosen so that the firmware reports the timeout first.
The two outcomes are recovered differently.
- If the firmware returns
rc=0x7, the device can still answer commands, and only that command ended in a timeout. When an application job is running, the driver marks the device FATAL, so an SDK device reset can still reach the firmware. - If the ceiling expires, the firmware did not send a response. If command processing has stopped, re-initialization does not run either, and the reset goes on to an SBR.
If you set ha_devcmd_ceiling_ms shorter than the firmware's time limit, the driver marks the device FATAL for no response before rc=0x7 arrives. This is the same constraint as the warning in Changing the ceiling.
When the ceiling expires¶
The driver judges the command as not responding and leaves the following records.
- The kernel log gets the error
Devcmd no response after <N> ms (device hang): <verdict>and two lines of the hang dump.<N>is the ceiling that was applied (in ms). How to read them is described in Hang dump. devcmd_timeoutsincreases by 1, and the batch progress information and the verdict are recorded inlast_hang.
If an application job was running, the driver switches the device to FATAL.
- The kernel or job that was running fails, and other waiting kernels or whole jobs are cancelled as well. The application receives the error through the error code returned by the runtime.
- Until a reset, the device accepts no new jobs. This is true even if you restart the process.
healthbecomesfaulted. Telemetry that is read periodically, such as temperature, power, and utilization, soon changes tounknownand stays that way until a reset.- If
ha_auto_reset=1, the driver attempts a device reset by itself. With the default0, the device stays FATAL until it is reset. See FATAL and recovery.
The same records can also be left during a reset
If the firmware does not respond to the re-initialization command during a reset, the ceiling expires, a ceiling timeout hang dump is left, and devcmd_timeouts increases. The driver then moves on to the next recovery step. A hang dump you see right after a reset may be a record of the reset, not of the original hang.
Long-running commands¶
Large memory fills and firmware updates can legitimately take longer than the default ceiling. When the driver sends such a command, it extends the ceiling by the command's expected duration plus a margin, and it extends the watchdog thresholds by the same amount. So these operations are not judged to be hangs when they take a long time normally. In exchange, if the device really stops during such an operation, the verdict is also delayed by the extended amount.
Changing the ceiling¶
ha_devcmd_ceiling_ms can be changed at run time, and the new value applies from the waits that start after the change. A value smaller than 100 is treated as 100 ms, and 0 is also 100 ms, not an infinite wait. In this case, /sys/module/ha/parameters/ha_devcmd_ceiling_ms shows the value you wrote unchanged.
Do not lower the ceiling below the default
The ceiling must be longer than the firmware's own timeout judgment. If it becomes shorter, the driver judges the command as not responding before the firmware reports the cause, so the device can become FATAL even for a command that would have completed normally. Lowering the ceiling can also keep the watchdog's arms from firing (Thresholds and the polling interval).
Raising the ceiling delays the verdict by the same amount when the device has really stopped.
Watchdog¶
One watchdog runs per device, and every ha_wd_poll_ms (default 500 ms) it checks how long the waits in progress have lasted. The waiting side reaches a verdict only when its own ceiling runs out, but the watchdog looks from outside the wait, so it reports a wait that is getting long before the ceiling. The watchdog is also the first to report a wait that has no ceiling.
Records left by the watchdog¶
When a wait crosses the threshold, the watchdog leaves one warning line in the kernel log.
1 | |
- The word after
watchdog:is the arm that crossed its threshold this time:devcmd,hdma, ordevcmd+hdmaif both crossed together. - The parentheses show how long the two waits had lasted at that moment.
0means there is no wait of that kind. - Only one warning is issued per wait. If another wait crosses the threshold after that wait has ended, it is reported again. The two arms are counted separately.
- The watchdog makes no judgment while the device is being reset or is FATAL.
- If the wait reported by the
devcmdarm is a wait without a ceiling, or ifha_wd_preempt=1, the warning is followed by a hang dump with thewatchdogtag. For a wait that has a ceiling, a dump with a verdict is left when the ceiling expires, so the watchdog does not leave a dump. Thehdmaarm does not leave a dump.
The time added by Long-running commands is added to the threshold of the devcmd arm. The threshold of the hdma arm applies to the wait for a single transfer handed to the DMA engine, not to the entire copy operation.
The watchdog only warns and gives no verdict¶
The watchdog looks only at elapsed time. It cannot distinguish a wait that legitimately takes long from a device that is not responding, so the warning text says not "hang" but "no completion within the threshold". Read a warning as an observation, and base the verdict on the records that follow. With the default (ha_wd_preempt=0), read it as follows.
- If a
devcmdwarning is not followed by the ceiling-expiry records (Devcmd no responseand theceiling timeouthang dump), that wait ended before the ceiling. - If
hdma_timeoutsdid not increase after anhdmawarning, that transfer finished within 5 seconds. - A wait without a ceiling produces a warning when it crosses the threshold even if it is normal operation. See Waits without a ceiling.
Thresholds and the polling interval¶
The watchdog checks the state of the waits once per polling interval, so it can take up to one more interval to notice a wait that crossed the threshold. If the waiting side's ceiling ends in the meantime, the watchdog never sees that wait. So for each arm to fire, the following must hold.
| Arm | Condition | Default |
|---|---|---|
devcmd |
ha_wd_hang_ms + ha_wd_poll_ms < ha_devcmd_ceiling_ms |
8000 + 500 < 12000 |
hdma |
ha_wd_hdma_hang_ms + ha_wd_poll_ms < 5000 |
4000 + 500 < 5000 |
The time added by long-running commands is added equally to the threshold and the ceiling, so this relationship holds as is.
If an arm condition is broken, the driver does not change the values and reports it in the kernel log. For example, if you raise ha_wd_poll_ms to 1000, the hdma arm cannot fire.
1 | |
This warning appears only once for the same configuration. If you change one of the two thresholds, the polling interval, or the response wait ceiling and the condition is still broken, it appears again. Lower the threshold or the polling interval to satisfy the condition. Watchdog parameters can be changed at run time and apply from the next poll.
The meaning of 0 differs from parameter to parameter.
| Parameter | If you use 0 |
|---|---|
ha_wd_hang_ms |
The devcmd arm is turned off |
ha_wd_hdma_hang_ms |
The hdma arm is turned off |
ha_wd_poll_ms |
It is not turned off. A value smaller than 50 is treated as 50 ms |
The only way to turn off an arm is to set its threshold, ha_wd_hang_ms or ha_wd_hdma_hang_ms, to 0. The arms keep running every interval, so if you set a threshold again, the arm comes back from the next poll. If both thresholds are 0, the watchdog judges nothing.
Waits without a ceiling¶
The response wait ceiling does not apply to a device command that the application asked to wait for without a ceiling. Even if the device does not respond, the driver does not end the wait by itself; such a wait ends when a response arrives, when the waiting process receives a termination signal, or when the device is reset or removed. If ha_wd_preempt=1, the watchdog cuts the wait off at the threshold (Cutting off a stalled wait).
The records left by such a wait without a ceiling are as follows.
- When the watchdog threshold (default 8 seconds) is exceeded, a watchdog warning and a hang dump with the
watchdogtag are left. This dump does not read progress information, so it ends withprogress not sampled here, so no verdictinstead of a verdict. - When the wait exceeds the same time as the response wait ceiling (default 12 seconds), a warning that starts with
Devcmd waiting <N> ms with no timeout requestedappears, and it repeats every 10 seconds after that. The name and ID of the waiting thread (name[ID]) and the verdict value are appended at the end of the line. Each time the warning is issued, the progress information is read anew and recorded inlast_hang.
This wait does not increase the devcmd_timeouts count and does not make the device FATAL. It does not show up in health alone, so check the kernel log.
Telling a long operation from a stopped device
Read last_hang twice at least 20 seconds apart. If batch_seq or exec_index has changed, the device is making progress. If they have not changed, a single command is taking a long time or has stopped. Judge by comparing with how long that command normally takes, and if you conclude that it has stopped, save the diagnostic information and then follow FATAL and recovery.
Cutting off a stalled wait¶
If you enable ha_wd_preempt (default 0) by setting it to 1, the watchdog does not stop at a warning but cuts off a wait that crossed the threshold by itself. A wait that is cut off ends in the same failure as a non-responding device, so if an application job was running, the device becomes FATAL. If the wait that was cut off is a device command wait without a ceiling, only that request fails and the device does not become FATAL.
| Situation | ha_wd_preempt=0 (default) |
ha_wd_preempt=1 |
|---|---|---|
| A device command wait crosses the threshold | Only a warning is left. When the ceiling expires, the command is judged as not responding | The wait is cut off on the spot |
| A device command wait without a ceiling crosses the threshold | Only a warning is left and the wait continues | The wait is cut off on the spot. That request fails, but the device does not become FATAL |
| A data transfer crosses the threshold | Only a warning is left. After 5 seconds, only the job that includes that transfer fails and the device does not become FATAL | The transfer is cut off on the spot and handled as a failure at the same level as a non-responding device |
Enabling ha_wd_preempt lets a non-response end sooner. A device command ends around the threshold (default 8 seconds) instead of the ceiling (default 12 seconds), and waits without a ceiling end too. In exchange, a data transfer that crosses the threshold is also cut off and the device becomes FATAL. With the default (0), only that job fails, and only after 5 seconds have passed; the device does not become FATAL.
A wait cut off by the watchdog with ha_wd_preempt=1 is not a ceiling expiry, so it does not increase devcmd_timeouts or hdma_timeouts, and it does not change the progress fields or the verdict of last_hang. The records left are the watchdog warning and the watchdog hang dump in the kernel log (for the devcmd arm), and faulted in health if the device became FATAL. If an automatic reset succeeds, faulted is cleared too, so on systems with ha_wd_preempt=1, monitor the kernel log as well. When a wait without a ceiling is cut off, the device does not become FATAL, so no automatic reset happens either. In this case, if the device appears to have stopped, reset it yourself following FATAL and recovery.
If you enable ha_wd_preempt while ha_auto_reset=0, a device that became FATAL stays that way until someone resets it. In this configuration, when the watchdog cuts off a wait, it leaves a warning that starts with watchdog: preempting without ha_auto_reset. If possible, use ha_wd_preempt together with ha_auto_reset=1. To enable them at run time, enable automatic reset first.
1 2 | |
To apply them every time the module is loaded, see Setting values in an option file. While a firmware update is in progress or the response to an update is unconfirmed, the watchdog does not cut off waits. This is to protect the boot flash that is being written.
Waits that legitimately take long are cut off too
The watchdog looks only at elapsed time, so it also cuts off normal operations that cross the threshold. Do not enable ha_wd_preempt on systems that run jobs that wait a long time without a ceiling.
Reading the records after a wait ceiling expires (hang)¶
RAS counters¶
| Node | When it increases |
|---|---|
devcmd_timeouts |
When the response wait ceiling of a device command expires. It also increases when the response deadline of a firmware update passes |
devcmd_fw_errors |
When the firmware returns an error result for a command. It counts not only the firmware's own timeouts but also per-command failures such as invalid arguments |
hdma_timeouts |
When a single data transfer does not finish within 5 seconds, or its completion signal does not arrive within 5 seconds |
- The three counters never decrease. A device reset does not bring them to 0; they start counting again from 0 only when the driver probes the device again (module reload,
ha-hotreset, reboot, and so on). - If any one of the three is not 0,
healthisdegraded. If the device is FATAL or the last reset failed,faultedtakes precedence. Sodegradeddoes not mean the device is broken now; it means that an error has occurred at least once since the driver probed the device. - In monitoring, watch for increases, not the value itself. If the value is greater than the one you last read, it is a new event. See Telemetry.
- Watchdog warnings, warnings for waits without a ceiling, and waits cut off by
ha_wd_preempt=1do not increase these counters.
last_hang¶
1 | |
1 | |
One line holds two records of different natures.
- The first three fields (
rc,cmd_idx,opcode) describe the last command for which the firmware returned an error result. Despite the name, it records every firmware error result, not only hangs. - The last four fields (
batch_seq,exec_index,resq_tail,verdict) are the batch progress information the firmware exposed at the time of the last hang observation, and its verdict. They are recorded when a response wait ceiling expires, when the response deadline of a firmware update expires, and at the periodic warnings of a wait without a ceiling; watchdog warnings do not change these values.
A response wait ceiling expiry is a case where the firmware did not respond at all, so it does not fill in the first three fields. They stay -, as in the example above, or point to an earlier, different error, so read a non-responding hang from verdict and the progress fields.
| Field | Value | Meaning when - |
|---|---|---|
rc |
Firmware error result code (hex). See Firmware error result | There has been no firmware error yet (devcmd_fw_errors is 0) |
cmd_idx |
Position of the failed command within the batch | Same as above |
opcode |
Firmware command code of the failed command (hex) | Same as above |
batch_seq |
<value just before the command was sent>-><value at observation time>. A number the firmware increments when it picks up a command batch |
There has been no hang observation yet, or no verdict can be made (verdict is none or unsupported) |
exec_index |
Position within the batch of the command the firmware was executing at observation time. none if no command was executing |
Same as above |
resq_tail |
The amount (in 32-bit words) the firmware had written to the response area at observation time. The firmware resets this value to 0 when it picks up a batch, so if verdict is not-picked-up, it is the value from the previous batch |
Same as above |
verdict |
Batch progress verdict. See Batch progress verdict | Not used |
The same values also appear in the RAS / Health section of hyperaccel-smi -q as Last Hang, Hang Verdict, and Batch Progress.
Batch progress verdict¶
When a hang is observed, the driver reads the progress information exposed by the firmware once and judges where in the batch the firmware stopped. The verdict value appears in verdict of last_hang and at the end of the error line in the kernel log, and a sentence with the same meaning appears at the end of the human-readable line of the hang dump. For example, stuck-in-command means that the firmware started the command at position exec_index but did not finish it. The full list of verdict values and sentences is in Kernel log messages.
The verdict is diagnostic information
The driver does not change its behavior according to the verdict. When the response wait ceiling expires, the same records are left whatever the verdict is, and if an application job was running, the device becomes FATAL and must be reset before it can be used again. The same applies to completed-irq-lost. The sentence for this verdict (a reset is NOT needed) means the device did not stop, and tells you that the cause is in the delivery of the completion interrupt, not in the device. If the same verdict repeats, report it to the support team with the diagnostic information attached.
Firmware error result¶
When the firmware returns an error result for a command, the following line is left in the kernel log, followed by a response area dump that starts with resq dump. The same values are recorded in the first three fields of last_hang, and devcmd_fw_errors increases by 1.
1 | |
If many errors occur in a short time, only some of them are left in the kernel log, but devcmd_fw_errors counts every one.
Depending on rc, a result either fails only that command and lets you keep using the device, or makes the device FATAL if an application job is running. 0x7 means the firmware gave up on that command at its time limit and wrote a response, and 0x9 is a fatal core error. Both make the device FATAL when an application job is running. The difference between 0x7 and an expired response wait ceiling is in Response wait ceiling. The meaning and consequence of each rc value are in Kernel log messages, and for firmware update results (0xa to 0xd), see fw_rc values of ha-fw-update.
The firmware keeps running the rest of a batch after an earlier command in that batch fails. Commands from the same batch may have run after the first Firmware error, so do not read that single line as the point where the device stopped.
Hang dump¶
Two lines are left in the kernel log when the response wait ceiling expires (ceiling timeout), when the response deadline of a firmware update passes (fw_update timeout), and when the watchdog reports a device command wait and that wait is a wait without a ceiling or ha_wd_preempt=1 is set (watchdog). The tag in parentheses is the reason the record was left. The first line is a human-readable summary, and the second line is the raw data to attach as is to a support request. Each line is self-contained, so you can read them even if other log lines come in between.
This is an example for a response wait ceiling expiry. The ha at the start of the line is the driver name, and 0000:3b:00.0 is the PCI address of that device.
1 2 3 | |
Interpret the human-readable line (the 2nd line of the example, hang (...)) as follows.
| Part | Meaning |
|---|---|
fw <name> (op 0x<code>) |
The name and code of the last firmware command sent. A command whose name is not known appears as unknown |
batch slot <n> (last staged, ...) |
The position of that command within the batch. It is the last command sent, which may not be the command that stopped. Look at exec_index of last_hang for the position where it stopped |
kernel@<address> size=<n> cores=<mask> buf_sel=<n> |
For a kernel launch command (LAUNCH_TCTRL_INST), the kernel's address, size, the mask of the cores used for execution, and the kernel binary buffer number |
abi@<address> size=<n> |
For a LAUNCH_IDMA command, the address and size of the data that the command moves |
opened by pid <PID> (<name>) |
The process that submitted the command to the device. If a process that inherited the file descriptor or was handed it submitted the job, this PID can differ from the actual submitter. It does not appear for commands that are not a specific process's job, such as a reset |
-- <sentence> |
The sentence of the Batch progress verdict. A line with the watchdog tag does not read progress information, so it ends with progress not sampled here, so no verdict |
Interpret the fields of the raw line (the 3rd line of the example, hang dump (...)) as follows.
| Field | Meaning |
|---|---|
stream |
A tag that identifies the stream that issued the command. 0 is a command that does not belong to a specific stream |
corr |
The ID that the submitter attached to the command if it used the profiler. 0 if there is none |
opcode, cmd_idx |
Same as op and batch slot of the human-readable line |
payload |
The first 8 words (hex) of that command's body |
staged, flushed |
The number of commands queued to be sent, and the number of commands last sent to the device |
mbox_head, db |
These are the values the driver wrote to the device when it sent the command, read back. If mbox_head differs from flushed or is 4294967295 (all bits are 1), the driver's write did not reach the device, which is a problem with the PCIe link or the access path, not a non-response of the device |
hdma_wr, hdma_rd |
The status/interrupt status register values of the DMA engine's write (device→host) channel and read (host→device) channel. If all four are ffffffff, (hdma unreadable: link down?) is appended to the end of the line. This means the link is down or the device is not responding to PCIe reads |
The full list of kernel log messages is in Kernel log messages.
When a hang is suspected¶
What to check first¶
The sysfs nodes below do not access the device, so you can read them without root privileges even while the device is stopped.
- Check whether
healthisfaulted. If it isfaulted, the device is FATAL and accepts no jobs until it is reset, or the last reset has failed. - Check whether
inflightstays at a value other than 0 without decreasing. It can be a sign that jobs are not progressing. - Check whether
devcmd_timeouts,devcmd_fw_errors, andhdma_timeoutshave increased since the values you last saw. - Use
verdictoflast_hangto see where the firmware stopped. - Look in the kernel log for watchdog warnings, ceiling expiries, hang dumps, and firmware error records.
1 2 3 | |
The kernel log refers to a device by its PCI address. Check the mapping to the node number with Bus-Id in hyperaccel-smi -q or with readlink -f /sys/class/ha/ha0/device.
Collecting diagnostic information¶
When you request support, send the driver version, the hang-related parameters, the sysfs nodes of each device, the output of hyperaccel-smi -q, and the entire kernel log together. An example script that collects all of these at once is in Collecting diagnostic information for a problem.
Attach the collected results as they are. Do not pick out only the lines that seem relevant. The cause is often in lines that come before the hang dump.
Save before resetting¶
A reset erases the state that explains the failure. Save the diagnostic information before resetting.
- A reset ends all stuck jobs and re-initializes the device. The stuck state itself is gone, and only the records remain.
- If the firmware does not respond during a reset, the progress fields and the verdict of
last_hangare replaced by the record of that reset; if the firmware returns an error for a reset command, the first three fields are replaced. - A recovery in which the driver probes the device again, such as
ha-hotresetor a module reload, returns the counters andlast_hangto their initial state. - The kernel log is a ring buffer, so older lines are pushed out first as time passes.
If ha_auto_reset=1, the driver resets by itself right after FATAL, so there is no chance to save the pre-reset state. In this case, diagnose from the kernel log and from the counters and last_hang that remain after the reset.
See also¶
- FATAL and recovery — the procedure for reviving a device that became FATAL, automatic reset
- Telemetry — the
healthverdict and integration with monitoring - Firmware update — response deadlines and reset restrictions during an update
- Troubleshooting
- Configuring and using parameters — how to configure them and the relationships between parameters
- Parameter reference — parameter definitions
- sysfs nodes
- Kernel log messages
- hyperaccel-smi