Collecting diagnostic information for a problem¶
This page explains how to collect, into a single archive, the information the support team needs to find the cause when a card does not respond (hang), becomes FATAL, or behaves differently from usual. A reset erases the state that explains the failure, so run the script on this page before you reset the card. The script does not open the device node and only reads state, so you can run it while workloads are running.
Collect first, then recover¶
When to run it¶
If the health check or monitoring reports one of the following, run the script before you take any action on the card. Taking action on a card means a reset requested through the SDK, ha-hotreset, a module reload or unbind, or a host reboot.
healthisfaulted, or Health inhyperaccel-smiisFAULTED.devcmd_timeouts,devcmd_fw_errors, orhdma_timeoutsis higher than the value you last saw.- The kernel log contains a watchdog warning (
watchdog: … wait stalled),Devcmd no response,Devcmd waiting, a hang dump (hang (…),hang dump (…)),Firmware error, orsys_mon: no response. - An application receives a device error from the SDK, or
inflightdoes not decrease although the workload is not making progress. - Temperature, power, and utilization stay
unknown. - Performance is lower than expected and you are contacting support. The archive also includes
abi_prefetch_stats.
After collecting, follow Card recovery.
Records lost when you take action¶
After you take action on a card, the state at that moment can no longer be seen. A reset ends the stalled jobs and leaves only records, and which records remain also depends on the method.
| Record | Device reset (SDK reset, automatic reset) | Card probed again (ha-hotreset, module reload, unbind then bind) |
Host reboot |
|---|---|---|---|
faulted in health |
Clears. Remains if the reset fails | Clears | Clears |
| Error counters | Remain | Restart from 0 | Restart from 0 |
last_hang |
Remains. If the firmware does not respond during the reset, the progress fields (batch_seq, exec_index, resq_tail) and the verdict (verdict) are replaced by the record of that reset; if the firmware returns an error for the reset command, the first three fields (rc, cmd_idx, opcode) are replaced by the record of that reset |
Returns to its initial state | Returns to its initial state |
| Kernel log | Remains, but messages logged afterward can push out earlier lines | Same as a device reset | The kernel log buffer is emptied. Lines remain in the journal only if the systemd journal is stored persistently |
Module reload and unbind do not reset the card, so they cannot revive a stalled card, but they erase the driver's records in this way. See Recovery methods.
Systems with automatic reset enabled
If ha_auto_reset=1, the driver resets the card by itself right after FATAL, so there is no opportunity to collect the state from before the reset. Run the script as soon as you notice. The kernel log, error counters, and last_hang, which remain after the reset, serve as the basis for diagnosis.
The script collects only the kernel log of the current boot. If you have already rebooted the host, save the previous boot's kernel log separately with sudo journalctl -k -b -1 > journal-prev.txt and send it along. On hosts that do not store the journal persistently, the previous boot's kernel log may remain in syslog files (for example /var/log/kern.log, /var/log/messages).
Collection script¶
Running the script¶
Save the script below as ha-collect-diag.sh and run it with root privileges. The argument is the output directory; if omitted, the output is created in the current directory.
1 | |
1 | |
The files before compression remain in a directory of the same name (/var/tmp/ha-diag-20261008-153012/). Send the .tar.gz file to the support team. In an environment where the host's /sys is mounted at a different path (for example, a container), pass that path in SYSFS, as in SYSFS=/host/sys sh ha-collect-diag.sh. Inside a container, the distribution information in host.txt and modinfo.txt belong to the container and the kernel log is easily missing, so run the script as root on the host if possible.
The script¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | |
What is collected¶
| File | Contents |
|---|---|
sysfs-haN.txt |
One per card. The PCI address on the first line, the values of the nodes listed in NODES, the IOMMU group number and its type |
host.txt |
Collection time, uname -a, /etc/os-release |
driver.txt |
The loaded driver version (/sys/module/ha/version) and the current values of the module parameters |
lsmod.txt, modinfo.txt |
The list of loaded modules, information about the installed ha module file |
dmesg.txt |
The entire kernel log buffer |
journal.txt |
The kernel log of the current boot remaining in the systemd journal |
hyperaccel-smi-q.txt, hyperaccel-smi.json |
The output of hyperaccel-smi -q and --json |
lspci-vvv.txt |
The PCI configuration space of the HyperAccel cards (PCI vendor ID 20fb). It includes the link status and error status registers |
lspci-tv.txt |
PCI topology |
status.txt |
The number of cards found and the result of each command |
status.txt records one line per command with its result. ok means normal, missing means the command was skipped because the host does not have it, and exit <code> means the command ended with an exit code other than 0. lspci is in the pciutils package, and hyperaccel-smi ends with exit 2 if no card is visible. A sysfs node that cannot be read is recorded as an error message in its place in sysfs-haN.txt. In every case the script does not stop and collects the rest. For the meaning of the IOMMU type value, see IOMMU mode and the device-to-device path.
Nodes that are not read¶
The script specifies the nodes to read by name, and it does not read the following two nodes.
| Node | Why it is not read |
|---|---|
fw_update_progress |
Every read fetches the value directly from the device. It must not be read on a card that does not respond |
help |
It is fixed description text built into the driver and contains no card state |
bypass_range_check, a debugging setting, is not telemetry, but when its value is 1 the driver does not check the range of device memory access requests, so it is needed for diagnosis and is collected as well (bypass_range_check). The remaining nodes report the state the driver keeps in host memory and the values the poller has collected, so reading them does not access the device and they respond even while the card is stalled. The exceptions are fw_version and serial. While fw_version is unknown, reading these two nodes checks again whether the firmware is ready; the conditions under which the driver skips this check are described in Monitoring suspension. Because the nodes are specified by name, a node that has not been verified is not read even if a new driver version adds nodes.
root privileges¶
Even without root, the script runs to the end, and the sysfs nodes, module parameters, hyperaccel-smi, and lspci -tv are collected as usual. The following three items are empty or only partly collected without root privileges. In that case the script prints a line starting with not root: at startup.
dmesg.txt: If the kernel settingkernel.dmesg_restrictis1, a user who is not root (more precisely, who lacks theCAP_SYSLOGcapability) cannot read the kernel log buffer. Check the current value withsysctl kernel.dmesg_restrict.journal.txt: Without root, the system journal cannot be read, so the kernel log may be missing.lspci-vvv.txt: Without root, only the first 64 bytes of the PCI configuration space can be read, soCapabilities: <access denied>appears instead of the link status and error status. Inside a container, the same can happen even as root.
Reading the archive¶
These are the things to check yourself before sending the archive to the support team. The meaning of each value is on the linked page. A per-card file starts as follows.
1 2 3 4 5 6 | |
- Card state: If
healthisfaulted, the device is FATAL or the last reset has failed. See health verdict. - Error counters: For
devcmd_timeouts,devcmd_fw_errors, andhdma_timeouts, look at whether the value has increased since the last time you saw it, rather than at the value itself. See RAS counters. last_hang:verdictis the verdict on where in the batch the firmware stopped. The first three fields and the last four fields can be records of different events. Seelast_hangand Batch progress verdict.- Kernel log: Look for the driver's hang records in
dmesg.txt(command below). Thehang (…)line is a human-readable summary, and thehang dump (…)line holds the raw values. See Hang dump and Kernel log messages. - Missing items: Check whether
status.txtcontainsmissingorexit. If the kernel log is missing, run the script again with root privileges.
1 | |
Telling slow from stalled
If Devcmd waiting keeps appearing every 10 seconds in the kernel log, an application is waiting without a ceiling. Run the script twice, at least 20 seconds apart, and compare last_hang in the two archives. If batch_seq or exec_index has changed, the device is making progress. See Waits without a ceiling.
What to check before sending¶
The archive contains as much host information as diagnosis requires.
- It contains the host name, the kernel and distribution versions (
host.txt), the list of loaded modules, and the PCI device list and configuration. - The kernel log (
dmesg.txt,journal.txt) also contains all messages from devices and services unrelated to this driver. In a hang record,opened by pid <PID> (<name>)shows the name of the process that opened the card. - The directory and archive the script creates are readable only by the user who ran the script. If you ran it with
sudo, root privileges are also needed to read or copy them.
Review the contents before sending. If your organization's policy requires masking some values, do not delete lines from the files in the directory; replace only those values and then re-create the archive (sudo sh -c 'umask 077; cd /var/tmp && tar -czf ha-diag-<time>.tar.gz ha-diag-<time>'). The archive the script first created still contains the values from before the replacement. The cause is sometimes on a line preceding the driver messages (for example, an error from an upstream PCIe port), so picking out individual lines makes diagnosis harder. What to provide along with the archive (when the symptom started, the jobs that were running at the time, and reproduction steps) is described in Requesting support.
See also¶
- Card recovery — Recovery procedure after the diagnostic information is collected
- Health check, Monitoring — Monitoring that tells you when to run this script
- Hang detection and the watchdog — Reading
last_hang, hang dumps, and RAS counters - Telemetry — How nodes obtain their values, and the meaning of
unknown - Kernel log messages
- Troubleshooting
- sysfs nodes
- hyperaccel-smi