Skip to content

Collecting diagnostic information for a problem

This page explains how to collect, into a single archive, the information the support team needs to find the cause when a card does not respond (hang), becomes FATAL, or behaves differently from usual. A reset erases the state that explains the failure, so run the script on this page before you reset the card. The script does not open the device node and only reads state, so you can run it while workloads are running.


Collect first, then recover

When to run it

If the health check or monitoring reports one of the following, run the script before you take any action on the card. Taking action on a card means a reset requested through the SDK, ha-hotreset, a module reload or unbind, or a host reboot.

  • health is faulted, or Health in hyperaccel-smi is FAULTED.
  • devcmd_timeouts, devcmd_fw_errors, or hdma_timeouts is higher than the value you last saw.
  • The kernel log contains a watchdog warning (watchdog: … wait stalled), Devcmd no response, Devcmd waiting, a hang dump (hang (…), hang dump (…)), Firmware error, or sys_mon: no response.
  • An application receives a device error from the SDK, or inflight does not decrease although the workload is not making progress.
  • Temperature, power, and utilization stay unknown.
  • Performance is lower than expected and you are contacting support. The archive also includes abi_prefetch_stats.

After collecting, follow Card recovery.

Records lost when you take action

After you take action on a card, the state at that moment can no longer be seen. A reset ends the stalled jobs and leaves only records, and which records remain also depends on the method.

Record Device reset (SDK reset, automatic reset) Card probed again (ha-hotreset, module reload, unbind then bind) Host reboot
faulted in health Clears. Remains if the reset fails Clears Clears
Error counters Remain Restart from 0 Restart from 0
last_hang Remains. If the firmware does not respond during the reset, the progress fields (batch_seq, exec_index, resq_tail) and the verdict (verdict) are replaced by the record of that reset; if the firmware returns an error for the reset command, the first three fields (rc, cmd_idx, opcode) are replaced by the record of that reset Returns to its initial state Returns to its initial state
Kernel log Remains, but messages logged afterward can push out earlier lines Same as a device reset The kernel log buffer is emptied. Lines remain in the journal only if the systemd journal is stored persistently

Module reload and unbind do not reset the card, so they cannot revive a stalled card, but they erase the driver's records in this way. See Recovery methods.

Systems with automatic reset enabled

If ha_auto_reset=1, the driver resets the card by itself right after FATAL, so there is no opportunity to collect the state from before the reset. Run the script as soon as you notice. The kernel log, error counters, and last_hang, which remain after the reset, serve as the basis for diagnosis.

The script collects only the kernel log of the current boot. If you have already rebooted the host, save the previous boot's kernel log separately with sudo journalctl -k -b -1 > journal-prev.txt and send it along. On hosts that do not store the journal persistently, the previous boot's kernel log may remain in syslog files (for example /var/log/kern.log, /var/log/messages).


Collection script

Running the script

Save the script below as ha-collect-diag.sh and run it with root privileges. The argument is the output directory; if omitted, the output is created in the current directory.

1
sudo sh ha-collect-diag.sh /var/tmp
1
saved: /var/tmp/ha-diag-20261008-153012.tar.gz

The files before compression remain in a directory of the same name (/var/tmp/ha-diag-20261008-153012/). Send the .tar.gz file to the support team. In an environment where the host's /sys is mounted at a different path (for example, a container), pass that path in SYSFS, as in SYSFS=/host/sys sh ha-collect-diag.sh. Inside a container, the distribution information in host.txt and modinfo.txt belong to the container and the kernel log is easily missing, so run the script as root on the host if possible.

The script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
#!/bin/sh
# ha-collect-diag.sh - Collects LPU diagnostic information into one directory and one .tar.gz.
# Usage: sudo sh ha-collect-diag.sh [output directory] (default: current directory)
# It only reads. It does not write to sysfs, does not open /dev/haN, and does not reset anything.

# If the host's /sys is mounted at a different path, pass that path in SYSFS.
SYSFS=${SYSFS:-/sys}
# Nodes to read for each card. fw_update_progress and help are not read.
NODES="health devcmd_timeouts devcmd_fw_errors hdma_timeouts last_hang inflight
stream_count fw_version hw_version serial dram_size dram_used dram_free sram_size
sram_used sram_free temp temp_avg temp_blocks power_mw rails util
abi_prefetch_stats peer_map peer_route peer_sram tw_window_size dmabuf_exports
bypass_range_check"

# The output contains host information, so make it readable only by the user who ran the script.
umask 077
base=${1:-.}
name=ha-diag-$(date +%Y%m%d-%H%M%S)
out=$base/$name
{ mkdir -p "$base" && mkdir "$out"; } || exit 1
[ "$(id -u)" -eq 0 ] || echo "not root: dmesg, journal and lspci output may be incomplete" >&2

# run <file> <command> [args...]: Saves the output to the file and records the result in status.txt.
run() {
    f=$1
    shift
    if ! command -v "$1" >/dev/null 2>&1; then
        echo "$f: missing ($1 not found)" >> "$out/status.txt"
    elif "$@" > "$out/$f" 2>&1; then
        echo "$f: ok" >> "$out/status.txt"
    else
        echo "$f: exit $?" >> "$out/status.txt"
    fi
}

# 1. Per-card sysfs. Read first, before the state changes.
cards=0
for d in "$SYSFS"/class/ha/ha*; do
    [ -d "$d" ] || continue
    cards=$((cards + 1))
    n=${d##*/}
    {
        echo "# $n $(basename "$(readlink -f "$d/device")")"
        for node in $NODES; do
            grep -H . "$d/$node" 2>&1
        done
        echo "iommu_group: $(basename "$(readlink "$d/device/iommu_group" 2>/dev/null || echo none)")"
        grep -H . "$d/device/iommu_group/type" 2>&1
    } > "$out/sysfs-$n.txt"
done
echo "sysfs: $cards card(s) under $SYSFS/class/ha" >> "$out/status.txt"

# 2. Host and driver
{
    date
    uname -a
    cat /etc/os-release
} > "$out/host.txt" 2>&1
{
    grep -H . "$SYSFS/module/ha/version"
    grep -H . "$SYSFS"/module/ha/parameters/*
} > "$out/driver.txt" 2>&1
run lsmod.txt lsmod
run modinfo.txt modinfo ha

# 3. Kernel log of the current boot. Lines pushed out of the buffer may remain in the journal, so save both.
run dmesg.txt dmesg -T
run journal.txt journalctl -k -b --no-pager

# 4. hyperaccel-smi. Shows all cards, reading the same sysfs as above.
unset HYPERACCEL_VISIBLE_DEVICES
run hyperaccel-smi-q.txt hyperaccel-smi --root "$SYSFS/class/ha" -q
run hyperaccel-smi.json hyperaccel-smi --root "$SYSFS/class/ha" --json

# 5. PCI. 20fb is HyperAccel's PCI vendor ID.
run lspci-vvv.txt lspci -D -nn -vvv -d 20fb:
run lspci-tv.txt lspci -D -tv

# 6. Archive
if (cd "$base" && tar -czf "$name.tar.gz" "$name"); then
    echo "saved: $out.tar.gz"
else
    echo "archive failed; files are in $out" >&2
    exit 1
fi

What is collected

File Contents
sysfs-haN.txt One per card. The PCI address on the first line, the values of the nodes listed in NODES, the IOMMU group number and its type
host.txt Collection time, uname -a, /etc/os-release
driver.txt The loaded driver version (/sys/module/ha/version) and the current values of the module parameters
lsmod.txt, modinfo.txt The list of loaded modules, information about the installed ha module file
dmesg.txt The entire kernel log buffer
journal.txt The kernel log of the current boot remaining in the systemd journal
hyperaccel-smi-q.txt, hyperaccel-smi.json The output of hyperaccel-smi -q and --json
lspci-vvv.txt The PCI configuration space of the HyperAccel cards (PCI vendor ID 20fb). It includes the link status and error status registers
lspci-tv.txt PCI topology
status.txt The number of cards found and the result of each command

status.txt records one line per command with its result. ok means normal, missing means the command was skipped because the host does not have it, and exit <code> means the command ended with an exit code other than 0. lspci is in the pciutils package, and hyperaccel-smi ends with exit 2 if no card is visible. A sysfs node that cannot be read is recorded as an error message in its place in sysfs-haN.txt. In every case the script does not stop and collects the rest. For the meaning of the IOMMU type value, see IOMMU mode and the device-to-device path.

Nodes that are not read

The script specifies the nodes to read by name, and it does not read the following two nodes.

Node Why it is not read
fw_update_progress Every read fetches the value directly from the device. It must not be read on a card that does not respond
help It is fixed description text built into the driver and contains no card state

bypass_range_check, a debugging setting, is not telemetry, but when its value is 1 the driver does not check the range of device memory access requests, so it is needed for diagnosis and is collected as well (bypass_range_check). The remaining nodes report the state the driver keeps in host memory and the values the poller has collected, so reading them does not access the device and they respond even while the card is stalled. The exceptions are fw_version and serial. While fw_version is unknown, reading these two nodes checks again whether the firmware is ready; the conditions under which the driver skips this check are described in Monitoring suspension. Because the nodes are specified by name, a node that has not been verified is not read even if a new driver version adds nodes.

root privileges

Even without root, the script runs to the end, and the sysfs nodes, module parameters, hyperaccel-smi, and lspci -tv are collected as usual. The following three items are empty or only partly collected without root privileges. In that case the script prints a line starting with not root: at startup.

  • dmesg.txt: If the kernel setting kernel.dmesg_restrict is 1, a user who is not root (more precisely, who lacks the CAP_SYSLOG capability) cannot read the kernel log buffer. Check the current value with sysctl kernel.dmesg_restrict.
  • journal.txt: Without root, the system journal cannot be read, so the kernel log may be missing.
  • lspci-vvv.txt: Without root, only the first 64 bytes of the PCI configuration space can be read, so Capabilities: <access denied> appears instead of the link status and error status. Inside a container, the same can happen even as root.

Reading the archive

These are the things to check yourself before sending the archive to the support team. The meaning of each value is on the linked page. A per-card file starts as follows.

1
2
3
4
5
6
# ha1 0000:65:00.0
/sys/class/ha/ha1/health:faulted
/sys/class/ha/ha1/devcmd_timeouts:1
/sys/class/ha/ha1/devcmd_fw_errors:0
/sys/class/ha/ha1/hdma_timeouts:0
/sys/class/ha/ha1/last_hang:rc=- cmd_idx=- opcode=- batch_seq=1041->1042 exec_index=0 resq_tail=0 verdict=stuck-in-command
  1. Card state: If health is faulted, the device is FATAL or the last reset has failed. See health verdict.
  2. Error counters: For devcmd_timeouts, devcmd_fw_errors, and hdma_timeouts, look at whether the value has increased since the last time you saw it, rather than at the value itself. See RAS counters.
  3. last_hang: verdict is the verdict on where in the batch the firmware stopped. The first three fields and the last four fields can be records of different events. See last_hang and Batch progress verdict.
  4. Kernel log: Look for the driver's hang records in dmesg.txt (command below). The hang (…) line is a human-readable summary, and the hang dump (…) line holds the raw values. See Hang dump and Kernel log messages.
  5. Missing items: Check whether status.txt contains missing or exit. If the kernel log is missing, run the script again with root privileges.
1
grep -E ' ha [0-9a-f]+:' dmesg.txt | grep -E 'watchdog|Devcmd|hang|Firmware error|sys_mon|auto-TDR|device reset failed|fw_update'

Telling slow from stalled

If Devcmd waiting keeps appearing every 10 seconds in the kernel log, an application is waiting without a ceiling. Run the script twice, at least 20 seconds apart, and compare last_hang in the two archives. If batch_seq or exec_index has changed, the device is making progress. See Waits without a ceiling.


What to check before sending

The archive contains as much host information as diagnosis requires.

  • It contains the host name, the kernel and distribution versions (host.txt), the list of loaded modules, and the PCI device list and configuration.
  • The kernel log (dmesg.txt, journal.txt) also contains all messages from devices and services unrelated to this driver. In a hang record, opened by pid <PID> (<name>) shows the name of the process that opened the card.
  • The directory and archive the script creates are readable only by the user who ran the script. If you ran it with sudo, root privileges are also needed to read or copy them.

Review the contents before sending. If your organization's policy requires masking some values, do not delete lines from the files in the directory; replace only those values and then re-create the archive (sudo sh -c 'umask 077; cd /var/tmp && tar -czf ha-diag-<time>.tar.gz ha-diag-<time>'). The archive the script first created still contains the values from before the replacement. The cause is sometimes on a line preceding the driver messages (for example, an error from an upstream PCIe port), so picking out individual lines makes diagnosis harder. What to provide along with the archive (when the symptom started, the jobs that were running at the time, and reproduction steps) is described in Requesting support.


See also