Skip to content

Troubleshooting

This page collects symptoms you may encounter while installing and operating the driver. Find the entry that matches your symptom to see the cause and resolution, and see the related documents at the end of the entry for more detail. Individual kernel log messages are described in Kernel log messages. For a problem that is not covered on this page, review Information to collect first and Collecting diagnostic information for a problem, and then contact the HyperAccel support team.


Information to collect first

Resets and module reloads clear the state that explains the earlier failure. Save the diagnostic information before you take any action. If the card is in a hang or FATAL state, create an archive with the script in Collecting diagnostic information for a problem. For installation or load problems, or when you cannot use the script, save the information into a single file with the commands below. sysfs telemetry is read without opening the device node and still responds while the card is stalled.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# Run as root (for example: sudo -i)
{
  date
  uname -a
  cat /sys/module/ha/version
  grep -H . /sys/module/ha/parameters/*
  dkms status 2>/dev/null
  lspci -D -tv
  lspci -D -nn -vvv -d 20fb:
  ls -l /dev/ha*
  for d in /sys/class/ha/ha*; do
    echo "== $(basename "$d") $(basename "$(readlink -f "$d/device")")"
    for n in fw_version serial health last_hang inflight stream_count devcmd_timeouts devcmd_fw_errors hdma_timeouts peer_map peer_route peer_sram; do
      echo "$n: $(cat "$d/$n")"
    done
  done
  hyperaccel-smi -q
  dmesg -T
} > ha-diag.txt 2>&1
  • If you have already rebooted the host, also save the kernel log of the previous boot with journalctl -k -b -1. It remains only if the systemd journal is stored persistently.
  • The meaning of each node is described in sysfs nodes.

Installation and loading

/dev/haN is missing

The driver did not probe the card. Use the three commands below to narrow down where it stopped.

1
2
3
4
5
6
7
8
# Is the module loaded?
lsmod | grep -w ha

# Is the card visible on the PCI bus, and which driver has claimed it?
lspci -D -nn -k -d 20fb:

# Where did probe stop?
sudo dmesg -T | grep -E '(^|\] )ha[ :]'

If everything is normal, Probe successful: /dev/ha<N> created is logged for each card. The criteria for a normal state are in Verifying the installation.

The module is not loaded

Cause: ha is not in the lsmod output. The module was not loaded automatically at boot, or loading failed.

Resolution: Load it with sudo modprobe ha. If that fails, use the message to narrow down the cause.

modprobe message Cause Resolution
Module ha not found in directory /lib/modules/<kernel> The module for the running kernel is not installed See There is no module for the running kernel
could not insert 'ha': Key was rejected by service Secure Boot is enabled and this host does not trust the signing key of the module dkms signs the module with its own key, so enroll that key once. On Ubuntu, run sudo mokutil --import /var/lib/shim-signed/mok/MOK.der; on other distributions, run sudo mokutil --import /var/lib/dkms/mok.pub; then reboot and complete the enrollment on the MOK management screen
could not insert 'ha': Invalid module format The module was built for a different kernel Install the installation package that matches the running kernel

The installation package creates /etc/modules-load.d/ha.conf so that the module is loaded at every boot. If this file does not exist, reinstall the driver with the installation package.

Related documents: Installation

The card is not visible

Cause:

  • If the card is not in the output of lspci -d 20fb:, the operating system failed to enumerate the card. This is a problem with seating, power, or the slot settings in the BIOS.
  • If the card appears as (rev ff), the card does not respond to PCI configuration space reads.
  • If Kernel driver in use: is not ha, another driver (for example, vfio-pci) has claimed the card.

Resolution:

  • Check the card seating, the power, and the BIOS settings.
  • If it is (rev ff), power-cycle the host.
  • If another driver has claimed the card, unbind it from that driver and then bind it to ha.
    1
    2
    echo <BDF> | sudo tee /sys/bus/pci/drivers/<driver>/unbind
    echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind
    

Related documents: Supported environments

The probe fails

Cause: dmesg contains Probing device but no Probe successful for the same <BDF>. The common cases are as follows.

  • ABI protocol mismatch: firmware v<n>, driver requires >= v<n>; refusing probe: The firmware of the card is older than this driver requires.
  • Failed to request <name> BAR (hw_index=<n>) or BAR<n> length is 0: The BIOS did not assign an address to the BAR of the card.
  • firmware booted but did not publish the BAR2 inbound base … or firmware has not booted, so the BAR2 inbound base is unknown …: The firmware did not report the card-side BAR2 address or has not booted, so the driver did not bind the card.
  • Failed to setup MSI-X: The interrupts of the card could not be set up.
  • No driver message and no kernel probe of <BDF> failed line after Probing device: There are more than 8 cards. The driver handles up to 8 cards on one host.

Resolution: Follow the action for each message in Errors that stop the probe. A card with a mismatched firmware version is not registered with the driver and cannot be fixed with ha-fw-update, so contact the support team.

Related documents: Kernel log messages, Supported environments

There is no module for the running kernel

Cause: When a new kernel is installed, DKMS rebuilds the module, but it cannot build it if the headers of that kernel are missing.

Resolution: Install the headers of the running kernel, and then build the module for the running kernel with sudo dkms autoinstall. If the build fails, see The DKMS build fails.

Related documents: Upgrading and removal

The page size is not 4 KiB

Cause: The driver supports only 4 KiB page kernels. On ARM64 kernels with 16K or 64K pages, DKMS does not build this module.

Resolution: Check that getconf PAGESIZE returns 4096, and if it does not, boot a 4 KiB page kernel.

Related documents: Supported environments

The DKMS build fails

Cause and resolution: Narrow it down from the messages that the installation left.

Message Cause Resolution
hyperdriver-dkms: kernel headers for <kernel> not found. The headers of the running kernel are missing Install the headers with sudo apt install linux-headers-$(uname -r), and then reinstall the package
ha/<version> failed to stage or build for kernel <kernel> The module build failed on this kernel. The driver that is already installed is unchanged Check the build log (make.log) that the message points to. A compile error related to the kernel API means that the kernel is outside the supported range
hyperdriver-dkms: no headers for installed kernel <kernel> - ha.ko will be MISSING if you boot it. The headers are missing for a kernel that is installed but has not been booted yet. If you boot that kernel, the module is missing Install the headers of that kernel and build with sudo dkms autoinstall -k <kernel>

On kernels with 16K or 64K pages, the build itself is skipped. See The page size is not 4 KiB.

Related documents: Installation, Supported environments

The module cannot be unloaded

Cause: rmmod: ERROR: Module ha is in use or modprobe: FATAL: Module ha is in use. appears. A process has /dev/haN open or has device memory mapped. Processes inside containers are included.

Resolution: Find the processes that have the node open, terminate them, and then unload the module again.

1
2
3
4
5
sudo lsof /dev/ha*

# If lsof is not available
sudo find /proc/[0-9]*/fd -lname '/dev/ha*' 2>/dev/null
sudo grep -l '/dev/ha' /proc/[0-9]*/maps 2>/dev/null
  • Do not force the module to unload.
  • Unloading the module takes down all cards together. To restart only one card, use ha-hotreset.
  • If a firmware update is in progress, wait until it finishes.
  • If cat /sys/module/ha/version does not show the new version after an installation or upgrade, the old module could not be unloaded because the cards are in use. In this case, the deb package leaves the message hyperdriver-dkms: WARNING - installed <new version>, but the kernel is still running <previous version>.. Stop the jobs, and then run sudo modprobe -r ha && sudo modprobe ha or reboot. On distributions that build the initial RAM disk with dracut, such as RHEL-family distributions and Fedora, if the old version is still running even after a reboot, the initial RAM disk may contain the old ha.ko. Run sudo dracut -f --regenerate-all and reboot again (Checking the running version).

Related documents: Upgrading and removal, ha-hotreset


Device nodes

The node exists but cannot be opened

Open the node as the user that will run the workload and check the error.

1
2
ls -l /dev/ha*
python3 -c "import os; os.close(os.open('/dev/ha0', os.O_RDWR)); print('ok')"
Error Cause Resolution
Permission denied The udev rule (99-ha.rules) has not been applied so only root can open the node, or the user does not belong to the group specified by a rule that was changed to be group-based. The default rule makes the node readable and writable by everyone (crw-rw-rw-) Check that the rule file exists (/lib/udev/rules.d/99-ha.rules, or /etc/udev/rules.d/99-ha.rules if you copied and edited it) and reapply it with sudo udevadm control --reload-rules && sudo udevadm trigger. If you changed the group, add the user to the group and log in again
No such device The driver is still preparing the card (right after loading or right after a hot reset) or is removing the card Open it again after a few seconds. If it persists, check dmesg for a probe failure or a removal record
No such device or address The device number that the node points to differs from the one the driver currently has. This can happen if the module was reloaded or the card was probed again after the node was created, and it is mostly seen on nodes inside containers Restart the container to create the node again

Related documents: Container support, Installation


Status and telemetry

health is faulted

Cause:

  • The device is FATAL. The firmware did not respond within the response wait ceiling or reported a fatal error, so the driver no longer assigns jobs to that card. FATAL is a state of the card, so restarting the application does not clear it.
  • The last reset (an SDK reset or the automatic reset) failed. device reset failed (...) is logged in dmesg.

Resolution:

  1. Save the information described in Information to collect first. In particular, save last_hang and the hang dump in dmesg.
  2. Stop the applications that use that card. Leave the applications that use other cards running.
  3. Reset the card. Use one of the following two methods. - The application resets the card with the reset function of the SDK. - After closing all handles of that card, hot reset it on the host with sudo ha-hotreset --bdf <BDF>. The card is probed again, so the error counters also become 0.
  4. Check whether the reset succeeded from the result received by the party that requested the reset and from the reset messages in dmesg. Do not judge from health alone. After a reset with the SDK, health leaves faulted, but it is normal for it to show degraded because the error counters from before the reset remain. Confirm a hot reset by Hot reset OK: <BDF> and by Probe successful for the same PCI address.
  5. If a reset failed, the device reset failed (...) line it left last tells you what to do next (run ha-hotreset, or power-cycle the host).

To follow along with resetting a card with ha-hotreset, see Recovering an unresponsive card. To have the driver reset the device by itself every time it becomes FATAL, set ha_auto_reset=1 as described in Enabling automatic recovery.

Related documents: FATAL and recovery, Hang detection and the watchdog, Parameter reference

health is degraded

Cause: At least one error or timeout has occurred since the driver probed the card. One of devcmd_timeouts, devcmd_fw_errors, and hdma_timeouts is not 0. These counters never decrease, so degraded does not mean that there is still a problem now. Errors in which only that command fails and the card can still be used, such as a command that the firmware rejected because of an invalid argument sent by the application, are also counted in devcmd_fw_errors.

Resolution:

  1. Check whether the counters are still increasing.

    1
    watch -n 5 grep -H . /sys/class/ha/ha0/devcmd_timeouts /sys/class/ha/ha0/devcmd_fw_errors /sys/class/ha/ha0/hdma_timeouts
    
  2. If they have stopped, it is a past event. Check what happened and when with last_hang and dmesg.

  3. If they are increasing, the problem is ongoing. Find the message that corresponds to the increasing counter in Kernel log messages.

degraded returns to ok when the card is probed again (hot reset, module reload, or reboot). It does not return to ok after a reset with the SDK.

Related documents: Telemetry, sysfs nodes

Telemetry is unknown

Cause: unknown means that the driver cannot trust the value. The driver does not substitute the previous value or 0. Check in the following order.

What to check Cause Resolution
health is faulted If the device is FATAL, polling stops, and the temperature, power, and utilization are unknown from 5 seconds later They return when you reset the card. See health is faulted
dmesg contains sys_mon: no response for … suspending telemetry polling The card did not respond, so polling stopped Resetting the card restarts polling
The card is being reset The card is not read during a reset They return when the reset finishes
fw_version is also unknown The firmware is not ready See fw_version is unknown
dmesg contains sys_mon telemetry unavailable The telemetry poller could not be started at probe Hot reset the card or reload the module
Firmware version The firmware is old and does not provide telemetry Check the firmware version in Supported environments
Only util is unknown The firmware has not finished the first measurement yet, or the utilization measurement is not progressing Read again after a moment. If it continues, notify the support team
Only power_mw is unknown and rails is none No power rail is measured —

If there are no telemetry nodes at all under /sys/class/ha/haN/ and dmesg contains telemetry sysfs unavailable, the nodes could not be created. Report it to the support team together with the kernel version.

Related documents: Telemetry, sysfs nodes

fw_version is unknown

Cause: The driver could not confirm that the firmware is ready or could not read the firmware version. Look for the following messages in dmesg.

Message Cause
firmware boot sentinel not seen within <ms> ms The firmware has not booted or is still booting
firmware never reported ready The firmware did not report that it is ready. It is still booting, or its self-test failed during boot
firmware version CSR reads 0x<value> The version was read but the value was not valid.
device came back but firmware did not report ready The firmware was not ready after a reset
sys_mon: no response for … The card did not respond, so polling stopped. For a card whose monitoring channel has been opened at least once, the version is not read again until it is reset. For a card that was still booting, the version is read again when fw_version is read after the firmware becomes ready

Resolution:

  1. If this is right after loading or right after a reset, wait a few seconds and read again. If the firmware becomes ready late, the driver reads the version by itself and logs firmware booted after probe gave up; CSR-derived telemetry refreshed.
  2. If it stays unknown, close all handles of that card and restart the card with sudo ha-hotreset --bdf <BDF>. If the configuration does not allow an SBR to be asserted, power-cycle the host.
  3. If it is still unknown, the firmware of the card cannot boot. Contact the support team.

While fw_version is unknown, ha-fw-update does not update that card and ends with exit code 2.

Related documents: Kernel log messages, FATAL and recovery

hyperaccel-smi does not show the card

Cause: The default output shows No HyperAccel LPU found under <path> and the exit code is 2. Even when run with -q or --json, the exit code is 2 if no card is visible.

  • The driver has not probed the card, so /sys/class/ha/ is empty.
  • The environment variable HYPERACCEL_VISIBLE_DEVICES restricts the visible cards. If the value is none or empty, no card is visible, and if it is a comma-separated list of numbers, only the cards with those numbers are visible.
  • A number that does not exist was given to -i.
  • Inside a container, the /sys/class/ha of the host is not visible or is at a different path.

Resolution:

1
2
ls /sys/class/ha/
env | grep HYPERACCEL
  • If /sys/class/ha/ is empty, follow /dev/haN is missing.
  • Unset HYPERACCEL_VISIBLE_DEVICES, set it to all, or correct it to the right numbers.
  • If sysfs is mounted at a different path (for example, /host/sys inside a container), tell the tool the class/ha directory under it with --root /host/sys/class/ha or the environment variable HYPERACCEL_SYSFS_ROOT. If you give the path where sysfs is mounted itself, such as /host/sys, the tool does not find the cards.
  • hyperaccel-smi runs only if python3 is available.

Related documents: hyperaccel-smi


Hang and recovery

A hang dump appears in dmesg

Cause: Lines like the following were logged.

1
2
3
ha <BDF>: Devcmd no response after <ms> ms (device hang): <verdict>
ha <BDF>: hang (<reason>): fw <command> (op 0x<op>), batch slot <n> (last staged, not necessarily the stuck one) … -- <verdict sentence>
ha <BDF>: hang dump (<reason>): stream=0x<tag> corr=<n> opcode=0x<op> cmd_idx=<n> payload=[<word>] …

The first line that is logged before the dump depends on <reason>. If it is ceiling timeout, the Devcmd no response after <ms> ms (device hang): <verdict> line above is logged; if it is fw_update timeout, fw_update: no response after <ms> ms: <verdict> is logged; if it is watchdog, watchdog: devcmd wait stalled (devcmd=<ms> ms hdma=<ms> ms) - no completion within the threshold is logged.

The firmware did not respond to a command within the response wait ceiling (default 12 seconds; correspondingly longer for commands that take a long time). If the reason is watchdog, a request that waits without a ceiling exceeded the watchdog threshold (default 8 seconds; with ha_wd_preempt=1, other requests are also subject to it), and if it is fw_update timeout, a firmware update exceeded its deadline.

Resolution:

  1. Save the two dump lines and the surrounding log as they are. See Information to collect first.
  2. If the reason is fw_update timeout, do not reset the card; follow Resets and jobs are refused after an update.
  3. Check health. If it is faulted, follow health is faulted.
  4. If the state is not faulted and the reason is watchdog: with ha_wd_preempt=1, the watchdog has already cut off that wait and the request failed. If the device appears to have stalled, reset the card. With the default (ha_wd_preempt=0), the request that waits without a ceiling is still waiting. Check whether Devcmd waiting … continues to appear in dmesg every 10 seconds, and read last_hang twice at an interval of at least 20 seconds to see whether batch_seq or exec_index changes. During this wait, last_hang is updated only when a Devcmd waiting line is logged, so if you read it at shorter intervals, you may get the same value even though the command is progressing. If it changes, the command is progressing slowly; if it stays the same, a single command is taking a long time or has stalled. If, comparing with how long that command normally takes, you judge that it has stalled, terminate that application and reset the card.
  5. If the same symptom recurs, send it to the support team together with the dump. The meaning of each field of the dump and of the verdicts is in Reading the hang dump.

Related documents: Hang detection and the watchdog, FATAL and recovery

DMA transfer errors appear in dmesg

Cause: The following lines were logged by the DMA engines that handle copies between the host and the card, or between cards.

1
2
3
4
ha <BDF>: ll: timeout (n=<n> channel=<ch> <direction>)
ha <BDF>: ll: engine stopped but its completion <mode> never arrived (status=<n> channel=<ch> <direction>)
ha <BDF>: ll: no response (channel=<ch> <direction>)
ha <BDF>: HDMA engine at 0x<offset> <state>; quarantining its pages until the engines stop (reset or remove)
  • ll: timeout means that one transfer did not finish within 5 seconds. hdma_timeouts increases. If the engine can be stopped, only the work that the transfer belongs to fails and the card does not become FATAL.
  • ll: engine stopped but its completion … never arrived means that the engine had already stopped, but its completion signal did not reach the driver within 5 seconds. hdma_timeouts increases and only the work that the transfer belongs to fails.
  • ll: no response and ring write: no response mean that the card does not respond to the status register of the DMA engine. The link may be down, or the card may have stopped or been removed, and the card becomes FATAL.
  • HDMA engine at … means that the engine could not be stopped after a timeout or abort. The engine may still be able to write to host memory, so the driver quarantines those host pages and blocks new transfers, and the card becomes FATAL.

Resolution:

  1. Save the logs before and after the lines, and the sysfs values. See Information to collect first.
  2. Check health. If it is faulted, follow health is faulted. When you reset the card, the engines stop, the quarantined pages are released, and transfers are accepted again.
  3. If every copy in a VM guest logs ll: engine stopped but its completion MSI-X never arrived, the driver mistook the VM for bare metal. Reload the driver inside the guest with ha_hdma_completion=poll.
  4. If only ll: timeout, ll: engine stopped …, or ll: n=… error=… was logged and health is not faulted, run the failed work again.
  5. If it repeats on the same card, check the PCIe error messages from the same time and the link speed and width (current_link_speed, current_link_width), and send the diagnostic information to the support team. If the other cards on the same server are fine and only this card repeats it even when moved to another slot, the problem is likely on the card side.

Related documents: Kernel log messages, FATAL and recovery

The application receives an error that the device cannot be recovered

Cause: The device is FATAL. FATAL is a state of the card, so it is not cleared by restarting the application or reopening the node. The sysfs health is faulted.

Resolution:

  1. Reset the card by the procedure in health is faulted.
  2. A reset invalidates all open handles of that card, so the other applications that were using that card must also reopen the node. If several applications share one card, decide that only one of them performs the reset.
  3. If the reset request returns a failure, or the same error occurs again after the reset, check the reset messages in dmesg to see which step failed. See Reset progress and results.

Applications that use only other cards are not affected. However, jobs on other cards that exchanged data with this card can fail (Multiple devices).

Related documents: FATAL and recovery

ha-hotreset refuses the reset

Cause: If the pre-flight check fails, the tool touches nothing and ends with Pre-flight failed - nothing was touched. The line directly above it gives the reason. The exit code is 3 for an open handle and 4 for a configuration in which an SBR cannot be issued. Path masking can also fail right before the reset; the tool then ends with Nothing was touched.

Output Cause Resolution
<BDF>: /dev/ha<N> still open by PID(s) <pid> A process has the node of that card open Terminate that process and run the tool again. With --force, the tool proceeds even if handles are open, but an application that holds a handle loses the card and must reopen the node. Use --force only in an emergency, and do not use it on a card whose firmware is being updated
<BDF>: parent is a <port type> whose Surprise Down mask cannot be read (no AER capability) - refusing SBR, <BDF>: parent has no PCIe capability - refusing SBR Whether the parent port masks Surprise Down errors cannot be determined. In this configuration, dropping the link with an SBR can cause the platform to reset the entire host. This cannot be bypassed even with --force Power-cycle the host to restart the card
<BDF>: cannot read <register> on <port> - the path cannot be masked, refusing SBR, <BDF>: refusing to SBR without a verified mask on the whole path The parent port does not mask Surprise Down errors, and the ports on the path cannot be masked. This cannot be bypassed even with --force Power-cycle the host to restart the card
<BDF>: parent is a <port type> with Surprise Down Error unmasked - refusing SBR --no-switch-link-drop was given, and the parent port does not mask Surprise Down errors Run without --no-switch-link-drop to reset with the path masked, or power-cycle the host
another ha-hotreset is running on this host - waiting up to 600s Another ha-hotreset is running, so the tool waits. If it does not finish within 600 seconds, the tool prints the other ha-hotreset did not finish in 600s - nothing was touched and ends with exit code 3 Run it again after the other run ends
<BDF>: parent bridge <bridge> has more than one device below it: There are other devices under the same parent bridge, so the SBR also resets those devices The card cannot be reset on its own. Power-cycle the host

Other cases are as follows.

  • Run as root: Run with sudo.
  • <n> cards present - pass --bdf or --all:: If there are multiple cards, specify one card with --bdf or use --all. The exit code is 2.
  • --bdf '<value>' is ambiguous: The same bus address exists in several PCI domains. Specify the full address including the domain (in the 0000:01:00.0 format). The exit code is 2.
  • setpci not found (install pciutils): Install the pciutils package. For flock not found (install util-linux), install the util-linux package.
  • If <BDF>: /dev/ha<N> did not reappear - check dmesg for probe failure appears after the reset, follow /dev/haN is missing.
  • the mask is LEFT IN PLACE on purpose: The reset card did not respond again, so the tool deliberately left the path masking in place. Real link errors on that path are not reported either, so after you deal with the card problem, restore it with the setpci commands printed right below.

Running sudo ha-hotreset --list lets you see in advance, for each card, whether the configuration allows an SBR to be asserted from the parent port. Cards for which it cannot be asserted are marked NOT resettable, and cards that are reset with the path masked are marked the path is masked for the reset. Whether other devices exist under the same parent bridge, and whether the registers on the path can be read, are not shown in this list; they are checked in the pre-flight check at reset time.

Related documents: ha-hotreset, FATAL and recovery


Firmware update

ha-fw-update fails

Cause and resolution: Narrow it down from the exit code and the last ERROR: line.

Exit code Meaning and common messages Flash What to do
1 A usage or environment problem. A -d address format error, the image file cannot be read (cannot read image), not root (root required), no reset script (no reset script), declined at the confirmation prompt (aborted) Unchanged Fix it as the message says and run the tool again
2 The card cannot be used or does not support updates. No card (no HyperAccel card at), in use by another process (is open by pid(s)), an update is already running on the same card (another ha-fw-update is running), fw_version is unknown (reports firmware 'unknown'), the firmware is not ready (device not ready), the driver is waiting for the response to an earlier update (upload failed: Device or resource busy), the firmware aborted before writing (firmware aborted before writing), the driver or firmware does not support updates, a command-line syntax error (an unknown option, -d and -i given together or neither given, and so on) Unchanged Follow the message. If another process is using the card, run the tool again after it finishes. If the driver is waiting for the response to an earlier update, follow Resets and jobs are refused after an update. If the firmware aborted before writing, reset the card and then run the tool again. If the driver does not support updates, update the driver first; if the firmware does not support them, contact the support team
3 The image was rejected. The image check of the tool failed, the firmware rejected the image (firmware rejected the image), the partition layout in the flash differs from the image (fip partition layout differs from the one in flash -- retry with --scope full) Unchanged If the message says the layout differs, run the tool again with --scope full. Otherwise, check the image file
4 The write failed or the state is unknown (FLASH WRITE FAILED, firmware did not answer within the budget, device went FATAL or stopped answering) May be partially written Do not power off or reboot the host; run the tool again with the same image. If no response was received, also see Resets and jobs are refused after an update
5 The flash was updated but the card could not be reset (flash updated but the reset failed) New image Fix the cause using the ha-hotreset messages printed along with it, and then run sudo ha-hotreset --bdf <BDF> or power-cycle the host. See ha-hotreset refuses the reset
6 After the reset, the card did not return within --boot-timeout (default 60 seconds), or the tool stopped at the verification step New image Check dmesg and fw_version. If the card does not boot, contact the support team
  • With --json, the result is output as JSON together with exit_code and error.
  • For exit code 4, do not reset the card; run the update again. Unless you restart the card, it does not boot from a partially written flash. If the tool ended because no response was received from the firmware, run it again after the firmware responds. See Resets and jobs are refused after an update.

When the tool mentions a debug tool

The tool may also state that a debug tool (e.g., Trace32) may be needed for recovery. This is a JTAG debugger, and the message means that the card is in a state that cannot be recovered with host software. If you see such a message, contact the support team.

Related documents: Firmware update, ha-fw-update

Resets and jobs are refused after an update

Cause: After ha-fw-update ends with exit code 4, the following line is logged in dmesg.

1
ha <BDF>: fw_update: no answer from firmware (<err>); resets stay refused until it answers or the update is retried

The driver sent the update command but could not confirm the response of the firmware. The firmware may still be writing the flash, so to avoid booting from a partially written flash, the driver refuses resets and application jobs for that card until the response is confirmed. A reset request logs reset refused: last fw_update unanswered …, and an automatic reset logs auto-TDR: skipped, …. The step in which ha-fw-update loads the image into the card memory is also refused, so running it again in this state prints upload failed: Device or resource busy and ends with exit code 2. This run does not change the flash.

Resolution:

  1. Do not power off or reboot the host, and do not reset the card. This includes ha-hotreset and resets by applications.
  2. If the firmware responds, even late, the refusal is lifted and fw_update: late completion from firmware; resets re-enabled is logged. This line does not contain the result of the write, so after you see it, run ha-fw-update again with the same image to complete the reset and the verification.
  3. If the firmware never responds, contact the support team. A module reload, ha-hotreset, or a power cycle risks the card booting from a flash whose write has not finished, so use them only after consulting the support team.

Related documents: Firmware update, Kernel log messages

The version is unchanged after the update

Cause:

  • It was run with --no-reset. Only the flash changed, and the old firmware keeps running until you hot reset the card.
  • An application reset the card with the SDK. This reset usually only reinitializes the running firmware and does not load the new image from the flash.
  • It succeeded with a same version warning. This is the case where an image with the same version number was written.

Resolution: Close all handles of that card, hot reset the card, and then check the version.

1
2
sudo ha-hotreset --bdf <BDF>
for d in /sys/class/ha/ha*; do echo "$(basename "$d") $(basename "$(readlink -f "$d/device")") $(cat "$d/fw_version")"; done

The card normally reappears with the same number, but to be sure, find it by PCI address.

Related documents: Firmware update, ha-hotreset


Requesting support

Contact the HyperAccel support team and attach the following.

  1. The archive created with Collecting diagnostic information for a problem, or the file created with Information to collect first
  2. The dmesg saved before a reset, a hot reset, or a module reload
  3. The driver version (/sys/module/ha/version) and the firmware version (fw_version)
  4. The time the symptom started and the jobs that were running then
  5. The reproduction steps, and whether the problem still reproduces after the card is reset or hot reset
  6. If the support team requests detailed logs, the dmesg captured while reproducing the problem with detailed logging turned on by following Enabling detailed logs

See also