Troubleshooting¶
This page collects symptoms you may encounter while installing and operating the driver. Find the entry that matches your symptom to see the cause and resolution, and see the related documents at the end of the entry for more detail. Individual kernel log messages are described in Kernel log messages. For a problem that is not covered on this page, review Information to collect first and Collecting diagnostic information for a problem, and then contact the HyperAccel support team.
Information to collect first¶
Resets and module reloads clear the state that explains the earlier failure. Save the diagnostic information before you take any action. If the card is in a hang or FATAL state, create an archive with the script in Collecting diagnostic information for a problem. For installation or load problems, or when you cannot use the script, save the information into a single file with the commands below. sysfs telemetry is read without opening the device node and still responds while the card is stalled.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | |
- If you have already rebooted the host, also save the kernel log of the previous boot with
journalctl -k -b -1. It remains only if the systemd journal is stored persistently. - The meaning of each node is described in sysfs nodes.
Installation and loading¶
/dev/haN is missing¶
The driver did not probe the card. Use the three commands below to narrow down where it stopped.
1 2 3 4 5 6 7 8 | |
If everything is normal, Probe successful: /dev/ha<N> created is logged for each card. The criteria for a normal state are in Verifying the installation.
The module is not loaded¶
Cause: ha is not in the lsmod output. The module was not loaded automatically at boot, or loading failed.
Resolution: Load it with sudo modprobe ha. If that fails, use the message to narrow down the cause.
modprobe message |
Cause | Resolution |
|---|---|---|
Module ha not found in directory /lib/modules/<kernel> |
The module for the running kernel is not installed | See There is no module for the running kernel |
could not insert 'ha': Key was rejected by service |
Secure Boot is enabled and this host does not trust the signing key of the module | dkms signs the module with its own key, so enroll that key once. On Ubuntu, run sudo mokutil --import /var/lib/shim-signed/mok/MOK.der; on other distributions, run sudo mokutil --import /var/lib/dkms/mok.pub; then reboot and complete the enrollment on the MOK management screen |
could not insert 'ha': Invalid module format |
The module was built for a different kernel | Install the installation package that matches the running kernel |
The installation package creates /etc/modules-load.d/ha.conf so that the module is loaded at every boot. If this file does not exist, reinstall the driver with the installation package.
Related documents: Installation
The card is not visible¶
Cause:
- If the card is not in the output of
lspci -d 20fb:, the operating system failed to enumerate the card. This is a problem with seating, power, or the slot settings in the BIOS. - If the card appears as
(rev ff), the card does not respond to PCI configuration space reads. - If
Kernel driver in use:is notha, another driver (for example,vfio-pci) has claimed the card.
Resolution:
- Check the card seating, the power, and the BIOS settings.
- If it is
(rev ff), power-cycle the host. - If another driver has claimed the card, unbind it from that driver and then bind it to
ha.1 2
echo <BDF> | sudo tee /sys/bus/pci/drivers/<driver>/unbind echo <BDF> | sudo tee /sys/bus/pci/drivers/ha/bind
Related documents: Supported environments
The probe fails¶
Cause: dmesg contains Probing device but no Probe successful for the same <BDF>. The common cases are as follows.
ABI protocol mismatch: firmware v<n>, driver requires >= v<n>; refusing probe: The firmware of the card is older than this driver requires.Failed to request <name> BAR (hw_index=<n>)orBAR<n> length is 0: The BIOS did not assign an address to the BAR of the card.firmware booted but did not publish the BAR2 inbound base …orfirmware has not booted, so the BAR2 inbound base is unknown …: The firmware did not report the card-side BAR2 address or has not booted, so the driver did not bind the card.Failed to setup MSI-X: The interrupts of the card could not be set up.- No driver message and no kernel
probe of <BDF> failedline afterProbing device: There are more than 8 cards. The driver handles up to 8 cards on one host.
Resolution: Follow the action for each message in Errors that stop the probe. A card with a mismatched firmware version is not registered with the driver and cannot be fixed with ha-fw-update, so contact the support team.
Related documents: Kernel log messages, Supported environments
There is no module for the running kernel¶
Cause: When a new kernel is installed, DKMS rebuilds the module, but it cannot build it if the headers of that kernel are missing.
Resolution: Install the headers of the running kernel, and then build the module for the running kernel with sudo dkms autoinstall. If the build fails, see The DKMS build fails.
Related documents: Upgrading and removal
The page size is not 4 KiB¶
Cause: The driver supports only 4 KiB page kernels. On ARM64 kernels with 16K or 64K pages, DKMS does not build this module.
Resolution: Check that getconf PAGESIZE returns 4096, and if it does not, boot a 4 KiB page kernel.
Related documents: Supported environments
The DKMS build fails¶
Cause and resolution: Narrow it down from the messages that the installation left.
| Message | Cause | Resolution |
|---|---|---|
hyperdriver-dkms: kernel headers for <kernel> not found. |
The headers of the running kernel are missing | Install the headers with sudo apt install linux-headers-$(uname -r), and then reinstall the package |
ha/<version> failed to stage or build for kernel <kernel> |
The module build failed on this kernel. The driver that is already installed is unchanged | Check the build log (make.log) that the message points to. A compile error related to the kernel API means that the kernel is outside the supported range |
hyperdriver-dkms: no headers for installed kernel <kernel> - ha.ko will be MISSING if you boot it. |
The headers are missing for a kernel that is installed but has not been booted yet. If you boot that kernel, the module is missing | Install the headers of that kernel and build with sudo dkms autoinstall -k <kernel> |
On kernels with 16K or 64K pages, the build itself is skipped. See The page size is not 4 KiB.
Related documents: Installation, Supported environments
The module cannot be unloaded¶
Cause: rmmod: ERROR: Module ha is in use or modprobe: FATAL: Module ha is in use. appears. A process has /dev/haN open or has device memory mapped. Processes inside containers are included.
Resolution: Find the processes that have the node open, terminate them, and then unload the module again.
1 2 3 4 5 | |
- Do not force the module to unload.
- Unloading the module takes down all cards together. To restart only one card, use
ha-hotreset. - If a firmware update is in progress, wait until it finishes.
- If
cat /sys/module/ha/versiondoes not show the new version after an installation or upgrade, the old module could not be unloaded because the cards are in use. In this case, the deb package leaves the messagehyperdriver-dkms: WARNING - installed <new version>, but the kernel is still running <previous version>.. Stop the jobs, and then runsudo modprobe -r ha && sudo modprobe haor reboot. On distributions that build the initial RAM disk with dracut, such as RHEL-family distributions and Fedora, if the old version is still running even after a reboot, the initial RAM disk may contain the oldha.ko. Runsudo dracut -f --regenerate-alland reboot again (Checking the running version).
Related documents: Upgrading and removal, ha-hotreset
Device nodes¶
The node exists but cannot be opened¶
Open the node as the user that will run the workload and check the error.
1 2 | |
| Error | Cause | Resolution |
|---|---|---|
Permission denied |
The udev rule (99-ha.rules) has not been applied so only root can open the node, or the user does not belong to the group specified by a rule that was changed to be group-based. The default rule makes the node readable and writable by everyone (crw-rw-rw-) |
Check that the rule file exists (/lib/udev/rules.d/99-ha.rules, or /etc/udev/rules.d/99-ha.rules if you copied and edited it) and reapply it with sudo udevadm control --reload-rules && sudo udevadm trigger. If you changed the group, add the user to the group and log in again |
No such device |
The driver is still preparing the card (right after loading or right after a hot reset) or is removing the card | Open it again after a few seconds. If it persists, check dmesg for a probe failure or a removal record |
No such device or address |
The device number that the node points to differs from the one the driver currently has. This can happen if the module was reloaded or the card was probed again after the node was created, and it is mostly seen on nodes inside containers | Restart the container to create the node again |
Related documents: Container support, Installation
Status and telemetry¶
health is faulted¶
Cause:
- The device is FATAL. The firmware did not respond within the response wait ceiling or reported a fatal error, so the driver no longer assigns jobs to that card. FATAL is a state of the card, so restarting the application does not clear it.
- The last reset (an SDK reset or the automatic reset) failed.
device reset failed (...)is logged indmesg.
Resolution:
- Save the information described in Information to collect first. In particular, save
last_hangand the hang dump indmesg. - Stop the applications that use that card. Leave the applications that use other cards running.
- Reset the card. Use one of the following two methods.
- The application resets the card with the reset function of the SDK.
- After closing all handles of that card, hot reset it on the host with
sudo ha-hotreset --bdf <BDF>. The card is probed again, so the error counters also become 0. - Check whether the reset succeeded from the result received by the party that requested the reset and from the reset messages in
dmesg. Do not judge fromhealthalone. After a reset with the SDK,healthleavesfaulted, but it is normal for it to showdegradedbecause the error counters from before the reset remain. Confirm a hot reset byHot reset OK: <BDF>and byProbe successfulfor the same PCI address. - If a reset failed, the
device reset failed (...)line it left last tells you what to do next (runha-hotreset, or power-cycle the host).
To follow along with resetting a card with ha-hotreset, see Recovering an unresponsive card. To have the driver reset the device by itself every time it becomes FATAL, set ha_auto_reset=1 as described in Enabling automatic recovery.
Related documents: FATAL and recovery, Hang detection and the watchdog, Parameter reference
health is degraded¶
Cause: At least one error or timeout has occurred since the driver probed the card. One of devcmd_timeouts, devcmd_fw_errors, and hdma_timeouts is not 0. These counters never decrease, so degraded does not mean that there is still a problem now. Errors in which only that command fails and the card can still be used, such as a command that the firmware rejected because of an invalid argument sent by the application, are also counted in devcmd_fw_errors.
Resolution:
-
Check whether the counters are still increasing.
1watch -n 5 grep -H . /sys/class/ha/ha0/devcmd_timeouts /sys/class/ha/ha0/devcmd_fw_errors /sys/class/ha/ha0/hdma_timeouts -
If they have stopped, it is a past event. Check what happened and when with
last_hanganddmesg. - If they are increasing, the problem is ongoing. Find the message that corresponds to the increasing counter in Kernel log messages.
degraded returns to ok when the card is probed again (hot reset, module reload, or reboot). It does not return to ok after a reset with the SDK.
Related documents: Telemetry, sysfs nodes
Telemetry is unknown¶
Cause: unknown means that the driver cannot trust the value. The driver does not substitute the previous value or 0. Check in the following order.
| What to check | Cause | Resolution |
|---|---|---|
health is faulted |
If the device is FATAL, polling stops, and the temperature, power, and utilization are unknown from 5 seconds later |
They return when you reset the card. See health is faulted |
dmesg contains sys_mon: no response for … suspending telemetry polling |
The card did not respond, so polling stopped | Resetting the card restarts polling |
| The card is being reset | The card is not read during a reset | They return when the reset finishes |
fw_version is also unknown |
The firmware is not ready | See fw_version is unknown |
dmesg contains sys_mon telemetry unavailable |
The telemetry poller could not be started at probe | Hot reset the card or reload the module |
| Firmware version | The firmware is old and does not provide telemetry | Check the firmware version in Supported environments |
Only util is unknown |
The firmware has not finished the first measurement yet, or the utilization measurement is not progressing | Read again after a moment. If it continues, notify the support team |
Only power_mw is unknown and rails is none |
No power rail is measured | — |
If there are no telemetry nodes at all under /sys/class/ha/haN/ and dmesg contains telemetry sysfs unavailable, the nodes could not be created. Report it to the support team together with the kernel version.
Related documents: Telemetry, sysfs nodes
fw_version is unknown¶
Cause: The driver could not confirm that the firmware is ready or could not read the firmware version. Look for the following messages in dmesg.
| Message | Cause |
|---|---|
firmware boot sentinel not seen within <ms> ms |
The firmware has not booted or is still booting |
firmware never reported ready |
The firmware did not report that it is ready. It is still booting, or its self-test failed during boot |
firmware version CSR reads 0x<value> |
The version was read but the value was not valid. |
device came back but firmware did not report ready |
The firmware was not ready after a reset |
sys_mon: no response for … |
The card did not respond, so polling stopped. For a card whose monitoring channel has been opened at least once, the version is not read again until it is reset. For a card that was still booting, the version is read again when fw_version is read after the firmware becomes ready |
Resolution:
- If this is right after loading or right after a reset, wait a few seconds and read again. If the firmware becomes ready late, the driver reads the version by itself and logs
firmware booted after probe gave up; CSR-derived telemetry refreshed. - If it stays
unknown, close all handles of that card and restart the card withsudo ha-hotreset --bdf <BDF>. If the configuration does not allow an SBR to be asserted, power-cycle the host. - If it is still
unknown, the firmware of the card cannot boot. Contact the support team.
While fw_version is unknown, ha-fw-update does not update that card and ends with exit code 2.
Related documents: Kernel log messages, FATAL and recovery
hyperaccel-smi does not show the card¶
Cause: The default output shows No HyperAccel LPU found under <path> and the exit code is 2. Even when run with -q or --json, the exit code is 2 if no card is visible.
- The driver has not probed the card, so
/sys/class/ha/is empty. - The environment variable
HYPERACCEL_VISIBLE_DEVICESrestricts the visible cards. If the value isnoneor empty, no card is visible, and if it is a comma-separated list of numbers, only the cards with those numbers are visible. - A number that does not exist was given to
-i. - Inside a container, the
/sys/class/haof the host is not visible or is at a different path.
Resolution:
1 2 | |
- If
/sys/class/ha/is empty, follow/dev/haNis missing. - Unset
HYPERACCEL_VISIBLE_DEVICES, set it toall, or correct it to the right numbers. - If sysfs is mounted at a different path (for example,
/host/sysinside a container), tell the tool theclass/hadirectory under it with--root /host/sys/class/haor the environment variableHYPERACCEL_SYSFS_ROOT. If you give the path where sysfs is mounted itself, such as/host/sys, the tool does not find the cards. hyperaccel-smiruns only ifpython3is available.
Related documents: hyperaccel-smi
Hang and recovery¶
A hang dump appears in dmesg¶
Cause: Lines like the following were logged.
1 2 3 | |
The first line that is logged before the dump depends on <reason>. If it is ceiling timeout, the Devcmd no response after <ms> ms (device hang): <verdict> line above is logged; if it is fw_update timeout, fw_update: no response after <ms> ms: <verdict> is logged; if it is watchdog, watchdog: devcmd wait stalled (devcmd=<ms> ms hdma=<ms> ms) - no completion within the threshold is logged.
The firmware did not respond to a command within the response wait ceiling (default 12 seconds; correspondingly longer for commands that take a long time). If the reason is watchdog, a request that waits without a ceiling exceeded the watchdog threshold (default 8 seconds; with ha_wd_preempt=1, other requests are also subject to it), and if it is fw_update timeout, a firmware update exceeded its deadline.
Resolution:
- Save the two dump lines and the surrounding log as they are. See Information to collect first.
- If the reason is
fw_update timeout, do not reset the card; follow Resets and jobs are refused after an update. - Check
health. If it isfaulted, followhealthisfaulted. - If the state is not
faultedand the reason iswatchdog: withha_wd_preempt=1, the watchdog has already cut off that wait and the request failed. If the device appears to have stalled, reset the card. With the default (ha_wd_preempt=0), the request that waits without a ceiling is still waiting. Check whetherDevcmd waiting …continues to appear indmesgevery 10 seconds, and readlast_hangtwice at an interval of at least 20 seconds to see whetherbatch_seqorexec_indexchanges. During this wait,last_hangis updated only when aDevcmd waitingline is logged, so if you read it at shorter intervals, you may get the same value even though the command is progressing. If it changes, the command is progressing slowly; if it stays the same, a single command is taking a long time or has stalled. If, comparing with how long that command normally takes, you judge that it has stalled, terminate that application and reset the card. - If the same symptom recurs, send it to the support team together with the dump. The meaning of each field of the dump and of the verdicts is in Reading the hang dump.
Related documents: Hang detection and the watchdog, FATAL and recovery
DMA transfer errors appear in dmesg¶
Cause: The following lines were logged by the DMA engines that handle copies between the host and the card, or between cards.
1 2 3 4 | |
ll: timeoutmeans that one transfer did not finish within 5 seconds.hdma_timeoutsincreases. If the engine can be stopped, only the work that the transfer belongs to fails and the card does not become FATAL.ll: engine stopped but its completion … never arrivedmeans that the engine had already stopped, but its completion signal did not reach the driver within 5 seconds.hdma_timeoutsincreases and only the work that the transfer belongs to fails.ll: no responseandring write: no responsemean that the card does not respond to the status register of the DMA engine. The link may be down, or the card may have stopped or been removed, and the card becomes FATAL.HDMA engine at …means that the engine could not be stopped after a timeout or abort. The engine may still be able to write to host memory, so the driver quarantines those host pages and blocks new transfers, and the card becomes FATAL.
Resolution:
- Save the logs before and after the lines, and the sysfs values. See Information to collect first.
- Check
health. If it isfaulted, followhealthisfaulted. When you reset the card, the engines stop, the quarantined pages are released, and transfers are accepted again. - If every copy in a VM guest logs
ll: engine stopped but its completion MSI-X never arrived, the driver mistook the VM for bare metal. Reload the driver inside the guest withha_hdma_completion=poll. - If only
ll: timeout,ll: engine stopped …, orll: n=… error=…was logged andhealthis notfaulted, run the failed work again. - If it repeats on the same card, check the PCIe error messages from the same time and the link speed and width (
current_link_speed,current_link_width), and send the diagnostic information to the support team. If the other cards on the same server are fine and only this card repeats it even when moved to another slot, the problem is likely on the card side.
Related documents: Kernel log messages, FATAL and recovery
The application receives an error that the device cannot be recovered¶
Cause: The device is FATAL. FATAL is a state of the card, so it is not cleared by restarting the application or reopening the node. The sysfs health is faulted.
Resolution:
- Reset the card by the procedure in
healthisfaulted. - A reset invalidates all open handles of that card, so the other applications that were using that card must also reopen the node. If several applications share one card, decide that only one of them performs the reset.
- If the reset request returns a failure, or the same error occurs again after the reset, check the reset messages in
dmesgto see which step failed. See Reset progress and results.
Applications that use only other cards are not affected. However, jobs on other cards that exchanged data with this card can fail (Multiple devices).
Related documents: FATAL and recovery
ha-hotreset refuses the reset¶
Cause: If the pre-flight check fails, the tool touches nothing and ends with Pre-flight failed - nothing was touched. The line directly above it gives the reason. The exit code is 3 for an open handle and 4 for a configuration in which an SBR cannot be issued. Path masking can also fail right before the reset; the tool then ends with Nothing was touched.
| Output | Cause | Resolution |
|---|---|---|
<BDF>: /dev/ha<N> still open by PID(s) <pid> |
A process has the node of that card open | Terminate that process and run the tool again. With --force, the tool proceeds even if handles are open, but an application that holds a handle loses the card and must reopen the node. Use --force only in an emergency, and do not use it on a card whose firmware is being updated |
<BDF>: parent is a <port type> whose Surprise Down mask cannot be read (no AER capability) - refusing SBR, <BDF>: parent has no PCIe capability - refusing SBR |
Whether the parent port masks Surprise Down errors cannot be determined. In this configuration, dropping the link with an SBR can cause the platform to reset the entire host. This cannot be bypassed even with --force |
Power-cycle the host to restart the card |
<BDF>: cannot read <register> on <port> - the path cannot be masked, refusing SBR, <BDF>: refusing to SBR without a verified mask on the whole path |
The parent port does not mask Surprise Down errors, and the ports on the path cannot be masked. This cannot be bypassed even with --force |
Power-cycle the host to restart the card |
<BDF>: parent is a <port type> with Surprise Down Error unmasked - refusing SBR |
--no-switch-link-drop was given, and the parent port does not mask Surprise Down errors |
Run without --no-switch-link-drop to reset with the path masked, or power-cycle the host |
another ha-hotreset is running on this host - waiting up to 600s |
Another ha-hotreset is running, so the tool waits. If it does not finish within 600 seconds, the tool prints the other ha-hotreset did not finish in 600s - nothing was touched and ends with exit code 3 |
Run it again after the other run ends |
<BDF>: parent bridge <bridge> has more than one device below it: |
There are other devices under the same parent bridge, so the SBR also resets those devices | The card cannot be reset on its own. Power-cycle the host |
Other cases are as follows.
Run as root: Run withsudo.<n> cards present - pass --bdf or --all:: If there are multiple cards, specify one card with--bdfor use--all. The exit code is 2.--bdf '<value>' is ambiguous: The same bus address exists in several PCI domains. Specify the full address including the domain (in the0000:01:00.0format). The exit code is 2.setpci not found (install pciutils): Install thepciutilspackage. Forflock not found (install util-linux), install theutil-linuxpackage.- If
<BDF>: /dev/ha<N> did not reappear - check dmesg for probe failureappears after the reset, follow/dev/haNis missing. the mask is LEFT IN PLACE on purpose: The reset card did not respond again, so the tool deliberately left the path masking in place. Real link errors on that path are not reported either, so after you deal with the card problem, restore it with thesetpcicommands printed right below.
Running sudo ha-hotreset --list lets you see in advance, for each card, whether the configuration allows an SBR to be asserted from the parent port. Cards for which it cannot be asserted are marked NOT resettable, and cards that are reset with the path masked are marked the path is masked for the reset. Whether other devices exist under the same parent bridge, and whether the registers on the path can be read, are not shown in this list; they are checked in the pre-flight check at reset time.
Related documents: ha-hotreset, FATAL and recovery
Firmware update¶
ha-fw-update fails¶
Cause and resolution: Narrow it down from the exit code and the last ERROR: line.
| Exit code | Meaning and common messages | Flash | What to do |
|---|---|---|---|
1 |
A usage or environment problem. A -d address format error, the image file cannot be read (cannot read image), not root (root required), no reset script (no reset script), declined at the confirmation prompt (aborted) |
Unchanged | Fix it as the message says and run the tool again |
2 |
The card cannot be used or does not support updates. No card (no HyperAccel card at), in use by another process (is open by pid(s)), an update is already running on the same card (another ha-fw-update is running), fw_version is unknown (reports firmware 'unknown'), the firmware is not ready (device not ready), the driver is waiting for the response to an earlier update (upload failed: Device or resource busy), the firmware aborted before writing (firmware aborted before writing), the driver or firmware does not support updates, a command-line syntax error (an unknown option, -d and -i given together or neither given, and so on) |
Unchanged | Follow the message. If another process is using the card, run the tool again after it finishes. If the driver is waiting for the response to an earlier update, follow Resets and jobs are refused after an update. If the firmware aborted before writing, reset the card and then run the tool again. If the driver does not support updates, update the driver first; if the firmware does not support them, contact the support team |
3 |
The image was rejected. The image check of the tool failed, the firmware rejected the image (firmware rejected the image), the partition layout in the flash differs from the image (fip partition layout differs from the one in flash -- retry with --scope full) |
Unchanged | If the message says the layout differs, run the tool again with --scope full. Otherwise, check the image file |
4 |
The write failed or the state is unknown (FLASH WRITE FAILED, firmware did not answer within the budget, device went FATAL or stopped answering) |
May be partially written | Do not power off or reboot the host; run the tool again with the same image. If no response was received, also see Resets and jobs are refused after an update |
5 |
The flash was updated but the card could not be reset (flash updated but the reset failed) |
New image | Fix the cause using the ha-hotreset messages printed along with it, and then run sudo ha-hotreset --bdf <BDF> or power-cycle the host. See ha-hotreset refuses the reset |
6 |
After the reset, the card did not return within --boot-timeout (default 60 seconds), or the tool stopped at the verification step |
New image | Check dmesg and fw_version. If the card does not boot, contact the support team |
- With
--json, the result is output as JSON together withexit_codeanderror. - For exit code 4, do not reset the card; run the update again. Unless you restart the card, it does not boot from a partially written flash. If the tool ended because no response was received from the firmware, run it again after the firmware responds. See Resets and jobs are refused after an update.
When the tool mentions a debug tool
The tool may also state that a debug tool (e.g., Trace32) may be needed for recovery. This is a JTAG debugger, and the message means that the card is in a state that cannot be recovered with host software. If you see such a message, contact the support team.
Related documents: Firmware update, ha-fw-update
Resets and jobs are refused after an update¶
Cause: After ha-fw-update ends with exit code 4, the following line is logged in dmesg.
1 | |
The driver sent the update command but could not confirm the response of the firmware. The firmware may still be writing the flash, so to avoid booting from a partially written flash, the driver refuses resets and application jobs for that card until the response is confirmed. A reset request logs reset refused: last fw_update unanswered …, and an automatic reset logs auto-TDR: skipped, …. The step in which ha-fw-update loads the image into the card memory is also refused, so running it again in this state prints upload failed: Device or resource busy and ends with exit code 2. This run does not change the flash.
Resolution:
- Do not power off or reboot the host, and do not reset the card. This includes
ha-hotresetand resets by applications. - If the firmware responds, even late, the refusal is lifted and
fw_update: late completion from firmware; resets re-enabledis logged. This line does not contain the result of the write, so after you see it, runha-fw-updateagain with the same image to complete the reset and the verification. - If the firmware never responds, contact the support team. A module reload,
ha-hotreset, or a power cycle risks the card booting from a flash whose write has not finished, so use them only after consulting the support team.
Related documents: Firmware update, Kernel log messages
The version is unchanged after the update¶
Cause:
- It was run with
--no-reset. Only the flash changed, and the old firmware keeps running until you hot reset the card. - An application reset the card with the SDK. This reset usually only reinitializes the running firmware and does not load the new image from the flash.
- It succeeded with a
same versionwarning. This is the case where an image with the same version number was written.
Resolution: Close all handles of that card, hot reset the card, and then check the version.
1 2 | |
The card normally reappears with the same number, but to be sure, find it by PCI address.
Related documents: Firmware update, ha-hotreset
Requesting support¶
Contact the HyperAccel support team and attach the following.
- The archive created with Collecting diagnostic information for a problem, or the file created with Information to collect first
- The
dmesgsaved before a reset, a hot reset, or a module reload - The driver version (
/sys/module/ha/version) and the firmware version (fw_version) - The time the symptom started and the jobs that were running then
- The reproduction steps, and whether the problem still reproduces after the card is reset or hot reset
- If the support team requests detailed logs, the
dmesgcaptured while reproducing the problem with detailed logging turned on by following Enabling detailed logs
See also¶
- Kernel log messages — meaning of and action for each message
- FATAL and recovery
- Hang detection and the watchdog
- Telemetry
- Firmware update
- Container support
- sysfs nodes
- Parameter reference
- Verifying the installation