Firmware update¶
This page covers how to update the firmware in a card's boot flash from the host with ha-fw-update. It explains, in order, how the update works, what to check before you begin, the update procedure, the restrictions during an update, how to verify that the card booted the new firmware, and how to respond when an update fails. Options, exit codes, and the JSON output are defined in ha-fw-update.
Do not power off or reboot the host during an update
A card that fails to boot this way cannot be recovered with host software. Even if the tool terminates abnormally or the terminal connection drops, the firmware carries the write through to the end, so in that case too, leave the host as it is and wait.
When to update¶
- When HyperAccel releases a new firmware image and announces the update
- When a card's firmware is older than the firmware that Supported environments specifies for use with the installed driver
Check each card's current firmware version in the sysfs fw_version or in Firmware Version of hyperaccel-smi -q. The value has the format major.minor.patch, and it is unknown if the firmware is not ready yet.
1 2 | |
First check that the new firmware matches the installed driver
If the card boots with firmware that is not compatible with the driver, the driver may not probe the card. The card then does not come back after the update, and you cannot update it again with this tool. Before you update, check the firmware and driver combination in Supported environments.
Before you begin¶
| Item | Requirement |
|---|---|
| Privileges | Run as root (sudo) |
| Tools | ha-fw-update and ha-hotreset are installed in /usr/bin/. Python 3.7 or later is required (check with python3 --version; if the version is lower, see Requirements), and ha-hotreset uses pciutils (lspci, setpci) |
| Image | A firmware image distributed by HyperAccel (image.gpt), or a full flash image that is exactly 16 MiB. The detailed conditions are in Image file |
| Firmware | The firmware running on the card must support updates from the host (Supported environments). If it does not, the tool ends with exit code 2 and the flash is not changed |
| Card state | The firmware must be ready (fw_version is not unknown), and no process may have the card's /dev/haN open |
| Hot reset | To boot the new firmware after the update, ha-hotreset must be able to reset the card. For a card shown as NOT resettable in sudo ha-hotreset --list, update with --no-reset and then power-cycle the host |
| Board | As the safety item (SAFETY) of ha-fw-update --help says, use the tool only on boards whose root-of-trust is in bypass mode. This cannot be verified in software, so contact HyperAccel if you are not sure |
Update procedure¶
1. Select the card¶
Specifying the card by its PCI BDF address, including the domain, is recommended. The number in /dev/haN can change after a hot reset or after a card is added, but the PCI address is fixed to the slot. Pass the value in the Bus-Id column printed by hyperaccel-smi to -d as it is.
1 | |
Bus-Id has the format domain:bus:device.function. If only one card across all PCI domains has that bus:device.function, you can omit the domain, as in -d 01:00.0. If cards with the same bus:device.function exist in several domains, the tool lists the candidates and refuses, so use the full address in that case. You can also give the /dev/haN number with -i N.
2. Stop the workloads running on the card¶
The update proceeds only when no process has the card's /dev/haN open. If another process has it open, the tool reports its PID and ends with exit code 2. You do not need to stop workloads that use only other cards, or monitoring that only reads sysfs, such as hyperaccel-smi.
3. Run the pre-flight check¶
--dry-run performs only the checks up to just before the write and writes nothing.
1 | |
The pre-flight check verifies root privileges, the image format, the card, other updates running on the same card, fw_version, other processes that have the card open, and the reset command. Finally, it opens the card and allocates the card memory to which the image is uploaded, to confirm that the card can accept requests. If the check passes, the tool prints a summary and ends with Dry run: nothing written. and exit code 0. The exit codes for each check item are in Pre-flight checks.
If the Reset line of the summary shows parent is NOT a root port and the warning WARNING: <BDF> is not directly under a root port; ... appears, the hot reset after the update proceeds with the path masked, or is refused if the path cannot be masked (Path masking). Check with sudo ha-hotreset --list whether the card can be reset. --list judges only the conditions of the parent port. If another device is under the same parent bridge, ha-hotreset refuses the reset even if --list shows the card as resettable.
4. Choose the scope¶
| Situation | --scope |
|---|---|
| Not sure | full (default) |
| Full 16 MiB flash image | full (this image accepts only full) |
| You know that only the fip partition (firmware application) of the image changed | app |
full writes the entire image, and app writes only the fip partition of the image. Either way, only the parts whose content differs from the flash are actually written, so the difference in time is small.
appdoes not write anything outside the fip partition. If other parts of the image changed, those changes are not applied.- If the location of the fip partition in the flash differs from the one in the image, the firmware rejects the
apprequest (exit code 3), and the tool tells you to run again with--scope full. The flash is not changed.
--rewrite erases and rewrites the entire scope without skipping even the sectors that are the same as the flash. It is usually not needed.
5. Run the update¶
1 | |
The tool finishes the pre-flight check, prints the summary, and then asks for confirmation with Proceed? [y/N]. Enter y to start. In scripts, skip the prompt with -y.
Example output:
1 2 3 4 5 6 7 8 9 10 11 12 | |
At [2/5], the tool uploads the image to the card memory, and at [3/5], it waits until the firmware finishes inspecting, comparing, writing, and read-back verification. If [3/5] is ok, the flash already holds the new image. Then at [4/5], the tool hot-resets the card, and at [5/5], it verifies that the card booted the new firmware. Each line is defined in Screen output.
The device timeout in the summary is the ceiling for how long the driver waits for the firmware's response, not the expected duration.
In a remote session, run the tool inside tmux or screen
If the connection drops, the tool may not run to completion. The firmware's write is not affected. After you reconnect, run the same command again; it skips the parts already written and completes the reset and verification. If the earlier run has not finished yet, the tool prints another ha-fw-update is running for <BDF>, and if the driver is still waiting for the response to the earlier update, it prints upload failed: Device or resource busy. In both cases it ends with exit code 2, so run it again after a while. For the latter case, see When no response is received.
6. Watch the progress¶
During [3/5], the terminal keeps updating the same line to show the progress.
1 | |
- The progress is read from the sysfs
fw_update_progress. The value is<completed sectors> <total sectors>(in units of 4 KiB), and sectors skipped because they are the same as the flash also count as completed. An image that is nearly identical therefore reaches 100% right away. - While the firmware inspects the image, the value is
0 0, and the tool shows the elapsed time instead of the progress. - If the progress cannot be read (the value is
unknownor stays at0 0), the tool shows only the elapsed time, and when[3/5]finishes it displaysno progress counter (old firmware)instead of the sector count. - If you send the output to a file or a pipe, the tool does not update the line and prints only the step name once.
You can also read the same value directly from another terminal. After the update finishes, the value remains until the next update or until the card restarts.
1 | |
From [3/5] until the reset finishes, Ctrl+C, SIGTERM, and SIGHUP are held pending. If you press Ctrl+C in this interval, the tool completes the write and the reset, skips the verification step, and ends with exit code 6. At that point the flash holds the new image and the card has been reset, so go on to Verifying that the card booted the new firmware. A pending SIGTERM or SIGHUP terminates the tool right after the reset succeeds, and in that case the output of the verification step and the --json result are not produced. In this case too, the flash holds the new image and the card has been reset. With --no-reset, the pending signals are not acted on and the tool simply ends.
7. Boot into the new firmware¶
At [4/5], the tool closes /dev/haN and resets the card with ha-hotreset --bdf <BDF>. After the reset, the card boots from the new image in the flash, and the driver probes the card again.
At [5/5], the tool waits for up to --boot-timeout (default 60 seconds) until the card at the same PCI address reports fw_version. The /dev/haN number can change after the reset.
Resetting later¶
With --no-reset, the tool updates only the flash and does not reset the card. Until the card restarts (hot reset, host reboot, power cycle), it keeps running on the previous firmware, and fw_version shows the previous version in the meantime. Use it in the following cases.
- The host operates under a policy of resetting regularly at fixed times
ha-hotresetcannot reset the card (NOT resettable), so the host must be power-cycled after the update finishes- Another process has the card open and you must proceed with
--force(in this case,--forcemust be used together with--no-reset)
1 2 3 | |
An application's reset is not a way to load new firmware
The SDK's device reset first re-initializes the running firmware. To boot from the new image in the flash, use ha-hotreset or a host power cycle.
Multiple cards and automation¶
Run the update once for each card. The following example stops if any one of the updates fails.
1 2 3 | |
To process the result in a script, use --json. --json must be used together with -y or --dry-run, and it writes only a single result JSON object to standard output instead of the screen output. Check the exit code and the result field. The fields are described in JSON output.
1 | |
An example script that saves the result JSON for each card and tells you what to do on a card where the run stopped and which command to run to continue is in Updating firmware on several cards in sequence.
Restrictions during an update¶
While an update is in progress, the following restrictions apply to that card. Other cards are not affected.
- Requests that send commands to the device being updated, such as an application submitting a job through the SDK or requesting a reset, are rejected immediately without waiting.
- The driver's automatic reset (
ha_auto_reset) and watchdog preemption (ha_wd_preempt) do not operate on that device. The kernel log may showauto-TDR: skipped, a fw_update is in flight or unanswered (flash may be mid-write). - Because the update tool has
/dev/haNopen,ha-hotresetrefuses to reset that card. - Running another
ha-fw-updateon the same card is rejected with exit code 2. - sysfs reads and
hyperaccel-smikeep working as usual.
Verifying that the card booted the new firmware¶
If the [5/5] Verify line is ok and shows firmware <previous> -> <new>, the following has been confirmed.
- After the reset, the driver probed the card at the same PCI address again.
- The card booted from the image in the flash, and the firmware reported that it is ready.
- The version reported by that firmware is the value to the right of the arrow in the log line
[5/5] Verify ........... ok firmware 1.0.5 -> 1.0.6 (ha0)(for example,1.0.6).
Whether the flash content matches the image has already been verified at [3/5], where the firmware read it back.
[5/5] compares version numbers. If the image you wrote has the same version number as before the update, the tool ends successfully (exit code 0) with the warning WARNING: same version -- confirm the new image on the UART boot log.. The version number alone cannot tell the new image apart. A release firmware image does not print its version on the UART, so the UART boot log named in that warning cannot show which image is running. Identify the image by the file you wrote.
To check directly, print the PCI address, node, and firmware version together.
1 2 3 4 | |
If fw_version is unknown, the card is still booting or the firmware is not ready. Read it again after a while.
When the update fails¶
When the update tool fails, it prints a message that starts with ERROR: (with --json, the message goes into error of the result JSON) and indicates the flash state through the exit code. The full definition is in Exit codes.
| Exit code | Flash | What to do |
|---|---|---|
1, 2, 3 |
Not changed by this run | Fix the cause as the message says, then run again |
4 |
May be partially written | Do not reset the card; run the same command again. See When no response is received |
5 |
New image | See When the reset fails |
6 |
New image | See When the card does not come back |
If the tool ended without reporting an exit code (the terminal connection dropped, the process was terminated, or it raised a Python error that starts with Traceback), you cannot tell how far that run wrote to the flash. Do not reset the card; run the same update command again.
When you contact HyperAccel, send the tool's output (or the --json result), sudo dmesg, and the output of hyperaccel-smi -q together.
When the update ends before writing¶
Exit codes 1, 2, and 3 mean that the run wrote nothing to the flash. The messages you see most often are as follows.
| Message | What to do |
|---|---|
/dev/haN is open by pid(s) ...; close them, or pass --force --no-reset |
End those processes and run again |
another ha-fw-update is running for <BDF>, another update is running on this device |
Wait for the other update on the same card to finish |
device haN reports firmware 'unknown' (not ready), a message that starts with device not ready |
The card is booting or has just been reset. Run again after a while, and if it persists, check dmesg |
device allocation failed: ... (for example, State not recoverable) |
The card is in a state where it cannot accept commands. Check the sysfs health, recover the card as described in FATAL and recovery, and run again |
upload failed: Device or resource busy |
An update is in progress on that card, or the driver is waiting for the response to an earlier update. Follow When no response is received |
this firmware has no update handler: flash it once with a debug tool (e.g., Trace32), firmware ABI predates FW_UPDATE (protocol < 7): flash it once with a debug tool (e.g., Trace32) |
The running firmware does not support updates from the host. Contact HyperAccel |
fip partition layout differs from the one in flash -- retry with --scope full |
Run again with --scope full (the default) |
Image check messages such as firmware rejected the image (GPT/CRC) -- flash untouched and not a GPT image ... |
The image file is corrupted or is a different file. Check the distributed image again |
the firmware could not read the flash ... |
Run again |
firmware aborted before writing (device busy/unhealthy) -- reset the card and retry |
The firmware aborted before erasing the flash. Reset the card with sudo ha-hotreset --bdf <BDF>, then run again |
no reset script: ... |
Install ha-hotreset, give its path with --hotreset, or run with --no-reset |
When no response is received¶
There are two typical cases of exit code 4.
FLASH WRITE FAILED -- flash may be partially written. Do NOT reboot; retry, then use a debug tool (e.g., Trace32): The firmware responded that writing or read-back verification failed.firmware did not answer within the budget -- state unknown, ...ordevice went FATAL or stopped answering during the update -- state unknown, ...: The host did not receive a response within the deadline. This does not mean that the firmware stopped writing.
In either case, the card does not boot from a partially written flash unless the card is restarted. The answer is therefore not a reset but running the update again.
- Do not power off or reboot the host, and do not reset the card. This includes
ha-hotresetand an application's reset. - Run the same command again with the same image. Only the parts whose content differs from the flash are rewritten, so the parts already written by the earlier write are skipped. If the host did not receive a response, run it again after the firmware responds, as described below.
If the host does not receive a response, the kernel log shows the following.
1 | |
Depending on the cause, this is preceded either by fw_update: no response after <N> ms: ..., which says the response deadline passed, together with a hang diagnostic tagged fw_update timeout (Hang detection and the watchdog), or by fw_update: device vanished during the write (all-ones), which says the card disappeared during the write.
From this point until the firmware responds, the driver keeps the same restrictions on that card as in Restrictions during an update. It rejects requests that send commands to the card and an application's reset requests, and it stops automatic reset. When an application's reset request is rejected, the following is logged.
1 | |
ha-fw-update uploading the image to the card memory is also one of the rejected requests. So if you run it again in this state, it prints upload failed: Device or resource busy at [2/5] and ends with exit code 2, and this run does not change the flash.
If the firmware responds, even late, this state is cleared and the following is logged.
1 | |
This message does not mean that the update to the new firmware image succeeded. After you see this message, run the update again with the same image to complete the reset and verification.
Even in the state where no response has been received, ha-hotreset proceeds with the reset, because no process has the card open after the tool has ended. Do not run ha-hotreset before the firmware responds.
If the card never responds, the remaining options are to reload the driver to clear the reset-refused state (rmmod), to reset with ha-hotreset, or to power-cycle the host. Each of these carries the risk that the card boots from a flash whose write has not finished, so contact HyperAccel first.
When the reset fails¶
Exit code 5 means that the flash was updated to the new image and the firmware's read-back verification finished, but ha-hotreset failed at the reset step. The error message includes the output of ha-hotreset. Common causes are as follows.
- A configuration in which SBR cannot be issued from the card's parent port (
NOT resettable), or path masking that is not applied - Another
ha-hotresetrun that does not finish within 600 seconds - Another device is under the same parent bridge
- Another process opened the card during the update
pciutilsis not installed
After you resolve the cause, run sudo ha-hotreset --bdf <BDF> again, or power-cycle the host. The flash write has already finished, so a reboot or power cycle is safe at this stage. The reset conditions are described in ha-hotreset.
When the card does not come back¶
Exit code 6 means that, after the reset, the card at the same PCI address did not report fw_version within --boot-timeout (default 60 seconds), or that the verification step was skipped with Ctrl+C. The flash holds the new image.
- Wait a while, then check again with the commands in Verifying that the card booted the new firmware. A card whose boot was delayed logs
firmware booted after probe gave up; CSR-derived telemetry refreshedin the kernel log and reportsfw_version. - Check the messages for that card in
sudo dmesg. -ABI protocol mismatch: firmware v..., driver requires >= v...; refusing probe: The installed driver does not support the new firmware. Check Supported environments. -firmware never reported ready: The firmware did not report that it was ready during boot. - If the card cannot boot, contact HyperAccel.
Safety rules¶
- Do not power off or reboot the host until
[3/5]finishes. Even if the tool terminates abnormally, the firmware carries the write through to the end. Thedevice timeoutin the summary is the ceiling for how long to wait. - Do not reset the card with
ha-hotreset --forceduring an update.--forceresets the card even if handles are open. - Do not reset the card with the PCI sysfs
reset(/sys/bus/pci/devices/<BDF>/reset). This card does not support FLR. For a reset from the host, use onlyha-hotreset. - The answer to exit code 4 is a retry, not a reset. A partially written flash is not a problem as long as the card is not restarted. If retries keep failing, do not reset the card; contact HyperAccel.
When the tool mentions a debug tool
The tool prints messages saying that recovery may require a debug tool (e.g., Trace32). This means recovery with a JTAG debugger and refers to a state that cannot be recovered with host software. If you end up in such a state, contact HyperAccel.
See also¶
- ha-fw-update — options, pre-flight checks, exit codes, JSON output, image requirements
- ha-hotreset — the hot reset command and reset conditions
- Updating firmware on several cards in sequence — an example script that updates several cards one at a time
- FATAL and recovery — automatic reset, an application's reset, hot reset
- sysfs —
fw_version,fw_update_progress - hyperaccel-smi — checking
Bus-Idand the firmware version - Parameter reference —
ha_auto_reset,ha_wd_preempt - Kernel log messages
- Supported environments — firmware and driver combinations
- Troubleshooting