Updating firmware on several cards in sequence¶
Update all the LPUs installed in the host to the same firmware image. The example script updates the cards one at a time in order with ha-fw-update and records the results; if any card does not finish normally, it stops without starting the next card. The procedure for updating a single card, the constraints during an update, and the safety rules are in Firmware update; this page covers only how to apply that procedure to several cards in sequence. You need root privileges on the host and ha-fw-update and ha-hotreset from the driver package (Install location). The example updates three cards (0000:01:00.0, 0000:41:00.0, 0000:81:00.0) from firmware 1.0.5 to 1.0.6. Replace the addresses, versions, and image names with your actual values.
Before you begin¶
- Check that the new firmware matches the driver. Check the combination of the new firmware and the installed driver in Supported environments. A card whose running firmware uses a command protocol below v7 cannot be updated with
ha-fw-update. - Stop all workloads that use the cards you are updating. If any process has the
/dev/haNof a card open, the update of that card ends with exit code 2, and the script stops there as well. Monitoring that only reads sysfs, such ashyperaccel-smi, can be left running. If you update only some of the cards, also stop jobs that run on other cards and exchange data with those cards (When one card is reset or removed). - Check that each card can be hot reset. If
sudo ha-hotreset --listmarks any card asNOT resettable, leave that card out by giving only the PCI addresses of the remaining cards when you run the script. If you give no addresses, all registered cards are updated. Update a card you left out separately, as described in Resetting later. A card markedthe path is masked for the resetis reset with the path masked during the reset. On a platform you are using for the first time that has such a card, check withsudo ha-hotreset --check-mask --bdf <PCI address>that the masking holds before you start. On a platform that monitors errors through another path, the entire host can restart when that card is reset, even with the path masked (Path masking).--listevaluates only the parent port, so for a card that has other devices under the same parent bridge, the reset is refused after the write finishes and the script stops with exit code 5. - Leave the host as it is until all cards finish. Do not power off or reboot the host during the update. For a remote session, run the script inside
tmuxorscreen. Read the Safety rules first. - Record the current versions. The script's
-nprints the PCI address, node, and firmware version of each card (1. Check the target cards).
The script handles only cards registered with the driver, that is, cards that have a /dev/haN. Compare against the number of cards from lspci -D -d 20fb: to check that every installed card is registered. The rules for numbering and registration are in Card numbers.
Script¶
- Cards are identified by PCI address. If you give no address, all registered cards are selected, using the PCI address that the
/sys/class/ha/haN/devicelink points to. The/dev/haNnumber is not used because it is not the identity of a card (Card numbers). - For each card,
ha-fw-update -d <PCI address> -y --json IMAGEis run, and the result JSON is saved as<PCI address>.jsonin the results directory.ha-fw-updateperforms the hot reset and the check of the new firmware. The script does not reset cards itself and does not use--force. - If the exit code of
ha-fw-updateis not 0, the script stops and prints what to do for that card and the command to resume with. Ifexit_codein the result JSON differs from the exit code, that is, ifha-fw-updateended without leaving a result, the result of that card is shown asunknownand the guidance is the same as for exit code 4. - On Ctrl+C,
SIGTERM, orSIGHUP, the script waits forha-fw-updateof the card in progress to finish and then stops. The next card is not started.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | |
| Environment variable | Default | Meaning |
|---|---|---|
LOGDIR |
ha-fw-update-<date>-<time> in the current directory |
The directory in which to save the per-card result JSON (for example, sudo env LOGDIR=/var/tmp/fw-update sh ha-fw-update-all.sh image.gpt) |
SYSFS |
/sys |
The sysfs path from which the card list and fw_version are read. Change it when the host's /sys is mounted at a different path. ha-fw-update reads /sys regardless of this value |
Running the script¶
1. Check the target cards¶
-n prints only the cards and their current firmware versions and does not run ha-fw-update. It reads only sysfs, so root privileges are not required.
1 | |
1 2 3 4 5 | |
- To update only some of the cards, give the PCI addresses after the image, in the format that includes the domain. The cards are updated in the order given.
- If any of the given addresses is not a registered card, the script prints
ERROR: <address> is not a registered cardand updates no card. - To also run the pre-flight check of
ha-fw-updatefor each card, runsudo ha-fw-update -d <PCI address> --dry-run image.gptfor each card. The script has no such option. See Run the pre-flight check.
2. Update the cards¶
1 | |
The script skips the confirmation prompt of ha-fw-update and also does not show the summary and warnings that the tool prints before an update. Run it only after you have checked everything in Before you begin. If all cards succeed, it prints the following and ends with exit code 0.
1 2 3 4 5 6 7 8 9 | |
- The
[n/total]line is printed when the update of that card starts. No progress is printed until one card finishes. To see progress, usefw_update_progressof that card from another terminal (for example,cat /sys/class/ha/ha1/fw_update_progress). BEFOREandAFTERare thefw_versionread immediately before and immediately afterha-fw-updateruns for that card (for a card that wasnot run,BEFOREis the current value, and for a card that is not registered it is-).RESULTis one of the following: the exit code ofha-fw-update(exit N), the result cannot be determined (unknown), or not run (not run).- The results directory holds the result JSON of
ha-fw-updatefor each card. The meaning of fields such asresult,exit_code,error,fw_version_before, andfw_version_afteris in JSON output.
Verifying after the update¶
Run sh ha-fw-update-all.sh -n image.gpt again, or check Firmware Version in hyperaccel-smi -q, to confirm that every card reports the new version.
- If you used an image with the same version number as before the update, it is a success even if
AFTERequalsBEFORE. In that caseversion_changedin the result JSON isfalse(grep '"version_changed"' ha-fw-update-20261008-101500/*.json), and the version number cannot tell the new image apart. See Verifying that the card booted the new firmware. - A card whose
fw_versionisunknownis still booting or its firmware is not ready. Check again after a while, and if it persists, follow When the card does not come back. - When the check is complete, restart the workloads you stopped. The hot reset recreates the host's nodes, so also restart the containers that used the updated cards.
When it stops partway¶
If ha-fw-update ends with an exit code other than 0 for any card, the script does not start the next card. Below the summary table it prints the error message of that card (the ERROR: line), what to do, and the command to resume with (the Resume with: line), and ends with exit code 1. The following is an example where the second card ended with exit code 4.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
The ERROR: line is a verbatim copy of the error message left by ha-fw-update. The debug tool in the message means recovery using a JTAG debugger and indicates a state that cannot be recovered with host software. Do not try it yourself; follow the actions in the table below. If the card is still not recovered, contact HyperAccel.
Act according to the RESULT of the card that stopped, as follows. The exit codes are defined in Exit codes.
RESULT |
Flash of that card | What to do |
|---|---|---|
exit 1, exit 2, exit 3 |
Not changed by this run | Fix the cause shown in the ERROR: line, and then run the Resume with: command. Actions for each message are in When the update ends before writing |
exit 4, unknown |
May be partially written | Do not reset the card or reboot the host; follow If the exit code is 4 |
exit 5 |
New image. The card is not reset | Following the guidance in When the reset fails, fix the cause and then run sudo ha-hotreset --bdf <PCI address> or power-cycle the host. Then update the remaining cards with the Resume with: command |
exit 6 |
New image. The card is reset | Check as described in When the card does not come back. If that card reports the new version, update the remaining cards with the Resume with: command |
unknownis the case whereha-fw-updateended without leaving a result (it was terminated by a signal or aborted by a Python error). Because it is impossible to tell how far that run wrote the flash, it is handled like exit code 4.- The
Resume with:command passes the cards that have not been run yet in their original order. For exit codes 1 to 4 andunknown, the card that stopped is put back at the front; for exit codes 5 and 6, it is not. Cards that already succeeded are not updated again. - Ctrl+C is delivered to the
ha-fw-updatein progress as well as to the script. If the write for that card has not started yet, the tool ends with exit code 1 and the flash is not changed; if the write has started, the tool finishes the write and the reset and then ends with exit code 6 (5 if the reset fails). The script prints the guidance that matches that exit code. If it stopped between cards, or the card in progress ended successfully, it printsStopped on request..
If the exit code is 4¶
If the host does not receive a response from the firmware, the first line below is logged in the kernel log. If the firmware later responds, even if late, the second line is logged.
1 2 | |
- Do not power off or reboot the host. Do not reset that card with
ha-hotresetor an application reset, either. Unless the card is restarted, it does not boot from a partially written flash. - If
sudo dmesgshows the first line for that card, wait until the second line is logged. In the meantime, the driver refuses requests that send commands to that card and the application's reset requests, and it also refusesha-fw-updateuploading the image. So if you run the script again before then, that card ends with exit code 2 together withupload failed: Device or resource busy, and the flash is not changed. - Run the
Resume with:command. It updates that card again with the same image (only the parts whose contents differ from the flash are rewritten) and, if that succeeds, moves on to the remaining cards. - If it fails again, or the firmware never responds, do not reset the card and contact HyperAccel. Send the JSON in the results directory, the output of
sudo dmesg, and the output ofhyperaccel-smi -qtogether. The background is in When no response is received.
See also¶
- Firmware update — Procedure for a single card, constraints during an update, safety rules
- ha-fw-update — Options, exit codes, JSON output
- ha-hotreset — Hot reset after the update and the conditions for reset
- Supported environments — Combinations of firmware and driver
- sysfs —
fw_version,fw_update_progress