Skip to content

Updating firmware on several cards in sequence

Update all the LPUs installed in the host to the same firmware image. The example script updates the cards one at a time in order with ha-fw-update and records the results; if any card does not finish normally, it stops without starting the next card. The procedure for updating a single card, the constraints during an update, and the safety rules are in Firmware update; this page covers only how to apply that procedure to several cards in sequence. You need root privileges on the host and ha-fw-update and ha-hotreset from the driver package (Install location). The example updates three cards (0000:01:00.0, 0000:41:00.0, 0000:81:00.0) from firmware 1.0.5 to 1.0.6. Replace the addresses, versions, and image names with your actual values.


Before you begin

  1. Check that the new firmware matches the driver. Check the combination of the new firmware and the installed driver in Supported environments. A card whose running firmware uses a command protocol below v7 cannot be updated with ha-fw-update.
  2. Stop all workloads that use the cards you are updating. If any process has the /dev/haN of a card open, the update of that card ends with exit code 2, and the script stops there as well. Monitoring that only reads sysfs, such as hyperaccel-smi, can be left running. If you update only some of the cards, also stop jobs that run on other cards and exchange data with those cards (When one card is reset or removed).
  3. Check that each card can be hot reset. If sudo ha-hotreset --list marks any card as NOT resettable, leave that card out by giving only the PCI addresses of the remaining cards when you run the script. If you give no addresses, all registered cards are updated. Update a card you left out separately, as described in Resetting later. A card marked the path is masked for the reset is reset with the path masked during the reset. On a platform you are using for the first time that has such a card, check with sudo ha-hotreset --check-mask --bdf <PCI address> that the masking holds before you start. On a platform that monitors errors through another path, the entire host can restart when that card is reset, even with the path masked (Path masking). --list evaluates only the parent port, so for a card that has other devices under the same parent bridge, the reset is refused after the write finishes and the script stops with exit code 5.
  4. Leave the host as it is until all cards finish. Do not power off or reboot the host during the update. For a remote session, run the script inside tmux or screen. Read the Safety rules first.
  5. Record the current versions. The script's -n prints the PCI address, node, and firmware version of each card (1. Check the target cards).

The script handles only cards registered with the driver, that is, cards that have a /dev/haN. Compare against the number of cards from lspci -D -d 20fb: to check that every installed card is registered. The rules for numbering and registration are in Card numbers.


Script

  • Cards are identified by PCI address. If you give no address, all registered cards are selected, using the PCI address that the /sys/class/ha/haN/device link points to. The /dev/haN number is not used because it is not the identity of a card (Card numbers).
  • For each card, ha-fw-update -d <PCI address> -y --json IMAGE is run, and the result JSON is saved as <PCI address>.json in the results directory. ha-fw-update performs the hot reset and the check of the new firmware. The script does not reset cards itself and does not use --force.
  • If the exit code of ha-fw-update is not 0, the script stops and prints what to do for that card and the command to resume with. If exit_code in the result JSON differs from the exit code, that is, if ha-fw-update ended without leaving a result, the result of that card is shown as unknown and the guidance is the same as for exit code 4.
  • On Ctrl+C, SIGTERM, or SIGHUP, the script waits for ha-fw-update of the card in progress to finish and then stops. The next card is not started.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
#!/bin/sh
# ha-fw-update-all.sh: Updates cards one at a time with the same firmware image and stops at the first problem.
# Usage: sh ha-fw-update-all.sh [-n] IMAGE [BDF...]
#   -n   Shows only the cards and their current firmware versions and writes nothing
#   BDF  PCI address of a card to update (for example 0000:01:00.0). If omitted, all cards registered with the driver
# Exit codes: 0 all cards succeeded, 1 stopped or error, 2 incorrect usage
# Environment variables: LOGDIR (directory for the per-card result JSON), SYSFS (sysfs path, default /sys)
set -u
SYSFS=${SYSFS:-/sys}

die() { echo "ERROR: $*" >&2; exit 1; }
row() { printf '%-14s %-9s %-9s %s\n' "$1" "$2" "$3" "$4"; }

# Prints one line "PCI-address node firmware-version" for each registered card.
cards() {
    for d in "$SYSFS"/class/ha/ha*; do
        [ -e "$d/device" ] || continue
        pci=$(basename "$(readlink -f "$d/device")")
        echo "$pci ${d##*/} $(cat "$d/fw_version" 2>/dev/null || echo unknown)"
    done
}

# Prints the current firmware version of the card at that PCI address. If it is not a registered card, prints -.
fwver() {
    cards | awk -v b="$1" '$1 == b { v = $3 } END { print (v == "" ? "-" : v) }'
}

dry=0
[ "${1:-}" = -n ] && dry=1 && shift
[ $# -ge 1 ] || { echo "usage: $0 [-n] IMAGE [BDF...]" >&2; exit 2; }
image=$1
shift
[ -f "$image" ] || die "$image: no such file"
if [ $# -eq 0 ]; then
    while read -r b _; do [ -n "$b" ] && set -- "$@" "$b"; done <<EOF
$(cards)
EOF
fi
[ $# -gt 0 ] || die "no card is registered in $SYSFS/class/ha"
for b in "$@"; do
    [ "$(fwver "$b")" != - ] || die "$b is not a registered card (list them with: sh $0 -n $image)"
done

if [ "$dry" = 1 ]; then
    printf '%-14s %-5s %s\n' BDF NODE FIRMWARE
    for b in "$@"; do
        cards | awk -v b="$b" '$1 == b { printf "%-14s %-5s %s\n", $1, $2, $3 }'
    done
    echo "dry run: nothing written; $# card(s) would be updated with $image"
    exit 0
fi

command -v ha-fw-update >/dev/null || die "ha-fw-update not found in PATH"
log=${LOGDIR:-ha-fw-update-$(date +%Y%m%d-%H%M%S)}
mkdir -p "$log" || die "cannot create $log"

# Ctrl+C or a similar signal makes the script wait for the running ha-fw-update to finish; the next card is not started.
stop=0
trap 'stop=1' INT TERM HUP
total=$#
rc=0
table=$(row BDF BEFORE AFTER RESULT)
while [ $# -gt 0 ]; do
    before=$(fwver "$1")
    [ "$stop" = 0 ] || break
    bdf=$1
    shift
    echo "[$((total - $#))/$total] $bdf: firmware $before, updating"
    ha-fw-update -d "$bdf" -y --json "$image" >"$log/$bdf.json"
    st=$?
    rc=$st res="exit $st"
    # Only an ha-fw-update that ran to completion leaves an exit_code in the JSON equal to its exit status. Otherwise the state is unknown.
    grep -q "\"exit_code\": $st," "$log/$bdf.json" || { rc=unknown; res=unknown; }
    table="$table
$(row "$bdf" "$before" "$(fwver "$bdf")" "$res")"
    [ "$rc" = 0 ] || break
done
echo
echo "$table"
for b in "$@"; do row "$b" "$(fwver "$b")" - "not run"; done
echo "results: $log/"
[ "$rc" = 0 ] && [ $# -eq 0 ] && exit 0
echo
[ "$rc" = 0 ] || sed -n 's/^  "error": "\(.*\)",$/\1/p' "$log/$bdf.json" | awk '{ gsub(/\\n/, "\n       "); gsub(/\\"/, "\""); print "ERROR: " $0 }'
case $rc in
    0) echo "Stopped on request." ;;
    1|2|3)
        echo "$bdf: exit $rc. This run did not change its flash. Fix the cause, then resume."
        set -- "$bdf" "$@" ;;
    5)
        echo "$bdf: exit 5. Its flash holds the new image, but the card was not reset."
        echo "Fix the cause, then run 'ha-hotreset --bdf $bdf' or power cycle the host." ;;
    6)
        echo "$bdf: exit 6. Its flash holds the new image and the card was reset,"
        echo "but the version check did not finish. Check it with: sh $0 -n $image" ;;
    *)
        echo "$bdf: exit status $st. Its flash may be partially written."
        echo "Do NOT reboot the host or reset the card. Update this card again; if dmesg shows"
        echo "'fw_update: no answer from firmware', first wait for 'late completion from firmware'."
        set -- "$bdf" "$@" ;;
esac
[ $# -eq 0 ] || echo "Resume with: sudo sh $0 $image $*"
exit 1
Environment variable Default Meaning
LOGDIR ha-fw-update-<date>-<time> in the current directory The directory in which to save the per-card result JSON (for example, sudo env LOGDIR=/var/tmp/fw-update sh ha-fw-update-all.sh image.gpt)
SYSFS /sys The sysfs path from which the card list and fw_version are read. Change it when the host's /sys is mounted at a different path. ha-fw-update reads /sys regardless of this value

Running the script

1. Check the target cards

-n prints only the cards and their current firmware versions and does not run ha-fw-update. It reads only sysfs, so root privileges are not required.

1
sh ha-fw-update-all.sh -n image.gpt
1
2
3
4
5
BDF            NODE  FIRMWARE
0000:01:00.0   ha0   1.0.5
0000:41:00.0   ha1   1.0.5
0000:81:00.0   ha2   1.0.5
dry run: nothing written; 3 card(s) would be updated with image.gpt
  • To update only some of the cards, give the PCI addresses after the image, in the format that includes the domain. The cards are updated in the order given.
  • If any of the given addresses is not a registered card, the script prints ERROR: <address> is not a registered card and updates no card.
  • To also run the pre-flight check of ha-fw-update for each card, run sudo ha-fw-update -d <PCI address> --dry-run image.gpt for each card. The script has no such option. See Run the pre-flight check.

2. Update the cards

1
sudo sh ha-fw-update-all.sh image.gpt

The script skips the confirmation prompt of ha-fw-update and also does not show the summary and warnings that the tool prints before an update. Run it only after you have checked everything in Before you begin. If all cards succeed, it prints the following and ends with exit code 0.

1
2
3
4
5
6
7
8
9
[1/3] 0000:01:00.0: firmware 1.0.5, updating
[2/3] 0000:41:00.0: firmware 1.0.5, updating
[3/3] 0000:81:00.0: firmware 1.0.5, updating

BDF            BEFORE    AFTER     RESULT
0000:01:00.0   1.0.5     1.0.6     exit 0
0000:41:00.0   1.0.5     1.0.6     exit 0
0000:81:00.0   1.0.5     1.0.6     exit 0
results: ha-fw-update-20261008-101500/
  • The [n/total] line is printed when the update of that card starts. No progress is printed until one card finishes. To see progress, use fw_update_progress of that card from another terminal (for example, cat /sys/class/ha/ha1/fw_update_progress).
  • BEFORE and AFTER are the fw_version read immediately before and immediately after ha-fw-update runs for that card (for a card that was not run, BEFORE is the current value, and for a card that is not registered it is -). RESULT is one of the following: the exit code of ha-fw-update (exit N), the result cannot be determined (unknown), or not run (not run).
  • The results directory holds the result JSON of ha-fw-update for each card. The meaning of fields such as result, exit_code, error, fw_version_before, and fw_version_after is in JSON output.

Verifying after the update

Run sh ha-fw-update-all.sh -n image.gpt again, or check Firmware Version in hyperaccel-smi -q, to confirm that every card reports the new version.

  • If you used an image with the same version number as before the update, it is a success even if AFTER equals BEFORE. In that case version_changed in the result JSON is false (grep '"version_changed"' ha-fw-update-20261008-101500/*.json), and the version number cannot tell the new image apart. See Verifying that the card booted the new firmware.
  • A card whose fw_version is unknown is still booting or its firmware is not ready. Check again after a while, and if it persists, follow When the card does not come back.
  • When the check is complete, restart the workloads you stopped. The hot reset recreates the host's nodes, so also restart the containers that used the updated cards.

When it stops partway

If ha-fw-update ends with an exit code other than 0 for any card, the script does not start the next card. Below the summary table it prints the error message of that card (the ERROR: line), what to do, and the command to resume with (the Resume with: line), and ends with exit code 1. The following is an example where the second card ended with exit code 4.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
[1/3] 0000:01:00.0: firmware 1.0.5, updating
[2/3] 0000:41:00.0: firmware 1.0.5, updating

BDF            BEFORE    AFTER     RESULT
0000:01:00.0   1.0.5     1.0.6     exit 0
0000:41:00.0   1.0.5     1.0.5     exit 4
0000:81:00.0   1.0.5     -         not run
results: ha-fw-update-20261008-101500/

ERROR: firmware did not answer within the budget -- state unknown, do NOT reboot; retry, then use a debug tool (e.g., Trace32)
0000:41:00.0: exit status 4. Its flash may be partially written.
Do NOT reboot the host or reset the card. Update this card again; if dmesg shows
'fw_update: no answer from firmware', first wait for 'late completion from firmware'.
Resume with: sudo sh ha-fw-update-all.sh image.gpt 0000:41:00.0 0000:81:00.0

The ERROR: line is a verbatim copy of the error message left by ha-fw-update. The debug tool in the message means recovery using a JTAG debugger and indicates a state that cannot be recovered with host software. Do not try it yourself; follow the actions in the table below. If the card is still not recovered, contact HyperAccel.

Act according to the RESULT of the card that stopped, as follows. The exit codes are defined in Exit codes.

RESULT Flash of that card What to do
exit 1, exit 2, exit 3 Not changed by this run Fix the cause shown in the ERROR: line, and then run the Resume with: command. Actions for each message are in When the update ends before writing
exit 4, unknown May be partially written Do not reset the card or reboot the host; follow If the exit code is 4
exit 5 New image. The card is not reset Following the guidance in When the reset fails, fix the cause and then run sudo ha-hotreset --bdf <PCI address> or power-cycle the host. Then update the remaining cards with the Resume with: command
exit 6 New image. The card is reset Check as described in When the card does not come back. If that card reports the new version, update the remaining cards with the Resume with: command
  • unknown is the case where ha-fw-update ended without leaving a result (it was terminated by a signal or aborted by a Python error). Because it is impossible to tell how far that run wrote the flash, it is handled like exit code 4.
  • The Resume with: command passes the cards that have not been run yet in their original order. For exit codes 1 to 4 and unknown, the card that stopped is put back at the front; for exit codes 5 and 6, it is not. Cards that already succeeded are not updated again.
  • Ctrl+C is delivered to the ha-fw-update in progress as well as to the script. If the write for that card has not started yet, the tool ends with exit code 1 and the flash is not changed; if the write has started, the tool finishes the write and the reset and then ends with exit code 6 (5 if the reset fails). The script prints the guidance that matches that exit code. If it stopped between cards, or the card in progress ended successfully, it prints Stopped on request..

If the exit code is 4

If the host does not receive a response from the firmware, the first line below is logged in the kernel log. If the firmware later responds, even if late, the second line is logged.

1
2
ha 0000:41:00.0: fw_update: no answer from firmware (<error number>); resets stay refused until it answers or the update is retried
ha 0000:41:00.0: fw_update: late completion from firmware; resets re-enabled
  1. Do not power off or reboot the host. Do not reset that card with ha-hotreset or an application reset, either. Unless the card is restarted, it does not boot from a partially written flash.
  2. If sudo dmesg shows the first line for that card, wait until the second line is logged. In the meantime, the driver refuses requests that send commands to that card and the application's reset requests, and it also refuses ha-fw-update uploading the image. So if you run the script again before then, that card ends with exit code 2 together with upload failed: Device or resource busy, and the flash is not changed.
  3. Run the Resume with: command. It updates that card again with the same image (only the parts whose contents differ from the flash are rewritten) and, if that succeeds, moves on to the remaining cards.
  4. If it fails again, or the firmware never responds, do not reset the card and contact HyperAccel. Send the JSON in the results directory, the output of sudo dmesg, and the output of hyperaccel-smi -q together. The background is in When no response is received.

See also