Skip to content

Verifying the installation

This is the procedure for checking, after you install the driver, that the driver has probed all the cards and is ready to operate normally. Check the steps in order, starting with step 1, and if a result differs from what you expect, find the cause at that step. Commands that read the state (lspci, dmesg, cat, hyperaccel-smi) can be run while workloads are running.


1. Check that the cards appear on the PCI bus

1
lspci -D -k -d 20fb:

20fb is the PCI vendor ID of the card. One entry that starts with a PCI address (for example, 0000:01:00.0) should appear for each card, and if the driver is installed correctly, Kernel driver in use: ha is printed with each card.

  • If there are no entries at all, the host did not find the card. Check the card seating, power, and BIOS settings. In this state, the driver cannot probe the card either.
  • If Kernel driver in use: ha is missing, the driver is not loaded or did not probe that card. Find the cause in steps 2 and 3.
  • If Kernel driver in use shows a different driver, that driver is using the card.

2. Check that the module is loaded

1
2
3
lsmod | grep '^ha '
cat /sys/module/ha/version
modinfo -F version ha
  • A line that starts with ha should appear in the lsmod output.
  • /sys/module/ha/version is the version of the driver that is currently loaded, and modinfo -F version ha is the version of the module file installed for the running kernel. After a new installation, both should match the version you installed.
  • If the two versions differ, the old driver is still loaded. See Upgrading and removal.

Also check that the module for the running kernel was installed with DKMS.

1
2
uname -r
dkms status | grep '^ha'

The entry with the same kernel version as uname -r should show installed.

If the module is not loaded, load it manually and look at the kernel log.

1
2
sudo modprobe ha
sudo dmesg | tail -n 30

If loading is rejected on a host with Secure Boot enabled, see Installation.

Automatic loading after a reboot

The installation creates /etc/modules-load.d/ha.conf so that the module is loaded at boot. If this is a host where you have just installed the driver for the first time, reboot once and check this step again.

3. Check the probe result in the kernel log

Filter the log to show only the messages that the driver left.

1
sudo dmesg | grep -E '(^|\] )ha[ :]'

Use the messages below to judge whether the card was probed normally.

1
sudo dmesg -t | grep -E 'firmware ready after|ABI protocol|Probe successful|Module initialized'

Example output (one card):

1
2
3
4
ha 0000:01:00.0: firmware ready after 0 ms
ha 0000:01:00.0: ABI protocol v8 (driver v9, min v6)
ha 0000:01:00.0: Probe successful: /dev/ha0 created
ha: Module initialized
Message Meaning
firmware ready after <N> ms The firmware of the card has finished getting ready
ABI protocol v<N> (driver v<N>, min v<N>) The command protocol version of the firmware is at least the minimum version that the driver requires
Probe successful: /dev/haN created The driver probed the card and created the device node /dev/haN
ha: Module initialized Module loading has finished

There should be one of each of the first three messages for every card. If warnings or errors appear along with them, check their meaning in Kernel log messages. The messages that appear when the firmware is not ready are in step 8.

If these messages have scrolled out of dmesg because the module was loaded a long time ago, look for the same messages in the output of sudo journalctl -k -b.

4. Check the device nodes

1
ls -l /dev/ha*

There should be one /dev/haN node for each card. If the default udev rule has been applied, the permissions are crw-rw-rw- (0666).

1
crw-rw-rw- 1 root root 239, 0 Oct 16 09:12 /dev/ha0

The device numbers and the time differ from system to system.

Result Meaning and action
There are no nodes The driver did not probe the card. Check the kernel log from step 3
There are fewer nodes than cards Some cards were not probed. See step 7
The permissions are not crw-rw-rw- If you have not changed the udev rule yourself, the rule has not been applied. Reapply it with the commands below
1
2
sudo udevadm control --reload-rules
sudo udevadm trigger --subsystem-match=ha

With the default permissions 0666, all local users of the host can use the card. For rules that restrict access, see Restricting access.

5. Check the sysfs state

The driver provides status nodes under /sys/class/ha/haN/ for each card. Every user can read them, so root privileges are not required.

1
2
3
4
ls /sys/class/ha/
cat /sys/class/ha/ha0/health
cat /sys/class/ha/ha0/fw_version
cat /sys/class/ha/ha0/serial
Node When normal When different
health ok If it is degraded or faulted, check its meaning in Telemetry and the action in Troubleshooting
fw_version Firmware version in the <major>.<minor>.<patch> format If it is unknown, the firmware is not ready. See step 8
serial The unique card ID that starts with HA- It is unknown if the firmware is not ready. If it is unknown even though fw_version shows a version, this card or firmware does not provide an ID, and this does not affect the use of the device

Check whether the firmware is ready with fw_version

health is determined by the errors that the driver has observed and the FATAL state. A card whose firmware is not ready yet is also ok if there are no errors.

For the definitions of the other nodes, see sysfs nodes.

6. Check with hyperaccel-smi

1
hyperaccel-smi

This is the first part of the output. The values are examples.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
+-----------------------------------------------------------------------------------------+
| HYPERACCEL-SMI 1.0.0      Tue Oct 06 11:58:59 2026                                      |
| Firmware Version: 1.0.5        Driver: ha                                               |
+-------------------------------------------+-----------------------+---------------------+
| LPU  Name                 Serial          | Bus-Id                | Health              |
| Temp   Power        Util(core)            | Memory-Usage (DRAM)   | Fw-Err / Timeouts   |
+===========================================+=======================+=====================+
|   0  HyperAccel LPU   HA-0a1b2c3d-000…    | 0000:01:00.0          | OK                  |
| 38C    21.5 W      0.0 %                  | 0.00 / 128.00 GiB     | 0 / 0               |
+-------------------------------------------+-----------------------+---------------------+
  • The firmware version should appear in Firmware Version at the top. This value is the version of the card with the lowest number, so view the per-card versions with hyperaccel-smi -q.
  • Each card should have a two-line row, and Bus-Id should match the PCI address you saw in step 1.
  • Health should be OK.

hyperaccel-smi runs on Python 3. If no card is visible, it ends with exit code 2, so you can use this to verify the installation in a script.

1
hyperaccel-smi > /dev/null; echo $?

If it prints 0, at least one card is visible. For the options and output items, see hyperaccel-smi.

7. Check multiple cards

If there are several cards, check that the number of cards visible on the PCI bus equals the number of devices that the driver created.

1
2
lspci -D -d 20fb: | wc -l
ls -d /sys/class/ha/ha* | wc -l

Look at the device number, PCI address, health, and fw_version of each card, one line per card.

1
2
3
for d in /sys/class/ha/ha*; do
  echo "$(basename "$d") $(basename "$(readlink -f "$d/device")") $(cat "$d/health") $(cat "$d/fw_version")"
done
1
2
ha0 0000:01:00.0 ok 1.0.5
ha1 0000:02:00.0 ok 1.0.5

All cards should be ok and show a firmware version. The LPU number and Bus-Id of hyperaccel-smi show the same correspondence.

  • The driver probes up to 8 cards on one host.
  • If a card has no node, search the kernel log for the PCI address of that card. Example: sudo dmesg | grep 0000:02:00.0
  • For how the numbers are assigned, see Multiple devices.

8. When the firmware is not ready

When the driver probes a card, it waits a few seconds for the firmware to become ready. Even if the firmware is not ready within that time, the driver still probes the card and creates /dev/haN, but until the firmware is ready, the following is observed.

  • A warning that starts with firmware boot sentinel not seen within or firmware never reported ready is logged in the kernel log.
  • fw_version and serial are unknown. Values that the firmware sends, such as temperature and power, are also unknown until the firmware starts sending them. health can be ok.
  • In the row for that card in hyperaccel-smi, Serial is N/A, and in hyperaccel-smi -q, Firmware Version and Serial of that card are unknown. If the temperature and power are unknown, Temp and Power are also N/A.
  • If an application tries to use the device, an error saying that the device is not ready is returned.
  • After a few seconds, an error that starts with sys_mon: no response for may be logged in the kernel log (dmesg). It means that polling to read temperature, power, and utilization information has stopped; when you read fw_version after the firmware becomes ready, polling restarts and sys_mon: resuming telemetry polling is logged.

If the firmware finishes getting ready later, you do not need to reload the driver. If you read fw_version again after a short while and a version appears, you can use the device from then on. At that point, firmware booted after probe gave up; CSR-derived telemetry refreshed is logged in the kernel log.

If fw_version stays unknown, see Troubleshooting and FATAL and recovery.


When the results differ from what you expect

Symptom What to check
The card is not in lspci Card seating, power, BIOS settings
The module is not loaded The error message from sudo modprobe ha, the kernel log, and the Secure Boot settings
The card is visible but /dev/haN is missing Find the PCI address of that card in the kernel log. If ABI protocol mismatch is present, the combination of the firmware and driver versions does not match. See Supported environments
The node permissions are not crw-rw-rw- Reapply the udev rule as described in step 4
fw_version stays unknown Step 8
health is not ok Troubleshooting
The loaded version differs from the installed version Upgrading and removal

When you request support, save the diagnostic information with Collecting diagnostic information for a problem and send it along. What else to include is described in Requesting support.


See also