Skip to content

Examples and tutorials

The tasks you perform often while operating the driver are organized here so that you can follow them from start to finish. Each tutorial includes example scripts and settings that you can use as they are or adapt. Why the features behave as they do is explained in Operations guide; command options and exit codes are on the page of each tool (hyperaccel-smi, ha-hotreset, ha-fw-update); and the definitions of the nodes are in sysfs nodes.


Tutorials

Tutorial What you will be able to do Example
Collecting telemetry with Prometheus Collect hwmon sensor values and the card state, memory, error counters, and utilization with Prometheus, and set up alerts ha-prom-textfile.sh, systemd timer, alert rules
Building a card health check Check that the driver is loaded, the number of cards, and the health and fw_version of each card, and report the result through the exit code ha-healthcheck.sh, systemd timer
Collecting diagnostic information for a problem Before resetting a card, save the information to send to the support team in a single archive ha-collect-diag.sh
Recovering an unresponsive card Follow the process from checking the symptom through resetting one card with ha-hotreset to confirming the recovery Command examples
Updating firmware on several cards in sequence Update the cards one at a time with the same firmware, and stop on the spot if a problem occurs ha-fw-update-all.sh

When a card has a problem, collect the diagnostic information first and then recover the card. This is because a reset erases the state that shows the cause.


Before you use the example scripts

  • The example scripts are not included in the driver package. Save the content of the page to a file and use it.
  • The scripts are written in POSIX sh syntax, so they also work with a /bin/sh that is not bash (for example, dash).
  • The health check and monitoring examples only read sysfs and do not open the device nodes, so they can run alongside workloads. The health check runs without root privileges.
  • In an environment where the /sys of the host is mounted at a different path (for example, a container), you can tell the scripts that path with the environment variable SYSFS.
  • Adapt values such as paths, the number of cards, and intervals to your environment.

See also