Skip to content

Collecting telemetry with Prometheus

This tutorial is for operators who monitor servers with Prometheus and node_exporter. If you follow it to the end, you can use node_exporter alone to collect each card's temperature, voltage, current, and power along with health, memory usage, job counts, error counters, and utilization, view the values grouped by card name, and apply basic alert rules. It does not change any driver settings. What each node means and the conditions under which unknown appears are described in Telemetry.


Before you begin

Temperature, voltage, current, and power are picked up by node_exporter's hwmon collector without any configuration (step 1); for the other values, the script in this tutorial reads them from /sys/class/ha/haN/ and writes them to a file, which node_exporter's textfile collector then reads (step 2). You need the following.

  • A host where the driver has probed the cards. See Verifying the installation.
  • node_exporter running on the host. This tutorial uses the default port 9100. The hwmon collector and the textfile collector are enabled by default, but the textfile collector reads the *.prom files in a directory only if you specify the directory with --collector.textfile.directory. This tutorial uses /var/lib/node_exporter/textfile_collector as the example.

1. Check the hwmon values

If the kernel supports hwmon, the driver registers an hwmon device whose name is ha for each card. First check that the hwmon devices exist.

1
grep -l '^ha$' /sys/class/hwmon/hwmon*/name

As many paths as there are cards should appear. If none appear, see hwmon integration. In that case temperature, voltage, current, and power are not collected, but you can proceed from step 2 as is.

node_exporter exposes the LPU's hwmon values as the following metrics and converts the values to base units.

hwmon file Metric sensor label Unit
temp1_input–temp15_input node_hwmon_temp_celsius temp1–temp15 °C
in0_input–in3_input node_hwmon_in_volts in0–in3 V
curr1_input–curr4_input node_hwmon_curr_amps curr1–curr4 A
power1_input node_hwmon_power_watt power1 W
  • The name of each channel (peak, avg, top_core, total, and so on) is in the label label of node_hwmon_sensor_label. The peak temperature is temp1 (peak), and the sum of rail power is power1 (total). For the full mapping, see sysfs nodes.
  • Cards are distinguished by the chip label. node_exporter builds chip from the sysfs path of the PCI device to which the hwmon device is attached. Find the chip of an LPU in node_hwmon_chip_names as the line with chip_name="ha" (you check this in step 4).
  • If the driver cannot trust a value, the hwmon file returns a read error and node_exporter leaves that sensor out of that scrape. The value does not drop to 0; the samples for that interval appear empty.

2. Install the textfile script

The script below exports the values that are not in hwmon. The script reads the nodes in /sys/class/ha/haN/ and writes an ha.prom file in the textfile collector format. It is written in POSIX sh, so it also runs on hosts without bash.

Metric Type Source Value
ha_cards gauge /sys/class/ha/haN Number of cards. 0 if the module is not loaded
ha_driver_info gauge /sys/module/ha/version 1. Driver version in the version label
ha_card_info gauge fw_version, serial, device link 1. chip, fw_version, and serial labels
ha_health gauge health One line per state label (ok, degraded, faulted, unknown). Only the current state is 1
ha_dram_used_bytes, ha_dram_free_bytes gauge dram_used, dram_free Bytes
ha_inflight, ha_streams gauge inflight, stream_count Count
ha_devcmd_timeouts_total, ha_devcmd_fw_errors_total, ha_hdma_timeouts_total counter devcmd_timeouts, devcmd_fw_errors, hdma_timeouts Number of times
ha_util_ratio gauge util One line per ip label. 0–1 (percentage ÷ 100)

The metrics other than ha_cards and ha_driver_info carry the card name card (haN) and the PCI address pci as labels. The haN number can change depending on the card configuration or probe order, so use pci or the serial of ha_card_info when you track a card over a long period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
#!/bin/sh
# Writes LPU telemetry to the ha.prom file that the node_exporter textfile collector reads.
# Usage: ha-prom-textfile.sh [textfile directory]
# If the host's /sys is mounted at a different path, specify that path in SYSFS.
DIR=${1:-/var/lib/node_exporter/textfile_collector}
SYSFS=${SYSFS:-/sys}
export LC_ALL=C

# Prints one line per card: "<common labels> <card directory>".
cards() {
  for d in "$SYSFS"/class/ha/ha*; do
    p=$(cd -P "$d/device" 2>/dev/null && pwd -P) || continue
    printf 'card="%s",pci="%s" %s\n' "${d##*/}" "${p##*/}" "$d"
  done
}
hdr() { printf '# HELP %s %s\n# TYPE %s %s\n' "$1" "$3" "$1" "$2"; }
# Escapes \ and " in label values.
esc() { printf '%s' "$1" | sed 's/[\\"]/\\&/g'; }

# Emits a node that yields one integer as one metric.
# unknown and other non-numeric values are not turned into 0; the sample is dropped instead.
num() { # metric type description node
  hdr "$1" "$2" "$3"
  cards | while read -r l d; do
    v=$(cat "$d/$4" 2>/dev/null) || continue
    case ${v#-} in '' | *[!0-9]*) continue ;; esac
    printf '%s{%s} %s\n' "$1" "$l" "$v"
  done
}

tmp=$(mktemp "$DIR/.ha.prom.XXXXXX") || exit 1
trap 'rm -f "$tmp"' EXIT
{
  hdr ha_cards gauge 'LPU cards found under /sys/class/ha.'
  echo "ha_cards $(cards | wc -l)"

  hdr ha_driver_info gauge 'Loaded ha.ko version. Always 1.'
  v=$(cat "$SYSFS/module/ha/version" 2>/dev/null) &&
    printf 'ha_driver_info{version="%s"} 1\n' "$(esc "$v")"

  hdr ha_card_info gauge 'Card identity. Always 1.'
  cards | while read -r l d; do
    p=$(cd -P "$d/device" 2>/dev/null && pwd -P) || continue
    fw=$(cat "$d/fw_version" 2>/dev/null) || continue
    sn=$(cat "$d/serial" 2>/dev/null) || continue
    # node_exporter hwmon collector's chip label: last two components of the PCI path, '.' becomes '_'
    b=${p%/*}
    chip=$(printf '%s_%s' "${b##*/}" "${p##*/}" | tr . _)
    printf 'ha_card_info{%s,chip="%s",fw_version="%s",serial="%s"} 1\n' \
      "$l" "$chip" "$(esc "$fw")" "$(esc "$sn")"
  done

  hdr ha_health gauge 'Driver health verdict. 1 for the current state.'
  cards | while read -r l d; do
    h=$(cat "$d/health" 2>/dev/null) || continue
    for s in ok degraded faulted unknown; do
      v=0
      [ "$h" = "$s" ] && v=1
      printf 'ha_health{%s,state="%s"} %s\n' "$l" "$s" "$v"
    done
  done

  num ha_dram_used_bytes gauge 'Allocated device DRAM.' dram_used
  num ha_dram_free_bytes gauge 'Free device DRAM.' dram_free
  num ha_inflight gauge 'Accepted work not finished yet.' inflight
  num ha_streams gauge 'Streams with work left.' stream_count
  num ha_devcmd_timeouts_total counter 'Device command timeouts.' devcmd_timeouts
  num ha_devcmd_fw_errors_total counter 'Firmware error responses.' devcmd_fw_errors
  num ha_hdma_timeouts_total counter 'DMA transfer timeouts.' hdma_timeouts

  hdr ha_util_ratio gauge 'Per-IP utilization, 0 to 1.'
  cards | while read -r l d; do
    # util being unknown means there is no "<IP> <percentage>%" line, so no sample is produced.
    awk -v l="$l" '$1 ~ /^[a-z0-9_]+$/ && $2 ~ /^[0-9.]+%$/ {
      printf "ha_util_ratio{%s,ip=\"%s\"} %g\n", l, $1, $2 / 100 }' \
      "$d/util" 2>/dev/null
  done
} >"$tmp"
chmod 0644 "$tmp" && mv -f "$tmp" "$DIR/ha.prom"

Install the script and run it once to try it.

1
2
3
4
sudo install -m 0755 ha-prom-textfile.sh /usr/local/bin/ha-prom-textfile.sh
sudo install -d -m 0755 /var/lib/node_exporter/textfile_collector
sudo /usr/local/bin/ha-prom-textfile.sh /var/lib/node_exporter/textfile_collector
cat /var/lib/node_exporter/textfile_collector/ha.prom

The script follows these rules.

  • Unknown values are not turned into 0. If a node that should yield a number yields a non-numeric value such as unknown, that metric of that card is left out of this file. Writing 0 would turn "the value is unknown" into "the value is 0" and distort graphs and alerts. If util is unknown, all ha_util_ratio samples of that card are omitted. String values are copied as they are, so if health is unknown, state="unknown" becomes 1, and before the firmware is ready it appears as fw_version="unknown".
  • It reads only nodes suited to periodic collection. Every node it reads reports a value that the driver maintains. The exceptions are fw_version and serial. While fw_version is unknown, the driver checks again whether the firmware is ready when these two nodes are read (Collection interval and cost). fw_update_progress is not read because the driver reads it directly from the device on every read, and help (description text) and bypass_range_check (debug setting) are not read either because they are not values to collect.
  • The file is written even if there are no cards. If the module is not loaded or there are no cards, it writes ha_cards 0 without per-card samples. This is so that the previous values of a card that has disappeared do not appear to remain.
  • The file is replaced in one step. It finishes writing a temporary file in the same directory and then replaces the target with mv, so node_exporter never reads a half-written file. The temporary file name does not end in .prom, so it is not collected.
  • The sysfs path can be changed. If the host's /sys is mounted at a different path (for example, /host/sys inside a container), specify that path in the environment variable SYSFS.

A node that yields a single integer can be exported by adding one more num line. For example, on a kernel without hwmon, adding num ha_temp_celsius gauge 'Peak on-die temperature.' temp collects the peak temperature (°C).


3. Run it periodically

Run it every 15 seconds with a systemd timer. Create a service unit and a timer unit.

1
2
3
4
5
6
7
# /etc/systemd/system/ha-prom-textfile.service
[Unit]
Description=Write LPU telemetry for the node_exporter textfile collector

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ha-prom-textfile.sh /var/lib/node_exporter/textfile_collector
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# /etc/systemd/system/ha-prom-textfile.timer
[Unit]
Description=Run ha-prom-textfile.service every 15 seconds

[Timer]
OnBootSec=30s
OnUnitActiveSec=15s
AccuracySec=1s

[Install]
WantedBy=timers.target
1
2
3
sudo systemctl daemon-reload
sudo systemctl enable --now ha-prom-textfile.timer
systemctl list-timers ha-prom-textfile.timer
  • If you do not specify AccuracySec, the default of 1 minute applies, and systemd can delay the run time by up to 1 minute.
  • The value that Prometheus scrapes is the value from the last file written, so it can lag by up to the timer interval. Match the interval to Prometheus's scrape interval.
  • The service runs as root. If there is a user who can write to the textfile directory, you can also specify User= in [Service] to run it as that user.

4. Check the collection

Check that node_exporter has read the file and the hwmon values.

1
2
curl -s http://localhost:9100/metrics |
  grep -E '^(ha_cards|ha_card_info|ha_health|node_textfile_scrape_error)|chip_name="ha"'

This is an example of the output on a host with one card. node_exporter outputs the labels sorted by name.

1
2
3
4
5
6
7
8
ha_card_info{card="ha0",chip="0000:00:01_0_0000:01:00_0",fw_version="1.0.5",pci="0000:01:00.0",serial="HA-1a2b3c4d-5e6f-7a8b-9c0d-1e2f00000000"} 1
ha_cards 1
ha_health{card="ha0",pci="0000:01:00.0",state="degraded"} 0
ha_health{card="ha0",pci="0000:01:00.0",state="faulted"} 0
ha_health{card="ha0",pci="0000:01:00.0",state="ok"} 1
ha_health{card="ha0",pci="0000:01:00.0",state="unknown"} 0
node_hwmon_chip_names{chip="0000:00:01_0_0000:01:00_0",chip_name="ha"} 1
node_textfile_scrape_error 0
  • If there are no ha_ metrics at all, check that node_exporter's --collector.textfile.directory is the same as the directory you gave the script.
  • If node_textfile_scrape_error is 1, node_exporter could not read the textfile directory or a .prom file in it. The cause is recorded in the node_exporter log.
  • The chip value of ha_card_info must equal the chip value of node_hwmon_chip_names. The join in the next section is based on this value. If there is no node_hwmon_chip_names line, see step 1.

Viewing hwmon values by card name

hwmon metrics have no card name. If you join them with ha_card_info on chip, you can attach the card and pci labels. The following is a PromQL query that gets the peak temperature of each card.

1
2
node_hwmon_temp_celsius{sensor="temp1"}
  * on (instance, chip) group_left (card, pci) ha_card_info

The value of ha_card_info is always 1, so the result is the temperature unchanged. Servers with the same configuration can have the same chip value, so instance is matched as well. For voltage, current, and power, use the same expression with the metric name and the sensor condition changed.


Alert rule example

This is an example that translates the recommendations in Alert criteria into Prometheus alert rules. Save it as a rule file (for example, ha-lpu.rules.yml), check the syntax with promtool check rules ha-lpu.rules.yml, and then add it to rule_files in the Prometheus configuration.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
groups:
  - name: hyperaccel-lpu
    rules:
      - alert: LpuFaulted
        expr: ha_health{state="faulted"} == 1
      - alert: LpuErrorCountersIncreased
        expr: |
          increase(ha_devcmd_timeouts_total[10m]) > 0
          or increase(ha_devcmd_fw_errors_total[10m]) > 0
          or increase(ha_hdma_timeouts_total[10m]) > 0
      - alert: LpuUtilUnknown
        expr: ha_card_info unless on (instance, card) ha_util_ratio
        for: 5m
      - alert: LpuCardMissing
        # Replace 4 with the number of cards installed in this host.
        expr: ha_cards < 4
        for: 5m
      - alert: LpuTextfileStale
        expr: |
          time() - node_textfile_mtime_seconds{file=~".*/ha.prom"} > 120
          or node_textfile_scrape_error == 1
Alert Meaning and response
LpuFaulted The card is FATAL or the last reset has failed. Before you reset, save the diagnostic information with Collecting diagnostic information for a problem and follow Recovering an unresponsive card
LpuErrorCountersIncreased A new timeout or firmware error has occurred within the last 10 minutes. Check last_hang and the kernel log. Counters restart from 0 when the driver probes the card again, so the alert is set on the increase, not on the value
LpuUtilUnknown util has been unknown for more than 5 minutes. Causes include the poller having stopped or FATAL continuing (Conditions under which unknown appears)
LpuCardMissing Fewer cards are visible than expected. The module is not loaded, the driver failed to probe some cards, or a card has disappeared from the PCI bus. The count also drops briefly while the module is reloaded or ha-hotreset runs
LpuTextfileStale ha.prom has not been updated for more than 2 minutes, or node_exporter could not read the textfile. node_exporter keeps exposing the last file even if the timer stops, so this alert is how you identify stale values

Do not set an alert on degraded

degraded persists until the driver probes the card again if even a single error has occurred. Judge whether the problem is continuing by the increase of counters, as LpuErrorCountersIncreased does, and use degraded for the status display on a dashboard.


See also