Collecting telemetry with Prometheus¶
This tutorial is for operators who monitor servers with Prometheus and node_exporter. If you follow it to the end, you can use node_exporter alone to collect each card's temperature, voltage, current, and power along with health, memory usage, job counts, error counters, and utilization, view the values grouped by card name, and apply basic alert rules. It does not change any driver settings. What each node means and the conditions under which unknown appears are described in Telemetry.
Before you begin¶
Temperature, voltage, current, and power are picked up by node_exporter's hwmon collector without any configuration (step 1); for the other values, the script in this tutorial reads them from /sys/class/ha/haN/ and writes them to a file, which node_exporter's textfile collector then reads (step 2). You need the following.
- A host where the driver has probed the cards. See Verifying the installation.
- node_exporter running on the host. This tutorial uses the default port 9100. The hwmon collector and the textfile collector are enabled by default, but the textfile collector reads the
*.promfiles in a directory only if you specify the directory with--collector.textfile.directory. This tutorial uses/var/lib/node_exporter/textfile_collectoras the example.
1. Check the hwmon values¶
If the kernel supports hwmon, the driver registers an hwmon device whose name is ha for each card. First check that the hwmon devices exist.
1 | |
As many paths as there are cards should appear. If none appear, see hwmon integration. In that case temperature, voltage, current, and power are not collected, but you can proceed from step 2 as is.
node_exporter exposes the LPU's hwmon values as the following metrics and converts the values to base units.
| hwmon file | Metric | sensor label |
Unit |
|---|---|---|---|
temp1_input–temp15_input |
node_hwmon_temp_celsius |
temp1–temp15 |
°C |
in0_input–in3_input |
node_hwmon_in_volts |
in0–in3 |
V |
curr1_input–curr4_input |
node_hwmon_curr_amps |
curr1–curr4 |
A |
power1_input |
node_hwmon_power_watt |
power1 |
W |
- The name of each channel (
peak,avg,top_core,total, and so on) is in thelabellabel ofnode_hwmon_sensor_label. The peak temperature istemp1(peak), and the sum of rail power ispower1(total). For the full mapping, see sysfs nodes. - Cards are distinguished by the
chiplabel. node_exporter buildschipfrom the sysfs path of the PCI device to which the hwmon device is attached. Find thechipof an LPU innode_hwmon_chip_namesas the line withchip_name="ha"(you check this in step 4). - If the driver cannot trust a value, the hwmon file returns a read error and node_exporter leaves that sensor out of that scrape. The value does not drop to 0; the samples for that interval appear empty.
2. Install the textfile script¶
The script below exports the values that are not in hwmon. The script reads the nodes in /sys/class/ha/haN/ and writes an ha.prom file in the textfile collector format. It is written in POSIX sh, so it also runs on hosts without bash.
| Metric | Type | Source | Value |
|---|---|---|---|
ha_cards |
gauge | /sys/class/ha/haN |
Number of cards. 0 if the module is not loaded |
ha_driver_info |
gauge | /sys/module/ha/version |
1. Driver version in the version label |
ha_card_info |
gauge | fw_version, serial, device link |
1. chip, fw_version, and serial labels |
ha_health |
gauge | health |
One line per state label (ok, degraded, faulted, unknown). Only the current state is 1 |
ha_dram_used_bytes, ha_dram_free_bytes |
gauge | dram_used, dram_free |
Bytes |
ha_inflight, ha_streams |
gauge | inflight, stream_count |
Count |
ha_devcmd_timeouts_total, ha_devcmd_fw_errors_total, ha_hdma_timeouts_total |
counter | devcmd_timeouts, devcmd_fw_errors, hdma_timeouts |
Number of times |
ha_util_ratio |
gauge | util |
One line per ip label. 0–1 (percentage ÷ 100) |
The metrics other than ha_cards and ha_driver_info carry the card name card (haN) and the PCI address pci as labels. The haN number can change depending on the card configuration or probe order, so use pci or the serial of ha_card_info when you track a card over a long period.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | |
Install the script and run it once to try it.
1 2 3 4 | |
The script follows these rules.
- Unknown values are not turned into 0. If a node that should yield a number yields a non-numeric value such as
unknown, that metric of that card is left out of this file. Writing 0 would turn "the value is unknown" into "the value is 0" and distort graphs and alerts. Ifutilisunknown, allha_util_ratiosamples of that card are omitted. String values are copied as they are, so ifhealthisunknown,state="unknown"becomes 1, and before the firmware is ready it appears asfw_version="unknown". - It reads only nodes suited to periodic collection. Every node it reads reports a value that the driver maintains. The exceptions are
fw_versionandserial. Whilefw_versionisunknown, the driver checks again whether the firmware is ready when these two nodes are read (Collection interval and cost).fw_update_progressis not read because the driver reads it directly from the device on every read, andhelp(description text) andbypass_range_check(debug setting) are not read either because they are not values to collect. - The file is written even if there are no cards. If the module is not loaded or there are no cards, it writes
ha_cards 0without per-card samples. This is so that the previous values of a card that has disappeared do not appear to remain. - The file is replaced in one step. It finishes writing a temporary file in the same directory and then replaces the target with
mv, so node_exporter never reads a half-written file. The temporary file name does not end in.prom, so it is not collected. - The sysfs path can be changed. If the host's
/sysis mounted at a different path (for example,/host/sysinside a container), specify that path in the environment variableSYSFS.
A node that yields a single integer can be exported by adding one more num line. For example, on a kernel without hwmon, adding num ha_temp_celsius gauge 'Peak on-die temperature.' temp collects the peak temperature (°C).
3. Run it periodically¶
Run it every 15 seconds with a systemd timer. Create a service unit and a timer unit.
1 2 3 4 5 6 7 | |
1 2 3 4 5 6 7 8 9 10 11 | |
1 2 3 | |
- If you do not specify
AccuracySec, the default of 1 minute applies, and systemd can delay the run time by up to 1 minute. - The value that Prometheus scrapes is the value from the last file written, so it can lag by up to the timer interval. Match the interval to Prometheus's scrape interval.
- The service runs as root. If there is a user who can write to the textfile directory, you can also specify
User=in[Service]to run it as that user.
4. Check the collection¶
Check that node_exporter has read the file and the hwmon values.
1 2 | |
This is an example of the output on a host with one card. node_exporter outputs the labels sorted by name.
1 2 3 4 5 6 7 8 | |
- If there are no
ha_metrics at all, check that node_exporter's--collector.textfile.directoryis the same as the directory you gave the script. - If
node_textfile_scrape_erroris1, node_exporter could not read the textfile directory or a.promfile in it. The cause is recorded in the node_exporter log. - The
chipvalue ofha_card_infomust equal thechipvalue ofnode_hwmon_chip_names. The join in the next section is based on this value. If there is nonode_hwmon_chip_namesline, see step 1.
Viewing hwmon values by card name¶
hwmon metrics have no card name. If you join them with ha_card_info on chip, you can attach the card and pci labels. The following is a PromQL query that gets the peak temperature of each card.
1 2 | |
The value of ha_card_info is always 1, so the result is the temperature unchanged. Servers with the same configuration can have the same chip value, so instance is matched as well. For voltage, current, and power, use the same expression with the metric name and the sensor condition changed.
Alert rule example¶
This is an example that translates the recommendations in Alert criteria into Prometheus alert rules. Save it as a rule file (for example, ha-lpu.rules.yml), check the syntax with promtool check rules ha-lpu.rules.yml, and then add it to rule_files in the Prometheus configuration.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | |
| Alert | Meaning and response |
|---|---|
LpuFaulted |
The card is FATAL or the last reset has failed. Before you reset, save the diagnostic information with Collecting diagnostic information for a problem and follow Recovering an unresponsive card |
LpuErrorCountersIncreased |
A new timeout or firmware error has occurred within the last 10 minutes. Check last_hang and the kernel log. Counters restart from 0 when the driver probes the card again, so the alert is set on the increase, not on the value |
LpuUtilUnknown |
util has been unknown for more than 5 minutes. Causes include the poller having stopped or FATAL continuing (Conditions under which unknown appears) |
LpuCardMissing |
Fewer cards are visible than expected. The module is not loaded, the driver failed to probe some cards, or a card has disappeared from the PCI bus. The count also drops briefly while the module is reloaded or ha-hotreset runs |
LpuTextfileStale |
ha.prom has not been updated for more than 2 minutes, or node_exporter could not read the textfile. node_exporter keeps exposing the last file even if the timer stops, so this alert is how you identify stale values |
Do not set an alert on degraded
degraded persists until the driver probes the card again if even a single error has occurred. Judge whether the problem is continuing by the increase of counters, as LpuErrorCountersIncreased does, and use degraded for the status display on a dashboard.
See also¶
- Telemetry — Update method,
unknown,healthverdict, alert criteria - sysfs nodes — Output format of each node and hwmon channels
- Hang detection and the watchdog — Interpreting
last_hang - FATAL and recovery
- Building a card health check
- Collecting diagnostic information for a problem
- Recovering an unresponsive card