Skip to content

Environment Variables

The haprof tool library reads environment variables exactly once, when the process starts (before the target program's main()). Changing a value during execution has no effect, so always set them in the shell before the run. The exception is the device-related variables, which are read when the device record ring is set up (right after the first device use, or when a Kineto profile() region opens).

Most settings are specified as CLI options, and the launcher translates them into the variables below and passes them to the tool. Names not listed here are not meant to be set by users.

Most invalid values are silently treated as defaults

  • HAPTI_TOOL_DEVICE_TIMELINE, HAPTI_TOOL_DEVICE_COUNTERS, HAPTI_PROFILE_CHILDREN — Treated as on if the value is present, non-empty, and not 0. HAPTI_PROFILING_ACTIVE is recognized only when the value is exactly 1.
  • HAPTI_TOOL_LOG_LEVEL — Any value other than error or info is treated as warning.
  • HAPTI_COUNTERS — For an unknown block or event, an out-of-range id, or a malformed token, prints [ha_profiling] HAPTI_COUNTERS: ... — skipped to stderr and skips only that token. It does not substitute another event.
  • HAPTI_TOOL_DEVICE_ID — Read as 0 if it is not a number.
  • HAPTI_KINETO_DEVICE — Prints a warning and is ignored if it is not an integer of 0 or greater.
  • HAPTI_DEVICE_RING_BYTES — If it cannot be read as a size, a warning is printed to stderr and the default of 1 MiB is used.

1. How to Set Them

The launcher builds the variables from the CLI options and passes them to the target process. Put only the variables you set yourself before the command.

1
2
HAPTI_TOOL_DEVICE_ID=1 HAPTI_COUNTERS=util:0:mpu_active,util:0:sample_total \
  haprof --device-trace -- python infer.py

When you preload the tool library directly without the launcher, you must also set the variables that the launcher would set. At minimum the sentinel variable must be present for the tool to work.

1
2
HAPTI_PROFILING_ACTIVE=1 HAPTI_TOOL_DEVICE_TIMELINE=1 \
  LD_PRELOAD=/path/to/hapti/libhapti_tool.so python infer.py

The variables must be in the process environment before the profile() region opens.

1
2
3
4
5
import os
os.environ["HAPTI_KINETO_DEVICE"] = "1"   # before entering profile()

import hapti.kineto
hapti.kineto.register()

2. Environment Variable Details

2.1 Variables Set by the Launcher

haprof sets all of the variables below on every run. Values left in the shell therefore do not leak into the tool, and the command line is the complete configuration. You normally do not set them directly.

Variable Set by Purpose
HAPTI_PROFILING_ACTIVE Always Sentinel variable checked by the tool. The tool clears it at startup so that child processes do not inherit profiling and create traces
HAPTI_PROFILE_CHILDREN --profile-children If 1, keeps the sentinel across fork and exec so that children also profile themselves
HAPTI_TOOL_OUTPUT_PATH -d Directory to write traces to (absolute path)
HAPTI_TOOL_OUTPUT_FILE_NAME -o File name without extension. If empty, <hostname>_<pid>_<starttime>
HAPTI_TOOL_OUTPUT_FORMAT -f Comma-separated list of pftrace, json. Both formats if empty or if every value is unknown
HAPTI_TOOL_LOG_LEVEL --log-level error / warning (default) / info
HAPTI_TOOL_DEVICE_TIMELINE --runtime-trace, --device-trace Collects the device kernel and transfer timeline
HAPTI_TOOL_DEVICE_COUNTERS --device-trace Additionally collects per-kernel hardware counters and the signal logger. Includes the timeline
LD_PRELOAD Always Injects the tool library. Placed after the libraries given with --preload
LD_LIBRARY_PATH Always Prepends the package directory so that only one copy of the runtime library is loaded

2.2 Variables You Set Directly

Variable Type Default Applies to Description
HAPTI_TOOL_DEVICE_ID Integer 0 haprof Device whose records are collected. There is no CLI flag
HAPTI_COUNTERS String AXIBRG 8 + DMC 2 + UTIL 10 of core 0 haprof --device-trace Counter events to register. See the format below
HAPTI_KINETO_DEVICE_TRACE Flag On Kineto plugin If 0/false/off, the device timeline is not enabled
HAPTI_KINETO_DEVICE Integer Current device of the thread that opened the region; 0 if none Kineto plugin Device whose timeline is collected
HAPTI_DEVICE_RING_BYTES Size 1M haprof, Kineto plugin Size of the host ring that receives device records. The number can take a K/M/G suffix (1024-based) and is rounded up to 4 KiB. Increase it if a record loss warning appears

HAPTI_COUNTERS format

1
<block>:<instance>:<event>[,<block>:<instance>:<event>...]
Element Value
Block axibrg, cdma, dmc, util, cgdma, mpu (case-insensitive)
Instance Core number (core group number for CGDMA)
Event Event name or numeric id from the Counter Reference
Count Up to 32. The rest are ignored
Off off or none — timeline and signal logger only, no counters
1
2
3
# Matrix unit utilization of core 0 and the CGDMA read path breakdown
HAPTI_COUNTERS=util:0:sample_total,util:0:mpu_active,cgdma:0:inrob_beat,cgdma:0:inrob_stall_supply,cgdma:0:inrob_stall_demand \
  haprof --device-trace -- python infer.py

Device records are collected from only one device. For behavior in multi-device processes and per-rank collection, see Device Tracing: Selecting the Target Device.

3. Settings That Environment Variables Cannot Specify

What you want How
A subset of activity kinds Not possible. All host-side activity kinds are collected (CLI: Option Status)
More than one device per trace Not possible. Use --profile-children per rank
Start and stop during execution Not possible. haprof works per process. For region-level control, use the Kineto plugin
Hardware counters on the Kineto path Not possible. Use haprof --device-trace

The profiled process also runs the rest of the stack, so those settings apply as well.

Layer Variables
Runtime logging HA_LOG_LEVEL, HA_LOG_PATH — see Runtime environment variables
PyTorch backend See torch_ha environment variables
Collective communication LCCL_* — see LCCL environment variables

See Also