Environment Variables¶
The haprof tool library reads environment variables exactly once, when the process starts (before the target
program's main()). Changing a value during execution has no effect, so always set them in the shell before the
run. The exception is the device-related variables, which are read when the device record ring is set up (right
after the first device use, or when a Kineto profile() region opens).
Most settings are specified as CLI options, and the launcher translates them into the variables below and passes them to the tool. Names not listed here are not meant to be set by users.
Most invalid values are silently treated as defaults
HAPTI_TOOL_DEVICE_TIMELINE,HAPTI_TOOL_DEVICE_COUNTERS,HAPTI_PROFILE_CHILDREN— Treated as on if the value is present, non-empty, and not0.HAPTI_PROFILING_ACTIVEis recognized only when the value is exactly1.HAPTI_TOOL_LOG_LEVEL— Any value other thanerrororinfois treated aswarning.HAPTI_COUNTERS— For an unknown block or event, an out-of-range id, or a malformed token, prints[ha_profiling] HAPTI_COUNTERS: ... — skippedto stderr and skips only that token. It does not substitute another event.HAPTI_TOOL_DEVICE_ID— Read as0if it is not a number.HAPTI_KINETO_DEVICE— Prints a warning and is ignored if it is not an integer of 0 or greater.HAPTI_DEVICE_RING_BYTES— If it cannot be read as a size, a warning is printed to stderr and the default of 1 MiB is used.
1. How to Set Them¶
The launcher builds the variables from the CLI options and passes them to the target process. Put only the variables you set yourself before the command.
1 2 | |
When you preload the tool library directly without the launcher, you must also set the variables that the launcher would set. At minimum the sentinel variable must be present for the tool to work.
1 2 | |
The variables must be in the process environment before the profile() region opens.
1 2 3 4 5 | |
2. Environment Variable Details¶
2.1 Variables Set by the Launcher¶
haprof sets all of the variables below on every run. Values left in the shell therefore do not leak into the
tool, and the command line is the complete configuration. You normally do not set them directly.
| Variable | Set by | Purpose |
|---|---|---|
HAPTI_PROFILING_ACTIVE |
Always | Sentinel variable checked by the tool. The tool clears it at startup so that child processes do not inherit profiling and create traces |
HAPTI_PROFILE_CHILDREN |
--profile-children |
If 1, keeps the sentinel across fork and exec so that children also profile themselves |
HAPTI_TOOL_OUTPUT_PATH |
-d |
Directory to write traces to (absolute path) |
HAPTI_TOOL_OUTPUT_FILE_NAME |
-o |
File name without extension. If empty, <hostname>_<pid>_<starttime> |
HAPTI_TOOL_OUTPUT_FORMAT |
-f |
Comma-separated list of pftrace, json. Both formats if empty or if every value is unknown |
HAPTI_TOOL_LOG_LEVEL |
--log-level |
error / warning (default) / info |
HAPTI_TOOL_DEVICE_TIMELINE |
--runtime-trace, --device-trace |
Collects the device kernel and transfer timeline |
HAPTI_TOOL_DEVICE_COUNTERS |
--device-trace |
Additionally collects per-kernel hardware counters and the signal logger. Includes the timeline |
LD_PRELOAD |
Always | Injects the tool library. Placed after the libraries given with --preload |
LD_LIBRARY_PATH |
Always | Prepends the package directory so that only one copy of the runtime library is loaded |
2.2 Variables You Set Directly¶
| Variable | Type | Default | Applies to | Description |
|---|---|---|---|---|
HAPTI_TOOL_DEVICE_ID |
Integer | 0 |
haprof |
Device whose records are collected. There is no CLI flag |
HAPTI_COUNTERS |
String | AXIBRG 8 + DMC 2 + UTIL 10 of core 0 | haprof --device-trace |
Counter events to register. See the format below |
HAPTI_KINETO_DEVICE_TRACE |
Flag | On | Kineto plugin | If 0/false/off, the device timeline is not enabled |
HAPTI_KINETO_DEVICE |
Integer | Current device of the thread that opened the region; 0 if none |
Kineto plugin | Device whose timeline is collected |
HAPTI_DEVICE_RING_BYTES |
Size | 1M |
haprof, Kineto plugin |
Size of the host ring that receives device records. The number can take a K/M/G suffix (1024-based) and is rounded up to 4 KiB. Increase it if a record loss warning appears |
HAPTI_COUNTERS format
1 | |
| Element | Value |
|---|---|
| Block | axibrg, cdma, dmc, util, cgdma, mpu (case-insensitive) |
| Instance | Core number (core group number for CGDMA) |
| Event | Event name or numeric id from the Counter Reference |
| Count | Up to 32. The rest are ignored |
| Off | off or none — timeline and signal logger only, no counters |
1 2 3 | |
Device records are collected from only one device. For behavior in multi-device processes and per-rank collection, see Device Tracing: Selecting the Target Device.
3. Settings That Environment Variables Cannot Specify¶
| What you want | How |
|---|---|
| A subset of activity kinds | Not possible. All host-side activity kinds are collected (CLI: Option Status) |
| More than one device per trace | Not possible. Use --profile-children per rank |
| Start and stop during execution | Not possible. haprof works per process. For region-level control, use the Kineto plugin |
| Hardware counters on the Kineto path | Not possible. Use haprof --device-trace |
4. Related Settings from Other Components¶
The profiled process also runs the rest of the stack, so those settings apply as well.
| Layer | Variables |
|---|---|
| Runtime logging | HA_LOG_LEVEL, HA_LOG_PATH — see Runtime environment variables |
| PyTorch backend | See torch_ha environment variables |
| Collective communication | LCCL_* — see LCCL environment variables |