Skip to content

haprof CLI

1
2
haprof [options] -- <application> [args...]
haprof merge [options] [traces...]

haprof runs an application under the profiler, and haprof merge combines per-rank traces into one. In the first form, -- separates the profiler options from the command and cannot be omitted. When to use each option is covered in Capturing a Trace.

Contents

Section Objective
1. Option Status What each option does and whether it works
2. Output Files Naming rules and formats
3. Exit Codes Which exit code is returned
4. Merging Per-Rank Traces Merging with haprof merge and the straggler report
5. Examples Common commands

1. Option Status

Legend — ✅ works · ❌ parsed but ignored

Ignored options produce no warning

Options marked ❌ are accepted on the command line but not passed to the profiler. The run proceeds as if the flag were absent, with no warning or error, so check the table below before using one.

Tracing Options

Option Status Effect
--runtime-trace ✅ Adds the device kernel and memory transfer timeline
--device-trace ✅ Adds per-kernel hardware counters and the signal logger on top of the above. Includes --runtime-trace
--sys-trace ❌ Ignored
--kernel-trace ❌ Ignored
--memory-copy-trace ❌ Ignored
--memory-allocation-trace ❌ Ignored
--driver-trace ❌ Ignored
--overhead-trace ❌ Ignored
--external-correlation ❌ Ignored
--marker-trace ❌ Ignored

The per-domain flags are not yet passed to the tool. Host-side activity kinds such as runtime (including memory allocation), driver, marker, and external correlation are always collected in full regardless of the flags, and the device kernel and memory transfer timeline is turned on only by --runtime-trace or --device-trace. The profiler's own execution overhead is not yet recorded as a separate activity record.

Output Options

Option Status Effect
-d, --output-directory ✅ Directory to write traces to. Default . (where haprof was run). Passed as an absolute path, so it holds even if the target calls chdir
-o, --output-file ✅ File name without extension. A path is rejected. Default <hostname>_<pid>_<starttime>. With --profile-children it becomes a prefix: <name>_<hostname>_<pid>_<starttime>
-f, --output-format ✅ A comma-separated subset of pftrace, json, or all (default). Unknown formats are rejected
--log-level ✅ Tool log level error / warning (default) / info

Process Control Options

Option Status Effect
--profile-children ✅ Also profiles child processes that the target forks or execs. Creates one trace per process. Warns if -f does not include json
--preload ✅ Library paths (colon : separated) to put in LD_PRELOAD before the tool library
--no-execv ✅ Runs the target as a subprocess instead of replacing the launcher with the target command (execv). Passes the exit code through

Detailed settings that have no dedicated CLI flag, such as device selection, are controlled through environment variables.


2. Output Files

Each profiled process produces the following files (changeable with -f, -o, -d).

1
2
<hostname>_<pid>_<starttime>.pftrace
<hostname>_<pid>_<starttime>.json
Format Viewer
.pftrace ui.perfetto.dev. Recommended
.json chrome://tracing and other Chrome trace tools. Input to the haprof merge report

Both files are generated from the same records. The start time is part of the name so that a pid reused during a long run does not overwrite the trace of an earlier process.

Written at process exit

A process killed with SIGKILL, or one that could not run its exit path, leaves no usable trace.


3. Exit Codes

haprof replaces itself with the target through execvpe, so the process exit code is the target's. With --no-execv, the target runs as a subprocess and its exit code is passed through explicitly.

If there is no command after --, the tool library is missing, or an option value is invalid (a path for -o, an unknown format for -f, and so on), haprof exits with a nonzero code before running anything.


4. Merging Per-Rank Traces

In a multi-rank run, each process produces its own trace file. The haprof merge command merges the per-rank trace files collected with --profile-children into a single timeline, analyzes the per-rank entry delay for each matching collective, and produces a straggler diagnostic report.

Command Options

Option Default Effect
traces... All traces under the -d path Files to merge
-d, --directory . Directory to search for trace files
-o, --output merged.pftrace Path of the merged trace file
--no-trace — Produces only the report, without a merged trace file
--report Depends on format (text goes to standard output, json to straggler.json) Report output path
--report-format text Report format (text, json, both)
--top 20 Number of top collective operations listed in the text report

Output Specification

Output Input source Contents
Merged .pftrace Per-rank .pftrace files A single timeline concatenated ordered by clock reference, earliest first. Each rank gets its own lane
Straggler report Per-rank .json files Per-collective entry skew, duration variance, last-arriving and last-finishing ranks, and a per-rank delay frequency histogram

Clock differences across machines

Traces collected on the same machine share the system clock and align exactly. When traces from different machines are merged, the top of the report shows a warning that this is a cross-machine merge (cross-node: yes) together with the per-process clock offsets. The clocks themselves are not corrected, so the spread of these offsets is the error margin of every delay analysis in the report.


5. Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# Host side only
haprof -- python infer.py

# Device timeline
haprof --runtime-trace -- python infer.py

# Device timeline + hardware counters
haprof --device-trace -- python infer.py

# Specify location, name, format → traces/decode.pftrace
haprof -d traces -o decode -f pftrace --device-trace -- python infer.py

# Multi-rank. One trace per rank, then merge
haprof --profile-children --device-trace -d traces -- torchrun --nproc-per-node 4 serve.py
haprof merge -d traces -o traces/merged.pftrace --report-format both

# Select a nonzero device (there is no CLI flag)
HAPTI_TOOL_DEVICE_ID=1 haprof --device-trace -- python infer.py

See Also