haprof CLI¶
1 2 | |
haprof runs an application under the profiler, and haprof merge combines per-rank traces into one. In the first
form, -- separates the profiler options from the command and cannot be omitted. When to use each option is
covered in Capturing a Trace.
Contents¶
| Section | Objective |
|---|---|
| 1. Option Status | What each option does and whether it works |
| 2. Output Files | Naming rules and formats |
| 3. Exit Codes | Which exit code is returned |
| 4. Merging Per-Rank Traces | Merging with haprof merge and the straggler report |
| 5. Examples | Common commands |
1. Option Status¶
Legend — ✅ works · ❌ parsed but ignored
Ignored options produce no warning
Options marked ❌ are accepted on the command line but not passed to the profiler. The run proceeds as if the flag were absent, with no warning or error, so check the table below before using one.
Tracing Options¶
| Option | Status | Effect |
|---|---|---|
--runtime-trace |
✅ | Adds the device kernel and memory transfer timeline |
--device-trace |
✅ | Adds per-kernel hardware counters and the signal logger on top of the above. Includes --runtime-trace |
--sys-trace |
❌ | Ignored |
--kernel-trace |
❌ | Ignored |
--memory-copy-trace |
❌ | Ignored |
--memory-allocation-trace |
❌ | Ignored |
--driver-trace |
❌ | Ignored |
--overhead-trace |
❌ | Ignored |
--external-correlation |
❌ | Ignored |
--marker-trace |
❌ | Ignored |
The per-domain flags are not yet passed to the tool. Host-side activity kinds such as runtime (including memory
allocation), driver, marker, and external correlation are always collected in full regardless of the flags, and the
device kernel and memory transfer timeline is turned on only by --runtime-trace or --device-trace. The
profiler's own execution overhead is not yet recorded as a separate activity record.
Output Options¶
| Option | Status | Effect |
|---|---|---|
-d, --output-directory |
✅ | Directory to write traces to. Default . (where haprof was run). Passed as an absolute path, so it holds even if the target calls chdir |
-o, --output-file |
✅ | File name without extension. A path is rejected. Default <hostname>_<pid>_<starttime>. With --profile-children it becomes a prefix: <name>_<hostname>_<pid>_<starttime> |
-f, --output-format |
✅ | A comma-separated subset of pftrace, json, or all (default). Unknown formats are rejected |
--log-level |
✅ | Tool log level error / warning (default) / info |
Process Control Options¶
| Option | Status | Effect |
|---|---|---|
--profile-children |
✅ | Also profiles child processes that the target forks or execs. Creates one trace per process. Warns if -f does not include json |
--preload |
✅ | Library paths (colon : separated) to put in LD_PRELOAD before the tool library |
--no-execv |
✅ | Runs the target as a subprocess instead of replacing the launcher with the target command (execv). Passes the exit code through |
Detailed settings that have no dedicated CLI flag, such as device selection, are controlled through environment variables.
2. Output Files¶
Each profiled process produces the following files (changeable with -f, -o, -d).
1 2 | |
| Format | Viewer |
|---|---|
.pftrace |
ui.perfetto.dev. Recommended |
.json |
chrome://tracing and other Chrome trace tools. Input to the haprof merge report |
Both files are generated from the same records. The start time is part of the name so that a pid reused during a long run does not overwrite the trace of an earlier process.
Written at process exit
A process killed with SIGKILL, or one that could not run its exit path, leaves no usable trace.
3. Exit Codes¶
haprof replaces itself with the target through execvpe, so the process exit code is the target's. With
--no-execv, the target runs as a subprocess and its exit code is passed through explicitly.
If there is no command after --, the tool library is missing, or an option value is invalid (a path for -o, an
unknown format for -f, and so on), haprof exits with a nonzero code before running anything.
4. Merging Per-Rank Traces¶
In a multi-rank run, each process produces its own trace file. The haprof merge command merges the per-rank trace
files collected with --profile-children into a single timeline, analyzes the per-rank entry delay for each
matching collective, and produces a straggler diagnostic report.
Command Options¶
| Option | Default | Effect |
|---|---|---|
traces... |
All traces under the -d path |
Files to merge |
-d, --directory |
. |
Directory to search for trace files |
-o, --output |
merged.pftrace |
Path of the merged trace file |
--no-trace |
— | Produces only the report, without a merged trace file |
--report |
Depends on format (text goes to standard output, json to straggler.json) |
Report output path |
--report-format |
text |
Report format (text, json, both) |
--top |
20 |
Number of top collective operations listed in the text report |
Output Specification¶
| Output | Input source | Contents |
|---|---|---|
Merged .pftrace |
Per-rank .pftrace files |
A single timeline concatenated ordered by clock reference, earliest first. Each rank gets its own lane |
| Straggler report | Per-rank .json files |
Per-collective entry skew, duration variance, last-arriving and last-finishing ranks, and a per-rank delay frequency histogram |
Clock differences across machines
Traces collected on the same machine share the system clock and align exactly. When traces from different
machines are merged, the top of the report shows a warning that this is a cross-machine merge
(cross-node: yes) together with the per-process clock offsets. The clocks themselves are not corrected, so
the spread of these offsets is the error margin of every delay analysis in the report.
5. Examples¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | |