2. Capturing a Trace¶
This chapter covers how to run a program under the profiler and the files it produces. It assumes the key concepts from Chapter 1 Overview.
Contents¶
| Section | Objective |
|---|---|
| 2.1 Basic Form | Structure of the haprof command |
| 2.2 Decisions at Run Time | Guide to choosing command-line options |
| 2.3 Output Files | File naming rules, location, and formats |
| 2.4 Multi-Process Runs | Collecting and merging multi-rank traces |
| 2.5 Choosing the Collection Scope | The lowest-cost command for each analysis goal |
| 2.6 Making a Trace You Can Analyze | Adding annotations and warming up |
| 2.7 Overhead | Performance impact of each mode |
2.1 Basic Form¶
1 | |
-- separates the profiler options from the command to run and cannot be omitted.
1 2 | |
haprof sets up environment variables and then replaces the process image with the target command (execvpe). The
system process list therefore shows the running user program as is, and the process exit code is the program's
return value.
2.2 Decisions at Run Time¶
There are three things to decide on the command line.
- Whether to include device data —
--runtime-trace(timeline) or--device-trace(timeline and hardware counters). Covered in detail in Chapter 3 Device Tracing. - Whether to trace child processes — The
--profile-childrenoption. Covered in Multi-Process Runs. - Output location and file format — The
-d,-o, and-foptions. Covered in Output Files.
Host-side activity kinds (runtime, driver, marker, external correlation) are always collected in full. You cannot
collect only some of them with per-domain flags such as --kernel-trace. For the detailed support status of each
option, see CLI: Option Status.
2.3 Output Files¶
Each profiled process produces the following two files.
1 2 | |
| Format | Purpose and characteristics |
|---|---|
.pftrace |
Perfetto Protobuf format. Open it at ui.perfetto.dev; recommended for analyzing large traces. |
.json |
Chrome trace JSON format. Opens in chrome://tracing and similar tools, and is the input data for the haprof merge report. |
Both files are generated from the same records and represent the same run. Change the output directory with -d,
the file name with -o, and the format with -f.
1 | |
When the trace file is written
Collected records are held in a memory buffer and written to the file at process exit. A process forcibly
killed with SIGKILL, or one that could not take the normal exit path because of a fatal crash, leaves no
valid trace. When profiling a program that hangs, do not force-kill the process; send an interrupt signal so
that it exits normally.
2.4 Multi-Process Runs¶
By default, child processes of a program that calls fork are excluded from profiling. This is because the tool
clears its own sentinel variable at startup so that child processes do not create duplicate trace files.
In multi-rank training and inference environments, enable the --profile-children option explicitly.
1 | |
The run creates files with a unique <hostname>_<pid>_<starttime> combination for each rank. With the -o NAME
option, the name is applied as a prefix and the output takes the form NAME_<hostname>_<pid>_<starttime>.
The per-rank traces can be merged into a combined timeline and a straggler report with the haprof merge command.
1 | |
| Output | Contents |
|---|---|
merged.pftrace |
A single timeline that concatenates the per-rank .pftrace files ordered by clock reference, earliest first |
| Straggler report | Groups the same collective ranges across ranks and reports entry delay, duration variance, and the last rank to arrive (straggler) |
Conditions for the straggler report
The straggler report is generated by reading the clock reference and process information from each trace's
.json file. If .json generation is skipped with -f pftrace, only the merged timeline file is produced.
Clock shared within a machine
Files produced on the same machine share the same system clock, so the merged result aligns exactly. When
traces collected on different machines are merged, the top of the report shows a warning that this is a
cross-machine merge (cross-node: yes) together with the per-process clock offsets. The clocks themselves are
not corrected, so the spread of these offsets is the error margin of every delay analysis in the report.
2.5 Choosing the Collection Scope¶
| Analysis goal | Recommended command |
|---|---|
| Analyze host time | haprof -- APP |
| Understand device work flow and timing | haprof --runtime-trace -- APP |
| Find the cause of slow kernels (hardware counters) | haprof --device-trace -- APP |
| Analyze per-rank delay in a multi-rank environment | haprof --profile-children --device-trace -- APP, then run haprof merge |
Start with the lowest-cost mode that meets the analysis goal and step up as needed. Device counter collection costs more than timeline collection, and timeline collection costs more than the host-only mode.
2.6 Making a Trace You Can Analyze¶
If you profile without adding annotations to the code, the timeline shows only repeated low-level API calls such as
HaLaunchKernel and HaMemCpy. From that alone it is hard to tell which model layer, compute block, or loop in the
code caused the operation, which limits your ability to pinpoint bottlenecks. For meaningful analysis, the
following two steps are recommended.
- Annotate key code ranges — Adding just a few Ranges for prefill, decode, sampling, and so on makes the timeline easy to navigate. See Chapter 5 Annotations.
- Warm up first — The first execution includes kernel binary loading time, and the hardware counters have no valid baseline. Profile the steady-state range after excluding the initial warm-up executions.
1 2 3 4 5 6 7 | |
2.7 Overhead¶
Running the profiling tool adds overhead. The cost increases by mode in the following order.
| Profiling mode | Added cost |
|---|---|
Host only (haprof -- APP) |
Interception cost per runtime and driver call |
--runtime-trace |
Host-only mode + device record ring collection cost |
--device-trace |
--runtime-trace + per-kernel-execution counter sampling and signal logger collection cost |
The profiler's own execution time cost is not yet recorded as a separate activity record in the trace. Before drawing final conclusions about a performance bottleneck, compare against a run without the profiler to estimate the overhead.
Next Steps¶
- Device Tracing — Collecting kernel executions and hardware counters
- Trace Data Analysis Guide — How to interpret profiling results
- CLI Reference — All command options and their status