Skip to content

2. Capturing a Trace

This chapter covers how to run a program under the profiler and the files it produces. It assumes the key concepts from Chapter 1 Overview.

Contents

Section Objective
2.1 Basic Form Structure of the haprof command
2.2 Decisions at Run Time Guide to choosing command-line options
2.3 Output Files File naming rules, location, and formats
2.4 Multi-Process Runs Collecting and merging multi-rank traces
2.5 Choosing the Collection Scope The lowest-cost command for each analysis goal
2.6 Making a Trace You Can Analyze Adding annotations and warming up
2.7 Overhead Performance impact of each mode

2.1 Basic Form

1
haprof [options] -- <command> [args...]

-- separates the profiler options from the command to run and cannot be omitted.

1
2
haprof -- python infer.py --model llama3
haprof --device-trace -- ./my_binary

haprof sets up environment variables and then replaces the process image with the target command (execvpe). The system process list therefore shows the running user program as is, and the process exit code is the program's return value.

2.2 Decisions at Run Time

There are three things to decide on the command line.

  1. Whether to include device data — --runtime-trace (timeline) or --device-trace (timeline and hardware counters). Covered in detail in Chapter 3 Device Tracing.
  2. Whether to trace child processes — The --profile-children option. Covered in Multi-Process Runs.
  3. Output location and file format — The -d, -o, and -f options. Covered in Output Files.

Host-side activity kinds (runtime, driver, marker, external correlation) are always collected in full. You cannot collect only some of them with per-domain flags such as --kernel-trace. For the detailed support status of each option, see CLI: Option Status.

2.3 Output Files

Each profiled process produces the following two files.

1
2
<hostname>_<pid>_<starttime>.pftrace
<hostname>_<pid>_<starttime>.json
Format Purpose and characteristics
.pftrace Perfetto Protobuf format. Open it at ui.perfetto.dev; recommended for analyzing large traces.
.json Chrome trace JSON format. Opens in chrome://tracing and similar tools, and is the input data for the haprof merge report.

Both files are generated from the same records and represent the same run. Change the output directory with -d, the file name with -o, and the format with -f.

1
haprof -d traces -o decode -f pftrace -- python infer.py   # creates traces/decode.pftrace

When the trace file is written

Collected records are held in a memory buffer and written to the file at process exit. A process forcibly killed with SIGKILL, or one that could not take the normal exit path because of a fatal crash, leaves no valid trace. When profiling a program that hangs, do not force-kill the process; send an interrupt signal so that it exits normally.

2.4 Multi-Process Runs

By default, child processes of a program that calls fork are excluded from profiling. This is because the tool clears its own sentinel variable at startup so that child processes do not create duplicate trace files.

In multi-rank training and inference environments, enable the --profile-children option explicitly.

1
haprof --profile-children --device-trace -- torchrun --nproc-per-node 4 serve.py

The run creates files with a unique <hostname>_<pid>_<starttime> combination for each rank. With the -o NAME option, the name is applied as a prefix and the output takes the form NAME_<hostname>_<pid>_<starttime>.

The per-rank traces can be merged into a combined timeline and a straggler report with the haprof merge command.

1
haprof merge -o merged.pftrace --report-format both
Output Contents
merged.pftrace A single timeline that concatenates the per-rank .pftrace files ordered by clock reference, earliest first
Straggler report Groups the same collective ranges across ranks and reports entry delay, duration variance, and the last rank to arrive (straggler)

Conditions for the straggler report

The straggler report is generated by reading the clock reference and process information from each trace's .json file. If .json generation is skipped with -f pftrace, only the merged timeline file is produced.

Clock shared within a machine

Files produced on the same machine share the same system clock, so the merged result aligns exactly. When traces collected on different machines are merged, the top of the report shows a warning that this is a cross-machine merge (cross-node: yes) together with the per-process clock offsets. The clocks themselves are not corrected, so the spread of these offsets is the error margin of every delay analysis in the report.

2.5 Choosing the Collection Scope

Analysis goal Recommended command
Analyze host time haprof -- APP
Understand device work flow and timing haprof --runtime-trace -- APP
Find the cause of slow kernels (hardware counters) haprof --device-trace -- APP
Analyze per-rank delay in a multi-rank environment haprof --profile-children --device-trace -- APP, then run haprof merge

Start with the lowest-cost mode that meets the analysis goal and step up as needed. Device counter collection costs more than timeline collection, and timeline collection costs more than the host-only mode.

2.6 Making a Trace You Can Analyze

If you profile without adding annotations to the code, the timeline shows only repeated low-level API calls such as HaLaunchKernel and HaMemCpy. From that alone it is hard to tell which model layer, compute block, or loop in the code caused the operation, which limits your ability to pinpoint bottlenecks. For meaningful analysis, the following two steps are recommended.

  1. Annotate key code ranges — Adding just a few Ranges for prefill, decode, sampling, and so on makes the timeline easy to navigate. See Chapter 5 Annotations.
  2. Warm up first — The first execution includes kernel binary loading time, and the hardware counters have no valid baseline. Profile the steady-state range after excluding the initial warm-up executions.
1
2
3
4
5
6
7
import hapti

for _ in range(3):
    model(x)                      # warm-up

with hapti.annotate("decode"):
    out = model(x)

2.7 Overhead

Running the profiling tool adds overhead. The cost increases by mode in the following order.

Profiling mode Added cost
Host only (haprof -- APP) Interception cost per runtime and driver call
--runtime-trace Host-only mode + device record ring collection cost
--device-trace --runtime-trace + per-kernel-execution counter sampling and signal logger collection cost

The profiler's own execution time cost is not yet recorded as a separate activity record in the trace. Before drawing final conclusions about a performance bottleneck, compare against a run without the profiler to estimate the overhead.


Next Steps