1. Overview¶
This guide describes how to profile applications that run on the HyperAccel LPU. It is for engineers who need to find out whether time is spent in the application code, the runtime, or the device.
Contents¶
| Section | Objective |
|---|---|
| 1.1 What the Tool Does | The five kinds of information it collects |
| 1.2 Design Background | Three choices that determine how it works |
| 1.3 Key Concepts | Terms used throughout the following chapters |
| 1.4 Capturing Your First Trace | From running to checking the output files |
| 1.5 Two Integration Methods | Comparison of haprof and the Kineto plugin |
Capturing (Chapter 2), device tracing (Chapter 3), and analysis (Chapter 4) are best read in order; annotations (Chapter 5), PyTorch integration (Chapter 6), and troubleshooting (Chapter 7) can be consulted as needed.
1.1 What the Tool Does¶
hapti records what happens on the host and the LPU in time order while a program runs, and saves it to a file
that a trace viewer can open. It collects the following five kinds of information.
- Host API activity — Records the runtime API calls (
Ha*) made by the application and the lower-layer call ranges inside the runtime. - Device activity — Records kernels and memory transfers executed on the LPU on the device timeline.
- Hardware counters — Collects the device's performance counter values during kernel execution.
- User annotations — Records Range and Mark annotations placed in application code.
- Correlation — Links API calls made on the host to the device work they caused.
hapti is a data collection tool only; it does not diagnose bottlenecks. How to interpret the collected trace is
covered in Chapter 4 Trace Data Analysis Guide.
1.2 Design Background¶
The profiler integrates through preload, not linking. haprof configures the LD_PRELOAD environment and then
runs the program, so application code does not need to change. No rebuild, import in the source, or initialization
call is needed, but in exchange all settings are decided before the process starts, so tracing cannot be turned on
dynamically during execution.
Host and device timelines are collected differently. Host calls are measured immediately at call time. Device activity, on the other hand, is collected by the profiler from data the device writes to the record ring, so it appears at batch boundaries rather than at call time. Most confusion in trace analysis comes from this difference; it is covered in detail in Trace Data Analysis Guide: Caveats When Reading the Device Timeline.
All timestamps use a single clock. Both host and device records are converted to host CLOCK_MONOTONIC
nanoseconds, so traces collected on the same machine align on one timeline without extra correction. Device cycles
are converted to nanoseconds using the host-device clock pairing provided by the runtime.
1.3 Key Concepts¶
| Concept | Meaning |
|---|---|
| Activity | One recorded range or event. The kinds are driver, runtime, kernel, memory copy, external correlation, and marker; memory allocation is included in runtime activity. |
| Host timeline | Runtime API calls and their lower-layer calls. Placed on per-process and per-thread lanes. |
| Device timeline | Kernel executions and data transfers. Placed on per-device and per-stream lanes. |
| correlation id | An identifier that links a host API call to the device activity it caused. |
| Batch boundary | The point at which the device completes a unit of work and emits records. This is where device time becomes visible. |
| Counter | A hardware event metric sampled for each kernel execution. See the Counter Reference. |
| stall | The number of cycles a data path spent waiting. Divided into supply-side stalls (waiting on DRAM) and demand-side stalls (waiting on the core). |
| Annotation | A Range or Mark placed in code to label a span or point in time. |
| Trace | The output file. One is created per profiled process. |
1.4 Capturing Your First Trace¶
1 | |
When the run finishes, two files are created in the working directory.
1 2 | |
Open the generated .pftrace file at ui.perfetto.dev. The trace data is processed only
inside the browser and is not uploaded.
To also collect device records, run as follows.
1 | |
Points to note when running:
- The
--separator is required. Everything before it ishaprofoptions; everything after it is the command to run. - One trace is created per process. Interception works per process, so forked child processes and multi-rank
runs are recorded only with the
--profile-childrenoption, and a separate file is written for each rank. - Warm up before measuring. The first execution includes kernel binary load time, and counter deltas have no valid baseline.
- The trace is written all at once at process exit. A process forcibly killed with
SIGKILLor similar leaves no trace. - The profiler's own overhead is not yet recorded as a separate activity. To measure overhead, see the method in Overhead.
1.5 Two Integration Methods¶
The method described so far uses the haprof launcher. Another integration method is also supported.
| Aspect | haprof method |
Kineto plugin method |
|---|---|---|
| Integration | LD_PRELOAD (no changes to the target code) |
One line of code: hapti.kineto.register() |
| Output | Separate .pftrace and .json files |
Integrated into the torch.profiler trace |
| Scope | Whole process (general programs outside PyTorch also work) | Only the profile() region (PyTorch only) |
| Hardware counters | Supported with --device-trace |
Not collected |
For systems that already have a torch.profiler-based analysis setup, the Kineto plugin is recommended. CPU
operations and the underlying device executions are connected automatically with flow arrows, which makes analysis
more efficient. For details, see PyTorch Profiler Integration.
However, do not enable both profiling methods in the same process. When both collection systems run at the same time, they contend for access to the device record ring, and device activity can be missing from one of the traces.
Next Steps¶
- Capturing a Trace — How to run and check the output files
- Device Tracing — Kernel executions and hardware counters
- Trace Data Analysis Guide — How to interpret profiling results
- PyTorch Profiler Integration — Using the Kineto plugin