Skip to content

1. Overview

This guide describes how to profile applications that run on the HyperAccel LPU. It is for engineers who need to find out whether time is spent in the application code, the runtime, or the device.

Contents

Section Objective
1.1 What the Tool Does The five kinds of information it collects
1.2 Design Background Three choices that determine how it works
1.3 Key Concepts Terms used throughout the following chapters
1.4 Capturing Your First Trace From running to checking the output files
1.5 Two Integration Methods Comparison of haprof and the Kineto plugin

Capturing (Chapter 2), device tracing (Chapter 3), and analysis (Chapter 4) are best read in order; annotations (Chapter 5), PyTorch integration (Chapter 6), and troubleshooting (Chapter 7) can be consulted as needed.


1.1 What the Tool Does

hapti records what happens on the host and the LPU in time order while a program runs, and saves it to a file that a trace viewer can open. It collects the following five kinds of information.

  • Host API activity — Records the runtime API calls (Ha*) made by the application and the lower-layer call ranges inside the runtime.
  • Device activity — Records kernels and memory transfers executed on the LPU on the device timeline.
  • Hardware counters — Collects the device's performance counter values during kernel execution.
  • User annotations — Records Range and Mark annotations placed in application code.
  • Correlation — Links API calls made on the host to the device work they caused.

hapti is a data collection tool only; it does not diagnose bottlenecks. How to interpret the collected trace is covered in Chapter 4 Trace Data Analysis Guide.

1.2 Design Background

The profiler integrates through preload, not linking. haprof configures the LD_PRELOAD environment and then runs the program, so application code does not need to change. No rebuild, import in the source, or initialization call is needed, but in exchange all settings are decided before the process starts, so tracing cannot be turned on dynamically during execution.

Host and device timelines are collected differently. Host calls are measured immediately at call time. Device activity, on the other hand, is collected by the profiler from data the device writes to the record ring, so it appears at batch boundaries rather than at call time. Most confusion in trace analysis comes from this difference; it is covered in detail in Trace Data Analysis Guide: Caveats When Reading the Device Timeline.

All timestamps use a single clock. Both host and device records are converted to host CLOCK_MONOTONIC nanoseconds, so traces collected on the same machine align on one timeline without extra correction. Device cycles are converted to nanoseconds using the host-device clock pairing provided by the runtime.

1.3 Key Concepts

Concept Meaning
Activity One recorded range or event. The kinds are driver, runtime, kernel, memory copy, external correlation, and marker; memory allocation is included in runtime activity.
Host timeline Runtime API calls and their lower-layer calls. Placed on per-process and per-thread lanes.
Device timeline Kernel executions and data transfers. Placed on per-device and per-stream lanes.
correlation id An identifier that links a host API call to the device activity it caused.
Batch boundary The point at which the device completes a unit of work and emits records. This is where device time becomes visible.
Counter A hardware event metric sampled for each kernel execution. See the Counter Reference.
stall The number of cycles a data path spent waiting. Divided into supply-side stalls (waiting on DRAM) and demand-side stalls (waiting on the core).
Annotation A Range or Mark placed in code to label a span or point in time.
Trace The output file. One is created per profiled process.

1.4 Capturing Your First Trace

1
haprof -- python your_script.py

When the run finishes, two files are created in the working directory.

1
2
<hostname>_<pid>_<starttime>.pftrace
<hostname>_<pid>_<starttime>.json

Open the generated .pftrace file at ui.perfetto.dev. The trace data is processed only inside the browser and is not uploaded.

To also collect device records, run as follows.

1
haprof --device-trace -- python your_script.py

Points to note when running:

  1. The -- separator is required. Everything before it is haprof options; everything after it is the command to run.
  2. One trace is created per process. Interception works per process, so forked child processes and multi-rank runs are recorded only with the --profile-children option, and a separate file is written for each rank.
  3. Warm up before measuring. The first execution includes kernel binary load time, and counter deltas have no valid baseline.
  4. The trace is written all at once at process exit. A process forcibly killed with SIGKILL or similar leaves no trace.
  5. The profiler's own overhead is not yet recorded as a separate activity. To measure overhead, see the method in Overhead.

1.5 Two Integration Methods

The method described so far uses the haprof launcher. Another integration method is also supported.

Aspect haprof method Kineto plugin method
Integration LD_PRELOAD (no changes to the target code) One line of code: hapti.kineto.register()
Output Separate .pftrace and .json files Integrated into the torch.profiler trace
Scope Whole process (general programs outside PyTorch also work) Only the profile() region (PyTorch only)
Hardware counters Supported with --device-trace Not collected

For systems that already have a torch.profiler-based analysis setup, the Kineto plugin is recommended. CPU operations and the underlying device executions are connected automatically with flow arrows, which makes analysis more efficient. For details, see PyTorch Profiler Integration.

However, do not enable both profiling methods in the same process. When both collection systems run at the same time, they contend for access to the device record ring, and device activity can be missing from one of the traces.


Next Steps