4. Trace Data Analysis Guide¶
A trace does not tell you the root cause of a problem; it is only a record of what ran. This chapter looks at what each track means and walks through the procedure for identifying performance bottlenecks step by step through timeline analysis. It assumes the record collection timing described in Chapter 3 Device Tracing.
Contents¶
| Section | Objective |
|---|---|
| 4.1 Opening a Viewer and Loading a Trace | Choosing an analysis viewer and loading data |
| 4.2 Timeline Tracks and Layout | Meaning of the data in each track and lane |
| 4.3 Caveats When Reading the Device Timeline | Avoiding causality mistakes caused by batch boundaries |
| 4.4 Step-by-Step Timeline Analysis | The standard order from the big picture down to the details |
| 4.5 Diagnosing Host-Bound Bottlenecks | Host bottleneck patterns that leave the device idle |
| 4.6 Device-Bound Bottlenecks and Counter Analysis | Diagnosing compute saturation and detailed hardware counter analysis |
| 4.7 Before-and-After Comparison Principles | Conditions for fair and reliable before-and-after comparison |
| 4.8 Unrecorded Items and Timeline Gaps | Correctly interpreting gaps and items outside the observed scope |
4.1 Opening a Viewer and Loading a Trace¶
Open the generated .pftrace file in the Perfetto web viewer (ui.perfetto.dev). The file is parsed entirely in the browser and is never uploaded, so the data does not leave your machine.
In a secured network environment where Perfetto is not reachable, load the .json file into the chrome://tracing
interface or a Chrome-based trace visualization tool.
4.2 Timeline Tracks and Layout¶
| Track | Lane structure | Contents and description |
|---|---|---|
| Runtime API | Per process · thread | Every Ha* runtime API call made by the application and its duration (memory allocation calls show the byte count) |
| Driver API | Per process · thread | Lower-layer calls made beneath the runtime API (sharing the same correlation id as the parent runtime call) |
| Markers | Per process · thread | User-defined annotations. A Range is shown as a range slice and a Mark as a point-in-time event |
| Device kernels | LPU Device <n> · Stream <s> |
Kernels executed on the LPU (includes hardware counters in --device-trace mode) |
| Device transfers | LPU Device <n> · Stream <s> |
Memory copy (Memcpy) transfers (shows transferred bytes and bandwidth) |
| Memcpy bandwidth | Per-device counter track | H2D and D2H transfer bandwidth (GB/s) over time |
| Signal logger | Device Trace <n> · CG<g>_<unit> |
On/off signal state of each internal unit during kernel execution (collected in --device-trace mode) |
Main Visual Elements¶
- Lane names — Host lanes are shown with the thread name (for example
rank 0, set by the collective communication library). Threads without a name are shown asThread <tid>. - Flow arrows — Runtime API calls that caused device work (kernel launches, memory copies) and the device slices they produced are connected by flow arrows based on the correlation id. Select either end to jump to the other.
- Collective ranges — The
lccl:<op>ranges recorded by LCCL (for examplelccl:AllReduce) are named uniformly per operation, so identical operations can be grouped and compared. The keys that link the same call across ranks are in the slice argumentscomm,op_count,rank, andn_ranks.
4.3 Caveats When Reading the Device Timeline¶
Device records are emitted at batch boundaries, not at call time
Asynchronous API calls return control right after enqueuing work. Device records are emitted only when the LPU completes a processing batch (see When Device Records Are Collected). Do not infer causality from the horizontal position of the host and device timelines side by side; always follow the correlation id arrows.
The following are common symptoms and their causes.
| What you see | Root cause |
|---|---|
| A device slice appears long after the host API call | The host call only enqueued the work, and the device ran it only after processing all preceding work |
| The device track area is completely empty | Device tracing was not enabled, or device profiling was rejected (see Troubleshooting) |
4.4 Step-by-Step Timeline Analysis¶
Always narrow the scope from the big picture (upper layers) down to the details (individual kernels). Starting from the duration of individual kernels makes it easy to miss the overall bottleneck.
- Identify the range of interest — If you annotated the source in advance, named Ranges identify the target range immediately. If there are no annotations, go back to the code and add them first (see Chapter 5 Annotations).
- Check how busy the device is — Look at the device track in that range and decide which of the three patterns below it matches.
- Drill into individual kernels — In strongly host-bound workloads, optimizing a single kernel has little effect on overall performance. Fix the higher-level bottleneck first, then move on to individual kernel analysis.
| Execution pattern | What the timeline shows | Diagnosis | Detailed analysis guide |
|---|---|---|---|
| Host-bound | The host track is busy, but the device track has only sparse short kernels | The device sits idle waiting for the host to supply the next command | Diagnosing Host-Bound Bottlenecks |
| Device-bound | The host enqueues work and waits on synchronization, while the device track is densely filled with kernels | Execution time is concentrated in the hardware kernel compute itself | Device-Bound Bottlenecks and Counter Analysis |
| Dispatch/synchronization-bound | API calls and synchronization frequently intervene between kernels, leaving large gaps between them | Command dispatch overhead or frequent synchronization delays | Diagnosing Host-Bound Bottlenecks |
4.5 Diagnosing Host-Bound Bottlenecks¶
The LPU accelerator stays idle because not enough work is supplied to the device.
While the host track is busy with Python preprocessing and similar work, only sparse short kernels appear on the device track.
Example host-bound timeline (dark bars: host activity / light bars: device kernels)
Dispatch/synchronization-bound cases, where API calls and synchronization intervene between kernels and create gaps, fall into the same category.
Example dispatch/synchronization-bound timeline (dark bars: host activity / light bars: device kernels)
The root causes most often seen in a trace are as follows.
| Pattern in the trace | Root cause and remedy |
|---|---|
| Long runtime API calls repeat between kernels | Per-call overhead accumulates. Consider switching from eager mode to compile mode |
| A synchronization (sync) is called right after each launch | The program code causes serialization. Reduce synchronization frequency and dispatch operations in groups |
| Long gaps with no host activity at all | Unannotated Python code or data preprocessing overhead is running |
| Frequent small memory transfers (Memcpy) | Host-device round trips inside the compute loop (check .cpu(), .item(), print, and so on) |
Most common fix
Repeated small memory transfers are the pattern most often seen in PyTorch-based code, and fixing the code can bring an immediate performance improvement.
4.6 Device-Bound Bottlenecks and Counter Analysis¶
The device timeline is densely filled with kernel execution slices while the host has enqueued work and is waiting on synchronization.
Example device-bound timeline (dark bars: host activity / light bars: device kernels)
Because execution time is concentrated in the hardware compute itself rather than in host-side delay, precise
diagnosis with hardware counter values is essential. If no counter data was collected, profile again with the
--device-trace option.
Identify the dominant kernels that make up most of the total execution time, then check the key hardware counter metrics. For detailed formulas and criteria, see Interpreting Counters.
| Hardware counter metric | Diagnosis | Recommended optimization |
|---|---|---|
High supply-stall% |
Memory-bound | Reduce data transfer volume, change the tensor layout, improve data reuse |
High demand-stall% |
Compute-bound | Restructure the compute logic and improve utilization of the core compute units |
High util% / low stall metrics |
Data path saturated | The kernel has reached the hardware's performance limit |
High UTIL.c<n>.mpu% |
Matrix unit heavily active | Normal behavior for MatMul-centric kernels |
Low UTIL.c<n>.core% |
Core idles during execution | The core is starved of data because of upstream supply delays |
High DMC.c<n> bank-skew% |
SRAM bank conflicts | The memory placement layout of the tensor data needs to be restructured |
4.7 Before-and-After Comparison Principles¶
Profiling traces are most reliable for relative comparison before and after a code change, rather than for analyzing absolute measurements of a single run. Follow these principles for accurate before-and-after comparison.
- Keep the collection environment identical — Hardware counter values are computed as the difference (delta)
from the previous execution, and the results depend on the set of collected counters. Use the same collection mode
and
HAPTI_COUNTERSsetting for everything you compare. - Warm up sufficiently — The first execution includes module initialization and kernel loading overhead, so exclude it from comparison. Compare steady-state data after warm-up (see The First Execution Is Not a Measurement Target).
- Compare the same annotated ranges — Instead of total wall-clock time, which includes overhead, measure performance changes by comparing the durations of identically defined annotation ranges.
Do not compare wall-clock times of profiled and unprofiled runs
The profiling tool adds its own overhead through API hooking and counter collection. Use profiled runs to diagnose bottlenecks, and verify the final speedup by comparing wall-clock time from runs without the profiler.
4.8 Unrecorded Items and Timeline Gaps¶
When analyzing the timeline, be aware of what is not recorded in the trace and the technical reasons why.
| Item not recorded in the trace | Technical reason |
|---|---|
| Unannotated Python code | The profiler intercepts calls at the runtime API level, not inside the Python interpreter |
| Work targeting another device | Device records are collected only from the single specified device |
| The profiler's own overhead | The profiler's own execution overhead is not aggregated as a separate activity track |
Processes terminated abnormally, for example by SIGKILL |
Collected records are flushed to the file all at once at normal process exit |
Child processes without --profile-children |
In multi-process environments, child processes are excluded from tracing by default |
Interpreting timeline gaps
An empty gap on the timeline does not mean "no work was done"; it means "no activity was recorded within the range the profiler can observe". Work outside the observed scope (the items in the table above) may have been running during that gap.
Next Steps¶
- Annotations — Making the timeline easier to read and navigate
- Counter Reference — Specification of all 41 hardware events
- Troubleshooting — Common trace problems and remedies