Skip to content

4. Trace Data Analysis Guide

A trace does not tell you the root cause of a problem; it is only a record of what ran. This chapter looks at what each track means and walks through the procedure for identifying performance bottlenecks step by step through timeline analysis. It assumes the record collection timing described in Chapter 3 Device Tracing.

Contents

Section Objective
4.1 Opening a Viewer and Loading a Trace Choosing an analysis viewer and loading data
4.2 Timeline Tracks and Layout Meaning of the data in each track and lane
4.3 Caveats When Reading the Device Timeline Avoiding causality mistakes caused by batch boundaries
4.4 Step-by-Step Timeline Analysis The standard order from the big picture down to the details
4.5 Diagnosing Host-Bound Bottlenecks Host bottleneck patterns that leave the device idle
4.6 Device-Bound Bottlenecks and Counter Analysis Diagnosing compute saturation and detailed hardware counter analysis
4.7 Before-and-After Comparison Principles Conditions for fair and reliable before-and-after comparison
4.8 Unrecorded Items and Timeline Gaps Correctly interpreting gaps and items outside the observed scope

4.1 Opening a Viewer and Loading a Trace

Open the generated .pftrace file in the Perfetto web viewer (ui.perfetto.dev). The file is parsed entirely in the browser and is never uploaded, so the data does not leave your machine.

In a secured network environment where Perfetto is not reachable, load the .json file into the chrome://tracing interface or a Chrome-based trace visualization tool.


4.2 Timeline Tracks and Layout

Track Lane structure Contents and description
Runtime API Per process · thread Every Ha* runtime API call made by the application and its duration (memory allocation calls show the byte count)
Driver API Per process · thread Lower-layer calls made beneath the runtime API (sharing the same correlation id as the parent runtime call)
Markers Per process · thread User-defined annotations. A Range is shown as a range slice and a Mark as a point-in-time event
Device kernels LPU Device <n> · Stream <s> Kernels executed on the LPU (includes hardware counters in --device-trace mode)
Device transfers LPU Device <n> · Stream <s> Memory copy (Memcpy) transfers (shows transferred bytes and bandwidth)
Memcpy bandwidth Per-device counter track H2D and D2H transfer bandwidth (GB/s) over time
Signal logger Device Trace <n> · CG<g>_<unit> On/off signal state of each internal unit during kernel execution (collected in --device-trace mode)

Main Visual Elements

  • Lane names — Host lanes are shown with the thread name (for example rank 0, set by the collective communication library). Threads without a name are shown as Thread <tid>.
  • Flow arrows — Runtime API calls that caused device work (kernel launches, memory copies) and the device slices they produced are connected by flow arrows based on the correlation id. Select either end to jump to the other.
  • Collective ranges — The lccl:<op> ranges recorded by LCCL (for example lccl:AllReduce) are named uniformly per operation, so identical operations can be grouped and compared. The keys that link the same call across ranks are in the slice arguments comm, op_count, rank, and n_ranks.

4.3 Caveats When Reading the Device Timeline

Device records are emitted at batch boundaries, not at call time

Asynchronous API calls return control right after enqueuing work. Device records are emitted only when the LPU completes a processing batch (see When Device Records Are Collected). Do not infer causality from the horizontal position of the host and device timelines side by side; always follow the correlation id arrows.

The following are common symptoms and their causes.

What you see Root cause
A device slice appears long after the host API call The host call only enqueued the work, and the device ran it only after processing all preceding work
The device track area is completely empty Device tracing was not enabled, or device profiling was rejected (see Troubleshooting)

4.4 Step-by-Step Timeline Analysis

Always narrow the scope from the big picture (upper layers) down to the details (individual kernels). Starting from the duration of individual kernels makes it easy to miss the overall bottleneck.

  1. Identify the range of interest — If you annotated the source in advance, named Ranges identify the target range immediately. If there are no annotations, go back to the code and add them first (see Chapter 5 Annotations).
  2. Check how busy the device is — Look at the device track in that range and decide which of the three patterns below it matches.
  3. Drill into individual kernels — In strongly host-bound workloads, optimizing a single kernel has little effect on overall performance. Fix the higher-level bottleneck first, then move on to individual kernel analysis.
Execution pattern What the timeline shows Diagnosis Detailed analysis guide
Host-bound The host track is busy, but the device track has only sparse short kernels The device sits idle waiting for the host to supply the next command Diagnosing Host-Bound Bottlenecks
Device-bound The host enqueues work and waits on synchronization, while the device track is densely filled with kernels Execution time is concentrated in the hardware kernel compute itself Device-Bound Bottlenecks and Counter Analysis
Dispatch/synchronization-bound API calls and synchronization frequently intervene between kernels, leaving large gaps between them Command dispatch overhead or frequent synchronization delays Diagnosing Host-Bound Bottlenecks

4.5 Diagnosing Host-Bound Bottlenecks

The LPU accelerator stays idle because not enough work is supplied to the device.

While the host track is busy with Python preprocessing and similar work, only sparse short kernels appear on the device track.

A timeline where the host track is filled with Python preprocessing and the device track has only sparse short kernels

Example host-bound timeline (dark bars: host activity / light bars: device kernels)

Dispatch/synchronization-bound cases, where API calls and synchronization intervene between kernels and create gaps, fall into the same category.

A timeline where an API call and synchronization come between every kernel, creating gaps between kernels

Example dispatch/synchronization-bound timeline (dark bars: host activity / light bars: device kernels)

The root causes most often seen in a trace are as follows.

Pattern in the trace Root cause and remedy
Long runtime API calls repeat between kernels Per-call overhead accumulates. Consider switching from eager mode to compile mode
A synchronization (sync) is called right after each launch The program code causes serialization. Reduce synchronization frequency and dispatch operations in groups
Long gaps with no host activity at all Unannotated Python code or data preprocessing overhead is running
Frequent small memory transfers (Memcpy) Host-device round trips inside the compute loop (check .cpu(), .item(), print, and so on)

Most common fix

Repeated small memory transfers are the pattern most often seen in PyTorch-based code, and fixing the code can bring an immediate performance improvement.


4.6 Device-Bound Bottlenecks and Counter Analysis

The device timeline is densely filled with kernel execution slices while the host has enqueued work and is waiting on synchronization.

A timeline where kernels run back to back on the device track while the host waits on synchronization after its API calls

Example device-bound timeline (dark bars: host activity / light bars: device kernels)

Because execution time is concentrated in the hardware compute itself rather than in host-side delay, precise diagnosis with hardware counter values is essential. If no counter data was collected, profile again with the --device-trace option.

Identify the dominant kernels that make up most of the total execution time, then check the key hardware counter metrics. For detailed formulas and criteria, see Interpreting Counters.

Hardware counter metric Diagnosis Recommended optimization
High supply-stall% Memory-bound Reduce data transfer volume, change the tensor layout, improve data reuse
High demand-stall% Compute-bound Restructure the compute logic and improve utilization of the core compute units
High util% / low stall metrics Data path saturated The kernel has reached the hardware's performance limit
High UTIL.c<n>.mpu% Matrix unit heavily active Normal behavior for MatMul-centric kernels
Low UTIL.c<n>.core% Core idles during execution The core is starved of data because of upstream supply delays
High DMC.c<n> bank-skew% SRAM bank conflicts The memory placement layout of the tensor data needs to be restructured

4.7 Before-and-After Comparison Principles

Profiling traces are most reliable for relative comparison before and after a code change, rather than for analyzing absolute measurements of a single run. Follow these principles for accurate before-and-after comparison.

  • Keep the collection environment identical — Hardware counter values are computed as the difference (delta) from the previous execution, and the results depend on the set of collected counters. Use the same collection mode and HAPTI_COUNTERS setting for everything you compare.
  • Warm up sufficiently — The first execution includes module initialization and kernel loading overhead, so exclude it from comparison. Compare steady-state data after warm-up (see The First Execution Is Not a Measurement Target).
  • Compare the same annotated ranges — Instead of total wall-clock time, which includes overhead, measure performance changes by comparing the durations of identically defined annotation ranges.

Do not compare wall-clock times of profiled and unprofiled runs

The profiling tool adds its own overhead through API hooking and counter collection. Use profiled runs to diagnose bottlenecks, and verify the final speedup by comparing wall-clock time from runs without the profiler.


4.8 Unrecorded Items and Timeline Gaps

When analyzing the timeline, be aware of what is not recorded in the trace and the technical reasons why.

Item not recorded in the trace Technical reason
Unannotated Python code The profiler intercepts calls at the runtime API level, not inside the Python interpreter
Work targeting another device Device records are collected only from the single specified device
The profiler's own overhead The profiler's own execution overhead is not aggregated as a separate activity track
Processes terminated abnormally, for example by SIGKILL Collected records are flushed to the file all at once at normal process exit
Child processes without --profile-children In multi-process environments, child processes are excluded from tracing by default

Interpreting timeline gaps

An empty gap on the timeline does not mean "no work was done"; it means "no activity was recorded within the range the profiler can observe". Work outside the observed scope (the items in the table above) may have been running during that gap.


Next Steps