3. Device Tracing¶
Host-only tracing shows the API requests the program made, while device tracing shows the compute that actually ran on the LPU. This chapter covers the collection modes, target device selection, when records are collected, hardware counters and how to read them, and the cost.
Contents¶
| Section | Objective |
|---|---|
| 3.1 Trace Collection Modes | Roles of and differences between the --runtime-trace and --device-trace options |
| 3.2 Selecting the Target Device | The single-device limit when collecting a trace |
| 3.3 When Device Records Are Collected | Batch boundaries and how the record ring buffer works |
| 3.4 Hardware Counters | Collected counters and how they are displayed |
| 3.5 Interpreting Counters | Telling memory-bound from compute-bound |
| 3.6 Transfer Bandwidth and the Signal Logger | Device tracks provided in addition to hardware counters |
| 3.7 The First Execution Is Not a Measurement Target | Why warm-up runs are needed for accurate measurement |
| 3.8 Cost and Loss | Performance overhead of device tracing and record drops |
3.1 Trace Collection Modes¶
| Flag | Additionally collected |
|---|---|
--runtime-trace |
Device kernel and memory transfer timeline |
--device-trace |
Timeline + per-kernel hardware counters and the signal logger |
The --device-trace option includes timeline collection, so use only one of the two flags. Both options are off by
default, so a run in the form haprof -- APP without either flag collects only host-side records and incurs no
device profiling cost.
1 2 | |
Unsupported driver or firmware
In environments where the driver or firmware does not support device profiling, the configuration is rejected and the trace records only host-side information (see Troubleshooting for details).
3.2 Selecting the Target Device¶
By default, device records are collected only from a single device, index 0. There is no CLI flag; specify it with
the HAPTI_TOOL_DEVICE_ID environment variable.
1 | |
Single Device per Trace¶
A multi-device process outputs records only for the selected device. The host timeline covers all calls, so work
done on other devices appears as host activity with an empty device track below it. In multi-rank runs that use one
device per rank, use the --profile-children option.
3.3 When Device Records Are Collected¶
This is the most important point to keep in mind when analyzing the device timeline.
Asynchronous APIs return control right after enqueuing work, so device activity is not yet recorded at API call time. Device records are written to the record ring when a processing batch completes, and the profiler's reader thread collects them continuously.
Even without an explicit synchronization call, the kernel driver (KMD) automatically appends a wait command at the end of a batch, so device tracks are not lost. However, records become visible at the batch boundary, not at the time of the host API call.
Records missing when the process exits early
The tool drains the record ring one last time at process exit, but no records exist for batches the device has not completed by then. To see device execution without gaps, synchronize the stream right before the program exits.
For details on interpreting trace results, see Trace Data Analysis Guide: Caveats When Reading the Device Timeline.
3.4 Hardware Counters¶
With the --device-trace option, hardware counter samples are recorded together with the kernel slice for each
kernel execution. The device supports 41 events across 6 blocks; the full catalog is in the
Counter Reference.
Default Counter Scope¶
The default configuration registers only 8 AXIBRG, 2 DMC, and 10 UTIL events of core 0. Counters of the CDMA, CGDMA,
and MPU blocks and of core 1 and above are not collected by default. If you need a different combination, specify up
to 32 directly with the
HAPTI_COUNTERS environment variable.
1 2 | |
Diagnostic Purpose by Block¶
| Block | Main diagnostic target |
|---|---|
| UTIL | Occupancy of each unit inside the core (matrix, vector, special function, DMA, and so on) |
| CGDMA | Data movement state of the core group DMA and which side causes stalls |
| AXIBRG | Transfer state of the core-DRAM AXI bridge and write completion latency |
| CDMA | Per-core DMA beat transfer volume |
| DMC | Whether SRAM bank conflicts occur |
| MPU | Raw work of the matrix compute unit |
Display rules
The values shown in the trace are not plain raw counts. Free-running counter values are converted to the difference (delta) from the previous execution, the UTIL block is shown as percentages, and some blocks show additional derived metrics. For the exact notation and formulas, see Counter Reference: Display Rules.
PyTorch Kineto integration
The PyTorch Kineto plugin path collects only the kernel timeline and does not collect hardware counters.
3.5 Interpreting Counters¶
In data collected with --device-trace, the kernel slice arguments (args) show percentage metrics such as
AXIBRG.c0.read util%. The % in a metric name means it is a percentage, mostly the ratio to the total cycles
(transfer + wait) in that transfer direction. For the formula of each metric, see
Counter Reference: Bandwidth and Stall Breakdown.
Of these, supply-stall% and demand-stall% tell you directly whether the bottleneck is memory (DRAM) or the
compute core, so check them first.
| Metric | Interpretation and diagnosis |
|---|---|
High util% |
The data path is actively transferring data |
High supply-stall% |
Waiting for DRAM responses — memory-bound |
High demand-stall% |
Waiting for the compute core to produce or consume data — compute-bound |
High bresp-wait% |
Delay in confirming the write response |
High bank-skew% |
Conflicts concentrate on a specific SRAM bank — the data layout needs improvement |
Interpreting derived values
CGDMA bandwidth (GB/s) is a utilization metric scaled by the peak defined by the firmware, not a direct measurement of bytes. UTIL percentages are a sampled duty cycle, not cycle-exact occupancy, so units that operate only in short bursts can be under-measured. Use both metrics only for relative comparison between executions, and do not interpret them as absolute physical quantities.
3.6 Transfer Bandwidth and the Signal Logger¶
In device tracing mode, two tracks are added in addition to the hardware counters. Transfer bandwidth is also recorded with --runtime-trace; the signal logger is recorded only with --device-trace.
- Memory transfer bandwidth (
memory bandwidth) — H2D (host-to-device) and D2H (device-to-host) transfer slices of 4 KiB or more carry a bandwidth value, and thehaproftrace gets per-deviceMemcpy H2D bandwidthandMemcpy D2H bandwidthcounter tracks. This value is a measured bandwidth: transferred bytes divided by the slice duration. The slice also covers host page pinning and the completion wait, so read it as a lower bound on what the link carried. D2D transfers and small transfers under 4 KiB are excluded from the bandwidth calculation because fixed costs dominate them. - Signal logger (
Signal Logger) — Records the on/off signal state of the main units inside the device for each kernel execution and shows it under theDevice Trace <n>group of the trace asCG<group>_<unit>(for exampleCG0_MPU) andcgdma_<n>tracks. The signal data of each execution is aligned to the range of the kernel slice with the same correlation id.
3.7 The First Execution Is Not a Measurement Target¶
Exclude the first execution from performance analysis for the following two independent reasons.
- Kernel binary loading time — The first call on each device includes loading the kernel binary, so the first execution time includes the initialization cost.
- No counter baseline — The counter delta of the first execution is taken against a baseline of 0, so it includes counts from before profiling started.
Profile executions that have reached steady state after enough warm-up.
1 2 3 4 5 | |
3.8 Cost and Loss¶
Device tracing adds performance overhead, and counter collection mode costs more than timeline collection.
| Profiling mode | Source of added overhead |
|---|---|
--runtime-trace |
Device record ring collection cost |
--device-trace |
Per-kernel-execution counter sampling and signal logger buffer read cost |
Because the cost per execution is fixed, the share of overhead in total run time grows for workloads that run small kernels frequently (fine-grained workloads). Results remain valid for relative comparison between executions, but be careful when judging absolute wall-clock performance.
Record Drops¶
If data is lost during record collection, the tool reports it with a [hapti-tool] warning log at exit.
| Warning message | Meaning and cause |
|---|---|
... bytes of device records were overwritten before the reader drained them |
The reader thread could not keep up with the record ring and an overrun occurred. Part of the timeline has gaps |
... device records were dropped by the driver before reaching the ring |
The driver lost records before writing them to the record ring |
signal logger: skipped ... launches whose device kernel record never arrived |
Signal logger data from executions with a missing kernel record is excluded from the visualization |
dropped ... records (buffer capacity exceeded) |
Records were discarded because the profiler's internal record buffer was full |
Next Steps¶
- Trace Data Analysis Guide — How to interpret the collected trace
- Counter Reference — All 41 hardware events and notation rules