Skip to content

3. Device Tracing

Host-only tracing shows the API requests the program made, while device tracing shows the compute that actually ran on the LPU. This chapter covers the collection modes, target device selection, when records are collected, hardware counters and how to read them, and the cost.

Contents

Section Objective
3.1 Trace Collection Modes Roles of and differences between the --runtime-trace and --device-trace options
3.2 Selecting the Target Device The single-device limit when collecting a trace
3.3 When Device Records Are Collected Batch boundaries and how the record ring buffer works
3.4 Hardware Counters Collected counters and how they are displayed
3.5 Interpreting Counters Telling memory-bound from compute-bound
3.6 Transfer Bandwidth and the Signal Logger Device tracks provided in addition to hardware counters
3.7 The First Execution Is Not a Measurement Target Why warm-up runs are needed for accurate measurement
3.8 Cost and Loss Performance overhead of device tracing and record drops

3.1 Trace Collection Modes

Flag Additionally collected
--runtime-trace Device kernel and memory transfer timeline
--device-trace Timeline + per-kernel hardware counters and the signal logger

The --device-trace option includes timeline collection, so use only one of the two flags. Both options are off by default, so a run in the form haprof -- APP without either flag collects only host-side records and incurs no device profiling cost.

1
2
haprof --runtime-trace -- python infer.py     # trace device compute and timing
haprof --device-trace  -- python infer.py     # analyze per-kernel time (with counters)

Unsupported driver or firmware

In environments where the driver or firmware does not support device profiling, the configuration is rejected and the trace records only host-side information (see Troubleshooting for details).


3.2 Selecting the Target Device

By default, device records are collected only from a single device, index 0. There is no CLI flag; specify it with the HAPTI_TOOL_DEVICE_ID environment variable.

1
HAPTI_TOOL_DEVICE_ID=1 haprof --device-trace -- python infer.py

Single Device per Trace

A multi-device process outputs records only for the selected device. The host timeline covers all calls, so work done on other devices appears as host activity with an empty device track below it. In multi-rank runs that use one device per rank, use the --profile-children option.


3.3 When Device Records Are Collected

This is the most important point to keep in mind when analyzing the device timeline.

Asynchronous APIs return control right after enqueuing work, so device activity is not yet recorded at API call time. Device records are written to the record ring when a processing batch completes, and the profiler's reader thread collects them continuously.

Even without an explicit synchronization call, the kernel driver (KMD) automatically appends a wait command at the end of a batch, so device tracks are not lost. However, records become visible at the batch boundary, not at the time of the host API call.

Records missing when the process exits early

The tool drains the record ring one last time at process exit, but no records exist for batches the device has not completed by then. To see device execution without gaps, synchronize the stream right before the program exits.

For details on interpreting trace results, see Trace Data Analysis Guide: Caveats When Reading the Device Timeline.


3.4 Hardware Counters

With the --device-trace option, hardware counter samples are recorded together with the kernel slice for each kernel execution. The device supports 41 events across 6 blocks; the full catalog is in the Counter Reference.

Default Counter Scope

The default configuration registers only 8 AXIBRG, 2 DMC, and 10 UTIL events of core 0. Counters of the CDMA, CGDMA, and MPU blocks and of core 1 and above are not collected by default. If you need a different combination, specify up to 32 directly with the HAPTI_COUNTERS environment variable.

1
2
HAPTI_COUNTERS="cgdma:0:inrob_beat,cgdma:0:inrob_stall_supply,cgdma:0:inrob_stall_demand,util:0:mpu_active,util:0:sample_total" \
  haprof --device-trace -- python infer.py

Diagnostic Purpose by Block

Block Main diagnostic target
UTIL Occupancy of each unit inside the core (matrix, vector, special function, DMA, and so on)
CGDMA Data movement state of the core group DMA and which side causes stalls
AXIBRG Transfer state of the core-DRAM AXI bridge and write completion latency
CDMA Per-core DMA beat transfer volume
DMC Whether SRAM bank conflicts occur
MPU Raw work of the matrix compute unit

Display rules

The values shown in the trace are not plain raw counts. Free-running counter values are converted to the difference (delta) from the previous execution, the UTIL block is shown as percentages, and some blocks show additional derived metrics. For the exact notation and formulas, see Counter Reference: Display Rules.

PyTorch Kineto integration

The PyTorch Kineto plugin path collects only the kernel timeline and does not collect hardware counters.


3.5 Interpreting Counters

In data collected with --device-trace, the kernel slice arguments (args) show percentage metrics such as AXIBRG.c0.read util%. The % in a metric name means it is a percentage, mostly the ratio to the total cycles (transfer + wait) in that transfer direction. For the formula of each metric, see Counter Reference: Bandwidth and Stall Breakdown. Of these, supply-stall% and demand-stall% tell you directly whether the bottleneck is memory (DRAM) or the compute core, so check them first.

Metric Interpretation and diagnosis
High util% The data path is actively transferring data
High supply-stall% Waiting for DRAM responses — memory-bound
High demand-stall% Waiting for the compute core to produce or consume data — compute-bound
High bresp-wait% Delay in confirming the write response
High bank-skew% Conflicts concentrate on a specific SRAM bank — the data layout needs improvement

Interpreting derived values

CGDMA bandwidth (GB/s) is a utilization metric scaled by the peak defined by the firmware, not a direct measurement of bytes. UTIL percentages are a sampled duty cycle, not cycle-exact occupancy, so units that operate only in short bursts can be under-measured. Use both metrics only for relative comparison between executions, and do not interpret them as absolute physical quantities.


3.6 Transfer Bandwidth and the Signal Logger

In device tracing mode, two tracks are added in addition to the hardware counters. Transfer bandwidth is also recorded with --runtime-trace; the signal logger is recorded only with --device-trace.

  1. Memory transfer bandwidth (memory bandwidth) — H2D (host-to-device) and D2H (device-to-host) transfer slices of 4 KiB or more carry a bandwidth value, and the haprof trace gets per-device Memcpy H2D bandwidth and Memcpy D2H bandwidth counter tracks. This value is a measured bandwidth: transferred bytes divided by the slice duration. The slice also covers host page pinning and the completion wait, so read it as a lower bound on what the link carried. D2D transfers and small transfers under 4 KiB are excluded from the bandwidth calculation because fixed costs dominate them.
  2. Signal logger (Signal Logger) — Records the on/off signal state of the main units inside the device for each kernel execution and shows it under the Device Trace <n> group of the trace as CG<group>_<unit> (for example CG0_MPU) and cgdma_<n> tracks. The signal data of each execution is aligned to the range of the kernel slice with the same correlation id.

3.7 The First Execution Is Not a Measurement Target

Exclude the first execution from performance analysis for the following two independent reasons.

  1. Kernel binary loading time — The first call on each device includes loading the kernel binary, so the first execution time includes the initialization cost.
  2. No counter baseline — The counter delta of the first execution is taken against a baseline of 0, so it includes counts from before profiling started.

Profile executions that have reached steady state after enough warm-up.

1
2
3
4
5
for _ in range(3):
    model(x)                      # initial warm-up

with hapti.annotate("steady-state"):
    model(x)                      # range to analyze

3.8 Cost and Loss

Device tracing adds performance overhead, and counter collection mode costs more than timeline collection.

Profiling mode Source of added overhead
--runtime-trace Device record ring collection cost
--device-trace Per-kernel-execution counter sampling and signal logger buffer read cost

Because the cost per execution is fixed, the share of overhead in total run time grows for workloads that run small kernels frequently (fine-grained workloads). Results remain valid for relative comparison between executions, but be careful when judging absolute wall-clock performance.

Record Drops

If data is lost during record collection, the tool reports it with a [hapti-tool] warning log at exit.

Warning message Meaning and cause
... bytes of device records were overwritten before the reader drained them The reader thread could not keep up with the record ring and an overrun occurred. Part of the timeline has gaps
... device records were dropped by the driver before reaching the ring The driver lost records before writing them to the record ring
signal logger: skipped ... launches whose device kernel record never arrived Signal logger data from executions with a missing kernel record is excluded from the visualization
dropped ... records (buffer capacity exceeded) Records were discarded because the profiler's internal record buffer was full

Next Steps