Skip to content

6. PyTorch Profiler Integration

haprof creates a separate trace file for the whole program process. For teams that already use torch.profiler, this adds a separate profiler output next to the existing PyTorch trace, so CPU operations must be checked in the PyTorch trace and device activity in the haprof trace, and the two compared by eye.

The Kineto plugin solves this. It integrates LPU activity into the PyTorch profiler's own trace, so an aten::matmul operation and the device kernels it caused are connected directly by flow arrows. This chapter covers how the plugin works, how to register and use it, what goes into the trace, and how to choose between it and haprof.

Contents

Section Objective
6.1 Kineto Integration Structure Where vendor tracers plug into the architecture
6.2 Registering the Plugin One line of code that loads the plugin
6.3 Running the Profiler How to specify the PrivateUse1 activity
6.4 Collected Items and Flow Arrows Activity kinds and flow arrow linkage
6.5 Device Timeline Collection Conditions for enabling device records, and diagnostic messages
6.6 When Records Are Converted When records are converted into the trace, and loss conditions
6.7 Choosing Between haprof and the Kineto Plugin Guide to choosing a profiler for each environment

6.1 Kineto Integration Structure

libkineto is PyTorch's hardware abstraction layer for profiling. torch.profiler outputs a single Chrome trace file, but the per-device events in it are filled in by each hardware vendor's tracer. The point where these tracers plug in is the IActivityProfiler ABI.

torch.profiler drives vendor tracers such as CUPTI, roctracer, and the hapti Kineto plugin through libkineto and the IActivityProfiler ABI

Structure of vendor tracers plugged into libkineto

The LPU backend connects through the same standard integration path that NVIDIA and AMD use. No dedicated viewer or format is needed, and the torch.profiler UI, output formats, and Python entry points work as is. The plugin's only role is to implement the IActivityProfiler interface so that Kineto can drive hapti just like any other backend.

6.2 Registering the Plugin

1
2
3
import hapti.kineto

hapti.kineto.register()

hapti.kineto.register() registers the plugin with Kineto as an IActivityProfiler. Call it once per process; calling it more than once is safe.

Why runtime calls are visible

The plugin uses the same instance of the hart runtime already loaded in the process, so it observes every Ha* runtime API call the application makes. If runtime calls do not appear, see Troubleshooting.

If the plugin build is missing, registration fails immediately with an error.

1
FileNotFoundError: kineto plugin not found at ...

6.3 Running the Profiler

To collect LPU event data, ProfilerActivity.PrivateUse1 must be included in the activities list.

1
2
3
4
5
6
7
8
9
import hapti.kineto
from torch.profiler import ProfilerActivity, profile

hapti.kineto.register()

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.PrivateUse1]) as prof:
    ...                                  # run a workload that makes Ha* runtime calls

prof.export_chrome_trace("trace.json")

PrivateUse1 is the dispatch key that PyTorch reserves for external accelerator backends, and it is the slot where the LPU is registered. It shares the same key as the lpu device described in Torch Device Semantics.

6.4 Collected Items and Flow Arrows

Depending on the activity kinds Kineto requests, the plugin includes the following data in the trace.

Kineto activity kind LPU mapping
PRIVATEUSE1_RUNTIME Runtime API calls (Ha*). Memory allocation calls include the number of bytes allocated
PRIVATEUSE1_DRIVER Runtime lower-layer calls
CONCURRENT_KERNEL Device kernel execution ranges (see Device Timeline Collection)
GPU_MEMCPY Host↔device memory transfers and bandwidth information
EXTERNAL_CORRELATION Links between CPU operations and LPU activity

GPU_MEMCPY is Kineto's generic name for this activity kind. The plugin reuses it because a custom kind would not be displayed correctly in the torch.profiler UI.

Items not collected on the Kineto path

  • Hardware counters — Only the kernel timeline is collected; counters are not included. If you need hardware counter analysis, use haprof --device-trace.
  • hapti annotations — Ranges specified with hapti.annotate() are not recorded on this path. To name ranges in the PyTorch trace, use the standard PyTorch API torch.profiler.record_function() (see PyTorch Profiler Integration).

Flow Arrows

CPU operations and LPU activity are linked in two steps. PyTorch assigns an external ID to each CPU operation, and the plugin maps the correlation id of each runtime call made inside that operation to the external ID. Kernels and memory transfers share the same correlation id as the runtime call that caused them, so torch.profiler automatically draws flow arrows from the CPU operation range to the device activity range.

The external ID of a CPU operation is mapped to the correlation id of a runtime call, and a flow arrow continues to the device kernel with the same correlation id

Linkage from external ID → correlation id → device record

With this structure you can trace back which Python/PyTorch code called a given device kernel.

6.5 Device Timeline Collection

When the CONCURRENT_KERNEL activity is requested, the plugin sets up the device record ring when the profiling region (profile() block) starts, and collects the records and releases the ring when the region ends. When the GPU_MEMCPY activity is also requested, memory transfer record collection is enabled as well.

Environment variable Effect and description
HAPTI_KINETO_DEVICE_TRACE Disables device timeline collection if the value is 0, false, or off (default: enabled)
HAPTI_KINETO_DEVICE Specifies the target device explicitly. If unset, uses the current device of the thread that opened the profiling region

If the device timeline cannot be created or a problem occurs during collection, the plugin reports it through a torch.profiler warning.

Warning message Meaning and action
No LPU device (S6) time in this trace: ... Device profiling setup failed. The cause is shown at the end of the message
LPU device (S6) time was captured from device N by default No device was bound when the region started, so the default device (N) was used. Open the region after the device is already in use, or specify it directly with HAPTI_KINETO_DEVICE
... LPU device activities fell outside the capture window and were dropped Kernels not yet completed when the region ended are excluded from collection
... bytes of device records were overwritten before the reader drained them A record ring overrun left gaps in the device timeline
N LPU devices in one session; only device M gets a named trace row The region used several devices, and only the lowest-numbered one gets a named row

6.6 When Records Are Converted

On the Kineto path, all records are converted at once when the region ends (stop()). No data appears in the trace during the profiling region, and if the process terminates abnormally (SIGKILL or similar), the data accumulated in the buffer is lost.

6.7 Choosing Between haprof and the Kineto Plugin

The two methods are alternatives to choose between by purpose, not tools to use together. If both are enabled in the same process, the same activity is collected twice, and contention for the single device record ring can cause device activity to be missing from one of the traces.

Aspect Kineto plugin haprof
Output trace A single integrated torch.profiler output file Separate .pftrace and .json files
Profiling scope Only the profile() code block The entire run of the target process
Linkage to CPU operations Automatic, through flow arrows Requires matching correlation ids manually
Source code changes Requires one line of register() code None (uses LD_PRELOAD)
Non-PyTorch workloads Not supported Supported
Device timeline collection Supported (enabled by default) Supported (--runtime-trace)
Hardware counter collection Not supported Supported (--device-trace)
hapti annotations Not supported (use record_function) Supported
Viewers TensorBoard, Perfetto, Chrome tracing Perfetto, Chrome tracing

When to use which:

  • The team already has a torch.profiler-based analysis pipeline → Kineto plugin
  • You need hardware counter analysis → haprof --device-trace
  • The profiling target is a general C++/Python program outside PyTorch → haprof

Next Steps