Skip to content

Profiling

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5, HAPTI 0.0.5 · Last checked: 2026-10-15

Overview

To get the most out of vLLM HyperAccel, serving metrics such as TTFT (Time To First Token) and TPOT (Time Per Output Token) (LPU metrics) are not enough on their own; you also need low-level profiling that shows what actually runs on the LPU and how long it takes.

LPU profiling is handled by the HyperAccel profiler HAPTI (HyperAccel Profiling/Tracing Interface). With vLLM HyperAccel it is used in two ways.

Method Output Use it when
1. PyTorch Profiler (Kineto) One torch.profiler Chrome trace with CPU ops and LPU activity together You want a chosen section only, with aten ops linked to LPU kernels
2. HAPTI Profiler (haprof) .pftrace · .json per process You want the whole process without changing code, or hardware counters

Both methods use the same HAPTI core to collect LPU runtime calls (Ha*), driver calls, kernel execution, and memory transfers. HAPTI itself is covered in detail in Profiler.

Do not enable both methods in the same process

The same activity is collected twice, and the two contend for the single device record ring, so device activity can go missing from one of the traces.


1. PyTorch Profiler (Kineto)

Tracing in vLLM HyperAccel uses PyTorch's standard profiler, torch.profiler, and its hardware abstraction layer, Kineto (libkineto). Just as NVIDIA plugs CUPTI and AMD plugs roctracer into Kineto, HAPTI plugs in as a Kineto plugin (libhapti_kineto_plugin.so) in the IActivityProfiler slot. The LPU uses the PrivateUse1 slot PyTorch reserves for out-of-tree accelerators, so adding ProfilerActivity.PrivateUse1 to the activity list collects LPU activity.

1
2
3
4
5
6
7
torch.profiler.profile(activities=[CPU, PrivateUse1])
      │
  libkineto ── CPU ops (aten::*, record_function ranges)
      │
  IActivityProfiler ── HAPTI Kineto plugin ── Ha* runtime · ha_* driver · LPU kernels · memory transfers
      │
  export_chrome_trace("trace.json")   ← one trace

The result is a single Chrome trace written by torch.profiler. The plugin maps the external id PyTorch attaches to each CPU op onto the correlation id of the runtime calls made inside that op, so flow arrows link an aten:: op directly to the LPU kernels it launched. How the plugin works is described in PyTorch Profiler Integration.

1.1 Registering the plugin

1
2
3
4
import hart          # load the LPU runtime (libha_runtime.so) first
import hapti.kineto

hapti.kineto.register()

hapti ships with the SDK. register() only needs to be called once per process and is safe to call repeatedly.

The plugin finds the same libha_runtime.so already loaded by the hart extension, by soname, and subscribes to its interceptor. Importing hart first guarantees that it attaches to the runtime vLLM actually uses.

1.2 Profiling offline inference

By default, vLLM V1 starts the engine core (including the worker) in a separate process. torch.profiler only sees activity in the process where it is opened, so for a single LPU the simplest approach is VLLM_ENABLE_V1_MULTIPROCESSING=0, which runs the worker in the same process.

profile_offline.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import os

os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0"   # run the worker in this process

import hart
import hapti.kineto
from torch.profiler import ProfilerActivity, profile, record_function
from vllm import LLM, SamplingParams

hapti.kineto.register()

llm = LLM(model="Qwen/Qwen3-8B", max_model_len=4096, enable_prefix_caching=False, async_scheduling=False)
params = SamplingParams(temperature=0, max_tokens=32)
prompts = ["The capital of France is", "What a KV cache does in LLM inference is"]

# Warm-up: keeps compilation and first-run costs out of the trace
llm.generate(prompts, params)

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.PrivateUse1]) as prof:
    with record_function("generate"):
        llm.generate(prompts, params)

prof.export_chrome_trace("qwen3_trace.json")

Naming a range with record_function() makes it easy to find in the trace. hapti.annotate() annotations are not recorded on the Kineto path, so use PyTorch's standard record_function().

1.3 Profiling multiple workers

When workers live in several processes, as with pipeline or data parallelism, use llm.collective_rpc() to open and close the profiler in each worker, leaving one trace per rank. Sending a function to the engine core process requires VLLM_ALLOW_INSECURE_SERIALIZATION=1.

profile_pp.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
import os

os.environ["VLLM_ALLOW_INSECURE_SERIALIZATION"] = "1"   # to send functions through collective_rpc

from contextlib import ExitStack
from pathlib import Path

from vllm import LLM, SamplingParams


def start_profile(worker):
    """Register HAPTI and start Kineto in the worker process."""
    import hart
    import hapti.kineto
    from torch.profiler import ProfilerActivity, profile

    hapti.kineto.register()
    stack = ExitStack()
    worker._hapti_profiler = stack.enter_context(
        profile(activities=[ProfilerActivity.CPU, ProfilerActivity.PrivateUse1])
    )
    worker._hapti_profile_stack = stack


def stop_profile(worker, output_dir):
    """Close the profiler and write this rank's trace."""
    profiler = worker._hapti_profiler
    worker._hapti_profile_stack.close()
    path = Path(output_dir) / f"pp_rank{worker.rank}_pid{os.getpid()}.json"
    profiler.export_chrome_trace(str(path))
    return str(path)


llm = LLM(model="Qwen/Qwen3-8B", max_model_len=4096, pipeline_parallel_size=2, dtype="bfloat16", enable_prefix_caching=False, async_scheduling=False)
params = SamplingParams(temperature=0, max_tokens=32)
llm.generate("The capital of France is", params)      # warm-up

out_dir = Path("hapti_profiles").resolve()
out_dir.mkdir(exist_ok=True)

llm.collective_rpc(start_profile, timeout=300)
try:
    llm.generate("The capital of France is", params)
finally:
    print(llm.collective_rpc(stop_profile, timeout=300, args=(str(out_dir),)))

Export the traces before the engine shuts down. Records are converted all at once when the profiling range closes, so if a process is killed while the range is open, no data is left.

1.4 Additional options

Option Effect
activities=[ProfilerActivity.PrivateUse1] Collect LPU activity only. The default collects CPU as well
profile(..., record_shapes=True) Record input shapes and dtypes per op. Useful for telling per-element copies from bulk copies, at the cost of a larger trace
HAPTI_KINETO_DEVICE_TRACE=0 Turn off the device kernel timeline (on by default), when you only need host-side calls
HAPTI_KINETO_DEVICE=<index> Select the LPU whose device timeline is collected. Defaults to the current device of the thread that opened the range

Set environment variables before opening the profile() range. The full list is in Profiler environment variables.

1.5 What goes into the trace

Trace category Contents
cpu_op PyTorch CPU ops such as aten::
user_annotation Ranges named with record_function()
privateuse1_runtime LPU runtime API calls (Ha*). Kernel launches are HaLaunchKernel / HaLaunchKernelAsync
privateuse1_driver Driver API calls (ha_*)
kernel LPU device kernel execution
gpu_memcpy Host ↔ LPU memory transfers. The name is Kineto's common term, kept as is

Hardware counters are not collected on the Kineto path. If you need counters, use haprof --device-trace.

1.6 Viewing the results

Open the exported .json in the Perfetto UI, chrome://tracing, or TensorBoard. See Reading a Trace for how to read it.


2. HAPTI Profiler (haprof)

haprof is HAPTI's launcher. It injects HAPTI into the target process with LD_PRELOAD, so it records the whole process without code changes, and with --device-trace it also collects per-kernel hardware counters.

1
haprof [options] -- <command> [args...]

2.1 Profiling offline inference

Running the worker in the same process puts all activity in a single trace.

1
2
VLLM_ENABLE_V1_MULTIPROCESSING=0 \
  haprof --runtime-trace -d traces -o qwen3 -- python generate.py

This produces traces/qwen3.pftrace and traces/qwen3.json.

2.2 Profiling online serving

vllm serve runs the API server and the engine core in different processes. haprof does not follow child processes by default, so enable --profile-children.

1
2
haprof --profile-children --runtime-trace -d traces -- \
  vllm serve Qwen/Qwen3-8B --max-model-len 4096 --no-enable-prefix-caching --no-async-scheduling

Once the server is ready, send the requests you want to measure, then stop the server cleanly with Ctrl+C (SIGINT). Traces are written when a process exits normally, so SIGKILL or a forced container stop leaves nothing.

Each process writes its own <hostname>_<pid>_<starttime> files. With several LPUs, as in pipeline parallelism, merge the per-rank traces into one timeline and a straggler report.

1
haprof merge -d traces -o traces/merged.pftrace --report-format both

2.3 Choosing what to collect

Goal Command
Host-side time haprof -- APP
LPU work flow and timing (kernel · transfer timeline) haprof --runtime-trace -- APP
Why a kernel is slow (hardware counters) haprof --device-trace -- APP
Per-rank delays across ranks haprof --profile-children --device-trace -- APP, then haprof merge

Start with the lightest mode that answers your question. Counter collection costs more than the timeline, and the timeline costs more than host-only mode. To separate execution time cleanly per kernel, make kernel launches synchronous with LEGATO_ASYNC_LAUNCH=0 while measuring. All options are in the haprof CLI reference.

2.4 Viewing the results

Format Open it in
.pftrace Perfetto UI. Recommended for large traces
.json chrome://tracing and similar. Input to the haprof merge report

3. Which One to Use

PyTorch Profiler (Kineto) haprof
Output One torch.profiler trace .pftrace · .json per process
Scope The profile() range The whole process
Link to CPU ops Automatic, via flow arrows Match correlation ids by hand
Code changes Add register() and profile() None
Device timeline Supported (on by default) --runtime-trace
Hardware counters — --device-trace
Range names record_function() hapti.annotate()
  • You already analyze with torch.profiler, or want a specific request range alongside aten ops → Kineto
  • You want the whole server without changing code, or need hardware counters → haprof

Next