Profiling¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5, HAPTI 0.0.5 · Last checked: 2026-10-15
Overview¶
To get the most out of vLLM HyperAccel, serving metrics such as TTFT (Time To First Token) and TPOT (Time Per Output Token) (LPU metrics) are not enough on their own; you also need low-level profiling that shows what actually runs on the LPU and how long it takes.
LPU profiling is handled by the HyperAccel profiler HAPTI (HyperAccel Profiling/Tracing Interface). With vLLM HyperAccel it is used in two ways.
| Method | Output | Use it when |
|---|---|---|
| 1. PyTorch Profiler (Kineto) | One torch.profiler Chrome trace with CPU ops and LPU activity together |
You want a chosen section only, with aten ops linked to LPU kernels |
2. HAPTI Profiler (haprof) |
.pftrace · .json per process |
You want the whole process without changing code, or hardware counters |
Both methods use the same HAPTI core to collect LPU runtime calls (Ha*), driver calls, kernel execution, and memory
transfers. HAPTI itself is covered in detail in Profiler.
Do not enable both methods in the same process
The same activity is collected twice, and the two contend for the single device record ring, so device activity can go missing from one of the traces.
1. PyTorch Profiler (Kineto)¶
Tracing in vLLM HyperAccel uses PyTorch's standard profiler, torch.profiler, and its hardware abstraction layer,
Kineto (libkineto). Just as NVIDIA plugs CUPTI and AMD plugs roctracer into Kineto, HAPTI plugs in as a Kineto
plugin (libhapti_kineto_plugin.so) in the IActivityProfiler slot. The LPU uses the PrivateUse1 slot PyTorch
reserves for out-of-tree accelerators, so adding ProfilerActivity.PrivateUse1 to the activity list collects LPU
activity.
1 2 3 4 5 6 7 | |
The result is a single Chrome trace written by torch.profiler. The plugin maps the external id PyTorch attaches to
each CPU op onto the correlation id of the runtime calls made inside that op, so flow arrows link an aten:: op
directly to the LPU kernels it launched. How the plugin works is described in
PyTorch Profiler Integration.
1.1 Registering the plugin¶
1 2 3 4 | |
hapti ships with the SDK. register() only needs to be called once per process and is safe to call repeatedly.
The plugin finds the same libha_runtime.so already loaded by the hart extension, by soname, and subscribes to
its interceptor. Importing hart first guarantees that it attaches to the runtime vLLM actually uses.
1.2 Profiling offline inference¶
By default, vLLM V1 starts the engine core (including the worker) in a separate process. torch.profiler only sees
activity in the process where it is opened, so for a single LPU the simplest approach is
VLLM_ENABLE_V1_MULTIPROCESSING=0, which runs the worker in the same process.
| profile_offline.py | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | |
Naming a range with record_function() makes it easy to find in the trace. hapti.annotate() annotations are not
recorded on the Kineto path, so use PyTorch's standard record_function().
1.3 Profiling multiple workers¶
When workers live in several processes, as with pipeline or data parallelism, use llm.collective_rpc() to open and
close the profiler in each worker, leaving one trace per rank. Sending a function to the engine core process
requires VLLM_ALLOW_INSECURE_SERIALIZATION=1.
| profile_pp.py | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | |
Export the traces before the engine shuts down. Records are converted all at once when the profiling range closes, so if a process is killed while the range is open, no data is left.
1.4 Additional options¶
| Option | Effect |
|---|---|
activities=[ProfilerActivity.PrivateUse1] |
Collect LPU activity only. The default collects CPU as well |
profile(..., record_shapes=True) |
Record input shapes and dtypes per op. Useful for telling per-element copies from bulk copies, at the cost of a larger trace |
HAPTI_KINETO_DEVICE_TRACE=0 |
Turn off the device kernel timeline (on by default), when you only need host-side calls |
HAPTI_KINETO_DEVICE=<index> |
Select the LPU whose device timeline is collected. Defaults to the current device of the thread that opened the range |
Set environment variables before opening the profile() range. The full list is in
Profiler environment variables.
1.5 What goes into the trace¶
| Trace category | Contents |
|---|---|
cpu_op |
PyTorch CPU ops such as aten:: |
user_annotation |
Ranges named with record_function() |
privateuse1_runtime |
LPU runtime API calls (Ha*). Kernel launches are HaLaunchKernel / HaLaunchKernelAsync |
privateuse1_driver |
Driver API calls (ha_*) |
kernel |
LPU device kernel execution |
gpu_memcpy |
Host ↔ LPU memory transfers. The name is Kineto's common term, kept as is |
Hardware counters are not collected on the Kineto path. If you need counters, use
haprof --device-trace.
1.6 Viewing the results¶
Open the exported .json in the Perfetto UI, chrome://tracing, or TensorBoard. See
Reading a Trace for how to read it.
2. HAPTI Profiler (haprof)¶
haprof is HAPTI's launcher. It injects HAPTI into the target process with LD_PRELOAD, so it records the whole
process without code changes, and with --device-trace it also collects per-kernel hardware counters.
1 | |
2.1 Profiling offline inference¶
Running the worker in the same process puts all activity in a single trace.
1 2 | |
This produces traces/qwen3.pftrace and traces/qwen3.json.
2.2 Profiling online serving¶
vllm serve runs the API server and the engine core in different processes. haprof does not follow child
processes by default, so enable --profile-children.
1 2 | |
Once the server is ready, send the requests you want to measure, then stop the server cleanly with Ctrl+C
(SIGINT). Traces are written when a process exits normally, so SIGKILL or a forced container stop leaves nothing.
Each process writes its own <hostname>_<pid>_<starttime> files. With several LPUs, as in pipeline parallelism, merge
the per-rank traces into one timeline and a straggler report.
1 | |
2.3 Choosing what to collect¶
| Goal | Command |
|---|---|
| Host-side time | haprof -- APP |
| LPU work flow and timing (kernel · transfer timeline) | haprof --runtime-trace -- APP |
| Why a kernel is slow (hardware counters) | haprof --device-trace -- APP |
| Per-rank delays across ranks | haprof --profile-children --device-trace -- APP, then haprof merge |
Start with the lightest mode that answers your question. Counter collection costs more than the timeline, and the
timeline costs more than host-only mode. To separate execution time cleanly per kernel, make kernel launches
synchronous with LEGATO_ASYNC_LAUNCH=0 while measuring. All options are in the haprof CLI
reference.
2.4 Viewing the results¶
| Format | Open it in |
|---|---|
.pftrace |
Perfetto UI. Recommended for large traces |
.json |
chrome://tracing and similar. Input to the haprof merge report |
3. Which One to Use¶
| PyTorch Profiler (Kineto) | haprof |
|
|---|---|---|
| Output | One torch.profiler trace |
.pftrace · .json per process |
| Scope | The profile() range |
The whole process |
| Link to CPU ops | Automatic, via flow arrows | Match correlation ids by hand |
| Code changes | Add register() and profile() |
None |
| Device timeline | Supported (on by default) | --runtime-trace |
| Hardware counters | — | --device-trace |
| Range names | record_function() |
hapti.annotate() |
- You already analyze with
torch.profiler, or want a specific request range alongsideatenops → Kineto - You want the whole server without changing code, or need hardware counters →
haprof