5. Annotations¶
The profiler can observe runtime API calls, but it cannot infer the application's logic or model structure. Annotations make that structure visible on the trace timeline.
This chapter covers how to apply Range and Mark annotations, where to place annotations, and how to manage overhead in production. For exact function signatures, see the Python API Reference.
Contents¶
| Section | Objective |
|---|---|
| 5.1 Why Annotations | The limits of an unannotated trace and the effect of adding annotations |
| 5.2 Range Annotations | Ranges with a context manager and push/pop |
| 5.3 Non-scoped Range | Measuring ranges that cross function boundaries or overlap each other |
| 5.4 Mark Annotations | Recording a single event in time |
| 5.5 Where to Place Annotations | Key boundaries to annotate for bottleneck analysis |
| 5.6 Annotations in Production Code | Conditional execution and overhead patterns |
| 5.7 PyTorch Profiler Integration | How annotations behave in Kineto integration mode, and caveats |
5.1 Why Annotations¶
The profiling trace of an inference program without annotations shows only a long list of low-level API calls. From such a timeline alone you cannot tell the boundary between the prefill and decode phases or the extent of each request, which makes it hard to diagnose how much latency a given operation or piece of logic adds.
Adding just a few Range annotations makes the structure of the timeline visible. It greatly improves analysis with almost no code changes.
5.2 Range Annotations¶
1 2 3 4 5 6 7 8 | |
hapti.annotate() works as a Python context manager. It opens a
Range when the block is entered and closes it automatically when the block exits, and still closes it if an exception is raised in the block.
Explicit Range Control with Push/Pop¶
When you need range control outside the usual Python block structure, use the explicit push/pop entry points.
1 2 3 | |
Push and pop must pair up
range_push() and
range_pop() are managed as a per-thread stack. If a
pop call is missed because of an early return or incomplete exception handling, the Range stays open and all
later ranges are wrongly nested under it. Prefer hapti.annotate(), which always closes the Range, and when
debugging, check the nesting depth returned by the two functions.
Hierarchical Nesting¶
Ranges can be nested, and a hierarchical sub-scope structure makes the timeline much easier to read.
1 2 3 4 5 | |
5.3 Non-scoped Range¶
range_start() and
range_stop() are APIs for measuring independent ranges that
are not constrained by strict stack nesting. Use them when a range starts in one function and ends in another, or
when multiple Ranges cross or overlap each other.
1 2 3 | |
Unlike the push/pop stack, they are managed by process-wide handles, so two range_start() ranges can overlap and a
range can be ended from a different thread.
5.4 Mark Annotations¶
A Mark shows a single event that occurred at a specific point in time on the timeline, not a range with a duration.
1 | |
Use it to quickly identify a specific event on the timeline (for example, major execution boundaries, the time the first token is generated, or the time a cache eviction occurs). If the goal is to measure the duration of an operation, use a Range annotation instead.
5.5 Where to Place Annotations¶
Annotate mainly the key boundaries that serve as reference points when you look for the cause of a performance bottleneck.
| Target area | Reason and benefit |
|---|---|
| Prefill / decode phase split | The two phases have completely different compute characteristics and overhead profiles |
| Request and batch boundaries | Lets you compare performance variance and latency between individual requests or batches |
| Major model layers (Attention, MLP, Sampling) | Shows the latency and share of each model compute block |
| CPU control and data preprocessing | Without annotations these appear as idle gaps on the timeline and are hard to explain |
Do not overuse annotations in hot inner loops
Avoid annotations inside hot inner loops that run at high frequency. Every annotation call produces a separate record. One or two annotations per token are fine, but adding them indiscriminately to every per-layer operation makes the trace overly complex and adds unnecessary measurement overhead.
5.6 Annotations in Production Code¶
In a normal run with profiling disabled, the overhead of annotation API calls is negligible, but to remove the overhead entirely and keep code dependencies tidy, you can apply a conditional pattern like the following.
1 2 3 4 5 6 7 8 9 10 11 12 | |
Wrapping the profiler integration code in your own helper module to isolate the package dependency is also recommended.
Safe behavior without the launcher
import hapti always works, including in normal environments outside the haprof launcher. When profiling is
not enabled, annotation API calls are safe no-ops that record no data.
5.7 PyTorch Profiler Integration¶
If you already use a torch.profiler-based observation setup, registering the Kineto plugin integrates LPU device
execution into the PyTorch profiler output timeline instead of a separate trace. For how the Kineto plugin works and
how to choose between it and haprof, see Chapter 6 PyTorch Profiler Integration.
hapti annotation limits with Kineto
Annotations written with hapti.annotate() are recorded only when collecting a standalone trace with haprof.
The PyTorch Kineto integration path does not collect them, so to show user ranges on the torch.profiler
timeline, use the standard PyTorch interface torch.profiler.record_function(). In a codebase that uses both
collection modes, applying both annotation styles side by side is recommended.
Next Steps¶
- PyTorch Profiler Integration
- Trace Data Analysis Guide
- Python API — Exact function signatures