5. Paged KV Cache¶
vLLM-style paged attention extends the KV cache each time a request grows longer. Per-request
KV addresses are translated, in units of HW pages, into addresses in a physical pool, and the torch.ops.lpu.vmm_*
operators manage these HW pages through the runtime VMM API. You do not manage the HW pages directly; you call only these operators.
Upstream vLLM kernels locate KV blocks through a block table; on LPU each request slot has its own KV address and the
hardware translates it per HW page.
These operators are mostly called by the workers of the vLLM integration. You call them yourself when you write a serving engine or a KV connector.
Contents¶
| Section | Objective |
|---|---|
| 5.1 Components | What the pool, KV address tensors, and mappings are |
| 5.2 Creating the Pool | Pool tensor shape, initialization, and lifetime |
| 5.3 Initialization Order | Order of address reservation, pool creation, and VMM state initialization |
| 5.4 Mapping Every Step | Unmap → map arguments |
| 5.5 Flow in vLLM | The order in which the worker calls the operators |
| 5.6 Rules | Pool lifetime, kernel completion |
5.1 Components¶
| Component | Created by | Role |
|---|---|---|
| Pool | vmm_create_kv_pool |
Physical memory that holds the KV. KEY and VALUE are split along dim 0 |
| KV address tensors | vmm_reserve (once each for KEY and VALUE) |
The addresses kernels read KV from. Addresses only, with no physical memory |
| Device VMM state | vmm_init_blocks |
Initializes the device VMM state from the pool. One per device |
| Mappings | vmm_map_blocks_batch / vmm_unmap_blocks_batch |
Grow or return each request slot's KEY and VALUE pages to match its token count |
| Block | — | Logical page unit in which the KV cache is mapped. In general bundles one or more HW pages. Its size is set by block_size per operator |
| HW page | — | Address-translation unit in which blocks are placed in device memory, managed by the runtime |
A request slot is the row number of one request's KV addresses, not a vLLM slot_mapping slot (a token position).
Engineering sample
The current device (engineering sample) has the following constraints.
- In the pool and in map/unmap, one block is one token (
block_size=1). Address reservation andvmm_init_blockstreat one request slot row as one block (block_size= the maximum tokens per request). HW pages are allocated per request, and one HW page holds many tokens. - A request's blocks only grow. While a request is live, mapping only adds blocks; they are released per request (slot) all at once.
- Prefix caching works as multi-turn continuation. When a new request begins with the full KV of an earlier
request (as the next turn of the same conversation does), it extends that KV. The vLLM examples are written for prefix caching turned off (
--no-enable-prefix-caching). See vLLM KV cache.
KV address tensors are passed to kernels like ordinary lpu tensors, but positions that are not mapped do not hold
valid data. For the behavior of the runtime entry points, see
Runtime memory model §3.7.
The torch operators and the HaVmm* runtime manage the HW pages
5.2 Creating the Pool¶
1 2 3 4 5 | |
- Shape — Dim 0 is
{KEY, VALUE}. One block holds one token, soblock_sizeis1, and dim 1 is the pool's token budget. Each token occupiesL * H * D * itemsizebytes. - Initialization — The pool is zero-filled with
zero_()before it is returned. The fill takes the same path described in Device Semantics §2.7. - Alignment —
page_size(default 4096) sets the alignment of the pool's base address. Keep the default. - Lifetime — The pool tensor owns the pool memory. The physical memory is freed when the last reference to the tensor is dropped; there is no separate operator to free it.
vmm_create_kv_pool has a fake kernel, so it does not touch the device under FakeTensorMode. For operator details,
see VMM API §Pool.
5.3 Initialization Order¶
vmm_init_blocks takes the pool, so create the pool first and pass it to vmm_init_blocks as the last step.
1 2 3 4 5 6 7 8 9 10 11 12 | |
- KEY and VALUE addresses are reserved as one block (
block_size = Ttokens) for each request slot row. - In
vmm_init_blocks,max_num_blocks_per_row × block_size(1 × T) is the maximum number of tokens per request. It must equal the token depth of the address reservation. - The pool must be large enough to hold the KEY and VALUE pages of one request. If it is too small,
vmm_init_blocksfails. num_rows, the first argument ofvmm_init_blocks, is the number of request slots, similar to vLLM'smax_num_seqs(request slots are currently limited to 1024).
To tear down, unmap every request, release the pool tensor, and then call vmm_deinit. KV address tensors have no physical memory behind them, so they need no separate release.
1 2 | |
5.4 Mapping Every Step¶
On every scheduler step, unmap finished requests first, then map the requests that grew. Unmapping first lets a slot freed in a step be reused in that same step. Use the batch variants for both calls.
Unlike CUDA's cuMemMap, vmm_map_blocks_batch does not take a caller-chosen physical location: you pass how many
tokens each request slot has grown by, and HW pages are assigned from the pool.
| Argument | Meaning |
|---|---|
vmm_tensors |
[k, v] |
row_indices |
Request slot rows 0–1023. Required |
cached_blocks_per_req |
Number of tokens already mapped |
new_block_ids |
Values are ignored. Leave it empty or make its length sum(new_blocks_per_req) |
new_blocks_per_req |
Number of new tokens |
pool |
Pass the pool tensor from vmm_create_kv_pool. On the current engineering sample, pages are allocated from the pool registered with vmm_init_blocks |
block_size |
1 |
unmap blocks_per_req |
The request's full mapped token count; returned per request all at once |
The values of cached_block_ids and of the unmap block_ids are ignored. Pass [], or, if you pass a list, make its
length equal to the sum of the counts.
1 2 3 4 5 6 7 | |
Map adds only the KEY and VALUE pages that cover the change in token count. Pass the exact number of tokens already mapped on every step.
5.5 Flow in vLLM¶
The workers of the vLLM integration manage the KV cache in the following order.
VMM call order in the vLLM worker
- KV cache initialization (once) — Call
vmm_reserve(1024, L, H, max_model_len, D, "key"/"value", dtype)twice, create the pool withvmm_create_kv_pool(num_blocks, L, H, 1, D, dtype=..., device=...), then register it withvmm_init_blocks(max_num_seqs, 1, L, H, max_model_len, D, bits, pool).num_blocksis the pool's total token budget. - Every step — Return released slots in a single
vmm_unmap_blocks_batchcall. Pass every slot whose token count grew tovmm_map_blocks_batch. The operator skips slots that need no new page. A slot is returned when its request finishes or is preempted. With prefix caching on, a finished request's slot may be kept for the next turn. - Attention metadata — The worker passes each request's hardware request slot number to the kernel. The kernel locates KV by this number instead of vLLM's block table.
The plugin fixes vLLM's --block-size at 1, so vLLM's num_blocks is the pool's token count. The max_model_len passed as block_size to address reservation and vmm_init_blocks is unrelated to this setting.
For serving arguments, see vLLM arguments.
5.6 Rules¶
Do not release the pool tensor first
The pool's physical memory is freed when the last reference to the pool tensor is dropped. Releasing the pool while requests are still mapped leaves the HW pages pointing at freed memory. Keep the pool tensor alive until every request is unmapped.
Unmap only after the kernels reading the pages have finished
Do not assume that unmap waits for running kernels. Make sure the kernels that read the slot have finished (for
example, with torch.accelerator.synchronize()), then unmap the slot, and only then hand it to another request.
torch.compile: vmm_reserve has no fake kernel
Calling vmm_reserve under FakeTensorMode or during torch.compile tracing issues a real device call. Reserve
addresses and create the pool during engine initialization, outside compiled regions.
- Current device — Every VMM operator is routed to the calling thread's current device. The pool must be a tensor on that device.
- Reinitialization — Calling
vmm_init_blocksagain tears down the existing state first. - Errors — If a runtime call fails, the operator raises a
RuntimeErrorof the form<operator>: <runtime entry point> failed: err=<code>.
Next Steps¶
- Migrating from CUDA: differences from CUDA code
- VMM API: per-operator signatures and examples
- Distributed: multi-device execution