Skip to content

5. Paged KV Cache

vLLM-style paged attention extends the KV cache each time a request grows longer. Per-request KV addresses are translated, in units of HW pages, into addresses in a physical pool, and the torch.ops.lpu.vmm_* operators manage these HW pages through the runtime VMM API. You do not manage the HW pages directly; you call only these operators. Upstream vLLM kernels locate KV blocks through a block table; on LPU each request slot has its own KV address and the hardware translates it per HW page.

These operators are mostly called by the workers of the vLLM integration. You call them yourself when you write a serving engine or a KV connector.

Contents

Section Objective
5.1 Components What the pool, KV address tensors, and mappings are
5.2 Creating the Pool Pool tensor shape, initialization, and lifetime
5.3 Initialization Order Order of address reservation, pool creation, and VMM state initialization
5.4 Mapping Every Step Unmap → map arguments
5.5 Flow in vLLM The order in which the worker calls the operators
5.6 Rules Pool lifetime, kernel completion

5.1 Components

Component Created by Role
Pool vmm_create_kv_pool Physical memory that holds the KV. KEY and VALUE are split along dim 0
KV address tensors vmm_reserve (once each for KEY and VALUE) The addresses kernels read KV from. Addresses only, with no physical memory
Device VMM state vmm_init_blocks Initializes the device VMM state from the pool. One per device
Mappings vmm_map_blocks_batch / vmm_unmap_blocks_batch Grow or return each request slot's KEY and VALUE pages to match its token count
Block — Logical page unit in which the KV cache is mapped. In general bundles one or more HW pages. Its size is set by block_size per operator
HW page — Address-translation unit in which blocks are placed in device memory, managed by the runtime

A request slot is the row number of one request's KV addresses, not a vLLM slot_mapping slot (a token position).

Engineering sample

The current device (engineering sample) has the following constraints.

  • In the pool and in map/unmap, one block is one token (block_size=1). Address reservation and vmm_init_blocks treat one request slot row as one block (block_size = the maximum tokens per request). HW pages are allocated per request, and one HW page holds many tokens.
  • A request's blocks only grow. While a request is live, mapping only adds blocks; they are released per request (slot) all at once.
  • Prefix caching works as multi-turn continuation. When a new request begins with the full KV of an earlier request (as the next turn of the same conversation does), it extends that KV. The vLLM examples are written for prefix caching turned off (--no-enable-prefix-caching). See vLLM KV cache.

KV address tensors are passed to kernels like ordinary lpu tensors, but positions that are not mapped do not hold valid data. For the behavior of the runtime entry points, see Runtime memory model §3.7.

Layers from the vLLM worker through the torch.ops.lpu.vmm_* operators and the HaVmm* runtime down to the HW pages

The torch operators and the HaVmm* runtime manage the HW pages

5.2 Creating the Pool

1
2
3
4
5
pool = torch.ops.lpu.vmm_create_kv_pool(
    num_tokens, num_attn_layers, num_kv_heads, 1, head_dim,
    dtype=torch.bfloat16, device="lpu:0",
)
# pool.shape == (2, num_tokens, num_attn_layers, num_kv_heads, 1, head_dim)
  • Shape — Dim 0 is {KEY, VALUE}. One block holds one token, so block_size is 1, and dim 1 is the pool's token budget. Each token occupies L * H * D * itemsize bytes.
  • Initialization — The pool is zero-filled with zero_() before it is returned. The fill takes the same path described in Device Semantics §2.7.
  • Alignment — page_size (default 4096) sets the alignment of the pool's base address. Keep the default.
  • Lifetime — The pool tensor owns the pool memory. The physical memory is freed when the last reference to the tensor is dropped; there is no separate operator to free it.

vmm_create_kv_pool has a fake kernel, so it does not touch the device under FakeTensorMode. For operator details, see VMM API §Pool.

5.3 Initialization Order

vmm_init_blocks takes the pool, so create the pool first and pass it to vmm_init_blocks as the last step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import torch

L, H, D, dtype, bits = 2, 2, 128, torch.bfloat16, 16
R = 1024        # number of request slot rows
T = 4096        # maximum tokens per request (max_model_len)
S = 4           # number of concurrent requests (max_num_seqs)
N = T * S       # the pool's token budget

k = torch.ops.lpu.vmm_reserve(R, L, H, T, D, "key", dtype)
v = torch.ops.lpu.vmm_reserve(R, L, H, T, D, "value", dtype)
pool = torch.ops.lpu.vmm_create_kv_pool(N, L, H, 1, D, dtype=dtype, device="lpu:0")
torch.ops.lpu.vmm_init_blocks(S, 1, L, H, T, D, bits, pool)
  • KEY and VALUE addresses are reserved as one block (block_size = T tokens) for each request slot row.
  • In vmm_init_blocks, max_num_blocks_per_row × block_size (1 × T) is the maximum number of tokens per request. It must equal the token depth of the address reservation.
  • The pool must be large enough to hold the KEY and VALUE pages of one request. If it is too small, vmm_init_blocks fails.
  • num_rows, the first argument of vmm_init_blocks, is the number of request slots, similar to vLLM's max_num_seqs (request slots are currently limited to 1024).

To tear down, unmap every request, release the pool tensor, and then call vmm_deinit. KV address tensors have no physical memory behind them, so they need no separate release.

1
2
del pool
torch.ops.lpu.vmm_deinit()

5.4 Mapping Every Step

On every scheduler step, unmap finished requests first, then map the requests that grew. Unmapping first lets a slot freed in a step be reused in that same step. Use the batch variants for both calls.

Unlike CUDA's cuMemMap, vmm_map_blocks_batch does not take a caller-chosen physical location: you pass how many tokens each request slot has grown by, and HW pages are assigned from the pool.

Argument Meaning
vmm_tensors [k, v]
row_indices Request slot rows 0–1023. Required
cached_blocks_per_req Number of tokens already mapped
new_block_ids Values are ignored. Leave it empty or make its length sum(new_blocks_per_req)
new_blocks_per_req Number of new tokens
pool Pass the pool tensor from vmm_create_kv_pool. On the current engineering sample, pages are allocated from the pool registered with vmm_init_blocks
block_size 1
unmap blocks_per_req The request's full mapped token count; returned per request all at once

The values of cached_block_ids and of the unmap block_ids are ignored. Pass [], or, if you pass a list, make its length equal to the sum of the counts.

1
2
3
4
5
6
7
# Return finished slot 5 (300 tokens)
torch.ops.lpu.vmm_unmap_blocks_batch([k, v], [5], [], [300])

# Slot 3 grows from 10 tokens to 12; slot 7 is a new request with 5 tokens
torch.ops.lpu.vmm_map_blocks_batch(
    [k, v], [3, 7], [], [10, 0], pool, [0] * 7, [2, 5]
)

Map adds only the KEY and VALUE pages that cover the change in token count. Pass the exact number of tokens already mapped on every step.

5.5 Flow in vLLM

The workers of the vLLM integration manage the KV cache in the following order.

The vLLM worker reserves, creates the pool, and initializes during KV cache initialization, then calls unmap and map on every scheduler step

VMM call order in the vLLM worker

  1. KV cache initialization (once) — Call vmm_reserve(1024, L, H, max_model_len, D, "key"/"value", dtype) twice, create the pool with vmm_create_kv_pool(num_blocks, L, H, 1, D, dtype=..., device=...), then register it with vmm_init_blocks(max_num_seqs, 1, L, H, max_model_len, D, bits, pool). num_blocks is the pool's total token budget.
  2. Every step — Return released slots in a single vmm_unmap_blocks_batch call. Pass every slot whose token count grew to vmm_map_blocks_batch. The operator skips slots that need no new page. A slot is returned when its request finishes or is preempted. With prefix caching on, a finished request's slot may be kept for the next turn.
  3. Attention metadata — The worker passes each request's hardware request slot number to the kernel. The kernel locates KV by this number instead of vLLM's block table.

The plugin fixes vLLM's --block-size at 1, so vLLM's num_blocks is the pool's token count. The max_model_len passed as block_size to address reservation and vmm_init_blocks is unrelated to this setting.

For serving arguments, see vLLM arguments.

5.6 Rules

Do not release the pool tensor first

The pool's physical memory is freed when the last reference to the pool tensor is dropped. Releasing the pool while requests are still mapped leaves the HW pages pointing at freed memory. Keep the pool tensor alive until every request is unmapped.

Unmap only after the kernels reading the pages have finished

Do not assume that unmap waits for running kernels. Make sure the kernels that read the slot have finished (for example, with torch.accelerator.synchronize()), then unmap the slot, and only then hand it to another request.

torch.compile: vmm_reserve has no fake kernel

Calling vmm_reserve under FakeTensorMode or during torch.compile tracing issues a real device call. Reserve addresses and create the pool during engine initialization, outside compiled regions.

  • Current device — Every VMM operator is routed to the calling thread's current device. The pool must be a tensor on that device.
  • Reinitialization — Calling vmm_init_blocks again tears down the existing state first.
  • Errors — If a runtime call fails, the operator raises a RuntimeError of the form <operator>: <runtime entry point> failed: err=<code>.

Next Steps