VMM API¶
The torch.ops.lpu.vmm_* operators manage the paged KV cache. They create the physical KV pool,
reserve KEY and VALUE addresses, and map and unmap per-request pages on every scheduler step. These operators manage the
HW pages through the runtime VMM API. If you are new to these operators, read
the Paged KV Cache chapter first.
A block is the logical page unit in which the KV cache is mapped. The runtime places blocks in device memory in units of HW pages, and in general one block bundles one or more HW pages.
Engineering sample
The current device (engineering sample) has the following constraints.
- In the pool and in map/unmap, one block is one token (
block_size=1). Address reservation andvmm_init_blockstreat one request slot row as one block (block_size= the maximum tokens per request). HW pages are allocated per request, and one HW page holds many tokens. - A request's blocks only grow. While a request is live, mapping only adds blocks; they are released per request (slot) all at once.
- Prefix caching works as multi-turn continuation. When a new request begins with the full KV of an earlier
request (as the next turn of the same conversation does), it extends that KV. The vLLM examples are written for prefix caching turned off (
--no-enable-prefix-caching). See vLLM KV cache.
Every operator runs on the current device of the calling thread. If a runtime call fails, the operator raises a
RuntimeError of the form <operator>: <runtime entry point> failed: err=<code>.
Pool¶
| Function | Signature | Notes |
|---|---|---|
torch.ops.lpu.vmm_create_kv_pool |
(int num_blocks, int num_attn_layers, int num_kv_heads, int block_size, int head_dim, *, ScalarType dtype, Device device, str kind="dram_word_interleaved", int page_size=4096) -> Tensor |
Creates a pool that holds both KEY and VALUE. Shape [2, N, L, H, block_size, D]. One block holds one token, so block_size is 1 |
Allocates the pool memory with the runtime HaVmmAllocatePool, zero-fills it with zero_(), and returns
it. num_blocks is the token budget of the pool. page_size is
the alignment of the pool base address. Keep the default of 4096.
As with torch.ops.lpu.empty_with_kind, a fake kernel is registered for FakeTensorMode.
The pool tensor's lifetime is the physical memory's lifetime
The pool memory is freed when the last reference to the pool tensor is dropped. Keep the pool tensor alive while any mapped requests remain.
KV Cost¶
| Function | Signature | Notes |
|---|---|---|
torch.ops.lpu.vmm_kv_pool_bytes |
(int head_dim, ScalarType dtype, int num_layers, int num_kv_heads, int budget_bytes) -> int |
The largest number of bytes a pool can use within budget_bytes |
torch.ops.lpu.vmm_kv_pool_units |
(int head_dim, ScalarType dtype, int num_blocks) -> int |
The number of units a pool of num_blocks tokens holds |
torch.ops.lpu.vmm_kv_cost_table |
(int head_dim, ScalarType dtype, int max_blocks) -> Tensor |
A CPU int64 tensor of the units a request of n tokens takes, for n = 0 through max_blocks |
A scheduler uses these to check whether requests fit in a pool. The requests fit when the sum of their units is at most the pool's units. They can be called without opening a device.
Address Reservation¶
| Function | Signature | Notes |
|---|---|---|
torch.ops.lpu.vmm_reserve |
(int num_blocks, int num_attn_layers, int num_kv_heads, int block_size, int head_dim, str kind, ScalarType dtype) -> Tensor |
Creates a single KV address tensor. kind is "key" or "value". Shape [num_blocks, L, H, block_size, D] |
Calls the runtime HaVmmReserveBlocks without attaching physical memory. The operator is stateless and can be called
at any time; call it once for KEY and once for VALUE. The "value" tensor carries the VALUE bit. dtype accepts
float16, bfloat16, float8_e4m3fn, float8_e5m2, and float4_e2m1fn_x2. For float4_e2m1fn_x2, the last
dimension is reduced by the packing factor. A typical setup uses one block per request slot row, with block_size set
to the maximum number of tokens per request (the vLLM plugin passes 1024 rows and block_size = max_model_len; this is separate from vLLM's --block-size, which the plugin fixes at 1). A reserved address
has no physical memory behind it and does not need to be released.
vmm_reserve has no fake kernel
Calling it under FakeTensorMode or during torch.compile tracing issues a real device call. Call it outside
compiled regions, during engine initialization.
Initialization and Teardown¶
| Function | Signature | Notes |
|---|---|---|
torch.ops.lpu.vmm_init_blocks |
(int num_rows, int max_num_blocks_per_row, int num_attn_layers, int num_kv_heads, int block_size, int head_dim, int elem_size_bits, Tensor? pool=None) -> () |
Creates the VMM state for the device from the pool. Pass the pool tensor from vmm_create_kv_pool as pool |
torch.ops.lpu.vmm_deinit |
() -> () |
Releases the VMM state, but not the pool. Subsequent calls are no-ops |
vmm_init_blocks calls the runtime HaVmmInitBlocks. Call it after vmm_reserve and pool creation.
max_num_blocks_per_row × block_size is the maximum number of tokens per request and must equal the token depth of the
address reservation (the vLLM plugin passes 1 and max_model_len). num_rows is the number of request slots, similar to vLLM's max_num_seqs (request slots are currently limited to 1024). elem_size_bits is the logical bit
width of the dtype (16 for float16, 8 for float8_*, 4 for float4_e2m1fn_x2). pool must be a
vmm_create_kv_pool tensor on the current device, large enough to hold the KEY and VALUE pages of one request. A
repeated call cleans up the existing state first.
Mapping¶
| Function | Signature | Notes |
|---|---|---|
torch.ops.lpu.vmm_map_blocks |
(Tensor[] vmm_tensors, int row_index, int[] cached_block_ids, Tensor pool, int[] new_block_ids, int block_size=1) -> () |
Maps additional pages for one request. The lengths of cached_block_ids and new_block_ids give the token counts |
torch.ops.lpu.vmm_map_blocks_batch |
(Tensor[] vmm_tensors, int[] row_indices, int[] cached_block_ids, int[] cached_blocks_per_req, Tensor pool, int[] new_block_ids, int[] new_blocks_per_req, int block_size=1) -> () |
Batched variant, called on every step. Leave new_block_ids empty or make its length sum(new_blocks_per_req) |
torch.ops.lpu.vmm_unmap_blocks |
(Tensor[] vmm_tensors, int row_index, int[] block_ids, int block_size=1, int block_offset=-1) -> () |
Unmaps the pages of one request. The length of block_ids gives the token count |
torch.ops.lpu.vmm_unmap_blocks_batch |
(Tensor[] vmm_tensors, int[] row_indices, int[] block_ids, int[] blocks_per_req, int block_size=1, int[] block_offsets=[]) -> () |
Batched unmap. Does not take a pool argument |
The map/unmap operators call the runtime HaVmmMapBlocksBatchFromPool / HaVmmUnmapBlocksBatchFromPool.
| Argument | Meaning |
|---|---|
vmm_tensors |
[key, value] returned by vmm_reserve |
row_indices / row_index |
Request slot rows 0–1023. Required |
cached_blocks_per_req |
Number of tokens already mapped |
new_blocks_per_req |
Number of tokens to map |
new_block_ids, cached_block_ids, unmap block_ids |
Values are ignored. In the single-request ops (vmm_map_blocks, vmm_unmap_blocks) the length is the token count; in the batch ops, match the length to the sum of the counts or pass an empty list |
pool |
Pass the pool tensor from vmm_create_kv_pool. On the current engineering sample, pages are allocated from the pool registered with vmm_init_blocks |
block_size |
Must be 1 |
unmap blocks_per_req |
The request's full mapped token count; returned per request all at once |
unmap block_offsets / block_offset |
Unused |
A map call adds only the KEY and VALUE pages needed for the change in token count. Pass the exact "number of tokens already mapped" on every step.
Examples¶
torch.ops.lpu.vmm_create_kv_pool¶
Creates the pool that holds both KEY and VALUE. Keep the pool tensor alive while any mapped requests remain.
Example
1 2 3 4 5 6 7 | |
torch.ops.lpu.vmm_deinit¶
Unmaps all requests, releases the pool, then cleans up the VMM state for the device.
Example
1 2 3 4 5 | |
torch.ops.lpu.vmm_init_blocks¶
Creates the VMM state for the device from the pool, after address reservation and pool creation.
Example
1 2 3 4 5 | |
torch.ops.lpu.vmm_kv_cost_table¶
Gets the units for each request token count as a table, and checks whether requests fit in a pool.
Example
1 2 3 4 5 | |
torch.ops.lpu.vmm_kv_pool_bytes¶
Sizes a pool within a memory budget.
Example
1 2 3 4 5 | |
torch.ops.lpu.vmm_kv_pool_units¶
Gets the number of units a pool holds. See vmm_kv_cost_table for an example.
torch.ops.lpu.vmm_map_blocks¶
Maps additional tokens for one request. The argument lengths give the token counts.
Example
1 2 3 4 | |
torch.ops.lpu.vmm_map_blocks_batch¶
Performs all mappings for one scheduler step in a single call.
Example
1 2 3 4 | |
torch.ops.lpu.vmm_reserve¶
Creates separate KEY and VALUE address tensors. No physical memory is attached.
Example
1 2 3 4 5 6 7 8 | |
torch.ops.lpu.vmm_unmap_blocks¶
Unmaps the pages of one request. The length of block_ids gives the token count.
Example
1 2 3 4 | |
torch.ops.lpu.vmm_unmap_blocks_batch¶
Unmaps the pages of finished requests in a single call.
Example
1 2 3 4 | |
See Also¶
- Paged KV Cache: call order and the vLLM flow
- Runtime memory model §3.7:
HaVmm*entry points - Custom Operators
- Device API
- Environment Variables