Skip to content

VMM API

The torch.ops.lpu.vmm_* operators manage the paged KV cache. They create the physical KV pool, reserve KEY and VALUE addresses, and map and unmap per-request pages on every scheduler step. These operators manage the HW pages through the runtime VMM API. If you are new to these operators, read the Paged KV Cache chapter first.

A block is the logical page unit in which the KV cache is mapped. The runtime places blocks in device memory in units of HW pages, and in general one block bundles one or more HW pages.

Engineering sample

The current device (engineering sample) has the following constraints.

  • In the pool and in map/unmap, one block is one token (block_size=1). Address reservation and vmm_init_blocks treat one request slot row as one block (block_size = the maximum tokens per request). HW pages are allocated per request, and one HW page holds many tokens.
  • A request's blocks only grow. While a request is live, mapping only adds blocks; they are released per request (slot) all at once.
  • Prefix caching works as multi-turn continuation. When a new request begins with the full KV of an earlier request (as the next turn of the same conversation does), it extends that KV. The vLLM examples are written for prefix caching turned off (--no-enable-prefix-caching). See vLLM KV cache.

Every operator runs on the current device of the calling thread. If a runtime call fails, the operator raises a RuntimeError of the form <operator>: <runtime entry point> failed: err=<code>.

Pool

Function Signature Notes
torch.ops.lpu.vmm_create_kv_pool (int num_blocks, int num_attn_layers, int num_kv_heads, int block_size, int head_dim, *, ScalarType dtype, Device device, str kind="dram_word_interleaved", int page_size=4096) -> Tensor Creates a pool that holds both KEY and VALUE. Shape [2, N, L, H, block_size, D]. One block holds one token, so block_size is 1

Allocates the pool memory with the runtime HaVmmAllocatePool, zero-fills it with zero_(), and returns it. num_blocks is the token budget of the pool. page_size is the alignment of the pool base address. Keep the default of 4096. As with torch.ops.lpu.empty_with_kind, a fake kernel is registered for FakeTensorMode.

The pool tensor's lifetime is the physical memory's lifetime

The pool memory is freed when the last reference to the pool tensor is dropped. Keep the pool tensor alive while any mapped requests remain.

KV Cost

Function Signature Notes
torch.ops.lpu.vmm_kv_pool_bytes (int head_dim, ScalarType dtype, int num_layers, int num_kv_heads, int budget_bytes) -> int The largest number of bytes a pool can use within budget_bytes
torch.ops.lpu.vmm_kv_pool_units (int head_dim, ScalarType dtype, int num_blocks) -> int The number of units a pool of num_blocks tokens holds
torch.ops.lpu.vmm_kv_cost_table (int head_dim, ScalarType dtype, int max_blocks) -> Tensor A CPU int64 tensor of the units a request of n tokens takes, for n = 0 through max_blocks

A scheduler uses these to check whether requests fit in a pool. The requests fit when the sum of their units is at most the pool's units. They can be called without opening a device.

Address Reservation

Function Signature Notes
torch.ops.lpu.vmm_reserve (int num_blocks, int num_attn_layers, int num_kv_heads, int block_size, int head_dim, str kind, ScalarType dtype) -> Tensor Creates a single KV address tensor. kind is "key" or "value". Shape [num_blocks, L, H, block_size, D]

Calls the runtime HaVmmReserveBlocks without attaching physical memory. The operator is stateless and can be called at any time; call it once for KEY and once for VALUE. The "value" tensor carries the VALUE bit. dtype accepts float16, bfloat16, float8_e4m3fn, float8_e5m2, and float4_e2m1fn_x2. For float4_e2m1fn_x2, the last dimension is reduced by the packing factor. A typical setup uses one block per request slot row, with block_size set to the maximum number of tokens per request (the vLLM plugin passes 1024 rows and block_size = max_model_len; this is separate from vLLM's --block-size, which the plugin fixes at 1). A reserved address has no physical memory behind it and does not need to be released.

vmm_reserve has no fake kernel

Calling it under FakeTensorMode or during torch.compile tracing issues a real device call. Call it outside compiled regions, during engine initialization.

Initialization and Teardown

Function Signature Notes
torch.ops.lpu.vmm_init_blocks (int num_rows, int max_num_blocks_per_row, int num_attn_layers, int num_kv_heads, int block_size, int head_dim, int elem_size_bits, Tensor? pool=None) -> () Creates the VMM state for the device from the pool. Pass the pool tensor from vmm_create_kv_pool as pool
torch.ops.lpu.vmm_deinit () -> () Releases the VMM state, but not the pool. Subsequent calls are no-ops

vmm_init_blocks calls the runtime HaVmmInitBlocks. Call it after vmm_reserve and pool creation. max_num_blocks_per_row × block_size is the maximum number of tokens per request and must equal the token depth of the address reservation (the vLLM plugin passes 1 and max_model_len). num_rows is the number of request slots, similar to vLLM's max_num_seqs (request slots are currently limited to 1024). elem_size_bits is the logical bit width of the dtype (16 for float16, 8 for float8_*, 4 for float4_e2m1fn_x2). pool must be a vmm_create_kv_pool tensor on the current device, large enough to hold the KEY and VALUE pages of one request. A repeated call cleans up the existing state first.

Mapping

Function Signature Notes
torch.ops.lpu.vmm_map_blocks (Tensor[] vmm_tensors, int row_index, int[] cached_block_ids, Tensor pool, int[] new_block_ids, int block_size=1) -> () Maps additional pages for one request. The lengths of cached_block_ids and new_block_ids give the token counts
torch.ops.lpu.vmm_map_blocks_batch (Tensor[] vmm_tensors, int[] row_indices, int[] cached_block_ids, int[] cached_blocks_per_req, Tensor pool, int[] new_block_ids, int[] new_blocks_per_req, int block_size=1) -> () Batched variant, called on every step. Leave new_block_ids empty or make its length sum(new_blocks_per_req)
torch.ops.lpu.vmm_unmap_blocks (Tensor[] vmm_tensors, int row_index, int[] block_ids, int block_size=1, int block_offset=-1) -> () Unmaps the pages of one request. The length of block_ids gives the token count
torch.ops.lpu.vmm_unmap_blocks_batch (Tensor[] vmm_tensors, int[] row_indices, int[] block_ids, int[] blocks_per_req, int block_size=1, int[] block_offsets=[]) -> () Batched unmap. Does not take a pool argument

The map/unmap operators call the runtime HaVmmMapBlocksBatchFromPool / HaVmmUnmapBlocksBatchFromPool.

Argument Meaning
vmm_tensors [key, value] returned by vmm_reserve
row_indices / row_index Request slot rows 0–1023. Required
cached_blocks_per_req Number of tokens already mapped
new_blocks_per_req Number of tokens to map
new_block_ids, cached_block_ids, unmap block_ids Values are ignored. In the single-request ops (vmm_map_blocks, vmm_unmap_blocks) the length is the token count; in the batch ops, match the length to the sum of the counts or pass an empty list
pool Pass the pool tensor from vmm_create_kv_pool. On the current engineering sample, pages are allocated from the pool registered with vmm_init_blocks
block_size Must be 1
unmap blocks_per_req The request's full mapped token count; returned per request all at once
unmap block_offsets / block_offset Unused

A map call adds only the KEY and VALUE pages needed for the change in token count. Pass the exact "number of tokens already mapped" on every step.


Examples

torch.ops.lpu.vmm_create_kv_pool

Creates the pool that holds both KEY and VALUE. Keep the pool tensor alive while any mapped requests remain.

Example
1
2
3
4
5
6
7
import torch

pool = torch.ops.lpu.vmm_create_kv_pool(
    num_tokens, num_attn_layers, num_kv_heads, 1, head_dim,
    dtype=torch.bfloat16, device="lpu:0",
)
assert pool.shape == (2, num_tokens, num_attn_layers, num_kv_heads, 1, head_dim)

torch.ops.lpu.vmm_deinit

Unmaps all requests, releases the pool, then cleans up the VMM state for the device.

Example
1
2
3
4
5
import torch

torch.ops.lpu.vmm_unmap_blocks_batch([k, v], live_slots, [], live_tokens)
del pool
torch.ops.lpu.vmm_deinit()

torch.ops.lpu.vmm_init_blocks

Creates the VMM state for the device from the pool, after address reservation and pool creation.

Example
1
2
3
4
5
import torch

torch.ops.lpu.vmm_init_blocks(
    max_num_seqs, 1, num_attn_layers, num_kv_heads, max_model_len, head_dim, 16, pool
)

torch.ops.lpu.vmm_kv_cost_table

Gets the units for each request token count as a table, and checks whether requests fit in a pool.

Example
1
2
3
4
5
import torch

costs = torch.ops.lpu.vmm_kv_cost_table(head_dim, torch.bfloat16, max_model_len)
capacity = torch.ops.lpu.vmm_kv_pool_units(head_dim, torch.bfloat16, num_tokens)
fits = sum(costs[n].item() for n in token_counts) <= capacity

torch.ops.lpu.vmm_kv_pool_bytes

Sizes a pool within a memory budget.

Example
1
2
3
4
5
import torch

pool_bytes = torch.ops.lpu.vmm_kv_pool_bytes(
    head_dim, torch.bfloat16, num_attn_layers, num_kv_heads, budget_bytes,
)

torch.ops.lpu.vmm_kv_pool_units

Gets the number of units a pool holds. See vmm_kv_cost_table for an example.


torch.ops.lpu.vmm_map_blocks

Maps additional tokens for one request. The argument lengths give the token counts.

Example
1
2
3
4
import torch

# Slot 3 grows from 10 tokens to 12
torch.ops.lpu.vmm_map_blocks([k, v], 3, [0] * 10, pool, [0, 0])

torch.ops.lpu.vmm_map_blocks_batch

Performs all mappings for one scheduler step in a single call.

Example
1
2
3
4
import torch

# Slot 3 goes from 10 to 12 tokens; slot 7 is a new request with 5 tokens
torch.ops.lpu.vmm_map_blocks_batch([k, v], [3, 7], [], [10, 0], pool, [0] * 7, [2, 5])

torch.ops.lpu.vmm_reserve

Creates separate KEY and VALUE address tensors. No physical memory is attached.

Example
1
2
3
4
5
6
7
8
import torch

k = torch.ops.lpu.vmm_reserve(
    1024, num_attn_layers, num_kv_heads, max_model_len, head_dim, "key", torch.bfloat16
)
v = torch.ops.lpu.vmm_reserve(
    1024, num_attn_layers, num_kv_heads, max_model_len, head_dim, "value", torch.bfloat16
)

torch.ops.lpu.vmm_unmap_blocks

Unmaps the pages of one request. The length of block_ids gives the token count.

Example
1
2
3
4
import torch

# Return slot 3's 12 tokens
torch.ops.lpu.vmm_unmap_blocks([k, v], 3, [0] * 12)

torch.ops.lpu.vmm_unmap_blocks_batch

Unmaps the pages of finished requests in a single call.

Example
1
2
3
4
import torch

# Return slot 5 (300 tokens) and slot 9 (40 tokens)
torch.ops.lpu.vmm_unmap_blocks_batch([k, v], [5, 9], [], [300, 40])

See Also