VMM API¶
Paged KV-cache virtual memory API. The call order and values on this page apply to the current hardware revision (engineering sample). The runtime configures the HW pages. A KV address carries the request slot and the KEY/VALUE bit, and each request's pages grow with its block count and are released when the request finishes. Follow the call order in Virtual Memory & HW Pages in Memory Model. Every call is routed to the calling thread's current device; the device is not selected by the virtual address value.
Contents¶
| API | Description |
|---|---|
HaVmmAllocatePool |
Allocates the physical KV pool |
HaVmmDeinit |
Releases the VMM state of the current device |
HaVmmFreePool |
Frees the segments of a pool |
HaVmmInitBlocks |
Initializes the HW pages from the pool and block geometry |
HaVmmMapBlocksBatchFromPool |
Grows per-request KV pages to the block count |
HaVmmMapBlocksBatchFromPools |
Allocates KV pages for several VAs at once by block count (multi-pool form) |
HaVmmReserveBlocks |
Returns the base address of one KV arena (KEY or VALUE) |
HaVmmUnmapBlocksBatchFromPool |
Releases per-request KV pages |
HaVmmAllocatePool¶
Allocates the physical KV pool.
| Parameter | Direction | Default | Description |
|---|---|---|---|
arena_bytes |
in | — | Size of the pool to allocate, in bytes. |
block_bytes |
in | — | Size of one block in bytes. |
alignment |
in | — | Base address alignment. Must be a power of two or 0, and is used as given. 0 if no alignment is needed. |
inout_pool |
in/out | — | HaVmmPool structure. In: segments points to the caller's buffer and num_segments to its capacity. Out: segments[0] is filled with the pool segment, and num_segments is the number of segments written to the buffer (always 1). |
out_block_stride |
out | — | Bytes one block occupies (block_bytes). |
Returns — HaSuccess on success, error code otherwise. Fails if capacity is insufficient.
Signature · Example
1 | |
Example
1 2 3 4 5 6 7 8 | |
Call it before HaVmmInitBlocks(); it can be called without initialization. A segment (HaVmmBlockSegment) holds the base address and size of the original allocation (raw_ptr / raw_size), which are used to free it. pa_start is raw_ptr rounded up to alignment.
HaVmmDeinit¶
Releases the VMM state of the current device. Must be called when no kernel is in flight.
Returns — HaSuccess on success, error code otherwise.
Signature · Example
1 | |
Example
1 2 | |
Releases only the host-side state and does not free the pool (use HaVmmFreePool()). A no-op when nothing is initialized.
HaVmmFreePool¶
Frees every segment of a pool created with HaVmmAllocatePool.
| Parameter | Direction | Default | Description |
|---|---|---|---|
pool |
in | — | Pool to free (segments and count). A no-op if the count is 0. |
Returns — HaSuccess on success, otherwise the free error of the first segment that failed. The remaining segments are freed even on failure.
Signature · Example
1 | |
Example
1 2 3 4 5 | |
Each segment is freed by its raw_ptr.
Do not free a pool with mapped pages
Unmap every request first. Freeing with pages still mapped returns the pool memory to the driver, and the next allocation may reuse that memory while the configured HW page base address still points to it. What is freed is raw_ptr, not the aligned base address pa_start. Do not free segments owned elsewhere (e.g. a torch pool tensor) here.
HaVmmInitBlocks¶
Initializes the current device's HW pages from the pool and block geometry, and writes the page size to out_page_size.
| Parameter | Direction | Default | Description |
|---|---|---|---|
out_page_size |
out | — | Page size per channel in bytes. |
num_rows |
in | — | Number of request slots, similar to vLLM's max_num_seqs; request slots are currently limited to 1024. Not a token length. |
max_num_blocks_per_row |
in | — | Maximum number of blocks one request can have. Multiplied by block_size, it gives the maximum tokens per request. |
num_attn_layers |
in | — | Number of attention layers. |
num_kv_heads |
in | — | Number of KV heads. |
block_size |
in | — | Tokens per block. Multiplied by max_num_blocks_per_row, it gives the maximum tokens per request. The current mapping unit is one token per block (block_size is 1 for map and unmap). |
head_dim |
in | — | Number of dimensions per head. |
elem_size_bits |
in | — | Size of one element in bits (mxfp4=4, fp8=8, bf16=16). numel*bits must be a whole number of bytes. |
pool_base |
in | HA_DEVICE_PTR_NULL |
Required. Base address of the physical KV pool (segments[0].pa_start returned by HaVmmAllocatePool()), 4 KiB aligned. |
pool_bytes |
in | 0 |
Required. Byte capacity of the physical KV pool. Must hold one request's KEY and VALUE pages at the maximum token count. |
page_size_in |
in | 0 |
Unused (the page size is fixed at 4096 per channel). |
shared_ch_in |
in | 0 |
Unused. |
Returns — HaSuccess on success, error code otherwise.
Signature · Example
1 | |
Example
1 2 3 4 5 | |
pool_base comes from HaVmmAllocatePool(), so call this after it. Calling it again releases the existing state first.
pool_base and pool_bytes have default values in the signature, but omitting them is currently an error.
HaVmmMapBlocksBatchFromPool¶
Grows each request's KEY and VALUE pages to hold the new block count.
| Parameter | Direction | Default | Description |
|---|---|---|---|
vas |
in | — | Array of KEY and VALUE arena base addresses ({KEY, VALUE}). |
va_count |
in | — | Number of arenas (2: KEY, VALUE). |
row_indices |
in | — | Per-request slot, 0..1023. Goes into bits 56:47 of the virtual address. Required. |
cached_block_ids |
in | — | Not read. May be null. |
cached_blocks_per_req |
in | — | Number of blocks already mapped per request. |
pool |
in | — | Not read (the pool is registered by HaVmmInitBlocks()). segments may be null. |
new_block_ids |
in | — | Not read, but must be non-null if there is anything to map. |
new_blocks_per_req |
in | — | Number of blocks to newly map per request. |
num_reqs |
in | — | Number of requests. |
block_size |
in | — | Tokens per block. Must be 1. |
Returns — HaSuccess on success, error code otherwise.
Signature · Example
1 | |
Example
1 2 3 4 5 6 7 8 | |
Grows each request's KEY and VALUE pages to the sum of the blocks already mapped (cached_blocks_per_req) and the blocks newly mapped (new_blocks_per_req). Pass the exact number of already-mapped blocks (cached_blocks_per_req) at every step. Call it only after the previous step's kernels have finished.
HaVmmMapBlocksBatchFromPools¶
The explicit multi-pool form of the map surface.
| Parameter | Direction | Default | Description |
|---|---|---|---|
vas |
in | — | va_count base VAs. Each VA must already carry the request slot (bits 56:47) and the KEY/VALUE bit (bit 57). |
va_count |
in | — | Number of VAs. 0 is a no-op. |
pools |
in | — | The pools. Not read. May be null. |
num_pools |
in | — | Pool count. Not read. |
block_ids |
in | — | sum(blocks_per_va) coordinates (HaVmmBlockId, (pool_idx, block_idx)), flat in VA order. The values are not read, but it must be non-null when there is anything to map. |
blocks_per_va |
in | — | Per VA: blocks to map (>= 0). |
block_size |
in | — | Tokens per block. Must be 1. |
Returns — HaSuccess on success; HaMemoryInvalidArgument if vas or blocks_per_va is null, block_size is not 1, blocks_per_va has a negative entry, or block_ids is null with blocks to map; an error code otherwise.
Signature
1 | |
Pages are managed by block count, so explicit placement is not applied. The pools and block coordinates are ignored, each VA is treated as having no block mapped yet, and request pages are allocated to hold blocks_per_va[v] blocks. To grow a request that already has pages, use HaVmmMapBlocksBatchFromPool().
HaVmmReserveBlocks¶
Returns the base address of one KV arena (KEY or VALUE).
| Parameter | Direction | Default | Description |
|---|---|---|---|
out_va |
out | — | Arena base address. The constant 1 << 58 (address translation enabled); no physical memory is allocated. For the VALUE arena the caller sets bit 57 to 1. |
out_block_stride_bytes |
out | — | Virtual address distance between blocks, in bytes. Equal to block_bytes (blocks are laid out contiguously). |
num_blocks |
in | — | Total number of blocks the arena spans. |
numel_per_block |
in | — | Number of elements per block. |
elem_size_bits |
in | — | Size of one element in bits (mxfp4=4, fp8=8, bf16=16). numel*bits must be a whole number of bytes. |
Returns — HaSuccess on success, error code otherwise.
Signature · Example
1 | |
Example
1 2 3 4 5 6 7 | |
Stateless, so it can be called at any time without initialization. The returned address is not a reservation, so there is nothing to free.
HaVmmUnmapBlocksBatchFromPool¶
Releases the KEY and VALUE pages of each finished request. The unmap counterpart of HaVmmMapBlocksBatchFromPool.
| Parameter | Direction | Default | Description |
|---|---|---|---|
vas |
in | — | Array of KEY and VALUE arena base addresses ({KEY, VALUE}). |
va_count |
in | — | Number of arenas (2: KEY, VALUE). |
row_indices |
in | — | Per-request slot, 0..1023. Required. |
block_ids |
in | — | Not read. May be null. |
block_offsets |
in | — | Not read. May be null. |
blocks_per_req |
in | — | The request's full mapped block count. |
num_reqs |
in | — | Number of requests. |
block_size |
in | — | Tokens per block. Must be 1. |
Returns — HaSuccess on success, error code otherwise.
Signature · Example
1 | |
Example
1 2 3 4 5 6 | |
Releases the request's KEY and VALUE pages. The pool is not needed.