Skip to content

VMM API

Paged KV-cache virtual memory API. The call order and values on this page apply to the current hardware revision (engineering sample). The runtime configures the HW pages. A KV address carries the request slot and the KEY/VALUE bit, and each request's pages grow with its block count and are released when the request finishes. Follow the call order in Virtual Memory & HW Pages in Memory Model. Every call is routed to the calling thread's current device; the device is not selected by the virtual address value.

Contents

API Description
HaVmmAllocatePool Allocates the physical KV pool
HaVmmDeinit Releases the VMM state of the current device
HaVmmFreePool Frees the segments of a pool
HaVmmInitBlocks Initializes the HW pages from the pool and block geometry
HaVmmMapBlocksBatchFromPool Grows per-request KV pages to the block count
HaVmmMapBlocksBatchFromPools Allocates KV pages for several VAs at once by block count (multi-pool form)
HaVmmReserveBlocks Returns the base address of one KV arena (KEY or VALUE)
HaVmmUnmapBlocksBatchFromPool Releases per-request KV pages

HaVmmAllocatePool

Allocates the physical KV pool.

Parameter Direction Default Description
arena_bytes in — Size of the pool to allocate, in bytes.
block_bytes in — Size of one block in bytes.
alignment in — Base address alignment. Must be a power of two or 0, and is used as given. 0 if no alignment is needed.
inout_pool in/out — HaVmmPool structure. In: segments points to the caller's buffer and num_segments to its capacity. Out: segments[0] is filled with the pool segment, and num_segments is the number of segments written to the buffer (always 1).
out_block_stride out — Bytes one block occupies (block_bytes).

Returns — HaSuccess on success, error code otherwise. Fails if capacity is insufficient.

Signature · Example
1
HaError HaVmmAllocatePool(uint64_t arena_bytes, int64_t block_bytes, uint32_t alignment, HaVmmPool *inout_pool, int64_t *out_block_stride);

Example

1
2
3
4
5
6
7
8
HaVmmBlockSegment seg[1];
HaVmmPool pool{seg, 1};             // in: capacity 1
int64_t block_stride = 0;
HaError err = HaVmmAllocatePool(pool_bytes, block_bytes, /*alignment=*/4096, &pool, &block_stride);
if (err != HaError::HaSuccess) {
  return err;
}
// out: one segment; seg[0].pa_start is the HW page base address for HaVmmInitBlocks()

Call it before HaVmmInitBlocks(); it can be called without initialization. A segment (HaVmmBlockSegment) holds the base address and size of the original allocation (raw_ptr / raw_size), which are used to free it. pa_start is raw_ptr rounded up to alignment.


HaVmmDeinit

Releases the VMM state of the current device. Must be called when no kernel is in flight.

Returns — HaSuccess on success, error code otherwise.

Signature · Example
1
HaError HaVmmDeinit(void);

Example

1
2
HaVmmFreePool(pool);   // every request unmapped first
HaVmmDeinit();         // no kernels in flight past this point

Releases only the host-side state and does not free the pool (use HaVmmFreePool()). A no-op when nothing is initialized.


HaVmmFreePool

Frees every segment of a pool created with HaVmmAllocatePool.

Parameter Direction Default Description
pool in — Pool to free (segments and count). A no-op if the count is 0.

Returns — HaSuccess on success, otherwise the free error of the first segment that failed. The remaining segments are freed even on failure.

Signature · Example
1
HaError HaVmmFreePool(HaVmmPool pool);

Example

1
2
3
4
5
// every request is unmapped by now
HaError err = HaVmmFreePool(pool);
if (err != HaError::HaSuccess) {
  return err;
}

Each segment is freed by its raw_ptr.

Do not free a pool with mapped pages

Unmap every request first. Freeing with pages still mapped returns the pool memory to the driver, and the next allocation may reuse that memory while the configured HW page base address still points to it. What is freed is raw_ptr, not the aligned base address pa_start. Do not free segments owned elsewhere (e.g. a torch pool tensor) here.


HaVmmInitBlocks

Initializes the current device's HW pages from the pool and block geometry, and writes the page size to out_page_size.

Parameter Direction Default Description
out_page_size out — Page size per channel in bytes.
num_rows in — Number of request slots, similar to vLLM's max_num_seqs; request slots are currently limited to 1024. Not a token length.
max_num_blocks_per_row in — Maximum number of blocks one request can have. Multiplied by block_size, it gives the maximum tokens per request.
num_attn_layers in — Number of attention layers.
num_kv_heads in — Number of KV heads.
block_size in — Tokens per block. Multiplied by max_num_blocks_per_row, it gives the maximum tokens per request. The current mapping unit is one token per block (block_size is 1 for map and unmap).
head_dim in — Number of dimensions per head.
elem_size_bits in — Size of one element in bits (mxfp4=4, fp8=8, bf16=16). numel*bits must be a whole number of bytes.
pool_base in HA_DEVICE_PTR_NULL Required. Base address of the physical KV pool (segments[0].pa_start returned by HaVmmAllocatePool()), 4 KiB aligned.
pool_bytes in 0 Required. Byte capacity of the physical KV pool. Must hold one request's KEY and VALUE pages at the maximum token count.
page_size_in in 0 Unused (the page size is fixed at 4096 per channel).
shared_ch_in in 0 Unused.

Returns — HaSuccess on success, error code otherwise.

Signature · Example
1
HaError HaVmmInitBlocks(int64_t *out_page_size, int64_t num_rows, int64_t max_num_blocks_per_row, int64_t num_attn_layers, int64_t num_kv_heads, int64_t block_size, int64_t head_dim, uint32_t elem_size_bits, HaDevicePtr pool_base = HA_DEVICE_PTR_NULL, uint64_t pool_bytes = 0, uint32_t page_size_in = 0, uint32_t shared_ch_in = 0);

Example

1
2
3
4
5
// after HaVmmAllocatePool: set the pool as the HW page base address
int64_t page_size = 0;
HaError err = HaVmmInitBlocks(&page_size, num_rows, /*max_num_blocks_per_row=*/1, num_attn_layers,
                              num_kv_heads, /*block_size=*/max_tokens, head_dim, elem_size_bits,
                              seg[0].pa_start, pool_bytes);   // page_size == 4096 (per channel)

pool_base comes from HaVmmAllocatePool(), so call this after it. Calling it again releases the existing state first.

pool_base and pool_bytes have default values in the signature, but omitting them is currently an error.


HaVmmMapBlocksBatchFromPool

Grows each request's KEY and VALUE pages to hold the new block count.

Parameter Direction Default Description
vas in — Array of KEY and VALUE arena base addresses ({KEY, VALUE}).
va_count in — Number of arenas (2: KEY, VALUE).
row_indices in — Per-request slot, 0..1023. Goes into bits 56:47 of the virtual address. Required.
cached_block_ids in — Not read. May be null.
cached_blocks_per_req in — Number of blocks already mapped per request.
pool in — Not read (the pool is registered by HaVmmInitBlocks()). segments may be null.
new_block_ids in — Not read, but must be non-null if there is anything to map.
new_blocks_per_req in — Number of blocks to newly map per request.
num_reqs in — Number of requests.
block_size in — Tokens per block. Must be 1.

Returns — HaSuccess on success, error code otherwise.

Signature · Example
1
HaError HaVmmMapBlocksBatchFromPool(const HaDevicePtr *vas, uint32_t va_count, const int64_t *row_indices, const int64_t *cached_block_ids, const int64_t *cached_blocks_per_req, HaVmmPool pool, const int64_t *new_block_ids, const int64_t *new_blocks_per_req, uint32_t num_reqs, int64_t block_size);

Example

1
2
3
4
5
6
7
8
// request slot 3 grows from 10 to 12 tokens
const HaDevicePtr vas[] = {k_va, v_va};
const int64_t row[] = {3};
const int64_t have[] = {10};
const int64_t add[] = {2};
const int64_t unused_ids[] = {0, 0};   // not read, but must be non-null
HaError err = HaVmmMapBlocksBatchFromPool(vas, 2, row, nullptr, have, pool, unused_ids, add,
                                          /*num_reqs=*/1, /*block_size=*/1);

Grows each request's KEY and VALUE pages to the sum of the blocks already mapped (cached_blocks_per_req) and the blocks newly mapped (new_blocks_per_req). Pass the exact number of already-mapped blocks (cached_blocks_per_req) at every step. Call it only after the previous step's kernels have finished.


HaVmmMapBlocksBatchFromPools

The explicit multi-pool form of the map surface.

Parameter Direction Default Description
vas in — va_count base VAs. Each VA must already carry the request slot (bits 56:47) and the KEY/VALUE bit (bit 57).
va_count in — Number of VAs. 0 is a no-op.
pools in — The pools. Not read. May be null.
num_pools in — Pool count. Not read.
block_ids in — sum(blocks_per_va) coordinates (HaVmmBlockId, (pool_idx, block_idx)), flat in VA order. The values are not read, but it must be non-null when there is anything to map.
blocks_per_va in — Per VA: blocks to map (>= 0).
block_size in — Tokens per block. Must be 1.

Returns — HaSuccess on success; HaMemoryInvalidArgument if vas or blocks_per_va is null, block_size is not 1, blocks_per_va has a negative entry, or block_ids is null with blocks to map; an error code otherwise.

Signature
1
HaError HaVmmMapBlocksBatchFromPools(const HaDevicePtr *vas, uint32_t va_count, const HaVmmPool *pools, uint32_t num_pools, const HaVmmBlockId *block_ids, const int64_t *blocks_per_va, int64_t block_size);

Pages are managed by block count, so explicit placement is not applied. The pools and block coordinates are ignored, each VA is treated as having no block mapped yet, and request pages are allocated to hold blocks_per_va[v] blocks. To grow a request that already has pages, use HaVmmMapBlocksBatchFromPool().


HaVmmReserveBlocks

Returns the base address of one KV arena (KEY or VALUE).

Parameter Direction Default Description
out_va out — Arena base address. The constant 1 << 58 (address translation enabled); no physical memory is allocated. For the VALUE arena the caller sets bit 57 to 1.
out_block_stride_bytes out — Virtual address distance between blocks, in bytes. Equal to block_bytes (blocks are laid out contiguously).
num_blocks in — Total number of blocks the arena spans.
numel_per_block in — Number of elements per block.
elem_size_bits in — Size of one element in bits (mxfp4=4, fp8=8, bf16=16). numel*bits must be a whole number of bytes.

Returns — HaSuccess on success, error code otherwise.

Signature · Example
1
HaError HaVmmReserveBlocks(HaDevicePtr *out_va, int64_t *out_block_stride_bytes, int64_t num_blocks, int64_t numel_per_block, uint32_t elem_size_bits);

Example

1
2
3
4
5
6
7
HaDevicePtr k_va = 0, v_va = 0;
int64_t k_stride = 0, v_stride = 0;
HaError err = HaVmmReserveBlocks(&k_va, &k_stride, num_blocks, numel_per_block, elem_size_bits);
if (err == HaError::HaSuccess) {
  err = HaVmmReserveBlocks(&v_va, &v_stride, num_blocks, numel_per_block, elem_size_bits);
}
v_va |= 1ull << 57;   // VALUE arena

Stateless, so it can be called at any time without initialization. The returned address is not a reservation, so there is nothing to free.


HaVmmUnmapBlocksBatchFromPool

Releases the KEY and VALUE pages of each finished request. The unmap counterpart of HaVmmMapBlocksBatchFromPool.

Parameter Direction Default Description
vas in — Array of KEY and VALUE arena base addresses ({KEY, VALUE}).
va_count in — Number of arenas (2: KEY, VALUE).
row_indices in — Per-request slot, 0..1023. Required.
block_ids in — Not read. May be null.
block_offsets in — Not read. May be null.
blocks_per_req in — The request's full mapped block count.
num_reqs in — Number of requests.
block_size in — Tokens per block. Must be 1.

Returns — HaSuccess on success, error code otherwise.

Signature · Example
1
HaError HaVmmUnmapBlocksBatchFromPool(const HaDevicePtr *vas, uint32_t va_count, const int64_t *row_indices, const int64_t *block_ids, const int64_t *block_offsets, const int64_t *blocks_per_req, uint32_t num_reqs, int64_t block_size);

Example

1
2
3
4
5
6
// request slot 3 finished with 12 tokens, after its kernels completed
const HaDevicePtr vas[] = {k_va, v_va};
const int64_t row[] = {3};
const int64_t done[] = {12};
HaError err = HaVmmUnmapBlocksBatchFromPool(vas, 2, row, nullptr, nullptr, done,
                                            /*num_reqs=*/1, /*block_size=*/1);

Releases the request's KEY and VALUE pages. The pool is not needed.