3. Memory Model¶
Device memory is subject to architectural constraints that do not apply to host (CPU) virtual memory. This chapter covers the three key constraints the runtime imposes when you work with device memory.
- No dereferencing — Even if you obtain a device pointer address, you cannot dereference it directly from a host thread.
- Freed memory stays in the pool — Freed memory may stay in the runtime pool for reuse instead of being returned right
away. Check actual usage with the memory statistics, and return the pool's empty segments to the device with
device_mem_trim(). - Per-kernel layout dependence — The same physical byte sequence must be aligned and laid out differently depending on the memory access pattern and layout specification of the target kernel.
To reduce the cost of device memory allocation, the runtime obtains memory in segments, keeps them in a pool, and serves allocations from that pool. Freed memory goes back to the pool and is reused, and empty segments the pool keeps return to the device's free memory only when the trim API is called.
Check the actual allocation and the amount held by the pool through the device memory statistics. For details, see Memory Statistics & Trim. Host buffers can be allocated from a host memory manager that works the same way.
Contents¶
| Section | Objective |
|---|---|
| 3.1 Device & Host Pointers | Difference between device and host addresses, and the dereferencing constraint |
| 3.2 Allocation & Deallocation | Device memory (pool caching, statistics and trim), host memory |
| 3.3 Tensor Layout | Memory strides and alignment according to the hardware architecture specification |
| 3.4 Transfer | Synchronous and asynchronous transfer models, multi-dimensional tensor transfer |
| 3.5 Direct File Load | Loading directly from a file descriptor into device memory |
| 3.6 Device Memory Dump | Verifying layout placement with a device memory dump |
| 3.7 Virtual Memory & HW Pages | Virtual memory for the KV cache: call order and rules |
3.1 Device & Host Pointers¶
The runtime API deals with two kinds of addresses.
- Device pointer: An address in device memory, represented by the 64-bit integer type
HaDevicePtr(Python:int). It is whatmem_alloc()(C++:HaMemAlloc()) returns. It is not a host pointer, so dereferencing it on the host is undefined behavior. To read or write its contents, copy them with a transfer function. - Host pointer: An address in host memory, which the host can read and write directly. Both ordinarily allocated
memory and memory allocated with
host_mem_alloc()(C++:HaHostMemAlloc()) can be used for transfers.
Address Arguments of Transfer Functions
The data transfer function mem_cpy() (C++: HaMemCpy()) takes HaDevicePtr for both
source and destination in C++, so host pointers are also cast to this type (Python passes a buffer object such as
bytes or an int address). Because the argument type does not show whether an address is on the host or the
device, the transfer direction is given by the direction argument.
Multi-dimensional transfers (HaMemCpyNd) take the direction from the scope of the layouts.
3.2 Allocation & Deallocation¶
Device memory is allocated from the pool of the current device, and host memory from a process-wide manager.
Device Memory¶
Allocation¶
1 2 3 4 5 | |
1 2 3 4 5 6 | |
The memory allocation APIs are mem_alloc() for linear buffers and mem_alloc_3d()
(C++: HaMemAlloc3D()) for multi-dimensional tensors.
Both let you choose, through the is_interleaved argument, an allocation strategy for the
multi-channel DRAM architecture of the HyperAccel LPU.
- Contiguous Allocation — Places data in a single channel, optimized for sequential access. The same amount of space is nevertheless occupied in every other channel, so the device memory it occupies is the requested size multiplied by the number of channels.
- Interleaved Allocation — Distributes data across multiple channels so that the kernel can make full use of the aggregate bandwidth of all channels.
Which strategy to use depends on how the kernel that uses the buffer accesses its operands. Rather than making every allocation interleaved, set it explicitly to match the memory layout the kernel expects.
Note: HaShape fields
The fields of HaShape, used for 3D tensor allocation, mean the
following.
w(Width) — The width along the contiguous direction.h(Height) — The height, split by channel and block.c(Channel) — Despite its name, it must be1in the current version; larger values are not supported.
Because padding is inserted for interleaved mode and chunk alignment, the actual physical
size can exceed w * h * c bytes.
Deallocation¶
1 | |
1 | |
Freed memory: Freed memory goes back to the pool and is reused automatically by later allocations. When a free
leaves a segment entirely empty, the runtime returns that segment right away if other allocations remain and the free
memory the pool holds exceeds half of the allocated amount. Otherwise, in particular when every allocation has been
freed, it keeps the segment for later allocations. When you need to free up the segments it keeps, check how much
additional capacity you can reclaim with releasable_bytes in the memory statistics, then call
device_mem_trim() (C++: HaDeviceMemTrim()). For the statistics fields and how to return
memory, see Memory Statistics & Trim.
Memory reuse through the runtime caching mechanism (C++ API. Python: mem_alloc(), mem_free(), device_mem_trim())
Performance guide: Avoid dynamic allocation in production inference serving loops to prevent overhead. Pre-allocate the large buffers the workload needs during the initialization phase and reuse them inside the serving loop.
If you must repeatedly allocate buffers larger than 1 MiB at run time, you can set
HA_HUGE_POOL_THRESHOLD_SIZE_MB. Allocations up to that size are
then carved out of regions of that size, which reduces how often device memory is obtained. Allocations larger than
that size are obtained separately as before. A region no longer in use can be returned with device_mem_trim(), but
not while any allocation in it remains.
Memory Statistics & Trim¶
Because of caching, the usage the system sees differs from the actual allocated amount, so check the memory state of the current device through statistics.
device_mem_get_stats() (C++: HaDeviceMemGetStats()) returns the runtime memory counters (bytes allocated, its peak, and so on) together with the free DRAM of the whole device (free_bytes, reflecting usage by
other processes). device_mem_reset_peak_stats() (C++: HaDeviceMemResetPeakStats()) restarts the peak values from the current values.
To return the empty segments held by the runtime and increase the device's free memory, call device_mem_trim() (C++: HaDeviceMemTrim()).
Trim keeps segments that still hold allocations. Memory freed with mem_free_async() (C++: HaMemFreeAsync()) can be trimmed only
after its stream is synchronized, so synchronize the stream before calling trim.
1 2 3 4 | |
1 2 3 4 5 | |
The fields most useful for monitoring usage are listed below. Byte values are summed over all DRAM channels.
| Field | Meaning |
|---|---|
total_bytes / free_bytes |
DRAM capacity of the device and its current free DRAM. Reflects usage by other processes |
allocated_bytes |
Size of the blocks the program has allocated and not yet freed. Rounded up to the pool's allocation unit, so it can be larger than the sum of requested sizes |
reserved_bytes |
Size the runtime pools have obtained and hold. Includes freed memory, so it is at least allocated_bytes |
releasable_bytes |
Size of the segments with no allocated block. The additional capacity device_mem_trim() can reclaim |
peak_* |
Highest value since the peaks were last reset |
To measure the peak usage of a section, reset the peaks before it starts and read peak_allocated_bytes or
peak_reserved_bytes after it ends. Setting HA_DEVICE_MEM_STATS_PRINT=1 also prints the statistics to standard error
on every query. For all field definitions, see HaDeviceMemStats.
Host Memory¶
Separately from device memory, the runtime provides a process-wide host memory manager. You can allocate the host buffers used for transfers from this manager, and it can be used without setting a current device.
Allocation¶
1 | |
1 2 | |
host_mem_alloc() (C++: HaHostMemAlloc()) allocates 64-byte aligned host memory whose
contents are not initialized. The upper limit of host memory the process obtains from the OS is set with
HA_HOST_MEM_MAX_BYTES. An allocation over the limit fails
with HaHostMemLimitExceeded (Python: RuntimeError) if returning empty regions to the OS still does not make room.
Deallocation¶
1 2 3 | |
1 2 3 | |
Freed memory: Memory freed with host_mem_free() (C++: HaHostMemFree()) goes back to
the pool and is reused automatically by later allocations. Do not free a buffer handed to an asynchronous copy until
the copy completes. When you need to free up memory, check the statistics, then call host_mem_trim(). For the
statistics fields and how to return memory, see Memory Statistics & Trim.
Memory Statistics & Trim¶
host_mem_get_stats() (C++: HaHostMemGetStats()) returns the host memory statistics of
the whole process. host_mem_reset_peak_stats() (C++: HaHostMemResetPeakStats())
restarts the peak values from the current values. host_mem_trim() (C++: HaHostMemTrim())
returns the freed memory held in the pool to the OS. Regions that still hold allocations are kept.
1 2 3 | |
1 2 3 4 | |
| Field | Meaning |
|---|---|
allocated_bytes |
Size of the memory allocated and not yet freed. Counted as requested |
reserved_bytes |
Size obtained from the OS and held. Includes freed memory, so it is at least allocated_bytes |
peak_* |
Highest value since the peaks were last reset |
alloc_count / free_count |
Number of allocations and frees so far |
Setting HA_HOST_MEM_STATS_PRINT=1 also prints the statistics to standard error each time they are queried. For all field
definitions, see HaHostMemStats.
3.3 Tensor Layout¶
A tensor that a kernel on the LPU reads must be stored in device memory in the layout the kernel requires.
The runtime queries hardware information of the currently selected device, such as the channel count, DRAM word size, and channel offset, and computes the optimal tensor layout.
You must, however, explicitly specify the tensor kind (HaTensorKind) and the logical dimensions (dims)
(see the table below).
The following example selects a device, computes a layout,
and inspects the returned shape and strides.
1 2 3 4 5 6 7 8 9 10 | |
1 2 3 4 5 6 7 | |
The returned layout.shape and layout.strides describe the data as it is actually stored in device memory, so their
rank or sizes can differ from the dims you passed. For example, SingleChannel in the example above prepends a
channel dimension, turning [34, 128] into (number of channels, 34, 128). For
multi-dimensional transfers (mem_cpy_nd()), pass the returned layout as is rather
than the input dims.
Per-channel data placement of SingleChannel and DramWordInterleaved
HaTensorKind |
Required rank | Meaning of and constraints on dims |
|---|---|---|
DramWordInterleaved |
>= 1 |
Depends only on the total element count; pass the contiguous logical shape as is |
SinglePassInterleaved |
>= 1 |
Same as above, with stricter per-channel alignment constraints |
SingleChannel |
>= 1 |
All data is placed contiguously in channel 0 |
RowInterleaved |
>= 2 |
The innermost dimension is the row width W, and the product of the remaining outer dimensions is the total row count |
MaxRowsInterleaved |
>= 2 |
Similar to RowInterleaved, with different chunk alignment rules |
MPUTensor |
>= 2 |
The innermost dimension is head_dim, the one before it is block_size, and the product of the remaining outer dimensions is the total block count |
KvCacheKey / KvCacheValue |
Exactly 5 | Must follow the structure [num_blocks, num_layers, num_kv_heads, block_size, head_dim] |
Besides the per-tensor shape constraints above, you must follow the hardware constraints and execution rules below when you define a layout and call the API.
- Interleaved kind alignment: For every interleaved kind, the per-channel chunk
size must be an exact integer multiple of the DRAM word size (for example, 64 bytes on the Bertha Cloud
architecture and 32 bytes on the Bertha Edge architecture). In particular,
SinglePassInterleavedhas the stricter constraint that the total bit count must be a multiple ofnum_channels * dram_word_size * 8. A layout definition that violates these conditions is rejected immediately, without silent padding. - MPUTensor and KV cache constraints:
head_dimmust be exactly divisible bychannels_per_token * elements_per_word; otherwiseHaMemoryInvalidArgumentis returned. - API call order: Platform constants are derived from the hardware specification of the target
device, so
set_device()(C++:HaSetDevice()) must be called beforebuild_device_tensor_layout()(C++:HaBuildDeviceTensorLayout()). OtherwiseHaDeviceInvalidArgumentorHaDeviceNotInitializedis returned.
1 2 3 | |
1 2 3 4 | |
You can also call it with dummy dimensions, without a real tensor, to query only the hardware constants of the device (see the example below).
1 2 3 4 5 | |
1 2 3 4 5 6 7 | |
build_device_tensor_layout() (C++: HaBuildDeviceTensorLayout()) also takes an optional strides argument
in addition to dims.
Stride parameter compatibility:
In the current version (0.0.6), the strides parameter accepts only standard contiguous logical strides, and
any other custom stride pattern returns HaNotSupported. The parameter is exposed in advance to keep the API
compatible with future runtime updates;
for a standard row-major contiguous layout, pass nullptr.
3.4 Transfer¶
Synchronous Transfer¶
1 2 3 4 | |
1 2 3 4 | |
The example above shows how to transfer data synchronously between host and device, or between
device buffers, with the copy direction given by HaMemCpyDirection.
Characteristics:
- Synchronous execution (Sync):
mem_cpy()does not return control to the calling thread until the requested transfer has fully finished at the hardware level. Once the function returns, it is safe to reuse the source memory or read the destination data. - Parameters: Takes the destination pointer (
dst), the source pointer (src), the number of bytes to transfer (size), and the copy direction (HaMemCpyDirection).
Transfer directions and inter-device constraints:
- Basic transfers (H2D / D2H): Uploads and downloads between host and device are supported.
- Device-to-device copy (D2D): Copies within the same device
(
DeviceToDevice) are supported natively by the runtime and hardware. - Inter-device transfer limits: Transfers between different devices
depend on whether the devices support peer transfer. Between devices that do not
support it, a cross-device transfer does not fall back to a slow software
path; it immediately returns
HaMemCpyNotSupported. For topology and transfer constraints in multi-device setups, see Multi-Device Programming for details.
Asynchronous Transfer¶
1 2 | |
1 2 | |
The example above shows the basic pattern: enqueue a copy command asynchronously on the target stream, and use the stream synchronization API to guarantee that the transfer has completed. This asynchronous transfer interface lets you overlap hardware compute and data transfer to maximize performance.
Characteristics:
- Asynchronous execution (Async): The copy command is enqueued on the given stream and the call returns immediately. At that point the hardware may not have started the transfer yet.
- Buffer lifetime and synchronization: Do not modify the host-side source buffer or access the
destination buffer until you have confirmed completion with
stream_synchronize()or an event. Accessing them earlier causes a data race with the copy, and the results are undefined.
Multi-dimensional Transfer¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | |
The example above shows the basic pattern for transferring multi-dimensional tensor data based on the N-dimensional view specifications of the source and destination. This multi-dimensional transfer interface handles complex memory layout transfers between two N-dimensional views.
Characteristics:
- Argument structure (
Pointer, Layoutpair): Each transfer argument is a pair of a pointer and a layout, which lets the API handle transfers between two N-dimensional views. The example above uses a 2D view of height (h) and width (w), but the rank is not fixed to this value and can be larger. - Separate pointer and layout: The pointer and the layout specification are deliberately separate so that one layout definition, such as a single KV cache block layout, can be reused for transfers of many independent buffer instances. You therefore do not need to recreate the layout object every time the buffer changes.
Layout validation rules and error codes:
| Check | Error or behavior on violation |
|---|---|
| Rank and shape match between the two layouts | HaMemoryInvalidArgument (this API does not support automatic broadcasting) |
Dimension count validity (1 <= ndim <= HA_MAX_NDIM) |
HaMemoryInvalidArgument |
| Strides are in elements of the data type (dtype), not bytes | Accesses the wrong memory address without an error |
| Pointer validity (null pointer passed) | HaMemoryInvalidAddress |
Additional transfer behavior and optimizations:
- Automatic direction inference: The transfer direction is inferred from the
scopefield of the two layout objects, so it is not a separate argument. It is also valid when both source and target refer to host memory; in that case the copy is done within host memory without a device transfer. - Automatic normalization and flattening: The runtime normalizes the two views internally and then automatically merges adjacent dimensions within each view that are contiguous in memory (whose strides line up). As a result, a logically multi-dimensional structure whose actual memory layout is contiguous is handled in a single-pass transfer, and callers do not need to flatten the view in advance for transfer efficiency.
3.5 Direct File Load¶
1 | |
1 | |
Data is loaded directly from an open file descriptor into device memory through a file system I/O stream, so there is no unnecessary copy through host system memory. This is the recommended path for the initial loading phase of large model weights.
3.6 Device Memory Dump¶
dump_for_each_channel() (C++: HaMemDumpForEachChannel()) reads the raw bytes of an allocation,
one DRAM channel at a time, without de-interleaving.
1 | |
1 2 | |
This is a debugging tool for checking that a layout landed where you intended. It transfers the whole size synchronously, so do not use it as a data path.
3.7 Virtual Memory & HW Pages¶
The call order and values in this section follow the current device (engineering sample).
The KV cache translates addresses through HW pages. The VMM API manages the HW pages, so use the VMM API for address reservation, pool allocation, and per-request page management.
Concepts¶
| Term | Meaning |
|---|---|
| Block | Logical page unit in which the KV cache is mapped; in general it bundles one or more HW pages. On the engineering sample, one block in the pool and in mapping is one token (block_size 1 for map and unmap). num_attn_layers × num_kv_heads × head_dim × element size bytes |
| Pool | Physical memory that holds all KV. The first half is K, the second half is V. Always a single contiguous segment (HaVmmBlockSegment); raw_ptr is what gets freed |
| KV address | The address kernels use. Made of the translation-enable bit (58), the K/V bit (57), and the request slot (bits 56:47, 0–1023); it is not a reservation |
| HW page | Address-translation unit in which blocks are placed; 4 KiB per channel on the engineering sample |
| Request page | KEY and VALUE pages allocated per request slot, made of HW pages. One HW page covers 4 KiB × the number of channels for each (layer, head). While the request is live they only grow with the token count, and they are returned per request |
Every call is routed to the current device of the calling thread (selected with HaSetDevice(), id 0–7). Address values
are not used to select the device.
Call Order¶
Create the pool first, then initialize the HW pages from that pool
The example below assumes L layers, H KV heads, head_dim D, and a bf16 (16-bit) KV cache.
R is the number of request slots, T the maximum number of tokens per request, and N the number of tokens the pool holds for each of K and V.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | |
KV addresses are not reservations, so there is nothing to free. The meaning of each argument is in the
VMM API. max_num_blocks_per_row × block_size (1 × T) is
the maximum number of tokens per request, and the pool must be large enough to hold the KEY and VALUE pages of one request.
Rules¶
Do not free a pool that still has mapped pages
HaVmmFreePool() does not check for mappings.
If you free the pool while request pages remain, the freed memory is reused by the next allocation while the HW pages still
point to those addresses. Unmap every request before freeing. What gets freed is raw_ptr, not
pa_start, and HaVmmFreePool() handles this.
Change mappings only after the previous step's kernels finish
Map and unmap do not wait for running kernels. Confirm that the previous step's kernels have finished before changing the mappings for the next step.
Pass the exact token count every step
Map grows pages by the difference between cached_blocks_per_req (the number of tokens already mapped) and the new token count.
Pass the request's full mapped token count to unmap; it is released per request all at once.
Calling HaVmmInitBlocks again discards existing state
A repeated call tears down the existing state and initializes again.
PyTorch Ops and Runtime Entry Points¶
The PyTorch torch.ops.lpu.vmm_* ops validate their arguments and then call the runtime entry points below. The op
list and signatures are in PyTorch VMM ops.
| PyTorch op | Runtime entry point |
|---|---|
vmm_create_kv_pool |
HaVmmAllocatePool |
vmm_reserve |
HaVmmReserveBlocks |
vmm_init_blocks |
HaVmmInitBlocks |
vmm_map_blocks, vmm_map_blocks_batch |
HaVmmMapBlocksBatchFromPool |
vmm_unmap_blocks, vmm_unmap_blocks_batch |
HaVmmUnmapBlocksBatchFromPool |
vmm_deinit |
HaVmmDeinit |
The order in which the vLLM plugin calls these ops is described in Paged KV Cache: Flow in vLLM.
Next Steps¶
- Streams and Events — handling asynchronous copies safely
- Error Handling — meaning of memory error codes