Skip to content

3. Memory Model

The driver API's memory functions return at different points.

  • Allocation and free complete within the call: ha_drv_malloc, ha_drv_malloc_3d, and ha_drv_free finish the allocation or free before they return. These three functions do not use their stream argument, so they do not follow that stream's work order. To free after the work previously enqueued on a stream has finished, use the stream operation HA_STREAM_OP_FREE.
  • Data transfers return right after enqueuing: ha_drv_memcpy and ha_drv_memcpy_nd are asynchronous on every stream, including HA_STREAM_DEFAULT. Returning only means that the transfer command has been placed on the stream.

Callers must therefore follow two rules.

  1. Do not call ha_drv_free on memory the device is still using. Free it after confirming with ha_drv_wait that the work using it has finished, or free it in stream order with HA_STREAM_OP_FREE.
  2. Do not free or modify a host buffer passed to a transfer until ha_drv_wait confirms completion. The only exception is the source of an ha_drv_memcpy_nd host-to-device copy that goes through staging (Section 3.2).

Contents

Section Objective
3.1 Allocation & Deallocation Synchronous allocation and free, and asynchronous free that follows stream order
3.2 Transfer 1-D copy, N-D tensor copy, and direct device-to-device transfer (P2P)
3.3 Direct File Load Direct DMA load from a file descriptor (FD) into device memory
3.4 TLB Entry points for the static KV-cache TLB and the page-table TLB
3.5 Host Memory Allocation and limits through the process host memory manager

3.1 Allocation & Deallocation

Allocation

ha_drv_malloc and ha_drv_malloc_3d are synchronous.

1
2
3
4
5
6
7
8
9
HaDevicePtr buf = 0;
// Allocate interleaved across DRAM channels
ha_drv_malloc(dev, &buf, size_bytes, /*is_interleaved=*/true, HA_STREAM_DEFAULT);

HaShape shape{/*w=*/4096, /*h=*/64, /*c=*/1};
HaDevicePtr shaped = 0;
size_t nbytes = 0;
// Allocate based on a 3D shape specification
ha_drv_malloc_3d(dev, &shaped, shape, /*is_interleaved=*/true, HA_STREAM_DEFAULT, &nbytes);

Deallocation

Use ha_drv_free for an immediate free. To free after the work previously enqueued on a stream has finished, enqueue a HA_STREAM_OP_FREE stream operation. Like CUDA's cudaFreeAsync, HA_STREAM_OP_FREE follows stream order. The memory returns to the pool when a later stream synchronization (HA_WAIT_STREAM_SYNC or HA_WAIT_DEVICE_SYNC) succeeds, or when the stream is destroyed. To free memory allocated on another device, set free.owner to that device's handle.

1
2
3
4
5
6
7
8
// 1. Immediate free: call only after all stream work using this memory has finished. The stream argument is not used.
ha_drv_wait(dev, HA_STREAM_DEFAULT, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);
ha_drv_free(dev, buf, HA_STREAM_DEFAULT);

// 2. Stream-ordered free: freed after the previously enqueued work, returned to the pool at a later sync.
HaDrvStreamOp free_op{HA_STREAM_OP_FREE, {}};
free_op.free.ptr = shaped;
ha_drv_stream_post(dev, stream, &free_op, nullptr);
API Sync / Async Behavior
ha_drv_malloc() Sync Allocates device memory. The stream argument is not used
ha_drv_malloc_3d() Sync Allocates device memory based on a 3D shape specification. The stream argument is not used
ha_drv_free() Sync Frees device memory immediately. The stream argument is not used
ha_drv_stream_post() + HA_STREAM_OP_FREE Async Asynchronous free that follows stream operation order

Memory Statistics & Trim

Freed device memory goes back to the runtime pool for reuse, and empty segments may also be kept in the pool depending on conditions, so the free memory seen by the driver may not increase right away. Check the device's memory state with ha_drv_device_mem_get_stats(), which returns the pool counters together with the device-wide free DRAM reported by the driver (free_bytes). Return pool segments that have no allocated blocks to the driver with ha_drv_device_mem_trim(). Memory freed with HA_STREAM_OP_FREE returns to the pool only after a synchronization following that operation, so synchronize before trimming. Restart peak counting with ha_drv_device_mem_reset_peak_stats().

3.2 Transfer

1-D Transfer

ha_drv_memcpy enqueues a 1-D host-device or device-to-device transfer on a stream. In a host-device copy whose device side is allocated interleaved across DRAM channels, the driver splits the work into one operation per channel piece.

1
2
3
4
5
ha_drv_memcpy(dev, buf, reinterpret_cast<HaDevicePtr>(host_in), n, HaMemCpyDirection::HostToDevice, stream);
ha_drv_memcpy(dev, reinterpret_cast<HaDevicePtr>(host_out), buf, n, HaMemCpyDirection::DeviceToHost, stream);

// Transfer completion is guaranteed only after the stream is synchronized.
ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);

Multi-dimensional Transfer

ha_drv_memcpy_nd splits an N-D copy described by HaLayout structures into sub-transfers and enqueues them on a stream. A host-device copy made of many small pieces goes through a host staging buffer that the driver allocates.

  • Host → device: When the call returns, the source host buffer has already been copied into the staging buffer.
  • Device → host: The destination host buffer holds the result once the stream synchronization (ha_drv_wait) after this copy completes.

A staging buffer is released at that synchronization, when the stream is destroyed, or when the device is reset. When the total of unsynchronized staging buffers exceeds HA_MEMCPYND_ASYNC_STAGING_MAX_UNSYNCED_BYTES, a new staged copy first synchronizes its own stream and then proceeds; a copy that would exceed HA_MEMCPYND_STAGING_MAX_BYTES or HA_HOST_MEM_MAX_BYTES is rejected with HaHostMemLimitExceeded. For the defaults, see Environment Variables.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
HaLayout dev_layout = HA_LAYOUT_INIT;
dev_layout.scope = HaMemScope::Device;
dev_layout.dtype = HaDataType::Bytes;
dev_layout.ndim = 1;
dev_layout.shape[0] = n;
dev_layout.strides[0] = 1;

HaLayout host_layout = dev_layout;
host_layout.scope = HaMemScope::Host;

ha_drv_memcpy_nd(dev, buf, &dev_layout, reinterpret_cast<HaDevicePtr>(host_in), &host_layout, stream);
ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);

Peer Transfer

A device-to-device copy with ha_drv_memcpy or ha_drv_memcpy_nd can take another device's address as its destination (P2P). In that case, pass the handle of the device that holds the source as dev. Check whether P2P is supported with the HA_DEV_ATTR_P2P_SUPPORTED attribute. For a P2P transfer with ha_drv_memcpy_nd, each individual transfer must meet the driver's minimum transfer size; otherwise it returns HaMemCpyNotSupported.

API Sync / Async Behavior
ha_drv_memcpy() Async Enqueues a 1-D copy on the stream
ha_drv_memcpy_nd() Async Enqueues an N-D layout copy on the stream (uses host staging when needed)
ha_drv_get_attr() + HA_DEV_ATTR_P2P_SUPPORTED Sync Checks whether direct device-to-device copy (P2P) is supported

Caution: Host buffer lifetime

Do not free or reuse a host buffer passed to an asynchronous transfer until the target stream is synchronized (ha_drv_wait). When the call returns, the device transfer may still be in progress.

3.3 Direct File Load

Loads data by DMA directly from a file descriptor into device memory without going through host memory. Enqueue it as a HA_STREAM_OP_LOAD_FILE stream operation. The file descriptor must remain open until the DMA completes.

1
2
3
4
5
6
HaDrvStreamOp load{HA_STREAM_OP_LOAD_FILE, {}};
load.load_file = {fd, /*off=*/0, buf, len};
ha_drv_stream_post(dev, stream, &load, nullptr);

ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);
close(fd);   // Close the file descriptor after confirming synchronization has completed

3.4 TLB

The TLB functions provide the same functionality as the runtime TLB API, per device handle (HaDrvDev).

API Behavior
ha_drv_init_tlb() Initializes the TLB with a static KV-cache geometry
ha_drv_alloc_tlb_page() / ha_drv_dealloc_tlb_page() Allocates and releases per-request key/value pages
ha_drv_configure_tlb() Configures the page-table TLB structure
ha_drv_set_page_table_range() Sets the page table address range and size
ha_drv_flush_tlb() Flushes and invalidates the entire TLB cache

3.5 Host Memory

The driver API provides a per-process host memory manager that is independent of any device. It does not require an open device, so it can be called even before ha_drv_init. Each allocation specifies a purpose tag (HaDrvHostMemTag); the manager keeps per-tag statistics and enforces a process-wide limit. The host staging buffers used by ha_drv_memcpy_nd are also allocated from the same manager, so they count toward this limit.

API Behavior
ha_drv_host_mem_alloc() Allocates 64-byte-aligned host memory with the given tag and flags
ha_drv_host_mem_free() Frees allocated host memory
ha_drv_host_mem_get_stats() Queries process-wide host memory statistics (current and peak allocation, amount reserved from the OS, allocation and free counts)
ha_drv_host_mem_reset_peak_stats() Restarts the peak statistics from the current values
ha_drv_host_mem_trim() Returns as much retained freed memory to the OS as possible

Freed memory stays in the pool and is reused for later allocations. When an allocation would exceed the limit, the manager first returns entirely empty pool segments to the OS and rejects the allocation with HaHostMemLimitExceeded if that is still not enough. An allocation with HA_HOST_MEM_FLAG_NO_POOL bypasses the pool and is returned to the OS as soon as it is freed. To return retained freed memory explicitly, call ha_drv_host_mem_trim(). Pool segments that still hold live allocations are kept, and buffers that have not been freed are not affected.

Environment variable Effect
HA_HOST_MEM_MAX_BYTES Upper limit (in bytes) on host memory the process reserves from the OS. No limit if unset or negative

Do not use after fork

The host memory manager is not fork-safe. A child process must not use host memory inherited from its parent, and the inherited statistics cannot be trusted.


Next Steps