3. Memory Model¶
The driver API's memory functions return at different points.
- Allocation and free complete within the call:
ha_drv_malloc,ha_drv_malloc_3d, andha_drv_freefinish the allocation or free before they return. These three functions do not use theirstreamargument, so they do not follow that stream's work order. To free after the work previously enqueued on a stream has finished, use the stream operationHA_STREAM_OP_FREE. - Data transfers return right after enqueuing:
ha_drv_memcpyandha_drv_memcpy_ndare asynchronous on every stream, includingHA_STREAM_DEFAULT. Returning only means that the transfer command has been placed on the stream.
Callers must therefore follow two rules.
- Do not call
ha_drv_freeon memory the device is still using. Free it after confirming withha_drv_waitthat the work using it has finished, or free it in stream order withHA_STREAM_OP_FREE. - Do not free or modify a host buffer passed to a transfer until
ha_drv_waitconfirms completion. The only exception is the source of anha_drv_memcpy_ndhost-to-device copy that goes through staging (Section 3.2).
Contents¶
| Section | Objective |
|---|---|
| 3.1 Allocation & Deallocation | Synchronous allocation and free, and asynchronous free that follows stream order |
| 3.2 Transfer | 1-D copy, N-D tensor copy, and direct device-to-device transfer (P2P) |
| 3.3 Direct File Load | Direct DMA load from a file descriptor (FD) into device memory |
| 3.4 TLB | Entry points for the static KV-cache TLB and the page-table TLB |
| 3.5 Host Memory | Allocation and limits through the process host memory manager |
3.1 Allocation & Deallocation¶
Allocation¶
ha_drv_malloc and ha_drv_malloc_3d are synchronous.
1 2 3 4 5 6 7 8 9 | |
Deallocation¶
Use ha_drv_free for an immediate free. To free after the work previously enqueued on a stream has finished, enqueue a
HA_STREAM_OP_FREE stream operation. Like CUDA's cudaFreeAsync, HA_STREAM_OP_FREE follows stream order. The memory
returns to the pool when a later stream synchronization (HA_WAIT_STREAM_SYNC or HA_WAIT_DEVICE_SYNC) succeeds, or when the stream is destroyed. To free memory allocated on another device, set free.owner to that
device's handle.
1 2 3 4 5 6 7 8 | |
| API | Sync / Async | Behavior |
|---|---|---|
ha_drv_malloc() |
Sync | Allocates device memory. The stream argument is not used |
ha_drv_malloc_3d() |
Sync | Allocates device memory based on a 3D shape specification. The stream argument is not used |
ha_drv_free() |
Sync | Frees device memory immediately. The stream argument is not used |
ha_drv_stream_post() + HA_STREAM_OP_FREE |
Async | Asynchronous free that follows stream operation order |
Memory Statistics & Trim¶
Freed device memory goes back to the runtime pool for reuse, and empty segments may also be kept in the pool depending
on conditions, so the free memory seen by the driver may not increase right away. Check the device's memory state with ha_drv_device_mem_get_stats(),
which returns the pool counters together with the device-wide free DRAM reported by the driver (free_bytes). Return
pool segments that have no allocated blocks to the driver with ha_drv_device_mem_trim().
Memory freed with HA_STREAM_OP_FREE returns to the pool only after a synchronization following that operation, so
synchronize before trimming. Restart peak counting with ha_drv_device_mem_reset_peak_stats().
3.2 Transfer¶
1-D Transfer¶
ha_drv_memcpy enqueues a 1-D host-device or device-to-device transfer on a stream. In a host-device copy whose device
side is allocated interleaved across DRAM channels, the driver splits the work into one operation per channel piece.
1 2 3 4 5 | |
Multi-dimensional Transfer¶
ha_drv_memcpy_nd splits an N-D copy described by HaLayout structures into sub-transfers and enqueues them on a
stream. A host-device copy made of many small pieces goes through a host staging buffer that the driver allocates.
- Host → device: When the call returns, the source host buffer has already been copied into the staging buffer.
- Device → host: The destination host buffer holds the result once the stream synchronization (
ha_drv_wait) after this copy completes.
A staging buffer is released at that synchronization, when the stream is destroyed, or when the device is reset. When
the total of unsynchronized staging buffers exceeds HA_MEMCPYND_ASYNC_STAGING_MAX_UNSYNCED_BYTES, a new staged copy
first synchronizes its own stream and then proceeds; a copy that would exceed HA_MEMCPYND_STAGING_MAX_BYTES or
HA_HOST_MEM_MAX_BYTES is rejected with HaHostMemLimitExceeded. For the defaults, see
Environment Variables.
1 2 3 4 5 6 7 8 9 10 11 12 | |
Peer Transfer¶
A device-to-device copy with ha_drv_memcpy or ha_drv_memcpy_nd can take another device's address as its destination
(P2P). In that case, pass the handle of the device that holds the source as dev. Check whether P2P is supported with
the HA_DEV_ATTR_P2P_SUPPORTED attribute. For a P2P transfer with ha_drv_memcpy_nd, each individual transfer must meet
the driver's minimum transfer size; otherwise it returns HaMemCpyNotSupported.
| API | Sync / Async | Behavior |
|---|---|---|
ha_drv_memcpy() |
Async | Enqueues a 1-D copy on the stream |
ha_drv_memcpy_nd() |
Async | Enqueues an N-D layout copy on the stream (uses host staging when needed) |
ha_drv_get_attr() + HA_DEV_ATTR_P2P_SUPPORTED |
Sync | Checks whether direct device-to-device copy (P2P) is supported |
Caution: Host buffer lifetime
Do not free or reuse a host buffer passed to an asynchronous transfer until the target stream is synchronized
(ha_drv_wait). When the call returns, the device transfer may still be in progress.
3.3 Direct File Load¶
Loads data by DMA directly from a file descriptor into device memory without going through host memory. Enqueue it as a
HA_STREAM_OP_LOAD_FILE stream operation. The file descriptor must remain open until the DMA completes.
1 2 3 4 5 6 | |
3.4 TLB¶
The TLB functions provide the same functionality as the runtime TLB API, per device handle (HaDrvDev).
| API | Behavior |
|---|---|
ha_drv_init_tlb() |
Initializes the TLB with a static KV-cache geometry |
ha_drv_alloc_tlb_page() / ha_drv_dealloc_tlb_page() |
Allocates and releases per-request key/value pages |
ha_drv_configure_tlb() |
Configures the page-table TLB structure |
ha_drv_set_page_table_range() |
Sets the page table address range and size |
ha_drv_flush_tlb() |
Flushes and invalidates the entire TLB cache |
3.5 Host Memory¶
The driver API provides a per-process host memory manager that is independent of any device. It does not require an
open device, so it can be called even before ha_drv_init. Each allocation specifies a purpose tag (HaDrvHostMemTag); the manager keeps per-tag statistics and enforces a process-wide
limit. The host staging buffers used by ha_drv_memcpy_nd are also allocated from the same
manager, so they count toward this limit.
| API | Behavior |
|---|---|
ha_drv_host_mem_alloc() |
Allocates 64-byte-aligned host memory with the given tag and flags |
ha_drv_host_mem_free() |
Frees allocated host memory |
ha_drv_host_mem_get_stats() |
Queries process-wide host memory statistics (current and peak allocation, amount reserved from the OS, allocation and free counts) |
ha_drv_host_mem_reset_peak_stats() |
Restarts the peak statistics from the current values |
ha_drv_host_mem_trim() |
Returns as much retained freed memory to the OS as possible |
Freed memory stays in the pool and is reused for later allocations. When an allocation would exceed the limit, the
manager first returns entirely empty pool segments to the OS and rejects the allocation with HaHostMemLimitExceeded
if that is still not enough. An
allocation with HA_HOST_MEM_FLAG_NO_POOL bypasses the pool and is returned to the OS as soon as it is freed. To return
retained freed memory explicitly, call ha_drv_host_mem_trim(). Pool segments that still hold live allocations are
kept, and buffers that have not been freed are not affected.
| Environment variable | Effect |
|---|---|
HA_HOST_MEM_MAX_BYTES |
Upper limit (in bytes) on host memory the process reserves from the OS. No limit if unset or negative |
Do not use after fork
The host memory manager is not fork-safe. A child process must not use host memory inherited from its parent, and the inherited statistics cannot be trusted.
Next Steps¶
- Streams and Events — Stream operations and waits
- Driver API: Memory · Driver API: TLB — Per-function signatures