Driver API: Memory¶
Device allocation, copies, device memory statistics, and host memory. Every copy function posts the copy on a stream and returns once it is queued. See Driver API for the concepts.
Contents¶
| Name | Kind | Description |
|---|---|---|
ha_drv_malloc |
API | Allocate device memory |
ha_drv_malloc_3d |
API | Allocate device memory from a 3D shape |
ha_drv_memcpy |
API | 1-D memory copy (asynchronous) |
ha_drv_memcpy_nd |
API | Memory copy based on an N-D layout (asynchronous) |
ha_drv_free |
API | Free device memory immediately |
ha_drv_device_mem_get_stats |
API | Get device memory statistics (runtime pool counters and the driver's free DRAM) |
ha_drv_device_mem_reset_peak_stats |
API | Reset device memory peak statistics |
ha_drv_device_mem_trim |
API | Return pool segments with no allocated block to the driver |
ha_drv_host_mem_alloc |
API | Allocate host memory (process limit applies) |
ha_drv_host_mem_free |
API | Free host memory allocated with ha_drv_host_mem_alloc |
ha_drv_host_mem_get_stats |
API | Get process host memory statistics |
ha_drv_host_mem_reset_peak_stats |
API | Reset host memory peak statistics |
ha_drv_host_mem_trim |
API | Return freed host memory held for reuse to the OS |
ha_drv_malloc¶
Allocate device memory on dev. Allocation is synchronous and not stream-ordered (like cudaMalloc), so stream
is ignored.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle from ha_drv_open. |
ptr |
out | — | Receives the allocated device pointer on success. |
size |
in | — | Allocation size in bytes. |
is_interleaved |
in | — | Whether to interleave the allocation across DRAM channels. |
stream |
in | — | Ignored (allocation is not stream-ordered). |
Returns — HaSuccess; HaInvalidArgument if dev is null; HaDeviceNotReady if a reset is in progress;
HaMemoryInvalidArgument for a zero size, or for an interleaved size that is not a multiple of the DRAM channel
count; HaMemoryAllocationFailed if the memory pool cannot allocate it.
Signature · Example
1 | |
Example
1 2 | |
ha_drv_malloc_3d¶
Allocate a 3D-shaped region on dev.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle from ha_drv_open. |
ptr |
out | — | Receives the allocated device pointer on success. |
shape |
in | — | Tensor shape to allocate. w and h must be non-zero and c must be 1. See HaShape. |
is_interleaved |
in | — | Whether to interleave the allocation across DRAM channels. |
stream |
in | — | Ignored (allocation is not stream-ordered). |
nbytes_out |
out | — | Optional; receives the actual byte size allocated. |
Returns — HaSuccess; HaInvalidArgument if dev is null; HaDeviceNotReady if a reset is in progress;
HaMemoryInvalidArgument for a shape that breaks the rule above; HaMemoryAllocationFailed if the memory pool
cannot allocate it.
Signature · Example
1 2 | |
Example
1 2 3 4 | |
ha_drv_memcpy¶
1-D copy, posted on a stream. Asynchronous on every stream, HA_STREAM_DEFAULT (the calling thread's default
stream on dev) included: it returns once queued. Wait with ha_drv_wait(HA_WAIT_STREAM_SYNC) on the same stream;
host memory must stay valid until then.
Host↔device: if the device-side allocation is interleaved across the DRAM channels, the copy is split into one op per channel. Device-to-device is one op; its destination may be another device's address (see
HA_DEV_ATTR_P2P_SUPPORTED), which the driver routes by the address.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle from ha_drv_open; owns the device side (for device-to-device, the source). |
dst |
in | — | Destination address (device pointer, or host pointer cast via HaDevicePtr). |
src |
in | — | Source address (device pointer, or host pointer cast via HaDevicePtr). |
size |
in | — | Byte count to copy. |
direction |
in | — | Copy direction. One of HostToDevice, DeviceToHost, DeviceToDevice (see HaMemCpyDirection). |
stream |
in | — | Stream from ha_drv_obj_create, or HA_STREAM_DEFAULT. |
Returns — HaSuccess; HaInvalidArgument if dev is null or direction is unknown; HaDeviceNotReady if a
reset is in progress; HaMemoryInvalidArgument if size bytes from the device address extend past the allocation; or an error from posting the op (e.g. the stream's sticky error, an invalid
stream). Nothing is posted when the split fails; if posting fails, the remaining pieces are not posted.
Signature · Example
1 2 | |
Example
1 2 | |
ha_drv_memcpy_nd¶
N-D layout-described copy, posted on a stream. Asynchronous on every stream, HA_STREAM_DEFAULT included: it
returns once queued (see ha_drv_memcpy).
The driver plans the copy into transfers (one stream op each). A host↔device copy whose transfers would be many
small pieces is staged through a host buffer the driver allocates and charges against the process-wide staging
budget (HA_MEMCPYND_ASYNC_STAGING_MAX_UNSYNCED_BYTES, HA_MEMCPYND_STAGING_MAX_BYTES): host-to-device gathers
into it before returning, so the source buffer can be reused or freed on return; device-to-host lands in the destination only at the
sync that covers the copy. The buffer is released at that sync, or when the stream is destroyed or the device reset.
Device-to-device: dst may be another device's address (see HA_DEV_ATTR_P2P_SUPPORTED); then each
transfer must meet the driver's minimum cross-device transfer size.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle from ha_drv_open; owns the device side (for device-to-device, the source). |
dst |
in | — | Destination address. |
dst_layout |
in | — | Destination tensor layout. See HaLayout. |
src |
in | — | Source address. |
src_layout |
in | — | Source tensor layout. See HaLayout. |
stream |
in | — | Stream from ha_drv_obj_create, or HA_STREAM_DEFAULT. |
Returns — HaSuccess; HaInvalidArgument if dev, dst_layout or src_layout is null; HaDeviceNotReady
if a reset is in progress; HaMemoryInvalidArgument for a host-to-host copy; the planner's error;
HaMemCpyNotSupported for a cross-device transfer below the minimum size; HaHostMemLimitExceeded if the staging budget or host memory limit would be exceeded; HaMemoryAllocationFailed if the staging buffer cannot be allocated; or the post's error (e.g. the stream's sticky error). Nothing is posted when planning or staging
fails.
Signature · Example
1 2 | |
Example
1 2 3 4 5 6 7 8 9 10 11 12 | |
ha_drv_free¶
Free device memory on dev immediately. To free it after a stream's preceding work completes, post HA_STREAM_OP_FREE
(ha_drv_stream_post).
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle from ha_drv_open. |
da |
in | — | Device pointer previously returned by ha_drv_malloc / ha_drv_malloc_3d. |
stream |
in | — | Ignored (this free is not stream-ordered). |
Returns — HaSuccess; HaInvalidArgument if dev is null; HaDeviceNotReady if a reset is in progress;
HaMemoryInvalidAddress if da is 0 or not a live allocation of dev.
Signature · Example
1 | |
Example
1 2 | |
ha_drv_device_mem_get_stats¶
Read the device memory statistics: the runtime's pool counters and the driver's free DRAM figure. struct_size is set
by the caller; see HaDeviceMemStats.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle. |
out |
in/out | — | In: struct_size set to sizeof(HaDeviceMemStats). Out: the statistics, and the bytes written in struct_size. |
Returns — HaSuccess; HaInvalidArgument for a null dev or out, or a struct_size too small for the header
and the first field; HaDeviceNotReady while the device resets; HaDeviceNotInitialized when the device's memory is
not set up; the driver's error if it cannot be read.
Signature · Example
1 | |
Example
1 2 3 4 5 | |
Note
With HA_DEVICE_MEM_STATS_PRINT=1 the statistics are also printed to standard error at each call
(Environment variables).
ha_drv_device_mem_reset_peak_stats¶
Reset every peak of HaDeviceMemStats to its current value. An allocation that runs concurrently with the call may or may not be counted in the new peak.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle. |
Returns — HaSuccess; HaInvalidArgument for a null dev; HaDeviceNotReady while the device resets.
Signature
1 | |
ha_drv_device_mem_trim¶
Return the pool segments that hold no allocated block to the driver. A segment that still holds a live block is kept
whole. Memory freed in stream order (HA_STREAM_OP_FREE) returns to the pool only when the stream is synchronized, so
synchronize first for it to count.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dev |
in | — | Device handle. |
Returns — HaSuccess; HaInvalidArgument for a null dev; HaDeviceNotReady while the device resets.
Signature · Example
1 | |
Example
1 2 | |
ha_drv_host_mem_alloc¶
Allocate host memory through the process's host memory manager, which keeps per-tag statistics and applies the
process limit (HA_HOST_MEM_MAX_BYTES). Needs no device: callable before ha_drv_init. Not fork-safe: a child
must not use host memory or statistics inherited from its parent.
| Parameter | Direction | Default | Description |
|---|---|---|---|
desc |
in | — | Request; see HaDrvHostMemAllocDesc. |
out_ptr |
out | — | Receives the 64-byte aligned address (HaHostPtr). |
Returns — HaSuccess; HaInvalidArgument for a null argument, a zero size, an unknown flag or tag, a
reserved field set or a malformed struct_size; HaHostMemLimitExceeded if the allocation would exceed HA_HOST_MEM_MAX_BYTES; HaMemoryAllocationFailed if the system allocation fails.
Signature · Example
1 | |
Example
1 2 3 4 5 6 7 8 9 | |
ha_drv_host_mem_free¶
Free an allocation from ha_drv_host_mem_alloc.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
in | — | Address returned by ha_drv_host_mem_alloc; 0 is a no-op, as free(NULL) is. |
Returns — HaSuccess, or HaMemoryInvalidArgument if ptr is neither 0 nor a live allocation from
ha_drv_host_mem_alloc.
Signature
1 | |
ha_drv_host_mem_get_stats¶
Read the process's host memory statistics (every tag summed). struct_size is set by the caller; see
HaHostMemStats.
| Parameter | Direction | Default | Description |
|---|---|---|---|
out |
in/out | — | In: struct_size set to sizeof(HaHostMemStats). Out: the statistics, and the bytes written in struct_size. See HaHostMemStats. |
Returns — HaSuccess, or HaInvalidArgument for a null out or a struct_size too small for the two header
fields.
Signature · Example
1 | |
Example
1 2 3 4 5 | |
Note
With HA_HOST_MEM_STATS_PRINT=1 the statistics are also printed to standard error at each call
(Environment variables).
ha_drv_host_mem_reset_peak_stats¶
Restart both peaks of HaHostMemStats (peak_allocated_bytes, peak_reserved_bytes)
from their current values. An allocation racing the call may land on either side of it.
Returns — HaSuccess.
Signature · Example
1 | |
Example
1 2 3 4 5 | |
ha_drv_host_mem_trim¶
Return to the OS, where possible, the freed memory the manager keeps for reuse: a pool segment that still holds a live allocation is kept whole. Only freed memory is touched, so a buffer an in-flight copy still uses, not yet freed by its owner, is unaffected.
Returns — HaSuccess.
Signature · Example
1 | |
Example
1 2 | |