Skip to content

Driver API: Memory

Device allocation, copies, device memory statistics, and host memory. Every copy function posts the copy on a stream and returns once it is queued. See Driver API for the concepts.

Contents

Name Kind Description
ha_drv_malloc API Allocate device memory
ha_drv_malloc_3d API Allocate device memory from a 3D shape
ha_drv_memcpy API 1-D memory copy (asynchronous)
ha_drv_memcpy_nd API Memory copy based on an N-D layout (asynchronous)
ha_drv_free API Free device memory immediately
ha_drv_device_mem_get_stats API Get device memory statistics (runtime pool counters and the driver's free DRAM)
ha_drv_device_mem_reset_peak_stats API Reset device memory peak statistics
ha_drv_device_mem_trim API Return pool segments with no allocated block to the driver
ha_drv_host_mem_alloc API Allocate host memory (process limit applies)
ha_drv_host_mem_free API Free host memory allocated with ha_drv_host_mem_alloc
ha_drv_host_mem_get_stats API Get process host memory statistics
ha_drv_host_mem_reset_peak_stats API Reset host memory peak statistics
ha_drv_host_mem_trim API Return freed host memory held for reuse to the OS

ha_drv_malloc

Allocate device memory on dev. Allocation is synchronous and not stream-ordered (like cudaMalloc), so stream is ignored.

Parameter Direction Default Description
dev in — Device handle from ha_drv_open.
ptr out — Receives the allocated device pointer on success.
size in — Allocation size in bytes.
is_interleaved in — Whether to interleave the allocation across DRAM channels.
stream in — Ignored (allocation is not stream-ordered).

Returns — HaSuccess; HaInvalidArgument if dev is null; HaDeviceNotReady if a reset is in progress; HaMemoryInvalidArgument for a zero size, or for an interleaved size that is not a multiple of the DRAM channel count; HaMemoryAllocationFailed if the memory pool cannot allocate it.

Signature · Example
1
HaError ha_drv_malloc(HaDrvDev dev, HaDevicePtr *ptr, size_t size, bool is_interleaved, HaStream_t stream);

Example

1
2
HaDevicePtr da = 0;
ha_drv_malloc(dev, &da, 1 << 20, /*is_interleaved=*/true, HA_STREAM_DEFAULT);

ha_drv_malloc_3d

Allocate a 3D-shaped region on dev.

Parameter Direction Default Description
dev in — Device handle from ha_drv_open.
ptr out — Receives the allocated device pointer on success.
shape in — Tensor shape to allocate. w and h must be non-zero and c must be 1. See HaShape.
is_interleaved in — Whether to interleave the allocation across DRAM channels.
stream in — Ignored (allocation is not stream-ordered).
nbytes_out out — Optional; receives the actual byte size allocated.

Returns — HaSuccess; HaInvalidArgument if dev is null; HaDeviceNotReady if a reset is in progress; HaMemoryInvalidArgument for a shape that breaks the rule above; HaMemoryAllocationFailed if the memory pool cannot allocate it.

Signature · Example
1
2
HaError ha_drv_malloc_3d(HaDrvDev dev, HaDevicePtr *ptr, const HaShape &shape, bool is_interleaved, HaStream_t stream,
                         size_t *nbytes_out);

Example

1
2
3
4
HaShape shape{/*w=*/4096, /*h=*/64, /*c=*/1};
HaDevicePtr da = 0;
size_t nbytes = 0;
ha_drv_malloc_3d(dev, &da, shape, /*is_interleaved=*/true, HA_STREAM_DEFAULT, &nbytes);

ha_drv_memcpy

1-D copy, posted on a stream. Asynchronous on every stream, HA_STREAM_DEFAULT (the calling thread's default stream on dev) included: it returns once queued. Wait with ha_drv_wait(HA_WAIT_STREAM_SYNC) on the same stream; host memory must stay valid until then.

Host↔device: if the device-side allocation is interleaved across the DRAM channels, the copy is split into one op per channel. Device-to-device is one op; its destination may be another device's address (see HA_DEV_ATTR_P2P_SUPPORTED), which the driver routes by the address.

Parameter Direction Default Description
dev in — Device handle from ha_drv_open; owns the device side (for device-to-device, the source).
dst in — Destination address (device pointer, or host pointer cast via HaDevicePtr).
src in — Source address (device pointer, or host pointer cast via HaDevicePtr).
size in — Byte count to copy.
direction in — Copy direction. One of HostToDevice, DeviceToHost, DeviceToDevice (see HaMemCpyDirection).
stream in — Stream from ha_drv_obj_create, or HA_STREAM_DEFAULT.

Returns — HaSuccess; HaInvalidArgument if dev is null or direction is unknown; HaDeviceNotReady if a reset is in progress; HaMemoryInvalidArgument if size bytes from the device address extend past the allocation; or an error from posting the op (e.g. the stream's sticky error, an invalid stream). Nothing is posted when the split fails; if posting fails, the remaining pieces are not posted.

Signature · Example
1
2
HaError ha_drv_memcpy(HaDrvDev dev, HaDevicePtr dst, HaDevicePtr src, size_t size, HaMemCpyDirection direction,
                      HaStream_t stream);

Example

1
2
ha_drv_memcpy(dev, da, reinterpret_cast<HaDevicePtr>(host), n, HaMemCpyDirection::HostToDevice, HA_STREAM_DEFAULT);
ha_drv_wait(dev, HA_STREAM_DEFAULT, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);   // keep the host buffer alive until here

ha_drv_memcpy_nd

N-D layout-described copy, posted on a stream. Asynchronous on every stream, HA_STREAM_DEFAULT included: it returns once queued (see ha_drv_memcpy).

The driver plans the copy into transfers (one stream op each). A host↔device copy whose transfers would be many small pieces is staged through a host buffer the driver allocates and charges against the process-wide staging budget (HA_MEMCPYND_ASYNC_STAGING_MAX_UNSYNCED_BYTES, HA_MEMCPYND_STAGING_MAX_BYTES): host-to-device gathers into it before returning, so the source buffer can be reused or freed on return; device-to-host lands in the destination only at the sync that covers the copy. The buffer is released at that sync, or when the stream is destroyed or the device reset.

Device-to-device: dst may be another device's address (see HA_DEV_ATTR_P2P_SUPPORTED); then each transfer must meet the driver's minimum cross-device transfer size.

Parameter Direction Default Description
dev in — Device handle from ha_drv_open; owns the device side (for device-to-device, the source).
dst in — Destination address.
dst_layout in — Destination tensor layout. See HaLayout.
src in — Source address.
src_layout in — Source tensor layout. See HaLayout.
stream in — Stream from ha_drv_obj_create, or HA_STREAM_DEFAULT.

Returns — HaSuccess; HaInvalidArgument if dev, dst_layout or src_layout is null; HaDeviceNotReady if a reset is in progress; HaMemoryInvalidArgument for a host-to-host copy; the planner's error; HaMemCpyNotSupported for a cross-device transfer below the minimum size; HaHostMemLimitExceeded if the staging budget or host memory limit would be exceeded; HaMemoryAllocationFailed if the staging buffer cannot be allocated; or the post's error (e.g. the stream's sticky error). Nothing is posted when planning or staging fails.

Signature · Example
1
2
HaError ha_drv_memcpy_nd(HaDrvDev dev, HaDevicePtr dst, const HaLayout *dst_layout, HaDevicePtr src,
                         const HaLayout *src_layout, HaStream_t stream);

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
HaLayout dev_layout = HA_LAYOUT_INIT;
dev_layout.scope = HaMemScope::Device;
dev_layout.dtype = HaDataType::Bytes;
dev_layout.ndim = 1;
dev_layout.shape[0] = n;
dev_layout.strides[0] = 1;

HaLayout host_layout = dev_layout;
host_layout.scope = HaMemScope::Host;

ha_drv_memcpy_nd(dev, da, &dev_layout, reinterpret_cast<HaDevicePtr>(host), &host_layout, HA_STREAM_DEFAULT);
ha_drv_wait(dev, HA_STREAM_DEFAULT, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);

ha_drv_free

Free device memory on dev immediately. To free it after a stream's preceding work completes, post HA_STREAM_OP_FREE (ha_drv_stream_post).

Parameter Direction Default Description
dev in — Device handle from ha_drv_open.
da in — Device pointer previously returned by ha_drv_malloc / ha_drv_malloc_3d.
stream in — Ignored (this free is not stream-ordered).

Returns — HaSuccess; HaInvalidArgument if dev is null; HaDeviceNotReady if a reset is in progress; HaMemoryInvalidAddress if da is 0 or not a live allocation of dev.

Signature · Example
1
HaError ha_drv_free(HaDrvDev dev, HaDevicePtr da, HaStream_t stream);

Example

1
2
ha_drv_wait(dev, HA_STREAM_DEFAULT, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);  // after the work that uses da has finished
ha_drv_free(dev, da, HA_STREAM_DEFAULT);

ha_drv_device_mem_get_stats

Read the device memory statistics: the runtime's pool counters and the driver's free DRAM figure. struct_size is set by the caller; see HaDeviceMemStats.

Parameter Direction Default Description
dev in — Device handle.
out in/out — In: struct_size set to sizeof(HaDeviceMemStats). Out: the statistics, and the bytes written in struct_size.

Returns — HaSuccess; HaInvalidArgument for a null dev or out, or a struct_size too small for the header and the first field; HaDeviceNotReady while the device resets; HaDeviceNotInitialized when the device's memory is not set up; the driver's error if it cannot be read.

Signature · Example
1
HaError ha_drv_device_mem_get_stats(HaDrvDev dev, HaDeviceMemStats *out);

Example

1
2
3
4
5
HaDeviceMemStats stats{};
stats.struct_size = sizeof(stats);
if (ha_drv_device_mem_get_stats(dev, &stats) == HaSuccess) {
  // stats.free_bytes, stats.total_bytes, stats.allocated_bytes ...
}

Note

With HA_DEVICE_MEM_STATS_PRINT=1 the statistics are also printed to standard error at each call (Environment variables).


ha_drv_device_mem_reset_peak_stats

Reset every peak of HaDeviceMemStats to its current value. An allocation that runs concurrently with the call may or may not be counted in the new peak.

Parameter Direction Default Description
dev in — Device handle.

Returns — HaSuccess; HaInvalidArgument for a null dev; HaDeviceNotReady while the device resets.

Signature
1
HaError ha_drv_device_mem_reset_peak_stats(HaDrvDev dev);

ha_drv_device_mem_trim

Return the pool segments that hold no allocated block to the driver. A segment that still holds a live block is kept whole. Memory freed in stream order (HA_STREAM_OP_FREE) returns to the pool only when the stream is synchronized, so synchronize first for it to count.

Parameter Direction Default Description
dev in — Device handle.

Returns — HaSuccess; HaInvalidArgument for a null dev; HaDeviceNotReady while the device resets.

Signature · Example
1
HaError ha_drv_device_mem_trim(HaDrvDev dev);

Example

1
2
ha_drv_wait(dev, stream, nullptr, HA_WAIT_STREAM_SYNC, -1, nullptr);   // returns the deferred frees to the pool
ha_drv_device_mem_trim(dev);

ha_drv_host_mem_alloc

Allocate host memory through the process's host memory manager, which keeps per-tag statistics and applies the process limit (HA_HOST_MEM_MAX_BYTES). Needs no device: callable before ha_drv_init. Not fork-safe: a child must not use host memory or statistics inherited from its parent.

Parameter Direction Default Description
desc in — Request; see HaDrvHostMemAllocDesc.
out_ptr out — Receives the 64-byte aligned address (HaHostPtr).

Returns — HaSuccess; HaInvalidArgument for a null argument, a zero size, an unknown flag or tag, a reserved field set or a malformed struct_size; HaHostMemLimitExceeded if the allocation would exceed HA_HOST_MEM_MAX_BYTES; HaMemoryAllocationFailed if the system allocation fails.

Signature · Example
1
HaError ha_drv_host_mem_alloc(const HaDrvHostMemAllocDesc *desc, HaHostPtr *out_ptr);

Example

1
2
3
4
5
6
7
8
9
HaDrvHostMemAllocDesc desc{};
desc.struct_size = sizeof(desc);
desc.flags = HA_HOST_MEM_FLAG_ZERO;
desc.tag = HA_HOST_MEM_TAG_OTHER;
desc.size = 4 << 20;

HaHostPtr host = 0;
ha_drv_host_mem_alloc(&desc, &host);
void *buf = reinterpret_cast<void *>(host);

ha_drv_host_mem_free

Free an allocation from ha_drv_host_mem_alloc.

Parameter Direction Default Description
ptr in — Address returned by ha_drv_host_mem_alloc; 0 is a no-op, as free(NULL) is.

Returns — HaSuccess, or HaMemoryInvalidArgument if ptr is neither 0 nor a live allocation from ha_drv_host_mem_alloc.

Signature
1
HaError ha_drv_host_mem_free(HaHostPtr ptr);

ha_drv_host_mem_get_stats

Read the process's host memory statistics (every tag summed). struct_size is set by the caller; see HaHostMemStats.

Parameter Direction Default Description
out in/out — In: struct_size set to sizeof(HaHostMemStats). Out: the statistics, and the bytes written in struct_size. See HaHostMemStats.

Returns — HaSuccess, or HaInvalidArgument for a null out or a struct_size too small for the two header fields.

Signature · Example
1
HaError ha_drv_host_mem_get_stats(HaHostMemStats *out);

Example

1
2
3
4
5
HaHostMemStats stats{};
stats.struct_size = sizeof(stats);
if (ha_drv_host_mem_get_stats(&stats) == HaSuccess) {
  // stats.allocated_bytes, stats.peak_allocated_bytes, stats.reserved_bytes ...
}

Note

With HA_HOST_MEM_STATS_PRINT=1 the statistics are also printed to standard error at each call (Environment variables).


ha_drv_host_mem_reset_peak_stats

Restart both peaks of HaHostMemStats (peak_allocated_bytes, peak_reserved_bytes) from their current values. An allocation racing the call may land on either side of it.

Returns — HaSuccess.

Signature · Example
1
HaError ha_drv_host_mem_reset_peak_stats(void);

Example

1
2
3
4
5
ha_drv_host_mem_reset_peak_stats();
// ... section to measure ...
HaHostMemStats stats{};
stats.struct_size = sizeof(stats);
ha_drv_host_mem_get_stats(&stats);    // peak usage during the section is stats.peak_allocated_bytes

ha_drv_host_mem_trim

Return to the OS, where possible, the freed memory the manager keeps for reuse: a pool segment that still holds a live allocation is kept whole. Only freed memory is touched, so a buffer an in-flight copy still uses, not yet freed by its owner, is unaffected.

Returns — HaSuccess.

Signature · Example
1
HaError ha_drv_host_mem_trim(void);

Example

1
2
ha_drv_host_mem_free(host);
ha_drv_host_mem_trim();               // return cached freed memory to the OS (reserved_bytes decreases)