Python Memory API¶
| Function | Signature | Notes |
|---|---|---|
mem_alloc |
(size: int, is_interleaved: bool = False) -> int |
Returns a device address |
mem_alloc_3d |
(shape: HaShape, is_interleaved: bool = False) -> int |
|
mem_alloc_async |
(size: int, is_interleaved: bool = False, stream: int = HA_STREAM_DEFAULT) -> int |
Synchronize before accessing from the host |
mem_free |
(ptr: int \| None) -> None |
None is a no-op |
mem_free_async |
(ptr: int, stream: int = HA_STREAM_DEFAULT) -> None |
Do not use ptr after the call |
mem_cpy |
(dst: int \| bytes, src: int \| bytes, size: int, direction: HaMemCpyDirection) -> None |
Blocks until complete |
mem_cpy_async |
(dst: int \| bytes, src: int \| bytes, size: int, direction: HaMemCpyDirection, stream: int = HA_STREAM_DEFAULT) -> None |
Only enqueues |
mem_cpy_nd |
(dst: int, dst_layout: HaLayout, src: int, src_layout: HaLayout) -> None |
Direction is inferred from the layout scope |
mem_cpy_nd_async |
(dst: int, dst_layout: HaLayout, src: int, src_layout: HaLayout, stream: int = HA_STREAM_DEFAULT) -> None |
Only enqueues. Host operands must stay alive until stream_synchronize(), and the device of stream must own the device operands |
mem_load_file_async |
(file_fd: int, file_off: int, dst: int, len: int, stream: int = HA_STREAM_DEFAULT) -> None |
Reads directly from a file descriptor into device memory |
dump_for_each_channel |
(start: int, bytes_total: int) -> list[bytes] |
One buffer per DRAM channel, not de-interleaved |
host_mem_alloc |
(size: int) -> int |
Allocates from the host memory manager. Returns a 64-byte-aligned address as int. No device required |
host_mem_free |
(ptr: int) -> None |
Frees a host_mem_alloc allocation (kept in the pool for reuse). 0 is a no-op |
host_mem_get_stats |
() -> dict[str, int] |
Process host memory statistics. Keys are allocated_bytes, peak_allocated_bytes, reserved_bytes, peak_reserved_bytes, alloc_count, free_count |
host_mem_reset_peak_stats |
() -> None |
Resets the two peak values in the statistics to the current values |
host_mem_trim |
() -> None |
Returns freed host memory held in the pool to the OS |
device_mem_get_stats |
() -> dict[str, int] |
Memory statistics of the current device. Keys are the field names of HaDeviceMemStats (excluding struct_size and reserved0) |
device_mem_reset_peak_stats |
() -> None |
Resets all peak values in the current device statistics to the current values |
device_mem_trim |
() -> None |
Returns segments with no allocated blocks in the current device pool to the driver |
build_device_tensor_layout |
(kind: HaTensorKind, dtype: HaDataType, dims: list[int], strides: list[int] = []) -> HaLayout |
Requires set_device() first |
Pass host buffers as bytes or as an object that implements the buffer protocol; the binding extracts the raw
pointer.
host_mem_alloc() raises RuntimeError if the size is 0, if the HA_HOST_MEM_MAX_BYTES limit is exceeded
(HaHostMemLimitExceeded), or if the system rejects. Passing host_mem_free() an address that is not a live
host_mem_alloc() allocation also raises RuntimeError. For the corresponding C++ API, see Memory API.
The keys of device_mem_get_stats() are total_bytes, free_bytes, releasable_bytes, allocated_bytes,
peak_allocated_bytes, reserved_bytes, peak_reserved_bytes, requested_bytes, peak_requested_bytes,
cached_free_bytes, active_blocks, peak_active_blocks, segments, peak_segments, device_allocs,
device_frees. Byte values are summed over all DRAM channels, and free_bytes is the free DRAM of the whole device as
reported by the driver. With HA_DEVICE_MEM_STATS_PRINT=1, the statistics are also printed to standard error. Memory freed
asynchronously (mem_free_async()) returns to the pool, and becomes eligible for release, only after its stream is
synchronized, so synchronize before calling device_mem_trim(). For the corresponding C++ API, see
Memory API.
Example¶
build_device_tensor_layout¶
The meaning of dims depends on the tensor kind. A KV cache key has exactly 5 dimensions.
Example
1 2 3 4 5 6 7 8 9 | |
dump_for_each_channel¶
Raw bytes per DRAM channel. For debugging, not a data path.
Example
1 2 3 4 | |
mem_alloc¶
Returns a device address as an int. It is not a host pointer.
Example
1 2 3 4 5 6 7 | |
mem_alloc_3d¶
A tensor allocation sized by shape rather than by byte count.
Example
1 2 3 | |
mem_alloc_async¶
Usable for enqueuing right away, but the host cannot access it until the stream reaches that point.
Example
1 2 3 4 5 6 7 | |
mem_cpy¶
Each side is either a device address or bytes; direction determines which side is the host.
Example
1 2 3 4 5 6 7 | |
mem_cpy_async¶
Returning does not mean the copy has started.
Example
1 2 3 4 | |
mem_cpy_nd¶
The direction comes from the layout scope, not from an argument.
The two layouts must have the same rank and shape -- this API does not broadcast. Strides are in elements, not bytes; a wrong stride reads the wrong address without raising.
Example
1 2 3 4 5 6 7 | |
mem_free¶
The block returns to the runtime pool and is reused by later allocations. A segment emptied by a free may be kept in
the pool depending on conditions, so freeing alone may not lower device memory usage. Return the kept empty segments
with device_mem_trim(). For the conditions, see Memory Model.
Example
1 2 3 4 | |
mem_free_async¶
The free is deferred, but the pointer becomes invalid as soon as the call returns.
Whether or not you synchronize, scratch cannot be used after this call.
Example
1 2 3 | |
mem_load_file_async¶
Reads directly from a file descriptor into device memory without going through a host buffer.
Example
1 2 3 4 5 6 7 8 | |