Skip to content

Python Memory API

Function Signature Notes
mem_alloc (size: int, is_interleaved: bool = False) -> int Returns a device address
mem_alloc_3d (shape: HaShape, is_interleaved: bool = False) -> int
mem_alloc_async (size: int, is_interleaved: bool = False, stream: int = HA_STREAM_DEFAULT) -> int Synchronize before accessing from the host
mem_free (ptr: int \| None) -> None None is a no-op
mem_free_async (ptr: int, stream: int = HA_STREAM_DEFAULT) -> None Do not use ptr after the call
mem_cpy (dst: int \| bytes, src: int \| bytes, size: int, direction: HaMemCpyDirection) -> None Blocks until complete
mem_cpy_async (dst: int \| bytes, src: int \| bytes, size: int, direction: HaMemCpyDirection, stream: int = HA_STREAM_DEFAULT) -> None Only enqueues
mem_cpy_nd (dst: int, dst_layout: HaLayout, src: int, src_layout: HaLayout) -> None Direction is inferred from the layout scope
mem_cpy_nd_async (dst: int, dst_layout: HaLayout, src: int, src_layout: HaLayout, stream: int = HA_STREAM_DEFAULT) -> None Only enqueues. Host operands must stay alive until stream_synchronize(), and the device of stream must own the device operands
mem_load_file_async (file_fd: int, file_off: int, dst: int, len: int, stream: int = HA_STREAM_DEFAULT) -> None Reads directly from a file descriptor into device memory
dump_for_each_channel (start: int, bytes_total: int) -> list[bytes] One buffer per DRAM channel, not de-interleaved
host_mem_alloc (size: int) -> int Allocates from the host memory manager. Returns a 64-byte-aligned address as int. No device required
host_mem_free (ptr: int) -> None Frees a host_mem_alloc allocation (kept in the pool for reuse). 0 is a no-op
host_mem_get_stats () -> dict[str, int] Process host memory statistics. Keys are allocated_bytes, peak_allocated_bytes, reserved_bytes, peak_reserved_bytes, alloc_count, free_count
host_mem_reset_peak_stats () -> None Resets the two peak values in the statistics to the current values
host_mem_trim () -> None Returns freed host memory held in the pool to the OS
device_mem_get_stats () -> dict[str, int] Memory statistics of the current device. Keys are the field names of HaDeviceMemStats (excluding struct_size and reserved0)
device_mem_reset_peak_stats () -> None Resets all peak values in the current device statistics to the current values
device_mem_trim () -> None Returns segments with no allocated blocks in the current device pool to the driver
build_device_tensor_layout (kind: HaTensorKind, dtype: HaDataType, dims: list[int], strides: list[int] = []) -> HaLayout Requires set_device() first

Pass host buffers as bytes or as an object that implements the buffer protocol; the binding extracts the raw pointer.

host_mem_alloc() raises RuntimeError if the size is 0, if the HA_HOST_MEM_MAX_BYTES limit is exceeded (HaHostMemLimitExceeded), or if the system rejects. Passing host_mem_free() an address that is not a live host_mem_alloc() allocation also raises RuntimeError. For the corresponding C++ API, see Memory API.

The keys of device_mem_get_stats() are total_bytes, free_bytes, releasable_bytes, allocated_bytes, peak_allocated_bytes, reserved_bytes, peak_reserved_bytes, requested_bytes, peak_requested_bytes, cached_free_bytes, active_blocks, peak_active_blocks, segments, peak_segments, device_allocs, device_frees. Byte values are summed over all DRAM channels, and free_bytes is the free DRAM of the whole device as reported by the driver. With HA_DEVICE_MEM_STATS_PRINT=1, the statistics are also printed to standard error. Memory freed asynchronously (mem_free_async()) returns to the pool, and becomes eligible for release, only after its stream is synchronized, so synchronize before calling device_mem_trim(). For the corresponding C++ API, see Memory API.


Example

build_device_tensor_layout

The meaning of dims depends on the tensor kind. A KV cache key has exactly 5 dimensions.

Example
1
2
3
4
5
6
7
8
9
import hart

hart.set_device(0)  # platform constants are read from the current device

layout = hart.build_device_tensor_layout(
    hart.HaTensorKind.KV_CACHE_KEY,
    hart.HaDataType.BFLOAT16,
    [num_blocks, num_layers, num_kv_heads, block_size, head_dim],
)

dump_for_each_channel

Raw bytes per DRAM channel. For debugging, not a data path.

Example
1
2
3
4
import hart

channels = hart.dump_for_each_channel(start, bytes_total)
print([len(c) for c in channels])  # transfers everything synchronously -- not a data path

mem_alloc

Returns a device address as an int. It is not a host pointer.

Example
1
2
3
4
5
6
7
import hart

hart.set_device(0)
buf = hart.mem_alloc(1024)                       # one DRAM channel
spread = hart.mem_alloc(1024, is_interleaved=True)  # spread across channels

hart.mem_free(buf)

mem_alloc_3d

A tensor allocation sized by shape rather than by byte count.

Example
1
2
3
import hart

tensor = hart.mem_alloc_3d(hart.HaShape(width, height, channels), is_interleaved=True)

mem_alloc_async

Usable for enqueuing right away, but the host cannot access it until the stream reaches that point.

Example
1
2
3
4
5
6
7
import hart

scratch = hart.mem_alloc_async(size, False, stream)
args = hart.HaLaunchArgs()
args.add_tensor(hart.HaLaunchArgRoute.UNIVERSAL, -1, scratch, [size])
hart.launch_kernel_async_with_binary_path(kernel_path, args, stream)
hart.stream_synchronize(stream)  # the host can read it from here on

mem_cpy

Each side is either a device address or bytes; direction determines which side is the host.

Example
1
2
3
4
5
6
7
import hart

payload = b"\x00" * 1024
hart.mem_cpy(buf, payload, len(payload), hart.HaMemCpyDirection.HostToDevice)

out = bytearray(1024)
hart.mem_cpy(out, buf, len(out), hart.HaMemCpyDirection.DeviceToHost)

mem_cpy_async

Returning does not mean the copy has started.

Example
1
2
3
4
import hart

hart.mem_cpy_async(buf, payload, len(payload), hart.HaMemCpyDirection.HostToDevice, stream)
hart.stream_synchronize(stream)  # do not touch payload before this

mem_cpy_nd

The direction comes from the layout scope, not from an argument.

The two layouts must have the same rank and shape -- this API does not broadcast. Strides are in elements, not bytes; a wrong stride reads the wrong address without raising.

Example
1
2
3
4
5
6
7
import hart

bf16 = hart.HaDataType.BFLOAT16
dev_layout = hart.HaLayout([rows, cols], scope=hart.HaMemScope.DEVICE, dtype=bf16)
# host buffer in column-major order; strides are in elements
host_layout = hart.HaLayout([rows, cols], strides=[1, rows], scope=hart.HaMemScope.HOST, dtype=bf16)
hart.mem_cpy_nd(dev_ptr, dev_layout, host_ptr, host_layout)  # host -> device, from the scopes

mem_free

The block returns to the runtime pool and is reused by later allocations. A segment emptied by a free may be kept in the pool depending on conditions, so freeing alone may not lower device memory usage. Return the kept empty segments with device_mem_trim(). For the conditions, see Memory Model.

Example
1
2
3
4
import hart

hart.mem_free(buf)
hart.mem_free(None)  # a no-op, so cleanup paths need no guard

mem_free_async

The free is deferred, but the pointer becomes invalid as soon as the call returns.

Whether or not you synchronize, scratch cannot be used after this call.

Example
1
2
3
import hart

hart.mem_free_async(scratch, stream)

mem_load_file_async

Reads directly from a file descriptor into device memory without going through a host buffer.

Example
1
2
3
4
5
6
7
8
import os

import hart

fd = os.open(weights_path, os.O_RDONLY)
hart.mem_load_file_async(fd, file_offset, dev_weights, length, stream)
hart.stream_synchronize(stream)
os.close(fd)

See also