Memory API¶
Device allocation, host/device transfers, tensor layout construction, device memory statistics, and host memory from the process's host memory manager. The concepts behind these entry points are in Memory Model.
Contents¶
| API | 설명 |
|---|---|
HaBuildDeviceTensorLayout |
현재 디바이스에서 지정한 kind에 맞는 표준 HaLayout 생성 |
HaDeviceMemGetStats |
현재 디바이스의 메모리 통계 조회 (런타임 풀 카운터와 드라이버의 여유 DRAM) |
HaDeviceMemResetPeakStats |
현재 디바이스 메모리 통계의 최대값(Peak) 초기화 |
HaDeviceMemTrim |
현재 디바이스 풀에서 할당된 블록이 없는 메모리를 드라이버에 반환 |
HaHostMemAlloc |
프로세스 호스트 메모리 관리자에서 호스트 메모리 할당 |
HaHostMemFree |
HaHostMemAlloc() 할당분 해제 (재사용을 위해 풀에 보관) |
HaHostMemGetStats |
프로세스 호스트 메모리 통계 조회 |
HaHostMemResetPeakStats |
호스트 메모리 통계의 최대값(Peak) 초기화 |
HaHostMemTrim |
풀에 보관 중인 해제된 호스트 메모리를 OS에 반환 |
HaMemAlloc |
디바이스 메모리 할당 (동기) |
HaMemAlloc3D |
3D 논리 셰이프(Shape)를 기준으로 디바이스 메모리 할당 |
HaMemAllocAsync |
스트림에서 디바이스 메모리 비동기 할당 |
HaMemCpy |
호스트와 디바이스 간, 또는 디바이스 간 메모리 복사 |
HaMemCpyAsync |
스트림에서 메모리 비동기 복사 |
HaMemCpyNd |
두 개의 N-D 레이아웃 기술자를 기반으로 한 다차원 메모리 복사 |
HaMemCpyNdAsync |
스트림에서 N-D 레이아웃 기반 다차원 메모리 비동기 복사 |
HaMemDumpForEachChannel |
DRAM 채널별로 디바이스 할당 영역의 원시 바이트(Raw Byte) 덤프 |
HaMemFree |
HaMemAlloc() 또는 HaMemAlloc3D()로 할당한 디바이스 메모리 해제 |
HaMemFreeAsync |
스트림에서 HaMemAllocAsync() 할당분 비동기 해제 |
HaMemLoadFileAsync |
스트림에서 파일을 디바이스 메모리로 비동기 로드 |
HaBuildDeviceTensorLayout¶
Builds the canonical HaLayout for a tensor of kind on the current device.
| Parameter | Direction | Default | Description |
|---|---|---|---|
out |
out | — | Receives the populated layout (required). See HaLayout. |
kind |
in | — | Which tensor's layout rule to apply. One of KvCacheKey, KvCacheValue, SingleChannel, RowInterleaved, MaxRowsInterleaved, DramWordInterleaved, MPUTensor, MpuTensorTransposed, SinglePassInterleaved (see HaTensorKind). |
dtype |
in | — | Element type of the resulting layout. See HaDataType. |
dims |
in | — | Logical dimensions (kind-specific length / order; see the per-kind rules below). |
strides |
in | nullptr |
Optional logical strides, in elements; nullptr means row-major-contiguous (default). |
ndim |
in | — | Caller-provided logical rank — the valid length of dims (and strides when non-null). See the per-kind rules below. |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 3 4 | |
Note
Requires a current device (HaSetDevice()) so the runtime can resolve the channel configuration; otherwise returns HaDeviceInvalidArgument or HaDeviceNotInitialized.
Note
Per-kind logical dim conventions (see HaTensorKind): KvCacheKey / KvCacheValue: logical dims describe [num_blocks, num_layers, num_kv_heads, block_size, head_dim]. head_dim must be divisible by channels_per_token * elements_per_word (else HaMemoryInvalidArgument). SingleChannel: any ndim >= 1. The whole product(dims) lives contiguously on channel 0; the input shape is preserved inside the channel reservation (output storage shape = (N, dims_in...)). RowInterleaved / MaxRowsInterleaved: 2+ dims; the last dim is w and all leading dims multiply into the effective row count. MPUTensor: 2+ dims. The last dim is head_dim, the second-to-last dim is block_size per block, and any leading dims multiply into the batching count (num_blocks). head_dim must be divisible by channels_per_token * elements_per_word. Output rank is 6 for ndim == 2 (Value-pattern inner core only) and 7 for ndim >= 3 ((batching, 6-D inner core)). All *Interleaved kinds share a granularity constraint: the per-channel chunk_size must be a whole multiple of dram_word_size_bytes (e.g. 64 B on BerthaCloud, 32 B on BerthaEdge). The kernel rejects layouts that violate this. The per-kind formulas below derive their chunk_size and inherit this check. DramWordInterleaved: any ndim >= 1; chunk_size = dram_word_size_bytes (trivially satisfies the granularity constraint). Layout depends only on total = product(dims) * dtype_bits. Caller may pass their natural contiguous logical shape without flattening. RowInterleaved / MaxRowsInterleaved: chunk_size derived from W (last dim) and dtype; caller must ensure W*dtype_bits/8 (Row) or M*C*W*dtype_bits/8 (MaxRows) is a dram_word_size_bytes multiple. SinglePassInterleaved: any ndim >= 1; chunk_size = total/N per channel. Stricter requirement: total * dtype_bits must be a multiple of num_dram_channels * dram_word_size_bytes * 8 (DRAM-word divisibility per channel — granularity constraint, stronger than the byte-level check). Returns rank-2 storage (num_dram_channels, total / num_dram_channels).
HaDeviceMemGetStats¶
Reads the current device's memory statistics: the runtime's pool counters, and the driver's own figure for the free DRAM of the whole device (every process, memory outside the pools included). See HaDeviceMemStats.
| Parameter | Direction | Default | Description |
|---|---|---|---|
stats |
in, out | — | Statistics to fill; the caller sets struct_size to sizeof(HaDeviceMemStats). See HaDeviceMemStats. |
Returns — HaSuccess; HaInvalidArgument for a null stats or a struct_size that is too small; HaDeviceInvalidArgument without a current device; HaDeviceNotReady while that device is being reset.
시그니처 · 예제
1 | |
예제
1 2 3 4 5 6 7 8 9 | |
Note
Byte figures are the sizes occupied in all DRAM channels of the device, summed, so a contiguous allocation is counted as its requested size multiplied by the number of channels. The exception is requested_bytes and its peak, which are the sizes the callers asked for and are not multiplied by the channel count. With HA_DEVICE_MEM_STATS_PRINT=1 the statistics are also printed to standard error at each call (Environment Variables).
HaDeviceMemResetPeakStats¶
Restarts every peak of the current device's HaDeviceMemStats from its current value.
Returns — HaSuccess.
시그니처
1 | |
HaDeviceMemTrim¶
Returns to the driver the pool memory of the current device that holds no allocated blocks, so that HaDeviceMemStats::free_bytes grows by about HaDeviceMemStats::releasable_bytes. A pool segment that still holds a live allocation is kept whole.
Returns — HaSuccess.
시그니처 · 예제
1 | |
예제
1 2 3 | |
Note
Memory freed with HaMemFreeAsync() returns to the pool only when its stream is synchronized: synchronize first for it to count.
HaHostMemAlloc¶
Allocates host memory from the process's host memory manager, which pools freed memory for reuse.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
out | — | Receives the address. |
size |
in | — | Bytes, non-zero. |
Returns — HaSuccess; HaInvalidArgument for a null ptr or a zero size; HaHostMemLimitExceeded if the allocation would exceed HA_HOST_MEM_MAX_BYTES; HaMemoryAllocationFailed if the system cannot allocate it; HaDeviceNotReady while the runtime is shutting down.
시그니처 · 예제
1 | |
예제
1 2 3 4 5 | |
Note
Needs no device. Like every Ha* call, it brings the runtime up on first use (a process that needs host memory without bringing up the runtime API uses the driver API's ha_drv_host_mem_alloc()). The memory is 64-byte aligned and its contents are undefined. Freed memory stays cached for reuse until HaHostMemTrim() returns it to the OS.
Note
A buffer handed to an asynchronous copy must not be freed until that copy completes.
HaHostMemFree¶
Frees host memory from HaHostMemAlloc(); it stays cached for reuse.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
in | — | Address HaHostMemAlloc() returned; nullptr is a no-op, as free(NULL) is. |
Returns — HaSuccess, or HaMemoryInvalidArgument if ptr is not nullptr or a live HaHostMemAlloc() allocation.
시그니처
1 | |
HaHostMemGetStats¶
Reads the process's host memory statistics: every host allocation of the process, the runtime's own staging included. See HaHostMemStats.
| Parameter | Direction | Default | Description |
|---|---|---|---|
stats |
in, out | — | Statistics to fill; the caller sets struct_size to sizeof(HaHostMemStats). See HaHostMemStats. |
Returns — HaSuccess, or HaInvalidArgument for a null stats or a struct_size that is too small.
시그니처 · 예제
1 | |
예제
1 2 3 4 5 6 | |
HaHostMemResetPeakStats¶
Restarts both peaks of HaHostMemStats (peak_allocated_bytes and peak_reserved_bytes) from their current values.
Returns — HaSuccess.
시그니처
1 | |
HaHostMemTrim¶
Returns to the OS, as far as possible, the freed host memory that the manager keeps for reuse (a pool segment that still holds a live allocation is kept). Memory not yet freed, such as a buffer an in-flight copy uses, is unaffected.
Returns — HaSuccess.
시그니처
1 | |
HaMemAlloc¶
Allocates device memory.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
out | — | Output pointer to receive the allocated device address (required). |
size |
in | — | Number of bytes to allocate. |
is_interleaved |
in | — | Whether to use interleaved layout (meaning is platform-dependent). |
Returns — Success or an error code (e.g., out-of-memory).
시그니처 · 예제
1 | |
예제
1 2 3 4 | |
Note
The returned pointer can be used with HaMemCpy() and passed as a kernel argument.
Note
Interleaved spreads the allocation across DRAM channels; contiguous keeps it on one. Which is right depends on how the consuming kernel addresses the operand, so match the kernel rather than defaulting to either.
HaMemAlloc3D¶
Allocates device memory using a 3D logical shape.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
out | — | Output pointer to receive the allocated device address (required). |
shape |
in | — | Shape describing the allocation. See HaShape. |
is_interleaved |
in | — | Whether to use interleaved layout (platform-dependent). |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 | |
Note
Convenience variant for tensor-like allocations. The interpretation of HaShape follows the runtime layout contract.
Note
For operands whose physical layout the hardware fixes (KV-cache blocks, MPU tensors, row-interleaved matrices), do not size the allocation by hand; build the layout with HaBuildDeviceTensorLayout() instead.
HaMemAllocAsync¶
Asynchronously allocates device memory on a stream.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
out | — | Output pointer to receive the allocated device address (required). |
size |
in | — | Number of bytes to allocate. |
is_interleaved |
in | — | Whether to use interleaved layout (platform-dependent). |
stream |
in | HA_STREAM_DEFAULT |
Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device. |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 3 | |
Note
The allocation is enqueued into the stream and may not be available until the stream reaches that point. Use HaStreamSynchronize() before accessing the allocated memory from the host. Free with HaMemFreeAsync() on the same stream, or with HaMemFree() after synchronization.
Note
The pointer can be passed to later operations on the stream immediately, but must not be accessed from the host until the stream reaches that point.
HaMemCpy¶
Copies memory between host and device, or between devices.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dst |
in | — | Destination address (host pointer or device address). |
src |
in | — | Source address (host or device). |
size |
in | — | Number of bytes to copy. |
direction |
in | — | Copy direction. One of HostToDevice, DeviceToHost, DeviceToDevice (see HaMemCpyDirection). |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 3 | |
Note
Typical usage is uploading inputs (HostToDevice) or downloading outputs (DeviceToHost). Both operands are HaDevicePtr, including the host one, so the direction argument is what says which side is which; passing the wrong direction is not a type error.
Note
Stream-ordered on the calling thread's default stream on the device that owns the device address, whatever the current device: the copy is queued there and this call waits for it, so whatever was already queued there runs first. A stream from HaStreamCreate() is a separate queue; synchronize it, or wait on an event recorded on it, before a copy that depends on its result.
Note
The returned error is not only this copy's. A failure of an earlier asynchronous op on that stream is sticky and surfaces here, and a per-thread default stream is never destroyed, so only HaResetDevice() clears it.
HaMemCpyAsync¶
Asynchronously copies memory on a stream.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dst |
in | — | Destination address (host pointer or device address). |
src |
in | — | Source address (host or device). |
size |
in | — | Number of bytes to copy. |
direction |
in | — | Copy direction. One of HostToDevice, DeviceToHost, DeviceToDevice (see HaMemCpyDirection). |
stream |
in | HA_STREAM_DEFAULT |
Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device. Its device must own the device address: the copy executes there, so a stream on another device is rejected with HaMemCpyNotSupported. |
Returns — HaSuccess if enqueued successfully; the copy may not be complete on return.
시그니처 · 예제
1 | |
예제
1 2 3 | |
Note
Use HaStreamSynchronize() or HaEventSynchronize() to confirm completion.
Note
Returning does not mean the copy started. Do not touch either host buffer before a synchronization point; bugs of this kind depend on timing and go unnoticed in most runs.
Note
Cross-device device-to-device: synchronize stream before resetting either device. The copy is ordered against a reset of stream's device only; a reset of the other device neither waits for it nor cancels it, so it can then read or write that device's memory after the reset.
HaMemCpyNd¶
Copies memory between host and device, or between devices, described by two N-D layout descriptors.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dst |
in | — | Destination buffer pointer (host pointer or device pointer per dst_layout->scope). |
dst_layout |
in | — | Destination layout (non-null). See HaLayout. |
src |
in | — | Source buffer pointer (host pointer or device pointer per src_layout->scope). |
src_layout |
in | — | Source layout (non-null). See HaLayout. |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 3 4 | |
Note
dst_layout and src_layout must share the same rank and shape. If a caller starts with mismatched ranks, re-view one side first (e.g. as_strided) — this API does not broadcast.
Note
Strides are in elements of dtype, not bytes. Getting that wrong is not rejected; it reads the wrong addresses.
Note
The direction is inferred from (dst_layout->scope, src_layout->scope): host-to-device, device-to-host, device-to-device, or host-to-host. Host-to-host runs entirely on the host (a parallel memcpy walk) with no device transfer.
Note
Stream-ordered on the calling thread's default stream on the device that owns the device address, whatever the current device, as with HaMemCpy(). The returned error can be a sticky failure of an earlier asynchronous op on that stream; only HaResetDevice() clears it.
HaMemCpyNdAsync¶
Asynchronously copies memory described by two N-D layout descriptors, on a stream. Plans the copy as HaMemCpyNd() does, but enqueues each resulting transfer on stream instead of issuing it immediately, so the call returns without draining the stream first.
| Parameter | Direction | Default | Description |
|---|---|---|---|
dst |
in | — | Destination buffer pointer (host pointer or device pointer per dst_layout->scope). |
dst_layout |
in | — | Destination layout (non-null). See HaLayout. |
src |
in | — | Source buffer pointer (host pointer or device pointer per src_layout->scope). |
src_layout |
in | — | Source layout (non-null). See HaLayout. |
stream |
in | HA_STREAM_DEFAULT |
Stream to enqueue into; omit or pass HA_STREAM_DEFAULT for the calling thread's default stream on the current device. Its device must own a device operand: the copy executes there, so a stream on another device is rejected with HaMemCpyNotSupported. |
Returns — HaSuccess if enqueued successfully; the copy may not be complete on return.
시그니처 · 예제
1 | |
예제
1 2 | |
Note
Source lifetime. A host source must stay valid and unmodified until the copy completes (HaStreamSynchronize() or HaEventSynchronize()), as with HaMemCpyAsync(). Whether a call stages its source is not reported, so never release or reuse the source before the copy completes.
Note
Channel-interleaved device-to-host. The reorder into the caller's logical order runs only after the device has filled its staging buffer, so the destination is filled only at HaStreamSynchronize(stream) or HaDeviceSynchronize(); HaEventSynchronize() or synchronizing another stream is not enough.
Note
Host-to-host is rejected (it cannot be ordered against stream); use HaMemCpyNd(). For a cross-device device-to-device copy, synchronize stream before resetting either device: the copy is ordered against a reset of stream's device only.
HaMemDumpForEachChannel¶
Dumps the raw bytes backing a device allocation, one buffer per DRAM channel.
| Parameter | Direction | Default | Description |
|---|---|---|---|
start |
in | — | Channel-0 device address. |
bytes_total |
in | — | Logical byte count to dump (must be > 0). |
out |
out | — | Receives num_channels buffers (required). |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 | |
Note
Pre-condition: a current device is set; start is a channel-0 address from HaMemAlloc().
Note
A debugging aid for checking that a layout landed where it was meant to. It transfers the whole range synchronously and does not de-interleave, so it is not a data path.
HaMemFree¶
Frees device memory previously allocated by HaMemAlloc()/HaMemAlloc3D().
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
in | — | Device pointer to free. |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 | |
Note
Freed memory goes back to the runtime pool for reuse. When a free leaves a segment empty, the segment is returned to the driver right away if other allocations remain and the free memory the pool holds exceeds half of the allocated amount; otherwise, for example after every allocation is freed, it is kept. Return kept segments with HaDeviceMemTrim().
HaMemFreeAsync¶
Asynchronously frees device memory previously allocated by HaMemAllocAsync() on a stream.
| Parameter | Direction | Default | Description |
|---|---|---|---|
ptr |
in | — | Device pointer to free (must have been allocated by HaMemAllocAsync()). |
stream |
in | HA_STREAM_DEFAULT |
Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device. |
Returns — Success or an error code.
시그니처 · 예제
1 | |
예제
1 2 3 | |
Note
The free is enqueued into the stream and deferred until all preceding stream operations complete. Do not access ptr after this call, even if the stream has not yet been synchronized.
HaMemLoadFileAsync¶
Asynchronously loads a file into device memory on a stream.
| Parameter | Direction | Default | Description |
|---|---|---|---|
file_fd |
in | — | Open file descriptor to read from. |
file_off |
in | — | Byte offset within the file to start reading. |
dst |
in | — | Destination device address. |
len |
in | — | Number of bytes to load. |
stream |
in | HA_STREAM_DEFAULT |
Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device. |
Returns — HaSuccess if enqueued successfully; the load may not be complete on return.
시그니처 · 예제
1 | |
예제
1 2 3 | |
Note
Use HaStreamSynchronize() or HaEventSynchronize() to confirm completion.