콘텐츠로 이동

Memory API

Device allocation, host/device transfers, tensor layout construction, device memory statistics, and host memory from the process's host memory manager. The concepts behind these entry points are in Memory Model.

Contents

API 설명
HaBuildDeviceTensorLayout 현재 디바이스에서 지정한 kind에 맞는 표준 HaLayout 생성
HaDeviceMemGetStats 현재 디바이스의 메모리 통계 조회 (런타임 풀 카운터와 드라이버의 여유 DRAM)
HaDeviceMemResetPeakStats 현재 디바이스 메모리 통계의 최대값(Peak) 초기화
HaDeviceMemTrim 현재 디바이스 풀에서 할당된 블록이 없는 메모리를 드라이버에 반환
HaHostMemAlloc 프로세스 호스트 메모리 관리자에서 호스트 메모리 할당
HaHostMemFree HaHostMemAlloc() 할당분 해제 (재사용을 위해 풀에 보관)
HaHostMemGetStats 프로세스 호스트 메모리 통계 조회
HaHostMemResetPeakStats 호스트 메모리 통계의 최대값(Peak) 초기화
HaHostMemTrim 풀에 보관 중인 해제된 호스트 메모리를 OS에 반환
HaMemAlloc 디바이스 메모리 할당 (동기)
HaMemAlloc3D 3D 논리 셰이프(Shape)를 기준으로 디바이스 메모리 할당
HaMemAllocAsync 스트림에서 디바이스 메모리 비동기 할당
HaMemCpy 호스트와 디바이스 간, 또는 디바이스 간 메모리 복사
HaMemCpyAsync 스트림에서 메모리 비동기 복사
HaMemCpyNd 두 개의 N-D 레이아웃 기술자를 기반으로 한 다차원 메모리 복사
HaMemCpyNdAsync 스트림에서 N-D 레이아웃 기반 다차원 메모리 비동기 복사
HaMemDumpForEachChannel DRAM 채널별로 디바이스 할당 영역의 원시 바이트(Raw Byte) 덤프
HaMemFree HaMemAlloc() 또는 HaMemAlloc3D()로 할당한 디바이스 메모리 해제
HaMemFreeAsync 스트림에서 HaMemAllocAsync() 할당분 비동기 해제
HaMemLoadFileAsync 스트림에서 파일을 디바이스 메모리로 비동기 로드

HaBuildDeviceTensorLayout

Builds the canonical HaLayout for a tensor of kind on the current device.

Parameter Direction Default Description
out out — Receives the populated layout (required). See HaLayout.
kind in — Which tensor's layout rule to apply. One of KvCacheKey, KvCacheValue, SingleChannel, RowInterleaved, MaxRowsInterleaved, DramWordInterleaved, MPUTensor, MpuTensorTransposed, SinglePassInterleaved (see HaTensorKind).
dtype in — Element type of the resulting layout. See HaDataType.
dims in — Logical dimensions (kind-specific length / order; see the per-kind rules below).
strides in nullptr Optional logical strides, in elements; nullptr means row-major-contiguous (default).
ndim in — Caller-provided logical rank — the valid length of dims (and strides when non-null). See the per-kind rules below.

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaBuildDeviceTensorLayout(HaLayout *out, HaTensorKind kind, HaDataType dtype, uint32_t ndim, const uint64_t *dims, const uint64_t *strides=nullptr);

예제

1
2
3
4
HaLayout layout;
uint64_t dims[] = {num_blocks, num_layers, num_kv_heads, block_size, head_dim};
HaSetDevice(0);   // platform constants are read from the current device
HaBuildDeviceTensorLayout(&layout, HaTensorKind::KvCacheKey, HaDataType::BFloat16, 5, dims);

Note

Requires a current device (HaSetDevice()) so the runtime can resolve the channel configuration; otherwise returns HaDeviceInvalidArgument or HaDeviceNotInitialized.

Note

Per-kind logical dim conventions (see HaTensorKind): KvCacheKey / KvCacheValue: logical dims describe [num_blocks, num_layers, num_kv_heads, block_size, head_dim]. head_dim must be divisible by channels_per_token * elements_per_word (else HaMemoryInvalidArgument). SingleChannel: any ndim >= 1. The whole product(dims) lives contiguously on channel 0; the input shape is preserved inside the channel reservation (output storage shape = (N, dims_in...)). RowInterleaved / MaxRowsInterleaved: 2+ dims; the last dim is w and all leading dims multiply into the effective row count. MPUTensor: 2+ dims. The last dim is head_dim, the second-to-last dim is block_size per block, and any leading dims multiply into the batching count (num_blocks). head_dim must be divisible by channels_per_token * elements_per_word. Output rank is 6 for ndim == 2 (Value-pattern inner core only) and 7 for ndim >= 3 ((batching, 6-D inner core)). All *Interleaved kinds share a granularity constraint: the per-channel chunk_size must be a whole multiple of dram_word_size_bytes (e.g. 64 B on BerthaCloud, 32 B on BerthaEdge). The kernel rejects layouts that violate this. The per-kind formulas below derive their chunk_size and inherit this check. DramWordInterleaved: any ndim >= 1; chunk_size = dram_word_size_bytes (trivially satisfies the granularity constraint). Layout depends only on total = product(dims) * dtype_bits. Caller may pass their natural contiguous logical shape without flattening. RowInterleaved / MaxRowsInterleaved: chunk_size derived from W (last dim) and dtype; caller must ensure W*dtype_bits/8 (Row) or M*C*W*dtype_bits/8 (MaxRows) is a dram_word_size_bytes multiple. SinglePassInterleaved: any ndim >= 1; chunk_size = total/N per channel. Stricter requirement: total * dtype_bits must be a multiple of num_dram_channels * dram_word_size_bytes * 8 (DRAM-word divisibility per channel — granularity constraint, stronger than the byte-level check). Returns rank-2 storage (num_dram_channels, total / num_dram_channels).


HaDeviceMemGetStats

Reads the current device's memory statistics: the runtime's pool counters, and the driver's own figure for the free DRAM of the whole device (every process, memory outside the pools included). See HaDeviceMemStats.

Parameter Direction Default Description
stats in, out — Statistics to fill; the caller sets struct_size to sizeof(HaDeviceMemStats). See HaDeviceMemStats.

Returns — HaSuccess; HaInvalidArgument for a null stats or a struct_size that is too small; HaDeviceInvalidArgument without a current device; HaDeviceNotReady while that device is being reset.

시그니처 · 예제
1
HaError HaDeviceMemGetStats(HaDeviceMemStats *stats);

예제

1
2
3
4
5
6
7
8
9
HaSetDevice(0);
HaDeviceMemStats stats{};
stats.struct_size = sizeof(stats);
if (HaDeviceMemGetStats(&stats) == HaError::HaSuccess) {
  std::printf("device 0: %llu of %llu bytes free, %llu allocated by this process\n",
              static_cast<unsigned long long>(stats.free_bytes),
              static_cast<unsigned long long>(stats.total_bytes),
              static_cast<unsigned long long>(stats.allocated_bytes));
}

Note

Byte figures are the sizes occupied in all DRAM channels of the device, summed, so a contiguous allocation is counted as its requested size multiplied by the number of channels. The exception is requested_bytes and its peak, which are the sizes the callers asked for and are not multiplied by the channel count. With HA_DEVICE_MEM_STATS_PRINT=1 the statistics are also printed to standard error at each call (Environment Variables).


HaDeviceMemResetPeakStats

Restarts every peak of the current device's HaDeviceMemStats from its current value.

Returns — HaSuccess.

시그니처
1
HaError HaDeviceMemResetPeakStats();

HaDeviceMemTrim

Returns to the driver the pool memory of the current device that holds no allocated blocks, so that HaDeviceMemStats::free_bytes grows by about HaDeviceMemStats::releasable_bytes. A pool segment that still holds a live allocation is kept whole.

Returns — HaSuccess.

시그니처 · 예제
1
HaError HaDeviceMemTrim();

예제

1
2
3
HaMemFreeAsync(buf, stream);
HaStreamSynchronize(stream);   // the freed block returns to the pool only here
HaDeviceMemTrim();             // now its segment can go back to the driver

Note

Memory freed with HaMemFreeAsync() returns to the pool only when its stream is synchronized: synchronize first for it to count.


HaHostMemAlloc

Allocates host memory from the process's host memory manager, which pools freed memory for reuse.

Parameter Direction Default Description
ptr out — Receives the address.
size in — Bytes, non-zero.

Returns — HaSuccess; HaInvalidArgument for a null ptr or a zero size; HaHostMemLimitExceeded if the allocation would exceed HA_HOST_MEM_MAX_BYTES; HaMemoryAllocationFailed if the system cannot allocate it; HaDeviceNotReady while the runtime is shutting down.

시그니처 · 예제
1
HaError HaHostMemAlloc(void **ptr, size_t size);

예제

1
2
3
4
5
void *host = nullptr;
HaHostMemAlloc(&host, n);
HaMemCpyAsync(dev, reinterpret_cast<HaDevicePtr>(host), n, HaMemCpyDirection::HostToDevice, stream);
HaStreamSynchronize(stream);   // the copy must complete before the buffer is freed
HaHostMemFree(host);

Note

Needs no device. Like every Ha* call, it brings the runtime up on first use (a process that needs host memory without bringing up the runtime API uses the driver API's ha_drv_host_mem_alloc()). The memory is 64-byte aligned and its contents are undefined. Freed memory stays cached for reuse until HaHostMemTrim() returns it to the OS.

Note

A buffer handed to an asynchronous copy must not be freed until that copy completes.


HaHostMemFree

Frees host memory from HaHostMemAlloc(); it stays cached for reuse.

Parameter Direction Default Description
ptr in — Address HaHostMemAlloc() returned; nullptr is a no-op, as free(NULL) is.

Returns — HaSuccess, or HaMemoryInvalidArgument if ptr is not nullptr or a live HaHostMemAlloc() allocation.

시그니처
1
HaError HaHostMemFree(void *ptr);

HaHostMemGetStats

Reads the process's host memory statistics: every host allocation of the process, the runtime's own staging included. See HaHostMemStats.

Parameter Direction Default Description
stats in, out — Statistics to fill; the caller sets struct_size to sizeof(HaHostMemStats). See HaHostMemStats.

Returns — HaSuccess, or HaInvalidArgument for a null stats or a struct_size that is too small.

시그니처 · 예제
1
HaError HaHostMemGetStats(HaHostMemStats *stats);

예제

1
2
3
4
5
6
HaHostMemStats stats{};
stats.struct_size = sizeof(stats);
HaHostMemGetStats(&stats);
std::printf("host: %llu bytes allocated, %llu held\n",
            static_cast<unsigned long long>(stats.allocated_bytes),
            static_cast<unsigned long long>(stats.reserved_bytes));

HaHostMemResetPeakStats

Restarts both peaks of HaHostMemStats (peak_allocated_bytes and peak_reserved_bytes) from their current values.

Returns — HaSuccess.

시그니처
1
HaError HaHostMemResetPeakStats();

HaHostMemTrim

Returns to the OS, as far as possible, the freed host memory that the manager keeps for reuse (a pool segment that still holds a live allocation is kept). Memory not yet freed, such as a buffer an in-flight copy uses, is unaffected.

Returns — HaSuccess.

시그니처
1
HaError HaHostMemTrim();

HaMemAlloc

Allocates device memory.

Parameter Direction Default Description
ptr out — Output pointer to receive the allocated device address (required).
size in — Number of bytes to allocate.
is_interleaved in — Whether to use interleaved layout (meaning is platform-dependent).

Returns — Success or an error code (e.g., out-of-memory).

시그니처 · 예제
1
HaError HaMemAlloc(HaDevicePtr *ptr, const size_t size, const bool is_interleaved);

예제

1
2
3
4
HaDevicePtr buf = 0;
if (HaMemAlloc(&buf, 1024, /*is_interleaved=*/false) != HaError::HaSuccess) {
  return AllocationFailed();   // buf is untouched on failure -- do not read it
}

Note

The returned pointer can be used with HaMemCpy() and passed as a kernel argument.

Note

Interleaved spreads the allocation across DRAM channels; contiguous keeps it on one. Which is right depends on how the consuming kernel addresses the operand, so match the kernel rather than defaulting to either.


HaMemAlloc3D

Allocates device memory using a 3D logical shape.

Parameter Direction Default Description
ptr out — Output pointer to receive the allocated device address (required).
shape in — Shape describing the allocation. See HaShape.
is_interleaved in — Whether to use interleaved layout (platform-dependent).

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemAlloc3D(HaDevicePtr *ptr, const HaShape &shape, const bool is_interleaved);

예제

1
2
HaDevicePtr tensor = 0;
HaMemAlloc3D(&tensor, HaShape{width, height, channels}, /*is_interleaved=*/true);

Note

Convenience variant for tensor-like allocations. The interpretation of HaShape follows the runtime layout contract.

Note

For operands whose physical layout the hardware fixes (KV-cache blocks, MPU tensors, row-interleaved matrices), do not size the allocation by hand; build the layout with HaBuildDeviceTensorLayout() instead.


HaMemAllocAsync

Asynchronously allocates device memory on a stream.

Parameter Direction Default Description
ptr out — Output pointer to receive the allocated device address (required).
size in — Number of bytes to allocate.
is_interleaved in — Whether to use interleaved layout (platform-dependent).
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device.

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemAllocAsync(HaDevicePtr *ptr, const size_t size, const bool is_interleaved, HaStream_t stream=HA_STREAM_DEFAULT);

예제

1
2
3
HaDevicePtr scratch = 0;
HaMemAllocAsync(&scratch, size, /*is_interleaved=*/false, stream);
HaLaunchKernelAsync(kernel_path, args, stream);   // usable for queueing straight away

Note

The allocation is enqueued into the stream and may not be available until the stream reaches that point. Use HaStreamSynchronize() before accessing the allocated memory from the host. Free with HaMemFreeAsync() on the same stream, or with HaMemFree() after synchronization.

Note

The pointer can be passed to later operations on the stream immediately, but must not be accessed from the host until the stream reaches that point.


HaMemCpy

Copies memory between host and device, or between devices.

Parameter Direction Default Description
dst in — Destination address (host pointer or device address).
src in — Source address (host or device).
size in — Number of bytes to copy.
direction in — Copy direction. One of HostToDevice, DeviceToHost, DeviceToDevice (see HaMemCpyDirection).

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemCpy(HaDevicePtr dst, const HaDevicePtr src, const size_t size, const HaMemCpyDirection direction);

예제

1
2
3
std::vector<uint8_t> host(size);
HaMemCpy(dev, reinterpret_cast<HaDevicePtr>(host.data()), size,
         HaMemCpyDirection::HostToDevice);

Note

Typical usage is uploading inputs (HostToDevice) or downloading outputs (DeviceToHost). Both operands are HaDevicePtr, including the host one, so the direction argument is what says which side is which; passing the wrong direction is not a type error.

Note

Stream-ordered on the calling thread's default stream on the device that owns the device address, whatever the current device: the copy is queued there and this call waits for it, so whatever was already queued there runs first. A stream from HaStreamCreate() is a separate queue; synchronize it, or wait on an event recorded on it, before a copy that depends on its result.

Note

The returned error is not only this copy's. A failure of an earlier asynchronous op on that stream is sticky and surfaces here, and a per-thread default stream is never destroyed, so only HaResetDevice() clears it.


HaMemCpyAsync

Asynchronously copies memory on a stream.

Parameter Direction Default Description
dst in — Destination address (host pointer or device address).
src in — Source address (host or device).
size in — Number of bytes to copy.
direction in — Copy direction. One of HostToDevice, DeviceToHost, DeviceToDevice (see HaMemCpyDirection).
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device. Its device must own the device address: the copy executes there, so a stream on another device is rejected with HaMemCpyNotSupported.

Returns — HaSuccess if enqueued successfully; the copy may not be complete on return.

시그니처 · 예제
1
HaError HaMemCpyAsync(HaDevicePtr dst, const HaDevicePtr src, const size_t size, const HaMemCpyDirection direction, HaStream_t stream=HA_STREAM_DEFAULT);

예제

1
2
3
HaMemCpyAsync(dev, reinterpret_cast<HaDevicePtr>(host.data()), size,
              HaMemCpyDirection::HostToDevice, stream);
HaStreamSynchronize(stream);   // only now is it safe to modify host

Note

Use HaStreamSynchronize() or HaEventSynchronize() to confirm completion.

Note

Returning does not mean the copy started. Do not touch either host buffer before a synchronization point; bugs of this kind depend on timing and go unnoticed in most runs.

Note

Cross-device device-to-device: synchronize stream before resetting either device. The copy is ordered against a reset of stream's device only; a reset of the other device neither waits for it nor cancels it, so it can then read or write that device's memory after the reset.


HaMemCpyNd

Copies memory between host and device, or between devices, described by two N-D layout descriptors.

Parameter Direction Default Description
dst in — Destination buffer pointer (host pointer or device pointer per dst_layout->scope).
dst_layout in — Destination layout (non-null). See HaLayout.
src in — Source buffer pointer (host pointer or device pointer per src_layout->scope).
src_layout in — Source layout (non-null). See HaLayout.

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemCpyNd(HaDevicePtr dst, const HaLayout *dst_layout, const HaDevicePtr src, const HaLayout *src_layout);

예제

1
2
3
4
HaLayout host_view = ...;   // scope = Host
HaLayout dev_view = ...;    // scope = Device, same rank and shape

HaMemCpyNd(dev_ptr, &dev_view, reinterpret_cast<HaDevicePtr>(host_buf), &host_view);

Note

dst_layout and src_layout must share the same rank and shape. If a caller starts with mismatched ranks, re-view one side first (e.g. as_strided) — this API does not broadcast.

Note

Strides are in elements of dtype, not bytes. Getting that wrong is not rejected; it reads the wrong addresses.

Note

The direction is inferred from (dst_layout->scope, src_layout->scope): host-to-device, device-to-host, device-to-device, or host-to-host. Host-to-host runs entirely on the host (a parallel memcpy walk) with no device transfer.

Note

Stream-ordered on the calling thread's default stream on the device that owns the device address, whatever the current device, as with HaMemCpy(). The returned error can be a sticky failure of an earlier asynchronous op on that stream; only HaResetDevice() clears it.


HaMemCpyNdAsync

Asynchronously copies memory described by two N-D layout descriptors, on a stream. Plans the copy as HaMemCpyNd() does, but enqueues each resulting transfer on stream instead of issuing it immediately, so the call returns without draining the stream first.

Parameter Direction Default Description
dst in — Destination buffer pointer (host pointer or device pointer per dst_layout->scope).
dst_layout in — Destination layout (non-null). See HaLayout.
src in — Source buffer pointer (host pointer or device pointer per src_layout->scope).
src_layout in — Source layout (non-null). See HaLayout.
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT for the calling thread's default stream on the current device. Its device must own a device operand: the copy executes there, so a stream on another device is rejected with HaMemCpyNotSupported.

Returns — HaSuccess if enqueued successfully; the copy may not be complete on return.

시그니처 · 예제
1
HaError HaMemCpyNdAsync(HaDevicePtr dst, const HaLayout *dst_layout, const HaDevicePtr src, const HaLayout *src_layout, HaStream_t stream=HA_STREAM_DEFAULT);

예제

1
2
HaMemCpyNdAsync(dev_dst, &dev_layout, reinterpret_cast<HaDevicePtr>(host_src), &host_layout, stream);
HaStreamSynchronize(stream);   // the host source must stay valid until here

Note

Source lifetime. A host source must stay valid and unmodified until the copy completes (HaStreamSynchronize() or HaEventSynchronize()), as with HaMemCpyAsync(). Whether a call stages its source is not reported, so never release or reuse the source before the copy completes.

Note

Channel-interleaved device-to-host. The reorder into the caller's logical order runs only after the device has filled its staging buffer, so the destination is filled only at HaStreamSynchronize(stream) or HaDeviceSynchronize(); HaEventSynchronize() or synchronizing another stream is not enough.

Note

Host-to-host is rejected (it cannot be ordered against stream); use HaMemCpyNd(). For a cross-device device-to-device copy, synchronize stream before resetting either device: the copy is ordered against a reset of stream's device only.


HaMemDumpForEachChannel

Dumps the raw bytes backing a device allocation, one buffer per DRAM channel.

Parameter Direction Default Description
start in — Channel-0 device address.
bytes_total in — Logical byte count to dump (must be > 0).
out out — Receives num_channels buffers (required).

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemDumpForEachChannel(HaDevicePtr start, size_t bytes_total, std::vector< std::vector< uint8_t >> *out);

예제

1
2
std::vector<std::vector<uint8_t>> per_channel;
HaMemDumpForEachChannel(buf, bytes_total, &per_channel);   // one buffer per DRAM channel

Note

Pre-condition: a current device is set; start is a channel-0 address from HaMemAlloc().

Note

A debugging aid for checking that a layout landed where it was meant to. It transfers the whole range synchronously and does not de-interleave, so it is not a data path.


HaMemFree

Frees device memory previously allocated by HaMemAlloc()/HaMemAlloc3D().

Parameter Direction Default Description
ptr in — Device pointer to free.

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemFree(const HaDevicePtr ptr);

예제

1
HaMemFree(buf);   // returns the block to the runtime pool, not to the driver

Note

Freed memory goes back to the runtime pool for reuse. When a free leaves a segment empty, the segment is returned to the driver right away if other allocations remain and the free memory the pool holds exceeds half of the allocated amount; otherwise, for example after every allocation is freed, it is kept. Return kept segments with HaDeviceMemTrim().


HaMemFreeAsync

Asynchronously frees device memory previously allocated by HaMemAllocAsync() on a stream.

Parameter Direction Default Description
ptr in — Device pointer to free (must have been allocated by HaMemAllocAsync()).
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device.

Returns — Success or an error code.

시그니처 · 예제
1
HaError HaMemFreeAsync(const HaDevicePtr ptr, HaStream_t stream=HA_STREAM_DEFAULT);

예제

1
2
3
HaMemAllocAsync(&scratch, size, false, stream);
HaLaunchKernelAsync(kernel_path, args, stream);
HaMemFreeAsync(scratch, stream);   // scratch is unusable from here, synchronized or not

Note

The free is enqueued into the stream and deferred until all preceding stream operations complete. Do not access ptr after this call, even if the stream has not yet been synchronized.


HaMemLoadFileAsync

Asynchronously loads a file into device memory on a stream.

Parameter Direction Default Description
file_fd in — Open file descriptor to read from.
file_off in — Byte offset within the file to start reading.
dst in — Destination device address.
len in — Number of bytes to load.
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device.

Returns — HaSuccess if enqueued successfully; the load may not be complete on return.

시그니처 · 예제
1
HaError HaMemLoadFileAsync(int file_fd, size_t file_off, HaDevicePtr dst, size_t len, HaStream_t stream=HA_STREAM_DEFAULT);

예제

1
2
3
int fd = open(weights_path, O_RDONLY);
HaMemLoadFileAsync(fd, file_offset, dev_weights, len, stream);
HaStreamSynchronize(stream);

Note

Use HaStreamSynchronize() or HaEventSynchronize() to confirm completion.