콘텐츠로 이동

Execution API

Kernel launch, kernel libraries, and the argument list a launch takes. The difference between the synchronous and asynchronous forms is described in Execution Model.

Contents

API 설명
HaLibraryLoadData 미리 컴파일된 커널 라이브러리 등록
HaLibraryGetKernel 라이브러리에서 이름으로 커널 핸들 조회
HaLaunchKernel 커널 동기 실행 (커널 핸들 또는 바이너리 경로 지정)
HaLaunchKernelAsync 커널 비동기 실행 (커널 핸들 또는 바이너리 경로 지정)
HaLaunchArgsAddTensor Tensor 인자 추가
HaLaunchArgsAddScalar Scalar 인자 추가
HaLaunchArgsAddArray Array 인자 추가
HaLaunchArgsSetTensor 추가한 Tensor 인자의 주소·크기 교체
HaLaunchArgsSetScalarBits 추가한 Scalar 인자의 값 교체
HaLaunchArgsSetArray 추가한 Array 인자의 크기·데이터 교체

Launch arguments

HaLaunchKernel() and HaLaunchKernelAsync() take a HaLaunchArgs list: the arguments of one launch, in the order they were appended. The records are built inside the runtime in a layout that belongs to the kernel ABI and is not published, so a caller never depends on that layout.

  • Object. HaLaunchArgs is a C++ class that owns its records: construct it to get an empty list, and the records are released when the object is destroyed. It is move-only; a list that has been moved from is rejected by every function with HaKernelInvalidArgument. Construction allocates and throws std::bad_alloc when memory is short.
  • Arguments. Tensor, scalar and array arguments, each with a HaLaunchArgRoute that picks the argument block (and a target core for Core). The route must be the one the kernel was compiled with.
  • Checked against the current device. Appending checks each argument against the device that is current on the calling thread (HaSetDevice()): a Core route must name one of that device's cores, and the call returns HaDeviceNotInitialized when the thread has no current device and HaDeviceNotReady when that device is not ready (a reset is in progress). HaLaunchArgsSet*() needs no device: it changes records the list already holds, so neither a missing current device nor a reset in progress affects it. The list does not remember which device it was built against: the launch checks it again against the device it launches on, so an argument naming a core that device does not have is rejected there instead of being dropped. To build a list for a device other than the current one, use the driver API's ha_drv_launch_args_*() functions.
  • Consumed before return. Both launch calls read every record on the calling thread before they return, so the list may be destroyed or updated right after either call; an asynchronous launch needs no different handling.
  • Reuse. A caller that launches the same argument shape again can keep one list and update its records in place with HaLaunchArgsSet*(), by the position each record was appended at (counting from 0).
  • Threading. Several threads may launch from one list at the same time, and may also append to it or update it: appends and updates take the list's lock exclusively and a launch reads under it shared, so racing calls are serialized in some order. Which update a racing launch sees is not defined. Moving or destroying a list is not synchronized with its use, so do not move or destroy a list while another thread uses it. Lists share no state.

A mismatch between the list and the kernel — a wrong type, a scalar where an array is read, a different argument count — is rejected at launch with HaKernelInvalidArgument, and the log names the argument.

예제
1
2
3
4
5
6
7
8
HaSetDevice(0);                      // arguments are checked against the current device
HaLaunchArgs args;                   // empty list; released when it goes out of scope

const uint32_t x_dims[] = {10, 512};
HaLaunchArgsAddTensor(args, HaLaunchArgRoute::Universal, -1, x_da, x_dims, 2, /*offset=*/0);
HaLaunchArgsAddScalar(args, HaLaunchArgRoute::Top, -1, HaDataType::Int64, /*bits=*/7);

HaError err = HaLaunchKernelAsync(kernel_path, args, stream);   // records are read before this returns

HaLibraryLoadData

Registers a library of precompiled kernels. data points to an HaLibraryData (defined in ha_types.h).

Parameter Direction Default Description
library out — Receives the library.
data in — The library data (a HaLibraryData).

Returns — HaSuccess; HaInvalidArgument if library or data is null; HaKernelInvalidArgument if data is not a valid HaLibraryData, or has no binary set for a device version this runtime knows; HaKernelUnsupported if the library was built against a version of this runtime whose library format this one does not read.

시그니처 · 예제
1
HaError HaLibraryLoadData(HaLibrary_t *library, const void *data);

예제

1
2
3
4
5
// my_kernels_data: the HaLibraryData built into this binary
HaLibrary_t library = nullptr;
if (HaLibraryLoadData(&library, &my_kernels_data) != HaError::HaSuccess) {
  return LibraryLoadFailed();
}

Note

The HaLibraryData structs are only read during the call, but the kernel names and ELF bytes they point to are not copied and must stay valid for the life of the process. Call once per library per process: every call registers a new library.

Note

No device is touched: kernels are loaded onto a device when first launched there, and their code is checked then. Binary sets for device versions this runtime does not recognize are ignored without being checked; a kernel that appears only in such sets is treated as absent from the library. data is not a typed pointer, so only its magic and version are checked: it must point to a complete HaLibraryData.


HaLibraryGetKernel

Looks up a kernel of library by name. The same name returns the same kernel on every call, whatever device it later runs on.

Parameter Direction Default Description
kernel out — Receives the kernel.
library in — Library from HaLibraryLoadData().
name in — Kernel name.

Returns — HaSuccess; HaInvalidArgument if an argument is null; HaKernelInvalidHandle if library did not come from HaLibraryLoadData(); HaKernelNotFound if the library has no kernel named name.

시그니처 · 예제
1
HaError HaLibraryGetKernel(HaKernel_t *kernel, HaLibrary_t library, const char *name);

예제

1
2
HaKernel_t matmul = nullptr;
HaLibraryGetKernel(&matmul, library, "matmul");

HaLaunchKernel

Launches a kernel of a library on the current device and waits for it.

Parameter Direction Default Description
kernel in — Kernel from HaLibraryGetKernel().
args in — Argument list (required); see Launch arguments.

Returns — HaSuccess; HaKernelInvalidHandle if kernel did not come from HaLibraryGetKernel(); HaKernelInvalidArgument if args was moved from or does not match the kernel's arguments, or if the kernel's code is invalid; HaKernelUnsupported for a library kernel built for a kernel ABI this runtime does not support; HaKernelNoBinaryForDevice if the library has no binary of kernel for the device; or a load or launch error.

시그니처 · 예제 (kernel handle)
1
HaError HaLaunchKernel(HaKernel_t kernel, const HaLaunchArgs &args);

예제

1
2
HaSetDevice(0);
HaLaunchKernel(matmul, args);   // first launch on device 0 loads the kernel there

Note

The first launch of a kernel on a device loads that kernel there; kernels never launched on a device take no device memory there. A kernel's code may be written to the device on the stream of its first launch there; if that write fails, launches of the kernel on any stream fail with that stream's error. Destroy that stream to recover (the next launch writes the kernel again), or reset the device if it is a default stream. The list is consumed before this returns, so it may be destroyed or updated right after.

Launches a device kernel from a kernel binary path (eager path). The list is consumed before this returns, so it may be destroyed or updated right after.

Parameter Direction Default Description
binary_path in — Path to the kernel binary.
args in — Argument list (required); see Launch arguments.

Returns — HaSuccess; HaKernelInvalidArgument for an empty path, a list that was moved from, does not fit the kernel or that the kernel does not take, or a malformed binary; HaKernelNotFound if binary_path does not exist or names no known kernel; HaKernelUnsupported for a kernel built for another target or a kernel ABI this runtime does not support; or a load or launch error.

시그니처 · 예제 (binary path)
1
HaError HaLaunchKernel(const std::string &binary_path, const HaLaunchArgs &args);

예제

1
HaLaunchKernel(kernel_path, args);   // returns after the device has finished

HaLaunchKernelAsync

Asynchronously launches a kernel of a library on a stream. Loading and argument semantics are those of HaLaunchKernel(HaKernel_t, const HaLaunchArgs &).

Parameter Direction Default Description
kernel in — Kernel from HaLibraryGetKernel().
args in — Argument list (required); see Launch arguments.
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device.

Returns — HaSuccess if enqueued, otherwise as HaLaunchKernel(HaKernel_t, const HaLaunchArgs &).

시그니처 · 예제 (kernel handle)
1
HaError HaLaunchKernelAsync(HaKernel_t kernel, const HaLaunchArgs &args, HaStream_t stream = HA_STREAM_DEFAULT);

예제

1
2
HaLaunchKernelAsync(matmul, args, stream);
HaStreamSynchronize(stream);

Asynchronously launches a device kernel from a binary path on a stream (eager path). Enqueueing returns as soon as the work is queued, so overlapping a transfer with a launch is a matter of putting both on one stream and synchronizing once at the end.

Parameter Direction Default Description
binary_path in — Path to the kernel binary.
args in — Argument list (required); see Launch arguments.
stream in HA_STREAM_DEFAULT Stream to enqueue into; omit or pass HA_STREAM_DEFAULT to use the calling thread's default stream on the current device.

Returns — HaSuccess if enqueued, otherwise as HaLaunchKernel(binary_path, ...).

시그니처 · 예제 (binary path)
1
2
HaError HaLaunchKernelAsync(const std::string &binary_path, const HaLaunchArgs &args,
                            HaStream_t stream = HA_STREAM_DEFAULT);

예제

1
2
3
4
HaMemCpyAsync(dev_in, host_in, n, HaMemCpyDirection::HostToDevice, stream);
HaLaunchKernelAsync(kernel_path, args, stream);   // records were read before the call returned
HaMemCpyAsync(host_out, dev_out, n, HaMemCpyDirection::DeviceToHost, stream);
HaStreamSynchronize(stream);   // all three have completed here

Note

Argument semantics are those of HaLaunchKernel(binary_path, const HaLaunchArgs &). The list is consumed before this returns.


HaLaunchArgsAddTensor

Appends a tensor argument.

Parameter Direction Default Description
args in, out — The list.
route in — Block the argument is written to. One of Universal, Top, Core (see HaLaunchArgRoute).
target_core in — Core whose block receives a HaLaunchArgRoute::Core argument, from 0 to one less than the core count of the current device. Ignored otherwise.
da in — Device address of the tensor's base.
dims in — rank extents, in the order the kernel declares them. Copied, so dims need not outlive this call.
rank in — Number of entries in dims.
offset in — Element offset from da.

Returns — HaSuccess; HaDeviceNotInitialized, appending nothing, when the calling thread has no current device; HaDeviceNotReady, appending nothing, when that device is not ready (a reset is in progress); or HaKernelInvalidArgument, appending nothing, when args was moved from, route is not a HaLaunchArgRoute, rank is negative, dims is nullptr with a positive rank, or route is HaLaunchArgRoute::Core and target_core names a core the current device does not have.

시그니처
1
2
HaError HaLaunchArgsAddTensor(HaLaunchArgs &args, HaLaunchArgRoute route, int target_core, HaDevicePtr da,
                              const uint32_t *dims, int rank, uint64_t offset);

HaLaunchArgsAddScalar

Appends a scalar argument.

Parameter Direction Default Description
args in, out — The list.
route in — Block the argument is written to. One of Universal, Top, Core (see HaLaunchArgRoute).
target_core in — Core whose block receives a HaLaunchArgRoute::Core argument. As for HaLaunchArgsAddTensor().
dtype in — Element type of the scalar: Int8, Int16, Int32, Int64, Float16, BFloat16, Float32, Float64 or Float8 (E4M3). Any other type is rejected.
bits in — The value, bit for bit. Only the low bytes the type occupies are read, so a negative Int32 is its two's complement in the low 4 bytes and a Float32 is its IEEE pattern in the low 4 bytes.

Returns — HaSuccess, or HaKernelInvalidArgument, appending nothing, under the conditions of HaLaunchArgsAddTensor() that apply, or when dtype is not one of the types above.

시그니처
1
2
HaError HaLaunchArgsAddScalar(HaLaunchArgs &args, HaLaunchArgRoute route, int target_core, HaDataType dtype,
                              uint64_t bits);

HaLaunchArgsAddArray

Appends an array argument: element bytes the kernel reads from its instruction table. Unlike a tensor, the data is not a device pointer; the bytes are embedded in the argument block.

Parameter Direction Default Description
args in, out — The list.
route in — Block the argument is written to. One of Universal, Top, Core (see HaLaunchArgRoute).
target_core in — Core whose block receives a HaLaunchArgRoute::Core argument. As for HaLaunchArgsAddTensor().
dims in — rank extents, in the order the kernel declares them. Copied.
rank in — Number of entries in dims.
strides in — rank strides, or nullptr for unit strides. Copied.
raw_data in — The element bytes. Copied.
raw_data_size in — Number of bytes in raw_data.

Returns — HaSuccess, or HaKernelInvalidArgument, appending nothing, under the conditions of HaLaunchArgsAddTensor() that apply, or when raw_data is nullptr with a positive raw_data_size.

시그니처 · 예제
1
2
HaError HaLaunchArgsAddArray(HaLaunchArgs &args, HaLaunchArgRoute route, int target_core, const uint32_t *dims,
                             int rank, const uint32_t *strides, const uint8_t *raw_data, size_t raw_data_size);

예제

1
2
3
4
5
// Four int32 values embedded in core 0's argument block
const int32_t table[4] = {1, 2, 3, 4};
const uint32_t dims[] = {4};
HaLaunchArgsAddArray(args, HaLaunchArgRoute::Core, 0, dims, 1, /*strides=*/nullptr,
                     reinterpret_cast<const uint8_t *>(table), sizeof(table));

HaLaunchArgsSetTensor

Replaces the address and extents of the tensor appended at index. Everything else about the record (route, target core, offset, strides) stays as it was appended.

Parameter Direction Default Description
args in, out — The list.
index in — Position the tensor was appended at, counting from 0.
da in — New device address of the tensor's base.
dims in — rank extents. Copied.
rank in — Number of entries in dims. Must equal the rank the tensor was appended with.

Returns — HaSuccess, or HaKernelInvalidArgument, changing nothing, when args was moved from, index is out of range or names a record that is not a tensor, rank differs from the record's, or dims is nullptr with a positive rank.

시그니처
1
HaError HaLaunchArgsSetTensor(HaLaunchArgs &args, int index, HaDevicePtr da, const uint32_t *dims, int rank);

HaLaunchArgsSetScalarBits

Replaces the value of the scalar appended at index.

Parameter Direction Default Description
args in, out — The list.
index in — Position the scalar was appended at, counting from 0.
bits in — New value, bit for bit, as for HaLaunchArgsAddScalar().

Returns — HaSuccess, or HaKernelInvalidArgument, changing nothing, when args was moved from, index is out of range or names a record that is not a scalar.

시그니처
1
HaError HaLaunchArgsSetScalarBits(HaLaunchArgs &args, int index, uint64_t bits);

HaLaunchArgsSetArray

Replaces the extents and element bytes of the array appended at index.

Parameter Direction Default Description
args in, out — The list.
index in — Position the array was appended at, counting from 0.
dims in — rank extents. Copied.
rank in — Number of entries in dims. Must equal the rank the array was appended with.
raw_data in — The element bytes. Copied.
raw_data_size in — Number of bytes in raw_data.

Returns — HaSuccess, or HaKernelInvalidArgument, changing nothing, when args was moved from, index is out of range or names a record that is not an array, rank differs from the record's, dims is nullptr with a positive rank, or raw_data is nullptr with a positive raw_data_size.

시그니처
1
2
HaError HaLaunchArgsSetArray(HaLaunchArgs &args, int index, const uint32_t *dims, int rank, const uint8_t *raw_data,
                             size_t raw_data_size);