2. Execution Model¶
The runtime does not interpret or optimize kernel code. It loads a compiled kernel onto the device, lays out the given arguments in the hardware format, and dispatches the work to the device.
Synchronous and asynchronous APIs dispatch the same way; they differ only in when they return control. A
synchronous API (for example, launch_kernel_with_binary_path() (C++: HaLaunchKernel()))
waits until the work finishes before it returns, so when it returns the operation is complete. An asynchronous
API (for example, launch_kernel_async_with_binary_path() (C++: HaLaunchKernelAsync()))
returns as soon as the work is enqueued on the given stream, so you check completion separately through a stream or an
event (Synchronous and Asynchronous Dispatch).
This document covers how work is dispatched and when calls return, not the internal computation of kernels.
Contents¶
| Section | Objective |
|---|---|
| 2.1 Operating Modes | The two modes (Compile, Eager) in which higher-level frameworks drive the runtime, and the trade-offs of each |
| 2.2 Device Management | How a thread selects the device it works on, and what happens on a device reset |
| 2.3 Kernel Loading and Execution | How kernels are loaded and cached on the device, how arguments are passed (HaLaunchArgs), and how to launch a kernel by file path or from a kernel library |
| 2.4 Synchronous and Asynchronous Dispatch | When synchronous and asynchronous APIs return control, and how to check that work has finished |
| 2.5 Notes on Measuring Execution Time | Three factors that distort manually measured execution time, and how to deal with them |
2.1 Operating Modes¶
Higher-level frameworks drive the runtime in one of two ways. This is not a flag on an individual API call but the architectural entry path a higher-level framework uses to drive the runtime, and each path trades execution overhead against observability.
| Mode | Behavior | Dispatch overhead | Observability | Primary use case |
|---|---|---|---|---|
| Compile | Runs the execution sequence of a model graph optimized by an higher-level compiler as is. | Low | Limited | Production inference serving |
| Eager | Dispatches each operation to the device in order, as soon as it is called. | High | Full | Initial bring-up, operator accuracy validation, debugging |
In PyTorch, for example, this choice is made through torch.compile(backend="legato"): regions the compiler supports are offloaded to Legato kernels, and unsupported regions fall back to Eager mode. For the architectural background, see the Compiler documentation.
Use Eager mode during system bring-up and unit validation, and switch to Compile mode in production once the graph is stable. If you see numerical mismatches or anomalies in Compile mode, rerun the same input in Eager mode and compare results operator by operator.
2.2 Device Management¶
Device Assignment¶
set_device() (C++: HaSetDevice()) sets the current device of the calling thread only. It does not affect other
threads, and a newly created thread does not inherit it. Each thread that uses a device must therefore call
set_device() first; calling an API that needs a device from a thread that has not done so returns an error.
1 2 | |
1 2 3 | |
- Valid device IDs are in the range
[0, hart.get_device_count())and use 0-based indexing. - Device orchestration in multi-threaded and multi-process environments is covered in Multi-Device Programming.
Device Reset¶
reset_device() (C++: HaResetDevice()) returns one device to its initial state, and reset_all_devices() (C++: HaResetAllDevices()) resets every device the process uses.
Constraints and caveats of device reset
- Resource invalidation: A reset immediately invalidates every allocation,
stream, and event handle that belongs to the device. Stream and event handles from before the reset
are rejected with
HaStreamInvalidHandleandHaEventInvalidHandle, and device pointers withHaMemoryInvalidAddressby a transfer andHaMemoryFreeFailedby a free. However, if memory allocated after the reset receives the same address, the old pointer refers to the new allocation and is not rejected, so you must explicitly release related resources when you call reset. - Selecting the device again: To use a reset device again, every thread that used it must call
set_device()again. Until then, calls that need a device returnHaDeviceInvalidArgumentorHaDeviceNotInitialized. - Not for production error recovery: This mechanism suits unit tests and benchmark loops that need a clean state on every iteration. It affects the whole process and is a heavy blocking operation that also resets devices other threads are using, so do not use it for error recovery in production serving.
2.3 Kernel Loading and Execution¶
Regardless of the operating mode above, the smallest unit of work that actually runs on the device is a
kernel. A kernel is loaded and launched either from the path of a compiled binary file or through a kernel handle
obtained from a precompiled kernel library, and the arguments passed to the kernel go in the argument list
object HaLaunchArgs. How to build an argument list is described in "Kernel Arguments" below, and
how to use a library in "Kernel Library and Handle-Based Launch".
Loading & Caching Behavior¶
- Lazy Loading: A kernel binary is loaded onto each device at its first launch there, and only once per process.
- Avoiding profiling noise (Warm-up Strategy): The first iteration includes the binary load cost, so latency measured in that iteration does not reflect actual compute performance. Always run a warm-up phase before benchmarking.
- Cache Scope & Lifecycle: The kernel cache is kept independently per device and lasts for the lifetime of the process. The same kernel is loaded separately on each target device. A device reset clears the kernel cache along with every internal resource and cache kept to reduce the overhead of repeated operations. See Device Reset above for details.
There is no separate kernel loading API
If the kernel is not yet loaded on the target device, it is loaded when you call
launch_kernel_with_binary_path() or launch_kernel_with_handle() (C++: HaLaunchKernel()) in "Kernel Execution"
below. library_load_data() (C++: HaLibraryLoadData()), which registers a kernel library, does not load kernels
onto the device either.
Kernel Arguments (HaLaunchArgs)¶
In the HyperAccel runtime, kernel launch functions receive all arguments at once through the argument list object HaLaunchArgs (C++: HaLaunchArgs class, Python:
hart.HaLaunchArgs).
The argument list keeps the arguments for a launch in the order they were added. Callers handle arguments only through the argument list interface (methods), so changes to the ABI layout of the underlying kernel do not affect calling code.
| Section | Objective |
|---|---|
| Argument Kinds and Data Format | Properties and format of Tensor, Scalar, and Array arguments |
Routing and Core Selection (Route) |
Universal, Top, Core routing rules |
| Object Lifetime and Release | Creating an argument list, reading it at launch, data copies, and release |
| Reusing an Argument List | Updating values for repeated launches without recreating the object |
| Thread Safety | Rules for sharing an argument list across threads, and independence between lists |
| Validation and Error Handling | Validation both when arguments are added and when the kernel is launched |
Argument Kinds and Data Format¶
Arguments are added in the order the kernel declares them and fall into three kinds.
| Argument kind | Method (C++ / Python) | Data and structure | Purpose |
|---|---|---|---|
| Tensor | HaLaunchArgsAddTensor() / add_tensor() |
Device memory address, dimensions (dims), offset in elements |
Input and output tensors in device memory |
| Scalar | HaLaunchArgsAddScalar() / add_scalar() |
Data type (HaDataType) and the bit pattern of the value |
Single scalar values such as sizes, flags, and coefficients |
| Array | HaLaunchArgsAddArray() / add_array() |
Dimensions, strides, element bytes (data) | Small host data written directly into the kernel argument block (not a device pointer) |
Tensor Format¶
Rules for Tensor arguments (HaLaunchArgsAddTensor() / add_tensor()).
- Device address (
da): The device address of the start of the tensor. Only the address value is recorded in the argument list; tensor data is not copied. - Dimensions (
dims, plusrankin C++): The size of each dimension, in the order the kernel declares them. The element type is interpreted as the kernel declares it, so it is not specified in the argument. - Offset (
offset): Offset fromdain elements. Defaults to0(Python default).
Scalar Format¶
Rules for Scalar arguments (HaLaunchArgsAddScalar() / add_scalar()).
- Supported types (
HaDataType):Int8,Int16,Int32,Int64,Float16,BFloat16,Float32,Float64,Float8(E4M3) - Bit-pattern passing: The value is passed as a raw bit pattern, and only the low bytes occupied by the given data type are read.
* For example, a negative
Int32is the two's complement in the low 4 bytes, and aFloat32is the IEEE 754 bit pattern in the low 4 bytes.
Array Format¶
Rules for Array arguments (HaLaunchArgsAddArray() / add_array()).
- Data (
raw_data): The element bytes, passed as is (plus the byte countraw_data_sizein C++;bytes,bytearray, ormemoryviewin Python). These bytes go directly into the kernel argument block, not into device memory, and the element type is interpreted as the kernel declares it. - Dimensions (
dims, plusrankin C++): The size of each dimension, in the order the kernel declares them. - Strides (
strides): Per-dimension strides. If omitted (C++:nullptr, Python:None), unit strides are used.
Routing and Core Selection (Route)¶
A kernel ABI consists of one argument block for the Top Executor and one argument block per core, and each execution unit reads only
the block assigned to it. HaLaunchArgRoute specifies which block each argument is written to.
| Route value | Target block | target_core value |
|---|---|---|
Universal |
Top block and every core block | Ignored (pass -1) |
Top |
Top block only | Ignored (pass -1) |
Core |
The block of the single core given by target_core |
Core index (from 0, less than the core count of the current device). For multiple cores, add the argument once per core |
Use the kernel's route
route must be the one the kernel was compiled with. An argument added with a different route is written to a
block the kernel does not read, so the kernel misbehaves.
Object Lifetime and Release¶
An argument list is used in the order create the list → add arguments → launch the kernel, and you can replace
values (Set*) and launch again. It is released automatically when the object is destroyed.
- Read immediately at launch: The launch functions (
HaLaunchKernel(),HaLaunchKernelAsync()) read the entire argument list on the calling thread and build the launch command before returning control. Only the built command is enqueued on the stream. - Lifetime with asynchronous launch: Even for an asynchronous launch, it is safe to release the argument list or change its values right after the launch call returns.
- Argument data copies: When you add or replace an argument (
Add*,Set*), thedims,strides, and Array host data passed with it are copied into the argument list at that point. You can therefore release or modify the source host buffer as soon as the call returns. For a Tensor, only the device address value is recorded, and the device memory it points to must remain valid until execution finishes. - Creation and release: In C++,
HaLaunchArgsis an object that owns the argument records. Constructing the object creates an empty list, and the records are released when it goes out of scope and is destroyed, so no separate release call is needed. They are also released automatically on early-return error paths. Python objects are also released automatically when destroyed. - Move-only: C++ objects cannot be copied, only moved. Passing a moved-from object to an argument list function or a launch
function is rejected with
HaKernelInvalidArgument. If memory runs out during construction,std::bad_allocis thrown.
Reusing an Argument List¶
When you launch a kernel repeatedly with the same argument structure, keep the existing list object and update only the values instead of creating a new object each time.
- Update methods:
set_tensor(),set_scalar_bits(),set_array()(C++:HaLaunchArgsSetTensor(),HaLaunchArgsSetScalarBits(),HaLaunchArgsSetArray()) - Behavior: Replaces the address, dimensions, or value of the argument identified by the order in which arguments were added (index starting at
0). - Constraints:
route,target_core, data type, and offset keep the values from when the argument was first added, and the number of dimensions (rank) cannot change.
Thread Safety¶
- Sharing a list: Multiple threads can launch with one list, or add and replace arguments, concurrently. Each list has a lock: adds and replacements take it exclusively and launches read the list in shared mode, so overlapping calls are processed one at a time in some order. Which of the overlapping replacements a launch sees is unspecified, so finalize the values a launch uses before launching.
- Move and destruction: Moving and destroying a list are not synchronized. Do not move or destroy a list that another thread is using.
- Python: Python objects follow the same rules, so one object can be shared across threads.
- Independence: Different argument list objects share no state.
Validation and Error Handling¶
Arguments are validated at two points.
When an argument is added, it is checked against the current device of the calling thread (HaSetDevice() / set_device()).
For example, the core number of a Core route must be a core on the current device. You must therefore set the current
device before adding arguments; otherwise HaDeviceNotInitialized is returned, and HaDeviceNotReady is returned if a device reset is
in progress. To build a list against a device other than the current device, use the driver API ha_drv_launch_args_add_*() to
specify the device directly. Replacing an argument (Set*) only changes the value of an existing record and does not need a device.
You can replace arguments even with no current device or while a reset is in progress.
An argument list is not bound to a particular device, so it is checked again at launch against the device that runs it. If an argument points to a core that device does not have, the launch is rejected. As long as the core range fits, the list can run on a device other than the one it was built for. A bad argument is therefore always reported as an error, either when it is added or at launch.
| Validation point | Errors detected | Result |
|---|---|---|
| 1. When arguments are added or replaced | On add: no current device (HaDeviceNotInitialized) or device reset in progress (HaDeviceNotReady); unsupported data type, invalid route, core number not on the current device, negative rank; on replace: kind or rank mismatch; moved-from list |
The method returns an error code (Python: RuntimeError) and the argument list is unchanged |
| 2. When the kernel is launched | Arguments whose kind, data type, or count differs from what the kernel declares; a core number the launching device does not have | The launch is rejected immediately with HaKernelInvalidArgument, and the log records the expected and passed argument specifications |
Per-function error conditions are listed in the Execution API reference (Python:
HaLaunchArgs).
Kernel Execution¶
1 2 3 4 5 | |
1 2 3 4 5 6 7 | |
Argument List Reuse Example¶
The following is a typical Python pattern: create a HaLaunchArgs object, bind arguments, then update addresses and values
inside a loop and launch asynchronously.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | |
Kernel Library and Handle-Based Launch¶
A kernel library lets you look up a kernel by name and launch it, instead of by file path. The steps are as follows.
- Register: Register the library with
library_load_data()(C++:HaLibraryLoadData()) and receive a library handle (HaLibrary_t). It does not use a device. Each call registers a new library, so call it only once per library per process. - Look up a kernel: Get the kernel handle (
HaKernel_t) for a name withlibrary_get_kernel()(C++:HaLibraryGetKernel()). The same name always returns the same handle, and the handle is the same on whichever device it runs. - Launch: Launch with
launch_kernel_with_handle()/launch_kernel_async_with_handle()(C++:HaLaunchKernel(kernel, args)/HaLaunchKernelAsync(kernel, args, stream)). On the first launch on each device, the build for that device's chip generation is loaded. If the library has no build for that chip generation,HaKernelNoBinaryForDeviceis returned.
Registration, lookup, and per-device load flow of a kernel library (C++ API)
Library handles and kernel handles remain valid until the process exits and can still be used after a device reset. After a reset, the kernel is loaded again on the first launch.
1 2 3 4 5 6 7 8 9 10 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
For API details, see the C++ Execution API and the Python Execution API.
2.4 Synchronous and Asynchronous Dispatch¶
The runtime supports both synchronous and asynchronous dispatch for kernel launches and memory operations.
When control returns for synchronous and asynchronous dispatch
Synchronous¶
launch_kernel_with_binary_path() (C++: HaLaunchKernel())
returns after the kernel has finished running on the device.
1 | |
Asynchronous¶
launch_kernel_async_with_binary_path() (C++: HaLaunchKernelAsync())
returns as soon as the launch is enqueued on the stream, and completion must be checked through a stream or an
event.
1 2 | |
Caution and measurement guide
A typical profiler that samples only call sites observes only the enqueue time of an asynchronous launch, not the actual compute time. Measure device execution time against stream or event boundaries. To view device execution on a timeline, use the HyperAccel Profiler.
Caution: Accessing Host Buffers Before Synchronization
An asynchronous call returns as soon as the work is enqueued on the stream, so the device work may not be finished when it returns. Reading a host buffer passed to that work may return values from before the result is written, and modifying or freeing it may corrupt the result. Use host buffers only after confirming completion through a stream or an event.
2.5 Notes on Measuring Execution Time¶
This section covers three factors that can distort results by several times or more when you measure execution time yourself without a profiler, and how to deal with them. To see device execution on a timeline, use the HyperAccel Profiler.
Measurement Point Misconfiguration¶
Simply wrapping an asynchronous dispatch API call measures only the host-side command enqueue overhead, not the actual device compute time.
1 2 3 4 5 6 7 8 9 10 11 12 | |
1 2 3 4 5 6 7 8 9 10 | |
To profile the latency of a specific pipeline region (granular latency), use event objects
instead of a global synchronization that blocks the whole stream.
HaStreamSynchronize() synchronizes all pending work on the stream at once,
so it measures the latency of the whole pipeline rather than the region of interest.
First Iteration Distortion¶
The first iteration also pays one-time initialization costs that steady-state runs do not, so do warm-up runs first and exclude them from your measurements.
| Category | Factor and mechanism | Benchmark impact and mitigation |
|---|---|---|
| Binary Loading | • Mechanism: the binary is loaded and cached on the first launch on each target device • Execution path: reading and checking the binary file and reserving its device memory run on the calling thread; copying the binary to the device is enqueued on the stream of the first launch |
• Impact: extreme latency outliers in the first iteration • Mitigation: discarding only the first iteration is not enough; a warm-up that completes binary caching is required |
| Initial Allocation | • Mechanism: initial memory requests, made while the pool is empty, take a cold path that reserves new device memory • Steady state: later requests of the same size are served immediately through the internal cache and an optimized reuse path |
• Impact: initialization noise prevents statistical convergence even as sample size grows • Mitigation: run enough warm-up cycles to finish conditioning the internal cache |
State Isolation & Reset Side-effects¶
Trying to return to a fully clean state on every iteration costs more than the operation being measured
and distorts the results, for the four reasons below. As a rule, do not use
reset_device() (C++: HaResetDevice()) inside a repeated measurement loop.
- Handle invalidation forces teardown and recreation:
HaResetDevice()invalidates all allocated resources, streams, and event handles of the device, effectively resetting the device's entire execution state. Calling it inside a loop therefore requires recreating streams and resources from scratch on every iteration, and that recreation cost completely outweighs the workload you meant to measure. - Cache flush and binary reload: A device reset also forcibly flushes internal caches. A loop that includes a reset therefore pays the binary reload cost on every iteration, so the initialization overhead described above recurs every time.
- Resource reuse limits isolation: Even after resources are explicitly returned with
HaMemFree(), residual state remains because of internal caching and reuse policies, so complete state isolation between iterations is not guaranteed. Forcing it by callingHaResetDevice()on every loop only adds severe overhead, so trying to reset state inside a benchmark loop is not effective. - Reset with multiple threads: A reset does not stop other threads using the same device. While it is in
progress, other threads' calls on that device return
HaDeviceNotReady, and calls waiting on synchronization also end withHaDeviceNotReady. After it completes, the stream and event handles and device pointers held by other threads are invalid as well. This makes it hard to tell which iteration failed and whether a measurement is valid. When multiple threads use the same device, stop and synchronize all of their work before resetting.
Instead of trying to remove the cached, steady state that forms after warm-up, measure it as is;
this reflects steady-state production performance far better.
Use HaResetDevice() only at the boundary outside the loop, where you change experiment settings and run separate
benchmarks, not inside the loop.
Next Steps¶
- Memory Model — how to move data to the device
- Streams and Events — ordering asynchronous work