1. Overview¶
The HyperAccel Runtime (hart) is an API that controls devices, memory, and kernel execution.
torch_ha (PyTorch) and vllm_hyperaccel (model serving) call this API internally, so if you use them you rarely
need to work with the runtime API directly. This document is the programming guide for developers who integrate
with the system at the runtime level, without a higher-level framework.
Contents¶
| Section | Objective |
|---|---|
| 1.1 Role of the Runtime API | Why the runtime API is needed and how it differs from the driver API |
| 1.2 Design Background | The six principles behind the API design and what they require of your code |
| 1.3 Key Concepts | Terms and definitions used throughout the guide |
| 1.4 Basic Usage Example | Basic code flow for memory allocation, copy, and free |
Execution Model, Memory Model, Streams and Events, and Multi-Device are written to be read in order. Error Handling and Troubleshooting are organized for lookup as needed. Converting weights into the device layout is covered separately in Weight Conversion. The device-handle-based driver API is covered in the separate Driver API Guide.
1.1 Role of the Runtime API¶
To run computation on an LPU, you load a kernel binary onto the device, place operands in device memory, and then run it. The runtime API provides this process as device, memory, execution, and synchronization APIs, and it takes care of the device selection and address routing needed when multiple threads and multiple devices are involved. Applications can therefore focus on the flow of computation without managing device handles themselves. To specify the device directly with a handle, use the driver API.
The figure below shows how software that uses the runtime API is structured. Each thread only sets the device it uses and calls the API without a device argument, and the runtime API takes care of per-device control and pointer routing.
Layering of software that uses the runtime API
1.2 Design Background¶
These are the six design principles that determine the shape of the API, and what each one requires when you write code.
- CUDA runtime compatibility — Function names and semantics follow the CUDA runtime. The
Haprefix replaces thecudaprefix (C++:HaMemAlloc, Python:mem_alloc), and a function with the corresponding name plays the same role. Where behavior differs from CUDA, the relevant chapter says so explicitly. - Per-thread device selection — The target device is determined by the current device set for each thread,
not by a call argument, so APIs such as allocation and launch take no device argument. Each thread must call
set_device()(C++:HaSetDevice()) before using the runtime API; otherwise calls returnHaDeviceInvalidArgument. - Memory pooling — Memory freed with
mem_free()(C++:HaMemFree()) can stay in the runtime pool for reuse by later allocations, so the device's free memory may not grow right after a free. If you need to give memory back for another process or a large allocation, calldevice_mem_trim()(C++:HaDeviceMemTrim()). - Pointer-based device routing — Functions that take a device pointer, such as
mem_free()andmem_cpy()(C++:HaMemCpy()), operate on the device the pointer belongs to. You can free or copy memory another thread allocated without changing the current device. The calling thread must still have a current device set. - Synchronous and asynchronous pairs — Memory allocation, free, and copy (including N-D copy) and kernel
launch come as both a synchronous function and an asynchronous function (suffix
_async, C++:Async). An asynchronous function enqueues the work on a stream and returns right away, so do not free or modify the host buffers you passed until the work completes. File load (mem_load_file_async()) has only an asynchronous function and 3D allocation (mem_alloc_3d()) only a synchronous one. For details, see Synchronous and Asynchronous Dispatch in the Execution Model. - Out-parameter-based error model — The C++ API does not throw exceptions. Every function returns
HaError, and results such as an allocated address or a queried value are written into a variable the caller passes in. For example,HaGetDeviceCount(&count)writes the device count tocount. Check that the return value isHaSuccesson every call. The Python API raises aRuntimeErrorthat carries the same code name. For details, see Error Handling.
1.3 Key Concepts¶
This section summarizes the key terms and definitions used throughout the rest of this guide.
| Concept | Meaning |
|---|---|
| Runtime | No explicit initialization is needed. The runtime initializes itself on the first runtime API call in the process; there is one per process, shared by all threads. For errors the first call can return, see The First Calls of a Process in Error Handling. |
| Device | One LPU, identified by a virtual device ID in the range [0, HaGetDeviceCount()). This differs from the physical ID used by the hardware (see Virtual Device ID in Multi-Device Programming). |
| Current device | The device a thread works on. Each thread sets it with set_device() (C++: HaSetDevice()); after that, allocations and kernel launches on that thread run on this device. |
| Device pointer | A 64-bit device address of type HaDevicePtr. Dereferencing it directly on the host is not allowed. |
| Tensor layout | An HaLayout object that holds a tensor's shape, stride, data type, and memory scope. It does not hold a data pointer, so one layout can be reused to copy several buffers of the same shape (see Tensor Layout in the Memory Model). |
| Stream | A queue for asynchronous work. Work enqueued on the same stream runs in the order it was enqueued; no order is guaranteed between different streams. |
| Event | A marker recorded on a stream. Use it to check or wait until the work enqueued before it has finished, or to make another stream wait until that point. |
| Kernel | A program written in Legato and compiled to run on the device. Its arguments are passed in an argument list (HaLaunchArgs). For how to launch it, see Execution Model. |
1.4 Basic Usage Example¶
A basic example that finds a device, allocates memory, moves data, and frees the memory.
1 2 3 4 5 6 7 8 9 10 11 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | |
C++: The direction argument determines the transfer direction
HaMemCpy() takes HaDevicePtr arguments in both directions.
Host pointers are also cast to the same type, so which side is the host and which is the
device is determined by the direction argument, not by type information. As a result, specifying the wrong
direction is not caught as a compile-time type error.
Next Steps¶
- Execution Model — Operating modes and execution paths
- Memory Model — Allocation, layout, and copy
- API Reference — Exact signatures