Skip to content

1. Overview

The HyperAccel Runtime (hart) is an API that controls devices, memory, and kernel execution. torch_ha (PyTorch) and vllm_hyperaccel (model serving) call this API internally, so if you use them you rarely need to work with the runtime API directly. This document is the programming guide for developers who integrate with the system at the runtime level, without a higher-level framework.

Contents

Section Objective
1.1 Role of the Runtime API Why the runtime API is needed and how it differs from the driver API
1.2 Design Background The six principles behind the API design and what they require of your code
1.3 Key Concepts Terms and definitions used throughout the guide
1.4 Basic Usage Example Basic code flow for memory allocation, copy, and free

Execution Model, Memory Model, Streams and Events, and Multi-Device are written to be read in order. Error Handling and Troubleshooting are organized for lookup as needed. Converting weights into the device layout is covered separately in Weight Conversion. The device-handle-based driver API is covered in the separate Driver API Guide.


1.1 Role of the Runtime API

To run computation on an LPU, you load a kernel binary onto the device, place operands in device memory, and then run it. The runtime API provides this process as device, memory, execution, and synchronization APIs, and it takes care of the device selection and address routing needed when multiple threads and multiple devices are involved. Applications can therefore focus on the flow of computation without managing device handles themselves. To specify the device directly with a handle, use the driver API.

The figure below shows how software that uses the runtime API is structured. Each thread only sets the device it uses and calls the API without a device argument, and the runtime API takes care of per-device control and pointer routing.

Layering of software that uses the runtime API

Layering of software that uses the runtime API

1.2 Design Background

These are the six design principles that determine the shape of the API, and what each one requires when you write code.

  1. CUDA runtime compatibility — Function names and semantics follow the CUDA runtime. The Ha prefix replaces the cuda prefix (C++: HaMemAlloc, Python: mem_alloc), and a function with the corresponding name plays the same role. Where behavior differs from CUDA, the relevant chapter says so explicitly.
  2. Per-thread device selection — The target device is determined by the current device set for each thread, not by a call argument, so APIs such as allocation and launch take no device argument. Each thread must call set_device() (C++: HaSetDevice()) before using the runtime API; otherwise calls return HaDeviceInvalidArgument.
  3. Memory pooling — Memory freed with mem_free() (C++: HaMemFree()) can stay in the runtime pool for reuse by later allocations, so the device's free memory may not grow right after a free. If you need to give memory back for another process or a large allocation, call device_mem_trim() (C++: HaDeviceMemTrim()).
  4. Pointer-based device routing — Functions that take a device pointer, such as mem_free() and mem_cpy() (C++: HaMemCpy()), operate on the device the pointer belongs to. You can free or copy memory another thread allocated without changing the current device. The calling thread must still have a current device set.
  5. Synchronous and asynchronous pairs — Memory allocation, free, and copy (including N-D copy) and kernel launch come as both a synchronous function and an asynchronous function (suffix _async, C++: Async). An asynchronous function enqueues the work on a stream and returns right away, so do not free or modify the host buffers you passed until the work completes. File load (mem_load_file_async()) has only an asynchronous function and 3D allocation (mem_alloc_3d()) only a synchronous one. For details, see Synchronous and Asynchronous Dispatch in the Execution Model.
  6. Out-parameter-based error model — The C++ API does not throw exceptions. Every function returns HaError, and results such as an allocated address or a queried value are written into a variable the caller passes in. For example, HaGetDeviceCount(&count) writes the device count to count. Check that the return value is HaSuccess on every call. The Python API raises a RuntimeError that carries the same code name. For details, see Error Handling.

1.3 Key Concepts

This section summarizes the key terms and definitions used throughout the rest of this guide.

Concept Meaning
Runtime No explicit initialization is needed. The runtime initializes itself on the first runtime API call in the process; there is one per process, shared by all threads. For errors the first call can return, see The First Calls of a Process in Error Handling.
Device One LPU, identified by a virtual device ID in the range [0, HaGetDeviceCount()). This differs from the physical ID used by the hardware (see Virtual Device ID in Multi-Device Programming).
Current device The device a thread works on. Each thread sets it with set_device() (C++: HaSetDevice()); after that, allocations and kernel launches on that thread run on this device.
Device pointer A 64-bit device address of type HaDevicePtr. Dereferencing it directly on the host is not allowed.
Tensor layout An HaLayout object that holds a tensor's shape, stride, data type, and memory scope. It does not hold a data pointer, so one layout can be reused to copy several buffers of the same shape (see Tensor Layout in the Memory Model).
Stream A queue for asynchronous work. Work enqueued on the same stream runs in the order it was enqueued; no order is guaranteed between different streams.
Event A marker recorded on a stream. Use it to check or wait until the work enqueued before it has finished, or to make another stream wait until that point.
Kernel A program written in Legato and compiled to run on the device. Its arguments are passed in an argument list (HaLaunchArgs). For how to launch it, see Execution Model.

1.4 Basic Usage Example

A basic example that finds a device, allocates memory, moves data, and frees the memory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import hart

try:
    hart.get_device_count()  # raises RuntimeError (HaDeviceNotAvailable) when there is no device
except RuntimeError:
    raise SystemExit("no LPU device found")
hart.set_device(0)

buf = hart.mem_alloc(1024)
hart.mem_cpy(buf, b"\x00" * 1024, 1024, hart.HaMemCpyDirection.HostToDevice)
hart.mem_free(buf)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
#include "ha_runtime_api.h"

#include <cstdint>
#include <cstdio>
#include <vector>

int main() {
  int count = 0;
  if (HaGetDeviceCount(&count) != HaError::HaSuccess || count == 0) {
    std::fprintf(stderr, "no LPU device found\n");
    return 1;
  }
  if (HaSetDevice(0) != HaError::HaSuccess) {
    std::fprintf(stderr, "set device failed\n");
    return 1;
  }

  HaDevicePtr buf = 0;
  if (HaMemAlloc(&buf, 1024, /*is_interleaved=*/false) != HaError::HaSuccess) {
    std::fprintf(stderr, "allocation failed\n");
    return 1;
  }

  std::vector<uint8_t> host(1024, 0);
  HaError err = HaMemCpy(buf, reinterpret_cast<HaDevicePtr>(host.data()), host.size(),
                         HaMemCpyDirection::HostToDevice);
  if (err != HaError::HaSuccess) {
    std::fprintf(stderr, "copy failed\n");
  }

  HaMemFree(buf);
  return err == HaError::HaSuccess ? 0 : 1;
}

C++: The direction argument determines the transfer direction

HaMemCpy() takes HaDevicePtr arguments in both directions. Host pointers are also cast to the same type, so which side is the host and which is the device is determined by the direction argument, not by type information. As a result, specifying the wrong direction is not caught as a compile-time type error.


Next Steps