Skip to content

HyperAccel Runtime (hart)

HyperAccel Runtime (hart) is the Resource Management & Execution Layer for HyperAccel LPUs (Language Processing Units). It manages device and memory resources and is responsible for kernel loading and dispatch and for stream- and event-based synchronous and asynchronous execution. It is designed to get the most out of the LPU's compute performance by using resources efficiently and minimizing execution overhead.

hart handles internally what multi-threaded, multi-device environments require: per-thread device selection, memory-address-based device routing, and low-overhead memory allocation. Higher-level frameworks and applications can therefore focus on building the flow of computation without handling the details of device management themselves.

hart is the core interface that sits between higher-level AI frameworks and the kernel mode driver (KMD). Major components of the HyperAccel AI software stack, such as the PyTorch backend (torch_ha), model serving (vllm_hyperaccel), collective communication (lccl), and the profiler (hapti), run on hart. Software that must work with devices directly through handles, such as custom runtimes or lightweight environments, uses the device-handle-based Driver API (ha_drv_*).


Architecture Overview

Internally, hart is organized into the following five areas. Each feature is listed under Core Features of hart below.

  • Device & Routing Engine: Discovers, initializes, and resets hardware devices, and assigns virtual IDs sequentially from 0 only to devices that initialized successfully. It also tracks the per-thread current device and the device that owns each memory pointer, and routes each call to the right target device.
  • Execution & Synchronization: Queues work per stream to provide synchronous and asynchronous execution, and synchronizes execution order and completion across streams through events. Kernels are looked up by name in a kernel library and launched, and each device loads the binary for its chip generation.
  • Memory Management: Reuses freed device memory to reduce allocation overhead, and supports memory usage queries and release. It also manages host transfer buffers, virtual memory for the KV cache, and direct P2P (peer-to-peer) transfer between devices.
  • Weight Conversion: Converts a model's weight tensors, together with their quantization metadata, into the device memory layout specific to the target architecture. This runs independently on the host without occupying device resources.
  • Observability & Diagnostics: A standardized error code scheme clearly identifies the cause of a failure, and execution diagnostics let the HyperAccel Profiler (hapti) correlate and trace host-side calls with compute activity inside the device.

The Driver API bypasses these areas and works on devices directly through handles.

HyperAccel Runtime architecture and features in the software stack

HyperAccel Runtime architecture and features in the software stack


Core Features of hart

The table below summarizes the main features of the areas above. For details, see the Programming Guide and the API Reference.

Feature Description
CUDA Runtime-Style API Naming and usage flow follow the CUDA runtime, so developers familiar with CUDA can understand the API easily (for example, cudaMalloc → HaMemAlloc).
Device Discovery & Reset Checks the available devices and resets one device or all of them. Devices that fail to initialize are skipped, and virtual IDs are assigned without gaps from 0 only to healthy devices.
Multi-Threaded Device Routing Multiple host threads can each pick a device and use it independently, and frees and copies are forwarded automatically to the device the pointer belongs to.
Synchronous & Asynchronous Execution Kernel launches and memory operations are available both synchronously and asynchronously, and streams and events control the ordering and completion of asynchronous work.
Kernel Library & Typed Launch Kernels are looked up by name in a library and launched, and each device loads the build for its chip generation. Arguments are passed as a list that records each argument's kind and type, so a mismatch with the kernel's signature is rejected before launch (Kernel Library and Handle-Based Launch).
Memory Reuse, Statistics & Trim Freed device memory is reused to reduce allocation overhead, and you can query usage or release memory that is not in use (Memory Statistics & Trim).
Host Memory Manager Allocates and reuses host transfer buffers per process and enforces a usage limit (Host Memory).
Peer-to-Peer (P2P) Transfer & KV Cache Virtual Memory Copies between devices are handled directly without going through the host, and virtual memory (VMM) for the KV cache manages the HW pages (Virtual Memory & HW Pages).
Weight Conversion Converts weight tensors, together with their quantization metadata, into the device memory layout of the target architecture (Weight Conversion).
Standardized Error Codes C++ reports HaError codes and Python raises a RuntimeError carrying the same code name, so failure causes can be distinguished by code.

At a Glance

Item Specification
Package hart
Version 0.0.6
Languages C++20 core, Python bindings (pybind11)
Artifacts C++ library (CMake package hart), hart Python module
Public headers ha_runtime_api.h, ha_driver_api.h, ha_driver_types.h, ha_types.h, ha_error_code.h, ha_error.h, ha_vmm.h, ha_callback_params.h, ha_weight_converter.h, ha_weight_converter_types.h
Python 3.10 – 3.12
Status Beta

API Overview

hart provides a C++ API (Ha*) and a Python API (hart module) with the same functionality. Python names drop the Ha prefix and use snake_case. Entry points by area are listed in the API Reference, and the device-handle-based API is listed in the Driver API Reference.


Installation

hart is not installed as a standalone package; it is distributed as a single HyperAccel SDK wheel package together with the compiler, profiler, and framework integration components.

For the detailed installation guide and the required KMD (kernel mode driver) setup, see the SDK Installation Guide.


Compatibility

This documentation is written for hart 0.0.6.

Category Verified versions
Python 3.10 – 3.12
PyTorch 2.10.0
Device versions Bertha Cloud EVT0, Bertha Cloud EVT1, Bertha Edge EVT0

For detailed specifications, see the Support Matrix.


Documentation

  • 📋 Release Notes

    Changes by version, deprecations, and known issues.