HyperAccel Runtime (hart)¶
HyperAccel Runtime (hart) is the Resource Management & Execution Layer for HyperAccel LPUs (Language Processing Units). It manages device and memory resources and is responsible for kernel loading and dispatch and for stream- and event-based synchronous and asynchronous execution. It is designed to get the most out of the LPU's compute performance by using resources efficiently and minimizing execution overhead.
hart handles internally what multi-threaded, multi-device environments require: per-thread device
selection, memory-address-based device routing, and low-overhead memory allocation. Higher-level frameworks and
applications can therefore focus on building the flow of computation without handling the details of device
management themselves.
hart is the core interface that sits between higher-level AI frameworks and the kernel mode driver (KMD). Major
components of the HyperAccel AI software stack, such as the PyTorch backend (torch_ha), model serving
(vllm_hyperaccel), collective communication (lccl), and the profiler (hapti), run on hart. Software
that must work with devices directly through handles, such as custom runtimes or lightweight environments,
uses the device-handle-based Driver API (ha_drv_*).
Architecture Overview¶
Internally, hart is organized into the following five areas. Each feature is listed under
Core Features of hart below.
- Device & Routing Engine: Discovers, initializes, and resets hardware devices, and assigns virtual IDs sequentially from 0 only to devices that initialized successfully. It also tracks the per-thread current device and the device that owns each memory pointer, and routes each call to the right target device.
- Execution & Synchronization: Queues work per stream to provide synchronous and asynchronous execution, and synchronizes execution order and completion across streams through events. Kernels are looked up by name in a kernel library and launched, and each device loads the binary for its chip generation.
- Memory Management: Reuses freed device memory to reduce allocation overhead, and supports memory usage queries and release. It also manages host transfer buffers, virtual memory for the KV cache, and direct P2P (peer-to-peer) transfer between devices.
- Weight Conversion: Converts a model's weight tensors, together with their quantization metadata, into the device memory layout specific to the target architecture. This runs independently on the host without occupying device resources.
- Observability & Diagnostics: A standardized error code scheme clearly identifies the cause of a
failure, and execution diagnostics let the HyperAccel Profiler (
hapti) correlate and trace host-side calls with compute activity inside the device.
The Driver API bypasses these areas and works on devices directly through handles.
HyperAccel Runtime architecture and features in the software stack
Core Features of hart¶
The table below summarizes the main features of the areas above. For details, see the Programming Guide and the API Reference.
| Feature | Description |
|---|---|
| CUDA Runtime-Style API | Naming and usage flow follow the CUDA runtime, so developers familiar with CUDA can understand the API easily (for example, cudaMalloc → HaMemAlloc). |
| Device Discovery & Reset | Checks the available devices and resets one device or all of them. Devices that fail to initialize are skipped, and virtual IDs are assigned without gaps from 0 only to healthy devices. |
| Multi-Threaded Device Routing | Multiple host threads can each pick a device and use it independently, and frees and copies are forwarded automatically to the device the pointer belongs to. |
| Synchronous & Asynchronous Execution | Kernel launches and memory operations are available both synchronously and asynchronously, and streams and events control the ordering and completion of asynchronous work. |
| Kernel Library & Typed Launch | Kernels are looked up by name in a library and launched, and each device loads the build for its chip generation. Arguments are passed as a list that records each argument's kind and type, so a mismatch with the kernel's signature is rejected before launch (Kernel Library and Handle-Based Launch). |
| Memory Reuse, Statistics & Trim | Freed device memory is reused to reduce allocation overhead, and you can query usage or release memory that is not in use (Memory Statistics & Trim). |
| Host Memory Manager | Allocates and reuses host transfer buffers per process and enforces a usage limit (Host Memory). |
| Peer-to-Peer (P2P) Transfer & KV Cache Virtual Memory | Copies between devices are handled directly without going through the host, and virtual memory (VMM) for the KV cache manages the HW pages (Virtual Memory & HW Pages). |
| Weight Conversion | Converts weight tensors, together with their quantization metadata, into the device memory layout of the target architecture (Weight Conversion). |
| Standardized Error Codes | C++ reports HaError codes and Python raises a RuntimeError carrying the same code name, so failure causes can be distinguished by code. |
At a Glance¶
| Item | Specification |
|---|---|
| Package | hart |
| Version | 0.0.6 |
| Languages | C++20 core, Python bindings (pybind11) |
| Artifacts | C++ library (CMake package hart), hart Python module |
| Public headers | ha_runtime_api.h, ha_driver_api.h, ha_driver_types.h, ha_types.h, ha_error_code.h, ha_error.h, ha_vmm.h, ha_callback_params.h, ha_weight_converter.h, ha_weight_converter_types.h |
| Python | 3.10 – 3.12 |
| Status | Beta |
API Overview¶
hart provides a C++ API (Ha*) and a Python API (hart module) with the same functionality. Python names drop
the Ha prefix and use snake_case. Entry points by area are listed in the
API Reference, and the device-handle-based API is listed in the
Driver API Reference.
Installation¶
hart is not installed as a standalone package; it is distributed as a single HyperAccel SDK wheel package together
with the compiler, profiler, and framework integration components.
For the detailed installation guide and the required KMD (kernel mode driver) setup, see the SDK Installation Guide.
Compatibility¶
This documentation is written for hart 0.0.6.
| Category | Verified versions |
|---|---|
| Python | 3.10 – 3.12 |
| PyTorch | 2.10.0 |
| Device versions | Bertha Cloud EVT0, Bertha Cloud EVT1, Bertha Edge EVT0 |
For detailed specifications, see the Support Matrix.
Documentation¶
Programming Guide
What the runtime is, how execution and memory work, and how to write correct code.
Execution Model · Memory Model · Streams and Events · Multi-Device · Error Handling · Weight Conversion · Troubleshooting · Driver API Guide
API Reference
Exact signatures and semantics of every entry point.
C++ API · Python API · Driver API · Environment Variables · Error Codes · Support Matrix
Release Notes
Changes by version, deprecations, and known issues.