Python API¶
1 | |
The semantics are the same as the C++ API; only the differences are listed in Differences from the C++ API.
Device pointers are int.
Sections¶
| Page | Contents |
|---|---|
| Device | Device discovery, selection, and reset |
| Memory | Allocation, transfers, tensor layout, host and device memory statistics |
| Execution | Kernel launch, kernel library loading, and launch by handle |
| Streams | Ordered queues of asynchronous work |
| Events | Stream markers and cross-stream ordering |
| TLB | Static KV-cache TLB initialization and page management |
| VMM | KV-cache virtual memory. On Bertha DC EVT0, use the C++ API or PyTorch operators |
| Sampling | Token sampling parameters |
| Weight Converter | Converts weight tensors to the device memory layout (hart.weight_converter) |
| Types | Enumerations and structs. Import the TLB types from hart.hart |
Differences from the C++ API¶
| C++ | Python | |
|---|---|---|
| Failure reporting | Returns HaError |
Raises RuntimeError whose message ends in the HaError name and value (... : HaKernelNotFound (err=196612)) |
| Device pointers | HaDevicePtr |
int |
| Default stream | HA_STREAM_DEFAULT (nullptr) |
HA_STREAM_DEFAULT (0) |
| Host buffers | HaDevicePtr cast from a host pointer |
bytes or any buffer-protocol object |
HaLaunchKernel overloads |
Overloaded on binary path vs kernel handle (HaKernel_t) |
Separate functions: launch_kernel_with_binary_path, launch_kernel_with_handle (and their _async_ forms) |
| Library and kernel handles | HaLibrary_t, HaKernel_t |
int |
| Launch arguments | HaLaunchArgs object filled with HaLaunchArgsAdd*(), released when destroyed |
HaLaunchArgs object (add_*), released with the object |
| Memory statistics | HaHostMemStats / HaDeviceMemStats struct with struct_size |
dict[str, int] keyed by field name |
Process Exit¶
hart registers an atexit hook that calls reset_all_devices(), but the hook runs only if the runtime
was actually initialized. A process that imports hart but never uses a device exits without opening the device.
Do not register an atexit handler that runs after this hook and touches the runtime.