7. Troubleshooting¶
A list of failures indexed by observed symptom. Before reasoning about the cause, search this page for the exact string or error code printed in the log.
Each entry has three parts: Symptom · Cause · Action. The cause is included so that you have a basis for your own judgment when you hit a variant that is not listed here.
Contents¶
| Section | Objective |
|---|---|
| Before You Start | What to collect before asking for support |
| Startup | Failures on the first call or in a newly created thread |
| Memory | Allocation is rejected or data is wrong |
| VMM | KV cache HW page initialization fails or results are corrupted after mapping |
| Execution | A kernel is not found or a launch fails |
| Hangs | A call does not return |
Most of the entries here are properties described in Chapter 2–Chapter 5 showing up as actual symptoms. If the same symptom keeps recurring, rereading the relevant chapter is faster.
Code definitions are in Error codes, and the criteria for categorizing failures are in Error Handling.
Before You Start¶
When asking for support, collect the following. It answers most of the questions we would ask.
1 2 3 | |
1 2 3 | |
Startup¶
HaDeviceNotAvailable on the first call¶
Symptom — get_device_count() (C++: HaGetDeviceCount()) or the first
device call fails with HaDeviceNotAvailable (Python: RuntimeError).
Cause — No device is visible to the process. The KMD is not loaded, or the process cannot access the device node.
Action
1 2 | |
If the module is missing, see the driver documentation. If the node exists but the user cannot read it, it is a permissions problem, not a runtime problem.
HaDeviceInvalidArgument only on worker threads¶
Symptom — A code path that works when called from the main thread fails on a thread pool thread.
Cause — The current device is set separately for each thread. The worker thread has never called
set_device() (C++: HaSetDevice()).
Action — Call hart.set_device(id) at the start of every thread that uses the runtime,
including threads created by frameworks or thread pools.
See Multi-Device Programming.
Rebuilt runtime is not picked up¶
Symptom — You modified and rebuilt runtime, but behavior is unchanged when running through torch_ha.
Cause — torch_ha bundles its own copy of the runtime shared library.
Rebuilding only runtime leaves that copy stale.
Action — After changing runtime, rebuild torch_ha as well.
Memory¶
HaMemoryAllocationFailed while device memory remains¶
Symptom — Allocation fails even though the total allocated amount is far below the device capacity.
Cause — Usually fragmentation, not exhaustion. Freed blocks remain in the runtime pool and are reused by size, so after a long sequence of allocations of varying sizes, no single block large enough may remain.
Action — Allocate large, long-lived buffers once at startup, not per request. If a test loop really needs a clean state, reset the device between iterations, but since a reset invalidates handles, discard all pointers and streams first.
HaMemoryAlignmentError or layout rejected¶
Symptom — build_device_tensor_layout() (C++: HaBuildDeviceTensorLayout()) or
mem_cpy_nd() (C++: HaMemCpyNd()) fails with an alignment or invalid-argument code.
Cause — The layout violates a DRAM alignment constraint. For interleaved kinds, the per-channel chunk must be
an integer multiple of the DRAM word size (Bertha Cloud 64 B, Bertha Edge 32 B).
SinglePassInterleaved is stricter. For MPUTensor and the KV cache kinds, head_dim must be
divisible by channels_per_token * elements_per_word.
Action — Check the constraints of the HaTensorKind you are using in
Memory Model. These are hardware requirements, so the runtime
rejects the layout instead of silently padding it.
HaMemoryInvalidArgument from HaMemCpyNd()¶
Cause — Almost always a rank or shape mismatch between the two layouts. This API does not broadcast.
Action — Before the call, re-view one side so that rank and shape match. Also check that strides are in elements, not bytes. This mistake produces wrong data, not an error.
Data is wrong after an asynchronous copy¶
Symptom — Results are intermittently wrong, and more often under higher load.
Cause — The host buffer was modified or read before the copy completed.
The return of mem_cpy_async() (C++: HaMemCpyAsync()) does not mean the copy has
finished.
Action — Before touching the host buffer, synchronize the stream, or record an event and wait on it. This class of bug depends on load, so it can pass hundreds of times and fail once. A passing test is not evidence that the synchronization is correct.
VMM¶
HaVmmInitBlocks() returns HaMemoryAllocationFailed¶
Symptom — Pool allocation succeeds, but HaVmmInitBlocks() fails with HaMemoryAllocationFailed.
Cause — pool_bytes is smaller than the KEY+VALUE pages of one request (based on maximum tokens =
max_num_blocks_per_row × block_size). The runtime derives the maximum number of HW pages from the pool
size, and rejects a pool that cannot hold even one request as a configuration error.
Action — Enlarge the pool or reduce max_num_blocks_per_row × block_size (maximum tokens per request). The pool
must hold both the K and V halves. If pool_base is not 4 KiB aligned, HaMemoryInvalidArgument is returned, so
allocate the pool with alignment=4096 and pass segments[0].pa_start.
Kernel results are corrupted after changing mappings¶
Symptom — In a serving loop that maps and unmaps request pages every step, attention results are intermittently wrong, and more often with larger batches.
Cause — The mapping was changed while the previous step's kernel was still running. Map and unmap do not
wait for running kernels, so a running kernel reads the changed translation. The same symptom also appears when
HaVmmFreePool() is called while request pages are still mapped and the
physical memory is reused by another allocation.
Action — Before changing mappings, synchronize the previous step's stream or confirm completion with an event. Free the pool only after unmapping all requests. The detailed rules are in Memory Model: Rules.
Execution¶
HaKernelNotFound¶
Cause — The kernel binary path cannot be resolved.
Action — Check that the path exists relative to the process's working directory. Kernel metadata is cached per device, so a kernel loaded successfully on device 0 is loaded again the first time it is used on device 1.
HaKernelExecutionFailed¶
Cause — The kernel ran and failed, or the device faulted. This code alone does not tell you which.
Action — Do not retry the launch in place. Device state after this error is not guaranteed. Check whether it reproduces with the same input, then report the kernel binary together with the input.
Only the first launch is slow¶
Cause — This is not a defect. A kernel binary is loaded and cached per device on its first launch.
Action — Warm up before measuring. See Loading & Caching Behavior in Execution Model.
Hangs¶
Stuck in HaStreamSynchronize()¶
Cause — Work on the stream cannot complete. Common cases are a cross-stream dependency that waits on an event that was never recorded, and a kernel that does not terminate.
Action
- Replace
stream_synchronize()(C++:HaStreamSynchronize()) withstream_query()(C++:HaStreamQuery()) polling to check whether it is really stuck or just slow. - Check that every
stream_wait_event()(C++:HaStreamWaitEvent()) has a matchingevent_record()(C++:HaEventRecord()). An event that is only waited on and never recorded blocks the waiting stream forever. - Configure logging to a file (Error Handling) and collect the logs.
Two threads deadlock on the same device¶
Cause — A common cause is assuming that HA_STREAM_DEFAULT is serialized across threads.
It is not. The default stream is per thread.
Action — Share an explicit stream or order the work with events. See Streams and Events.
Getting Support¶
If the entries above do not resolve the problem, contact HyperAccel support with the following attached.
- Diagnostic information from Before You Start
- Runtime logs (Error Handling: Logging)
- Minimal reproduction code