Skip to content

7. Troubleshooting

A list of failures indexed by observed symptom. Before reasoning about the cause, search this page for the exact string or error code printed in the log.

Each entry has three parts: Symptom · Cause · Action. The cause is included so that you have a basis for your own judgment when you hit a variant that is not listed here.

Contents

Section Objective
Before You Start What to collect before asking for support
Startup Failures on the first call or in a newly created thread
Memory Allocation is rejected or data is wrong
VMM KV cache HW page initialization fails or results are corrupted after mapping
Execution A kernel is not found or a launch fails
Hangs A call does not return

Most of the entries here are properties described in Chapter 2–Chapter 5 showing up as actual symptoms. If the same symptom keeps recurring, rereading the relevant chapter is faster.

Code definitions are in Error codes, and the criteria for categorizing failures are in Error Handling.


Before You Start

When asking for support, collect the following. It answers most of the questions we would ask.

1
2
3
import hart

print("device count :", hart.get_device_count())
1
2
3
python -c "import hart, os; print(os.path.dirname(hart.__file__))"
lsmod | grep ha
ls -l /dev/ha*

Startup

HaDeviceNotAvailable on the first call

Symptom — get_device_count() (C++: HaGetDeviceCount()) or the first device call fails with HaDeviceNotAvailable (Python: RuntimeError).

Cause — No device is visible to the process. The KMD is not loaded, or the process cannot access the device node.

Action

1
2
lsmod | grep ha          # Is ha.ko loaded?
ls -l /dev/ha*           # Does the node exist, and can this user open it?

If the module is missing, see the driver documentation. If the node exists but the user cannot read it, it is a permissions problem, not a runtime problem.

HaDeviceInvalidArgument only on worker threads

Symptom — A code path that works when called from the main thread fails on a thread pool thread.

Cause — The current device is set separately for each thread. The worker thread has never called set_device() (C++: HaSetDevice()).

Action — Call hart.set_device(id) at the start of every thread that uses the runtime, including threads created by frameworks or thread pools. See Multi-Device Programming.

Rebuilt runtime is not picked up

Symptom — You modified and rebuilt runtime, but behavior is unchanged when running through torch_ha.

Cause — torch_ha bundles its own copy of the runtime shared library. Rebuilding only runtime leaves that copy stale.

Action — After changing runtime, rebuild torch_ha as well.


Memory

HaMemoryAllocationFailed while device memory remains

Symptom — Allocation fails even though the total allocated amount is far below the device capacity.

Cause — Usually fragmentation, not exhaustion. Freed blocks remain in the runtime pool and are reused by size, so after a long sequence of allocations of varying sizes, no single block large enough may remain.

Action — Allocate large, long-lived buffers once at startup, not per request. If a test loop really needs a clean state, reset the device between iterations, but since a reset invalidates handles, discard all pointers and streams first.

HaMemoryAlignmentError or layout rejected

Symptom — build_device_tensor_layout() (C++: HaBuildDeviceTensorLayout()) or mem_cpy_nd() (C++: HaMemCpyNd()) fails with an alignment or invalid-argument code.

Cause — The layout violates a DRAM alignment constraint. For interleaved kinds, the per-channel chunk must be an integer multiple of the DRAM word size (Bertha Cloud 64 B, Bertha Edge 32 B). SinglePassInterleaved is stricter. For MPUTensor and the KV cache kinds, head_dim must be divisible by channels_per_token * elements_per_word.

Action — Check the constraints of the HaTensorKind you are using in Memory Model. These are hardware requirements, so the runtime rejects the layout instead of silently padding it.

HaMemoryInvalidArgument from HaMemCpyNd()

Cause — Almost always a rank or shape mismatch between the two layouts. This API does not broadcast.

Action — Before the call, re-view one side so that rank and shape match. Also check that strides are in elements, not bytes. This mistake produces wrong data, not an error.

Data is wrong after an asynchronous copy

Symptom — Results are intermittently wrong, and more often under higher load.

Cause — The host buffer was modified or read before the copy completed. The return of mem_cpy_async() (C++: HaMemCpyAsync()) does not mean the copy has finished.

Action — Before touching the host buffer, synchronize the stream, or record an event and wait on it. This class of bug depends on load, so it can pass hundreds of times and fail once. A passing test is not evidence that the synchronization is correct.


VMM

HaVmmInitBlocks() returns HaMemoryAllocationFailed

Symptom — Pool allocation succeeds, but HaVmmInitBlocks() fails with HaMemoryAllocationFailed.

Cause — pool_bytes is smaller than the KEY+VALUE pages of one request (based on maximum tokens = max_num_blocks_per_row × block_size). The runtime derives the maximum number of HW pages from the pool size, and rejects a pool that cannot hold even one request as a configuration error.

Action — Enlarge the pool or reduce max_num_blocks_per_row × block_size (maximum tokens per request). The pool must hold both the K and V halves. If pool_base is not 4 KiB aligned, HaMemoryInvalidArgument is returned, so allocate the pool with alignment=4096 and pass segments[0].pa_start.

Kernel results are corrupted after changing mappings

Symptom — In a serving loop that maps and unmaps request pages every step, attention results are intermittently wrong, and more often with larger batches.

Cause — The mapping was changed while the previous step's kernel was still running. Map and unmap do not wait for running kernels, so a running kernel reads the changed translation. The same symptom also appears when HaVmmFreePool() is called while request pages are still mapped and the physical memory is reused by another allocation.

Action — Before changing mappings, synchronize the previous step's stream or confirm completion with an event. Free the pool only after unmapping all requests. The detailed rules are in Memory Model: Rules.


Execution

HaKernelNotFound

Cause — The kernel binary path cannot be resolved.

Action — Check that the path exists relative to the process's working directory. Kernel metadata is cached per device, so a kernel loaded successfully on device 0 is loaded again the first time it is used on device 1.

HaKernelExecutionFailed

Cause — The kernel ran and failed, or the device faulted. This code alone does not tell you which.

Action — Do not retry the launch in place. Device state after this error is not guaranteed. Check whether it reproduces with the same input, then report the kernel binary together with the input.

Only the first launch is slow

Cause — This is not a defect. A kernel binary is loaded and cached per device on its first launch.

Action — Warm up before measuring. See Loading & Caching Behavior in Execution Model.


Hangs

Stuck in HaStreamSynchronize()

Cause — Work on the stream cannot complete. Common cases are a cross-stream dependency that waits on an event that was never recorded, and a kernel that does not terminate.

Action

  1. Replace stream_synchronize() (C++: HaStreamSynchronize()) with stream_query() (C++: HaStreamQuery()) polling to check whether it is really stuck or just slow.
  2. Check that every stream_wait_event() (C++: HaStreamWaitEvent()) has a matching event_record() (C++: HaEventRecord()). An event that is only waited on and never recorded blocks the waiting stream forever.
  3. Configure logging to a file (Error Handling) and collect the logs.

Two threads deadlock on the same device

Cause — A common cause is assuming that HA_STREAM_DEFAULT is serialized across threads. It is not. The default stream is per thread.

Action — Share an explicit stream or order the work with events. See Streams and Events.


Getting Support

If the entries above do not resolve the problem, contact HyperAccel support with the following attached.

  1. Diagnostic information from Before You Start
  2. Runtime logs (Error Handling: Logging)
  3. Minimal reproduction code