Skip to content

8. Troubleshooting

This page lists failures by observable symptom. Search for the exact string from your traceback.

Contents

Section Objective
Before You Start Initial checks of versions and registration status
Device lpu device not registered, backend library fails to load
Operators Unregistered operators, eager compute failures, directly created random tensors, view and reshape failures
Results Results that differ from the CPU, all-zero outputs
Performance Slow inference, multi-device throughput that does not scale
Distributed Rendezvous hangs, kLcclNotSupported, collective hangs
Paged KV Cache KV cache initialization failures
Getting Support What to attach when asking for help

Before You Start

1
2
3
4
import torch
print("torch      :", torch.__version__)
print("lpu module :", hasattr(torch, "lpu"))
print("available  :", torch.accelerator.is_available())
1
python -c "import torch; print(torch.zeros(1).to('lpu'))" 2>&1 | head -40

Registration failures are recorded in the import-time log. See Error Handling.


Device

The lpu Device Does Not Exist

Symptom — torch.device("lpu") or .to("lpu") fails with an unknown device type.

Cause — The backend did not register. Registration happens during import torch, and failures are logged rather than raised, so the result looks exactly like a missing package.

Fix

1
python -c "import torch" 2>&1 | grep -i "torch_ha\|backend"

The log shows the actual cause. Common causes are a missing libha_backend.so or one of its dependencies (next entry), and a PyTorch version mismatch. If TORCH_DEVICE_BACKEND_AUTOLOAD=0 is set, registration never runs and nothing is logged, so unset the variable.

Failed to load torch_ha aten backend: Could not load this library

Symptom — During import torch, the log shows Failed to load torch_ha aten backend: Could not load this library: …/torch_ha/libha_backend.so, and the lpu device is unavailable afterward.

Cause — The shared libraries of the hart package that libha_backend.so depends on (libha_runtime.so, libha_umd.so, libha_weight_converter.so) could not be found. libha_backend.so looks for them only in its RUNPATH, $ORIGIN/../hart, which is the hart directory installed next to torch_ha in the same site-packages. If hart is installed somewhere else (another site-packages, a user directory, and so on), import hart can succeed in Python while the dynamic linker still cannot find those libraries.

Fix — First, check which library is missing.

1
ldd "$(python -c 'import torch_ha, os; print(os.path.dirname(torch_ha.__file__))')/libha_backend.so" | grep "not found"

The default fix is to install hart into the same site-packages as torch_ha. If you cannot, add the hart package directory to LD_LIBRARY_PATH.

1
export LD_LIBRARY_PATH="$(python -c 'import hart, os; print(os.path.dirname(hart.__file__))'):$LD_LIBRARY_PATH"

PYTHONPATH affects only Python's module search and is not read by the dynamic linker, so setting it does not help. LD_LIBRARY_PATH is read when a process starts, so set it in the shell and then launch a new process.

import torch Fails with a Message About the triton Namespace

Symptom — Only a single TORCH_LIBRARY can be used to register the namespace triton. This usually happens on the second import in a test session.

Cause — An earlier import torch failed partway through and Python removed the module. The re-import then re-ran registrations that the C++ registry still held.

Fix — The first failure is the actual cause. To find it, import only torch in a fresh process and read the first failure in the log.


Operators

Could not run 'aten::<op>' with arguments from the 'lpu' backend

Cause — The operator is not a registered operator for the LPU device. Some low-level messages may show the backend under its internal key name, PrivateUse1; it is the same backend.

Fix — Check Operator Coverage. Rewrite the code with operators that compile, or handle the operator explicitly by moving the data to the host with .cpu() and back.

Find all gaps up front instead of hitting them one at a time. The snippet is in Operators.

LPU host-pointer fallback … is unsupported or No eager kernel took this op

Cause — A compute operator with no device kernel ran in eager mode. In eager mode, add, sub, mul, and mm on bfloat16 compute on the device, when a prebuilt kernel exists for the shape and every operand is contiguous. The reason reads <name> is not contiguous for a non-contiguous operand, or operand <i> is neither a whole tensor nor a view laid out and addressed as one for a slice that device compute does not take as is.

Fix — Run model computation with torch.compile(backend="legato"). If one of those four operators fails, check the reason after No eager kernel took this op -- and torch.ops.lpu.eager_status(). For the first, call .contiguous() on the non-contiguous view (a transpose, permute, or expand, or a stepped slice like x[::2]); for the second, call .clone() on a slice that device compute does not take as is (Device Semantics §Slicing). See Operators §3.1.

A Random Tensor Created Directly on lpu Fails

Symptom — torch.randn(..., device="lpu") or torch.rand(..., device="lpu") raises NotImplementedError (aten::normal_ / aten::uniform_).

Cause — Random operators (such as aten::normal_) do not run eagerly on lpu.

Fix — Create random tensors on the CPU and move them, as in torch.randn(32, 128, dtype=torch.bfloat16).to("lpu"). Value factories (zeros, ones, full, arange, empty) work as is with device="lpu" (the current device) or an explicit index.

view or reshape Fails on a Tensor That Was Working

Cause — The backend stores the device layout with the tensor's storage. An operation that reinterprets the storage in a way the layout cannot express fails at that view, but the cause lies in how the tensor was created.

Fix — Trace back to where the tensor was created. Weights moved as part of nn.Linear or nn.Embedding have a specific layout; a standalone weight tensor moved directly with .to("lpu") does not. See Device Semantics.


Results

Results Differ from the CPU

Cause — Usually the dtype, not a bug. Comparing bfloat16 on the device with float32 on the host measures the dtype difference. Reduction order also differs, and the resulting differences accumulate in deep models.

Fix

  1. Match the dtype on both sides before comparing.
  2. Use a tolerance that fits the dtype — roughly 1e-2 for bfloat16.
  3. If the difference is larger than that, compare layer by layer instead of only the final logits. Report the first layer that diverges.

The Output Is All Zeros

Cause — Often an operation produced an empty result without an error and the buffer was never written.

Fix — Read the value with .cpu() at the point you check it, and confirm that the shape at each step matches what you expect.


Performance

Inference Is Much Slower Than Expected

Check in this order.

  1. Timing without a synchronization point. model(x) returns before the device finishes. Include out.cpu() in the measurement, or profile with haprof.
  2. First-run cost. Kernel binaries are loaded the first time they are used on each device. Warm up before measuring.
  3. CPU round trips inside the loop. Every .cpu(), .item(), and print is both a synchronization point and a transfer. A single one inside the token loop can dominate the total time.
  4. Steps left on the CPU as a coverage workaround. Explicit fallbacks are easy to forget once they work.

Multi-Device Throughput Does Not Scale

Cause — Usually a collective bottleneck. Check how the underlying library runs the collectives that your parallelism strategy uses on the hot path.

Fix — See Distributed and the LCCL support matrix.


Distributed

init_process_group Hangs

Cause — Not all ranks reached the rendezvous, or world_size differs across ranks.

Fix — Check MASTER_ADDR, MASTER_PORT, RANK, and WORLD_SIZE on every rank, and confirm that rank 0 is reachable. The full procedure is in LCCL Troubleshooting.

A Collective Raises kLcclNotSupported

Cause — There is no kernel for that combination of operation, reduction, and dtype. With 2 or more ranks, MAX and MIN have no kernel for any operation, and reduce_scatter supports SUM only. Collectives that perform a reduction take float16 and bfloat16 only, and float32 is rejected as well.

Fix — See the LCCL support matrix.

All Ranks Hang in a Collective

Cause — The ranks called collectives in different orders. Collectives are matched across ranks by call order, so a rank that skips one on an empty batch breaks synchronization for the whole group.

Fix — Log the collective name and sequence number on every rank. The first point where they diverge is the bug.

Paged KV Cache

vmm_init_blocks Fails with HaMemoryAllocationFailed

Symptom — torch.ops.lpu.vmm_init_blocks fails with HaVmmInitBlocks failed: err=…, where the code is the numeric value of HaMemoryAllocationFailed.

Cause — The pool is smaller than the KEY and VALUE pages of one request. The maximum number of tokens per request is max_num_blocks_per_row × block_size, and the pool must hold both KEY and VALUE for that many tokens.

Fix — Increase the token budget (num_blocks) of vmm_create_kv_pool, or reduce the maximum number of tokens per request. For the call order, see Paged KV Cache §5.3.


Getting Support

Contact HyperAccel support and include the following:

  1. The exception and full traceback
  2. The log written to stderr from import onward
  3. torch.__version__ and the SDK version
  4. A minimal reproduction, and whether it also reproduces on the CPU