8. Troubleshooting¶
This page lists failures by observable symptom. Search for the exact string from your traceback.
Contents¶
| Section | Objective |
|---|---|
| Before You Start | Initial checks of versions and registration status |
| Device | lpu device not registered, backend library fails to load |
| Operators | Unregistered operators, eager compute failures, directly created random tensors, view and reshape failures |
| Results | Results that differ from the CPU, all-zero outputs |
| Performance | Slow inference, multi-device throughput that does not scale |
| Distributed | Rendezvous hangs, kLcclNotSupported, collective hangs |
| Paged KV Cache | KV cache initialization failures |
| Getting Support | What to attach when asking for help |
Before You Start¶
1 2 3 4 | |
1 | |
Registration failures are recorded in the import-time log. See Error Handling.
Device¶
The lpu Device Does Not Exist¶
Symptom — torch.device("lpu") or .to("lpu") fails with an unknown device type.
Cause — The backend did not register. Registration happens during import torch, and failures are logged rather
than raised, so the result looks exactly like a missing package.
Fix
1 | |
The log shows the actual cause. Common causes are a missing libha_backend.so or one of its dependencies (next
entry), and a PyTorch version mismatch.
If TORCH_DEVICE_BACKEND_AUTOLOAD=0 is set, registration never runs and nothing is logged, so unset the variable.
Failed to load torch_ha aten backend: Could not load this library¶
Symptom — During import torch, the log shows Failed to load torch_ha aten backend: Could not load this library:
…/torch_ha/libha_backend.so, and the lpu device is unavailable afterward.
Cause — The shared libraries of the hart package that libha_backend.so depends on (libha_runtime.so,
libha_umd.so, libha_weight_converter.so) could not be found. libha_backend.so looks for them only in its RUNPATH,
$ORIGIN/../hart, which is the hart directory installed next to torch_ha in the same site-packages. If hart
is installed somewhere else (another site-packages, a user directory, and so on), import hart can succeed in Python
while the dynamic linker still cannot find those libraries.
Fix — First, check which library is missing.
1 | |
The default fix is to install hart into the same site-packages as torch_ha. If you cannot, add the hart
package directory to LD_LIBRARY_PATH.
1 | |
PYTHONPATH affects only Python's module search and is not read by the dynamic linker, so setting it does not help.
LD_LIBRARY_PATH is read when a process starts, so set it in the shell and then launch a new process.
import torch Fails with a Message About the triton Namespace¶
Symptom — Only a single TORCH_LIBRARY can be used to register the namespace triton. This usually happens on the
second import in a test session.
Cause — An earlier import torch failed partway through and Python removed the module. The re-import then re-ran
registrations that the C++ registry still held.
Fix — The first failure is the actual cause. To find it, import only torch in a fresh process and read the first failure in the log.
Operators¶
Could not run 'aten::<op>' with arguments from the 'lpu' backend¶
Cause — The operator is not a registered operator for the LPU device. Some low-level messages may show the backend
under its internal key name, PrivateUse1; it is the same backend.
Fix — Check Operator Coverage. Rewrite the code with operators that
compile, or handle the operator explicitly by moving the data to the host with .cpu() and back.
Find all gaps up front instead of hitting them one at a time. The snippet is in Operators.
LPU host-pointer fallback … is unsupported or No eager kernel took this op¶
Cause — A compute operator with no device kernel ran in eager mode. In eager mode, add, sub, mul, and mm
on bfloat16 compute on the device, when a prebuilt kernel exists for the shape and every operand is contiguous. The
reason reads <name> is not contiguous for a non-contiguous operand, or
operand <i> is neither a whole tensor nor a view laid out and addressed as one for a slice that device compute does not take as is.
Fix — Run model computation with torch.compile(backend="legato"). If one of those four operators fails, check the
reason after No eager kernel took this op -- and torch.ops.lpu.eager_status(). For the first, call .contiguous()
on the non-contiguous view (a transpose, permute, or expand, or a stepped slice like x[::2]); for the second,
call .clone() on a slice that device compute does not take as is (Device Semantics §Slicing). See Operators §3.1.
A Random Tensor Created Directly on lpu Fails¶
Symptom — torch.randn(..., device="lpu") or torch.rand(..., device="lpu") raises NotImplementedError (aten::normal_ / aten::uniform_).
Cause — Random operators (such as aten::normal_) do not run eagerly on lpu.
Fix — Create random tensors on the CPU and move them, as in
torch.randn(32, 128, dtype=torch.bfloat16).to("lpu"). Value factories (zeros, ones, full, arange, empty)
work as is with device="lpu" (the current device) or an explicit index.
view or reshape Fails on a Tensor That Was Working¶
Cause — The backend stores the device layout with the tensor's storage. An operation that reinterprets the storage in a way the layout cannot express fails at that view, but the cause lies in how the tensor was created.
Fix — Trace back to where the tensor was created. Weights moved as part of nn.Linear or nn.Embedding have a
specific layout; a standalone weight tensor moved directly with .to("lpu") does not.
See Device Semantics.
Results¶
Results Differ from the CPU¶
Cause — Usually the dtype, not a bug. Comparing bfloat16 on the device with float32 on the host measures the
dtype difference. Reduction order also differs, and the resulting differences accumulate in deep models.
Fix
- Match the dtype on both sides before comparing.
- Use a tolerance that fits the dtype — roughly
1e-2forbfloat16. - If the difference is larger than that, compare layer by layer instead of only the final logits. Report the first layer that diverges.
The Output Is All Zeros¶
Cause — Often an operation produced an empty result without an error and the buffer was never written.
Fix — Read the value with .cpu() at the point you check it, and confirm that the shape at each step matches what
you expect.
Performance¶
Inference Is Much Slower Than Expected¶
Check in this order.
- Timing without a synchronization point.
model(x)returns before the device finishes. Includeout.cpu()in the measurement, or profile withhaprof. - First-run cost. Kernel binaries are loaded the first time they are used on each device. Warm up before measuring.
- CPU round trips inside the loop. Every
.cpu(),.item(), andprintis both a synchronization point and a transfer. A single one inside the token loop can dominate the total time. - Steps left on the CPU as a coverage workaround. Explicit fallbacks are easy to forget once they work.
Multi-Device Throughput Does Not Scale¶
Cause — Usually a collective bottleneck. Check how the underlying library runs the collectives that your parallelism strategy uses on the hot path.
Fix — See Distributed and the LCCL support matrix.
Distributed¶
init_process_group Hangs¶
Cause — Not all ranks reached the rendezvous, or world_size differs across ranks.
Fix — Check MASTER_ADDR, MASTER_PORT, RANK, and WORLD_SIZE on every rank, and confirm that rank 0 is
reachable. The full procedure is in LCCL Troubleshooting.
A Collective Raises kLcclNotSupported¶
Cause — There is no kernel for that combination of operation, reduction, and dtype. With 2 or more ranks, MAX
and MIN have no kernel for any operation, and reduce_scatter supports SUM only. Collectives that perform a
reduction take float16 and bfloat16 only, and float32 is rejected as well.
Fix — See the LCCL support matrix.
All Ranks Hang in a Collective¶
Cause — The ranks called collectives in different orders. Collectives are matched across ranks by call order, so a rank that skips one on an empty batch breaks synchronization for the whole group.
Fix — Log the collective name and sequence number on every rank. The first point where they diverge is the bug.
Paged KV Cache¶
vmm_init_blocks Fails with HaMemoryAllocationFailed¶
Symptom — torch.ops.lpu.vmm_init_blocks fails with HaVmmInitBlocks failed: err=…, where the code is the numeric value of HaMemoryAllocationFailed.
Cause — The pool is smaller than the KEY and VALUE pages of one request. The maximum number of tokens per request
is max_num_blocks_per_row × block_size, and the pool must hold both KEY and VALUE for that many tokens.
Fix — Increase the token budget (num_blocks) of vmm_create_kv_pool, or reduce the maximum number of tokens per
request. For the call order, see Paged KV Cache §5.3.
Getting Support¶
Contact HyperAccel support and include the following:
- The exception and full traceback
- The log written to stderr from import onward
torch.__version__and the SDK version- A minimal reproduction, and whether it also reproduces on the CPU