Skip to content

7. Error Handling

torch_ha reports errors as ordinary PyTorch exceptions. This chapter explains how to read them and where to find the underlying message.

Contents

Section Objective
7.1 lpu and PrivateUse1 Are the Same Thing How the two backend names in error messages relate
7.2 Kinds of Failure How to distinguish unregistered operators, unregistered devices, compile failures, and runtime failures
7.3 Registration Failures Are Logged How to check for import-time registration failures
7.4 Runtime Failures Reading RuntimeErrors raised below the dispatcher, and HAL error codes
7.5 Asynchrony and Where Errors Surface Why a traceback points at a synchronization point rather than the actual cause
7.6 Reporting Failures What to include in a report

7.1 lpu and PrivateUse1 Are the Same Thing

The backend uses PyTorch's PrivateUse1 dispatch key and renames it to lpu. Dispatch errors use the new name.

1
2
NotImplementedError: Could not run 'aten::<op>' with arguments from the
'lpu' backend.

Some low-level messages may still show the internal key name PrivateUse1. Treat it as lpu.

7.2 Kinds of Failure

Kind What it looks like See
Unregistered operator NotImplementedError mentioning an aten:: operator and the lpu backend Operators
Unregistered device An error reporting that lpu is an unknown device type §7.3
Compile failure Verification or compile errors from torch.compile(backend="legato") (UnsupportedOpError, VerificationError, and so on) Compile PyTorch Model with Legato
Runtime failure A RuntimeError raised below the dispatcher (allocation, layout, kernels) §7.4

7.3 Registration Failures Are Logged

torch_ha registers itself during import torch. If registration fails, the failure is logged, not raised. The reason is that an exception escaping PyTorch's backend autoloading would make import torch itself fail and cause unrelated failures on later import attempts.

As a result, a backend that failed to register looks exactly like one that is not installed: import torch succeeds, and torch.device("lpu") fails later.

Two lines are enough for a quick health check.

1
2
torch.accelerator.is_available()
torch.distributed.is_backend_available("lccl")

If lpu is an unknown device, read the log first

The actual cause is recorded at import time. Registration failures are written to stderr through the Python logger (loguru) and are visible with the default settings.

7.4 Runtime Failures

Errors raised below the dispatcher surface as a RuntimeError that carries the underlying message. The message includes the HAL error code together with its name and description, so each code explains itself, as in err=<value> (HaMemoryAllocationFailed): Memory allocation failed.

What the message mentions Cause See
Allocation, out of memory Device memory exhausted or fragmented hart memory model
Alignment, layout, stride A tensor layout the hardware cannot express Device Semantics
NotImplementedError: LPU host-pointer fallback … An eager compute operator with no device kernel was called, or a tensor whose layout prevents a single-piece copy was copied. Use the torch.compile path for computation Operators §3.1
No eager kernel took this op -- … The experimental eager AOT kernels did not accept the operator. Check the reason after -- and torch.ops.lpu.eager_status() Operators §3.1
kLccl… Collective failure LCCL error codes

If a tensor ends up on lpu:0 instead of the intended device, the thread allocated it before selecting a device. Device selection is per thread; see Distributed §4.2.

For the detailed meaning of a message, see the documentation of the layer that produced it (the See column above).

Telling a software problem from a device fault

You can extract the error codes from a log or an error message and classify them.

1
2
3
4
from torch_ha.error_triage import is_device_fault, scan

codes = scan(log_text)
any(is_device_fault(c) for c in codes)   # True means a device fault

7.5 Asynchrony and Where Errors Surface

Device execution is asynchronous with respect to the host, so an error may be reported by a later call instead of the one that caused it, usually at the next synchronization point such as .cpu() or .item().

The traceback points at the synchronization point, not the cause

A RuntimeError raised from out.cpu() often originates in an operation a few lines earlier. To narrow it down, insert a read after each suspect step so that each step has its own synchronization point.

1
2
h = layer1(x); h.cpu()
h = layer2(h); h.cpu()

Remove the reads once you have located the failure; each one costs a host round trip.

7.6 Reporting Failures

Include the following in the report:

  1. The exact exception and traceback
  2. The log written to stderr from import onward
  3. torch.__version__ and the SDK version
  4. A minimal reproduction, and whether it also reproduces on the CPU

Next Steps