7. Error Handling¶
torch_ha reports errors as ordinary PyTorch exceptions. This chapter explains how to read them and where to find the
underlying message.
Contents¶
| Section | Objective |
|---|---|
7.1 lpu and PrivateUse1 Are the Same Thing |
How the two backend names in error messages relate |
| 7.2 Kinds of Failure | How to distinguish unregistered operators, unregistered devices, compile failures, and runtime failures |
| 7.3 Registration Failures Are Logged | How to check for import-time registration failures |
| 7.4 Runtime Failures | Reading RuntimeErrors raised below the dispatcher, and HAL error codes |
| 7.5 Asynchrony and Where Errors Surface | Why a traceback points at a synchronization point rather than the actual cause |
| 7.6 Reporting Failures | What to include in a report |
7.1 lpu and PrivateUse1 Are the Same Thing¶
The backend uses PyTorch's PrivateUse1 dispatch key and renames it to lpu. Dispatch errors use the new name.
1 2 | |
Some low-level messages may still show the internal key name PrivateUse1. Treat it as lpu.
7.2 Kinds of Failure¶
| Kind | What it looks like | See |
|---|---|---|
| Unregistered operator | NotImplementedError mentioning an aten:: operator and the lpu backend |
Operators |
| Unregistered device | An error reporting that lpu is an unknown device type |
§7.3 |
| Compile failure | Verification or compile errors from torch.compile(backend="legato") (UnsupportedOpError, VerificationError, and so on) |
Compile PyTorch Model with Legato |
| Runtime failure | A RuntimeError raised below the dispatcher (allocation, layout, kernels) |
§7.4 |
7.3 Registration Failures Are Logged¶
torch_ha registers itself during import torch. If registration fails, the failure is logged, not raised.
The reason is that an exception escaping PyTorch's backend autoloading would make import torch itself fail and cause
unrelated failures on later import attempts.
As a result, a backend that failed to register looks exactly like one that is not installed: import torch
succeeds, and torch.device("lpu") fails later.
Two lines are enough for a quick health check.
1 2 | |
If lpu is an unknown device, read the log first
The actual cause is recorded at import time. Registration failures are written to stderr through the Python logger (loguru) and are visible with the default settings.
7.4 Runtime Failures¶
Errors raised below the dispatcher surface as a RuntimeError that carries the underlying message. The message includes
the HAL error code together with its name and description, so each code explains itself, as in
err=<value> (HaMemoryAllocationFailed): Memory allocation failed.
| What the message mentions | Cause | See |
|---|---|---|
| Allocation, out of memory | Device memory exhausted or fragmented | hart memory model |
| Alignment, layout, stride | A tensor layout the hardware cannot express | Device Semantics |
NotImplementedError: LPU host-pointer fallback … |
An eager compute operator with no device kernel was called, or a tensor whose layout prevents a single-piece copy was copied. Use the torch.compile path for computation |
Operators §3.1 |
No eager kernel took this op -- … |
The experimental eager AOT kernels did not accept the operator. Check the reason after -- and torch.ops.lpu.eager_status() |
Operators §3.1 |
kLccl… |
Collective failure | LCCL error codes |
If a tensor ends up on lpu:0 instead of the intended device, the thread allocated it before selecting a device.
Device selection is per thread; see Distributed §4.2.
For the detailed meaning of a message, see the documentation of the layer that produced it (the See column above).
Telling a software problem from a device fault
You can extract the error codes from a log or an error message and classify them.
1 2 3 4 | |
7.5 Asynchrony and Where Errors Surface¶
Device execution is asynchronous with respect to the host, so an error may be reported by a later call instead of the
one that caused it, usually at the next synchronization point such as .cpu() or .item().
The traceback points at the synchronization point, not the cause
A RuntimeError raised from out.cpu() often originates in an operation a few lines earlier. To narrow it down,
insert a read after each suspect step so that each step has its own synchronization point.
1 2 | |
Remove the reads once you have located the failure; each one costs a host round trip.
7.6 Reporting Failures¶
Include the following in the report:
- The exact exception and traceback
- The log written to stderr from import onward
torch.__version__and the SDK version- A minimal reproduction, and whether it also reproduces on the CPU
Next Steps¶
- Troubleshooting: index by symptom
- Operators