3. Operators¶
This chapter describes which operators run on the LPU, how operators that do not run fail, and how to check in advance the operators your model uses. Per-operator support is listed in Operator Coverage.
Contents¶
| Section | Objective |
|---|---|
| 3.1 Two Execution Paths | The roles of the eager path (including experimental eager AOT kernels) and the torch.compile path |
| 3.2 Supported Eager Operators | Operators for the torch.compile path and experimental eager compute |
| 3.3 Unregistered Operators | How unregistered operators fail, and how to check for them in advance |
| 3.4 Numerical Differences | dtype and tolerance guidelines for comparing against CPU results |
| 3.5 Custom Operators | Layout and debugging operators in the torch.ops.lpu namespace |
| 3.6 When an Operator You Need Is Missing | What to report |
3.1 Two Execution Paths¶
PyTorch operators run on the LPU through one of two paths.
| Path | Package | Contents |
|---|---|---|
| Eager dispatch | torch_ha |
Handles operators such as allocation, copy, views, and serialization. Computation runs on the device only for the few operators that have eager AOT kernels |
torch.compile(backend="legato") |
torch_legato |
Compiles operators that have a legato resolver into Legato kernels and runs them. Model computation uses this path |
On the eager path, the operators computed by device kernels are those handled by the eager AOT kernels below. Calling
other compute operators in eager mode raises NotImplementedError. multinomial and exponential_ are exceptions: they return fixed values without an error (Operator Coverage).
Model computation is compiled with torch.compile(backend="legato") and then run. The legato resolvers, not this
documentation, decide which operators compile. See
Compile PyTorch Model with Legato.
Eager AOT Kernels (Experimental)¶
This experimental feature runs some computation as device kernels without torch.compile. It selects a prebuilt kernel
that matches the operator, dtype, and shape, and runs it. It covers add, sub, mul, and mm on bfloat16, and
supports only the shapes fixed when the kernels were built.
Device compute currently takes contiguous operands. Every operand, including the output, must be contiguous; the mm
weight may also be w.t() of a contiguous weight. A view must be a slice that device compute takes as is
(Device Semantics §Slicing). For a non-contiguous view (a transpose, permute, or
expand, or a stepped slice like x[::2]), call .contiguous() first; this goes through a host copy. Call .clone() on
a slice that device compute does not take as is.
If no kernel matches, the operator fails with NotImplementedError, and the No eager kernel took this op -- <reason>
part of the message gives the reason. To check whether this feature is available, call eager_status().
1 2 3 4 | |
Kernels of the same kind also handle fill_ of 1 MiB or more and copy_ within one device. When no kernel matches,
these two operators run on their existing paths without an error. See
Device Semantics §2.7.
3.2 Supported Eager Operators¶
The eager path supports two kinds of operators.
| Kind | Contents |
|---|---|
Operators for the torch.compile path |
Allocation, copy and transfer, views and slices, and value fills. Only large fill_ calls and copy_ within one device use eager kernels, and the rest run without kernels. They have restrictions that depend on the tensor layout |
| Experimental eager compute | add, sub, mul, and mm on bfloat16 (§3.1) |
Other registered compute operators have no eager device kernel. The full list and its classification are in Operator Coverage.
3.3 Unregistered Operators¶
An operator that has neither an LPU registration nor a PyTorch composite implementation raises a dispatch error.
Composite operators (for example transpose, permute, select, split, matmul) decompose into the operators in
Operator Coverage. CPU fallback (running the operator on the CPU instead) is not provided.
Unregistered operators fail with a dispatch error
The error names the operator and the backend.
1 2 | |
This behavior is intentional. If execution automatically fell back to the CPU, a model running at host speed would be hard to notice. The downside is that you encounter missing operators one at a time as they execute, so check for them in advance as described below.
A registered operator that does not run on the device fails with a NotImplementedError. The message lists the
operand's sizes, strides, dtype, and kind. For an operator that the eager AOT kernels target, the
No eager kernel took this op -- <reason> part gives the reason no kernel accepted it.
Finding the Operators You Need in Advance¶
Before running the model on the device, check it with a dry run. A dry run captures the graph and splits its nodes
between the legato path and extern nodes, without compiling legato kernels. Nodes without a resolver are classified as
extern nodes that run in eager mode. The captured graph runs eagerly on the device that holds the inputs, so keep the
model and the input on the CPU. On lpu, compute operators that have no device kernel raise NotImplementedError.
1 2 3 4 5 6 | |
A passing dry run does not mean that every operator runs on the device. Compute operators classified as extern can fail during actual execution.
There is not yet a report of which nodes were classified as extern. Only nodes that a resolver examined and rejected appear in the debug log. Nodes without a resolver do not appear in the log at all. If you need an exact list, compare against the legato resolver registration list or contact the team.
Working Around Operators Without a Device Kernel¶
| Approach | When |
|---|---|
| Rewrite with operators that compile | The same computation can be expressed as a combination of other operators |
| Compute only that step on the CPU | The step is not on the hot path. Move it with x.cpu(), compute, then call .to("lpu") |
| Request the operator | The step is on the hot path and cannot be rewritten |
Steps moved to the CPU are visible in both the code and the profile, so they are easy to find and fix later.
3.4 Numerical Differences¶
Device results may not be bit-identical to CPU results.
- Reduction order differs. Sums and means can differ in the last digit, and the differences accumulate as models get deeper.
- The LPU computes in
bfloat16. Comparing abfloat16LPU result with afloat32CPU result shows the dtype difference, not a backend difference.
To compare against a reference, use the same dtype on both sides and a tolerance appropriate for that dtype.
1 2 3 4 5 | |
If the difference is larger than the dtype explains, compare per-layer outputs instead of the final logits to find the first layer that diverges. See Troubleshooting.
3.5 Custom Operators¶
torch_ha adds its own operators in the torch.ops.lpu namespace.
| Operator | Purpose |
|---|---|
torch.ops.lpu.empty_with_kind |
Allocates a tensor with a specified device layout (kind) |
torch.ops.lpu.set_storage_meta_ / get_storage_meta |
Inspect and set the layout metadata of a tensor's storage |
torch.ops.lpu.eager_status |
Checks whether the eager AOT kernels are available (§3.1) |
Most of these are for specifying and inspecting layouts, not for computation. The full list and signatures, including
quantized weights and collectives, are in the Custom Operators API. For the vmm_*
operators for the KV cache, see Paged KV Cache and the VMM API.
3.6 When an Operator You Need Is Missing¶
Include the following in your report:
- The exact operator and overload from the dispatch error
- The model and the call site
- Whether it is on the hot path, and whether it can be moved to the CPU
Next Steps¶
- Operator Coverage: per-operator support
- Custom Operators API:
torch.ops.lpuoperator signatures - Migrating from CUDA: differences from CUDA code