Skip to content

3. Operators

This chapter describes which operators run on the LPU, how operators that do not run fail, and how to check in advance the operators your model uses. Per-operator support is listed in Operator Coverage.

Contents

Section Objective
3.1 Two Execution Paths The roles of the eager path (including experimental eager AOT kernels) and the torch.compile path
3.2 Supported Eager Operators Operators for the torch.compile path and experimental eager compute
3.3 Unregistered Operators How unregistered operators fail, and how to check for them in advance
3.4 Numerical Differences dtype and tolerance guidelines for comparing against CPU results
3.5 Custom Operators Layout and debugging operators in the torch.ops.lpu namespace
3.6 When an Operator You Need Is Missing What to report

3.1 Two Execution Paths

PyTorch operators run on the LPU through one of two paths.

Path Package Contents
Eager dispatch torch_ha Handles operators such as allocation, copy, views, and serialization. Computation runs on the device only for the few operators that have eager AOT kernels
torch.compile(backend="legato") torch_legato Compiles operators that have a legato resolver into Legato kernels and runs them. Model computation uses this path

On the eager path, the operators computed by device kernels are those handled by the eager AOT kernels below. Calling other compute operators in eager mode raises NotImplementedError. multinomial and exponential_ are exceptions: they return fixed values without an error (Operator Coverage).

Model computation is compiled with torch.compile(backend="legato") and then run. The legato resolvers, not this documentation, decide which operators compile. See Compile PyTorch Model with Legato.

Eager AOT Kernels (Experimental)

This experimental feature runs some computation as device kernels without torch.compile. It selects a prebuilt kernel that matches the operator, dtype, and shape, and runs it. It covers add, sub, mul, and mm on bfloat16, and supports only the shapes fixed when the kernels were built.

Device compute currently takes contiguous operands. Every operand, including the output, must be contiguous; the mm weight may also be w.t() of a contiguous weight. A view must be a slice that device compute takes as is (Device Semantics §Slicing). For a non-contiguous view (a transpose, permute, or expand, or a stepped slice like x[::2]), call .contiguous() first; this goes through a host copy. Call .clone() on a slice that device compute does not take as is.

If no kernel matches, the operator fails with NotImplementedError, and the No eager kernel took this op -- <reason> part of the message gives the reason. To check whether this feature is available, call eager_status().

1
2
3
4
import torch

ready, reason = torch.ops.lpu.eager_status()[:2]
print(ready, reason or "-")

Kernels of the same kind also handle fill_ of 1 MiB or more and copy_ within one device. When no kernel matches, these two operators run on their existing paths without an error. See Device Semantics §2.7.

3.2 Supported Eager Operators

The eager path supports two kinds of operators.

Kind Contents
Operators for the torch.compile path Allocation, copy and transfer, views and slices, and value fills. Only large fill_ calls and copy_ within one device use eager kernels, and the rest run without kernels. They have restrictions that depend on the tensor layout
Experimental eager compute add, sub, mul, and mm on bfloat16 (§3.1)

Other registered compute operators have no eager device kernel. The full list and its classification are in Operator Coverage.

3.3 Unregistered Operators

An operator that has neither an LPU registration nor a PyTorch composite implementation raises a dispatch error. Composite operators (for example transpose, permute, select, split, matmul) decompose into the operators in Operator Coverage. CPU fallback (running the operator on the CPU instead) is not provided.

Unregistered operators fail with a dispatch error

The error names the operator and the backend.

1
2
NotImplementedError: Could not run 'aten::<op>' with arguments from the
'lpu' backend.

This behavior is intentional. If execution automatically fell back to the CPU, a model running at host speed would be hard to notice. The downside is that you encounter missing operators one at a time as they execute, so check for them in advance as described below.

A registered operator that does not run on the device fails with a NotImplementedError. The message lists the operand's sizes, strides, dtype, and kind. For an operator that the eager AOT kernels target, the No eager kernel took this op -- <reason> part gives the reason no kernel accepted it.

Finding the Operators You Need in Advance

Before running the model on the device, check it with a dry run. A dry run captures the graph and splits its nodes between the legato path and extern nodes, without compiling legato kernels. Nodes without a resolver are classified as extern nodes that run in eager mode. The captured graph runs eagerly on the device that holds the inputs, so keep the model and the input on the CPU. On lpu, compute operators that have no device kernel raise NotImplementedError.

1
2
3
4
5
6
compiled = torch.compile(
    model,
    backend="legato",
    options={"legato.device": "bertha", "legato.frontend.dry_run": True},
)
compiled(example_input)   # model and example_input on the CPU

A passing dry run does not mean that every operator runs on the device. Compute operators classified as extern can fail during actual execution.

There is not yet a report of which nodes were classified as extern. Only nodes that a resolver examined and rejected appear in the debug log. Nodes without a resolver do not appear in the log at all. If you need an exact list, compare against the legato resolver registration list or contact the team.

Working Around Operators Without a Device Kernel

Approach When
Rewrite with operators that compile The same computation can be expressed as a combination of other operators
Compute only that step on the CPU The step is not on the hot path. Move it with x.cpu(), compute, then call .to("lpu")
Request the operator The step is on the hot path and cannot be rewritten

Steps moved to the CPU are visible in both the code and the profile, so they are easy to find and fix later.

3.4 Numerical Differences

Device results may not be bit-identical to CPU results.

  • Reduction order differs. Sums and means can differ in the last digit, and the differences accumulate as models get deeper.
  • The LPU computes in bfloat16. Comparing a bfloat16 LPU result with a float32 CPU result shows the dtype difference, not a backend difference.

To compare against a reference, use the same dtype on both sides and a tolerance appropriate for that dtype.

1
2
3
4
5
# model_lpu is a model compiled with torch.compile(backend="legato").
ref = model_cpu(x_cpu.to(torch.bfloat16))
out = model_lpu(x_cpu.to("lpu").to(torch.bfloat16)).cpu()

torch.testing.assert_close(out, ref, rtol=1e-2, atol=1e-2)

If the difference is larger than the dtype explains, compare per-layer outputs instead of the final logits to find the first layer that diverges. See Troubleshooting.

3.5 Custom Operators

torch_ha adds its own operators in the torch.ops.lpu namespace.

Operator Purpose
torch.ops.lpu.empty_with_kind Allocates a tensor with a specified device layout (kind)
torch.ops.lpu.set_storage_meta_ / get_storage_meta Inspect and set the layout metadata of a tensor's storage
torch.ops.lpu.eager_status Checks whether the eager AOT kernels are available (§3.1)

Most of these are for specifying and inspecting layouts, not for computation. The full list and signatures, including quantized weights and collectives, are in the Custom Operators API. For the vmm_* operators for the KV cache, see Paged KV Cache and the VMM API.

3.6 When an Operator You Need Is Missing

Include the following in your report:

  1. The exact operator and overload from the dispatch error
  2. The model and the call site
  3. Whether it is on the hot path, and whether it can be moved to the CPU

Next Steps