Compile PyTorch Model with Legato¶
This page describes how users integrate Legato with PyTorch models with
torch.compile(..., backend="legato"): registering custom Legato primitives, and
how the backend behaves when a graph is fully compilable or only partially
compilable. To call a hand-written Legato kernel directly on device tensors
instead, see PyTorch Integration.
Prerequisites¶
Use an SDK environment with legato, torch_legato, and PyTorch 2.10.0.
The separate torch_legato package supplies the "legato" TorchDynamo backend;
installing legato alone does not register it. Launching on an LPU also needs
hart, torch_ha, and a matching device/runtime setup from
PyTorch Integration. The compile-only example below uses CPU
inputs and legato.session.launch=False.
Writing Primitive Functions¶
A resolver returns a @legato.primitive(...) JitFunction implementing the
PyTorch op. Operator arguments become primitive parameters, and the primitive
returns the operator result. See Decorators And Sessions
for the language syntax.
The current frontend inserts input loads and distributes token rows across cores. Primitive signatures must match the post-load layout, per-core tile capacity, and placement. A fixed-shape, single-core primitive is not a drop-in replacement for a core-split resolver. Reuse or adapt the backend's primitive builders when implementing a new resolver.
For matmul, the built-in resolver supports BF16 and expects a replicated RHS
loaded as "mpu_weight", from an "mpu_dram" or
"transposed_mpu_dram" operand. A row-major SRAM RHS is not supported by that
resolver. See Choosing the Kind for the host
layout names and Bertha Layouts.
Registering a resolver¶
legato.registry.register(op=...) accepts a compatible primitive directly or a
resolver that selects one from the current input metadata. The example below
delegates to the current matmul builder, preserving its layout and core-split
checks. A custom resolver can choose a different compatible primitive for its
supported inputs.
Load the built-in resolver libraries before registering an override. The registry keeps the last registration; first-use discovery can otherwise replace an earlier override.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
Resolver behavior and fallback¶
Legato now probes resolver results before deciding whether a node can stay on the compiled path.
- If a resolver exists and returns a primitive
JitFunction, the node stays compilable. - If a resolver is missing, the node can become an extern fallback candidate.
- If a resolver raises
AssertionErrorfor the current dtype/shape combination, the node can also become an extern fallback candidate. - If a resolver returns an invalid object or raises an unexpected exception, Legato still treats that as an error rather than silently falling back.
This means a missing resolver does not always hard-fail compilation anymore when mixed execution is enabled. Unsupported call_function nodes can be routed to eager extern execution while compilable regions are still lowered through Legato.
Expose Functions for the Compiler¶
Legato cannot see primitive functions unless modules are imported. There are three ways to import modules that contain implemented primitive functions.
-
Import a module directly.
1 2
# model.py import my.lib -
Use
load_libraryfunction.1 2 3 4
# model.py import legato.registry legato.registry.load_library("my.lib") -
Use entrypoints.
Add
torch_legato_libraryentrypoint in yourpyproject.toml.1 2 3 4 5
[project] # your project configurations... [project.entry-points.torch_legato_library] my_lib = "my.lib"
Compile the Model¶
You can compile the model using torch.compile with backend="legato". Pass
dynamic=False for static-shape models; to allow dynamic dimensions, see
Dynamic shape inputs.
Using torch.nn.Module¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | |
Using a function¶
1 2 3 4 5 6 7 8 9 10 11 12 | |
Execution model¶
All graphs go through LegatoGraphWrapper, which provides a unified execution model:
- Classifies
call_functionnodes as compiled (Legato primitive) or extern (eager fallback). - Groups contiguous compiled nodes into segments.
- Compiles each segment into a Legato kernel.
- Executes unsupported nodes eagerly as extern calls.
- Returns one callable that orchestrates both compiled and extern paths.
For fully compilable graphs (all nodes lower to Legato primitives), the wrapper produces a single segment. For mixed graphs (some nodes need eager fallback), it interleaves multiple compiled segments with extern calls.
In practice, this allows models with custom ops or unsupported ATen ops to avoid immediate hard failure as long as the unsupported nodes can run eagerly.
Note
Current mixed execution support is intentionally conservative. Segment boundaries are expected to be tensor-like values, and the implementation focuses on call_function fallback.
Compile without a device¶
This complete example lowers the built-in add path to a binary with CPU inputs.
launch=False runs the compiler without executing the binary. The returned
output buffer has no computed result, so this checks artifacts rather than
numerical correctness. Device execution needs initialized LPU tensors and
stream synchronization.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | |
Options¶
legato.device¶
Type: str
Target device name to compile. This option is mandatory.
Example:
1 2 3 4 5 6 | |
legato.session.output_type / legato.session.output_path¶
Type: legato.OutputType / str
Select the compiler artifact and its destination, mirroring the output_type and
output_path arguments of legato.session(...) (see Session).
Both default to None.
legato.session(...) cannot be opened inside torch.compiled code, so the backend
opens it internally and reads its arguments from these options.
Example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
Dynamic shape inputs¶
Each compiled variant needs a finite buffer capacity. There are two ways to obtain it:
- Supply an explicit bound with
torch._dynamo.mark_dynamic(tensor, dim, min=1, max=N). The frontend reads the bound from Dynamo'sShapeEnv. - Leave
maxunspecified. The current frontend sizes these dimensions with size buckets, including dimensions introduced bydynamic=Trueor automatic dynamic-shape recompilation. The first call for a bucket assignment compiles a variant; subsequent calls in the same bucket reuse it.
The default bucket rule grows without a fixed upper limit; it does not compile
an infinitely sized tensor. Set legato.size_buckets to a nonempty, strictly
increasing list of positive integers to choose capacities explicitly. An input
larger than the largest explicit bucket raises ValueError. This option applies
to dimensions without an explicit max; backend layout and memory limits still
apply to each compiled capacity.
The examples below use frontend dry_run=True so CPU values can be checked
without compiling or launching a binary. For a bounded dimension, pass both
min and max on the SDK's PyTorch 2.10:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
For a dimension without a max, an explicit bucket list makes the chosen capacity and limit predictable:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
legato.session.launch¶
Type: bool, default True.
False still compiles the selected artifact but does not submit the kernel.
It is separate from frontend dry_run, which skips backend compilation. Do not
interpret output buffers from a compile-only call as computed results.
Core distribution options¶
| Option | Default | Meaning |
|---|---|---|
legato.frontend.core_num |
32 |
Number of participating cores, from 1 through 32; match the initialized device. |
legato.frontend.token_dim |
-2 |
Signed token-axis index; use 0 for a leading token axis such as [tokens, heads, head_dim]. |
legato.frontend.dump_before_pass¶
Type: bool
Print torch FX graph for each iteration of passes. Debugging purpose only.
legato.frontend.dump_ir¶
Type: bool
Print IR (intermediate representation) code of the model. Debugging and testing purpose only.
legato.frontend.dump_legato¶
Type: bool
Print synthesized Legato function or generated mixed wrapper source. Debugging and testing purpose only.
legato.frontend.dry_run¶
Type: bool
Skip backend execution while keeping frontend validation behavior. Debugging and testing purpose only.
- For fully compilable graphs, dry-run mode still runs the transformed frontend path and returns
gm.forwardinstead of compiling backend code. - For mixed graphs, dry-run mode still runs segmentation planning and subgraph extraction validation, but does not compose or compile segments. It then returns
gm.forward.
This makes dry-run mode useful for validating frontend behavior without requiring backend code generation or runtime execution.