Skip to content

Compile PyTorch Model with Legato

This page describes how users integrate Legato with PyTorch models with torch.compile(..., backend="legato"): registering custom Legato primitives, and how the backend behaves when a graph is fully compilable or only partially compilable. To call a hand-written Legato kernel directly on device tensors instead, see PyTorch Integration.

Prerequisites

Use an SDK environment with legato, torch_legato, and PyTorch 2.10.0. The separate torch_legato package supplies the "legato" TorchDynamo backend; installing legato alone does not register it. Launching on an LPU also needs hart, torch_ha, and a matching device/runtime setup from PyTorch Integration. The compile-only example below uses CPU inputs and legato.session.launch=False.

Writing Primitive Functions

A resolver returns a @legato.primitive(...) JitFunction implementing the PyTorch op. Operator arguments become primitive parameters, and the primitive returns the operator result. See Decorators And Sessions for the language syntax.

The current frontend inserts input loads and distributes token rows across cores. Primitive signatures must match the post-load layout, per-core tile capacity, and placement. A fixed-shape, single-core primitive is not a drop-in replacement for a core-split resolver. Reuse or adapt the backend's primitive builders when implementing a new resolver.

For matmul, the built-in resolver supports BF16 and expects a replicated RHS loaded as "mpu_weight", from an "mpu_dram" or "transposed_mpu_dram" operand. A row-major SRAM RHS is not supported by that resolver. See Choosing the Kind for the host layout names and Bertha Layouts.

Registering a resolver

legato.registry.register(op=...) accepts a compatible primitive directly or a resolver that selects one from the current input metadata. The example below delegates to the current matmul builder, preserving its layout and core-split checks. A custom resolver can choose a different compatible primitive for its supported inputs.

Load the built-in resolver libraries before registering an override. The registry keeps the last registration; first-use discovery can otherwise replace an earlier override.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# my/lib.py
import torch

import legato.registry
from torch_legato.primitive_builder.matmul import mm_resolver as build_mm
from torch_legato.torch_support import discover_resolver_libraries


discover_resolver_libraries()


@legato.registry.register(op=torch.ops.aten.mm.default)
def matmul_resolver(a, b):
    return build_mm(a, b)

Resolver behavior and fallback

Legato now probes resolver results before deciding whether a node can stay on the compiled path.

  • If a resolver exists and returns a primitive JitFunction, the node stays compilable.
  • If a resolver is missing, the node can become an extern fallback candidate.
  • If a resolver raises AssertionError for the current dtype/shape combination, the node can also become an extern fallback candidate.
  • If a resolver returns an invalid object or raises an unexpected exception, Legato still treats that as an error rather than silently falling back.

This means a missing resolver does not always hard-fail compilation anymore when mixed execution is enabled. Unsupported call_function nodes can be routed to eager extern execution while compilable regions are still lowered through Legato.

Expose Functions for the Compiler

Legato cannot see primitive functions unless modules are imported. There are three ways to import modules that contain implemented primitive functions.

  1. Import a module directly.

    1
    2
    # model.py
    import my.lib
    

  2. Use load_library function.

    1
    2
    3
    4
    # model.py
    import legato.registry
    
    legato.registry.load_library("my.lib")
    

  3. Use entrypoints.

    Add torch_legato_library entrypoint in your pyproject.toml.

    1
    2
    3
    4
    5
    [project]
    # your project configurations...
    
    [project.entry-points.torch_legato_library]
    my_lib = "my.lib"
    

Compile the Model

You can compile the model using torch.compile with backend="legato". Pass dynamic=False for static-shape models; to allow dynamic dimensions, see Dynamic shape inputs.

Using torch.nn.Module

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# model.py
import torch


OPTIONS = {
    "legato.device": "bertha",
}


class Model(torch.nn.Module):
    def forward(self, x, y):
        ...


model = Model()
model = torch.compile(model, backend="legato", dynamic=False, options=OPTIONS)

Using a function

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# model.py
import torch


OPTIONS = {
    "legato.device": "bertha",
}


@torch.compile(backend="legato", dynamic=False, options=OPTIONS)
def model(x, y):
    ...

Execution model

All graphs go through LegatoGraphWrapper, which provides a unified execution model:

  • Classifies call_function nodes as compiled (Legato primitive) or extern (eager fallback).
  • Groups contiguous compiled nodes into segments.
  • Compiles each segment into a Legato kernel.
  • Executes unsupported nodes eagerly as extern calls.
  • Returns one callable that orchestrates both compiled and extern paths.

For fully compilable graphs (all nodes lower to Legato primitives), the wrapper produces a single segment. For mixed graphs (some nodes need eager fallback), it interleaves multiple compiled segments with extern calls.

In practice, this allows models with custom ops or unsupported ATen ops to avoid immediate hard failure as long as the unsupported nodes can run eagerly.

Note

Current mixed execution support is intentionally conservative. Segment boundaries are expected to be tensor-like values, and the implementation focuses on call_function fallback.

Compile without a device

This complete example lowers the built-in add path to a binary with CPU inputs. launch=False runs the compiler without executing the binary. The returned output buffer has no computed result, so this checks artifacts rather than numerical correctness. Device execution needs initialized LPU tensors and stream synchronization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
from pathlib import Path

import torch

import legato


output = Path("legato_torch_output")
options = {
    "legato.device": "bertha",
    "legato.session.output_type": legato.OutputType.BINARY,
    "legato.session.output_path": str(output),
    "legato.session.launch": False,
}


@torch.compile(backend="legato", dynamic=False, options=options)
def add(a, b):
    return a + b


a = torch.randn(34, 256, dtype=torch.bfloat16)
b = torch.randn_like(a)
_ = add(a, b)
assert any(path.stat().st_size > 0 for path in output.rglob("*.bin"))

Options

legato.device

Type: str

Target device name to compile. This option is mandatory.

Example:

1
2
3
4
5
6
import torch


@torch.compile(backend="legato", options={"legato.device": "bertha"})
def model(x):
    ...

legato.session.output_type / legato.session.output_path

Type: legato.OutputType / str

Select the compiler artifact and its destination, mirroring the output_type and output_path arguments of legato.session(...) (see Session). Both default to None.

legato.session(...) cannot be opened inside torch.compiled code, so the backend opens it internally and reads its arguments from these options.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import torch

import legato


OPTIONS = {
    "legato.device": "bertha",
    "legato.session.output_type": legato.OutputType.BINARY,
    "legato.session.output_path": "/tmp/legato_out",
}


@torch.compile(backend="legato", options=OPTIONS)
def model(x):
    ...

Dynamic shape inputs

Each compiled variant needs a finite buffer capacity. There are two ways to obtain it:

  • Supply an explicit bound with torch._dynamo.mark_dynamic(tensor, dim, min=1, max=N). The frontend reads the bound from Dynamo's ShapeEnv.
  • Leave max unspecified. The current frontend sizes these dimensions with size buckets, including dimensions introduced by dynamic=True or automatic dynamic-shape recompilation. The first call for a bucket assignment compiles a variant; subsequent calls in the same bucket reuse it.

The default bucket rule grows without a fixed upper limit; it does not compile an infinitely sized tensor. Set legato.size_buckets to a nonempty, strictly increasing list of positive integers to choose capacities explicitly. An input larger than the largest explicit bucket raises ValueError. This option applies to dimensions without an explicit max; backend layout and memory limits still apply to each compiled capacity.

The examples below use frontend dry_run=True so CPU values can be checked without compiling or launching a binary. For a bounded dimension, pass both min and max on the SDK's PyTorch 2.10:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import torch


@torch.compile(
    backend="legato",
    options={"legato.device": "bertha", "legato.frontend.dry_run": True},
)
def model(x):
    return x + x


x = torch.randn(16, 256, dtype=torch.bfloat16)
torch._dynamo.mark_dynamic(x, 0, min=1, max=128)
torch.testing.assert_close(model(x), x + x)

For a dimension without a max, an explicit bucket list makes the chosen capacity and limit predictable:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
@torch.compile(
    backend="legato",
    options={
        "legato.device": "bertha",
        "legato.frontend.dry_run": True,
        "legato.size_buckets": [32, 64, 128],
    },
)
def bucketed_model(x):
    return x + x


x = torch.randn(33, 256, dtype=torch.bfloat16)
torch._dynamo.mark_dynamic(x, 0)
torch.testing.assert_close(bucketed_model(x), x + x)  # capacity bucket: 64

legato.session.launch

Type: bool, default True.

False still compiles the selected artifact but does not submit the kernel. It is separate from frontend dry_run, which skips backend compilation. Do not interpret output buffers from a compile-only call as computed results.

Core distribution options

Option Default Meaning
legato.frontend.core_num 32 Number of participating cores, from 1 through 32; match the initialized device.
legato.frontend.token_dim -2 Signed token-axis index; use 0 for a leading token axis such as [tokens, heads, head_dim].

legato.frontend.dump_before_pass

Type: bool

Print torch FX graph for each iteration of passes. Debugging purpose only.

legato.frontend.dump_ir

Type: bool

Print IR (intermediate representation) code of the model. Debugging and testing purpose only.

legato.frontend.dump_legato

Type: bool

Print synthesized Legato function or generated mixed wrapper source. Debugging and testing purpose only.

legato.frontend.dry_run

Type: bool

Skip backend execution while keeping frontend validation behavior. Debugging and testing purpose only.

  • For fully compilable graphs, dry-run mode still runs the transformed frontend path and returns gm.forward instead of compiling backend code.
  • For mixed graphs, dry-run mode still runs segmentation planning and subgraph extraction validation, but does not compose or compile segments. It then returns gm.forward.

This makes dry-run mode useful for validating frontend behavior without requiring backend code generation or runtime execution.