Skip to content

Legato

Legato is a Python eDSL for writing kernels for HyperAccel LPUs. It keeps the programming model close to Python while giving the compiler enough structure to place data, select executors, and lower operations to the target backend.

Why Legato?

Legato is intended for kernels where programmers want to describe algorithmic data flow explicitly:

  • which executor runs each phase,
  • when tensors move between memories,
  • what tensor types, layouts, and dynamic dimensions are expected,
  • and which values are compile-time parameters.

The compiler then verifies the program, lowers it through Legato IR, runs backend passes, and writes the requested artifact.

Minimal Bertha Example

Most examples use a backend factory that receives the active MLIR context and returns a compute model instance.

See Running the examples for the shared setup and simulator/hardware requirements.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
"""Single-core BF16 matmul with a bounded dynamic activation row count."""

import os
from pathlib import Path

import torch

import legato
import legato.model.bertha as bertha

# Store compiler artifacts and optional simulator logs in a configurable directory.
OUTPUT = Path(os.environ.get("LEGATO_EXAMPLE_OUTPUT", "legato_example_output"))


@legato.compile(
    backend=lambda ctx: bertha.Bertha(ctx, "bertha"),
    param_kinds={"a": "universal", "b": "top", "out": "core"},
)
def simple_matmul(
    a: legato.get_model().tensor_type(legato.types.float("bfloat16"), ((-1, 10), 256), "shared_dram"),
    b: legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_dram"),
    out: legato.get_model().tensor_type(legato.types.float("bfloat16"), [10, 128], "shared_dram"),
):
    """Load BF16 operands, multiply on core zero, and store the output."""
    device = legato.get_context().get_device()

    with device.get_top():
        legato.tensor.request_load(a, device.get_core(0), "sram")
        legato.tensor.request_load(b, device.get_core(0), "mpu_buffer")

    with device.get_core(0):
        weight_type = legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_weight")
        loaded_a = legato.tensor.receive(a, 0, "sram")
        loaded_b = legato.tensor.receive_type(weight_type, 0, None, "mpu_buffer")
        result = loaded_a @ loaded_b
        legato.tensor.memcpy(out, result)


def main():
    """Compare the complete output with a CPU matrix product."""
    # Register the LPU PyTorch backend before allocating device tensors.
    import hart
    import torch_ha

    torch_ha._autoload()
    # HA_BACKEND=sim selects EVT0 Torch FSim; otherwise use the hardware runtime.
    if os.environ.get("HA_BACKEND") == "sim":
        OUTPUT.mkdir(parents=True, exist_ok=True)
        hart.set_fsim_env(
            hart.HaDeviceVersion.BERTHA_CLOUD_EVT0,
            hart.HaFSimComputeMode.TORCH,
            str(OUTPUT / "fsim.log"),
            "error",
        )
    hart.set_device(0)
    # Keep CPU inputs reproducible without initializing a device RNG.
    torch.random.default_generator.manual_seed(0)
    a_cpu = (torch.randn(10, 256) * 0.25).to(torch.bfloat16)
    b_cpu = (torch.randn(256, 128) * 0.25).to(torch.bfloat16)
    # Allocate the physical layout expected by the kernel and copy contiguous host data.
    a = torch.ops.lpu.empty_with_kind(list(a_cpu.shape), "dram_word_interleaved", dtype=a_cpu.dtype, device="lpu:0")
    a.copy_(a_cpu.contiguous())
    # Allocate the physical layout expected by the kernel and copy contiguous host data.
    b = torch.ops.lpu.empty_with_kind(list(b_cpu.shape), "mpu_tensor", dtype=b_cpu.dtype, device="lpu:0")
    b.copy_(b_cpu.contiguous())
    out_cpu = torch.zeros(10, 128, dtype=torch.bfloat16)
    # Allocate the physical layout expected by the kernel and copy contiguous host data.
    out = torch.ops.lpu.empty_with_kind(
        list(out_cpu.shape), "dram_word_interleaved", dtype=out_cpu.dtype, device="lpu:0"
    )
    out.copy_(out_cpu.contiguous())
    # Compile a binary and launch it on the selected device.
    with legato.session(output_type=legato.OutputType.BINARY, output_path=str(OUTPUT), launch=True):
        simple_matmul(a, legato.Arg(b, bertha.top), legato.Arg(out, bertha.core(0)))
    # Surface asynchronous execution errors before reading the result.
    hart.stream_synchronize()
    # Read back the synchronized result and compare every element with the CPU reference.
    actual = out.cpu().float()
    assert torch.isfinite(actual).all(), "Non-finite device result"
    torch.testing.assert_close(actual, (a_cpu.float() @ b_cpu.float()).float(), atol=0.05, rtol=0.05)


if __name__ == "__main__":
    main()

Core Concepts

Sessions

Compiled functions must run inside legato.session(...). The session selects the output artifact kind and where compiler artifacts and logs are written.

1
2
with legato.session(output_type=legato.OutputType.MLIR, output_path="test-output"):
    kernel(*args)

Supported output types are MLIR, CORE_IR, BACKEND_IR, ASM, and BINARY.

Contexts

Legato uses Python with statements to select the executor context:

  • device.get_top() issues top-level orchestration and load requests.
  • device.get_core(i) runs on one core.
  • device.get_all_cores() runs the region for every core.

Values produced in one region must be yielded when they are needed by a later region.

Tensor Movement

Data movement is explicit. request_load is issued from a source context, and receive or receive_type materializes the data in the target context.

1
2
3
4
5
with device.get_top():
    legato.tensor.request_load(input_data, device.get_core(0), "sram")

with device.get_core(0):
    loaded = legato.tensor.receive(input_data, 0, "sram")

Types And Layouts

Function annotations use legato.get_model().tensor_type(...) when the tensor is bound to a backend-specific memory/layout kind.

Common Bertha kinds include:

  • "shared_memory" and "shared_dram"
  • "row_major"
  • "value_dram" and "mpu_dram" for the same value-style DRAM layout
  • "key_dram" and "transposed_mpu_dram" for the same key-style DRAM layout
  • "mpu_weight", "mpu_packed_buffer", and "sliced_dram"

Where To Go Next