Legato
Legato is a Python eDSL for writing kernels for HyperAccel LPUs. It keeps the
programming model close to Python while giving the compiler enough structure to
place data, select executors, and lower operations to the target backend.
Why Legato?
Legato is intended for kernels where programmers want to describe algorithmic
data flow explicitly:
- which executor runs each phase,
- when tensors move between memories,
- what tensor types, layouts, and dynamic dimensions are expected,
- and which values are compile-time parameters.
The compiler then verifies the program, lowers it through Legato IR, runs
backend passes, and writes the requested artifact.
Minimal Bertha Example
Most examples use a backend factory that receives the active MLIR context and
returns a compute model instance.
See Running the examples for the shared setup and simulator/hardware requirements.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84 | """Single-core BF16 matmul with a bounded dynamic activation row count."""
import os
from pathlib import Path
import torch
import legato
import legato.model.bertha as bertha
# Store compiler artifacts and optional simulator logs in a configurable directory.
OUTPUT = Path(os.environ.get("LEGATO_EXAMPLE_OUTPUT", "legato_example_output"))
@legato.compile(
backend=lambda ctx: bertha.Bertha(ctx, "bertha"),
param_kinds={"a": "universal", "b": "top", "out": "core"},
)
def simple_matmul(
a: legato.get_model().tensor_type(legato.types.float("bfloat16"), ((-1, 10), 256), "shared_dram"),
b: legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_dram"),
out: legato.get_model().tensor_type(legato.types.float("bfloat16"), [10, 128], "shared_dram"),
):
"""Load BF16 operands, multiply on core zero, and store the output."""
device = legato.get_context().get_device()
with device.get_top():
legato.tensor.request_load(a, device.get_core(0), "sram")
legato.tensor.request_load(b, device.get_core(0), "mpu_buffer")
with device.get_core(0):
weight_type = legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_weight")
loaded_a = legato.tensor.receive(a, 0, "sram")
loaded_b = legato.tensor.receive_type(weight_type, 0, None, "mpu_buffer")
result = loaded_a @ loaded_b
legato.tensor.memcpy(out, result)
def main():
"""Compare the complete output with a CPU matrix product."""
# Register the LPU PyTorch backend before allocating device tensors.
import hart
import torch_ha
torch_ha._autoload()
# HA_BACKEND=sim selects EVT0 Torch FSim; otherwise use the hardware runtime.
if os.environ.get("HA_BACKEND") == "sim":
OUTPUT.mkdir(parents=True, exist_ok=True)
hart.set_fsim_env(
hart.HaDeviceVersion.BERTHA_CLOUD_EVT0,
hart.HaFSimComputeMode.TORCH,
str(OUTPUT / "fsim.log"),
"error",
)
hart.set_device(0)
# Keep CPU inputs reproducible without initializing a device RNG.
torch.random.default_generator.manual_seed(0)
a_cpu = (torch.randn(10, 256) * 0.25).to(torch.bfloat16)
b_cpu = (torch.randn(256, 128) * 0.25).to(torch.bfloat16)
# Allocate the physical layout expected by the kernel and copy contiguous host data.
a = torch.ops.lpu.empty_with_kind(list(a_cpu.shape), "dram_word_interleaved", dtype=a_cpu.dtype, device="lpu:0")
a.copy_(a_cpu.contiguous())
# Allocate the physical layout expected by the kernel and copy contiguous host data.
b = torch.ops.lpu.empty_with_kind(list(b_cpu.shape), "mpu_tensor", dtype=b_cpu.dtype, device="lpu:0")
b.copy_(b_cpu.contiguous())
out_cpu = torch.zeros(10, 128, dtype=torch.bfloat16)
# Allocate the physical layout expected by the kernel and copy contiguous host data.
out = torch.ops.lpu.empty_with_kind(
list(out_cpu.shape), "dram_word_interleaved", dtype=out_cpu.dtype, device="lpu:0"
)
out.copy_(out_cpu.contiguous())
# Compile a binary and launch it on the selected device.
with legato.session(output_type=legato.OutputType.BINARY, output_path=str(OUTPUT), launch=True):
simple_matmul(a, legato.Arg(b, bertha.top), legato.Arg(out, bertha.core(0)))
# Surface asynchronous execution errors before reading the result.
hart.stream_synchronize()
# Read back the synchronized result and compare every element with the CPU reference.
actual = out.cpu().float()
assert torch.isfinite(actual).all(), "Non-finite device result"
torch.testing.assert_close(actual, (a_cpu.float() @ b_cpu.float()).float(), atol=0.05, rtol=0.05)
if __name__ == "__main__":
main()
|
Core Concepts
Sessions
Compiled functions must run inside legato.session(...). The session selects
the output artifact kind and where compiler artifacts and logs are written.
| with legato.session(output_type=legato.OutputType.MLIR, output_path="test-output"):
kernel(*args)
|
Supported output types are MLIR, CORE_IR, BACKEND_IR, ASM, and BINARY.
Contexts
Legato uses Python with statements to select the executor context:
device.get_top() issues top-level orchestration and load requests.
device.get_core(i) runs on one core.
device.get_all_cores() runs the region for every core.
Values produced in one region must be yielded when they are needed by a later
region.
Tensor Movement
Data movement is explicit. request_load is issued from a source context, and
receive or receive_type materializes the data in the target context.
| with device.get_top():
legato.tensor.request_load(input_data, device.get_core(0), "sram")
with device.get_core(0):
loaded = legato.tensor.receive(input_data, 0, "sram")
|
Types And Layouts
Function annotations use legato.get_model().tensor_type(...) when the tensor
is bound to a backend-specific memory/layout kind.
Common Bertha kinds include:
"shared_memory" and "shared_dram"
"row_major"
"value_dram" and "mpu_dram" for the same value-style DRAM layout
"key_dram" and "transposed_mpu_dram" for the same key-style DRAM layout
"mpu_weight", "mpu_packed_buffer", and "sliced_dram"
Where To Go Next