PyTorch Integration
Legato kernels run on HyperAccel LPU devices and interoperate with PyTorch
tensors. There are two ways to use Legato from PyTorch:
- Call a Legato kernel directly on device-resident tensors you create and
fill yourself — described on this page.
- Compile a PyTorch model with
torch.compile(..., backend="legato") — see
Compile PyTorch Model with Legato.
This page covers the direct path: creating a device tensor, filling it with data,
and handing it to a compiled Legato kernel.
Creating a Device Tensor
A Legato kernel reads its inputs from device memory. The host creates a
device-resident tensor with torch.ops.lpu.empty_with_kind, which allocates the
tensor on the LPU and stamps it with a layout kind describing how the data is
laid out in memory.
| import torch
import torch_ha
torch_ha._autoload() # idempotent, including when Torch backend autoload is disabled
weight = torch.ops.lpu.empty_with_kind(
[256, 128], "mpu_tensor", dtype=torch.bfloat16, device="lpu:0"
)
|
Filling a Device Tensor
empty_with_kind returns an uninitialized tensor. Create the data on CPU and
copy it onto the device tensor:
| weight.copy_(torch.randn(256, 128, dtype=torch.bfloat16))
|
An output tensor is created the same way and left for the kernel to write.
Choosing the Kind
The kind string passed to empty_with_kind must match the annotation kind
of the kernel parameter the tensor is passed to — the third argument of
tensor_type(dtype, shape, kind). The two line up one-to-one:
empty_with_kind name |
Annotation kind |
Typical operand |
"dram_word_interleaved" |
"shared_dram" / "shared_memory" |
activation |
"mpu_tensor" |
"mpu_dram" / "value_dram" |
weight / value |
"mpu_tensor_transposed" |
"transposed_mpu_dram" / "key_dram" |
transposed weight / key |
Note
The two alias names per kind (e.g. "shared_dram" / "shared_memory") are aliases in the current compiler.
A tensor created with any other empty_with_kind name, or one created without a kind at all, takes the
default shared-memory → SRAM path; passing such a tensor to a parameter that
expects one of the three kinds above raises at launch.
The "mpu_tensor" and "mpu_tensor_transposed" layouts require rank 2 or
larger, a dtype bit width of 4, 8, or 16, and a shape compatible with the
selected backend. See
Bertha Layouts
for the kind and allocation requirements.
Physical storage
A type such as (4, (70, 256)) needs an actual 256-element physical row. It does
not pad a compact [4, 70] tensor. Allocate/copy the full backing tensor first,
then pass its logical view:
| cpu = torch.randn(4, 70, dtype=torch.bfloat16)
padded_cpu = torch.zeros(4, 256, dtype=torch.bfloat16)
padded_cpu[:, :70] = cpu
storage = torch.ops.lpu.empty_with_kind(
[4, 256], "dram_word_interleaved", dtype=torch.bfloat16, device="lpu:0"
)
storage.copy_(padded_cpu)
logical = storage[:, :70] # shape [4, 70], row stride 256
# Pass logical to a matching (4, (70, 256)) kernel annotation.
# For outputs, copy the whole backing to CPU, then crop the CPU result.
|
Shared-DRAM launch validation rejects a compact device tensor whose row stride
disagrees with the annotation. Copy/read complete backing tensors rather than
using host transfers through noncontiguous device views. Keep backing storage
alive through stream synchronization.
Passing Tensors to a Kernel
Wrap each tensor according to the parameter's param_kinds and call the kernel
inside a legato.session. The wrapping rules — top / core / universal placement,
Arg versus Args — are described in
Argument Binding.
Full Example
See Running the examples for the shared setup and simulator/hardware requirements.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84 | """Single-core BF16 matmul with a bounded dynamic activation row count."""
import os
from pathlib import Path
import torch
import legato
import legato.model.bertha as bertha
# Store compiler artifacts and optional simulator logs in a configurable directory.
OUTPUT = Path(os.environ.get("LEGATO_EXAMPLE_OUTPUT", "legato_example_output"))
@legato.compile(
backend=lambda ctx: bertha.Bertha(ctx, "bertha"),
param_kinds={"a": "universal", "b": "top", "out": "core"},
)
def simple_matmul(
a: legato.get_model().tensor_type(legato.types.float("bfloat16"), ((-1, 10), 256), "shared_dram"),
b: legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_dram"),
out: legato.get_model().tensor_type(legato.types.float("bfloat16"), [10, 128], "shared_dram"),
):
"""Load BF16 operands, multiply on core zero, and store the output."""
device = legato.get_context().get_device()
with device.get_top():
legato.tensor.request_load(a, device.get_core(0), "sram")
legato.tensor.request_load(b, device.get_core(0), "mpu_buffer")
with device.get_core(0):
weight_type = legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_weight")
loaded_a = legato.tensor.receive(a, 0, "sram")
loaded_b = legato.tensor.receive_type(weight_type, 0, None, "mpu_buffer")
result = loaded_a @ loaded_b
legato.tensor.memcpy(out, result)
def main():
"""Compare the complete output with a CPU matrix product."""
# Register the LPU PyTorch backend before allocating device tensors.
import hart
import torch_ha
torch_ha._autoload()
# HA_BACKEND=sim selects EVT0 Torch FSim; otherwise use the hardware runtime.
if os.environ.get("HA_BACKEND") == "sim":
OUTPUT.mkdir(parents=True, exist_ok=True)
hart.set_fsim_env(
hart.HaDeviceVersion.BERTHA_CLOUD_EVT0,
hart.HaFSimComputeMode.TORCH,
str(OUTPUT / "fsim.log"),
"error",
)
hart.set_device(0)
# Keep CPU inputs reproducible without initializing a device RNG.
torch.random.default_generator.manual_seed(0)
a_cpu = (torch.randn(10, 256) * 0.25).to(torch.bfloat16)
b_cpu = (torch.randn(256, 128) * 0.25).to(torch.bfloat16)
# Allocate the physical layout expected by the kernel and copy contiguous host data.
a = torch.ops.lpu.empty_with_kind(list(a_cpu.shape), "dram_word_interleaved", dtype=a_cpu.dtype, device="lpu:0")
a.copy_(a_cpu.contiguous())
# Allocate the physical layout expected by the kernel and copy contiguous host data.
b = torch.ops.lpu.empty_with_kind(list(b_cpu.shape), "mpu_tensor", dtype=b_cpu.dtype, device="lpu:0")
b.copy_(b_cpu.contiguous())
out_cpu = torch.zeros(10, 128, dtype=torch.bfloat16)
# Allocate the physical layout expected by the kernel and copy contiguous host data.
out = torch.ops.lpu.empty_with_kind(
list(out_cpu.shape), "dram_word_interleaved", dtype=out_cpu.dtype, device="lpu:0"
)
out.copy_(out_cpu.contiguous())
# Compile a binary and launch it on the selected device.
with legato.session(output_type=legato.OutputType.BINARY, output_path=str(OUTPUT), launch=True):
simple_matmul(a, legato.Arg(b, bertha.top), legato.Arg(out, bertha.core(0)))
# Surface asynchronous execution errors before reading the result.
hart.stream_synchronize()
# Read back the synchronized result and compare every element with the CPU reference.
actual = out.cpu().float()
assert torch.isfinite(actual).all(), "Non-finite device result"
torch.testing.assert_close(actual, (a_cpu.float() @ b_cpu.float()).float(), atol=0.05, rtol=0.05)
if __name__ == "__main__":
main()
|