Skip to content

PyTorch Integration

Legato kernels run on HyperAccel LPU devices and interoperate with PyTorch tensors. There are two ways to use Legato from PyTorch:

  • Call a Legato kernel directly on device-resident tensors you create and fill yourself — described on this page.
  • Compile a PyTorch model with torch.compile(..., backend="legato") — see Compile PyTorch Model with Legato.

This page covers the direct path: creating a device tensor, filling it with data, and handing it to a compiled Legato kernel.

Creating a Device Tensor

A Legato kernel reads its inputs from device memory. The host creates a device-resident tensor with torch.ops.lpu.empty_with_kind, which allocates the tensor on the LPU and stamps it with a layout kind describing how the data is laid out in memory.

1
2
3
4
5
6
7
8
import torch
import torch_ha

torch_ha._autoload()  # idempotent, including when Torch backend autoload is disabled

weight = torch.ops.lpu.empty_with_kind(
    [256, 128], "mpu_tensor", dtype=torch.bfloat16, device="lpu:0"
)

Filling a Device Tensor

empty_with_kind returns an uninitialized tensor. Create the data on CPU and copy it onto the device tensor:

1
weight.copy_(torch.randn(256, 128, dtype=torch.bfloat16))

An output tensor is created the same way and left for the kernel to write.

Choosing the Kind

The kind string passed to empty_with_kind must match the annotation kind of the kernel parameter the tensor is passed to — the third argument of tensor_type(dtype, shape, kind). The two line up one-to-one:

empty_with_kind name Annotation kind Typical operand
"dram_word_interleaved" "shared_dram" / "shared_memory" activation
"mpu_tensor" "mpu_dram" / "value_dram" weight / value
"mpu_tensor_transposed" "transposed_mpu_dram" / "key_dram" transposed weight / key

Note

The two alias names per kind (e.g. "shared_dram" / "shared_memory") are aliases in the current compiler.

A tensor created with any other empty_with_kind name, or one created without a kind at all, takes the default shared-memory → SRAM path; passing such a tensor to a parameter that expects one of the three kinds above raises at launch.

The "mpu_tensor" and "mpu_tensor_transposed" layouts require rank 2 or larger, a dtype bit width of 4, 8, or 16, and a shape compatible with the selected backend. See Bertha Layouts for the kind and allocation requirements.

Physical storage

A type such as (4, (70, 256)) needs an actual 256-element physical row. It does not pad a compact [4, 70] tensor. Allocate/copy the full backing tensor first, then pass its logical view:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cpu = torch.randn(4, 70, dtype=torch.bfloat16)
padded_cpu = torch.zeros(4, 256, dtype=torch.bfloat16)
padded_cpu[:, :70] = cpu
storage = torch.ops.lpu.empty_with_kind(
    [4, 256], "dram_word_interleaved", dtype=torch.bfloat16, device="lpu:0"
)
storage.copy_(padded_cpu)
logical = storage[:, :70]  # shape [4, 70], row stride 256
# Pass logical to a matching (4, (70, 256)) kernel annotation.
# For outputs, copy the whole backing to CPU, then crop the CPU result.

Shared-DRAM launch validation rejects a compact device tensor whose row stride disagrees with the annotation. Copy/read complete backing tensors rather than using host transfers through noncontiguous device views. Keep backing storage alive through stream synchronization.

Passing Tensors to a Kernel

Wrap each tensor according to the parameter's param_kinds and call the kernel inside a legato.session. The wrapping rules — top / core / universal placement, Arg versus Args — are described in Argument Binding.

Full Example

See Running the examples for the shared setup and simulator/hardware requirements.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
"""Single-core BF16 matmul with a bounded dynamic activation row count."""

import os
from pathlib import Path

import torch

import legato
import legato.model.bertha as bertha

# Store compiler artifacts and optional simulator logs in a configurable directory.
OUTPUT = Path(os.environ.get("LEGATO_EXAMPLE_OUTPUT", "legato_example_output"))


@legato.compile(
    backend=lambda ctx: bertha.Bertha(ctx, "bertha"),
    param_kinds={"a": "universal", "b": "top", "out": "core"},
)
def simple_matmul(
    a: legato.get_model().tensor_type(legato.types.float("bfloat16"), ((-1, 10), 256), "shared_dram"),
    b: legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_dram"),
    out: legato.get_model().tensor_type(legato.types.float("bfloat16"), [10, 128], "shared_dram"),
):
    """Load BF16 operands, multiply on core zero, and store the output."""
    device = legato.get_context().get_device()

    with device.get_top():
        legato.tensor.request_load(a, device.get_core(0), "sram")
        legato.tensor.request_load(b, device.get_core(0), "mpu_buffer")

    with device.get_core(0):
        weight_type = legato.get_model().tensor_type(legato.types.float("bfloat16"), [256, 128], "mpu_weight")
        loaded_a = legato.tensor.receive(a, 0, "sram")
        loaded_b = legato.tensor.receive_type(weight_type, 0, None, "mpu_buffer")
        result = loaded_a @ loaded_b
        legato.tensor.memcpy(out, result)


def main():
    """Compare the complete output with a CPU matrix product."""
    # Register the LPU PyTorch backend before allocating device tensors.
    import hart
    import torch_ha

    torch_ha._autoload()
    # HA_BACKEND=sim selects EVT0 Torch FSim; otherwise use the hardware runtime.
    if os.environ.get("HA_BACKEND") == "sim":
        OUTPUT.mkdir(parents=True, exist_ok=True)
        hart.set_fsim_env(
            hart.HaDeviceVersion.BERTHA_CLOUD_EVT0,
            hart.HaFSimComputeMode.TORCH,
            str(OUTPUT / "fsim.log"),
            "error",
        )
    hart.set_device(0)
    # Keep CPU inputs reproducible without initializing a device RNG.
    torch.random.default_generator.manual_seed(0)
    a_cpu = (torch.randn(10, 256) * 0.25).to(torch.bfloat16)
    b_cpu = (torch.randn(256, 128) * 0.25).to(torch.bfloat16)
    # Allocate the physical layout expected by the kernel and copy contiguous host data.
    a = torch.ops.lpu.empty_with_kind(list(a_cpu.shape), "dram_word_interleaved", dtype=a_cpu.dtype, device="lpu:0")
    a.copy_(a_cpu.contiguous())
    # Allocate the physical layout expected by the kernel and copy contiguous host data.
    b = torch.ops.lpu.empty_with_kind(list(b_cpu.shape), "mpu_tensor", dtype=b_cpu.dtype, device="lpu:0")
    b.copy_(b_cpu.contiguous())
    out_cpu = torch.zeros(10, 128, dtype=torch.bfloat16)
    # Allocate the physical layout expected by the kernel and copy contiguous host data.
    out = torch.ops.lpu.empty_with_kind(
        list(out_cpu.shape), "dram_word_interleaved", dtype=out_cpu.dtype, device="lpu:0"
    )
    out.copy_(out_cpu.contiguous())
    # Compile a binary and launch it on the selected device.
    with legato.session(output_type=legato.OutputType.BINARY, output_path=str(OUTPUT), launch=True):
        simple_matmul(a, legato.Arg(b, bertha.top), legato.Arg(out, bertha.core(0)))
    # Surface asynchronous execution errors before reading the result.
    hart.stream_synchronize()
    # Read back the synchronized result and compare every element with the CPU reference.
    actual = out.cpu().float()
    assert torch.isfinite(actual).all(), "Non-finite device result"
    torch.testing.assert_close(actual, (a_cpu.float() @ b_cpu.float()).float(), atol=0.05, rtol=0.05)


if __name__ == "__main__":
    main()