Skip to content

Argument Binding

When a compiled kernel is called from Python, each argument can be wrapped to tell the launcher where on the device its data should be placed — at the top executor, scattered one value per core, or both. legato.Arg and legato.Args are those call-time wrappers, and legato.model.bertha provides the device markers they bind to.

1
2
3
4
5
6
7
import legato
import legato.model.bertha as bertha

kernel(
    legato.Arg(weight, bertha.top),          # one tensor at top
    legato.Args(per_core_out, bertha.all_cores),  # one tensor per core
)

Two Axes: param_kinds vs. Markers

param_kinds (see Decorators) and device markers share the words "top" and "core", but they describe two different things and must not be conflated:

  • param_kinds is "compile-time" and baked into the IR. It says where the parameter symbol is referenced inside the kernel body — which executor scope uses it.
  • Device markers are "runtime", at the call site. They say where and how the argument's data is placed by the launcher.

At launch, Legato cross-checks the two for consistency — for example, a parameter the kernel only uses on cores cannot be bound to top.

Device Markers

Markers live in legato.model.bertha. Use top and all_cores directly; core takes core indices.

Marker Placement
bertha.top The top executor.
bertha.all_cores Every core — the per-core scatter target.
bertha.core(*idx), e.g. core(0, 1, 2) A specific subset of cores.

core(...) reports the same "core" kind as all_cores; the difference is that only the listed cores are bound. At launch, cores outside the subset are filled with a garbage slot so every core keeps the same argument count.

legato.Arg

Arg(value, target) binds a single value to a device target.
The target is either one marker or a tuple of markers.

1
2
3
legato.Arg(b, bertha.top)                     # single value at top
legato.Arg(b, bertha.all_cores)               # single core-distributed buffer
legato.Arg(w, (bertha.top, bertha.all_cores)) # one value placed on top AND cores

A multi-marker target — a tuple of markers such as (top, all_cores) — binds one value to several executors at once. That is only valid for a universal parameter; a parameter pinned to one executor cannot accept a multi-executor placement. Note that core(0, 1, 2) is not a multi-marker target: it is a single core-kind target naming a subset of cores.

legato.Args

Args(values, target) is the core-distributed wrapper: it carries a sequence of per-core values that are scattered one per core. It is a single logical kernel argument whose values map one-per-core — each core's kernel instance sees its own value through the one parameter.

1
2
legato.Args([t0, t1, ...], bertha.all_cores)  # i-th value -> core i
legato.Args([t0, t1, t2], bertha.core(0, 1, 2))  # i-th value -> i-th listed core

Only per-core scatter targets are valid: all_cores or a core(...) subset. Passing top or a multi-marker target raises ValueError at construction time, because a single-executor (or multi-executor) target cannot consume multiple values through one parameter. To place data on a single or multiple executors, use Arg instead (duplicating the argument separately if needed).

With a core(...) subset, the number of values must equal the number of listed cores.

Bare Arguments

An argument passed without a wrapper is a bare argument. It is only valid for a universal parameter (or a kernel that declared no param_kinds at all, which makes every parameter implicitly universal). The launcher places bare arguments automatically.

What Binds to What

param_kinds (columns) constrains which wrapper form (rows) is accepted. Below, ✅ compiles, ❌ raises RuntimeError at launch, and ⛔ raises ValueError while constructing the Args.

Argument form universal / undeclared top core
bare value ✅ ❌ ❌
Arg(t, top) ❌ ✅ ❌
Arg(t, all_cores) ❌ ❌ ✅
Arg(t, core(...)) ❌ ❌ ✅ subset
Arg(t, (top, all_cores)) ✅ ❌ ❌
Arg(t, (top, core(...))) ✅ ❌ ❌
Args(vs, all_cores) ❌ ❌ ✅ scatter
Args(vs, core(...)) ❌ ❌ ✅ subset scatter
Args(vs, top) ⛔ ⛔ ⛔
Args(vs, (top, all_cores)) ⛔ ⛔ ⛔

The rules behind the table:

  1. Bare values are accepted only by universal (or undeclared) parameters.
  2. A single-kind target must match the parameter's kind exactly. A universal parameter rejects a single-executor target — it is used across executors, so pinning it to one is ambiguous. Pass it bare, or give it a full multi-executor target.
  3. A multi-executor tuple such as (top, all_cores) is valid only on a universal parameter, and only with Arg.
  4. core(*idx) behaves like all_cores (a "core"-kind target) but binds only the listed cores; unbound cores are garbage-filled at launch.
  5. Args is per-core scatter only (all_cores or a core(...) subset).

Instruction Table (Host) Arguments

The instruction table is a small buffer for passing CPU parameters using values from host memory. It is mainly used for pointers and constants; this section explains how to place tensors in it. Because the instruction table is very small, avoid passing large tensors through it, as doing so may cause a compilation error.

A parameter annotated with the "instruction_table" kind — legato.get_model().tensor_type(dtype, shape, "instruction_table") — is not a device-resident tensor. Its element data is embedded verbatim into the kernel's instruction table heap, so the launcher reads the argument's raw host bytes rather than a device pointer (it becomes a hart LegatoArrayArg, not a LegatoTensorArg).

Because of that, the value you pass must be a CPU torch.Tensor:

1
2
3
4
5
6
7
8
9
@legato.compile(backend=get_bertha)
def rank1_it_tensor(
    a: legato.get_model().tensor_type(legato.types.integer("int32"), [32], "instruction_table"),
):
    ...


with legato.session(output_type=legato.OutputType.BINARY, output_path="read_it_tensor"):
    rank1_it_tensor(torch.randint(0, 32, (32,), dtype=torch.int32))  # host CPU tensor
  • The value must be a torch.Tensor — a non-tensor (e.g. a Python list/int) raises TypeError.
  • It must be CPU-resident — a device (non-CPU) tensor raises ValueError, because an instruction table argument is host constant data, not device memory.
  • Its bytes are taken little-endian in the tensor's own dtype, which must match the annotation's dtype and shape (the tensor type check below still applies).
  • Placement wrappers (Arg / Args / bare) and routing work exactly as for any other argument — the only difference is host bytes vs. a device pointer.

EVT1 supports scalar reads from instruction table tensors and integer casts of those values, including indices used for a DRAM view. It also supports tensor.gather with instruction table integer indices and a sliced_dram weight table whose row count is divisible by the DRAM channel count. Instruction table tensor views are not supported. The CPU-resident input requirement above applies to both generations.

See Instruction Table ABI for the on-device layout.

Launch-Time Validation

Wrappers are stripped on every call, so the compiler only ever sees raw values. Two checks run just before launch:

  • Target check — each wrapper's device target is cross-checked against the parameter's param_kinds, per the table above.
  • Tensor type check — every value's dtype, rank, and static dimensions are compared against the parameter annotation. For an Args, every value is checked, and dynamic dimensions (declared as -1) must agree across the per-core values. A device-resident tensor's layout kind is also cross-checked against the annotation's memory kind — see Choosing the Kind for the empty_with_kind ↔ annotation kind matching table.

Example

A kernel that declares a as universal, b as top, and out as core binds each argument at the call site to match. a, b, and out are device-resident tensors; see Simple Matmul for the full kernel and the device-tensor setup.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
@legato.compile(
    backend=get_bertha,
    param_kinds={"a": "universal", "b": "top", "out": "core"},
)
def simple_matmul(a, b, out):
    ...


with legato.session(output_type=legato.OutputType.BINARY, output_path="simple_matmul"):
    simple_matmul(
        a,                                # universal parameter -> bare
        legato.Arg(b, bertha.top),        # top parameter
        legato.Arg(out, bertha.core(0)),  # core parameter (kernel uses core 0)
    )

Common Mistakes

  • Passing a bare value to a top or core parameter — wrap it with legato.Arg(..., target).
  • Pinning a universal parameter to a single executor (e.g. Arg(t, bertha.top)) — pass it bare, or give it a multi-executor target such as (bertha.top, bertha.all_cores).
  • Using Args with a top or tuple target — Args is per-core scatter only; this fails at construction.
  • Confusing the two axes: param_kinds describes the kernel body, markers describe runtime placement.