Argument Binding¶
When a compiled kernel is called from Python, each argument can be wrapped to
tell the launcher where on the device its data should be placed — at the top
executor, scattered one value per core, or both. legato.Arg and legato.Args
are those call-time wrappers, and legato.model.bertha provides the device
markers they bind to.
1 2 3 4 5 6 7 | |
Two Axes: param_kinds vs. Markers¶
param_kinds (see Decorators) and device markers share the
words "top" and "core", but they describe two different things and must not
be conflated:
param_kindsis "compile-time" and baked into the IR. It says where the parameter symbol is referenced inside the kernel body — which executor scope uses it.- Device markers are "runtime", at the call site. They say where and how the argument's data is placed by the launcher.
At launch, Legato cross-checks the two for consistency — for example, a parameter
the kernel only uses on cores cannot be bound to top.
Device Markers¶
Markers live in legato.model.bertha. Use top and all_cores directly;
core takes core indices.
| Marker | Placement |
|---|---|
bertha.top |
The top executor. |
bertha.all_cores |
Every core — the per-core scatter target. |
bertha.core(*idx), e.g. core(0, 1, 2) |
A specific subset of cores. |
core(...) reports the same "core" kind as all_cores; the difference is that
only the listed cores are bound. At launch, cores outside the subset are filled
with a garbage slot so every core keeps the same argument count.
legato.Arg¶
Arg(value, target) binds a single value to a device target.
The target is either one marker or a tuple of markers.
1 2 3 | |
A multi-marker target — a tuple of markers such as (top, all_cores) — binds one
value to several executors at once. That is only valid for a universal
parameter; a parameter pinned to one executor cannot accept a multi-executor
placement. Note that core(0, 1, 2) is not a multi-marker target: it is a single core-kind target naming a subset of cores.
legato.Args¶
Args(values, target) is the core-distributed wrapper: it carries a sequence
of per-core values that are scattered one per core. It is a single logical
kernel argument whose values map one-per-core — each core's kernel instance sees
its own value through the one parameter.
1 2 | |
Only per-core scatter targets are valid: all_cores or a core(...) subset.
Passing top or a multi-marker target raises ValueError at construction time,
because a single-executor (or multi-executor) target cannot consume multiple
values through one parameter. To place data on a single or multiple executors, use Arg
instead (duplicating the argument separately if needed).
With a core(...) subset, the number of values must equal the number of listed
cores.
Bare Arguments¶
An argument passed without a wrapper is a bare argument. It is only valid for
a universal parameter (or a kernel that declared no param_kinds at all, which
makes every parameter implicitly universal). The launcher places bare arguments
automatically.
What Binds to What¶
param_kinds (columns) constrains which wrapper form (rows) is accepted. Below,
✅ compiles, ❌ raises RuntimeError at launch, and ⛔ raises ValueError while
constructing the Args.
| Argument form | universal / undeclared |
top |
core |
|---|---|---|---|
bare value |
✅ | ❌ | ❌ |
Arg(t, top) |
❌ | ✅ | ❌ |
Arg(t, all_cores) |
❌ | ❌ | ✅ |
Arg(t, core(...)) |
❌ | ❌ | ✅ subset |
Arg(t, (top, all_cores)) |
✅ | ❌ | ❌ |
Arg(t, (top, core(...))) |
✅ | ❌ | ❌ |
Args(vs, all_cores) |
❌ | ❌ | ✅ scatter |
Args(vs, core(...)) |
❌ | ❌ | ✅ subset scatter |
Args(vs, top) |
⛔ | ⛔ | ⛔ |
Args(vs, (top, all_cores)) |
⛔ | ⛔ | ⛔ |
The rules behind the table:
- Bare values are accepted only by
universal(or undeclared) parameters. - A single-kind target must match the parameter's kind exactly. A
universalparameter rejects a single-executor target — it is used across executors, so pinning it to one is ambiguous. Pass it bare, or give it a full multi-executor target. - A multi-executor tuple such as
(top, all_cores)is valid only on auniversalparameter, and only withArg. core(*idx)behaves likeall_cores(a"core"-kind target) but binds only the listed cores; unbound cores are garbage-filled at launch.Argsis per-core scatter only (all_coresor acore(...)subset).
Instruction Table (Host) Arguments¶
The instruction table is a small buffer for passing CPU parameters using values from host memory. It is mainly used for pointers and constants; this section explains how to place tensors in it. Because the instruction table is very small, avoid passing large tensors through it, as doing so may cause a compilation error.
A parameter annotated with the "instruction_table" kind —
legato.get_model().tensor_type(dtype, shape, "instruction_table") — is not
a device-resident tensor. Its element data is embedded verbatim into the kernel's
instruction table heap, so the launcher reads the argument's raw host bytes
rather than a device pointer (it becomes a hart LegatoArrayArg, not a
LegatoTensorArg).
Because of that, the value you pass must be a CPU torch.Tensor:
1 2 3 4 5 6 7 8 9 | |
- The value must be a
torch.Tensor— a non-tensor (e.g. a Pythonlist/int) raisesTypeError. - It must be CPU-resident — a device (non-CPU) tensor raises
ValueError, because an instruction table argument is host constant data, not device memory. - Its bytes are taken little-endian in the tensor's own dtype, which must match the annotation's dtype and shape (the tensor type check below still applies).
- Placement wrappers (
Arg/Args/ bare) and routing work exactly as for any other argument — the only difference is host bytes vs. a device pointer.
EVT1 supports scalar reads from instruction table tensors and integer casts of
those values, including indices used for a DRAM view. It also supports
tensor.gather with instruction table integer indices and a sliced_dram
weight table whose row count is divisible by the DRAM channel count.
Instruction table tensor views are not supported. The CPU-resident input
requirement above applies to both generations.
See Instruction Table ABI for the on-device layout.
Launch-Time Validation¶
Wrappers are stripped on every call, so the compiler only ever sees raw values. Two checks run just before launch:
- Target check — each wrapper's device target is cross-checked against the
parameter's
param_kinds, per the table above. - Tensor type check — every value's dtype, rank, and static dimensions are
compared against the parameter annotation. For an
Args, every value is checked, and dynamic dimensions (declared as-1) must agree across the per-core values. A device-resident tensor's layout kind is also cross-checked against the annotation's memory kind — see Choosing the Kind for theempty_with_kind↔ annotationkindmatching table.
Example¶
A kernel that declares a as universal, b as top, and out as core
binds each argument at the call site to match. a, b, and out are
device-resident tensors; see Simple Matmul for
the full kernel and the device-tensor setup.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
Common Mistakes¶
- Passing a bare value to a
toporcoreparameter — wrap it withlegato.Arg(..., target). - Pinning a
universalparameter to a single executor (e.g.Arg(t, bertha.top)) — pass it bare, or give it a multi-executor target such as(bertha.top, bertha.all_cores). - Using
Argswith atopor tuple target —Argsis per-core scatter only; this fails at construction. - Confusing the two axes:
param_kindsdescribes the kernel body, markers describe runtime placement.