Math Operations¶
Math operations live under legato.math. They emit Core compute.* ops
and keep the logical tensor shape: a physically padded hole
[logical, physical) is not an extra element.
This page is the language / Core contract. Shape broadcast and rank decomposition are in Broadcasting. Worked kernels are in Math And Masking and Causal Attention. Target-specific lowering belongs under Language/Bertha, not here.
Shared conventions¶
Compiled vs interpreted¶
Tensor operands (input, weight, bias, query, key, a supplied
softmax maximum) must be compiled values: results of receive,
allocate, @, or another legato.math call.
Geometry and control arguments are interpreted at compile time:
axis, keepdim, stride, dilation, padding, padding_mode,
kernel_size, epsilon, scale, causal, group_size. They are
Python / constexpr values, not tensors.
mask_start on attention_scores is the exception: it is a compiled
index so decode can pass a dynamic cache offset.
Axis¶
Negative axis counts from the end, then the frontend rewrites it to
[0, rank):
1 2 3 | |
Logical vs physical¶
Math uses the logical extent. On a last dim written (70, 256),
RMSNorm / LayerNorm / softmax / reductions see length 70. The hole
[70, 256) is filled with the op's identity (0 for sum and norms,
-inf for max / softmax shift) so a physical sweep matches a logical
one. See Broadcasting.
Channels-last spatial tensors¶
Conv, depthwise conv, and pooling are channels-last:
| Rank class | Activation | Weight |
|---|---|---|
| 1D | [N*, X, C] or [N*, X, Cin] |
[KX, C] or [KX, Cin, Cout] |
| 2D | [N*, H, W, C] or [N*, H, W, Cin] |
[KH, KW, C] or [KH, KW, Cin, Cout] |
N* is zero or more leading batch dims. They are copied to the result
and become a prefix loop of unbatched leaves
([X, C] / [H, W, C]). Weights and bias have no N* prefix:
the same SSA value is forwarded to every leaf. See
Broadcasting: spatial N*.
Element-wise unary operations¶
Each call maps one function over every logical element and returns a tensor of the same rank, extents, and dtype. There is no axis and no cross-element reduction, so a packed rank-N tensor may flatten to one vector leaf; a padded last dim must stay a prefix of rank-1 leaves.
1 2 3 | |
| Call | f(x) |
Domain |
|---|---|---|
exp |
e^x |
float |
sqrt |
√x |
float; x < 0 is non-finite |
abs |
\|x\| |
float or integer |
log |
ln(x) (natural log) |
float; x ≤ 0 is non-finite |
sin / cos / tan |
trig in radians | float |
The compiler does not insert domain checks. Out-of-domain inputs follow
IEEE-754 for the operand dtype. log is not log2 / log10.
1 2 3 | |
exp¶
1 | |
Compute element-wise exponential of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with exp applied element-wise.
sqrt¶
1 | |
Compute element-wise square root of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with sqrt applied element-wise.
abs¶
1 | |
Compute element-wise absolute value of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with abs applied element-wise.
log¶
1 | |
Compute element-wise natural logarithm of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with log applied element-wise.
sin¶
1 | |
Compute element-wise sine of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with sin applied element-wise.
cos¶
1 | |
Compute element-wise cosine of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with cos applied element-wise.
tan¶
1 | |
Compute element-wise tangent of a tensor.
Parameters:
tensor(Value): The input tensor.
Returns:
result(Value): Tensor with tan applied element-wise.
Backend support¶
This API reference describes frontend operations and mathematical semantics.
Current EVT0 lowering supports recognized attention/normalization patterns,
softmax with a supplied maximum, and standalone reduce_max under the
constraints below. A known unsupported standalone
reduction normally fails executor assignment with headline LE0121 and a
recorded reason such as LE2119 (non-last axis) or LE2120 (missing
keepdim=True). softmax(scores, -1) emits a maximum reduction and must satisfy
its target constraints too. See the compile diagnostics example
and the validated
Math and masking and
Causal attention examples.
Likewise, standalone full/full_like and general mask compositions do not
currently lower to EVT0 binaries. Prefer the supported attention_scores
pattern for causal attention, or supply masked scores and their post-mask
maximum explicitly. Frontend MLIR alone does not test these backend paths.
On EVT1, last-axis BF16 softmax can start with reduce_max(scores, -1, True)
and continue through sub → exp → reduce_sum(..., -1, True) → div.
The simulator regression uses [32, 256] inputs, and the reduced width must
be a multiple of 64 BF16 elements. This recognized pattern does not establish
standalone support for every reduction or dtype on EVT1. See the
EVT1 coverage table.
Reduction operations¶
A reduction collapses one axis. It is not a full-tensor fold: every
other dimension is kept. keepdim=False (default) deletes axis;
keepdim=True replaces it with size 1 so the result still broadcasts
against the input (scores - reduce_max(scores, -1, True)).
On Bertha EVT0, standalone reduce_max uses the VPU for row-major
FP16, BF16, and FP32 tensors when keepdim=True and the reduced axis is the
last axis. Its logical width must be static and between 1 and 16384 elements.
Prefix dimensions may be dynamic, and aligned row views may have a larger
physical pitch. Physical capacity must hold the logical width rounded up to
64 elements. Aligned rows need no fill; otherwise lowering fills only the
consumed source tail with -inf as required by the input contract.
1 2 3 4 5 6 7 8 9 | |
fold is:
1 2 3 | |
Only logical elements participate. A physically padded tail is the
identity of that fold (-inf / +inf / 0), not extra data.
axis must be an interpreted integer. The frontend accepts rank-dropping
keepdim=False, but Bertha physical-shape inference can assert when that result
has an incompatible physical extent. Use keepdim=True in attention/softmax
patterns; this does not add standalone backend support to generic reductions. The _KEEPDIM_DEFAULT binding is listed with each API below.
1 2 | |
reduce_max¶
1 | |
Compute maximum along an axis.
Parameters:
tensor(Value): The input tensor.axis(Value): Axis to reduce along. Negative values count from the end.keepdim(Value): If True, the reduced axis is retained with size 1. Defaults to False. Default:_KEEPDIM_DEFAULT.
Returns:
result(Value): Reduced tensor. If keepdim=False (default), the axis is removed; if keepdim=True, the axis dimension becomes 1.
1 2 | |
reduce_min¶
1 | |
Compute minimum along an axis.
Parameters:
tensor(Value): The input tensor.axis(Value): Axis to reduce along. Negative values count from the end.keepdim(Value): If True, the reduced axis is retained with size 1. Defaults to False. Default:_KEEPDIM_DEFAULT.
Returns:
result(Value): Reduced tensor. If keepdim=False (default), the axis is removed; if keepdim=True, the axis dimension becomes 1.
1 2 | |
reduce_sum¶
1 | |
Compute sum along an axis.
Parameters:
tensor(Value): The input tensor.axis(Value): Axis to reduce along. Negative values count from the end.keepdim(Value): If True, the reduced axis is retained with size 1. Defaults to False. Default:_KEEPDIM_DEFAULT.
Returns:
result(Value): Reduced tensor. If keepdim=False (default), the axis is removed; if keepdim=True, the axis dimension becomes 1.
1 2 | |
Softmax¶
softmax is shape-preserving: result.shape == tensor.shape. It is
the numerically stable map
1 2 3 4 5 | |
When maximum is omitted, the frontend emits a keepdim=True
reduce_max on axis and then compute.softmax. When you pass
maximum, only compute.softmax is emitted — the compiler does
not check that maximum equals max(X, axis).
maximum may keep axis as size 1 or squeeze it. Dtype and the
non-reduced extents must match X. Its values are the caller's
contract:
- Prefer a finite max of the post-mask tensor.
- Reusing a pre-mask max is only an upper bound. A large gap, or a
row that is all
-infafter masking, can underflow to0/0 = NaN.
Softmax is group-agnostic. A GQA score tensor [group_size * q_len, seq_k]
is ordinary rows; it does not unpack heads.
Last-axis softmax becomes rank-1 row leaves; see Broadcasting: last-axis rows.
1 2 | |
softmax¶
1 | |
Numerically-stable softmax over axis.
Emits a keepdim compute.reduce_max before compute.softmax when
maximum is omitted or None. A supplied maximum emits only
compute.softmax. Group-agnostic: it treats the grouped
[group_size * q_len, seq_k] score tensor as ordinary rows.
Parameters:
tensor(Value): The input scores (compiled).axis(Value): Softmax axis. Negative values count from the end.maximum(Value | None): Optional compiled precomputed shift with matching dtype and runtime non-reduced extents. It may retainaxiswith size one or squeeze it, and must have sufficient storage and bounds. The caller is responsible for its numerical values. For numerical stability, prefer a finite maximum of the actual post-mask tensor. Reusing a pre-mask maximum across nonpositive masks supplies only an upper bound; large gaps or all-masked rows can underflow or produce NaN. These numerical preconditions are not checked by the compiler. Default:None.
Returns:
result(Value): Softmax probabilities, the same shape astensor.
Attention scores¶
attention_scores is scaled Q @ Kᵀ with optional causal masking.
key is already matmul-ready (Kᵀ): there is no implicit transpose.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
G = 1 is ordinary single-head / per-head attention:
[q_len, d_k] @ [d_k, seq_k] → [q_len, seq_k].
G > 1 is grouped-query packing. Adjacent d_k bands of query share
one key. The result stacks those G score matrices token-major
([G * q_len, seq_k]), not head-major. The causal diagonal is the
token index i inside each band, not the packed row
g*q_len + i.
mask_start shifts the diagonal for decode: a new token at cache
offset past_len uses mask_start=past_len so it attends
[0, past_len] and masks the rest of the K window. A constant
mask_start must satisfy 0 <= mask_start < seq_k when seq_k is
static.
scale is an interpreted positive float (typically 1/sqrt(d_k)).
causal and group_size are interpreted. Query and key must share a
dtype and have rank ≥ 2. Static inner dims must satisfy
query.shape[-1] == group_size * key.shape[-2].
1 2 3 | |
attention_scores¶
1 | |
Compute scaled grouped-query attention scores (QK^T) with optional causal masking.
For group_size > 1, query packs adjacent d_k bands that share
one K head. Scores are token-major [group_size * q_len, seq_k].
Parameters:
query(Value): Query tensor[q_len, group_size * d_k](compiled).key(Value): Key tensor in matmul-ready K^T form[d_k, seq_k](compiled).scale(Value): Score multiplier, e.g.1/sqrt(d_k)(interpreted float).causal(Value): Whether to apply the causal diagonal mask (interpreted bool).group_size(Value): Grouped-query reuse factor,>= 1(interpreted int).mask_start(Value | None): Optional causal-mask offset index (compiled); valid only whencausalis true. Defaults to no offset. Default:None.
Returns:
result(Value): The scaled (and optionally masked) scores[group_size * q_len, seq_k].
Grouped matrix multiplication¶
legato.math.matmul(lhs, rhs, group_size) is not Python @.
@ is ordinary matmul (Arithmetic Operations).
This helper is the GQA P @ V companion to attention_scores:
token-major lhs rows are partitioned into G bands that share one
rhs.
1 2 3 4 5 6 7 8 9 | |
G = 1 is ordinary [M, K] @ [K, N] → [M, N]. G > 1 writes the
G head results as adjacent column bands ([M, G*N]), which is
the concat order used by grouped-query attention.
1 2 3 | |
matmul¶
1 | |
Grouped P@V matrix multiplication.
For group_size > 1, lhs is token-major [group_size * M, K] and
the result is banded [M, group_size * N].
Parameters:
lhs(Value): Left tensor[group_size * M, K](compiled).rhs(Value): Right (shared) tensor[K, N](compiled).group_size(Value): Grouped reuse factor,>= 1(interpreted int).
Returns:
result(Value): The product[M, group_size * N].
Spatial geometry¶
Conv, depthwise conv, and pooling share one geometry: channels-last layout, per-side padding tuples, and one output-length formula per spatial axis. The subsections below are the contract for defaults and result shape. Window sampling is after that.
Tuple layout¶
Numeric geometry is always a Python tuple (or list) of ints. It is
not PyTorch's scalar padding=1 or symmetric (H, W) pad pair.
| Rank | stride / dilation / kernel_size |
padding |
|---|---|---|
| 1D | (X,) |
(left, right) |
| 2D | (H, W) |
(top, bottom, left, right) |
Length is checked. A 1D call with stride=(1, 1) or a 2D call with
padding=(1, 1) is a frontend error. Each call's default constants
are listed under that API.
Output shape¶
The result rank equals the input rank. Leading N* (everything before
the spatial suffix) is copied unchanged. Each spatial axis is replaced
by the formula below. The last axis is the output channel.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | |
padding_mode does not change O. Reflect / replicate / circular
only change which input element a tap reads; the padded length used in
the formula is still I + pad_before + pad_after.
If a spatial I is dynamic (-1 / ?), that O stays dynamic. A
static I and K always produce a static O, and that O must be
> 0. stride, dilation, and kernel_size must be positive;
padding entries must be ≥ 0.
N* may be empty ([X, C] is already a leaf) or several batch dims
([2, 4, 8, 64] 1D conv keeps [2, 4] and only rewrites 8 → Xout).
The weight / kernel_size never contribute a batch axis. Worked
numbers for each call sit with that API.
How a padded coordinate is sampled¶
A spatial index p lives in padded space
[-pad_before, I + pad_after). The source index is s = p.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | |
2D applies the 1D rule independently to H and W.
padding_mode is a conv-only string from _VALID_PADDING_MODES
("zeros", "reflect", "replicate", "circular"). Depthwise conv
is implicit "zeros". Pooling has no mode: max pooling uses
pad_value = -inf so a pad never wins the window; average pooling
uses pad_value = 0 in the sum but still divides by the full
kernel area (see Pooling). A target may reject some
modes; that is a backend rule, not part of this contract.
Window walk¶
For one output coordinate o on one axis:
1 2 3 4 | |
Depthwise convolution¶
Each channel is an independent FIR. There is no Cin → Cout mix, no
bias, and no dilation / padding_mode argument (dilation is always
1; pads are zeros).
1 2 3 4 5 6 7 | |
Unrolled 1D:
1 2 3 4 5 6 7 8 | |
2D adds a KH loop over H with (pad_top, pad_bottom, stride_h).
C on the weight must match C on the input. Kernel dims are taps,
not batch.
dwconv_1d¶
1 | |
Channels-last 1D depthwise convolution.
input is [N*, X, C], weight is [KX, C], and the result is
[N*, Xout, C]. stride has length 1; padding is (left, right).
1 2 3 4 5 6 7 8 9 10 | |
dwconv_2d¶
1 | |
Channels-last 2D depthwise convolution.
input is [N*, H, W, C], weight is [KH, KW, C], and the
result is [N*, Hout, Wout, C]. stride is (H, W); padding is
(top, bottom, left, right).
1 2 3 4 5 6 7 8 9 | |
Convolution¶
Dense (non-depthwise) convolution mixes Cin into Cout and then
optionally adds a rank-1 bias. Same channels-last layout and N* rule
as depthwise; the weight gains an output-channel axis.
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Unrolled 1D with padding_mode:
1 2 3 4 5 6 7 8 9 10 11 | |
Cin on the input must match Cin on the weight. Bias, if present,
must be rank-1 [Cout] with the same dtype. The frontend emits
compute.conv.
conv_1d¶
1 | |
Channels-last 1D convolution.
input is [N*, X, Cin], weight is [KX, Cin, Cout], optional
bias is [Cout], and the result is [N*, Xout, Cout]. stride
and dilation have length 1; padding is (left, right).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | |
conv_2d¶
1 | |
Channels-last 2D convolution.
input is [N*, H, W, Cin], weight is [KH, KW, Cin, Cout],
optional bias is [Cout], and the result is [N*, Hout, Wout, Cout].
stride and dilation are (H, W); padding is
(top, bottom, left, right).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
Pooling¶
Pooling walks the same windows as a unit-dilation conv, then reduces the window per channel. There is no weight.
1 2 3 4 5 6 7 8 9 10 11 12 | |
Average pooling always divides by the full kernel area, including
pads (count_include_pad=True in PyTorch terms). A window that hangs
off the edge is therefore smaller in the numerator than in the
denominator. This is the Core contract, not avg_pool with
count_include_pad=False.
Max pooling pads with -inf, so a pad cannot become the window max
unless the whole window is padded (that geometry is rejected because
O would be non-positive or the window would be empty of real taps
only if pads fill it — the -inf then stays).
max_pool_1d¶
1 | |
Channels-last 1D max pooling over [N*, X, C].
1 2 3 4 5 6 7 8 | |
avg_pool_1d¶
1 | |
Channels-last 1D average pooling over [N*, X, C].
1 2 3 4 5 6 | |
max_pool_2d¶
1 | |
Channels-last 2D max pooling over [N*, H, W, C].
1 2 3 4 5 6 7 | |
avg_pool_2d¶
1 | |
Channels-last 2D average pooling over [N*, H, W, C].
1 2 3 4 | |
Normalization¶
Both ops normalize one axis and keep the input shape. epsilon is
an interpreted finite ≥ 0 scalar added under the square root for
stability. The divisor is the logical axis length D, never the
physically padded width. _AXIS_DEFAULT and _EPSILON_DEFAULT are
listed with each API below.
Affine vectors are per-channel along that axis: rank-1 [D], or the
same rank as input with leading dims 1 and last dim D. They
broadcast across the non-reduced dims. Dtype must match input.
Last-axis tensors become rank-1 row leaves; see Broadcasting: last-axis rows.
RMSNorm¶
weight is required. There is no mean subtraction and no bias.
1 2 3 4 5 | |
This is x / sqrt(mean(x²) + ε) * weight.
LayerNorm¶
Mean-center, then scale by the biased standard deviation
(/ D, not / (D-1)). weight and bias are independently optional.
1 2 3 4 5 6 7 8 9 10 11 | |
Order is fixed: center → divide by sqrt(var + ε) → scale → shift.
Omitting both affines is still LayerNorm (normalize only).
1 2 3 4 | |
rms_norm¶
1 | |
Root-mean-square normalization along axis with a required affine weight.
1 2 3 4 5 | |
layer_norm¶
1 | |
Mean-centered layer normalization along axis with optional affine operands.
1 2 3 4 5 | |