Math And Masking¶
Operations live under legato.math; tensor selection is legato.tensor.where.
A language operation being accepted by the frontend does not imply that its
standalone form can lower to an EVT0 binary.
This runnable example supplies already-masked scores and their row maxima
from the host, then performs legato.math.softmax(scores, -1, maximum) on the
device. It keeps mask and maximum preparation on the host and avoids a
standalone full_like fill. Standalone last-axis reduce_max is also supported
under the EVT0 constraints below.
For a mask generated on device as part of a supported attention pattern, see
Causal Attention.
maximum must be the maximum of each post-mask row. The caller must keep at
least one finite score in every row. The example uses a finite -30 mask for
BF16 scores; it checks the exact same finite-mask formulation on CPU.
Download math_and_masking.py, or run the checked-out files. See Running the examples for environment setup.
1 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | |
legato.math.exp(tensor)legato.math.sqrt(tensor)legato.math.reduce_max(tensor, axis)legato.math.reduce_min(tensor, axis)legato.math.reduce_sum(tensor, axis)legato.tensor.tril(tensor, diagonal)legato.tensor.where(condition, true_value, false_value)
Joined attention scores on Bertha EVT0¶
For attention split across head bands, add the partial score tensors before
subtracting legato.math.reduce_max(scores, -1, True). The sum of the separate
band maxima can exceed every joined score and underflow all exponentials.
The standalone VPU maximum lowering supports FP16, BF16, and FP32 last-axis
reductions with keepdim=True, and row-major static logical rows of 1–16384
elements. Physical capacity must hold the row rounded up to a 64-element group. Lowering
fills an unaligned consumed tail with -inf, while aligned rows need no fill.
Capacity beyond the reduction extent is untouched. Each row address retains
its physical pitch; dynamic prefix dimensions are allowed, but dynamic logical
reduction widths remain unsupported. Attention-specific MPU maximum fusion
remains unchanged. A known unsupported axis or dtype is recorded by the
component check and normally reported at executor assignment. See
the compile diagnostics example for headline and
candidate reason codes. For
sliding-window attention, reduce the maximum after applying the window mask so an
excluded attention sink cannot determine the shift.