Broadcasting¶
Broadcasting lets you write one operation over tensors whose shapes do not
match element-for-element. This page is hardware-agnostic: it is about
shapes, Core IR, and the PerformBroadcast pass. How those results run on
Bertha (VPU, MPU, SRAM vs mpu_buffer) is on
Broadcasting on Bertha. Copy-paste kernels are in
Examples: Broadcasting.
What broadcasting is¶
In NumPy and PyTorch, a + b does not require a and b to have the same
shape. The runtime aligns dimensions from the right. A size-1 dimension
stretches to match the other operand. A missing leading dimension is treated
as size 1.
a |
b |
a + b |
|---|---|---|
[256] |
[256] |
[256] |
[1, 256] |
[256, 256] |
[256, 256] |
[4, 128] |
[128] |
[4, 128] |
1 2 3 4 | |
[4, 3] + [4, 5] is illegal: the last dimensions are both greater than 1 and
they disagree.
Legato uses the same idea for +, *, and the other elementwise operators,
and for the batch dimensions of matmul.
Two meanings in Legato¶
The word covers two related compiler jobs. The first is the NumPy rule. The second is how a rank-N Core op becomes a smaller leaf the backend can lower.
- Shape broadcast. Binary ops and matmul batch dims must be broadcast-compatible. The frontend rejects a proven mismatch. Core IR then infers the result shape.
- Rank decomposition. Even when both operands already have the same
shape,
--legato-perform-broadcastmay still rewrite the op. A rank-3layer_normbecomes four rank-1 row ops. A batchedconv_1dbecomes a loop of unbatched[X, C]convs. This is also how padded last-dim elementwise add is handled.
If you only remember one sentence: shape broadcast decides whether the program is legal; rank decomposition decides how many lower-rank ops the compiler emits.
Shape rules¶
Dimensions are compared from the right. For each pair:
- size
1matches anything (that side stretches); - a missing rank is treated as
1; - two static sizes that are both greater than 1 must be equal;
- a dynamic size (
-1) is allowed unless a static pair already proves a mismatch.
The frontend check is
shapes_have_proven_broadcast_mismatch
(legato/src/legato/diagnostics/validation.py). Core IR uses
isBinaryOpCompatible / inferBinaryOpShape
(legato/csrc/Sources/Utils/ShapeUtils.cpp).
Matmul applies the same rule to the dimensions before the last two
([N*, M, K] @ [N*, K, N]). The matrix pair (M, K) / (K, N) is not
broadcast; only the leading N* batch is.
These shapes are not supported yet:
- Column broadcast
[N, 1] + [N, C]when the last dim is physically padded (SOFT-3724).[1, C] + [N, C](a leading 1) is fine.
A different use of the word broadcast — request_load sending one
buffer to many cores, or SplitCoreBroadcastFunctions cloning a function
per core — is not this page. See the Bertha memory guide for data-loading requirements.
Logical vs physical¶
A Core dimension is a pair (logical, physical).
- Logical is the size the math uses (
70channels,128hidden). - Physical is the storage the layout allocated (
256after padding).
You write that in Python as (70, 256) or (-1, 4) (dynamic logical,
physical upper bound 4).
1 2 | |
The gap [logical, physical) is a hole. The compiler must not treat
those bytes as real elements. That is why a padded last-dim add cannot be
flattened into one long vector: flatten would mix holes into the math.
How the compiler does it¶
1 2 3 4 5 6 7 8 9 | |
PerformBroadcast (--legato-perform-broadcast) does not look at a
compute model. It reads shapes and layouts only. The plan is
DecompositionKind in
legato/csrc/Includes/Dialects/Core/Compute/ShapeDecomposition.hpp:
| Kind | Meaning |
|---|---|
None |
Already a leaf (rank at or below the leaf rank), or the op cannot be split. |
Flatten |
Reinterpret a packed prefix as one rank-1 vector (or one rank-2 matmul). One invocation. |
PrefixLoop |
hscf.for over the leading dims; each iteration runs a suffix leaf. Later hscf → scf.for. |
Which analyzer runs:
| Analyzer | Ops | Leaf |
|---|---|---|
analyzeElementwiseDecomposition |
+, *, cast, exp, … |
rank-1 |
analyzeLastAxisRowDecomposition |
last-axis softmax, RMSNorm, LayerNorm, keepdim reduce_sum |
rank-1 row |
analyzeSpatialBatchDecomposition |
conv, pool, dwconv | [X, C] or [H, W, C] |
analyzeMatmulBatchDecomposition |
matmul / grouped matmul | rank-2 matrix |
A dynamic or padded prefix must PrefixLoop. Flatten is allowed only
when every merged dimension is statically packed (logical == physical
and logical >= 0). A rank-1 elementwise vector also requires a packed
last dim. A dynamic extent is not merged, because its runtime size is not
known to match the other operand.
Affine operands (RMSNorm / LayerNorm weight and bias, softmax max) are sliced per row. Spatial weights and bias have no batch prefix: the same SSA value is forwarded to every leaf.
Worked shapes¶
Snippets below are the math only. Full request_load kernels are on
Broadcasting on Bertha and
Examples: Broadcasting.
Leading-1 add — [1, 256] + [256, 256]¶
Right-align: last dim 256 == 256, first dim 1 stretches to 256.
Result [256, 256]. Rank 2 is above the elementwise leaf, so
PerformBroadcast emits a prefix loop (or flatten, if every dim is
packed) of rank-1 adds.
1 | |
Packed same-shape add vs padded last dim¶
[4, 128] + [4, 128] with logical == physical on every dim can
Flatten into one rank-1 add of length 512.
(4, (128, 256)) + (4, (128, 256)) cannot. The last dim has holes, so
the plan is PrefixLoop: four rank-1 adds of logical length 128. The
holes stay out of the vector.
Spatial N* — conv, dwconv, pool¶
Channels-last. The hardware leaf is the unbatched suffix:
- 1D:
[X, C](conv / dwconv / pool) - 2D:
[H, W, C]
Zero or more leading dims are N*. [8, 64] conv is already a leaf
(None). [2, 8, 64] is PrefixLoop with leafRank=2 and two
invocations. Weights stay [KX, Cin, Cout] or [KX, C]; they are not
sliced with the batch.
1 2 3 4 5 6 7 8 9 | |
Last-axis rows — LayerNorm, RMSNorm, softmax¶
These ops reduce or normalize one axis. On the last axis, the leaf is
rank-1 (axis rewritten to 0). [4, 128] LayerNorm is four [128]
leaves. Affine weight / bias are rank-1 [D] and are applied per row.
1 2 3 | |
The RMSNorm / LayerNorm divisor is the logical length. Padding
[70, 256) is identity zeros, not extra channels.
Matmul batch prefix¶
[B, M, K] @ [K, N] (RHS has unit or missing batch) shares one weight
matrix across B left-hand tiles. [B, M, K] @ [B, K, N] walks both
prefixes. The leaf is a rank-2 matmul.
1 | |
Whether a backend may replay that shared RHS from a streamed buffer is a target rule, not a Core rule. Bertha's rule is on Broadcasting on Bertha.