Skip to content

Broadcasting

Broadcasting lets you write one operation over tensors whose shapes do not match element-for-element. This page is hardware-agnostic: it is about shapes, Core IR, and the PerformBroadcast pass. How those results run on Bertha (VPU, MPU, SRAM vs mpu_buffer) is on Broadcasting on Bertha. Copy-paste kernels are in Examples: Broadcasting.

What broadcasting is

In NumPy and PyTorch, a + b does not require a and b to have the same shape. The runtime aligns dimensions from the right. A size-1 dimension stretches to match the other operand. A missing leading dimension is treated as size 1.

a b a + b
[256] [256] [256]
[1, 256] [256, 256] [256, 256]
[4, 128] [128] [4, 128]
1
2
3
4
  [1, 256]     each of the 256 columns is reused down every row
+ [256, 256]
-----------
  [256, 256]

[4, 3] + [4, 5] is illegal: the last dimensions are both greater than 1 and they disagree.

Legato uses the same idea for +, *, and the other elementwise operators, and for the batch dimensions of matmul.

Two meanings in Legato

The word covers two related compiler jobs. The first is the NumPy rule. The second is how a rank-N Core op becomes a smaller leaf the backend can lower.

  1. Shape broadcast. Binary ops and matmul batch dims must be broadcast-compatible. The frontend rejects a proven mismatch. Core IR then infers the result shape.
  2. Rank decomposition. Even when both operands already have the same shape, --legato-perform-broadcast may still rewrite the op. A rank-3 layer_norm becomes four rank-1 row ops. A batched conv_1d becomes a loop of unbatched [X, C] convs. This is also how padded last-dim elementwise add is handled.

If you only remember one sentence: shape broadcast decides whether the program is legal; rank decomposition decides how many lower-rank ops the compiler emits.

Shape rules

Dimensions are compared from the right. For each pair:

  • size 1 matches anything (that side stretches);
  • a missing rank is treated as 1;
  • two static sizes that are both greater than 1 must be equal;
  • a dynamic size (-1) is allowed unless a static pair already proves a mismatch.

The frontend check is shapes_have_proven_broadcast_mismatch (legato/src/legato/diagnostics/validation.py). Core IR uses isBinaryOpCompatible / inferBinaryOpShape (legato/csrc/Sources/Utils/ShapeUtils.cpp).

Matmul applies the same rule to the dimensions before the last two ([N*, M, K] @ [N*, K, N]). The matrix pair (M, K) / (K, N) is not broadcast; only the leading N* batch is.

These shapes are not supported yet:

  • Column broadcast [N, 1] + [N, C] when the last dim is physically padded (SOFT-3724). [1, C] + [N, C] (a leading 1) is fine.

A different use of the word broadcast — request_load sending one buffer to many cores, or SplitCoreBroadcastFunctions cloning a function per core — is not this page. See the Bertha memory guide for data-loading requirements.

Logical vs physical

A Core dimension is a pair (logical, physical).

  • Logical is the size the math uses (70 channels, 128 hidden).
  • Physical is the storage the layout allocated (256 after padding).

You write that in Python as (70, 256) or (-1, 4) (dynamic logical, physical upper bound 4).

1
2
# 4 rows, last dim logical 70 stored in a 256-wide row
x: (4, (70, 256))

The gap [logical, physical) is a hole. The compiler must not treat those bytes as real elements. That is why a padded last-dim add cannot be flattened into one long vector: flatten would mix holes into the math.

How the compiler does it

1
2
3
4
5
6
7
8
9
Python  legato.math.*  /  a + b
   ↓
Core IR
   ↓
AssignExecutor / AssignMemory / InferPhysicalShape
   ↓
PerformBroadcast     ← Flatten | PrefixLoop | None
   ↓
backend (Bertha: CoreToBertha → HiToLo)

PerformBroadcast (--legato-perform-broadcast) does not look at a compute model. It reads shapes and layouts only. The plan is DecompositionKind in legato/csrc/Includes/Dialects/Core/Compute/ShapeDecomposition.hpp:

Kind Meaning
None Already a leaf (rank at or below the leaf rank), or the op cannot be split.
Flatten Reinterpret a packed prefix as one rank-1 vector (or one rank-2 matmul). One invocation.
PrefixLoop hscf.for over the leading dims; each iteration runs a suffix leaf. Later hscf → scf.for.

Which analyzer runs:

Analyzer Ops Leaf
analyzeElementwiseDecomposition +, *, cast, exp, … rank-1
analyzeLastAxisRowDecomposition last-axis softmax, RMSNorm, LayerNorm, keepdim reduce_sum rank-1 row
analyzeSpatialBatchDecomposition conv, pool, dwconv [X, C] or [H, W, C]
analyzeMatmulBatchDecomposition matmul / grouped matmul rank-2 matrix

A dynamic or padded prefix must PrefixLoop. Flatten is allowed only when every merged dimension is statically packed (logical == physical and logical >= 0). A rank-1 elementwise vector also requires a packed last dim. A dynamic extent is not merged, because its runtime size is not known to match the other operand.

Affine operands (RMSNorm / LayerNorm weight and bias, softmax max) are sliced per row. Spatial weights and bias have no batch prefix: the same SSA value is forwarded to every leaf.

Worked shapes

Snippets below are the math only. Full request_load kernels are on Broadcasting on Bertha and Examples: Broadcasting.

Leading-1 add — [1, 256] + [256, 256]

Right-align: last dim 256 == 256, first dim 1 stretches to 256. Result [256, 256]. Rank 2 is above the elementwise leaf, so PerformBroadcast emits a prefix loop (or flatten, if every dim is packed) of rank-1 adds.

1
y = a + b  # a is [1, 256], b is [256, 256]

Packed same-shape add vs padded last dim

[4, 128] + [4, 128] with logical == physical on every dim can Flatten into one rank-1 add of length 512.

(4, (128, 256)) + (4, (128, 256)) cannot. The last dim has holes, so the plan is PrefixLoop: four rank-1 adds of logical length 128. The holes stay out of the vector.

Spatial N* — conv, dwconv, pool

Channels-last. The hardware leaf is the unbatched suffix:

  • 1D: [X, C] (conv / dwconv / pool)
  • 2D: [H, W, C]

Zero or more leading dims are N*. [8, 64] conv is already a leaf (None). [2, 8, 64] is PrefixLoop with leafRank=2 and two invocations. Weights stay [KX, Cin, Cout] or [KX, C]; they are not sliced with the batch.

1
2
3
4
5
6
7
8
9
y = legato.math.conv_1d(x, w, None, (1,), (1,), (1, 1), "zeros")
# x: [8, 64]        → one leaf
# x: [2, 8, 64]     → loop of two [8, 64] leaves

y = legato.math.dwconv_1d(x, w)
# x: [2, 19, 64], w: [4, 64] → two [19, 64] leaves, same w

y = legato.math.avg_pool_1d(x, (2,), (2,))
# x: [8, 64]        → one leaf

Last-axis rows — LayerNorm, RMSNorm, softmax

These ops reduce or normalize one axis. On the last axis, the leaf is rank-1 (axis rewritten to 0). [4, 128] LayerNorm is four [128] leaves. Affine weight / bias are rank-1 [D] and are applied per row.

1
2
3
y = legato.math.layer_norm(x, w, b)   # x: [4, 128]
y = legato.math.rms_norm(x, w)        # x: (4, (70, 256)) — divisor is 70, not 256
y = legato.math.softmax(x, -1, row_max)

The RMSNorm / LayerNorm divisor is the logical length. Padding [70, 256) is identity zeros, not extra channels.

Matmul batch prefix

[B, M, K] @ [K, N] (RHS has unit or missing batch) shares one weight matrix across B left-hand tiles. [B, M, K] @ [B, K, N] walks both prefixes. The leaf is a rank-2 matmul.

1
y = lhs @ rhs  # lhs: [2, 32, 64], rhs: [1, 64, 64] → shared RHS

Whether a backend may replay that shared RHS from a streamed buffer is a target rule, not a Core rule. Bertha's rule is on Broadcasting on Bertha.