Collective Operations¶
A collective coordinates data exchange among the ranks of a DeviceMesh.
Create a CollectiveGroup outside the device contexts, then call the same
operation in the same order from each participating rank's core. The six
operations use channels internally; the kernel names
the group, not individual channels.
The current implementation uses direct exchange. For example, a four-rank
all_reduce sends each full input to three peers, making twelve transfers.
This describes the communication pattern, not a synchronization barrier or a
bandwidth guarantee.
Example¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | |
The input and output are rank-local DRAM tensors. memcpy loads the input into
SRAM and stores the reduced value directly into the output. A fused kernel can
instead pass its computed SRAM value to the collective and consume the result
in its next operation; see tp_matmul_all_reduce_two below.
The group is built outside both with blocks so the separately traced rank
programs share its channels. Write each rank's core block explicitly:
legato.static.for_each repeats operations within a block, not across ranks.
Constraints¶
| Rule | Why | What you get if you break it |
|---|---|---|
| Every rank makes the same calls in the same order | The k-th collective on every participant has the same operation and root | Mismatched calls are rejected during tracing; missing calls report every message needs both |
| Every rank passes the same element type and shape | One message has one payload type | passes bf16[2, 256], but another rank passed bf16[1, 256] |
| A value you sent may be read afterwards but not overwritten | The compiler conservatively rejects modifying source storage after its first send | DIC source storage is modified after first send |
| The last dimension must fill whole SRAM words | The transfer unit is a 128 B word, so bf16 needs a multiple of 64 | is not a multiple of the 128 B SRAM word |
| The call cannot sit in a runtime loop or branch | Each message reserves its storage and address slot at compile time | a help pointing at legato.static.for_each |
| One call per core, from that core's own block | The transport lowers channel operations in single-core contexts | call it from each core's own get_core block |
A group spanning several cores is that many independent collectives: lanes pair equal core ids only, so core 1 of rank 0 talks to core 1 of rank 1 and never to core 0.
Payload pieces¶
Legato has no concat, and a view cannot be sent. So all_gather and
all_to_all return one value per rank rather than a joined tensor, and
reduce_scatter and all_to_all take one piece per rank as separate
arguments. Joining or splitting is the caller's, through legato.tensor.view
and memcpy, which are allowed as a copy's source and destination. The
all_to_all_two kernel shows these copies explicitly: input row j is copied
into the piece sent to rank j, and the received piece from rank r is copied
into output row r.
Group¶
CollectiveGroup(mesh) and CollectiveGroup(mesh, None) select every active
core in the mesh. None means no restriction on the mesh's active cores; the
selection therefore follows the mesh configuration. Pass a core ID or a list
or tuple of distinct IDs to fix the participating cores:
1 2 3 | |
Core selection is resolved at compile time. Empty selections, duplicates, and inactive core IDs are rejected. Each selected core communicates with the same core ID on other ranks; a collective must still be called from each selected core's own block. Selecting all cores does not replicate the call automatically or reduce values across different core IDs.
CollectiveGroup¶
1 | |
The participants every collective in this kernel refers to. Call it outside every device context.
Parameters:
mesh: TheComputeModelMeshthis kernel compiles for.cores(int | list[int] | tuple[int, ...] | None): A core ID or a nonempty collection of distinct core IDs. Omitted orNoneselects every active core in the mesh. Call each collective from each selected core's ownget_core(c)block. Default:None.
Rooted operations¶
reduce¶
1 | |
Sum value over every rank onto root.
Parameters:
group(Value): A collective group created bylegato.collective.CollectiveGroup.value(Value): This rank's contribution, with the same element type and shape on every rank.root(Value | int): The rank that receives the sum. Defaults to rank 0. Default:0.
Returns:
- (
Callable[[CodeGenerator], Value | None]): The sum on the root and the originalvalueon other ranks.
broadcast¶
1 | |
Give every rank the root's value.
Parameters:
group(Value): A collective group created bylegato.collective.CollectiveGroup.value(Value): The root's input. Other ranks pass a value with the same element type and shape; its contents are ignored.root(Value | int): The rank supplying the value. Defaults to rank 0. Default:0.
Returns:
- (
Callable[[CodeGenerator], Value | None]): The root's value on every rank.
Symmetric operations¶
all_reduce¶
1 | |
Sum value over every rank.
Parameters:
group(Value): A collective group created bylegato.collective.CollectiveGroup.value(Value): This rank's contribution, with the same element type and shape on every rank.
Returns:
- (
Callable[[CodeGenerator], Value | None]): The sum on every rank, accumulated in source-rank order.
all_gather¶
1 | |
Gather every rank's value on every rank.
Parameters:
group(Value): A collective group created bylegato.collective.CollectiveGroup.value(Value): This rank's contribution, with the same element type and shape on every rank.
Returns:
- (
Callable[[CodeGenerator], Value | None]): A tuple of values in source-rank order, not a concatenated tensor.
reduce_scatter¶
1 | |
Sum piece r over every rank onto rank r.
Parameters:
group(Value): A collective group created bylegato.collective.CollectiveGroup.*pieces(Value): This rank's contributions, one per destination rank in rank order. All pieces on all ranks must have the same element type and shape. Default:().
Returns:
- (
Callable[[CodeGenerator], Value | None]): This rank's reduced piece, accumulated in source-rank order.
all_to_all¶
1 | |
Send piece j to rank j and receive one piece from every rank.
Parameters:
group(Value): A collective group created bylegato.collective.CollectiveGroup.*pieces(Value): This rank's inputs, one per destination rank in rank order. All pieces on all ranks must have the same element type and shape. Default:().
Returns:
- (
Callable[[CodeGenerator], Value | None]): A tuple of received pieces in source-rank order, including this rank's own piece.
Numerics¶
Reductions add contributions in source-rank order. For all_reduce, this
gives every rank the same addition sequence. This matters because bf16
addition is not associative: with inputs (256, 1, 1, -256), a chain starting
from its own contribution can give rank 3 a different answer from ranks 0 to 2.
Tensor-parallel example¶
multi_device_test/dic_collective.py::tp_matmul_all_reduce_two shows how to
pass computed SRAM values directly to a collective. Each rank computes a
hidden shard with X @ W_column[r], then a [10,128] partial with
hidden[r] @ W_row[r]. all_reduce sums the two partials and returns the same
output on both ranks.
tp_matmul_reduce_scatter_all_gather_two uses the same matmuls, then splits
each partial into two [5,128] pieces. reduce_scatter sums corresponding
pieces, leaving one per rank; all_gather returns both pieces to each rank.
Pass a peer-window backend explicitly with legato.session(transport_backend=backend).
MeshLauncher enqueues every rank before waiting for runtime completion; it adds
no stagger and returns after completion and cleanup. It does not establish grant
ordering or extend the hardware's remote-read timeout.
The dense TP hardware test binds rank-local tensors with RankArgs, maps ranks
to device ordinals with MeshPlacement, and compares the output with PyTorch.
Run it on two allocated devices from legato/:
1 2 3 4 | |
A passing case establishes that kernel and launch configuration, not arbitrary launch skew or every collective.