Skip to content

Collective Operations

A collective coordinates data exchange among the ranks of a DeviceMesh. Create a CollectiveGroup outside the device contexts, then call the same operation in the same order from each participating rank's core. The six operations use channels internally; the kernel names the group, not individual channels.

The current implementation uses direct exchange. For example, a four-rank all_reduce sends each full input to three peers, making twelve transfers. This describes the communication pattern, not a synchronization barrier or a bandwidth guarantee.

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import legato
from legato.model.bertha import BerthaDeviceMesh


def two_rank_mesh(ctx):
    """Two ranks, one compute core and one memory channel each."""
    return BerthaDeviceMesh(ctx, "coll2", 2, num_cores=1, num_memory_channels=1)


@legato.compile(backend=two_rank_mesh, param_kinds={"value": "universal", "out": "universal"})
def all_reduce_two(
    value: legato.get_model().tensor_type(legato.types.float("bfloat16"), (1, 256), "sliced_dram"),
    out: legato.get_model().tensor_type(legato.types.float("bfloat16"), (1, 256), "sliced_dram"),
):
    """Both ranks end up holding the sum of both rows."""
    mesh = legato.get_context().get_mesh()
    group = legato.collective.CollectiveGroup(mesh, 0)
    row = legato.get_model().tensor_type(legato.types.float("bfloat16"), (1, 256), "row_major")
    with mesh.get_device(0).get_core(0):
        x = legato.tensor.allocate(row, None)
        legato.tensor.memcpy(x, value)
        y = legato.collective.all_reduce(group, x)
        legato.tensor.memcpy(out, y)
    with mesh.get_device(1).get_core(0):
        x = legato.tensor.allocate(row, None)
        legato.tensor.memcpy(x, value)
        y = legato.collective.all_reduce(group, x)
        legato.tensor.memcpy(out, y)

The input and output are rank-local DRAM tensors. memcpy loads the input into SRAM and stores the reduced value directly into the output. A fused kernel can instead pass its computed SRAM value to the collective and consume the result in its next operation; see tp_matmul_all_reduce_two below.

The group is built outside both with blocks so the separately traced rank programs share its channels. Write each rank's core block explicitly: legato.static.for_each repeats operations within a block, not across ranks.

Constraints

Rule Why What you get if you break it
Every rank makes the same calls in the same order The k-th collective on every participant has the same operation and root Mismatched calls are rejected during tracing; missing calls report every message needs both
Every rank passes the same element type and shape One message has one payload type passes bf16[2, 256], but another rank passed bf16[1, 256]
A value you sent may be read afterwards but not overwritten The compiler conservatively rejects modifying source storage after its first send DIC source storage is modified after first send
The last dimension must fill whole SRAM words The transfer unit is a 128 B word, so bf16 needs a multiple of 64 is not a multiple of the 128 B SRAM word
The call cannot sit in a runtime loop or branch Each message reserves its storage and address slot at compile time a help pointing at legato.static.for_each
One call per core, from that core's own block The transport lowers channel operations in single-core contexts call it from each core's own get_core block

A group spanning several cores is that many independent collectives: lanes pair equal core ids only, so core 1 of rank 0 talks to core 1 of rank 1 and never to core 0.

Payload pieces

Legato has no concat, and a view cannot be sent. So all_gather and all_to_all return one value per rank rather than a joined tensor, and reduce_scatter and all_to_all take one piece per rank as separate arguments. Joining or splitting is the caller's, through legato.tensor.view and memcpy, which are allowed as a copy's source and destination. The all_to_all_two kernel shows these copies explicitly: input row j is copied into the piece sent to rank j, and the received piece from rank r is copied into output row r.

Group

CollectiveGroup(mesh) and CollectiveGroup(mesh, None) select every active core in the mesh. None means no restriction on the mesh's active cores; the selection therefore follows the mesh configuration. Pass a core ID or a list or tuple of distinct IDs to fix the participating cores:

1
2
3
all_cores = legato.collective.CollectiveGroup(mesh)
core_zero = legato.collective.CollectiveGroup(mesh, 0)
selected_cores = legato.collective.CollectiveGroup(mesh, [0, 2])

Core selection is resolved at compile time. Empty selections, duplicates, and inactive core IDs are rejected. Each selected core communicates with the same core ID on other ranks; a collective must still be called from each selected core's own block. Selecting all cores does not replicate the call automatically or reduce values across different core IDs.

CollectiveGroup

1
CollectiveGroup(mesh, cores = None) -> Callable[[CodeGenerator], Value | None]

The participants every collective in this kernel refers to. Call it outside every device context.

Parameters:

  • mesh: The ComputeModelMesh this kernel compiles for.
  • cores (int | list[int] | tuple[int, ...] | None): A core ID or a nonempty collection of distinct core IDs. Omitted or None selects every active core in the mesh. Call each collective from each selected core's own get_core(c) block. Default: None.

Rooted operations

reduce

1
reduce(group: Value, value: Value, root: Value | int = 0) -> Callable[[CodeGenerator], Value | None]

Sum value over every rank onto root.

Parameters:

  • group (Value): A collective group created by legato.collective.CollectiveGroup.
  • value (Value): This rank's contribution, with the same element type and shape on every rank.
  • root (Value | int): The rank that receives the sum. Defaults to rank 0. Default: 0.

Returns:

  • (Callable[[CodeGenerator], Value | None]): The sum on the root and the original value on other ranks.

broadcast

1
broadcast(group: Value, value: Value, root: Value | int = 0) -> Callable[[CodeGenerator], Value | None]

Give every rank the root's value.

Parameters:

  • group (Value): A collective group created by legato.collective.CollectiveGroup.
  • value (Value): The root's input. Other ranks pass a value with the same element type and shape; its contents are ignored.
  • root (Value | int): The rank supplying the value. Defaults to rank 0. Default: 0.

Returns:

  • (Callable[[CodeGenerator], Value | None]): The root's value on every rank.

Symmetric operations

all_reduce

1
all_reduce(group: Value, value: Value) -> Callable[[CodeGenerator], Value | None]

Sum value over every rank.

Parameters:

  • group (Value): A collective group created by legato.collective.CollectiveGroup.
  • value (Value): This rank's contribution, with the same element type and shape on every rank.

Returns:

  • (Callable[[CodeGenerator], Value | None]): The sum on every rank, accumulated in source-rank order.

all_gather

1
all_gather(group: Value, value: Value) -> Callable[[CodeGenerator], Value | None]

Gather every rank's value on every rank.

Parameters:

  • group (Value): A collective group created by legato.collective.CollectiveGroup.
  • value (Value): This rank's contribution, with the same element type and shape on every rank.

Returns:

  • (Callable[[CodeGenerator], Value | None]): A tuple of values in source-rank order, not a concatenated tensor.

reduce_scatter

1
reduce_scatter(group: Value, *pieces: Value) -> Callable[[CodeGenerator], Value | None]

Sum piece r over every rank onto rank r.

Parameters:

  • group (Value): A collective group created by legato.collective.CollectiveGroup.
  • *pieces (Value): This rank's contributions, one per destination rank in rank order. All pieces on all ranks must have the same element type and shape. Default: ().

Returns:

  • (Callable[[CodeGenerator], Value | None]): This rank's reduced piece, accumulated in source-rank order.

all_to_all

1
all_to_all(group: Value, *pieces: Value) -> Callable[[CodeGenerator], Value | None]

Send piece j to rank j and receive one piece from every rank.

Parameters:

  • group (Value): A collective group created by legato.collective.CollectiveGroup.
  • *pieces (Value): This rank's inputs, one per destination rank in rank order. All pieces on all ranks must have the same element type and shape. Default: ().

Returns:

  • (Callable[[CodeGenerator], Value | None]): A tuple of received pieces in source-rank order, including this rank's own piece.

Numerics

Reductions add contributions in source-rank order. For all_reduce, this gives every rank the same addition sequence. This matters because bf16 addition is not associative: with inputs (256, 1, 1, -256), a chain starting from its own contribution can give rank 3 a different answer from ranks 0 to 2.

Tensor-parallel example

multi_device_test/dic_collective.py::tp_matmul_all_reduce_two shows how to pass computed SRAM values directly to a collective. Each rank computes a hidden shard with X @ W_column[r], then a [10,128] partial with hidden[r] @ W_row[r]. all_reduce sums the two partials and returns the same output on both ranks.

tp_matmul_reduce_scatter_all_gather_two uses the same matmuls, then splits each partial into two [5,128] pieces. reduce_scatter sums corresponding pieces, leaving one per rank; all_gather returns both pieces to each rank.

Pass a peer-window backend explicitly with legato.session(transport_backend=backend). MeshLauncher enqueues every rank before waiting for runtime completion; it adds no stagger and returns after completion and cleanup. It does not establish grant ordering or extend the hardware's remote-read timeout.

The dense TP hardware test binds rank-local tensors with RankArgs, maps ranks to device ordinals with MeshPlacement, and compares the output with PyTorch. Run it on two allocated devices from legato/:

1
2
3
4
HA_BACKEND=hw HA_TOOLCHAIN=1 \
  ../.venv/bin/python -m pytest \
  src/tests/multi_device_test/test_collective_hardware.py \
  -k test_dense_tp_reduction_variants_match

A passing case establishes that kernel and launch configuration, not arbitrary launch skew or every collective.