Skip to content

3. Collective operations

A collective operation is an operation in which every rank of a communicator takes part. This chapter covers what each operation does and the rules that apply to all operations.

Exact signatures are in the Collective Operations reference, and the implemented combinations are in the Support matrix.


3.1 List of operations

Operation Provided by each rank Where the result lands Reduction LPU execution
LcclAllReduce count elements All ranks Yes ✅
LcclReduce count elements Root only Yes ✅
LcclReduceScatter recv_count * nranks elements A different piece on each rank Yes ✅
LcclBroadcast Provided by the root All ranks No ✅
LcclAllGather send_count elements All ranks, concatenated in rank order No ✅
LcclGather count elements Root, concatenated in rank order No ✅
LcclScatter Provided by the root A different piece on each rank No ✅
LcclAllToAll A piece for each peer The piece received from each peer No ✅

Every operation runs as a device kernel on the LPU. The LPU's limits on reduction types and sizes are in 3.3 and the Support matrix.

Buffer sizes

The most common cause of memory corruption in collective operations is a receive buffer that is too small. The rules differ from operation to operation.

Operation Send buffer Receive buffer
LcclAllReduce count count
LcclReduce count count, root only
LcclReduceScatter recv_count * nranks recv_count
LcclBroadcast count, root only count
LcclAllGather send_count send_count * nranks
LcclGather count count * nranks, root only
LcclScatter count * nranks, root only count
LcclAllToAll count * nranks count * nranks

This example calls LcclAllGather from Python, where the receive buffer is nranks times the send buffer.

1
2
3
4
5
count = 1024                                   # elements per rank (Float16)
nbytes = count * 2                             # Float16 is 2 bytes per element
send = hart.mem_alloc(nbytes, is_interleaved=True)
recv = hart.mem_alloc(nbytes * comm.n_ranks, is_interleaved=True)   # the receive buffer is nranks times as large
lccl.all_gather(send, recv, count, lccl.LcclDataType.LcclFloat16, comm)

Where the data lands

All offsets are in units of elements. i and j are rank numbers.

Operation Location
LcclAllGather The data of rank i lands at recv + i*send_count on every rank
LcclReduceScatter The result of reducing the send buffers of all ranks element by element is divided into pieces of recv_count elements, and rank i receives the i-th block
LcclAllToAll The piece to send to rank j is taken from send + j*count, and the piece that comes from rank i lands at recv + i*count
LcclGather The data of rank i lands at recv + i*count on the root
LcclScatter The piece at send + i*count on the root goes to recv on rank i

For example, in LcclAllToAll with 3 ranks and count = 1, if send is [a0, a1, a2] on rank 0, [b0, b1, b2] on rank 1, and [c0, c1, c2] on rank 2, then recv is [a0, b0, c0] on rank 0, [a1, b1, c1] on rank 1, and [a2, b2, c2] on rank 2.

Operations that have a root must also be called by every rank

LcclReduce, LcclBroadcast, LcclGather, and LcclScatter center on one rank (the root), but every rank of the communicator must call them. If any rank skips the call, the others wait.

A buffer marked "root only" in the tables above may be null on ranks other than the root. If any other buffer is null, the result is kLcclInvalidArgument.

3.2 Rules that apply to all collective operations

All ranks call in the same order. Collective operations are matched by order. If the order differs between ranks, the operations are mismatched, and the result is a hang or wrong data rather than an error. The same applies when the size or data type differs from rank to rank. A hang does not go on forever. When a kernel exceeds LCCL_KERNEL_TIMEOUT_MS (default 5 seconds), the watchdog aborts the kernel and the call returns kLcclKernelSpinTimeout. The communicator is then unusable, so destroy it and create it again.

count is the number of elements, not bytes, and cannot be 0. The library converts it using the data type. count == 0 is not a no-op as it is in NCCL; it returns kLcclInvalidArgument.

Buffers are device pointers. Allocate them with hart. lccl does not allocate them and does not check whether a pointer is large enough. In LPU execution, LCCL assumes that user buffers are allocations interleaved across the DRAM channels, so allocate them with hart.mem_alloc(nbytes, is_interleaved=True). The lpu tensors that torch.distributed passes are always allocated that way.

On the LPU, sizes are subject to a 512-byte unit rule. Depending on the operation, a size that does not fit this unit is rejected. LcclAllToAll and LcclReduceScatter reject it with kLcclInvalidArgument, and LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv reject it with kLcclNotSupported, all at call time, and the communicator is left as it was. LcclAllReduce and LcclAllGather accept any count. The table is in the Support matrix.

In-place calls differ from operation to operation. LcclAllReduce and LcclBroadcast accept the same pointer for the send and receive buffers. LcclAllToAll does not support in-place calls, so the send and receive buffers must not overlap. If they overlap, the result is kLcclInvalidArgument. LcclAllGather rejects an in-place call whose count is not aligned to the 512-byte boundary; an in-place call whose count is aligned has not been verified. The remaining operations have not been verified either.

A communicator with 1 rank only copies and is done. When nranks == 1, the send buffer is copied to the receive buffer and the call returns, and no transfer takes place. Because no kernel is involved, any data type and reduction (including Max and Min) passes. This makes single-device execution cheap, but it also means that a single-rank test verifies neither the transfer path nor the supported range at all.

3.3 Data types and reductions

Values
Data types Int8/Char, Uint8, Int32/Int, Uint32, Int64, Uint64, Float16/Half, Float32/Float, Float64/Double, Bfloat16
Reductions Sum, Prod, Max, Min, Avg

The enumerations contain NCCL's main data types and reductions. What the enumerations accept and what has a kernel behind it are different things. The combinations that have kernels are as follows.

LPU execution
Reductions Sum, Prod, Avg (Prod and Avg for LcclAllReduce and LcclReduce only)
Data types for reductions Float16, Bfloat16
Operations without a reduction All data types

Collective operations without a reduction move data byte by byte regardless of the data type, so they accept every data type.

Unsupported combinations are rejected

  • A (operation, reduction, data type) combination with no registered kernel fails with kLcclNotSupported at the call (at LcclGroupEnd() inside a group). Max and Min are typical examples. Nothing is transferred, and nothing silently goes wrong either.
  • On the LPU, reductions on types other than Float16 and Bfloat16 (including Float32) are also rejected with kLcclNotSupported at the call.
  • With 1 rank, any combination passes (the copy-only case above).

The full table is in the Support matrix.

3.4 Streams and completion timing

Most collective operations take an HaStream_t, and the default is the calling thread's default stream. In the current implementation, however, a collective operation called outside a group has already finished when the call returns. Asynchronous collective operations (the style that enqueues onto the stream and returns immediately) are not supported yet. An operation called inside a group runs at the outermost LcclGroupEnd(), so read its results only after that execution has finished, and do not overwrite or free its buffers before then. See Groups.

1
2
LcclAllReduce(send, recv, count, kLcclFloat16, kLcclSum, comm);
// The result is in recv here. No further synchronization is needed.

The reverse direction is not guaranteed. The stream argument does not order the collective operation with other work, so if a kernel or copy that writes the send buffer is still running, LCCL can read data that has not been fully written. On the LPU, the operation starts only after the work already queued on the calling thread's default stream, and that wait counts toward LCCL_KERNEL_TIMEOUT_MS, so synchronize that stream before the call as well.

Synchronize the producer before the call

Check that the work that produced the send buffer has finished before you call. Calling on the same stream is not enough by itself.

1
2
HaStreamSynchronize(stream);   // Drain the stream that wrote the send buffer first
LcclAllReduce(send, recv, count, kLcclFloat16, kLcclSum, comm);

The lccl backend of torch.distributed synchronizes the calling thread's default stream on your behalf before the call. This is a temporary implementation, so work enqueued from other threads or on other streams is not included.

LcclAllGather, LcclAllToAll, LcclSend, and LcclRecv have no stream argument

These four functions do not take a stream argument and use the calling thread's default stream. In Python, only all_reduce accepts a stream keyword (default 0).

Stream semantics (including the fact that the default stream is per thread) come from the runtime. See Streams and events.

3.5 Choosing an operation

Tensor-parallel inference uses only a few of these patterns.

Need Operation
Combine the per-device partial outputs, and everyone needs the result LcclAllReduce
Concatenate the per-device pieces so that everyone has the full tensor LcclAllGather
Combine and distribute in one step (sequence parallelism) LcclReduceScatter
Distribute a prompt or settings from one rank LcclBroadcast
Collect the result on one rank for output LcclGather (LcclReduce if a reduction is needed)

The lccl backend of torch.distributed blocks gather, scatter, reduce, and all_to_all on the LPU. On that path, write reduce as all_reduce, gather as all_gather, and scatter as broadcast or send/recv. To write all_to_all with send/recv, see Troubleshooting.


Next steps