3. Collective operations¶
A collective operation is an operation in which every rank of a communicator takes part. This chapter covers what each operation does and the rules that apply to all operations.
Exact signatures are in the Collective Operations reference, and the implemented combinations are in the Support matrix.
3.1 List of operations¶
| Operation | Provided by each rank | Where the result lands | Reduction | LPU execution |
|---|---|---|---|---|
LcclAllReduce |
count elements |
All ranks | Yes | ✅ |
LcclReduce |
count elements |
Root only | Yes | ✅ |
LcclReduceScatter |
recv_count * nranks elements |
A different piece on each rank | Yes | ✅ |
LcclBroadcast |
Provided by the root | All ranks | No | ✅ |
LcclAllGather |
send_count elements |
All ranks, concatenated in rank order | No | ✅ |
LcclGather |
count elements |
Root, concatenated in rank order | No | ✅ |
LcclScatter |
Provided by the root | A different piece on each rank | No | ✅ |
LcclAllToAll |
A piece for each peer | The piece received from each peer | No | ✅ |
Every operation runs as a device kernel on the LPU. The LPU's limits on reduction types and sizes are in 3.3 and the Support matrix.
Buffer sizes¶
The most common cause of memory corruption in collective operations is a receive buffer that is too small. The rules differ from operation to operation.
| Operation | Send buffer | Receive buffer |
|---|---|---|
LcclAllReduce |
count |
count |
LcclReduce |
count |
count, root only |
LcclReduceScatter |
recv_count * nranks |
recv_count |
LcclBroadcast |
count, root only |
count |
LcclAllGather |
send_count |
send_count * nranks |
LcclGather |
count |
count * nranks, root only |
LcclScatter |
count * nranks, root only |
count |
LcclAllToAll |
count * nranks |
count * nranks |
This example calls LcclAllGather from Python, where the receive buffer is nranks times the send buffer.
1 2 3 4 5 | |
Where the data lands¶
All offsets are in units of elements. i and j are rank numbers.
| Operation | Location |
|---|---|
LcclAllGather |
The data of rank i lands at recv + i*send_count on every rank |
LcclReduceScatter |
The result of reducing the send buffers of all ranks element by element is divided into pieces of recv_count elements, and rank i receives the i-th block |
LcclAllToAll |
The piece to send to rank j is taken from send + j*count, and the piece that comes from rank i lands at recv + i*count |
LcclGather |
The data of rank i lands at recv + i*count on the root |
LcclScatter |
The piece at send + i*count on the root goes to recv on rank i |
For example, in LcclAllToAll with 3 ranks and count = 1, if send is [a0, a1, a2] on rank 0, [b0, b1, b2] on rank 1, and [c0, c1, c2] on rank 2, then recv is [a0, b0, c0] on rank 0, [a1, b1, c1] on rank 1, and [a2, b2, c2] on rank 2.
Operations that have a root must also be called by every rank
LcclReduce, LcclBroadcast, LcclGather, and LcclScatter center on one rank (the root), but every rank of the communicator must call them. If any rank skips the call, the others wait.
A buffer marked "root only" in the tables above may be null on ranks other than the root. If any other buffer is null, the result is kLcclInvalidArgument.
3.2 Rules that apply to all collective operations¶
All ranks call in the same order. Collective operations are matched by order. If the order differs between ranks, the operations are mismatched, and the result is a hang or wrong data rather than an error. The same applies when the size or data type differs from rank to rank. A hang does not go on forever. When a kernel exceeds LCCL_KERNEL_TIMEOUT_MS (default 5 seconds), the watchdog aborts the kernel and the call returns kLcclKernelSpinTimeout. The communicator is then unusable, so destroy it and create it again.
count is the number of elements, not bytes, and cannot be 0. The library converts it using the data type. count == 0 is not a no-op as it is in NCCL; it returns kLcclInvalidArgument.
Buffers are device pointers. Allocate them with hart. lccl does not allocate them and does not check whether a pointer is large enough. In LPU execution, LCCL assumes that user buffers are allocations interleaved across the DRAM channels, so allocate them with hart.mem_alloc(nbytes, is_interleaved=True). The lpu tensors that torch.distributed passes are always allocated that way.
On the LPU, sizes are subject to a 512-byte unit rule. Depending on the operation, a size that does not fit this unit is rejected. LcclAllToAll and LcclReduceScatter reject it with kLcclInvalidArgument, and LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv reject it with kLcclNotSupported, all at call time, and the communicator is left as it was. LcclAllReduce and LcclAllGather accept any count. The table is in the Support matrix.
In-place calls differ from operation to operation. LcclAllReduce and LcclBroadcast accept the same pointer for the send and receive buffers. LcclAllToAll does not support in-place calls, so the send and receive buffers must not overlap. If they overlap, the result is kLcclInvalidArgument. LcclAllGather rejects an in-place call whose count is not aligned to the 512-byte boundary; an in-place call whose count is aligned has not been verified. The remaining operations have not been verified either.
A communicator with 1 rank only copies and is done. When nranks == 1, the send buffer is copied to the receive buffer and the call returns, and no transfer takes place. Because no kernel is involved, any data type and reduction (including Max and Min) passes. This makes single-device execution cheap, but it also means that a single-rank test verifies neither the transfer path nor the supported range at all.
3.3 Data types and reductions¶
| Values | |
|---|---|
| Data types | Int8/Char, Uint8, Int32/Int, Uint32, Int64, Uint64, Float16/Half, Float32/Float, Float64/Double, Bfloat16 |
| Reductions | Sum, Prod, Max, Min, Avg |
The enumerations contain NCCL's main data types and reductions. What the enumerations accept and what has a kernel behind it are different things. The combinations that have kernels are as follows.
| LPU execution | |
|---|---|
| Reductions | Sum, Prod, Avg (Prod and Avg for LcclAllReduce and LcclReduce only) |
| Data types for reductions | Float16, Bfloat16 |
| Operations without a reduction | All data types |
Collective operations without a reduction move data byte by byte regardless of the data type, so they accept every data type.
Unsupported combinations are rejected
- A (operation, reduction, data type) combination with no registered kernel fails with
kLcclNotSupportedat the call (atLcclGroupEnd()inside a group).MaxandMinare typical examples. Nothing is transferred, and nothing silently goes wrong either. - On the LPU, reductions on types other than
Float16andBfloat16(includingFloat32) are also rejected withkLcclNotSupportedat the call. - With 1 rank, any combination passes (the copy-only case above).
The full table is in the Support matrix.
3.4 Streams and completion timing¶
Most collective operations take an HaStream_t, and the default is the calling thread's default stream. In the current implementation, however, a collective operation called outside a group has already finished when the call returns. Asynchronous collective operations (the style that enqueues onto the stream and returns immediately) are not supported yet. An operation called inside a group runs at the outermost LcclGroupEnd(), so read its results only after that execution has finished, and do not overwrite or free its buffers before then. See Groups.
1 2 | |
The reverse direction is not guaranteed. The stream argument does not order the collective operation with other work, so if a kernel or copy that writes the send buffer is still running, LCCL can read data that has not been fully written. On the LPU, the operation starts only after the work already queued on the calling thread's default stream, and that wait counts toward LCCL_KERNEL_TIMEOUT_MS, so synchronize that stream before the call as well.
Synchronize the producer before the call
Check that the work that produced the send buffer has finished before you call. Calling on the same stream is not enough by itself.
1 2 | |
The lccl backend of torch.distributed synchronizes the calling thread's default stream on your behalf before the call. This is a temporary implementation, so work enqueued from other threads or on other streams is not included.
LcclAllGather, LcclAllToAll, LcclSend, and LcclRecv have no stream argument
These four functions do not take a stream argument and use the calling thread's default stream. In Python, only all_reduce accepts a stream keyword (default 0).
Stream semantics (including the fact that the default stream is per thread) come from the runtime. See Streams and events.
3.5 Choosing an operation¶
Tensor-parallel inference uses only a few of these patterns.
| Need | Operation |
|---|---|
| Combine the per-device partial outputs, and everyone needs the result | LcclAllReduce |
| Concatenate the per-device pieces so that everyone has the full tensor | LcclAllGather |
| Combine and distribute in one step (sequence parallelism) | LcclReduceScatter |
| Distribute a prompt or settings from one rank | LcclBroadcast |
| Collect the result on one rank for output | LcclGather (LcclReduce if a reduction is needed) |
The lccl backend of torch.distributed blocks gather, scatter, reduce, and all_to_all on the LPU. On that path, write reduce as all_reduce, gather as all_gather, and scatter as broadcast or send/recv. To write all_to_all with send/recv, see Troubleshooting.
Next steps¶
- Groups — bundling several operations into a single execution
- Point-to-point — directed transfers
- Support matrix