Skip to content

Collective Operations

Collective communication across every rank in a communicator. Which operation, reduction, and data type combinations are supported is in the Support Matrix — an unsupported combination is rejected with kLcclNotSupported, not silently ignored.

Operations that take a stream argument, when called outside a group, have also finished when the call returns, but they are not guaranteed to wait for work queued on that stream before the call. Synchronize the work that produced the send buffer first; see Collective operations.


LcclAllGather

Each device gathers sendcount values from other devices into recvbuff, receiving data from rank i at offset i*sendcount.

1
LcclResult LcclAllGather(const void *send_buff, void *recv_buff, size_t send_count, LcclDataType datatype, LcclComm comm);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
send_count in Number of elements to all-gather.
datatype in Data type of the element
comm in LCCL communicator

Returns — kLcclSuccess if the data is all gathered successfully, error code otherwise

Any count is legal. A multi-rank count off the 512-byte super-block is gathered at the rounded-up count into a staging buffer on the comm (so sendbuff is read up to that boundary, which an interleaved device allocation always spans) and then compacted into recvbuff, written only up to nranks*sendcount. That compaction follows the collective, so such a count is refused with kLcclInvalidArgument inside a group, on a non-blocking comm, or in place (sendbuff inside recvbuff).

Note

Assumes recvcount is equal to nranks*sendcount, which means that recvbuff should have a size of at least nranks*sendcount elements.


LcclAllReduce

Reduces data arrays of length count in sendbuff using op operation, and leaves identical copies of result on each recvbuff.

1
LcclResult LcclAllReduce(const void *send_buff, void *recv_buff, size_t count, LcclDataType datatype, LcclRedOp op, LcclComm comm, HaStream_t stream=HA_STREAM_DEFAULT);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
count in Number of elements in the sendBuff.
datatype in Data type of the element
op in Reduction operation
comm in LCCL communicator
stream in HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream.

Returns — kLcclSuccess if the data is all reduced successfully, error code otherwise

Any count is legal. The kernel moves a count off the 512-byte super-block as whole super-blocks, so it reads sendbuff up to that boundary (an interleaved device allocation always spans it), but writes recvbuff only up to count.


LcclAllToAll

Each device sends count values to all other devices and receives count values from all other devices.

1
LcclResult LcclAllToAll(const void *send_buff, void *recv_buff, size_t count, LcclDataType datatype, LcclComm comm);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
count in Number of elements sent to and received from each rank.
datatype in Data type of the element
comm in LCCL communicator

Returns — kLcclSuccess if the data is all to alled successfully, error code otherwise

Data to send to destination rank j is taken from sendbuff+j*count and data received from source rank i is placed at recvbuff+i*count.

Note

The send and receive buffers each hold nranks*count elements.

Note

In-place operation is not supported: sendbuff and recvbuff must be distinct and must not overlap, because every non-self slice is both read (send to peer) and written (recv from peer) at the same offset with no ordering between the two. Overlapping buffers return kLcclInvalidArgument for any rank count.


LcclBroadcast

Copies count values from root to all other devices.

1
LcclResult LcclBroadcast(const void *send_buff, void *recv_buff, size_t count, LcclDataType datatype, int root, LcclComm comm, HaStream_t stream=HA_STREAM_DEFAULT);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
count in Number of elements in the sendBuff.
datatype in Data type of the element
root in The rank where the data resides before the operation is started
comm in LCCL communicator
stream in HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream.

Returns — kLcclSuccess if the data is broadcasted successfully, error code otherwise

Note

root is the rank (not the device) where data resides before the operation is started.


LcclGather

Each rank sends count elements from sendbuff to the root rank.

1
LcclResult LcclGather(const void *send_buff, void *recv_buff, size_t count, LcclDataType datatype, int root, LcclComm comm, HaStream_t stream=HA_STREAM_DEFAULT);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
count in Number of elements in the sendBuff.
datatype in Data type of the element
root in Global rank of the root
comm in LCCL communicator
stream in HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream.

Returns — kLcclSuccess if the data is gathered successfully, error code otherwise

On the root rank, data from rank i is placed at recvbuff + i*count. On non-root ranks, recvbuff is not used. root is the rank where data will be gathered.


LcclReduce

Reduces data arrays of length count in sendbuff into recvbuff using op operation.

1
LcclResult LcclReduce(const void *send_buff, void *recv_buff, size_t count, LcclDataType datatype, LcclRedOp op, int root, LcclComm comm, HaStream_t stream=HA_STREAM_DEFAULT);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
count in Number of elements in the sendBuff.
datatype in Data type of the element
op in Reduction operation
root in The rank where the data resides after the operation is complete
comm in LCCL communicator
stream in HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream.

Returns — kLcclSuccess if the data is reduced successfully, error code otherwise

Note

recvbuff may be NULL on all calls except for root device. root is the rank (not the device) where data will reside after the operation is complete.


LcclReduceScatter

Reduces data in sendbuff using op operation and leaves reduced result scattered over the devices so that recvbuff on rank i will contain the i-th block of the result.

1
LcclResult LcclReduceScatter(const void *send_buff, void *recv_buff, size_t recv_count, LcclDataType datatype, LcclRedOp op, LcclComm comm, HaStream_t stream=HA_STREAM_DEFAULT);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
recv_count in Number of elements to receive. Number of elements in the recvBuff.
datatype in Data type of the element
op in Reduction operation
comm in LCCL communicator
stream in HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream.

Returns — kLcclSuccess if the data is reduced scattered successfully, error code otherwise

On the LPU, a multi-rank recv_count off the 512-byte super-block is refused with kLcclInvalidArgument before any launch: each rank's partition would start inside a super-block, which the Legato ring primitives cannot address.

Note

Assumes sendcount is equal to nranks*recvcount, which means that sendbuff should have a size of at least nranks*recvcount elements.


LcclScatter

On the root rank, count elements from sendbuff+i*count are sent to rank i.

1
LcclResult LcclScatter(const void *send_buff, void *recv_buff, size_t count, LcclDataType datatype, int root, LcclComm comm, HaStream_t stream=HA_STREAM_DEFAULT);
Parameter Direction Description
send_buff in Pointer to the send buffer
recv_buff out Pointer to the receive buffer
count in Number of elements in the recvBuff.
datatype in Data type of the element
root in Global rank of the root
comm in LCCL communicator
stream in HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream.

Returns — kLcclSuccess if the data is scattered successfully, error code otherwise

On non-root ranks, sendbuff is not used. Each rank receives count elements into recvbuff. root is the rank that will distribute the data.

Note

(Number of elements in the sendBuff) = worldSize * count.