Collective Operations¶
Collective communication across every rank in a communicator. Which operation, reduction, and data type combinations are supported is in the Support Matrix — an unsupported combination is rejected with kLcclNotSupported, not silently ignored.
Operations that take a stream argument, when called outside a group, have also finished when the call returns, but they are not guaranteed to wait for work queued on that stream before the call. Synchronize the work that produced the send buffer first; see Collective operations.
LcclAllGather¶
Each device gathers sendcount values from other devices into recvbuff, receiving data from rank i at offset i*sendcount.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
send_count |
in | Number of elements to all-gather. |
datatype |
in | Data type of the element |
comm |
in | LCCL communicator |
Returns — kLcclSuccess if the data is all gathered successfully, error code otherwise
Any count is legal. A multi-rank count off the 512-byte super-block is gathered at the rounded-up count into a staging buffer on the comm (so sendbuff is read up to that boundary, which an interleaved device allocation always spans) and then compacted into recvbuff, written only up to nranks*sendcount. That compaction follows the collective, so such a count is refused with kLcclInvalidArgument inside a group, on a non-blocking comm, or in place (sendbuff inside recvbuff).
Note
Assumes recvcount is equal to nranks*sendcount, which means that recvbuff should have a size of at least nranks*sendcount elements.
LcclAllReduce¶
Reduces data arrays of length count in sendbuff using op operation, and leaves identical copies of result on each recvbuff.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
count |
in | Number of elements in the sendBuff. |
datatype |
in | Data type of the element |
op |
in | Reduction operation |
comm |
in | LCCL communicator |
stream |
in | HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream. |
Returns — kLcclSuccess if the data is all reduced successfully, error code otherwise
Any count is legal. The kernel moves a count off the 512-byte super-block as whole super-blocks, so it reads sendbuff up to that boundary (an interleaved device allocation always spans it), but writes recvbuff only up to count.
LcclAllToAll¶
Each device sends count values to all other devices and receives count values from all other devices.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
count |
in | Number of elements sent to and received from each rank. |
datatype |
in | Data type of the element |
comm |
in | LCCL communicator |
Returns — kLcclSuccess if the data is all to alled successfully, error code otherwise
Data to send to destination rank j is taken from sendbuff+j*count and data received from source rank i is placed at recvbuff+i*count.
Note
The send and receive buffers each hold nranks*count elements.
Note
In-place operation is not supported: sendbuff and recvbuff must be distinct and must not overlap, because every non-self slice is both read (send to peer) and written (recv from peer) at the same offset with no ordering between the two. Overlapping buffers return kLcclInvalidArgument for any rank count.
LcclBroadcast¶
Copies count values from root to all other devices.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
count |
in | Number of elements in the sendBuff. |
datatype |
in | Data type of the element |
root |
in | The rank where the data resides before the operation is started |
comm |
in | LCCL communicator |
stream |
in | HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream. |
Returns — kLcclSuccess if the data is broadcasted successfully, error code otherwise
Note
root is the rank (not the device) where data resides before the operation is started.
LcclGather¶
Each rank sends count elements from sendbuff to the root rank.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
count |
in | Number of elements in the sendBuff. |
datatype |
in | Data type of the element |
root |
in | Global rank of the root |
comm |
in | LCCL communicator |
stream |
in | HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream. |
Returns — kLcclSuccess if the data is gathered successfully, error code otherwise
On the root rank, data from rank i is placed at recvbuff + i*count. On non-root ranks, recvbuff is not used. root is the rank where data will be gathered.
LcclReduce¶
Reduces data arrays of length count in sendbuff into recvbuff using op operation.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
count |
in | Number of elements in the sendBuff. |
datatype |
in | Data type of the element |
op |
in | Reduction operation |
root |
in | The rank where the data resides after the operation is complete |
comm |
in | LCCL communicator |
stream |
in | HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream. |
Returns — kLcclSuccess if the data is reduced successfully, error code otherwise
Note
recvbuff may be NULL on all calls except for root device. root is the rank (not the device) where data will reside after the operation is complete.
LcclReduceScatter¶
Reduces data in sendbuff using op operation and leaves reduced result scattered over the devices so that recvbuff on rank i will contain the i-th block of the result.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
recv_count |
in | Number of elements to receive. Number of elements in the recvBuff. |
datatype |
in | Data type of the element |
op |
in | Reduction operation |
comm |
in | LCCL communicator |
stream |
in | HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream. |
Returns — kLcclSuccess if the data is reduced scattered successfully, error code otherwise
On the LPU, a multi-rank recv_count off the 512-byte super-block is refused with kLcclInvalidArgument before any launch: each rank's partition would start inside a super-block, which the Legato ring primitives cannot address.
Note
Assumes sendcount is equal to nranks*recvcount, which means that sendbuff should have a size of at least nranks*recvcount elements.
LcclScatter¶
On the root rank, count elements from sendbuff+i*count are sent to rank i.
1 | |
| Parameter | Direction | Description |
|---|---|---|
send_buff |
in | Pointer to the send buffer |
recv_buff |
out | Pointer to the receive buffer |
count |
in | Number of elements in the recvBuff. |
datatype |
in | Data type of the element |
root |
in | Global rank of the root |
comm |
in | LCCL communicator |
stream |
in | HA stream the operation is enqueued on; omit or pass HA_STREAM_DEFAULT to use the calling thread's per-device default stream. |
Returns — kLcclSuccess if the data is scattered successfully, error code otherwise
On non-root ranks, sendbuff is not used. Each rank receives count elements into recvbuff. root is the rank that will distribute the data.
Note
(Number of elements in the sendBuff) = worldSize * count.