5. Point-to-point¶
LcclSend and LcclRecv move data between two designated ranks. This chapter covers how to register both sides without a deadlock.
5.1 Operations¶
1 2 | |
peer is the other rank within the same communicator. Both sides must agree on count and datatype.
Unlike a collective operation, only the two designated ranks take part. The rest of the communicator calls nothing.
Neither function takes a stream argument. A LcclSend or LcclRecv called outside a group executes on the spot and returns only after it finishes. The deadlock in the next section comes from this.
5.2 Registering both sides¶
A send waits for the corresponding receive to be registered. If both ranks call send first and block, neither reaches its receive. A transfer smaller than the capacity that can be buffered on the other side can complete without a receive, but do not rely on that behavior.
Unpaired send/receive is a deadlock
1 2 3 | |
Both ranks hang in the send. They do not hang forever. After LCCL_KERNEL_TIMEOUT_MS (default 5 seconds) elapses, the watchdog aborts the kernel and returns kLcclKernelSpinTimeout. That communicator cannot be used afterwards, so destroy it and create it again.
The solution is a group. Inside the region, both operations are registered before either side needs to proceed.
1 2 3 4 | |
This is the standard pattern for an exchange, and it is the reason groups exist. In torch.distributed, dist.batch_isend_irecv plays this role.
5.3 Pipeline parallelism¶
The most common use is passing activations between pipeline stages. Each rank receives from the previous stage and sends to the next stage.
1 2 3 4 5 6 7 8 | |
The conditionals keep the ranks at both ends from naming a rank that does not exist. This does not violate the same-order rule for collective operations, because point-to-point operations are paired by peer, not by an order spanning the whole communicator. Sends with the same peer and in the same direction (and likewise receives) are paired in call order, so make the calls in the same order on both sides.
5.4 Relation to all-to-all¶
LcclAllToAll produces the same result as exchanging one send and one receive with each peer. Its transfer characteristics and hang behavior are the same as point-to-point, and on the LPU the byte count of each per-peer piece must be a multiple of 512, as with point-to-point. If this is violated, LcclAllToAll rejects the call with kLcclInvalidArgument at call time.
In-place calls are not supported, so the send and receive buffers must not overlap. Overlapping buffers produce kLcclInvalidArgument.
5.5 Sending to yourself¶
Do not send to or receive from your own rank. An unpaired self send/receive is reported as kLcclEnqueueSelfP2PUnmatched, and pairing the two in one group is not supported either. Copy your own share of the data directly.
5.6 Limitations¶
| Data type | All. Transfers are in bytes |
| Reduction | Not applicable |
| Size | The byte count must be a multiple of 512. If violated, the call is rejected with kLcclNotSupported at the point of the call (at LcclGroupEnd() inside a group). Support matrix |
| Gap between matching calls | If the peer rank makes its matching call later than LCCL_KERNEL_TIMEOUT_MS (5 seconds by default), the call fails with kLcclKernelSpinTimeout and the communicator is aborted |
| Transport | Shared memory within the same server (P2P between ranks in the same process), network between servers |
| Order relative to collective operations | Outside a group, call order is followed; inside the same group, collective operations execute first, then point-to-point operations |
Next steps¶
- Groups
- Troubleshooting — Diagnosing hangs