4. Groups¶
A group collects several operations and executes them when the region closes. This chapter covers when a group is needed and what a region guarantees.
4.1 What a group does¶
Operations called between LcclGroupStart() and LcclGroupEnd() are collected and then executed together when the region closes. LcclGroupEnd() executes the collected operations, waits until they finish, and then returns.
1 2 3 4 | |
There are two reasons to use a group.
- Fewer executions (point-to-point). On the LPU,
LcclSend/LcclRecvcalls inside a group are combined into fewer device kernel runs. Collective operations, by contrast, run one at a time even inside a group, so grouping them does not reduce the number of runs. - Deadlock avoidance. A send and a receive must both be registered before either side can proceed. A group is what lets a rank register both without blocking on the first one. See Point-to-point.
4.2 What a group does not do¶
A group is not a reordering barrier
A group means "these operations are independent of each other", not "run them in this order". Collective operations are sorted by transfer size and scheduled ahead of point-to-point operations, so they usually do not execute in the order they were called. Buffers must therefore not overlap between operations in a group. Overlapping buffers only produce undefined results; the overlap is not detected. If operation B needs the result of A, put them in different groups. Consecutive groups are ordered.
The execution order determined for a batch depends on the whole call sequence. Every rank must therefore call the same operations, with the same sizes, in the same order; if they differ between ranks, different orders are produced and the ranks go out of step. Point-to-point is stricter: per peer and direction, LcclSend/LcclRecv keep their call order, because they are paired in call order with the peer rank's calls in the same direction.
A rejected call and a discarded group are different cases
- If an argument check at call time fails (a null buffer,
count == 0, an unknown data type or reduction, arootorpeerout of range, the size rule on the LPU forLcclAllToAllandLcclReduceScatter, or aLcclAllGatherthat is not on a 512-byte boundary inside a group), only that call returns an error. The group is not affected, so operations queued earlier still execute, andLcclGroupEnd()can returnkLcclSuccess. You therefore need to check the return value of every call, even inside a group. If you do not, a rejected operation is silently dropped. - If a failure occurs in the stage where the group prepares for execution (for example
kLcclNotSupportedfor an unsupported combination or for the LPU's reduction-type and size rules onLcclBroadcast,LcclReduce,LcclGather,LcclScatter, andLcclSend/LcclRecv; this error surfaces atLcclGroupEnd()), the error is recorded on the group, andLcclGroupEnd()returns that error, discarding all of the collected operations. Operations requested earlier are not executed either. - A failure in the execution stage (a kernel error, the watchdog) does not undo the results of executions that already finished before it.
LcclGroupEnd()reports a kernel that was aborted or cut off by the watchdog.
4.3 Nesting¶
Groups nest. The library tracks the depth per thread, and the work is executed when the outermost region closes.
1 2 3 4 | |
Because of this, it is safe for a helper function to open its own group even if it does not know whether the caller has already opened one. If LcclAllToAll is called inside the caller's group, it joins that group and is not executed separately.
Every start must have an end
The depth is managed per thread and is not reset when an error occurs. A LcclGroupStart() whose LcclGroupEnd() is skipped by an early return or an exception leaves that thread in a group permanently, and all later operations are queued instead of executed. The symptom is a process that stops making progress without an error.
An unmatched LcclGroupEnd() is reported as kLcclGroupNotInGroup.
In C++, close the group with a scope guard rather than by hand. In Python, a failed call raises an exception, so call group_end() in a finally block.
4.4 Mixing communicators¶
Operations from several communicators can share one group. This is how a rank handles tensor-parallel and pipeline-parallel traffic with a single LcclGroupEnd().
Do not mix blocking and nonblocking communicators in one group
Put only communicators with the same blocking setting in one group. The blocking flag is set through LcclConfig at creation time. See Communicators.
The library rejects the mix with kLcclInvalidArgument only when communicator creation or destruction operations are mixed (with the warning Blocking and nonblocking communicators are not allowed in the same group.). Mixed collective operations are not checked; they follow the mode of the communicator that joined last, so you must avoid the mix yourself.
4.5 Errors from a group¶
A failure caught by the argument check is returned immediately by the call concerned, and a failure in the preparation or execution stage surfaces at LcclGroupEnd(). When diagnosing the latter, the error code tells you what the problem is but not which operation caused it.
If you need to attribute a failure to a specific call, call the operations without a group while debugging, accept the execution overhead, and restore the group after you are done.
On a nonblocking communicator, LcclGroupEnd() can return early with kLcclInProgress without waiting. This happens when the group still contains a communicator that is waiting for connections or an asynchronous job. Check completion with LcclCommGetAsyncError().
Next steps¶
- Point-to-point — Where a group is needed
- Error handling