6. Error handling¶
Every C++ entry point returns LcclResult. This chapter covers how to classify failures and what state the communicator is in afterwards.
The full list of codes is in Error codes.
6.1 Return convention¶
1 2 3 4 | |
- The return value is always
LcclResult; outputs are passed through pointer parameters. - Exceptions are not propagated. The Python binding raises
RuntimeError, and the message has the form<function name> failed: <message>. An example isLcclAllReduce failed: Operation not supported. The code number is not included, so if you need the code, look it up by message in Error codes. A unique ID whose length is not 128 bytes raisesValueError. LcclGetErrorToString()converts a code into a string for logging.- A failed call usually logs its cause first, on a WARN or ERROR line. Several ERROR lines of the form
<file>:<line> -> <message>can follow; they print the message sentence above, not the code name.kLcclInProgressis not a failure, so it writes nothing.
6.2 Classifying failures¶
LcclResult divides its codes into ranges by subsystem, so a single unfamiliar value is enough to tell at which layer and in what way the failure occurred.
| Range | Subsystem | What went wrong |
|---|---|---|
0x00000 |
General | Arguments, not supported, internal |
0x10000 |
Communicator | Handle, rank range, state, ranks that claimed the same card |
0x20000 |
Socket | Connect, send, receive, timeout |
0x30000 |
Bootstrap | Rendezvous, interface discovery |
0x40000 |
Memory | Host/device allocation, shared memory |
0x50000 |
Topology | Topology file parsing, limits, lookup |
0x60000 |
Net plugin | Plugin selection, device index, ibverbs completion error |
0x70000 |
Transport | Shared memory or network data path, configuration mismatch between ranks |
0x80000 |
Proxy | Progress error inside the communication path |
0x90000 |
Group | Asynchronous work, unmatched start/end |
0xA0000 |
Enqueue | Operation preparation: no algorithm or protocol available, unpaired self send/receive |
0xB0000 |
Kernel | Kernel arguments, watchdog, abort |
1 2 3 | |
The range tells you where to look. A 0x20000 (socket) or 0x30000 (bootstrap) code means that the ranks could not connect to each other, so there is no need to look at anything past that point, while a 0xB0000 code means that they met but the transfer itself failed.
Note that when a rank does not arrive, no code is produced. If a rank cannot connect to the process that created the handle, it fails with a socket code, but if the connection succeeds and some ranks do not arrive, LcclCommInitRank() waits without an error. See Communicators and Troubleshooting.
6.3 Codes that are not failures¶
| Code | Meaning |
|---|---|
kLcclInProgress |
A nonblocking operation has not finished yet. Check again |
Every other value that is not 0 is a failure.
6.4 kLcclNotSupported is a code you encounter in normal use¶
This is the code you see most often while migrating from NCCL, and it is not a defect. It means that there is no kernel for the (operation, reduction, data type) combination, and it occurs at the call (at LcclGroupEnd() inside a group) before anything is transferred. Avg on integer types or on Float64 returns the same code.
Unlike a socket error, the remedy is clear, so handle it separately instead of lumping it in with general failures. See the Support matrix.
This code does not catch every unsupported combination
A communicator with 1 rank passes any combination. So not receiving kLcclNotSupported in a 1-rank test does not mean that the combination works with several ranks.
6.5 Asynchronous failures¶
The default communicator is blocking, and a collective operation has already finished when it returns. In that case the return value is the result.
There are two cases in which you must read the asynchronous error.
- A nonblocking communicator (
blocking = 0inLcclConfig, orLCCL_BLOCKING=0). The call returnskLcclInProgressand the work proceeds in the background. The result is recorded on the communicator. - When you receive
kLcclKernelAborted. For a failure inside the communication path, the cause code is recorded on the communicator. On the LPU, the failed call returns the cause code itself.
1 2 3 4 5 6 | |
A recorded error is not cleared
LcclCommGetAsyncError() returns the value recorded last, and reading it does not clear it. There is no way to recover a communicator on which an error has been recorded, so destroy it and create it again. Release a nonblocking communicator with LcclCommAbort() instead.
If a call seems to have stopped for no reason, also read the value of LcclCommGetAsyncError(). In some cases an internal failure is left only in this value.
The Python binding does not have LcclCommGetAsyncError(). Nonblocking mode cannot be used either. The binding treats kLcclInProgress as a failure too and throws an exception.
6.6 State after a failure¶
| Failure type | State afterwards |
|---|---|
kLcclNotSupported (an unregistered combination, the LPU's reduction-type and size rules, etc.; raised at the call or at LcclGroupEnd()) |
Nothing is transferred. A blocking communicator is healthy, and only that call is unusable. On a nonblocking communicator, the error is recorded and every later operation returns it (6.5). Inside a group, all operations of the group are discarded. If LcclSend/LcclRecv inside a group were rejected, the other ranks can fail with kLcclKernelSpinTimeout, so create the communicator again on all ranks |
Argument checks at call time (kLcclInvalidArgument, kLcclComm*), size rules (LcclAllToAll, LcclReduceScatter), and LcclAllGather called inside a group, on a nonblocking communicator, or in place with a count that is not on a 512-byte boundary |
Nothing is transferred. The communicator is healthy. Even inside a group, only that call is rejected |
| Bootstrap and socket errors | The communicator is not created |
Execution-stage errors: kernel arguments (kLcclKernelInvalidArgs), failures inside the transfer path, failures the runtime reports while an operation runs on the LPU (such as kLcclSystemError) |
The communicator is unusable. This cannot be undone |
kLcclKernelSpinTimeout |
The watchdog (LCCL_KERNEL_TIMEOUT_MS) cuts off the kernel. The communicator is aborted. Find the rank that stopped participating. On the LPU, it also happens when the ranks enter the operation further apart than this time (5 seconds by default) |
kLcclKernelAborted |
A communicator that has already been aborted. The first cause may be in LcclCommGetAsyncError() |
There is no per-operation recovery
A failure in the transfer path means that the ranks have progressed to different points. Calling the operation again does not restore synchronization. The unit of recovery is the communicator. Clean up on all ranks and create it again.
When one rank fails, the other ranks may not know it and keep waiting for it. This wait continues until the watchdog cuts it off, and the rank that was cut off receives kLcclKernelSpinTimeout. So the same incident can show a different code on each rank.
6.7 Cleanup after a failure¶
Release the resources of an aborted communicator with LcclCommDestroy(). This covers a communicator that the watchdog cut off or that was aborted by an error. If a peer has disappeared and an in-flight operation cannot finish, the release can hang, so use LcclCommAbort() instead.
1 | |
LcclCommAbort() and LcclCommDestroy() both release resources, so call only one of them on a handle. Do not use the handle after the call. If one process contains several ranks that belong to the same communicator (LcclCommInitAll, etc.), all of those ranks must be cleaned up before the resources are released. See Communicators.
6.8 Logging¶
lccl writes logs through the shared HyperAccel logger. Send the log to a separate file for each rank. When the output of several ranks is mixed, it is almost unreadable, yet what you actually need is often the order across ranks.
1 2 3 4 5 | |
Release builds do not emit Info-level and lower logs
A release build is set up to raise anything below Warn (3) to Warn, so even if you lower the level with halogger.set_level("trace") or HA_LOG_LEVEL=0, you usually cannot get Trace, Debug, or Info logs. Information such as the selected network interface and the number of channels is logged only at INFO level, so it cannot be checked from the log in a release build. Instead, in Python, read attributes such as comm.n_nodes and comm.n_channels. The logging variables are shared with the runtime. See Logging in the runtime's error handling.
When you report an issue, attach the WARN and ERROR logs, the LCCL_* environment variables you used, and the per-rank n_nodes and n_channels.
Next steps¶
- Troubleshooting — symptom index
- Error codes — full list