7. Troubleshooting¶
A list of failures indexed by the symptoms you observe. Search for the string printed in the log, or for the error code, exactly as it appears. Code definitions are in Error codes.
Most failures in a distributed environment take one of two forms: the ranks did not meet, or the ranks met but one rank does not participate. Determining which one applies is the first step of every diagnosis below.
Release builds print only WARN and above
Release builds do not print INFO and lower logs, so the diagnoses below are written to be followed with only the WARN and ERROR logs and the comm attributes. Even if you call halogger.set_level("trace"), the level is raised to Warn. See Error handling.
Before you begin¶
Collect the information from all ranks, not only from the rank that reported the error. The rank that failed is often not the rank that caused the failure.
1 2 3 4 5 | |
1 2 3 4 5 | |
Collect the logs in a separate file for each rank (Error handling). Read the log of the process that created the unique ID as well, not only that of rank 0. Errors from the stage where the ranks gather are printed there.
Startup stage¶
LcclCommInitRank does not return¶
Symptom — Some or all ranks hang at communicator creation.
Cause — The ranks did not meet. There is no timeout while ranks wait for each other, so if the ranks are mismatched, they wait without an error. Common reasons are listed below.
- Ranks received different unique IDs (no log)
- Fewer ranks called than the agreed number (no log)
nranksdiffers between ranks (ERROR log in the process that created the unique ID)- The same rank number connected twice (ERROR log in the process that created the unique ID)
Resolution
- In the log of the process that created the unique ID, search for
Bootstrap Root : mismatch in rank countorrank N of M ranks has already checked in. These are ERROR logs, so they are visible in release builds as well. If either line is present, the connection wait has already ended: ranks that have already connected wait forever, and ranks that have not yet connected fail with a socket error. Stop all ranks, makenranksand the rank numbers consistent, and restart. - If that log is absent, call
lccl.bootstrap_is_root_alive()in that process. If it returns1, the process is still waiting for the ranks to connect. This means that some ranks are missing or that the ranks received different unique IDs. - Check that all ranks received the same
nranksand the same unique ID bytes. Different IDs create separate gatherings, each waiting only for its own ranks, so no error log is left on either side. - If there are multiple servers, check that the other servers can reach the server that created the unique ID over the network. See Multi-node execution.
Creation fails with Communicator unknown or unclassified error (kLcclCommError)¶
Cause — Usually one of the following two.
- The device was not selected before the communicator was created (
hart.set_device()/HaSetDevice()was not called). - Two ranks on the same server claimed the same card. The log contains
Duplicate LPU detected : rank X and rank Y both on LPU device N.
Resolution — Select a different card for each rank with set_device, then create the communicator. On one server, a card can be claimed by only one rank.
Bootstrap could not find a usable network interface (kLcclBootstrapNoNetIf)¶
Cause — No usable network interface was found. If you set LCCL_SOCKET_IFNAME, the most common reason is that no interface matches that name. Once a name is set, the library does not fall back to automatic selection. With automatic selection, loopback is also picked as a last resort, so this error does not occur even if only loopback exists.
Resolution
- Check that
LCCL_SOCKET_IFNAMEmatches a name shown byip -o link(ethis a prefix,=eth0is an exact match,^docker,lois an exclusion). If you are not sure, remove it and leave the choice to automatic selection. - In a container, check that the network namespace contains an interface.
LCCL_SOCKET_FAMILY(AF_INET/AF_INET6) filters the candidate interfaces by address family. Specifying a family that the server does not have is enough to cause this error. Remove the value or set it to a family the server has.- Within a process, this failure is not retried once it has been determined. If you have fixed the environment variable, restart the process.
kLcclInvalidArgument from a failed LcclConfig validation¶
Cause — The LcclConfig was not initialized with the provided initializer macro, or a value is out of range. The struct has fields that the library validates. Search the log for LcclConfig argument not initialized via LCCL_CONFIG_INITIALIZER or Invalid config.
Resolution — Start from LCCL_CONFIG_INITIALIZER and set only the fields you want to change. Setting only min_num_chnl is rejected, so set max_num_chnl as well.
kLcclCommInvalidArgs or kLcclInvalidArgument from an invalid rank¶
Cause — myrank is outside the range [0, nranks). LcclCommInitRank returns kLcclCommInvalidArgs, and LcclCommInitRankConfig returns kLcclInvalidArgument. Search the log for Invalid rank requested.
Resolution — This is usually an off-by-one in the launcher, or ranks numbered from 1.
Startup fails with kLcclMemShmMapFail¶
Cause — Shared memory could not be created or mapped. The log contains one of failed to create shared memory file, failed to open shared memory file, failed to extend, or Could not map. Common reasons are listed below.
/dev/shmis too small. Ranks on the same server that run in separate processes communicate through/dev/shm, and the usage per server grows with the square of the number of ranks on that server. With the defaultLCCL_BUFFSIZE, 4 ranks take about 80 MiB, so the container default of 64 MiB holds at most 3 ranks.- Ranks on one server received different
LCCL_SHM_PATHvalues - Directory permissions
Resolution — Check the capacity and usage with df -h /dev/shm. The space in use does not show in ls or du. Give a container a larger /dev/shm (docker run --shm-size; on Kubernetes, mount an emptyDir with medium: Memory at /dev/shm), or point LCCL_SHM_PATH at a larger filesystem. All ranks on the same server must receive the same LCCL_SHM_PATH.
Startup fails due to a configuration mismatch between ranks¶
Symptom — kLcclTransportBuffSizeMismatch or kLcclTransportShmLocalityMismatch, together with a ranks disagree on buff_size or ranks disagree on LCCL_SHM_LOCALITY log. If LCCL_BUFFSIZE differs between ranks on different servers, startup fails with kLcclNetSizeMismatch instead, together with a Net received size does not match expected log.
Cause — The LCCL_* environment variables differ between ranks. Each process reads these values from its own environment, so they are not aligned between ranks automatically.
Resolution — Give all ranks the same LCCL_* environment. Also check that the config files (~/.lccl.conf, /etc/lccl.conf) do not differ between servers.
--network host containers on one server are treated as separate servers¶
Symptom — comm.n_nodes is larger than the number of servers.
Cause — Shared memory can be used only when both ranks see the same /dev/shm. If each container has its own /dev/shm, LCCL treats each container as a separate server and connects the containers over the network transport.
Resolution — To connect the containers through shared memory, have them share /dev/shm (for example, docker run --ipc=host). See Multi-node execution.
The communicator of N ranks on hw uses 1 channel, not M warning appears on the LPU¶
Cause — On the LPU, a communicator with 2 or more ranks uses only 1 channel. If more than 1 channel is requested (LCCL_MAX_NCHANNELS, LCCL_MIN_NCHANNELS, and so on), the library logs this warning and sets it to 1.
Resolution — Behavior is not affected, so you can ignore it. To remove it, drop the channel-count settings on the LPU.
Hangs after startup¶
A collective operation that looks hung does not hang forever. After LCCL_KERNEL_TIMEOUT_MS (default 5 seconds) elapses, the watchdog aborts the kernel, the call returns kLcclKernelSpinTimeout, and the log contains a device loop exceeded LCCL_KERNEL_TIMEOUT_MS ... warning. After that, that communicator cannot be used. Destroy it and create it again.
All ranks hang in a collective operation¶
Cause — The ranks disagree about the next collective operation. Collective operations are paired by order, so if the order diverges, they are paired incorrectly. The same happens when the size or data type differs between ranks.
Resolution
- Before each call, log the operation name, sequence number, and size on all ranks. Compare the logs across ranks; the first point where they differ is the bug.
- Look for control flow that depends on per-rank data, such as an early
return, acontinueon an empty batch, or a conditional collective operation. Every rank must call in the same order even when its own share is empty.
Two ranks hang in send/receive¶
Cause — The send and the receive were not registered inside a group. Both ranks hang in the send, and this continues until the watchdog cuts it off.
Resolution — Wrap the pair in LcclGroupStart() / LcclGroupEnd(). In torch.distributed, use dist.batch_isend_irecv. See Point-to-point.
The remaining ranks hang after one rank finishes or dies¶
Cause — A collective operation finishes only when all ranks participate. One rank destroyed the communicator first, or its process died.
Resolution — Bring the ranks down together. If one rank has to clean up first, the others also clean up with abort. See Communicators. You can find out which rank stopped first from the last call in each rank's log.
The watchdog cuts off a healthy operation¶
Symptom — kLcclKernelSpinTimeout even though all ranks called correctly.
Cause — A single operation takes longer than LCCL_KERNEL_TIMEOUT_MS (default 5 seconds). A very large transfer, a slow connection between servers, or a difference in start times between ranks can be the cause. On the LPU, the time also includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation.
Resolution — Have the ranks call the operation at about the same time, synchronize the calling thread's default stream before the call, split a very large transfer into several calls, or increase LCCL_KERNEL_TIMEOUT_MS. On the LPU, keep it at 9999 or below. A larger value, or 0 or less, can let a stuck kernel reach the driver's 12-second limit and leave the card needing a reset.
Progress stops with no error and no blocked call¶
Cause — An unmatched LcclGroupStart(). The thread stays in a group permanently, so operations are queued instead of executed.
Resolution — Check that there is no early return or exception between every LcclGroupStart() and its LcclGroupEnd(). In C++, close the group with a scope guard. In torch.distributed, if dist.batch_isend_irecv raised an exception, do not continue in that process; restart the processes of all ranks.
Rejected operations¶
kLcclNotSupported from a collective operation¶
Cause — There is no kernel for the (operation, reduction, data type) combination. The error occurs at the point of the call (at LcclGroupEnd() if inside a group), and nothing was transferred. The same applies to Avg with integer or Float64 types.
On the LPU, the following are also rejected with this code. The reason is in the ERROR log (does not fit a device loop and so on).
- Reductions on types other than
Float16andBfloat16 LcclBroadcast,LcclReduce,LcclGather,LcclScatter, andLcclSend/LcclRecvwhose byte size is not a multiple of 512- Calls larger than 4 GiB (
LcclAllReduce,LcclReduceScatter) or 8 GiB (the other operations) - Every collective and
LcclSend/LcclRecvwhen there are 2 or more ranks andLCCL_BUFFSIZEis not a multiple of 4096 (set a multiple of 4096, or drop the setting) - Collectives on LPUs other than EVT0
- Communicator creation with more than 16 ranks, or when
hartcannot read the format of the kernel library inlccl(installlcclandhartfrom the same release)
Resolution — Check the combination in the Support matrix. The usual surprises are the Max/Min reductions, Prod and Avg outside all-reduce and reduce, and the LPU's size rules.
kLcclInvalidArgument from a collective operation¶
Cause — A null pointer buffer, count == 0, an unknown LcclDataType or LcclRedOp, a root/peer outside the range [0, nranks), or overlapping LcclAllToAll buffers. On the LPU, a size that violates the 512-byte unit rule (LcclAllToAll, LcclReduceScatter) also returns this code. A LcclAllGather that is not on a 512-byte boundary and is called inside a group, on a nonblocking communicator, or in place also returns this code. The reason is logged on one line in the ERROR log. This rejection happens at call time, so the communicator is unaffected, and even inside a group only that call is dropped.
Resolution — Check that no code path calls with a null buffer or a count of 0 on an empty batch, and check that root and peer are within the communicator's rank range.
kLcclKernelInvalidArgs on the LPU¶
Cause — The LPU kernel cannot handle the arguments. The reason is in the ERROR log. Float32 reductions and LcclBroadcast and LcclSend/LcclRecv whose byte count is not a multiple of 512, the common causes of this code up to 0.0.5, are now rejected at call time with kLcclNotSupported.
Resolution — Check the reason in the logs of all ranks. If you set LCCL_CHUNK_SIZE, check that it meets the conditions in Environment variables. The communicator cannot be used after this failure, so destroy it and create it again.
kLcclEnqueueNoAlgo from a collective operation¶
Cause — No algorithm/protocol combination is left to use. Most cases are an incorrectly specified LCCL_PROTO or LCCL_ALGO. Unknown names are ignored without an error, so the allow list can end up empty, and because there is no LL kernel for the LPU, specifying only LCCL_PROTO=LL makes every collective operation on the LPU fail with this code.
Resolution — Remove LCCL_PROTO and LCCL_ALGO, or set them to valid names such as LCCL_PROTO=Simple and LCCL_ALGO=Ring. NCCL's tree and LL128 values do not exist in this library. See Environment variables.
[HA_LCCL_NO_HW_KERNEL] exception from torch.distributed¶
Cause — On the LPU, the lccl backend of torch.distributed blocks gather, scatter, reduce, and all_to_all in advance with this exception.
Resolution — Replace reduce with all_reduce, gather with all_gather, and scatter with broadcast or send/recv. For all_to_all, issue one send and one recv for each other rank in a single dist.batch_isend_irecv call, and copy your own rank's piece directly. Each piece you send or receive must be a separately allocated tensor, so copy the pieces of x.chunk(n) with .clone() before passing them. See Support matrix.
Wrong results¶
Results differ between ranks after all-reduce¶
Cause — The ranks did not contribute inputs of the same shape, or the send buffer was still being written when the collective operation was called. LCCL is not guaranteed to wait for work that was queued on a stream before the call, so calling from the same stream is not enough by itself.
Resolution — Synchronize the stream that produced the send buffer (HaStreamSynchronize), then call the collective operation. See Collective operations.
Only some ranks have a result and the rest are 0¶
Cause — For an operation that has a root, the result is placed only on the root. This is normal behavior for LcclReduce and LcclGather.
Resolution — If all ranks need the result, use the all- variants.
Performance¶
Collective operations are much slower than expected¶
Cause — See the items below.
Resolution — You can see where a collective operation spends its time in the lccl:<operation> span of the profiler trace. See Reading a trace and Support matrix.
Throughput does not increase when devices are added¶
Cause — With one channel, transfers are serialized. On the LPU, a communicator with 2 or more ranks has its channel count fixed at 1.
Resolution — Check the transfer size before drawing a conclusion.
Collectives slow down after the LPU flag BAR of device N stopped for this process warning¶
Cause — When a collective fails during execution or LcclCommAbort() is called on the device named in the warning, collectives on that device run slower on the LPU until the process exits. This also applies to communicators re-created in the same process.
Resolution — If you need full throughput on the LPU, restart the process instead of only re-creating the communicator.
Connections between servers¶
Symptoms of configurations with multiple servers are collected separately in Multi-node execution. Only the commonly encountered ones are repeated here.
| Symptom | Most common cause |
|---|---|
comm.n_nodes is smaller than the number of servers you expect |
Containers started on one machine with --network host share /dev/shm (they are treated as one server) |
| Rendezvous does not complete | The specified NIC cannot reach the other server, a firewall, or a container bridge address (172.17.x.x) |
kLcclSocketTimeout, kLcclSocketNotConnect |
The unique ID is stale or the address is unreachable |
Traffic between servers goes over lo |
LCCL_SOCKET_IFNAME is not set |
Cleanup¶
Shared memory files are left behind after a crash or a failed creation¶
Symptom — lccl-XXXXXX files accumulate in /dev/shm, and later runs may fail with kLcclMemShmMapFail because of insufficient space.
Cause — A rank died before both ranks had connected, or communicator creation failed (for example, with kLcclMemShmMapFail). Once the connection completes, the library deletes this file itself. File names differ from run to run, so they do not collide with a new run; they only take up space.
Resolution — Once no process on the server is using LCCL, delete the leftover lccl-* files.
Requesting support¶
If the items above do not resolve the problem, contact the HyperAccel support team and attach the following.
- The WARN and above logs of all ranks, in a separate file per rank, including the log of the process that created the unique ID
- The number of ranks, the number of ranks per server, the number of servers, and
comm.n_nodesandcomm.n_channelsfor each rank - The exact operation, reduction, data type, and size that failed
- The
LCCL_*environment variables you set and the contents of the config files - The
lccl,hart, and driver versions - Whether the problem also reproduces with
nranks == 2