Error codes
Every C++ entry point returns LcclResult. The values of existing codes do not change between releases, and no code is reused with a different meaning, but a release can add new codes. Treat an unknown code as a general failure.
Classification and handling are described in Error handling. LcclGetErrorToString() converts a code into a string for logs.
How to find a code
Often you see only a message, not the code itself.
| What you see |
What it contains |
Python exception RuntimeError: LcclAllReduce failed: Operation not supported |
The function name and the message. There is no code number |
ERROR ... -> <message> in the log |
The same message as above. It is a sentence, not a code name |
C++ return value, torch exception failed with LCCL result 720897 |
A number. Convert it to hexadecimal (printf '%X\n' 720897 → B0001) |
So each table below lists the code, the value, and the message of that code (the string returned by LcclGetErrorToString()). Search this page by message or number.
Ranges
A value has the structure (subsystem << 16) | number. The upper hexadecimal digits tell you at which layer the failure occurred. The base value of each subsystem (the …Error code, such as kLcclSocketError) is used as the general failure of that layer.
| Base value |
Subsystem |
0x00000 |
General |
0x10000 |
Communicator |
0x20000 |
Socket |
0x30000 |
Bootstrap |
0x40000 |
Memory / allocation |
0x50000 |
Topology / graph / XML |
0x60000 |
Net plugin |
0x70000 |
Transport |
0x80000 |
Proxy (inside the communication path) |
0x90000 |
Group |
0xA0000 |
Enqueue (operation preparation) |
0xB0000 |
Kernel / device |
Reserved codes
The twelve codes marked Reserved code that is not currently returned are present in the header, but the library does not return them anywhere. The same situations are reported with other codes (such as kLcclInvalidArgument). They may be used in the future, so include a default case in your switch.
General
| Code / Value |
Message |
Cause |
Resolution |
kLcclSuccess
0 |
Success |
Success. |
— |
kLcclSystemError
1 |
Underlying system call failed |
An underlying system call failed. A failed ibverbs call is also reported here. On the LPU, this code is also returned when the runtime reports a failure while an operation runs (for example, when the card stops responding or is being reset), and the communicator is then aborted. |
Check the log for the failed call. If an operation on the LPU returned this code, destroy the communicator and create it again. If the card is FATAL, recover the card first. |
kLcclInvalidArgument
2 |
Invalid argument |
An invalid argument. Typical cases are a null buffer, count == 0, an unknown data type or reduction, root or peer out of range, an uninitialized LcclConfig, a communicator that has already been destroyed, a backend named by LCCL_NET that does not exist, overlapping LcclAllToAll buffers, a size that breaks the 512-byte rule for LcclAllToAll or LcclReduceScatter on the LPU, and an LcclAllGather that is not on a 512-byte boundary and is called inside a group, in nonblocking mode, or in place. |
The cause is logged at WARN or ERROR level. count is in elements and cannot be 0, and the receive buffer size must follow the rules in Collective operations. |
kLcclNotSupported
3 |
Operation not supported |
No kernel is registered for the (operation, reduction, data type) combination (this includes Avg for integer types and Float64; such calls are rejected at call time). This code is also returned when the runtime answers "not supported". On the LPU, calls that the kernels cannot handle (reductions on types other than Float16 and Bfloat16; LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv whose byte count is not a multiple of 512; calls above the size limit; a paired LcclSend/LcclRecv in one group whose peer is the calling rank itself, on a communicator with 2 or more ranks; every collective and LcclSend/LcclRecv when there are 2 or more ranks and LCCL_BUFFSIZE is not a multiple of 4096) are also rejected with this code at call time. Communicator creation returns this code when an LPU communicator would have more than 16 ranks or when hart cannot read the format of the kernel library in lccl, and collectives return it on LPUs other than EVT0. |
Check the Support matrix. Nothing has been sent. |
kLcclInProgress
4 |
Work in progress |
This is not an error. The work of a nonblocking communicator has not finished yet. |
Wait for completion with LcclCommGetAsyncError. |
kLcclInternalError
5 |
Internal error: unreachable branch reached |
A branch that should be unreachable was reached. During communicator creation, this code is also returned when the two ends of an ibverbs connection differ in LCCL_IB_QPS_PER_CONNECTION or in port kind (InfiniBand or RoCE). |
In that case the ERROR log names the cause; align the settings as described in Environment variables. Otherwise, this is a defect. Collect the logs and report it. |
Communicator
| Code / Value |
Message |
Cause |
Resolution |
kLcclCommError
0x10000 (65536) |
Communicator unknown or unclassified error |
General failure of the communicator. Returned when two ranks on the same server claim the same LPU (Duplicate LPU detected), when the device lookup fails, or when setting up the communication path fails. |
Check that each rank binds a different LPU. Otherwise, look for the point of failure in the log. |
kLcclCommNotInit
0x10001 (65537) |
Communicator not initialized |
Reserved code that is not currently returned. |
— |
kLcclCommInvalidArgs
0x10002 (65538) |
Communicator got invalid arguments |
LcclCommInitRank was given nranks < 1, or the rank is outside [0, nranks). The same applies when ndev in LcclCommInitAll is negative or larger than the device count, or when devlist holds an out-of-range or duplicated device index. The registration calls (LcclCommRegister, LcclCommDeregister) also return this code when they reject a buffer, size, or handle argument. |
Ranks run from 0 to nranks - 1. This is usually an off-by-one error in the launcher. Device indices for LcclCommInitAll run from 0 to the device count minus 1 and must be distinct. Check the argument rules for the registration calls in Communicator API. |
kLcclCommNullPtr
0x10003 (65539) |
Communicator got nullptr arguments |
The communicator or an output pointer is null. The same applies when devlist of LcclCommInitAll is null. |
— |
kLcclCommConfigInvalid
0x10004 (65540) |
Communicator config invalid (missing LCCL_CONFIG_INITIALIZER or out-of-range field) |
Reserved code that is not currently returned. An invalid LcclConfig is reported as kLcclInvalidArgument. |
Start from LCCL_CONFIG_INITIALIZER and set only the fields you want to change. |
kLcclCommRankInvalid
0x10005 (65541) |
Communicator rank outside [0, n_ranks) |
Reserved code that is not currently returned. A rank range error is reported as kLcclCommInvalidArgs or kLcclInvalidArgument. |
— |
kLcclCommStateInvalid
0x10006 (65542) |
Communicator in unexpected state for this call |
The same communicator was called again before a nonblocking operation finished, or LcclCommFinalize() was called twice. It is also returned during initialization when the ranks within a server cannot be determined. |
Wait until LcclCommGetAsyncError returns kLcclSuccess, then make the call. |
kLcclCommAlreadyDestroyed
0x10007 (65543) |
Communicator has already been destroyed |
Reserved code that is not currently returned. Using a communicator that has already been destroyed may be reported as kLcclInvalidArgument, but this is not guaranteed. |
Do not reuse a destroyed handle. |
kLcclCommSchedulerError
0x10008 (65544) |
Communicator channel/ring scheduler produced invalid plan |
Determining the execution order of point-to-point operations failed. |
This is a defect. Collect the logs and report it. |
Socket
A socket error during startup almost always means that the ranks cannot reach the address of the process that created the unique ID. Check that a connection to the address can be made, that no firewall blocks TCP between servers, and that LCCL_SOCKET_IFNAME points to the correct interface. Which interface was chosen is recorded only in the INFO log, so it is not visible in release builds. See Multi-node execution.
| Code / Value |
Message |
Cause |
kLcclSocketError
0x20000 (131072) |
Socket unknown or unclassified error |
General failure of the socket. |
kLcclSocketNullPtr
0x20001 (131073) |
Socket got nullptr arguments |
Null socket pointer. |
kLcclSocketInvalidData
0x20002 (131074) |
Socket got invalid data |
Validation of the received data failed. |
kLcclSocketNotReady
0x20003 (131075) |
Socket not ready |
The socket is not ready yet. |
kLcclSocketNotAccept
0x20004 (131076) |
Socket failed to accept connection |
accept failed. |
kLcclSocketTimeout
0x20005 (131077) |
Socket operation timed out |
A connection, send, or receive did not finish within the wait limit. |
kLcclSocketNotConnect
0x20006 (131078) |
Socket failed to connect |
connect failed (a stale unique ID that has already been used, an unreachable network, a connection reset, and so on). It fails immediately if nobody is listening at the address, and after the operating system's TCP connect timeout (about 2 minutes with default Linux settings) if the address does not respond. The ERROR log contains Connect to <address> failed : <reason>. |
kLcclSocketNotListen
0x20007 (131079) |
Socket failed to listen |
listen failed. |
kLcclSocketNotCreate
0x20008 (131080) |
Socket failed to create |
Socket creation failed. |
kLcclSocketAborted
0x20009 (131081) |
Socket operation aborted |
The operation was aborted. |
kLcclSocketRecvFail
0x2000A (131082) |
Socket recv failed mid-stream |
recv failed mid-stream. |
kLcclSocketSendFail
0x2000B (131083) |
Socket send failed mid-stream |
send failed mid-stream. |
Bootstrap
| Code / Value |
Message |
Cause |
Resolution |
kLcclBootstrapError
0x30000 (196608) |
Bootstrap unknown or unclassified error |
General failure of the bootstrap. |
— |
kLcclBootstrapNullptr
0x30001 (196609) |
Bootstrap got nullptr arguments |
Null pointer in the bootstrap state. |
— |
kLcclBootstrapNoNetIf
0x30002 (196610) |
Bootstrap could not find a usable network interface |
There is no usable network interface. This code is also returned when LCCL_SOCKET_IFNAME matches no interface. |
Check that LCCL_SOCKET_IFNAME matches an actual interface name and that a non-loopback interface is up. |
kLcclBootstrapWrongRecv
0x30003 (196611) |
Bootstrap received unexpected data |
The received message is larger than the buffer meant to receive it. The log contains Message truncated : received N bytes instead of M. |
Check that all ranks use the same version of lccl. |
kLcclBootstrapTruncated
0x30004 (196612) |
Bootstrap message truncated |
A message exchanged during the rendezvous is larger than the buffer meant to receive it (a Message truncated warning). |
Check that all ranks use the same version of lccl. |
kLcclBootstrapRingFail
0x30005 (196613) |
Bootstrap ring construction failed |
The connections through which the ranks exchange information with each other could not be set up. |
— |
kLcclBootstrapUnexpectedConn
0x30006 (196614) |
Bootstrap saw unexpected pending connection |
An unexpected connection is pending. |
Check whether more ranks than nranks have connected, or whether ranks from a previous run are still alive. |
Memory
| Code / Value |
Message |
Cause |
Resolution |
kLcclMemError
0x40000 (262144) |
Memory unknown or unclassified error |
General failure of memory. |
— |
kLcclMemAllocFail
0x40001 (262145) |
Host malloc/calloc/realloc failed |
Host allocation failed. |
— |
kLcclMemDeviceAllocFail
0x40002 (262146) |
Device (HaMemAlloc) allocation failed |
Device allocation failed. |
Device memory is exhausted or fragmented. See Memory model. |
kLcclMemCopyFail
0x40003 (262147) |
HaMemCpy failed |
Device copy failed. |
— |
kLcclMemFreeFail
0x40004 (262148) |
HaMemFree failed |
Device free failed. |
— |
kLcclMemShmMapFail
0x40005 (262149) |
shm_open / ftruncate / mmap failed |
Shared memory could not be created or mapped. |
Check the /dev/shm capacity (64 MiB by default in containers), whether the ranks on one server received the same LCCL_SHM_PATH, and the directory permissions. |
kLcclMemShmUnmapFail
0x40006 (262150) |
munmap / shm_unlink failed |
Shared memory could not be released. |
— |
kLcclMemSizeOverflow
0x40007 (262151) |
Reallocation requested smaller size than current |
The size to reallocate is smaller than the current size. |
— |
Topology
| Code / Value |
Message |
Cause |
Resolution |
kLcclTopoError
0x50000 (327680) |
Topology unknown or unclassified error |
General failure of the topology. |
— |
kLcclTopoLimitExceeded
0x50001 (327681) |
Topology static limit exceeded |
The static limit on nodes, links, or channels was exceeded. |
— |
kLcclTopoNodeNotFound
0x50002 (327682) |
Topology lookup did not find node/LPU/NET/CPU |
The lookup for a node, LPU, network, or CPU returned no result. This code is also returned when the topology file does not contain the card claimed by a rank on this server. |
Check that LPU.busId in the file is the same as the PCI address of the actual card. See Topology file. |
kLcclTopoXmlParseError
0x50003 (327683) |
Topology XML parse error |
The topology document could not be parsed. A topology JSON file uses the same code. |
Check the JSON format of the file. |
kLcclTopoXmlVersionMismatch
0x50004 (327684) |
Topology XML graph version mismatch |
The version of the topology XML graph differs from that of the library. |
— |
kLcclTopoFileError
0x50005 (327685) |
Topology file open / read / size failure |
The topology file could not be opened or read. This code is also returned with LCCL_TOPO_STRICT=1 when the file given by LCCL_PREDEFINED_SYSINFO does not exist. |
Check the path and the file permissions. |
kLcclTopoConfigInvalid
0x50006 (327686) |
Topology config invalid |
The topology configuration is invalid. Typical causes are a problem with the number of channels and contradictions within the topology file itself (a duplicated busId, a missing NIC.busId, and so on). |
Check the warnings in the log and inspect the topology file. |
kLcclTopoLookupNotFound
0x50007 (327687) |
Topology key/value or attribute lookup failed |
An attribute lookup failed. |
— |
kLcclTopoSchemaVersionMismatch
0x50008 (327688) |
Topology JSON schemaVersion missing or unsupported |
The schemaVersion of the topology file is missing (LCCL_TOPO_STRICT=1), has an unsupported value, or is not a number. |
Add "schemaVersion": 1 at the top level of the file. |
kLcclTopoPeerMismatch
0x50009 (327689) |
Topology contradicts the local device (host hash / busId) |
Reserved code that is not currently returned. |
— |
Net plugin
| Code / Value |
Message |
Cause |
Resolution |
kLcclNetError
0x60000 (393216) |
Net plugin unknown or unclassified error |
General failure of the network backend. |
— |
kLcclNetNotFound
0x60001 (393217) |
Net plugin or named network not available |
The ibverbs backend could not be initialized (when there is no RDMA device or IP interface). It is handled internally and falls back to sockets, so it is usually not returned to the caller. |
— |
kLcclNetInvalidDevice
0x60002 (393218) |
Net device index out of range |
The device index is out of range. |
— |
kLcclNetNoSlot
0x60003 (393219) |
Net request/subtask slot pool exhausted |
The number of network requests that can be handled at the same time was exceeded. |
— |
kLcclNetThreadFail
0x60004 (393220) |
Net helper pthread_create failed |
The thread that assists network transfers could not be created. |
— |
kLcclNetReadFail
0x60005 (393221) |
Net plugin read syscall failed |
The socket read failed. |
— |
kLcclNetSendCountInvalid
0x60006 (393222) |
Net iSendv/iRecvv called with n != 1 |
Multi-buffer send and receive are not supported. |
— |
kLcclNetTestNullRequest
0x60007 (393223) |
Net test() called with NULL request |
The transfer status was queried with a null request. |
— |
kLcclNetSizeMismatch
0x60008 (393224) |
Net received size does not match expected |
The received size differs from the expected size. When communicator creation fails with this code, LCCL_BUFFSIZE differs between ranks on different servers. |
If it appears during creation, give all ranks the same LCCL_BUFFSIZE. If it appears during an operation, check that the sending side and the receiving side called with the same count and data type. |
kLcclNetCommNotInit
0x60009 (393225) |
Net comm guard yielded no comm |
The network connection has not been initialized. |
— |
kLcclNetUnsupportedMemType
0x6000A (393226) |
Net plugin does not support requested memory type |
The backend does not support this kind of memory. |
— |
kLcclNetFlushUnsupported
0x6000B (393227) |
Net flush operation not supported |
Flush is not supported. |
— |
kLcclNetGdrUnsupported
0x6000C (393228) |
Net GDR/RDMA feature not supported on this platform |
GDR/RDMA is not supported on this platform. |
— |
kLcclNetRemoteError
0x6000D (393229) |
Net work completion reported a failure from the peer |
An ibverbs work completion reported a failure on the peer side. |
Check the peer server, the RNIC state, and the cables, and review LCCL_IB_TIMEOUT and LCCL_IB_RETRY_CNT. |
Transport
| Code / Value |
Message |
Cause |
Resolution |
kLcclTransportError
0x70000 (458752) |
Transport unknown or unclassified error |
General failure of the transport layer. |
— |
kLcclTransportNotFound
0x70001 (458753) |
No transport matched the requested peer/operation |
No transport can be used between the two ranks (P2P, shared memory, and network are all unavailable). |
— |
kLcclTransportShmInvalidArg
0x70002 (458754) |
SHM transport got mismatched proxy info sizes |
The shared memory transport received connection information sizes that do not match each other. |
— |
kLcclTransportShmDestroyFail
0x70003 (458755) |
SHM transport destroy of stream/fifo/event failed |
The resources of the shared memory transport could not be released. |
— |
kLcclTransportShmConnectFail
0x70004 (458756) |
SHM transport connect/accept polling failed |
The connection of the shared memory transport could not be established. |
— |
kLcclTransportShmChannelGeometryUnknown
0x70005 (458757) |
SHM transport could not read device DRAM channel count/stride |
The DRAM channel configuration (channel count and stride) could not be read from the active device. |
— |
kLcclTransportNetDestroyFail
0x70006 (458758) |
NET transport destroy of stream/fifo/event failed |
The resources of the network transport could not be released. |
— |
kLcclTransportNetUnsupported
0x70007 (458759) |
NET transport data path reached an unwired protocol/buffer mode |
The protocol or buffer mode is not supported by the network transport. On the LPU, a communicator creation that exceeds the per-process communicator limit also fails with this code. |
If creation failed, destroy communicators you no longer use, then create it again. |
kLcclTransportShmUnsupported
0x70008 (458760) |
SHM transport data path reached an unwired protocol |
The protocol is not supported by the shared memory transport. On the LPU, a communicator creation that exceeds the per-process communicator limit also fails with this code. |
If creation failed, destroy communicators you no longer use, then create it again. |
kLcclTransportP2pChannelMismatch
0x70009 (458761) |
P2P peer interleaves its fifo across a different DRAM channel count |
The peer device to be connected over P2P has a different number of DRAM channels than this device. |
— |
kLcclTransportBuffSizeMismatch
0x7000A (458762) |
ranks disagree on buff_size (LCCL_BUFFSIZE must match across ranks) |
LCCL_BUFFSIZE differs between ranks. Ranks with different values cannot connect, so the connection is rejected. |
Give all ranks the same LCCL_* environment. |
kLcclTransportShmLocalityMismatch
0x7000B (458763) |
ranks disagree on LCCL_SHM_LOCALITY (must match across ranks) |
The shared memory transport setting differs between ranks. Ranks with different values cannot connect, so the connection is rejected. |
Give all ranks the same LCCL_* environment. |
Proxy
An error inside the communication path aborts the communicator and is accompanied by a [Progress Thread] progressOps failed warning. On the LPU, the operation that fails in this case returns the code that caused the abort. Most of the codes below indicate an internal defect or resource exhaustion, so collect the logs and send an inquiry.
| Code / Value |
Message |
Cause |
kLcclProxyError
0x80000 (524288) |
Proxy unknown or unclassified error |
General failure inside the communication path. |
kLcclProxyResponseMismatch
0x80001 (524289) |
Proxy response size or state does not match request |
The size or state of an internal response does not match the request. |
kLcclProxyOpNotFound
0x80002 (524290) |
Proxy could not find an op matching the given op_id |
No in-progress operation matches the id. |
kLcclProxyOpsNull
0x80003 (524291) |
Proxy per-rank ops table is null |
There is no per-rank operation list. |
kLcclProxyEnqueueFail
0x80004 (524292) |
Proxy failed to enqueue op into fifo |
The operation to be progressed could not be put into the internal queue. |
kLcclProxyConnInvalid
0x80005 (524293) |
Proxy connector for peer/channel is invalid |
The connection for the peer and channel is not valid. |
kLcclProxyMsgUnknown
0x80006 (524294) |
Proxy message subcommand is unknown or unhandled |
Unknown internal message. |
kLcclProxyOpProgressFail
0x80007 (524295) |
Proxy op state machine could not progress |
Reserved code that is not currently returned. |
kLcclProxyThreadCreateFail
0x80008 (524296) |
Proxy service pthread_create failed |
The service thread that drives the communication path could not be created. |
kLcclProxyPeerSocksNull
0x80009 (524297) |
Proxy peer_socks not initialized |
The peer connection information has not been initialized. |
Group
| Code / Value |
Message |
Cause |
Resolution |
kLcclGroupError
0x90000 (589824) |
Group unknown or unclassified error |
General failure of the group. |
— |
kLcclGroupBlockingMixed
0x90001 (589825) |
Blocking and non-blocking communicators mixed in same group |
Reserved code that is not currently returned. Mixing blocking and nonblocking communicators in one group results in kLcclInvalidArgument together with the warning Blocking and nonblocking communicators are not allowed in the same group. |
Put only communicators of the same mode in one group. |
kLcclGroupSetDeviceFail
0x90002 (589826) |
HaSetDevice failed inside group dispatch |
Setting the device failed during group dispatch. |
— |
kLcclGroupAsyncJobUnknown
0x90003 (589827) |
Async job is in an unknown state |
An asynchronous job reached an unknown state. |
This is a defect. Collect the logs and report it. |
kLcclGroupJoinFail
0x90004 (589828) |
pthread_join on group thread failed |
join on the group thread failed. |
— |
kLcclGroupCaptureMixed
0x90005 (589829) |
Group has mix of HA-graph captured and uncaptured comms |
Only some of the communicators in the group are in the graph capture state. |
— |
kLcclGroupNotInGroup
0x90006 (589830) |
LcclGroupEnd called without matching LcclGroupStart |
LcclGroupEnd was called without LcclGroupStart. |
Check that starts and ends are paired. |
kLcclGroupSimInfoInvalid
0x90007 (589831) |
LcclSimInfo missing LCCL_SIM_INFO_INITIALIZER |
Reserved code that is not currently returned. |
— |
Enqueue
| Code / Value |
Message |
Cause |
Resolution |
kLcclEnqueueError
0xA0000 (655360) |
Enqueue/plan unknown or unclassified error |
General failure in the preparation stage before an operation runs. A failure while building the kernel arguments belongs here. |
— |
kLcclEnqueueNoAlgo
0xA0001 (655361) |
No algorithm/protocol pair available for op |
There is no algorithm/protocol pair for the operation. The typical case is allowing only LL, as in LCCL_PROTO=LL. |
Restore LCCL_PROTO to its default value or allow Simple, as in LCCL_PROTO=Simple. See Support matrix. |
kLcclEnqueueUnknownPattern
0xA0002 (655362) |
Unknown LcclPattern or collective+algorithm combination |
An unknown combination of collective operation and algorithm. |
This is a defect. Collect the logs and report it. |
kLcclEnqueueRegBufInvalid
0xA0003 (655363) |
Invalid LcclRegBufferType on queued task |
Reserved code that is not currently returned. A registered buffer error is reported as kLcclInvalidArgument. |
— |
kLcclEnqueueSelfP2PUnmatched
0xA0004 (655364) |
Self-rank send/recv lacks matching peer op |
An LcclSend to the sending rank itself has no matching LcclRecv (and vice versa). |
Do not send to or receive from your own rank; copy your own share of the data directly. Pairing the two in one group is not supported either. |
kLcclEnqueueDefault
0xA0005 (655365) |
Enqueue dispatch fell through default branch |
Reserved code that is not currently returned. |
— |
Kernel
| Code / Value |
Message |
Cause |
Resolution |
kLcclKernelError
0xB0000 (720896) |
Kernel unknown or unclassified error |
General failure of the kernel. |
— |
kLcclKernelInvalidArgs
0xB0001 (720897) |
Kernel args struct or pointers malformed |
The kernel argument structure or pointers are malformed. That communicator can no longer be used after this failure. Float32 reductions on the LPU and LcclBroadcast and LcclSend/LcclRecv whose byte count is not a multiple of 512, which failed with this code up to 0.0.5, are now rejected at call time with kLcclNotSupported. |
Check the buffer pointers and sizes, then destroy the communicator and create it again. |
kLcclKernelSpinTimeout
0xB0002 (720898) |
Launch watchdog deadline elapsed; host aborted a kernel stuck in a flag spin-wait |
The watchdog deadline (LCCL_KERNEL_TIMEOUT_MS, 5 seconds by default) elapsed and the stalled kernel was aborted. It occurs when a peer rank never arrives or when the call order or sizes differ between ranks. On the LPU, it also occurs when the ranks enter the operation further apart than this time. An aborted communicator cannot be used again. |
Find the stalled rank, check that all ranks called in the same order with the same sizes, then destroy the communicator and create it again. |
kLcclKernelChannelMask
0xB0003 (720899) |
Plan channel_mask did not match comm topology |
Reserved code that is not currently returned. |
— |
kLcclKernelInternalError
0xB0004 (720900) |
Unreachable branch hit inside kernel |
Reserved code that is not currently returned. |
— |
kLcclKernelAborted
0xB0005 (720901) |
Kernel spin-wait observed an externally requested abort |
The kernel saw the abort signal set on this communicator and finished. Either LcclCommAbort was called, or the communicator had already been aborted by an error inside the communication path or by an earlier kernel failure. A call aborted by the watchdog receives kLcclKernelSpinTimeout. |
If you called LcclCommAbort, do not use that handle again and create a new communicator. Otherwise the first cause is the code that an earlier failed call returned (for an error inside the communication path, the value of LcclCommGetAsyncError); check it, then destroy the communicator and create it again. |
See also