Release notes¶
Changes to the Legato Collective Communication Library (lccl). lccl is released with the same version as the other SDK components. For changes to the whole stack, see SDK release notes.
0.0.6¶
Requires Python 3.10 and Runtime (hart) 0.0.6.
Major changes¶
LcclAllReduceandLcclAllGatheraccept anycount. The constraint that requiredcounton the LPU to be aligned to a unit is removed. The receive buffer is written only up to the result size. Ifcountis not on a 512-byte boundary,LcclAllGathergathers through an extra copy, so it rejects the call withkLcclInvalidArgumentinside a group, on a nonblocking communicator, and in an in-place call.- The C++ public header becomes the single
lccl.hand ships in thelcclpackage. Includelccl/include/lccl.hof the site-packages wherelcclis installed, and addinclude/of thehartpackage to the include path as well.collectives.handgroup.hare internal headers. For how to build and link, see C++ API. - An
lcclpackage for the LPU.lcclis released as the+hwversion. Installhartas the+hwbuild of the same version too. The packages to install are listed on the home page. - The kernels are inside
libha_lccl.so. Every operation finds its kernels even when the library is installed from a package, and there are no separate kernel files. In 0.0.5, with 2 or more ranks,LcclAllReduce,LcclReduceScatter, andLcclBroadcastlooked for the kernel files at a path on the machine that built them, sotorch_hainstalled from a package could not launch their kernels. LcclReduce,LcclGather, andLcclScatterrun on the LPU.LcclReducetakesSum,Prod, andAvgonFloat16andBfloat16. In 0.0.5, these three operations had no device kernel. Thetorch.distributedbackend still blocksgather,scatter,reduce, andall_to_allon the LPU.- LPU collectives run in one device kernel run. An LPU communicator with 2 or more ranks finishes collectives and
LcclSend/LcclRecvinside one launched kernel instead of launching a kernel for each step. With fewer kernel launches, operations on small sizes are faster. - LPU Direct RDMA between servers. On the ibverbs transport, when the driver (2.6.0 or later), the RNIC, and the PCIe layout meet the conditions, the RNIC reads and writes LPU memory directly without moving the data through host memory. Connections that do not meet the conditions go through host memory as before. It is controlled with
LCCL_NET_RDMA_LEVEL,LCCL_NET_RDMA_READ, andLCCL_GDR_FLUSH_DISABLE. A process that uses LPU Direct RDMA allocates 1 GiB of LPU memory per LPU by default and keeps it until the process exits. See Multi-node execution.
Changes¶
- Removed:
LcclCommRingGranularity,LcclCommChannelGranularity, and thetorch_haoperators that wrapped them,torch.ops.lpu.lccl_ring_granularityandtorch.ops.lpu.lccl_channel_granularity. Code that paddedcountor a tensor size to the unit they returned can drop that part. There is no replacement function or operator. For the operations whose byte size must be a multiple of 512, see Known issues below. - On the LPU,
LcclReduceScatterwith 2 or more ranks rejects the call withkLcclInvalidArgumentbefore launching the kernel if the byte size ofrecv_countis not a multiple of 512. Previously, it failed during execution withkLcclKernelInvalidArgs. import lcclno longer imports PyTorch. The extension module islcclitself, and thelccl.lcclpath is gone. The public names are unchanged.lcclfinds thehartinstalled in the same Python environment, soLD_LIBRARY_PATHis not needed. The way to link a C++ program has changed as well. See C++ API.- When you use
lcclandtorch_hain one process, installhart,lccl, andtorch_hafrom the same release. See Python API. - A communicator cannot mix 0.0.6 ranks with 0.0.5 ranks. If you run across several servers, upgrade
lccl(ortorch_ha) to 0.0.6 on every server together. - The only dependency is now
hart, sopytestand others are no longer installed with it. The test executables are removed from the package. - The license notices of NCCL, rdma-core, and pybind11 are included in
dist-info/licenses/of the package. They are also on the home page. - An LPU communicator has at most 16 ranks. Beyond that,
LcclCommInit*fails withkLcclNotSupported. - On the LPU, a communicator with 2 or more ranks uses 1 channel. If
LCCL_MAX_NCHANNELSorLCCL_MIN_NCHANNELSis set above 1, the library logs a warning and ignores it. The warning that was logged at every communicator creation whenLCCL_MAX_NCHANNELSwas not set is gone. - On the LPU, the default of
LCCL_KERNEL_TIMEOUT_MSis 5000 (5 seconds). It applies to each device kernel run, and the time includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation. A value outside 1 to 9999 makes the library log a warning, because a stuck kernel that reaches the driver's 12-second limit leaves the card needing a reset. - On the LPU, a call that the kernels cannot handle is rejected with
kLcclNotSupportedat the point of the call (atLcclGroupEnd()inside a group). This covers reductions on types other thanFloat16andBfloat16;LcclBroadcast,LcclReduce,LcclGather,LcclScatter, andLcclSend/LcclRecvwhose byte size is not a multiple of 512; and calls larger than 4 GiB (LcclAllReduce,LcclReduceScatter) or 8 GiB (the other operations). In 0.0.5,Float32reductions andLcclBroadcastandLcclSend/LcclRecvwhose byte size was not a multiple of 512 failed during execution withkLcclKernelInvalidArgsand left the communicator unusable. - On LPUs other than EVT0, collectives return
kLcclNotSupported. - On the LPU with 2 or more ranks,
LCCL_BUFFSIZEmust be a multiple of 4096. Otherwise every collective andLcclSend/LcclRecvreturnskLcclNotSupported. - If
hartcannot read the format of the kernel library inlccl,LcclCommInit*fails withkLcclNotSupported. Installlcclandhartfrom the same release. - New environment variables:
LCCL_DEVICE_LOOP_LAUNCH_BYTES(the most bytes one device kernel run ofLcclAllReduce,LcclReduceScatter, orLcclSend/LcclRecvhandles, 256 MiB by default) andLCCL_DMABUF_POOL_SIZE_PER_CHANNEL(the per-DRAM-channel size of the buffers used for LPU Direct RDMA, 64 MiB by default). See Environment variables. - In the topology file (
LCCL_PREDEFINED_SYSINFO),LPUandNICcan havelinkspeedas well asPCI. If it is missing ornull, 12 GB/s is used instead of the parent's value. Numeric fields take JSON numbers only, and a file with a broken structure is not read partially but is treated as an initialization error. In 0.0.5, initialization did not finish in some of those cases. - In the topology file, a
PCI,LPU, orNICentry whosebusIdis missing or not written as a PCI address makes initialization fail withkLcclTopoConfigInvalid.
Fixes¶
- On an LPU communicator with 1 rank, an
LcclAllReduce,LcclAllGather,LcclReduceScatter,LcclBroadcast, orLcclReducethat was not in place filled the result only in the first DRAM channel or failed withBad address. Fixed. - A large
LcclBroadcastbetween LPU ranks connected through shared memory occasionally hung or received wrong values. Fixed. - If the topology file had a CPU with
socketId-1, runs between servers could not find the NIC. Fixed. - With a topology file, network devices without a PCI address (veth, bond, etc.) were attached to the first CPU in the file. They are now attached to the CPU of the card used by the lowest rank on this server, the same as when LCCL discovers the hardware itself.
Known issues¶
| Issue | Impact | Workaround |
|---|---|---|
LPU reductions are Float16 and Bfloat16 only |
Reductions on other types return kLcclNotSupported |
Change the data type |
On the LPU, the byte size of LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv must be a multiple of 512 |
Otherwise kLcclNotSupported |
Align the byte size to a multiple of 512 |
No kernel for the Max and Min reductions |
kLcclNotSupported with 2 or more ranks |
Use Sum or reduce on the host |
| Running operations of two communicators on the same LPU at the same time is not supported | — | Run the operations of one communicator at a time on an LPU |
On the LPU, the time the ranks take to enter an operation counts toward LCCL_KERNEL_TIMEOUT_MS (5 seconds by default) |
If the ranks are too far apart, the call fails with kLcclKernelSpinTimeout and the communicator must be re-created |
Have the ranks call the operation at about the same time. If needed, raise the value, up to 9999 |
| Collectives do not establish ordering with streams | Work queued before the call is not guaranteed to finish first | Synchronize yourself before the call |
| P2P only within the same process | No P2P benefit when each rank uses its own process | — |
No Python bindings for LcclCommGetAsyncError and LcclCommAbort |
A Python process cannot check for asynchronous failures or abort | Drive it from C++ if it matters |
| Release builds do not print logs at INFO or below | You cannot check the selected interface, the number of channels, or whether LPU Direct RDMA is used from the logs | Read the comm.n_nodes and comm.n_channels attributes. To check whether LPU Direct RDMA is used, read the driver's dmabuf_exports node (Multi-node execution) |
| A transport-path failure or a watchdog timeout requires re-creating the communicator | No per-operation recovery | Clean up on all ranks and re-create |
0.0.5¶
2026-10-14. Requires Python 3.10 and Runtime (hart) 0.0.5.
Major changes¶
- P2P transport between ranks in the same process. If the driver allows memory mapping between two cards, data movement for
LcclAllGather,LcclAllReduce, andLcclReduceScattergoes over direct DMA between the cards instead of through host shared memory. In configurations where each rank uses its own process, it is not selected and shared memory is used. - ibverbs network transport (InfiniBand, RoCE). For communication between servers, ibverbs is used before sockets if an RDMA device is present. Data goes through host memory.
LCCL_IB_*variables for configuration were added. See Environment variables. Bfloat16reduction.LcclAllReduce(Sum,Prod,Avg) andLcclReduceScatter(Sum) acceptBfloat16.LcclBroadcastandLcclAllToAllexecute on the LPU. Previously, they malfunctioned on the LPU. This has been verified on 2 and 4 LPUs in one server.- Buffer registration API.
LcclCommRegister,LcclCommDeregister, andLCCL_LOCAL_REGISTER. This API letsLcclSend/LcclRecvon a network transport use registered buffers directly, but the bundled network transports (socket, ibverbs) do not use it, so for now the calls succeed but have no effect. - Fixed-topology JSON contract. Added
schemaVersionto theLCCL_PREDEFINED_SYSINFOfile and added the strict modeLCCL_TOPO_STRICT. See Multi-node execution.
New¶
- 6 error codes:
kLcclTopoSchemaVersionMismatch,kLcclTopoPeerMismatch,kLcclNetRemoteError,kLcclTransportP2pChannelMismatch,kLcclTransportBuffSizeMismatch,kLcclTransportShmLocalityMismatch. The values of existing codes did not change. LcclCommChannelGranularity. (Removed in 0.0.6; see 0.0.6.)
Changes¶
- Rank, device number, and ring are no longer written to or read from the topology file. The file describes only the hardware layout, and LCCL fills in the rest at runtime.
- A topology file in which an LPU and an RNIC are under the same PCIe switch is accepted.
Known issues¶
| Issue | Impact | Workaround |
|---|---|---|
On the LPU, count of LcclAllReduce must be a multiple of 512 bytes × number of channels × number of ranks, and count of LcclAllGather a multiple of 512 bytes × number of channels |
If violated, kLcclKernelInvalidArgs during execution. The unit differs from server to server |
Query the unit with LcclCommRingGranularity and LcclCommChannelGranularity and align to it. Resolved in 0.0.6 |
On the LPU, recv_count of LcclReduceScatter does not match the unit |
Not rejected at call time; fails during execution with kLcclKernelInvalidArgs, and the communicator can no longer be used |
Align recv_count to the unit. Rejected at call time with kLcclInvalidArgument in 0.0.6 |
On the LPU, the byte count of LcclBroadcast or LcclSend/LcclRecv is not a multiple of 512 |
Rejected during execution with kLcclKernelInvalidArgs, and the communicator can no longer be used |
Align the byte count to a multiple of 512 |
On the LPU, LcclReduce, LcclGather, and LcclScatter have no device kernel |
Not rejected; access error or hang | Do not call them. Replace them with LcclAllReduce, LcclAllGather, etc. |
LPU reductions support only Float16 and Bfloat16 |
Float32 reductions fail with kLcclKernelInvalidArgs during execution |
Change the data type |
No kernel for Max and Min reductions |
kLcclNotSupported with 2 or more ranks |
Use Sum, or reduce on the host |
| Collectives do not establish ordering with streams; no asynchronous execution | Does not wait for work queued before the call | Synchronize yourself before the call |
| P2P works only within the same process; ibverbs goes through host memory | No P2P benefit if each rank uses its own process. No direct RDMA of LPU memory between servers | — |
| ibverbs has been verified only with 2 RNICs connected within one server | Behavior and performance of ibverbs configurations between servers have not been verified | Use sockets with LCCL_IB_DISABLE=1 |
No Python bindings for LcclCommGetAsyncError and LcclCommAbort |
A Python process cannot check for asynchronous failures or abort | If this matters, run it from C++ |
| Release builds do not print logs at INFO level or below | Cannot check the selected interface, number of channels, etc. in the logs | Read the comm.n_nodes and comm.n_channels attributes |
| A transport-path failure or watchdog timeout requires re-creating the communicator | No per-operation recovery | Clean up on all ranks, then re-create |
0.0.4¶
2026-08-28. Requires Python 3.10 and Runtime (hart) 0.0.4.
Major changes¶
- More operations run as device kernels on the LPU.
LcclAllReduce,LcclAllGather, andLcclReduceScatterrun as device kernels on 2 LPUs, andLcclSend/LcclRecvalso run on the LPU. - TCP socket transport between servers. Ranks on different servers perform collectives over the network. This has been verified on 2 servers.
- Unsupported combinations are rejected. Fixed the problem where a combination with no registered kernel silently returned success; it now fails with
kLcclNotSupported. - Hang-prevention watchdog. If a kernel hangs because one rank does not arrive, the watchdog aborts the kernel after
LCCL_KERNEL_TIMEOUT_MS(30 seconds by default) and returnskLcclKernelSpinTimeout. The aborted kernel returnskLcclKernelAborted.
New¶
ProdandAvgreductions (LcclAllReduce).- Profiler integration. Collectives and point-to-point operations are recorded as profiler ranges
lccl:<operation>, with rank and worker thread labels attached. - Python accepts the integer device address returned by
hart.mem_alloc()as a buffer. - Collectives on a communicator with 1 rank.
- 5 error codes:
kLcclTransportShmChannelGeometryUnknown,kLcclTransportNetDestroyFail,kLcclTransportNetUnsupported,kLcclTransportShmUnsupported,kLcclKernelAborted. LcclCommRingGranularity. (Still present in 0.0.5; removed in 0.0.6; see 0.0.6.)
Changes¶
LcclAllToAllrejects overlapping send and receive buffers.- Collectives without a reduction (broadcast, all-gather, gather, scatter) move data byte by byte, so they accept every
LcclDataType. - Group nesting tracks the depth per thread, and the work runs when the outermost region closes.
Known issues¶
| Issue | Impact | Workaround |
|---|---|---|
LcclBroadcast and LcclAllToAll do not run on the LPU |
Calling them on the LPU malfunctions | Use 0.0.5 or later |
No Bfloat16 reduction |
Bfloat16 reductions are rejected |
Use 0.0.5 or later, or Float16 |
No kernel for Max and Min reductions |
kLcclNotSupported |
Use Sum, or reduce on the host |
| The P2P and ibverbs transports are stubs | Shared memory on the same server, TCP sockets between servers | Use 0.0.5 or later |
| No buffer registration entry point | No fast path for registered buffers | — |
No Python binding for LcclCommGetAsyncError |
A Python process cannot check for asynchronous failures | If this matters, run it from C++ |
0.0.3¶
2026-06-18. The first release of lccl. It provides rank rendezvous, topology and ring discovery, and 6 collectives (LcclAllReduce, LcclReduce, LcclReduceScatter, LcclBroadcast, LcclGather, LcclScatter). LcclAllGather, LcclAllToAll, LcclSend, and LcclRecv were in the headers but only checked their arguments and returned success without moving any data, and collectives on a communicator with 1 rank also did nothing.
Version support¶
| Version | Status |
|---|---|
| 0.0.6 | Current |
| 0.0.5 | Previous |
| 0.0.4 | Previous |
| 0.0.3 | Previous |
lccl is in beta. The API follows NCCL, but the supported scope is still expanding, and public functions may be added or removed between releases (the removal of LcclCommRingGranularity in 0.0.6 is an example). Error code values have never changed and have not been reused with a different meaning, but a release can add new codes.