Skip to content

Release notes

Changes to the Legato Collective Communication Library (lccl). lccl is released with the same version as the other SDK components. For changes to the whole stack, see SDK release notes.


0.0.6

Requires Python 3.10 and Runtime (hart) 0.0.6.

Major changes

  • LcclAllReduce and LcclAllGather accept any count. The constraint that required count on the LPU to be aligned to a unit is removed. The receive buffer is written only up to the result size. If count is not on a 512-byte boundary, LcclAllGather gathers through an extra copy, so it rejects the call with kLcclInvalidArgument inside a group, on a nonblocking communicator, and in an in-place call.
  • The C++ public header becomes the single lccl.h and ships in the lccl package. Include lccl/include/lccl.h of the site-packages where lccl is installed, and add include/ of the hart package to the include path as well. collectives.h and group.h are internal headers. For how to build and link, see C++ API.
  • An lccl package for the LPU. lccl is released as the +hw version. Install hart as the +hw build of the same version too. The packages to install are listed on the home page.
  • The kernels are inside libha_lccl.so. Every operation finds its kernels even when the library is installed from a package, and there are no separate kernel files. In 0.0.5, with 2 or more ranks, LcclAllReduce, LcclReduceScatter, and LcclBroadcast looked for the kernel files at a path on the machine that built them, so torch_ha installed from a package could not launch their kernels.
  • LcclReduce, LcclGather, and LcclScatter run on the LPU. LcclReduce takes Sum, Prod, and Avg on Float16 and Bfloat16. In 0.0.5, these three operations had no device kernel. The torch.distributed backend still blocks gather, scatter, reduce, and all_to_all on the LPU.
  • LPU collectives run in one device kernel run. An LPU communicator with 2 or more ranks finishes collectives and LcclSend/LcclRecv inside one launched kernel instead of launching a kernel for each step. With fewer kernel launches, operations on small sizes are faster.
  • LPU Direct RDMA between servers. On the ibverbs transport, when the driver (2.6.0 or later), the RNIC, and the PCIe layout meet the conditions, the RNIC reads and writes LPU memory directly without moving the data through host memory. Connections that do not meet the conditions go through host memory as before. It is controlled with LCCL_NET_RDMA_LEVEL, LCCL_NET_RDMA_READ, and LCCL_GDR_FLUSH_DISABLE. A process that uses LPU Direct RDMA allocates 1 GiB of LPU memory per LPU by default and keeps it until the process exits. See Multi-node execution.

Changes

  • Removed: LcclCommRingGranularity, LcclCommChannelGranularity, and the torch_ha operators that wrapped them, torch.ops.lpu.lccl_ring_granularity and torch.ops.lpu.lccl_channel_granularity. Code that padded count or a tensor size to the unit they returned can drop that part. There is no replacement function or operator. For the operations whose byte size must be a multiple of 512, see Known issues below.
  • On the LPU, LcclReduceScatter with 2 or more ranks rejects the call with kLcclInvalidArgument before launching the kernel if the byte size of recv_count is not a multiple of 512. Previously, it failed during execution with kLcclKernelInvalidArgs.
  • import lccl no longer imports PyTorch. The extension module is lccl itself, and the lccl.lccl path is gone. The public names are unchanged.
  • lccl finds the hart installed in the same Python environment, so LD_LIBRARY_PATH is not needed. The way to link a C++ program has changed as well. See C++ API.
  • When you use lccl and torch_ha in one process, install hart, lccl, and torch_ha from the same release. See Python API.
  • A communicator cannot mix 0.0.6 ranks with 0.0.5 ranks. If you run across several servers, upgrade lccl (or torch_ha) to 0.0.6 on every server together.
  • The only dependency is now hart, so pytest and others are no longer installed with it. The test executables are removed from the package.
  • The license notices of NCCL, rdma-core, and pybind11 are included in dist-info/licenses/ of the package. They are also on the home page.
  • An LPU communicator has at most 16 ranks. Beyond that, LcclCommInit* fails with kLcclNotSupported.
  • On the LPU, a communicator with 2 or more ranks uses 1 channel. If LCCL_MAX_NCHANNELS or LCCL_MIN_NCHANNELS is set above 1, the library logs a warning and ignores it. The warning that was logged at every communicator creation when LCCL_MAX_NCHANNELS was not set is gone.
  • On the LPU, the default of LCCL_KERNEL_TIMEOUT_MS is 5000 (5 seconds). It applies to each device kernel run, and the time includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation. A value outside 1 to 9999 makes the library log a warning, because a stuck kernel that reaches the driver's 12-second limit leaves the card needing a reset.
  • On the LPU, a call that the kernels cannot handle is rejected with kLcclNotSupported at the point of the call (at LcclGroupEnd() inside a group). This covers reductions on types other than Float16 and Bfloat16; LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv whose byte size is not a multiple of 512; and calls larger than 4 GiB (LcclAllReduce, LcclReduceScatter) or 8 GiB (the other operations). In 0.0.5, Float32 reductions and LcclBroadcast and LcclSend/LcclRecv whose byte size was not a multiple of 512 failed during execution with kLcclKernelInvalidArgs and left the communicator unusable.
  • On LPUs other than EVT0, collectives return kLcclNotSupported.
  • On the LPU with 2 or more ranks, LCCL_BUFFSIZE must be a multiple of 4096. Otherwise every collective and LcclSend/LcclRecv returns kLcclNotSupported.
  • If hart cannot read the format of the kernel library in lccl, LcclCommInit* fails with kLcclNotSupported. Install lccl and hart from the same release.
  • New environment variables: LCCL_DEVICE_LOOP_LAUNCH_BYTES (the most bytes one device kernel run of LcclAllReduce, LcclReduceScatter, or LcclSend/LcclRecv handles, 256 MiB by default) and LCCL_DMABUF_POOL_SIZE_PER_CHANNEL (the per-DRAM-channel size of the buffers used for LPU Direct RDMA, 64 MiB by default). See Environment variables.
  • In the topology file (LCCL_PREDEFINED_SYSINFO), LPU and NIC can have linkspeed as well as PCI. If it is missing or null, 12 GB/s is used instead of the parent's value. Numeric fields take JSON numbers only, and a file with a broken structure is not read partially but is treated as an initialization error. In 0.0.5, initialization did not finish in some of those cases.
  • In the topology file, a PCI, LPU, or NIC entry whose busId is missing or not written as a PCI address makes initialization fail with kLcclTopoConfigInvalid.

Fixes

  • On an LPU communicator with 1 rank, an LcclAllReduce, LcclAllGather, LcclReduceScatter, LcclBroadcast, or LcclReduce that was not in place filled the result only in the first DRAM channel or failed with Bad address. Fixed.
  • A large LcclBroadcast between LPU ranks connected through shared memory occasionally hung or received wrong values. Fixed.
  • If the topology file had a CPU with socketId -1, runs between servers could not find the NIC. Fixed.
  • With a topology file, network devices without a PCI address (veth, bond, etc.) were attached to the first CPU in the file. They are now attached to the CPU of the card used by the lowest rank on this server, the same as when LCCL discovers the hardware itself.

Known issues

Issue Impact Workaround
LPU reductions are Float16 and Bfloat16 only Reductions on other types return kLcclNotSupported Change the data type
On the LPU, the byte size of LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv must be a multiple of 512 Otherwise kLcclNotSupported Align the byte size to a multiple of 512
No kernel for the Max and Min reductions kLcclNotSupported with 2 or more ranks Use Sum or reduce on the host
Running operations of two communicators on the same LPU at the same time is not supported — Run the operations of one communicator at a time on an LPU
On the LPU, the time the ranks take to enter an operation counts toward LCCL_KERNEL_TIMEOUT_MS (5 seconds by default) If the ranks are too far apart, the call fails with kLcclKernelSpinTimeout and the communicator must be re-created Have the ranks call the operation at about the same time. If needed, raise the value, up to 9999
Collectives do not establish ordering with streams Work queued before the call is not guaranteed to finish first Synchronize yourself before the call
P2P only within the same process No P2P benefit when each rank uses its own process —
No Python bindings for LcclCommGetAsyncError and LcclCommAbort A Python process cannot check for asynchronous failures or abort Drive it from C++ if it matters
Release builds do not print logs at INFO or below You cannot check the selected interface, the number of channels, or whether LPU Direct RDMA is used from the logs Read the comm.n_nodes and comm.n_channels attributes. To check whether LPU Direct RDMA is used, read the driver's dmabuf_exports node (Multi-node execution)
A transport-path failure or a watchdog timeout requires re-creating the communicator No per-operation recovery Clean up on all ranks and re-create

0.0.5

2026-10-14. Requires Python 3.10 and Runtime (hart) 0.0.5.

Major changes

  • P2P transport between ranks in the same process. If the driver allows memory mapping between two cards, data movement for LcclAllGather, LcclAllReduce, and LcclReduceScatter goes over direct DMA between the cards instead of through host shared memory. In configurations where each rank uses its own process, it is not selected and shared memory is used.
  • ibverbs network transport (InfiniBand, RoCE). For communication between servers, ibverbs is used before sockets if an RDMA device is present. Data goes through host memory. LCCL_IB_* variables for configuration were added. See Environment variables.
  • Bfloat16 reduction. LcclAllReduce (Sum, Prod, Avg) and LcclReduceScatter (Sum) accept Bfloat16.
  • LcclBroadcast and LcclAllToAll execute on the LPU. Previously, they malfunctioned on the LPU. This has been verified on 2 and 4 LPUs in one server.
  • Buffer registration API. LcclCommRegister, LcclCommDeregister, and LCCL_LOCAL_REGISTER. This API lets LcclSend/LcclRecv on a network transport use registered buffers directly, but the bundled network transports (socket, ibverbs) do not use it, so for now the calls succeed but have no effect.
  • Fixed-topology JSON contract. Added schemaVersion to the LCCL_PREDEFINED_SYSINFO file and added the strict mode LCCL_TOPO_STRICT. See Multi-node execution.

New

  • 6 error codes: kLcclTopoSchemaVersionMismatch, kLcclTopoPeerMismatch, kLcclNetRemoteError, kLcclTransportP2pChannelMismatch, kLcclTransportBuffSizeMismatch, kLcclTransportShmLocalityMismatch. The values of existing codes did not change.
  • LcclCommChannelGranularity. (Removed in 0.0.6; see 0.0.6.)

Changes

  • Rank, device number, and ring are no longer written to or read from the topology file. The file describes only the hardware layout, and LCCL fills in the rest at runtime.
  • A topology file in which an LPU and an RNIC are under the same PCIe switch is accepted.

Known issues

Issue Impact Workaround
On the LPU, count of LcclAllReduce must be a multiple of 512 bytes × number of channels × number of ranks, and count of LcclAllGather a multiple of 512 bytes × number of channels If violated, kLcclKernelInvalidArgs during execution. The unit differs from server to server Query the unit with LcclCommRingGranularity and LcclCommChannelGranularity and align to it. Resolved in 0.0.6
On the LPU, recv_count of LcclReduceScatter does not match the unit Not rejected at call time; fails during execution with kLcclKernelInvalidArgs, and the communicator can no longer be used Align recv_count to the unit. Rejected at call time with kLcclInvalidArgument in 0.0.6
On the LPU, the byte count of LcclBroadcast or LcclSend/LcclRecv is not a multiple of 512 Rejected during execution with kLcclKernelInvalidArgs, and the communicator can no longer be used Align the byte count to a multiple of 512
On the LPU, LcclReduce, LcclGather, and LcclScatter have no device kernel Not rejected; access error or hang Do not call them. Replace them with LcclAllReduce, LcclAllGather, etc.
LPU reductions support only Float16 and Bfloat16 Float32 reductions fail with kLcclKernelInvalidArgs during execution Change the data type
No kernel for Max and Min reductions kLcclNotSupported with 2 or more ranks Use Sum, or reduce on the host
Collectives do not establish ordering with streams; no asynchronous execution Does not wait for work queued before the call Synchronize yourself before the call
P2P works only within the same process; ibverbs goes through host memory No P2P benefit if each rank uses its own process. No direct RDMA of LPU memory between servers —
ibverbs has been verified only with 2 RNICs connected within one server Behavior and performance of ibverbs configurations between servers have not been verified Use sockets with LCCL_IB_DISABLE=1
No Python bindings for LcclCommGetAsyncError and LcclCommAbort A Python process cannot check for asynchronous failures or abort If this matters, run it from C++
Release builds do not print logs at INFO level or below Cannot check the selected interface, number of channels, etc. in the logs Read the comm.n_nodes and comm.n_channels attributes
A transport-path failure or watchdog timeout requires re-creating the communicator No per-operation recovery Clean up on all ranks, then re-create

0.0.4

2026-08-28. Requires Python 3.10 and Runtime (hart) 0.0.4.

Major changes

  • More operations run as device kernels on the LPU. LcclAllReduce, LcclAllGather, and LcclReduceScatter run as device kernels on 2 LPUs, and LcclSend/LcclRecv also run on the LPU.
  • TCP socket transport between servers. Ranks on different servers perform collectives over the network. This has been verified on 2 servers.
  • Unsupported combinations are rejected. Fixed the problem where a combination with no registered kernel silently returned success; it now fails with kLcclNotSupported.
  • Hang-prevention watchdog. If a kernel hangs because one rank does not arrive, the watchdog aborts the kernel after LCCL_KERNEL_TIMEOUT_MS (30 seconds by default) and returns kLcclKernelSpinTimeout. The aborted kernel returns kLcclKernelAborted.

New

  • Prod and Avg reductions (LcclAllReduce).
  • Profiler integration. Collectives and point-to-point operations are recorded as profiler ranges lccl:<operation>, with rank and worker thread labels attached.
  • Python accepts the integer device address returned by hart.mem_alloc() as a buffer.
  • Collectives on a communicator with 1 rank.
  • 5 error codes: kLcclTransportShmChannelGeometryUnknown, kLcclTransportNetDestroyFail, kLcclTransportNetUnsupported, kLcclTransportShmUnsupported, kLcclKernelAborted.
  • LcclCommRingGranularity. (Still present in 0.0.5; removed in 0.0.6; see 0.0.6.)

Changes

  • LcclAllToAll rejects overlapping send and receive buffers.
  • Collectives without a reduction (broadcast, all-gather, gather, scatter) move data byte by byte, so they accept every LcclDataType.
  • Group nesting tracks the depth per thread, and the work runs when the outermost region closes.

Known issues

Issue Impact Workaround
LcclBroadcast and LcclAllToAll do not run on the LPU Calling them on the LPU malfunctions Use 0.0.5 or later
No Bfloat16 reduction Bfloat16 reductions are rejected Use 0.0.5 or later, or Float16
No kernel for Max and Min reductions kLcclNotSupported Use Sum, or reduce on the host
The P2P and ibverbs transports are stubs Shared memory on the same server, TCP sockets between servers Use 0.0.5 or later
No buffer registration entry point No fast path for registered buffers —
No Python binding for LcclCommGetAsyncError A Python process cannot check for asynchronous failures If this matters, run it from C++

0.0.3

2026-06-18. The first release of lccl. It provides rank rendezvous, topology and ring discovery, and 6 collectives (LcclAllReduce, LcclReduce, LcclReduceScatter, LcclBroadcast, LcclGather, LcclScatter). LcclAllGather, LcclAllToAll, LcclSend, and LcclRecv were in the headers but only checked their arguments and returned success without moving any data, and collectives on a communicator with 1 rank also did nothing.


Version support

Version Status
0.0.6 Current
0.0.5 Previous
0.0.4 Previous
0.0.3 Previous

lccl is in beta. The API follows NCCL, but the supported scope is still expanding, and public functions may be added or removed between releases (the removal of LcclCommRingGranularity in 0.0.6 is an example). Error code values have never changed and have not been reused with a different meaning, but a release can add new codes.