Skip to content

Support matrix

These are the items verified on lccl 0.0.6. Anything not listed here has not been verified.

Legend — ✅ Supported · 🟡 Limited · ❌ Not supported · 📅 Planned

Check this page before you finalize your design

Support is uneven, and the gaps are large enough to change implementation choices. In particular, the LPU limits the data types of reductions, the size unit, and the number of ranks.


Execution environments

Environment Description
LPU execution Communicates through the card's device kernels. This is the path the lccl backend of torch.distributed uses

The standalone lccl package (import lccl) is released as an LPU hardware build (+hw). The packages to install are listed on the home page. torch_ha also includes LCCL, which is used through torch.distributed.


Operations

The unit of support is the (operation, reduction, data type) combination. A combination with no registered kernel is rejected with kLcclNotSupported at the point of the call (at LcclGroupEnd() if it is inside a group), and nothing is transferred.

Operation LPU execution
LcclAllReduce ✅ Sum, Prod, Avg × Float16, Bfloat16
LcclReduceScatter ✅ Sum × Float16, Bfloat16
LcclAllGather ✅ All types
LcclBroadcast ✅ All types
LcclSend / LcclRecv ✅ All types
LcclAllToAll ✅ All types
LcclReduce ✅ Sum, Prod, Avg × Float16, Bfloat16
LcclGather ✅ All types
LcclScatter ✅ All types

All types — Operations without a reduction move data byte by byte regardless of the data type, so every LcclDataType works. Operations with a reduction are limited to the types listed in the table.

Some operations are not available through torch.distributed

On the LPU, the lccl backend of torch.distributed blocks gather, scatter, reduce, and all_to_all up front by raising an [HA_LCCL_NO_HW_KERNEL] exception. The library's LcclGather, LcclScatter, LcclReduce, and LcclAllToAll run on the LPU.

LPU reductions work only with Float16 and Bfloat16

AllReduce, ReduceScatter, and Reduce on Float32 or any other type are rejected with kLcclNotSupported at the point of the call (at LcclGroupEnd() if it is inside a group). A blocking communicator remains usable; for a nonblocking communicator, see Error handling.

Communicators with 1 rank

With only one rank, a collective skips the kernel and finishes with a single copy. As a result, any type and reduction (including Max and Min) passes. A passing 1-rank test does not tell you whether the operation works with multiple ranks.

Reductions

Reduction LPU execution
Sum LcclAllReduce, LcclReduceScatter, LcclReduce
Prod LcclAllReduce, LcclReduce
Avg LcclAllReduce, LcclReduce
Max, Min ❌

Max and Min are in LcclRedOp so that the interface matches NCCL's, but no operation has a kernel for them. Using them on a communicator with 2 or more ranks returns kLcclNotSupported.

Avg is for Float16 and Bfloat16 only. Avg on other types is rejected with kLcclNotSupported at call time.

Data types

Data type LPU execution
Float16 / Half ✅ Reduction, copy
Bfloat16 ✅ Reduction, copy
Float32 / Float 🟡 Copy only. Reductions return kLcclNotSupported
Float64 / Double 🟡 Copy only
Int8 / Char, Uint8 🟡 Copy only
Int32 / Int, Uint32 🟡 Copy only
Int64, Uint64 🟡 Copy only

Copy — Operations without a reduction, such as LcclAllGather, LcclBroadcast, LcclSend / LcclRecv, LcclAllToAll, LcclGather, and LcclScatter.

Counts and sizes

On the LPU, with 2 or more ranks, some operations require sizes to be aligned to a 512-byte unit. A size that does not fit this unit is rejected, depending on the operation.

Operation Rejected when Timing and return value Communicator after rejection
LcclAllReduce None. Accepts any count — —
LcclAllGather None. However, if the byte size of count is not a multiple of 512, the data is gathered through an extra copy, so the call is rejected inside a group, on a nonblocking communicator, and in an in-place call At call time. kLcclInvalidArgument Normal
LcclReduceScatter With 2 or more ranks, the byte size of recv_count is not a multiple of 512 At call time. kLcclInvalidArgument Normal
LcclAllToAll The per-rank piece (count × type size) is not a multiple of 512 At call time. kLcclInvalidArgument Normal
LcclBroadcast, LcclReduce, LcclSend / LcclRecv The byte size is not a multiple of 512 At call time (at LcclGroupEnd() inside a group). kLcclNotSupported Normal for a blocking communicator
LcclGather, LcclScatter The byte size of the per-rank piece is not a multiple of 512 At call time (at LcclGroupEnd() inside a group). kLcclNotSupported Normal for a blocking communicator

The size rules apply the same way on every rank, so when all ranks call with the same size, they are rejected together and no single side is left hanging. LcclSend/LcclRecv inside a group, however, are judged on each rank by that rank's own sends and receives only, so even when one rank gets kLcclNotSupported from LcclGroupEnd(), the other ranks can run their groups and fail with kLcclKernelSpinTimeout. In that case, re-create the communicator on all ranks.

On the LPU, the size one call handles also has an upper limit. LcclAllReduce and LcclReduceScatter are rejected with kLcclNotSupported above 4 GiB in total, and the other operations above 8 GiB.

LcclAllGather that is not on a 512-byte boundary is slow

It gathers at the rounded-up size and then goes through an extra step that copies the data back into recv_buff. On the LPU, this step is a round-trip copy through the host, so it is especially slow. If performance matters, align the size to a multiple of 512 bytes.

How far the send buffer is read

If count falls across a 512-byte boundary, the kernel reads the send buffer up to that boundary. A device allocation interleaved across DRAM channels always extends to that boundary, so buffers allocated on this premise have no problem. The receive buffer is written only up to the result size.

The torch.distributed path is subject to this rule as well: broadcast, send/recv, and reduce_scatter raise RuntimeError on any number of ranks unless the tensor's byte size (the output tensor's for reduce_scatter) is a multiple of 512. Pad to the element multiple named in the message and slice the result.

Algorithms and protocols

LPU execution Notes
Ring algorithm ✅ The only algorithm
Tree algorithm ❌
Simple protocol ✅ Default
Low-latency (LL) protocol ❌

LPU execution has no LL kernel. If you allow only LL, as in LCCL_PROTO=LL, collectives have no protocol to use and fail with kLcclEnqueueNoAlgo. For the variables, see Environment variables.

Transport layers

At connection time, the library tries the transports in order and uses the first one that is possible. The order is P2P, shared memory, network.

Transport Scope Status Notes
P2P (direct device-to-device DMA) Between ranks on the same server and in the same process 🟡 Requires peer mapping by the driver. If each rank uses its own process (torchrun, multiprocessing), it is not selected and falls back to shared memory
Shared memory Same server ✅ Requires /dev/shm capacity
Network — socket (TCP) Between servers ✅ Choose the interface to use with LCCL_SOCKET_IFNAME
Network — ibverbs (InfiniBand, RoCE) Between servers 🟡 Tried before sockets. When the driver, the RNIC, and the PCIe layout meet the conditions, it performs RDMA on LPU memory directly; otherwise it goes through host memory (Multi-node execution). If libibverbs is missing at runtime, it silently falls back to sockets

Scale

Item Limit or verified range
Ranks per server 1 per card. Placing two ranks on the same card of the same server results in kLcclCommError
/dev/shm per server When each rank uses its own process, the size needed grows with the number of ranks on that server and can exceed a container's default. See Environment variables
torch.distributed (lccl backend) Supported within one server only. In each process group (including groups made with dist.new_group), rank r of the group must use lpu:<r>
Ranks per communicator LPU: up to 16. Beyond that, communicator creation fails with kLcclNotSupported
Communicators per process LPU: the communicators with 2 or more ranks that one process holds on one LPU at the same time can have at most 511 ranks in total (31 communicators of 16 ranks, or 63 of 8). Beyond that, communicator creation fails with kLcclTransportShmUnsupported, kLcclTransportNetUnsupported, or kLcclNotSupported, so destroy communicators you no longer use
Number of channels LPU (2 or more ranks): 1
Verified LPU configurations 2 and 4 LPUs in one server. 2 servers with 1 card each (TCP sockets, including containers)
Verified ibverbs configuration 2 RNICs connected by a cable within one server

Larger configurations have not been verified. See Known limitations.

Platform

Item Status
Linux, x86_64 ✅
Python 3.10 ✅ The SDK wheels are built for 3.10
PyTorch 2.10.0 Needed when you use torch.distributed. It must be exactly this version, so when you install torch_ha, install torch==2.10.0 with it. lccl alone does not need it
Runtime (hart) 0.0.6 ✅ Must be the same version. hart is also a +hw build. If hart cannot read the format of the kernel library in lccl, communicator creation fails with kLcclNotSupported
LPU ✅ EVT0. On other chips, collectives return kLcclNotSupported
Windows, macOS ❌

API surface

Feature Status Notes
LcclGetUniqueId, LcclCommInitRank ✅
LcclCommInitRankConfig ✅ The config must be initialized with LCCL_CONFIG_INITIALIZER
LcclCommInitAll 🟡 Single process, single node. Bind a device to the thread before the call. On the LPU, the device-side resources of all ranks are placed on that device
LcclCommDestroy, LcclCommAbort, LcclCommFinalize ✅
LcclCommCount, LcclCommUserRank, LcclCommHaDevice ✅
LcclCommGetAsyncError ✅ An error, once recorded, is not cleared
LcclGroupStart / LcclGroupEnd, including nesting ✅
LcclCommRegister / LcclCommDeregister 🟡 The calls succeed but currently have no effect. The bundled network transports (socket, ibverbs) do not use buffer registration, and it is not used in LPU execution either
Asynchronous (stream-ordered) collectives ❌ The operation has already finished when the call returns. Collective operations
User-defined reduction operators ❌
Graph capture ❌
Communicator variants such as Split and Shrink ❌

For the scope that the Python bindings expose, see Python API. Abort, finalize, asynchronous error query, config, and registration are not available in Python.


Known limitations

  • LPU reductions work only with Float16 and Bfloat16. Other types are rejected with kLcclNotSupported.
  • There is no kernel for Max and Min.
  • The only algorithm is ring, and the only protocol on the LPU is Simple. There is no Tree.
  • Direct RDMA to LPU memory needs the conditions to be met. If any of the driver, the RNIC, or the PCIe layout does not meet them, traffic between servers goes through host memory.
  • An LPU communicator has at most 16 ranks and 1 channel.
  • On the LPU, one device kernel run must finish within LCCL_KERNEL_TIMEOUT_MS (5 seconds by default). The time also includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation. Beyond it, the communicator is aborted. LcclAllGather, LcclBroadcast, LcclReduce, LcclGather, and LcclScatter handle a whole call in one run, so split large data across several calls.
  • Do not run operations of two communicators that use the same LPU at the same time.
  • Collectives do not establish ordering with streams. The operation has finished when the call returns, but it is not guaranteed to wait for work queued before the call. Synchronize the producer first. The torch.distributed backend synchronizes the calling thread's default stream before the call, but because it is a temporary implementation, it does not cover work queued on other threads or streams.
  • There is no per-operation recovery. A transport-path failure or a watchdog timeout aborts the communicator, and you must re-create it. See Error handling.
  • Some configurations have not been verified. Neither the behavior nor the performance has been verified for more than 4 LPUs in one server, 3 or more servers, configurations between servers with multiple cards per server, configurations in which one process creates multiple communicators over the same set of ranks, and real server-to-server ibverbs connections.

See also