Support matrix¶
These are the items verified on lccl 0.0.6. Anything not listed here has not been verified.
Legend — ✅ Supported · 🟡 Limited · ❌ Not supported · 📅 Planned
Check this page before you finalize your design
Support is uneven, and the gaps are large enough to change implementation choices. In particular, the LPU limits the data types of reductions, the size unit, and the number of ranks.
Execution environments¶
| Environment | Description |
|---|---|
| LPU execution | Communicates through the card's device kernels. This is the path the lccl backend of torch.distributed uses |
The standalone lccl package (import lccl) is released as an LPU hardware build (+hw). The packages to install are listed on the home page. torch_ha also includes LCCL, which is used through torch.distributed.
Operations¶
The unit of support is the (operation, reduction, data type) combination. A combination with no registered kernel is rejected with kLcclNotSupported at the point of the call (at LcclGroupEnd() if it is inside a group), and nothing is transferred.
| Operation | LPU execution |
|---|---|
LcclAllReduce |
✅ Sum, Prod, Avg × Float16, Bfloat16 |
LcclReduceScatter |
✅ Sum × Float16, Bfloat16 |
LcclAllGather |
✅ All types |
LcclBroadcast |
✅ All types |
LcclSend / LcclRecv |
✅ All types |
LcclAllToAll |
✅ All types |
LcclReduce |
✅ Sum, Prod, Avg × Float16, Bfloat16 |
LcclGather |
✅ All types |
LcclScatter |
✅ All types |
All types — Operations without a reduction move data byte by byte regardless of the data type, so every LcclDataType works. Operations with a reduction are limited to the types listed in the table.
Some operations are not available through torch.distributed
On the LPU, the lccl backend of torch.distributed blocks gather, scatter, reduce, and all_to_all up front by raising an [HA_LCCL_NO_HW_KERNEL] exception. The library's LcclGather, LcclScatter, LcclReduce, and LcclAllToAll run on the LPU.
LPU reductions work only with Float16 and Bfloat16
AllReduce, ReduceScatter, and Reduce on Float32 or any other type are rejected with kLcclNotSupported at the point of the call (at LcclGroupEnd() if it is inside a group). A blocking communicator remains usable; for a nonblocking communicator, see Error handling.
Communicators with 1 rank
With only one rank, a collective skips the kernel and finishes with a single copy. As a result, any type and reduction (including Max and Min) passes. A passing 1-rank test does not tell you whether the operation works with multiple ranks.
Reductions¶
| Reduction | LPU execution |
|---|---|
Sum |
LcclAllReduce, LcclReduceScatter, LcclReduce |
Prod |
LcclAllReduce, LcclReduce |
Avg |
LcclAllReduce, LcclReduce |
Max, Min |
❌ |
Max and Min are in LcclRedOp so that the interface matches NCCL's, but no operation has a kernel for them. Using them on a communicator with 2 or more ranks returns kLcclNotSupported.
Avg is for Float16 and Bfloat16 only. Avg on other types is rejected with kLcclNotSupported at call time.
Data types¶
| Data type | LPU execution |
|---|---|
Float16 / Half |
✅ Reduction, copy |
Bfloat16 |
✅ Reduction, copy |
Float32 / Float |
🟡 Copy only. Reductions return kLcclNotSupported |
Float64 / Double |
🟡 Copy only |
Int8 / Char, Uint8 |
🟡 Copy only |
Int32 / Int, Uint32 |
🟡 Copy only |
Int64, Uint64 |
🟡 Copy only |
Copy — Operations without a reduction, such as LcclAllGather, LcclBroadcast, LcclSend / LcclRecv, LcclAllToAll, LcclGather, and LcclScatter.
Counts and sizes¶
On the LPU, with 2 or more ranks, some operations require sizes to be aligned to a 512-byte unit. A size that does not fit this unit is rejected, depending on the operation.
| Operation | Rejected when | Timing and return value | Communicator after rejection |
|---|---|---|---|
LcclAllReduce |
None. Accepts any count |
— | — |
LcclAllGather |
None. However, if the byte size of count is not a multiple of 512, the data is gathered through an extra copy, so the call is rejected inside a group, on a nonblocking communicator, and in an in-place call |
At call time. kLcclInvalidArgument |
Normal |
LcclReduceScatter |
With 2 or more ranks, the byte size of recv_count is not a multiple of 512 |
At call time. kLcclInvalidArgument |
Normal |
LcclAllToAll |
The per-rank piece (count × type size) is not a multiple of 512 |
At call time. kLcclInvalidArgument |
Normal |
LcclBroadcast, LcclReduce, LcclSend / LcclRecv |
The byte size is not a multiple of 512 | At call time (at LcclGroupEnd() inside a group). kLcclNotSupported |
Normal for a blocking communicator |
LcclGather, LcclScatter |
The byte size of the per-rank piece is not a multiple of 512 | At call time (at LcclGroupEnd() inside a group). kLcclNotSupported |
Normal for a blocking communicator |
The size rules apply the same way on every rank, so when all ranks call with the same size, they are rejected together and no single side is left hanging. LcclSend/LcclRecv inside a group, however, are judged on each rank by that rank's own sends and receives only, so even when one rank gets kLcclNotSupported from LcclGroupEnd(), the other ranks can run their groups and fail with kLcclKernelSpinTimeout. In that case, re-create the communicator on all ranks.
On the LPU, the size one call handles also has an upper limit. LcclAllReduce and LcclReduceScatter are rejected with kLcclNotSupported above 4 GiB in total, and the other operations above 8 GiB.
LcclAllGather that is not on a 512-byte boundary is slow
It gathers at the rounded-up size and then goes through an extra step that copies the data back into recv_buff. On the LPU, this step is a round-trip copy through the host, so it is especially slow. If performance matters, align the size to a multiple of 512 bytes.
How far the send buffer is read
If count falls across a 512-byte boundary, the kernel reads the send buffer up to that boundary. A device allocation interleaved across DRAM channels always extends to that boundary, so buffers allocated on this premise have no problem. The receive buffer is written only up to the result size.
The torch.distributed path is subject to this rule as well: broadcast, send/recv, and reduce_scatter raise RuntimeError on any number of ranks unless the tensor's byte size (the output tensor's for reduce_scatter) is a multiple of 512. Pad to the element multiple named in the message and slice the result.
Algorithms and protocols¶
| LPU execution | Notes | |
|---|---|---|
| Ring algorithm | ✅ | The only algorithm |
| Tree algorithm | ❌ | |
| Simple protocol | ✅ | Default |
| Low-latency (LL) protocol | ❌ |
LPU execution has no LL kernel. If you allow only LL, as in LCCL_PROTO=LL, collectives have no protocol to use and fail with kLcclEnqueueNoAlgo. For the variables, see Environment variables.
Transport layers¶
At connection time, the library tries the transports in order and uses the first one that is possible. The order is P2P, shared memory, network.
| Transport | Scope | Status | Notes |
|---|---|---|---|
| P2P (direct device-to-device DMA) | Between ranks on the same server and in the same process | 🟡 | Requires peer mapping by the driver. If each rank uses its own process (torchrun, multiprocessing), it is not selected and falls back to shared memory |
| Shared memory | Same server | ✅ | Requires /dev/shm capacity |
| Network — socket (TCP) | Between servers | ✅ | Choose the interface to use with LCCL_SOCKET_IFNAME |
| Network — ibverbs (InfiniBand, RoCE) | Between servers | 🟡 | Tried before sockets. When the driver, the RNIC, and the PCIe layout meet the conditions, it performs RDMA on LPU memory directly; otherwise it goes through host memory (Multi-node execution). If libibverbs is missing at runtime, it silently falls back to sockets |
Scale¶
| Item | Limit or verified range |
|---|---|
| Ranks per server | 1 per card. Placing two ranks on the same card of the same server results in kLcclCommError |
/dev/shm per server |
When each rank uses its own process, the size needed grows with the number of ranks on that server and can exceed a container's default. See Environment variables |
torch.distributed (lccl backend) |
Supported within one server only. In each process group (including groups made with dist.new_group), rank r of the group must use lpu:<r> |
| Ranks per communicator | LPU: up to 16. Beyond that, communicator creation fails with kLcclNotSupported |
| Communicators per process | LPU: the communicators with 2 or more ranks that one process holds on one LPU at the same time can have at most 511 ranks in total (31 communicators of 16 ranks, or 63 of 8). Beyond that, communicator creation fails with kLcclTransportShmUnsupported, kLcclTransportNetUnsupported, or kLcclNotSupported, so destroy communicators you no longer use |
| Number of channels | LPU (2 or more ranks): 1 |
| Verified LPU configurations | 2 and 4 LPUs in one server. 2 servers with 1 card each (TCP sockets, including containers) |
| Verified ibverbs configuration | 2 RNICs connected by a cable within one server |
Larger configurations have not been verified. See Known limitations.
Platform¶
| Item | Status |
|---|---|
| Linux, x86_64 | ✅ |
| Python 3.10 | ✅ The SDK wheels are built for 3.10 |
| PyTorch 2.10.0 | Needed when you use torch.distributed. It must be exactly this version, so when you install torch_ha, install torch==2.10.0 with it. lccl alone does not need it |
Runtime (hart) 0.0.6 |
✅ Must be the same version. hart is also a +hw build. If hart cannot read the format of the kernel library in lccl, communicator creation fails with kLcclNotSupported |
| LPU | ✅ EVT0. On other chips, collectives return kLcclNotSupported |
| Windows, macOS | ❌ |
API surface¶
| Feature | Status | Notes |
|---|---|---|
LcclGetUniqueId, LcclCommInitRank |
✅ | |
LcclCommInitRankConfig |
✅ | The config must be initialized with LCCL_CONFIG_INITIALIZER |
LcclCommInitAll |
🟡 | Single process, single node. Bind a device to the thread before the call. On the LPU, the device-side resources of all ranks are placed on that device |
LcclCommDestroy, LcclCommAbort, LcclCommFinalize |
✅ | |
LcclCommCount, LcclCommUserRank, LcclCommHaDevice |
✅ | |
LcclCommGetAsyncError |
✅ | An error, once recorded, is not cleared |
LcclGroupStart / LcclGroupEnd, including nesting |
✅ | |
LcclCommRegister / LcclCommDeregister |
🟡 | The calls succeed but currently have no effect. The bundled network transports (socket, ibverbs) do not use buffer registration, and it is not used in LPU execution either |
| Asynchronous (stream-ordered) collectives | ❌ | The operation has already finished when the call returns. Collective operations |
| User-defined reduction operators | ❌ | |
| Graph capture | ❌ | |
Communicator variants such as Split and Shrink |
❌ |
For the scope that the Python bindings expose, see Python API. Abort, finalize, asynchronous error query, config, and registration are not available in Python.
Known limitations¶
- LPU reductions work only with
Float16andBfloat16. Other types are rejected withkLcclNotSupported. - There is no kernel for
MaxandMin. - The only algorithm is ring, and the only protocol on the LPU is Simple. There is no Tree.
- Direct RDMA to LPU memory needs the conditions to be met. If any of the driver, the RNIC, or the PCIe layout does not meet them, traffic between servers goes through host memory.
- An LPU communicator has at most 16 ranks and 1 channel.
- On the LPU, one device kernel run must finish within
LCCL_KERNEL_TIMEOUT_MS(5 seconds by default). The time also includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation. Beyond it, the communicator is aborted.LcclAllGather,LcclBroadcast,LcclReduce,LcclGather, andLcclScatterhandle a whole call in one run, so split large data across several calls. - Do not run operations of two communicators that use the same LPU at the same time.
- Collectives do not establish ordering with streams. The operation has finished when the call returns, but it is not guaranteed to wait for work queued before the call. Synchronize the producer first. The
torch.distributedbackend synchronizes the calling thread's default stream before the call, but because it is a temporary implementation, it does not cover work queued on other threads or streams. - There is no per-operation recovery. A transport-path failure or a watchdog timeout aborts the communicator, and you must re-create it. See Error handling.
- Some configurations have not been verified. Neither the behavior nor the performance has been verified for more than 4 LPUs in one server, 3 or more servers, configurations between servers with multiple cards per server, configurations in which one process creates multiple communicators over the same set of ranks, and real server-to-server ibverbs connections.