Legato Collective Communication Library¶
The Legato Collective Communication Library (lccl) provides collective and point-to-point communication between HyperAccel LPU devices. The API follows NCCL, so NCCL-based distributed code can be ported with minimal changes.
Position in the stack¶
lccl sits below the framework integration layer (torch_ha) and above the runtime (hart). It handles rank rendezvous, topology, and the transfers themselves, while device memory and kernel execution are the job of hart.
The usual way to use lccl on the LPU is the lccl backend of torch.distributed. torch_ha calls this library on your behalf, so you never need to call the lccl API directly. This documentation covers the library underneath, and is intended for people who build a backend or integrate the library into a framework.
At a glance¶
| Package | lccl (+hw build) |
| Version | 0.0.6 |
| Language | C++ core, Python bindings |
| Public header | lccl.h (shipped in the lccl package) |
| API model | NCCL-compatible naming and semantics |
| Transport layers | P2P (within the same process), shared memory, TCP sockets, ibverbs (InfiniBand, RoCE) |
| Python | 3.10 |
| Platform | Linux x86_64 |
| Status | Beta |
Documentation¶
Programming guide
How ranks find each other, what each collective operation guarantees, and how to write correct distributed code.
Overview · Communicators · Collective operations · Groups · Point-to-point · Error handling · Troubleshooting · Multi-node execution
API reference
The exact signature and meaning of every entry point.
C++ API · Python API · Environment variables · Error codes · Support matrix
Release notes
Changes and known issues for each version.
Getting started¶
On the LPU, through torch.distributed. On one server, one process per rank picks its own card and then creates a process group with the lccl backend.
1 2 3 4 5 6 7 8 | |
How to launch and the parallelization patterns are covered in Distributed.
Using lccl directly. The complete sequence of creating and distributing a unique ID, picking a device for each rank, and then creating a communicator is in First program in the programming guide.
What you must check before you design¶
lccl is in beta, and its coverage of supported features is uneven. The following points change design decisions.
Not every operation × reduction × data type combination is implemented
A combination with no registered kernel is usually rejected at call time with kLcclNotSupported. It neither silently returns success nor silently produces wrong data. It does mean, however, that code that worked with NCCL can fail here. Before finalizing the design, check the combinations your workload actually uses against the Support matrix.
The LPU has limits
On the LPU, every collective and LcclSend/LcclRecv run as device kernels. However, reductions work only with Float16 and Bfloat16, and LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv need byte sizes that are multiples of 512. A communicator has at most 16 ranks. Calls outside these limits are rejected with kLcclNotSupported. The byte sizes of count in LcclAllToAll and of recv_count in LcclReduceScatter must also be multiples of 512, and other sizes are rejected with kLcclInvalidArgument. The size rules for each operation are in the Support matrix.
Collective operations are not ordered with respect to streams
A collective operation called outside a group has already completed when the call returns. On the other hand, it is not guaranteed to wait for work queued on a stream before the call, so synchronize the work that produced the send buffer yourself before calling. See Collective operations.
Installation¶
For the driver installation and the SDK container image, see the SDK installation guide. The SDK container image, however, does not include the lccl package, so to call the lccl API directly (import lccl, lccl.h) in that image, install it as the table below shows.
Install lccl into the same Python environment as hart of the same version, and pin both packages to an exact version with ===. PyPI has an unrelated package that is also named hart, so if you do not pin the version of hart, that package can be installed instead.
| Execution environment | Packages to install |
|---|---|
| LPU | lccl===<version>+hw, hart===<version>+hw |
halogger, which sets the log level and the log file from Python code, is a separate package, so if you installed only lccl and hart, also install halogger===<version>. halogger has no +hw build, so do not add +hw to it.
Compatibility¶
This documentation is based on lccl 0.0.6.
| Verified version | |
|---|---|
| Python | 3.10 |
Runtime (hart) |
0.0.6 |
| PyTorch | 2.10.0 |
For details, see the Support matrix.
Third-party licenses¶
lccl is built on code from NVIDIA NCCL, carries code from rdma-core (libibverbs) in its ibverbs transport, and its Python bindings use pybind11. The license notices of the three components follow. The NCCL and rdma-core notices also apply to the LCCL included in torch_ha. The same files are also in dist-info/licenses/ of the lccl package.
NCCL (BSD-3-Clause)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | |
rdma-core (OpenIB.org BSD)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | |
pybind11 (BSD-3-Clause)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | |