Skip to content

Legato Collective Communication Library

The Legato Collective Communication Library (lccl) provides collective and point-to-point communication between HyperAccel LPU devices. The API follows NCCL, so NCCL-based distributed code can be ported with minimal changes.


Position in the stack

lccl sits below the framework integration layer (torch_ha) and above the runtime (hart). It handles rank rendezvous, topology, and the transfers themselves, while device memory and kernel execution are the job of hart.

Diagram

The usual way to use lccl on the LPU is the lccl backend of torch.distributed. torch_ha calls this library on your behalf, so you never need to call the lccl API directly. This documentation covers the library underneath, and is intended for people who build a backend or integrate the library into a framework.


At a glance

Package lccl (+hw build)
Version 0.0.6
Language C++ core, Python bindings
Public header lccl.h (shipped in the lccl package)
API model NCCL-compatible naming and semantics
Transport layers P2P (within the same process), shared memory, TCP sockets, ibverbs (InfiniBand, RoCE)
Python 3.10
Platform Linux x86_64
Status Beta

Documentation


Getting started

On the LPU, through torch.distributed. On one server, one process per rank picks its own card and then creates a process group with the lccl backend.

1
2
3
4
5
6
7
8
import torch
import torch.distributed as dist

torch.accelerator.set_device_index(local_rank)   # Pick the card before creating the process group
dist.init_process_group(backend="lccl")

x = torch.randn(32, 1024, dtype=torch.bfloat16).to("lpu")
dist.all_reduce(x, op=dist.ReduceOp.SUM)

How to launch and the parallelization patterns are covered in Distributed.

Using lccl directly. The complete sequence of creating and distributing a unique ID, picking a device for each rank, and then creating a communicator is in First program in the programming guide.


What you must check before you design

lccl is in beta, and its coverage of supported features is uneven. The following points change design decisions.

Not every operation × reduction × data type combination is implemented

A combination with no registered kernel is usually rejected at call time with kLcclNotSupported. It neither silently returns success nor silently produces wrong data. It does mean, however, that code that worked with NCCL can fail here. Before finalizing the design, check the combinations your workload actually uses against the Support matrix.

The LPU has limits

On the LPU, every collective and LcclSend/LcclRecv run as device kernels. However, reductions work only with Float16 and Bfloat16, and LcclBroadcast, LcclReduce, LcclGather, LcclScatter, and LcclSend/LcclRecv need byte sizes that are multiples of 512. A communicator has at most 16 ranks. Calls outside these limits are rejected with kLcclNotSupported. The byte sizes of count in LcclAllToAll and of recv_count in LcclReduceScatter must also be multiples of 512, and other sizes are rejected with kLcclInvalidArgument. The size rules for each operation are in the Support matrix.

Collective operations are not ordered with respect to streams

A collective operation called outside a group has already completed when the call returns. On the other hand, it is not guaranteed to wait for work queued on a stream before the call, so synchronize the work that produced the send buffer yourself before calling. See Collective operations.


Installation

For the driver installation and the SDK container image, see the SDK installation guide. The SDK container image, however, does not include the lccl package, so to call the lccl API directly (import lccl, lccl.h) in that image, install it as the table below shows.

Install lccl into the same Python environment as hart of the same version, and pin both packages to an exact version with ===. PyPI has an unrelated package that is also named hart, so if you do not pin the version of hart, that package can be installed instead.

Execution environment Packages to install
LPU lccl===<version>+hw, hart===<version>+hw

halogger, which sets the log level and the log file from Python code, is a separate package, so if you installed only lccl and hart, also install halogger===<version>. halogger has no +hw build, so do not add +hw to it.


Compatibility

This documentation is based on lccl 0.0.6.

Verified version
Python 3.10
Runtime (hart) 0.0.6
PyTorch 2.10.0

For details, see the Support matrix.


Third-party licenses

lccl is built on code from NVIDIA NCCL, carries code from rdma-core (libibverbs) in its ibverbs transport, and its Python bindings use pybind11. The license notices of the three components follow. The NCCL and rdma-core notices also apply to the LCCL included in torch_ha. The same files are also in dist-info/licenses/ of the lccl package.

NCCL (BSD-3-Clause)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 Copyright (c) 2015-2020, NVIDIA CORPORATION. All rights reserved.

 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions
 are met:
  * Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.
  * Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.
  * Neither the name of NVIDIA CORPORATION, Lawrence Berkeley National
    Laboratory, the U.S. Department of Energy, nor the names of their
    contributors may be used to endorse or promote products derived
    from this software without specific prior written permission.

 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
 EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
 OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

 The U.S. Department of Energy funded the development of this software
 under subcontract 7078610 with Lawrence Berkeley National Laboratory.


This code also includes files from the NVIDIA Tools Extension SDK project.

See:

   https://github.com/NVIDIA/NVTX

for more information and license details.
rdma-core (OpenIB.org BSD)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Copyright (c) 2004, 2005 Topspin Communications.  All rights reserved.
Copyright (c) 2004, 2011-2012 Intel Corporation.  All rights reserved.
Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc.  All rights reserved.
Copyright (c) 2005 PathScale, Inc.  All rights reserved.

This software is available to you under the OpenIB.org BSD license below:

    Redistribution and use in source and binary forms, with or
    without modification, are permitted provided that the following
    conditions are met:

     - Redistributions of source code must retain the above
       copyright notice, this list of conditions and the following
       disclaimer.

     - Redistributions in binary form must reproduce the above
       copyright notice, this list of conditions and the following
       disclaimer in the documentation and/or other materials
       provided with the distribution.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
pybind11 (BSD-3-Clause)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
Copyright (c) 2016 Wenzel Jakob <wenzel.jakob@epfl.ch>, All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
   list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors
   may be used to endorse or promote products derived from this software
   without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Please also refer to the file .github/CONTRIBUTING.md, which clarifies licensing of
external contributions to this project including patches, pull requests, etc.