Skip to content

1. Overview

This guide covers how to program the Legato Collective Communication Library (lccl) directly. It is written for integrators who implement distributed inference on LPU devices.

On the LPU, you usually use torch.distributed

torch_ha calls this library through the lccl backend of torch.distributed. If you use that path, you never call the APIs in this guide directly. See Distributed.

The standalone lccl package (import lccl) is released as the +hw build for the LPU (Installation).

Read this chapter first. The later chapters assume the concepts defined here.


1.1 What the library does

In distributed inference, all devices must combine or exchange data with each other at fixed points in the model. lccl provides those operations.

  • Rendezvous — ranks discover each other and form a communicator.
  • Collective operations — all-reduce, all-gather, reduce, reduce-scatter, broadcast, gather, scatter, all-to-all.
  • Point-to-point — directed send and receive between two ranks.
  • Groups — several operations are collected and executed when the group closes.
  • Topology — the server's hardware layout is detected and the communication paths are decided from it. It can also be specified directly with a file.

1.2 Design rationale

The API follows NCCL directly. Names, argument order, and semantics are close enough that existing distributed code can be ported by changing only the prefix. This is a deliberate constraint, not an accident. Had a new API been created, every framework integration would have had to be written twice.

The communicator is the unit of everything. The set of ranks and all the resources needed for communication are bound to the communicator. Operations take a communicator, not a list of ranks, and every rank in it must call the same operations in the same order.

Support coverage is uneven. Not every combination of operation × reduction × data type has a kernel. The library does not approximate, and it rejects an unregistered combination at the call with kLcclNotSupported. The LPU also limits reduction types, the size unit, and the number of ranks, so check the supported range beforehand. Designing with this in mind is, for now, a real part of using this library. See the Support matrix.

1.3 Key concepts

Concept Meaning
Rank One participant. It is identified by an integer in the range [0, nranks), and one rank drives one device. Claiming the same card from two ranks on the same server is rejected.
Communicator (LcclComm) The handle that every operation takes. It binds together the set of ranks and the communication resources.
Unique ID (LcclUniqueId) A 128-byte handle that one rank creates and distributes to the other ranks out of band.
Root In operations that have a root, such as broadcast, reduce, gather, and scatter, the rank that sends or receives the data.
Channel The unit of a path that processes transfers in parallel. On the LPU, a communicator with 2 or more ranks uses 1 channel, and in Python you can check the count with comm.n_channels.
Group The span between LcclGroupStart / LcclGroupEnd. The operations inside are collected and executed when the span closes.
Collective operation An operation in which every rank of the communicator takes part.

Every rank must call the same operations in the same order

Collective operations are matched by order, not by name or tag. If rank 0 calls all-reduce and then all-gather while rank 1 calls all-gather and then all-reduce, the two are mismatched. The result is not an error return but a hang (until the watchdog cuts it off) or silently wrong data. This is the most common cause of distributed bugs, and the library cannot catch it for you.

1.4 First program

One process per rank. The process that created the unique ID must stay alive while it waits for the ranks to connect, and the other ranks can take part only once they have received that value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import multiprocessing as mp

import hart
import lccl

NRANKS = 2
COUNT = 1024                                   # number of elements
DT = lccl.LcclDataType.LcclFloat16             # LPU reductions support only Float16 and Bfloat16


def worker(rank, id_bytes):
    hart.set_device(rank)                      # select the device before creating the communicator
    comm = lccl.comm_init_rank(NRANKS, lccl.LcclUniqueId(id_bytes), rank)
    try:
        nbytes = COUNT * 2                     # Float16 is 2 bytes per element
        send = hart.mem_alloc(nbytes, is_interleaved=True)
        recv = hart.mem_alloc(nbytes, is_interleaved=True)
        # For how to fill send, see the memory model of hart
        lccl.all_reduce(send, recv, COUNT, DT, lccl.LcclRedOp.LcclSum, comm)
        hart.mem_free(send)
        hart.mem_free(recv)
    finally:
        lccl.comm_destroy(comm)


if __name__ == "__main__":
    # The process that created the unique ID must stay alive until the ranks have connected.
    unique_id = lccl.get_unique_id()
    id_bytes = bytes(unique_id.internal)       # 128 bytes. A form that can be sent to another process

    ctx = mp.get_context("spawn")              # fork is avoided in favor of spawn
    procs = [ctx.Process(target=worker, args=(r, id_bytes)) for r in range(NRANKS)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#include "lccl.h"
#include "ha_runtime_api.h"  // HaSetDevice, HaMemAlloc

// Run this in every process. local_dev is the device number this rank uses.
HaSetDevice(local_dev);                 // Select the device before creating the communicator

LcclUniqueId id;
if (rank == 0) {
  LcclGetUniqueId(&id);                 // id is distributed to the other ranks
}
// ... the launcher passes id to all ranks ...

LcclComm comm;
LcclCommInitRank(&comm, nranks, id, rank);

HaDevicePtr send_da = 0, recv_da = 0;
HaMemAlloc(&send_da, count * 2, /*is_interleaved=*/true);
HaMemAlloc(&recv_da, count * 2, /*is_interleaved=*/true);

LcclAllReduce(reinterpret_cast<const void *>(send_da), reinterpret_cast<void *>(recv_da),
              count, kLcclFloat16, kLcclSum, comm);

HaMemFree(send_da);
HaMemFree(recv_da);
LcclCommDestroy(comm);

Four things recur throughout the API, so they are noted here in advance.

  1. Every C++ entry point returns LcclResult. There are no exceptions, and no result takes the place of the return value either. See Error handling.
  2. Buffers are device pointers. You obtain them from hart. lccl does not allocate the buffers you pass in. Use allocations that are interleaved across the DRAM channels.
  3. The unique ID must be delivered by other means. The library does not distribute it; that is the launcher's job.
  4. Select the device first. The device bound to the calling thread becomes that rank's device. When you later call or clean up that communicator from another thread, select the same device on that thread first.

1.5 Behaviors to know about

  • Creating a communicator waits until all ranks have gathered. This wait has no timeout, so if the ranks do not line up, it hangs without an error. Troubleshooting distinguishes the causes.
  • If a rank does not show up during an operation, the watchdog (LCCL_KERNEL_TIMEOUT_MS, default 5 seconds) cuts it off and the call returns kLcclKernelSpinTimeout. That communicator can no longer be used.
  • An unsupported combination shows up at the call to that operation, not when the communicator is created.
  • Creating a communicator prepares all connections between ranks in advance, so as the number of ranks grows, the creation time and the shared memory (/dev/shm) usage grow quickly. When ranks on the same server run in separate processes, they communicate through /dev/shm. The usage per server grows with the square of the number of ranks on that server; with the default LCCL_BUFFSIZE, it is about 22 MiB for 2 ranks, about 80 MiB for 4 ranks, and about 300 MiB for 8 ranks. The container default /dev/shm (64 MiB) holds at most 3 ranks.

Next steps