Skip to content

2. Communicators

A communicator is the handle that every operation takes. This chapter covers how ranks find each other and create a communicator, and the communicator's lifetime.


2.1 Bootstrap sequence

Ranks cannot discover each other on their own. One rank creates a handle, the launcher distributes it, and the remaining ranks connect with it.

Diagram

Call LcclGetUniqueId() once, on one rank only. It returns a 128-byte handle, and the process that created this handle must stay alive until all ranks have finished connecting. All other ranks must receive exactly the same handle before they call LcclCommInitRank(). If there are multiple servers, the other servers must be able to connect over the network to the server that created the handle. See Multi-node execution.

Distributing the unique ID is your responsibility

The library does not distribute the handle. Use a mechanism the launcher already has (MPI broadcast, a shared file, an environment variable, a key-value store). In Python, extract the 128 bytes with bytes(unique_id.internal) and send them, then restore the handle on the receiving side with lccl.LcclUniqueId(data). The handle object itself cannot be sent with pickle.

Use a handle only once, to create one communicator. After all ranks have connected, the handle accepts no more connections, so connecting again with a handle that has already been used fails with kLcclSocketNotConnect. Create a new handle each time you create a new communicator.

Mismatched ranks hang without an error

There is no timeout while ranks wait to gather. In the cases below, LcclCommInitRank() does not return. The log is written in the process that created the handle.

Situation Log
Two ranks received different handles None. Ranks gather separately for each handle, so each handle waits for nranks ranks to gather
Fewer than nranks ranks gathered None
nranks differs between ranks Logs Bootstrap Root : mismatch in rank count from procs N : M (ERROR) and ends the connection wait
The same rank number connected twice Logs rank N of M ranks has already checked in (ERROR) and ends the connection wait

When the connection wait ends with an error, ranks that have not yet connected fail with a socket error, and ranks that have already connected keep waiting. You can check whether the process that created the handle is still waiting for connections by calling lccl.bootstrap_is_root_alive() in that process (1 means it is still waiting).

2.2 Creating a communicator

Each process creates a communicator with its own rank index. This is the only configuration that scales to multiple servers, and it also matches the usual deployment form of a serving process. torch.distributed uses this configuration as well.

1
2
hart.set_device(local_device)                       # first, select this rank's device
comm = lccl.comm_init_rank(nranks, unique_id, rank)

A single process creates all the communicators at once. There is no rendezvous between processes, so the unique ID is not handled separately.

1
comms = lccl.comm_init_all(ndev, devlist)

This is convenient for tests and for work within a single server. A device must be bound to the calling thread first (otherwise the call fails with kLcclMemDeviceAllocFail), and in Python len(devlist) must equal ndev. If the call fails, stop using LCCL in that process; fix the cause and start again in a new process. It does not scale across servers, and a configuration in which one process per server takes care of all the cards of that server is not supported.

When you drive these communicators from one thread, call each operation once per communicator between group_start() and group_end() (LcclGroupStart() and LcclGroupEnd() in C++; see Groups). If you call them one after another without a group, the first call waits for the remaining ranks until the watchdog cuts it off and fails with kLcclKernelSpinTimeout, and the communicators must be created again.

Select the device before creating the communicator

The device currently bound to the calling thread becomes this rank's device. Call hart.set_device() in Python or HaSetDevice() in C++ first. If you skip it, LcclCommInitRank() fails with Communicator unknown or unclassified error (kLcclCommError). It fails with the same code when two ranks on the same server claim the same card, and the log contains Duplicate LPU detected. On one server, a card can be claimed by only one rank.

In Python, use a separate process for each rank, and start the processes with spawn, not fork

The binding's blocking calls do not release the GIL. If you assign one rank per thread within a single process, one thread waiting for its peer prevents the other threads from running, and they deadlock. Also, calling fork on a process that has initialized the runtime is unsafe, because the child inherits the runtime's threads and locks. Start the processes in a new interpreter, as multiprocessing.get_context("spawn") does.

In C++, you can assign one rank per thread. Call LcclCommInitRank() after each thread has selected its own device with HaSetDevice(). If you call it one after another from a single thread, the first call waits for the remaining ranks and does not return, so use a thread or process per rank as above.

On the LPU, LcclCommInitAll uses the calling thread's card

LcclCommInitAll() allocates the device-side auxiliary resources of all ranks on the device bound to the calling thread. As a result, on LPU systems with multiple cards, one set of resources per rank ends up on a single card. In configurations where the resources must be on each rank's own card, use a thread or process per rank as above and use LcclCommInitRank().

Creating with explicit configuration

LcclCommInitRankConfig() takes an LcclConfig for callers that need to change the defaults. It is not available in the Python binding.

Field Default Description
blocking 1 Nonblocking if 0. Communicator creation, LcclCommFinalize(), and the submission of a group that needs connections can return early with kLcclInProgress. Outside a group, LcclCommDestroy() and LcclCommAbort() complete within the call
min_num_chnl 1 Lower limit on the number of channels
max_num_chnl 8 Upper limit on the number of channels
net_name Automatic Network backend name ("IB", "Socket"). Automatic selection if NULL

The LCCL_BLOCKING, LCCL_MIN_CHANNELS, LCCL_MAX_CHANNELS, and LCCL_NET environment variables override the same fields. See Environment variables.

Initialize LcclConfig with the provided initializer macro

The struct has fields that the library validates. A config that is initialized to 0 or only partially filled is rejected with kLcclInvalidArgument. Start from LCCL_CONFIG_INITIALIZER and set only the fields you want to change. Fields left at their sentinel values follow the library defaults.

Values are validated as well. blocking must be 0 or 1, and the number of channels must be positive. If you set only min_num_chnl and leave max_num_chnl at the sentinel, it is judged as min_num_chnl > max_num_chnl and rejected, so set both fields together.

2.3 Querying a communicator

Entry point Returns
LcclCommCount() Number of ranks
LcclCommUserRank() This rank's index
LcclCommHaDevice() The device this communicator drives
LcclCommGetAsyncError() The asynchronous error recorded on the communicator

Python's LcclComm object exposes the read-only attributes rank, n_ranks, ha_dev, node, n_nodes, local_rank, local_ranks, and n_channels. Use n_nodes to see how many servers the ranks span, and n_channels to see the actual number of channels.

LcclCommGetAsyncError() is needed in two cases.

  • Nonblocking communicators (blocking = 0). The call returns early with kLcclInProgress, and the result of the work is recorded in this value. Check it until it becomes kLcclSuccess.
  • When kLcclKernelAborted is returned. For a failure inside the communication path, the cause code is recorded in this value. The failed operation returns the cause code itself, but if you receive kLcclKernelAborted, read this value to learn the cause.

On a default (blocking) communicator, a collective operation has already finished when it returns, so you can read a successful return as completion. Once this value is recorded, it is not cleared. A communicator with a recorded error must be created again. Release a nonblocking one with LcclCommAbort().

2.4 Cleanup

Entry point Purpose
LcclCommFinalize() Finishes pending work and stops the communicator. Does not release resources
LcclCommDestroy() Releases resources. The normal path
LcclCommAbort() Aborts without waiting for pending work, then releases resources

LcclCommAbort() is for the case where a peer has disappeared and a normal shutdown would block forever. It stops the operations in progress on this communicator and reclaims the resources. LcclCommDestroy() and LcclCommAbort() both release resources, so call only one of them on a given handle. After the call, the handle cannot be used.

The Python binding has only comm_destroy(); LcclCommAbort() and LcclCommFinalize() are not exposed.

Destroying the communicator on only one rank

Although it does not look like one, cleanup is effectively a collective operation. If one rank destroys its communicator while the other ranks keep calling collective operations, those ranks wait for their peer and hang until the watchdog (LCCL_KERNEL_TIMEOUT_MS, default 5 seconds) cuts them off. Bring the ranks down together, or abort all of them.

When one process holds several ranks that belong to the same communicator (LcclCommInitAll, or a configuration that assigns one rank per thread), you must destroy or abort all of those ranks for the resources to actually be released. If you clean up only some of them, the resources remain until the rest are finished. Separate communicators (for example one for tensor parallelism and one for pipeline parallelism) are released independently.

2.5 Lifetime rules

  • A communicator is tied to the device bound to the calling thread at creation time. Do not make calls from a thread that has a different device bound. A newly created thread has no device bound. Before calling an operation or a cleanup function (LcclCommFinalize(), LcclCommDestroy(), LcclCommAbort(), or comm_destroy() in Python) from a thread other than the one that created the communicator (for example the main thread after joining the rank threads), select the communicator's device on that thread with HaSetDevice() (hart.set_device() in Python).
  • Handles are not reference counted. A handle becomes invalid immediately on destruction, and any later use produces kLcclInvalidArgument if the library can detect it, and undefined behavior if it cannot.
  • Python's LcclComm is not released by the garbage collector. Always call comm_destroy(). Wrapping it in try/finally cleans up on exception paths as well.
  • Serialize calls that use a communicator so that only one is in progress at a time. A configuration in which each thread owns the communicator of a different rank has been verified, but a configuration in which several threads use one communicator concurrently has not been verified.
  • One process can hold several communicators over different sets of ranks. An example is a setup in which tensor parallelism and pipeline parallelism each have their own communicator. Creating several communicators over the same set of ranks has not been verified.

Next steps