Skip to content

Python API

1
import lccl

The meaning is the same as in the C++ API; only the differences are summarized in Differences from the C++ API.

Things to know first

  • import lccl finds the hart installed in the same Python environment. If you keep hart in another directory, for example through PYTHONPATH, run import hart first.
  • The standalone lccl package is released as an LPU hardware build (+hw). torch_ha also includes LCCL, which you can use through torch.distributed. See the Support matrix.
  • To use lccl and torch_ha in one process, install hart, lccl, and torch_ha from the same release. lccl and torch_ha each ship their own library with the same name (libha_lccl.so), and both use whichever is loaded first.

Communicators

Function Signature Notes
get_unique_id () -> LcclUniqueId Call on one rank only and distribute the result out of band. This process must stay alive until all ranks have connected
comm_init_rank (nranks: int, comm_id: LcclUniqueId, myrank: int) -> LcclComm Blocks until all nranks ranks have joined. Call hart.set_device() first
comm_init_all (ndev: int, devlist: list[int]) -> list[LcclComm] Single process, single server. Bind a device to the thread before the call. devlist must be a list whose length equals ndev, with no duplicate or out-of-range values. If it raises, stop using lccl in that process and start again in a new process
comm_destroy (comm: LcclComm) -> None Releases the resources. The garbage collector does not do this for you, so always call it
bootstrap_is_root_alive () -> int Whether ranks are still connecting to the unique ID created by this process (1 means it is still waiting). There is one value for the whole process, and it becomes 0 once all ranks have connected

Groups

Function Signature Notes
group_start () -> None Can be nested. The depth is tracked per thread
group_end () -> None Executes when the outermost region closes

Collective operations

Function Signature
all_reduce (send_buff, recv_buff, count: int, datatype: LcclDataType, op: LcclRedOp, comm: LcclComm, stream: int = 0) -> None
reduce (send_buff, recv_buff, count, datatype, op, root: int, comm) -> None
reduce_scatter (send_buff, recv_buff, recv_count, datatype, op, comm) -> None
broadcast (send_buff, recv_buff, count, datatype, root: int, comm) -> None
all_gather (send_buff, recv_buff, send_count, datatype, comm) -> None
gather (send_buff, recv_buff, count, datatype, root: int, comm) -> None
scatter (send_buff, recv_buff, count, datatype, root: int, comm) -> None
all_to_all (send_buff, recv_buff, count, datatype, comm) -> None

The receive buffer size differs for each operation. See Collective operations. The operations that run on the LPU are listed in the Support matrix.

Buffer arguments

send_buff and recv_buff accept the device address (an integer) returned by hart.mem_alloc(). An object that becomes an integer through __index__ (a NumPy or PyTorch integer scalar) is treated the same way. The size is not checked, and count is trusted. bool is rejected.

On a non-root rank, a buffer that is not used must still be given a value. None does not work; pass the integer 0.

Point-to-point

Function Signature Notes
send (send_buff, count, datatype, peer: int, comm) -> None Pair it with the corresponding recv inside a group
recv (recv_buff, count, datatype, peer: int, comm) -> None

For two ranks to exchange data with each other, both sides must put both calls in the same group; otherwise they deadlock. See Point-to-point.

1
2
3
4
5
6
lccl.group_start()
try:
    lccl.send(send_buf, count, dt, peer, comm)
    lccl.recv(recv_buf, count, dt, peer, comm)
finally:
    lccl.group_end()

Types

LcclComm

Read-only attributes that describe the communicator. They are useful for diagnostics.

Attribute Meaning
rank, n_ranks The index of this rank and the size of the communicator
ha_dev The device this communicator drives
bus_id The bus identifier of that device
node, n_nodes The index of this server and the number of servers
local_rank, local_ranks The rank index within the server and the number of ranks on this server
n_channels The number of channels this communicator created

n_nodes shows how many servers LCCL decided the ranks are spread over, and n_channels shows the number of channels actually created. A release build cannot provide this information through the INFO log, so check it with these attributes. See Environment variables.

LcclUniqueId

Form Meaning
LcclUniqueId() An empty handle. Use the one that get_unique_id() fills in and returns
LcclUniqueId(data: bytes) Reconstructs a handle from the internal value. It must be exactly 128 bytes, otherwise ValueError
internal Read-only. The raw 128-byte handle, as bytes

What you distribute to the other ranks is bytes(unique_id.internal), and the receiving side restores the handle with LcclUniqueId(data). The handle object itself cannot be serialized with pickle.

LcclDataType

LcclInt8 / LcclChar, LcclUint8, LcclInt32 / LcclInt, LcclUint32, LcclInt64, LcclUint64, LcclFloat16 / LcclHalf, LcclFloat32 / LcclFloat, LcclFloat64 / LcclDouble, LcclBfloat16.

Paired names are aliases with the same value and match the NCCL enumeration.

LcclRedOp

LcclSum, LcclProd, LcclMax, LcclMin, LcclAvg.

The enumeration is wider than the implementation

LcclMax and LcclMin have no kernel for any operation, so they raise a RuntimeError (kLcclNotSupported). LcclProd and LcclAvg work only with all_reduce and reduce. See the Support matrix.


Differences from the C++ API

C++ Python
Failure reporting Returns LcclResult RuntimeError. The message is <function name> failed: <message>
Invalid unique ID length — ValueError
Buffers void * device pointer Integer (device address)
Streams Most take an HaStream_t Only all_reduce has a stream keyword (integer handle, default 0). The rest use the default stream
Communicator queries LcclCommCount(), LcclCommUserRank(), LcclCommHaDevice() Attributes of LcclComm
LcclCommInitRankConfig, LcclCommFinalize, LcclCommAbort Available Not exposed
LcclCommGetAsyncError Available Not exposed
LcclCommRegister, LcclCommDeregister Available Not exposed
LcclConfig Available Not available. Environment variables and configuration files are the only configuration path
LcclResult values Enumeration Not exposed. Present only as a sentence in the exception message
String helpers LcclGetErrorToString() and others Not exposed
GIL — Blocking calls do not release the GIL

Asynchronous errors cannot be checked from Python

LcclCommGetAsyncError() has no Python binding. In nonblocking mode (LCCL_BLOCKING=0), a call that returns kLcclInProgress turns into an exception, so use only the default blocking mode from Python. The cause code cannot be read either when kLcclKernelAborted is returned. If this matters in your deployment, drive the library from C++.

The GIL is not released, so splitting ranks across threads deadlocks

Collective operations and comm_init_rank block while waiting for the other ranks, and they do not release the GIL in the meantime, so other Python threads in the same process stall. Assigning one rank per thread within a single process deadlocks. Use a separate process for each rank.

Device selection

A communicator is tied to the device bound to the calling thread at the time of creation. Set the device before creating it. If you omit this, the error is LcclCommInitRank failed: Communicator unknown or unclassified error.

1
2
3
import hart
hart.set_device(rank_local_device)
comm = lccl.comm_init_rank(nranks, unique_id, rank)

The current device is thread-local. See hart. Before calling an operation or comm_destroy() from a thread other than the one that created the communicator, select the same device on that thread with hart.set_device(). A child created by fork from a process that has initialized the runtime is not safe, so start rank processes with spawn.