4. Distributed¶
Multi-device execution uses torch.distributed with the lccl backend. This chapter covers the architecture, setup,
and common usage patterns.
Contents¶
| Section | Objective |
|---|---|
| 4.1 Architecture | The two layers: the c10d backend lccl and the torch.ops.lpu.lccl_* custom operators |
| 4.2 Device Binding | Per-rank device selection and how allocation without an index resolves |
| 4.3 Launch Setup | Environment variables, torchrun, and process group initialization order |
| 4.4 Parallelism Patterns | Tensor parallelism, pipeline parallelism, and collectives under torch.compile |
| 4.5 Supported Scope | Which collectives and dtype/reduction combinations are supported |
| 4.6 Correctness Rules | Call order, producer synchronization, and shutting down ranks together |
| 4.7 Debugging Hangs | How to distinguish rendezvous failures from ranks that do not participate |
4.1 Architecture¶
LCCL, the LPU collective communication library, handles the underlying communication. The PyTorch integration consists of two layers that share the same communicator.
| Layer | Entry point | Use |
|---|---|---|
c10d backend lccl |
Standard torch.distributed APIs such as dist.all_reduce |
General use. Existing distributed code works unchanged |
Custom operators torch.ops.lpu.lccl_* |
Called directly or from a compiled graph | Fast path that captures collectives during torch.compile tracing without a graph break |
During import torch, torch_ha registers "lccl" as the torch.distributed backend for the lpu device type,
which makes it the default backend for process groups that use lpu tensors.
The custom operator layer reuses the communicator that the process group created (torch.ops.lpu.lccl_comm_handle).
Rendezvous happens once, and both layers run over the same connection. The lccl_* operators that take tensors are
registered with Meta kernels, so they work under FakeTensor tracing.
4.2 Device Binding¶
A process can see multiple devices. Each rank selects its device once at startup, then allocates without an index.
1 2 3 4 | |
torch.device("lpu") without an index resolves to the current device of the calling thread, the same semantics as
CUDA. To select, query, and synchronize devices, use the torch.accelerator APIs: set_device_index,
device_count, and synchronize.
The lccl backend creates its communicator at the first collective and binds rank r of the process group to
lpu:<r>. As a result, torch.distributed is supported within one server only, and in groups made with
dist.new_group as well, rank r of the group must use lpu:<r>.
See LCCL support matrix §Scale.
4.3 Launch Setup¶
The standard torch.distributed environment variables (MASTER_ADDR, MASTER_PORT, RANK, WORLD_SIZE, and
others) apply unchanged, and the standard launchers work.
1 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
4.4 Parallelism Patterns¶
Tensor parallelism (TP). The hot path is all-reduce and all-gather, which run on the ring implementation in the device kernels. Each collective is issued once over the tensor's entire channel-interleaved allocation.
Pipeline parallelism (PP). Activations move between stages through point-to-point send/recv. This is a copy
path, so it has no dtype restriction, but the tensor's byte size must be a multiple of 512 (§4.5). For bidirectional
exchange, use dist.batch_isend_irecv. The backend supports coalescing, so both directions are fused into a single plan
and proceed without deadlock.
Under torch.compile. Collectives inside a compiled graph can be captured through the custom operator layer.
1 2 3 4 | |
When you call point-to-point operators directly, wrap a bidirectional exchange in torch.ops.lpu.lccl_group_start() /
lccl_group_end(). This serves the same purpose as c10d coalescing.
4.5 Supported Scope¶
The dtype/reduction combinations available through torch.distributed are the ones the underlying LCCL implements.
However, on the LPU the lccl backend blocks reduce, gather, scatter, and all_to_all with a RuntimeError
tagged [HA_LCCL_NO_HW_KERNEL]. The available collectives are all_reduce, all_gather, reduce_scatter,
broadcast, and send/recv. The supported set differs from NCCL, so check the
LCCL support matrix before designing around collectives.
broadcast, send/recv, and reduce_scatter raise RuntimeError on any number of ranks unless the tensor's byte
size (the output tensor's for reduce_scatter) is a multiple of 512. Pad to the element multiple named in the message
and slice the result. See
LCCL support matrix §Counts and sizes.
A combination outside the support matrix raises an exception, and no data is corrupted.
4.6 Correctness Rules¶
These rules come from the collective communication library. The PyTorch wrapper does not relax them.
- Every rank must call the same collectives in the same order. Collectives are matched by call order. A rank that skips one on an empty batch breaks the group's synchronization, and the result is a hang, not an error.
- Synchronize the producer before a collective. If the send buffer is still being written, the result is
nondeterministic. Reading the buffer with
.cpu()or issuing the collective on the same stream is sufficient. - Shut down ranks together. If one rank destroys the process group while other ranks are still calling collectives, the remaining ranks stall.
4.7 Debugging Hangs¶
A distributed hang almost always has one of two causes: the ranks never completed rendezvous, or one rank is not participating.
- Check that every rank reached
init_process_group. If one did not, the problem is rendezvous: checkMASTER_ADDR,MASTER_PORT, and reachability. - If all ranks initialized, log the collective name and sequence number on every rank before each call. The bug is at the first point where the ranks diverge.
A hang in a point-to-point bidirectional exchange is usually caused by missing grouping. First check that the exchange
is wrapped in dist.batch_isend_irecv or lccl_group_start/end.
The full procedure, including how to interpret error codes, is in LCCL Troubleshooting.
Next Steps¶
- Paged KV Cache: mapping KV cache blocks with virtual memory
- LCCL documentation: the underlying library
- Migrating from CUDA: differences from CUDA code