Skip to content

4. Distributed

Multi-device execution uses torch.distributed with the lccl backend. This chapter covers the architecture, setup, and common usage patterns.

Contents

Section Objective
4.1 Architecture The two layers: the c10d backend lccl and the torch.ops.lpu.lccl_* custom operators
4.2 Device Binding Per-rank device selection and how allocation without an index resolves
4.3 Launch Setup Environment variables, torchrun, and process group initialization order
4.4 Parallelism Patterns Tensor parallelism, pipeline parallelism, and collectives under torch.compile
4.5 Supported Scope Which collectives and dtype/reduction combinations are supported
4.6 Correctness Rules Call order, producer synchronization, and shutting down ranks together
4.7 Debugging Hangs How to distinguish rendezvous failures from ranks that do not participate

4.1 Architecture

LCCL, the LPU collective communication library, handles the underlying communication. The PyTorch integration consists of two layers that share the same communicator.

Layer Entry point Use
c10d backend lccl Standard torch.distributed APIs such as dist.all_reduce General use. Existing distributed code works unchanged
Custom operators torch.ops.lpu.lccl_* Called directly or from a compiled graph Fast path that captures collectives during torch.compile tracing without a graph break

During import torch, torch_ha registers "lccl" as the torch.distributed backend for the lpu device type, which makes it the default backend for process groups that use lpu tensors.

The custom operator layer reuses the communicator that the process group created (torch.ops.lpu.lccl_comm_handle). Rendezvous happens once, and both layers run over the same connection. The lccl_* operators that take tensors are registered with Meta kernels, so they work under FakeTensor tracing.

4.2 Device Binding

A process can see multiple devices. Each rank selects its device once at startup, then allocates without an index.

1
2
3
4
torch.accelerator.set_device_index(rank)             # once per rank

x = torch.randn(32, 1024, dtype=torch.bfloat16)
x = x.to("lpu")                                      # resolves to the selected device

torch.device("lpu") without an index resolves to the current device of the calling thread, the same semantics as CUDA. To select, query, and synchronize devices, use the torch.accelerator APIs: set_device_index, device_count, and synchronize.

The lccl backend creates its communicator at the first collective and binds rank r of the process group to lpu:<r>. As a result, torch.distributed is supported within one server only, and in groups made with dist.new_group as well, rank r of the group must use lpu:<r>. See LCCL support matrix §Scale.

4.3 Launch Setup

The standard torch.distributed environment variables (MASTER_ADDR, MASTER_PORT, RANK, WORLD_SIZE, and others) apply unchanged, and the standard launchers work.

1
torchrun --nproc-per-node 2 your_script.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import os

import torch
import torch.distributed as dist

# The backend binds rank r to lpu:<r>.
# Select the same device first so that tensors are placed there too.
local_rank = int(os.environ["LOCAL_RANK"])
torch.accelerator.set_device_index(local_rank)

dist.init_process_group(backend="lccl")

x = torch.randn(32, 1024, dtype=torch.bfloat16).to("lpu")

dist.all_reduce(x, op=dist.ReduceOp.SUM)

dist.destroy_process_group()

4.4 Parallelism Patterns

Tensor parallelism (TP). The hot path is all-reduce and all-gather, which run on the ring implementation in the device kernels. Each collective is issued once over the tensor's entire channel-interleaved allocation.

Pipeline parallelism (PP). Activations move between stages through point-to-point send/recv. This is a copy path, so it has no dtype restriction, but the tensor's byte size must be a multiple of 512 (§4.5). For bidirectional exchange, use dist.batch_isend_irecv. The backend supports coalescing, so both directions are fused into a single plan and proceed without deadlock.

Under torch.compile. Collectives inside a compiled graph can be captured through the custom operator layer.

1
2
3
4
group_name = dist.distributed_c10d._get_default_group().group_name
handle = torch.ops.lpu.lccl_comm_handle(group_name)

out = torch.ops.lpu.lccl_all_reduce(x, "sum", handle)

When you call point-to-point operators directly, wrap a bidirectional exchange in torch.ops.lpu.lccl_group_start() / lccl_group_end(). This serves the same purpose as c10d coalescing.

4.5 Supported Scope

The dtype/reduction combinations available through torch.distributed are the ones the underlying LCCL implements. However, on the LPU the lccl backend blocks reduce, gather, scatter, and all_to_all with a RuntimeError tagged [HA_LCCL_NO_HW_KERNEL]. The available collectives are all_reduce, all_gather, reduce_scatter, broadcast, and send/recv. The supported set differs from NCCL, so check the LCCL support matrix before designing around collectives.

broadcast, send/recv, and reduce_scatter raise RuntimeError on any number of ranks unless the tensor's byte size (the output tensor's for reduce_scatter) is a multiple of 512. Pad to the element multiple named in the message and slice the result. See LCCL support matrix §Counts and sizes.

A combination outside the support matrix raises an exception, and no data is corrupted.

4.6 Correctness Rules

These rules come from the collective communication library. The PyTorch wrapper does not relax them.

  • Every rank must call the same collectives in the same order. Collectives are matched by call order. A rank that skips one on an empty batch breaks the group's synchronization, and the result is a hang, not an error.
  • Synchronize the producer before a collective. If the send buffer is still being written, the result is nondeterministic. Reading the buffer with .cpu() or issuing the collective on the same stream is sufficient.
  • Shut down ranks together. If one rank destroys the process group while other ranks are still calling collectives, the remaining ranks stall.

4.7 Debugging Hangs

A distributed hang almost always has one of two causes: the ranks never completed rendezvous, or one rank is not participating.

  1. Check that every rank reached init_process_group. If one did not, the problem is rendezvous: check MASTER_ADDR, MASTER_PORT, and reachability.
  2. If all ranks initialized, log the collective name and sequence number on every rank before each call. The bug is at the first point where the ranks diverge.

A hang in a point-to-point bidirectional exchange is usually caused by missing grouping. First check that the exchange is wrapped in dist.batch_isend_irecv or lccl_group_start/end.

The full procedure, including how to interpret error codes, is in LCCL Troubleshooting.


Next Steps