8. Multi-node execution¶
This chapter covers what changes when ranks span several servers. It explains which transport is used, which network interface the ranks connect over, what gets in the way with containers and firewalls, and how to confirm that traffic has really crossed between servers. How to pin the topology with a file is also covered here.
8.1 What changes with several servers¶
When LCCL connects two ranks, it compares their host identifiers. The identifier is the hostname with a boot id (which changes on every boot) appended, and it can be overridden with LCCL_HOSTID. If the identifiers are the same, the ranks are treated as being on the same server; if they differ, as being on different servers.
| Location of the two ranks | Transport |
|---|---|
| Same server, same process | P2P (direct device-to-device DMA) |
| Same server, different processes | Shared memory |
| Different servers | Network: tries ibverbs first, and falls back to a TCP socket if it cannot be used |
Two cards on the same server are connected over the network transport instead when each card has a NIC under its PCIe switch and the path to that NIC has more bandwidth than the path between the two cards, for example two cards on different CPU sockets that each share a switch with a 400 Gb/s NIC. This is decided from the PCIe layout and link bandwidths, as discovered or as written in the topology file, and cards connected this way count as separate nodes, so comm.n_nodes is larger than the number of servers.
Between servers, only the configuration in which one process handles one rank is supported. The configuration in which one process per server handles all the cards of that server is not supported.
Verified configurations
AllGather and AllReduce, including LPU memory and device kernels, have been verified in a configuration with 2 servers, each with 1 LPU, connected over TCP sockets. This includes running inside containers (host network). For ibverbs, AllGather, AllReduce, ReduceScatter, and Broadcast have been verified with 2 RNICs cabled together within a single server, and an ibverbs connection between 2 servers has not been verified. Configurations with 3 or more servers, and configurations that connect several cards per server across servers, have not been verified either. See the Support matrix.
8.2 Setup sequence¶
Install lccl (or torch_ha) from the same release on every server. Mixing ranks of different releases in one communicator is not supported.
- Start a rank process on each server. All ranks use the same
nranks, and rank numbers are assigned from0tonranks - 1without duplicates. - One rank creates a unique ID and sends it to the ranks on all servers. This handle contains the connection address of the process that created the unique ID, so the ranks on the other servers must be able to connect to that address. Any means of delivery works, such as MPI or a key-value store. The
lcclbackend oftorch.distributedsupports runs within one server only, so set up runs across several servers with thelcclAPI, following this sequence (Support matrix). - Pin which network interface to use on each server (8.3).
- Each rank selects its own card and then calls
LcclCommInitRank(). - Confirm that traffic has crossed between servers (8.6).
8.3 Choosing the network interface¶
LCCL_SOCKET_IFNAME pins every interface used for bootstrap, the socket transport, and ibverbs connection setup. If a server has several NICs, be sure to set it.
| Notation | Meaning |
|---|---|
eth |
Prefix match. Both eth0 and eth1 |
=eth0 |
Exactly eth0 only |
^docker,lo |
Everything except docker* and lo |
If the NIC names differ from server to server, such as eno1 and enp3s0f0, give each server its own value with the exact name of its NIC. A prefix such as en can also match a management NIC, which may then be selected.
1 2 3 4 | |
If it is not set, the first matching interface is chosen in this order: ib*, anything that is not docker or lo, docker*, lo. On a server with several NICs, it may pick a management NIC or an unconnected ib0.
Once you set it, selection never falls back to automatic. If no interface matches, it fails with a warning. This is because a silent fallback would lead to the worst case: the connection between servers ends up on loopback and still reports success.
8.4 Firewalls and containers¶
Firewall. The process that created the unique ID and the socket transport accept connections on ephemeral ports. If a firewall or security group blocks TCP between servers, the connection never completes. There is no setting that uses a fixed port, so allow all TCP between servers.
Containers: bridge network. On the default bridge network, each container advertises a 172.17.x.x address that is valid only inside its own server. Other servers cannot reach that address, so the rendezvous never finishes. For containers on different servers, use the host network (docker run --network host). The hostname differs from server to server, so the distinction between servers is preserved.
Containers: two on one server. If you start two containers on the same server with --network host, both report the host's name. If they share /dev/shm (for example, docker run --ipc=host), they are treated as one server (n_nodes == 1) and connect through shared memory. If each container has its own /dev/shm (the default), each container is treated as a separate server (n_nodes == 2), and communication between the containers goes over the network transport.
Containers: ibverbs. To use ibverbs inside a container, pass the RDMA devices into the container and lift the locked-memory limit (docker run --device /dev/infiniband --ulimit memlock=-1). In the image, install libibverbs together with the same RNIC provider library that the host uses, and check that ibverbs is selected by pinning it with LCCL_NET=IB (8.5).
8.5 ibverbs (InfiniBand, RoCE)¶
If an RDMA device is present and libibverbs is available at run time, LCCL uses ibverbs before sockets. If either one is missing, it moves on to sockets without an error.
- When the conditions are met, LCCL performs RDMA on LPU memory directly (LPU Direct RDMA). The RNIC reads and writes LPU memory without moving the data through host memory. The servers on both ends of a connection must meet all of the following conditions; if any one is not met, the data of that connection goes through host memory as before.
- The driver is 2.6.0 or later and its translation window (
tw_window_size) is 64 MiB. The driver must be able to export LPU memory as a dma-buf. - The RNIC driver supports dma-buf memory registration. - The PCIe path between the LPU and the RNIC is withinLCCL_NET_RDMA_LEVEL. The default,PXB, is a path where both sit under PCIe switches and do not go through the CPU. -LCCL_BUFFSIZEis a multiple of 256 KiB (262144). The default (4 MiB) meets this. - A process that uses LPU Direct RDMA allocates 1 GiB of LPU memory per LPU by default and keeps it until the process exits; if this memory runs short, communicator creation fails instead of falling back to host memory. For the size and the error codes, see
LCCL_DMABUF_POOL_SIZE_PER_CHANNELin Environment variables. - To keep LCCL from using direct RDMA, set
LCCL_NET_RDMA_LEVEL=LOC. To turn it off only on the send side, setLCCL_NET_RDMA_READ=0; only on the receive side, setLCCL_GDR_FLUSH_DISABLE=1. - If
LCCL_IB_DISABLE=1, ibverbs is not used. - If there are several devices, select the device and port with
LCCL_IB_HCA(for example,mlx5_0:1), and if GID selection is a problem on RoCE, setLCCL_IB_GID_INDEX. - Connection setup (the handshake) uses an IP interface, so
LCCL_SOCKET_IFNAMEapplies here too.
Which backend was selected is recorded only in the INFO log, so it is not visible in release builds. To check, pin the backend with LCCL_NET=IB. If ibverbs cannot be used, communicator creation fails with kLcclInvalidArgument. For the variables, see Environment variables.
8.6 Confirming that traffic actually crossed the network¶
A collective operation that passes proves nothing about communication between servers. The result is equally correct whether it went through shared memory or the network, and whether the ranks are on the same server or on different servers. A test passes even if communication silently falls back to shared memory. Check the following.
| Check | How |
|---|---|
| Were the servers actually split? | Check that comm.n_nodes equals the number of servers on every rank. In a layout where cards on the same server are connected over the network transport (8.1), it is larger than the number of servers |
| Is the split not forged? | Check that LCCL_HOSTID has not been set and that no server runs several containers with separate /dev/shm. Such a split connects over loopback within the same server and verifies nothing about the network |
| Which interface did the traffic leave on? | Check whether the send and receive counters (ip -s link) of the NIC pinned with LCCL_SOCKET_IFNAME, or the RNIC port counters, have increased |
| Was ibverbs selected? | Pin it with LCCL_NET=IB and check that creation succeeds |
| Is LPU Direct RDMA used? | After the communicator is created, read /sys/class/ha/ha*/dmabuf_exports on the host and check for live windows whose pid is that of the rank's process. If there are any, at least one of the rank's network connections uses LPU Direct RDMA; if there are none, all of them go through host memory (sysfs nodes) |
n_nodes alone is not enough, because a split faked with LCCL_HOSTID reports the same value.
8.7 Topology file¶
By default, LCCL reads /sys at initialization and discovers the server's hardware layout (CPUs, PCIe structure, LPUs, NICs) by itself. If your orchestration knows this layout when it creates a pod, it can hand it over as a single JSON file. If you put a file path in LCCL_PREDEFINED_SYSINFO, LCCL uses that file instead of discovering the layout. The connection paths between ranks are not written in the file. LCCL decides the paths based on the hardware layout that the file describes.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
| Key | Description |
|---|---|
schemaVersion |
Integer. The current value is 1 |
Top-level members |
List of CPU entries |
CPU |
socketId (NUMA node number), arch, vendor, familyId, modelId, affinity (that NUMA node's cpumap), and members |
PCI |
busId (required), linkspeed, and nested members (PCI, LPU, NIC) |
LPU, NIC |
busId (required), linkspeed |
The rules are as follows.
busIdis the only key that ties the file to the execution environment.LPU.busIdmust be the PCI address of the card that the rank actually opened. For machines whose domain is not0000, write the domain as well (0001:d8:00.0). Write thebusIdof everyPCI,LPU, andNICentry in the form thatlspcior sysfs shows (19:00.0,0000:19:00.0). A missingbusId, or one in any other form, makes initialization fail withkLcclTopoConfigInvalid.- Neither a rank nor a device number is written. LCCL fills them in at run time.
- A NIC is described per card. Ports are not written; each card gets one
NICline. The address is the last component of the path thatls -l /sys/class/net/*/deviceorls -l /sys/class/infiniband/*/devicepoints to. - List the card of every network device the ranks use (RNICs for ibverbs, interfaces for the socket transport) as a
NIC. A device whose card is not in the file is ignored with a warning, even in a job on a single server; if no device remains for traffic between servers, initialization fails withkLcclCommError. - There is one
CPUentry per NUMA node, not per socket. Put each card under theCPUwhosesocketIdequals the card'snuma_node(/sys/bus/pci/devices/<busId>/numa_node), and write that NUMA node's/sys/devices/system/node/node<N>/cpumapinaffinityas is (lowercase hexadecimal, a comma every 32 bits, no0x). - Write
archasx86_64(orx86) andvendorasGenuineIntel(orIntel),AuthenticAMD(orAMD), orCentaurHauls(orZhaoxin), with the exact case. For any other value, LCCL logs a warning and treats the CPU type as unknown. linkspeedis the bandwidth (GB/s) of the link that connects the entry to the entry right above it.PCI,LPU, andNICcan all have it. Write the value that LCCL's own discovery gives the link, for example 48 for a PCIe Gen5 x16 link and 24 for a Gen4 x16 or Gen5 x8 link. If it is missing ornull, the entry does not inherit its parent's value and uses the default of 12 GB/s, the same as when LCCL discovers the hardware itself. A value that is not positive makes initialization fail withkLcclTopoConfigInvalid.- Numeric fields take JSON numbers only. Values such as
true, hexadecimal numbers,inf, and"48GB", and fractions or out-of-range values in integer fields, are initialization errors. A file with unbalanced brackets, a trailing comma, or a cut-off end is not read partially; initialization ends with an error. - Virtual interfaces (veth, bond, etc.) are not written. LCCL finds them at run time and attaches them to the CPU of the card used by the lowest rank on this server, the same as when it discovers the hardware without a file.
- A file describes one server. If a card claimed by a rank on this server is not in the file, initialization fails with
kLcclTopoNodeNotFound, and a card that is in the file but not claimed by any rank is removed with a warning. - If
schemaVersiondoes not match or is not a number, it always fails withkLcclTopoSchemaVersionMismatch.
In production, LCCL_TOPO_STRICT=1 is recommended. In lenient mode (the default), a wrong file path leaves only a one-line warning and falls back to direct discovery, and a file without a version is read by guesswork. In both cases init succeeds, so a single log line is the only evidence. In strict mode these two cases fail with kLcclTopoFileError and kLcclTopoSchemaVersionMismatch, respectively.
8.8 Troubleshooting¶
| Symptom | Cause |
|---|---|
comm.n_nodes is smaller than the number of servers you expect |
The two ranks see the same host identifier. --network host containers that share /dev/shm were started on one server as if they were separate servers (they are treated as one server), or LCCL_HOSTID is the same |
Traffic between servers goes over loopback (lo) |
LCCL_SOCKET_IFNAME was not set, so automatic selection picked the wrong interface |
| The rendezvous does not finish | The peer cannot be reached over the specified NIC. Firewall, container bridge address (172.17.x.x) |
kLcclSocketTimeout, kLcclSocketNotConnect |
The unique ID is stale (connecting with an ID whose use has already ended). Start with a new ID. Or the address cannot be reached |
kLcclMemShmMapFail during initialization |
The ranks on one server received different LCCL_SHM_PATH values |
kLcclNetSizeMismatch or kLcclTransportBuffSizeMismatch during initialization |
LCCL_BUFFSIZE differs between ranks |
kLcclTopoNodeNotFound |
The topology file does not contain the card claimed by a rank on this server |
| The watchdog aborts the kernel | The connection between servers is slow, or the ranks enter the operation too far apart, and LCCL_KERNEL_TIMEOUT_MS (5 seconds by default) was exceeded; or a rank on one server did not arrive |
Next steps¶
- Error handling
- Environment variables — network and topology variables
- Support matrix — verified configurations