Environment variables¶
lccl reads values from environment variables and two configuration files. LCCL_* variables not listed on this page are for development, so setting them has no effect or they change without notice.
Applies at — when the value is read. Library load is when import lccl runs or the library is loaded, and First use in the process means the first value read stays fixed until the process ends. Changing an environment variable after the value has already been read has no effect.
Precedence and configuration files¶
A value is taken from the first of the following places where it is defined.
- Environment variable
~/.lccl.conf/etc/lccl.conf- Built-in default
Configuration files use the NAME=VALUE format on each line. Note the following.
- The files are read at the first lookup and loaded into the process environment. Edits to the files after the process has started have no effect.
- Values from the files do not overwrite environment variables that are already set.
- Comments are not supported, and whitespace around a value is not trimmed. Trailing whitespace or
\rat the end of a line also becomes part of the value. ~is the home directory of the user running the process (thepasswdentry), not$HOME.- If you set a value that is not an integer on an integer variable, it silently reverts to the default.
LCCL_DMABUF_POOL_SIZE_PER_CHANNELis an exception: with an invalid value, creating a communicator that uses LPU Direct RDMA fails. - Every
NAME=VALUEin a file becomes a process environment variable. This includes names that do not start withLCCL_, and child processes inherit them. As a result,LCCL_SHM_PATH, which is read directly withgetenv, usually sees the file value too, but variables read when the library is loaded, such asHA_LOG_*, do not see file values. Set such variables as environment variables.
The Python bindings have no LcclConfig, so in Python, environment variables and configuration files are the only way to configure a communicator.
Topology¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
LCCL_PREDEFINED_SYSINFO |
Path | None | Communicator creation | Reads the hardware layout from this JSON file instead of discovering it itself. If the file does not exist or is not a regular file, it warns and discovers the layout itself (it fails if LCCL_TOPO_STRICT=1). For the format, see Topology file |
LCCL_TOPO_STRICT |
0 | 1 |
0 |
Communicator creation | If 1, a topology file that does not exist, is not a regular file, or has no schemaVersion fails initialization instead of being passed over with a warning. Only integers are accepted, so true and yes do not turn it on |
LCCL_TOPO_DUMP_FILE |
Path | None | Communicator creation | Rank 0 writes the topology it discovered itself to this file as XML. When a topology file is used, nothing is written |
LCCL_IGNORE_CPU_AFFINITY |
Whether it is set (value ignored) | Not set | Communicator creation | Ignores the inherited CPU affinity and uses the set of CPUs close to the LPU. Does not apply when a topology file is used |
LCCL_HOSTID |
String | Hostname + boot id | Communicator creation | Overrides the identifier used to decide which ranks are on the same server |
LCCL_HOSTID matters in containers. Ranks that report the same identifier are treated as being on one server and are connected through shared memory. The default identifier is the hostname plus a boot id that changes with every boot. Containers on the default network and Kubernetes pods each have a different hostname, so they are already classified as different servers. However, containers started on one server with the host network (host UTS) have the same hostname and boot id, so they are classified as one server, but shared memory can be used only when the two ranks see the same /dev/shm. If /dev/shm is separate per container, LCCL treats each container as a separate server and connects the containers over the network transport.
Channels and buffers¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
LCCL_MIN_NCHANNELS |
Integer | Not set (0) | Communicator creation | Lower limit on the number of channels. On the LPU with 2 or more ranks, a value above 1 is ignored with a warning |
LCCL_MAX_NCHANNELS |
Integer | 1 | Communicator creation | Upper limit on the number of channels. A value below 1 is ignored with a warning and the default is used. On the LPU with 2 or more ranks, a value above 1 is ignored with a warning |
LCCL_MIN_CHANNELS |
Integer | 1 | Communicator creation | Overrides min_num_chnl in LcclConfig |
LCCL_MAX_CHANNELS |
Integer | 8 | Communicator creation | Overrides max_num_chnl in LcclConfig |
LCCL_BUFFSIZE |
Bytes | 4 MiB (4194304) | Communicator creation | Size of the transfer buffer used by the Simple protocol. It must be a multiple of 64 and cannot be made larger than the default (it can only be reduced). If either of these conditions is violated, it warns and uses the default. On the LPU with 2 or more ranks, it must also be a multiple of 4096; otherwise every collective and LcclSend/LcclRecv returns kLcclNotSupported. All ranks must use the same value |
LCCL_CHUNK_SIZE |
Bytes | 0 (not set) | Operation submission | Chunk size for point-to-point transfers (LcclSend/LcclRecv, LcclAllToAll). Does not apply to collectives. If you set it, use a multiple of 512 no larger than ⅛ of LCCL_BUFFSIZE (512 KiB by default), with the same value on all ranks. Other values can make point-to-point transfers fail with kLcclKernelInvalidArgs, and that communicator can no longer be used after this failure |
On the LPU, a communicator with 2 or more ranks uses 1 channel regardless of these variables. A value above 1 makes the library log a communicator of N ranks on hw uses 1 channel warning.
Algorithms and protocols¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
LCCL_PROTO |
Name list | Simple |
Communicator creation | Protocols to allow. List LL and Simple separated by commas; a leading ^ excludes the listed ones. Case-insensitive. An empty value is treated as not set. Unknown names are ignored without an error |
LCCL_ALGO |
Name list | Ring |
Communicator creation | Algorithms to allow. Ring is the only one, so there is no reason to change it. The format is the same as LCCL_PROTO |
Be careful when you set these variables. If the allow list becomes empty (a misspelled name, or only LCCL_PROTO=LL on the LPU, which has no LL kernel), every collective fails with kLcclEnqueueNoAlgo. Values such as NCCL's tree and LL128 do not exist in this library. For the basis and scope, see Support matrix.
Transport layers¶
P2P and shared memory¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
LCCL_P2P_DISABLE |
0 | 1 |
0 |
Communicator creation | If 1, the P2P transport is not used |
LCCL_P2P_LEVEL |
LOC | PIX | PXB | PHB | SYS |
Auto | Communicator creation | Maximum interconnect distance at which P2P is allowed. The legacy numeric form (0 and up) is also accepted. When auto, it is PXB on Intel and Zhaoxin x86 hosts and on ARM hosts, and SYS otherwise |
LCCL_IGNORE_DISABLED_P2P |
0 | 1 |
0 |
Communicator creation | Hides the INFO message (P2P is disabled between connected LPUs) logged for each pair of LPUs for which the driver does not allow P2P. P2P for that pair is turned off regardless of this variable |
LCCL_SHM_PATH |
Path | /dev/shm |
When communication connections are created | Directory where shared memory files are created. All ranks on a server must use the same value |
P2P can be used only between ranks in the same process. In the common configuration where each rank uses its own process (torchrun, multiprocessing), the P2P transport is not selected and falls back to shared memory. LCCL_P2P_DISABLE and LCCL_P2P_LEVEL can affect how the communication path is chosen even in this configuration, where P2P is not used, so if you change them, check the result.
Shared memory is created as lccl-XXXXXX files under LCCL_SHM_PATH. When ranks on the same server run in separate processes, they communicate through this shared memory, and the capacity needed per server grows with the square of the number of ranks on that server. With the default LCCL_BUFFSIZE, 2 ranks need about 22 MiB, 4 ranks about 80 MiB, and 8 ranks about 300 MiB. The default /dev/shm of a container (64 MiB) holds at most 3 ranks, so for more ranks, increase --shm-size or point LCCL_SHM_PATH at a larger filesystem. If space or permissions are insufficient, the result is kLcclMemShmMapFail.
Network¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
LCCL_SOCKET_IFNAME |
Name list | Auto | First use in the process | Network interface to use for bootstrap and for socket and ibverbs connections. eth (prefix), =eth0 (exact match), ^docker,lo (exclude). Once set, it does not fall back to automatic selection |
LCCL_SOCKET_FAMILY |
AF_INET | AF_INET6 |
Auto | First use in the process | Filters candidate interfaces to those with an IPv4 or IPv6 address. If you specify a kind that the server does not have, no interface matches and the result is kLcclBootstrapNoNetIf. Other values are ignored |
LCCL_NET |
IB | Socket |
Auto | Communicator creation | Pins the network backend. Case-insensitive. If that backend cannot be used, it fails with kLcclInvalidArgument |
If you do not set LCCL_SOCKET_IFNAME, the first matching interface is chosen in the order ib*; anything that is not docker or lo; docker*; lo. Loopback is the last resort. If no interface matches the name you specify, it logs a warning and treats this as no interface found (kLcclBootstrapNoNetIf); it does not substitute another interface. This selection is made once per process and then stays fixed.
The network backend tries ibverbs (IB) first and uses sockets (Socket) if ibverbs cannot be used. If libibverbs is missing at runtime or there is no RDMA device, it silently falls back to sockets.
ibverbs (InfiniBand, RoCE)¶
These variables come from NCCL and have the same meaning as the NCCL_IB_* variables of the same name. The value is fixed when it is first read in the process. For the configurations that have been verified, see Support matrix.
| Variable | Default | Description |
|---|---|---|
LCCL_IB_DISABLE |
0 |
If 1, ibverbs is not used and sockets are used |
LCCL_IB_HCA |
All | List the devices and ports to use, separated by commas (mlx5_0:1,mlx5_1). A leading = means exact match, and ^ means exclude. Communicator creation fails if the two ends of a connection use different kinds of port (InfiniBand and RoCE), so if both kinds are present, select ports of the same kind on every server |
LCCL_IB_GID_INDEX |
-1 (auto) |
GID index to use with RoCE |
LCCL_IB_ROCE_VERSION_NUM |
2 |
RoCE version preferred in automatic selection |
LCCL_IB_ADDR_FAMILY |
AF_INET |
Address family preferred in automatic GID selection (AF_INET/AF_INET6). On a RoCE network that has only IPv6 addresses, set AF_INET6 |
LCCL_IB_ADDR_RANGE |
Not set | Address range preferred in automatic GID selection (for example, 10.0.0.0/16). Specify an address in the family set by LCCL_IB_ADDR_FAMILY |
LCCL_IB_TIMEOUT |
20 |
QP transmission timeout exponent |
LCCL_IB_RETRY_CNT |
7 |
QP retry count |
LCCL_IB_SL, LCCL_IB_TC, LCCL_IB_FIFO_TC |
-1 (not set) |
Service level, traffic class, and traffic class of the control FIFO |
LCCL_IB_QPS_PER_CONNECTION |
1 |
Number of QPs per connection. All ranks must use the same value; otherwise communicator creation fails |
LCCL_IB_SPLIT_DATA_ON_QPS |
0 |
If 1, data is split across QPs |
The other tuning variables are listed with their defaults. Do not touch them unless you need to: LCCL_IB_PKEY (0), LCCL_IB_USE_INLINE (0), LCCL_IB_PCI_RELAXED_ORDERING (2, auto), LCCL_IB_ADAPTIVE_ROUTING (-2, not set), LCCL_IB_AR_THRESHOLD (8192), LCCL_IB_MERGE_VFS (1), LCCL_IB_ECE_ENABLE (1), LCCL_IB_ROUTABLE_FLID_GID_INDEX (1), LCCL_IB_MQP_RETRY_ALL (0), LCCL_IB_MQP_RETRY_CNT (34), LCCL_IB_MQP_RETRY_SLEEP_MSEC (100).
The variables below control LPU Direct RDMA. For the conditions under which it is used, see Multi-node execution.
| Variable | Default | Description |
|---|---|---|
LCCL_NET_RDMA_LEVEL |
PXB |
Maximum distance of the PCIe path between the LPU and the RNIC for using LPU Direct RDMA. One of LOC, PIX, PXB, PHB, and SYS; LOC turns it off |
LCCL_NET_RDMA_READ |
Not set | 0 turns off LPU Direct RDMA on the send side |
LCCL_GDR_FLUSH_DISABLE |
0 |
A value other than 0 turns off LPU Direct RDMA on the receive side |
LCCL_NET_FORCE_FLUSH |
1 |
0 turns off LPU Direct RDMA on the receive side. This differs from what NCCL_NET_FORCE_FLUSH=0 means in NCCL, so do not carry that setting over |
LCCL_DMABUF_POOL_SIZE_PER_CHANNEL |
64 MiB (67108864) | Per-DRAM-channel size, in bytes, of the buffers used for LPU Direct RDMA. Must be a positive multiple of 4096 that is 64 MiB or less; otherwise it does not revert to the default, and communicator creation fails with kLcclInvalidArgument when LPU Direct RDMA is used |
When a process makes its first LPU Direct RDMA connection on an LPU, it allocates (LCCL_DMABUF_POOL_SIZE_PER_CHANNEL + 64 MiB) × the number of DRAM channels of LPU memory on that LPU (1 GiB with the default), keeps it until the process exits, and gives each connection LCCL_BUFFSIZE of it. If communicator creation fails with kLcclMemDeviceAllocFail because this memory cannot be allocated or has no room left, destroy communicators you no longer use, or turn off LPU Direct RDMA with LCCL_NET_RDMA_LEVEL=LOC.
Execution¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
LCCL_BLOCKING |
0 | 1 |
1 |
Communicator creation | If 0, communicator creation, LcclCommFinalize, and the submission of a group that needs connections can return early with kLcclInProgress, and you check completion with LcclCommGetAsyncError. LcclCommDestroy and LcclCommAbort finish inside the call when they are called outside a group. Values other than 0 and 1 are ignored, and when set, it overrides blocking in LcclConfig |
LCCL_KERNEL_TIMEOUT_MS |
Milliseconds | 5000 |
Kernel execution | Maximum time to wait for one kernel to finish. On the LPU, it applies to each device kernel run, and the time includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation. When it is exceeded, the communicator is aborted and kLcclKernelSpinTimeout is returned. A value of 0 or less turns off the watchdog |
LCCL_DEVICE_LOOP_LAUNCH_BYTES |
Bytes | 256 MiB (268435456) | Operation call | Maximum number of bytes one device kernel run of LcclAllReduce, LcclReduceScatter, or LcclSend/LcclRecv (including LcclAllToAll) handles on the LPU. Larger calls of these operations run in several parts. The value is chosen so that one run finishes within LCCL_KERNEL_TIMEOUT_MS. Other operations are not split |
LCCL_LOCAL_REGISTER |
0 | 1 |
1 |
Registration call | If 0, LcclCommRegister does nothing and returns success. With the bundled network backends, registration has no effect even when this is 1 |
HA_TOOLCHAIN |
0 | 1 |
Not set | Kernel execution | A variable shared with Runtime; on the LPU, set it to 1 (Legato). If it is not set, a warning is logged once and it behaves as 1; with 0, point-to-point operations and collectives on communicators with 2 or more ranks fail |
LCCL_BLOCKING=0 is a mode in which you must check completion with LcclCommGetAsyncError. Python has no function to query asynchronous errors, so do not change it to 0 in Python.
LCCL_KERNEL_TIMEOUT_MS is a safeguard that keeps a collective from waiting forever when it was called with different arguments or a peer rank never arrives. A communicator that is aborted after the time elapses is not recovered, so destroy it and re-create it. If a normal operation is cut off in a slow environment, increase this value. On the LPU, keep it within 1 to 9999. Outside that range, the library logs a warning, and a stuck kernel can reach the driver's 12-second limit and leave the card needing a reset.
Diagnostics¶
| Variable | Type | Default | Applies at | Description |
|---|---|---|---|---|
HA_LOG_LEVEL |
0 to 6 |
2 (Info). 3 (Warn) in release builds |
Library load | Log level. Release builds are set up to raise levels below Warn to Warn, so lowering the level usually has no effect |
HA_LOG_OUTPUT_MODE |
0 | 1 |
0 (console) |
First use in the process | If 1, writes the log to the HA_LOG_PATH file |
HA_LOG_PATH |
Path | ha_runtime.log |
Library load | Log file path. Takes effect when HA_LOG_OUTPUT_MODE=1 |
HA_LOG_* are variables shared with Runtime. For the detailed values and the constraints of release builds, see Logging in the Runtime environment variables. Release builds do not emit logs at INFO level or below, so the diagnostic procedures in Troubleshooting are written so that you can follow them without INFO logs.
What environment variables do not control¶
| Setting | How to specify |
|---|---|
| Current device | Call HaSetDevice() (hart.set_device() in Python) on the calling thread before creating the communicator |
| Log settings in code | halogger.set_level() / set_file(), and the environment variables under Diagnostics in C++. The runtime reconfigures logging from the HA_LOG_* environment variables when it is first called, so call the halogger functions after hart.set_device() |
NCCL_* variables are not read
Setting them has no effect, and no warning appears. The corresponding variables are mostly the same except that the prefix is LCCL_.