Skip to content

Environment variables

lccl reads values from environment variables and two configuration files. LCCL_* variables not listed on this page are for development, so setting them has no effect or they change without notice.

Applies at — when the value is read. Library load is when import lccl runs or the library is loaded, and First use in the process means the first value read stays fixed until the process ends. Changing an environment variable after the value has already been read has no effect.


Precedence and configuration files

A value is taken from the first of the following places where it is defined.

  1. Environment variable
  2. ~/.lccl.conf
  3. /etc/lccl.conf
  4. Built-in default

Configuration files use the NAME=VALUE format on each line. Note the following.

  • The files are read at the first lookup and loaded into the process environment. Edits to the files after the process has started have no effect.
  • Values from the files do not overwrite environment variables that are already set.
  • Comments are not supported, and whitespace around a value is not trimmed. Trailing whitespace or \r at the end of a line also becomes part of the value.
  • ~ is the home directory of the user running the process (the passwd entry), not $HOME.
  • If you set a value that is not an integer on an integer variable, it silently reverts to the default. LCCL_DMABUF_POOL_SIZE_PER_CHANNEL is an exception: with an invalid value, creating a communicator that uses LPU Direct RDMA fails.
  • Every NAME=VALUE in a file becomes a process environment variable. This includes names that do not start with LCCL_, and child processes inherit them. As a result, LCCL_SHM_PATH, which is read directly with getenv, usually sees the file value too, but variables read when the library is loaded, such as HA_LOG_*, do not see file values. Set such variables as environment variables.

The Python bindings have no LcclConfig, so in Python, environment variables and configuration files are the only way to configure a communicator.


Topology

Variable Type Default Applies at Description
LCCL_PREDEFINED_SYSINFO Path None Communicator creation Reads the hardware layout from this JSON file instead of discovering it itself. If the file does not exist or is not a regular file, it warns and discovers the layout itself (it fails if LCCL_TOPO_STRICT=1). For the format, see Topology file
LCCL_TOPO_STRICT 0 | 1 0 Communicator creation If 1, a topology file that does not exist, is not a regular file, or has no schemaVersion fails initialization instead of being passed over with a warning. Only integers are accepted, so true and yes do not turn it on
LCCL_TOPO_DUMP_FILE Path None Communicator creation Rank 0 writes the topology it discovered itself to this file as XML. When a topology file is used, nothing is written
LCCL_IGNORE_CPU_AFFINITY Whether it is set (value ignored) Not set Communicator creation Ignores the inherited CPU affinity and uses the set of CPUs close to the LPU. Does not apply when a topology file is used
LCCL_HOSTID String Hostname + boot id Communicator creation Overrides the identifier used to decide which ranks are on the same server

LCCL_HOSTID matters in containers. Ranks that report the same identifier are treated as being on one server and are connected through shared memory. The default identifier is the hostname plus a boot id that changes with every boot. Containers on the default network and Kubernetes pods each have a different hostname, so they are already classified as different servers. However, containers started on one server with the host network (host UTS) have the same hostname and boot id, so they are classified as one server, but shared memory can be used only when the two ranks see the same /dev/shm. If /dev/shm is separate per container, LCCL treats each container as a separate server and connects the containers over the network transport.

Channels and buffers

Variable Type Default Applies at Description
LCCL_MIN_NCHANNELS Integer Not set (0) Communicator creation Lower limit on the number of channels. On the LPU with 2 or more ranks, a value above 1 is ignored with a warning
LCCL_MAX_NCHANNELS Integer 1 Communicator creation Upper limit on the number of channels. A value below 1 is ignored with a warning and the default is used. On the LPU with 2 or more ranks, a value above 1 is ignored with a warning
LCCL_MIN_CHANNELS Integer 1 Communicator creation Overrides min_num_chnl in LcclConfig
LCCL_MAX_CHANNELS Integer 8 Communicator creation Overrides max_num_chnl in LcclConfig
LCCL_BUFFSIZE Bytes 4 MiB (4194304) Communicator creation Size of the transfer buffer used by the Simple protocol. It must be a multiple of 64 and cannot be made larger than the default (it can only be reduced). If either of these conditions is violated, it warns and uses the default. On the LPU with 2 or more ranks, it must also be a multiple of 4096; otherwise every collective and LcclSend/LcclRecv returns kLcclNotSupported. All ranks must use the same value
LCCL_CHUNK_SIZE Bytes 0 (not set) Operation submission Chunk size for point-to-point transfers (LcclSend/LcclRecv, LcclAllToAll). Does not apply to collectives. If you set it, use a multiple of 512 no larger than ⅛ of LCCL_BUFFSIZE (512 KiB by default), with the same value on all ranks. Other values can make point-to-point transfers fail with kLcclKernelInvalidArgs, and that communicator can no longer be used after this failure

On the LPU, a communicator with 2 or more ranks uses 1 channel regardless of these variables. A value above 1 makes the library log a communicator of N ranks on hw uses 1 channel warning.

Algorithms and protocols

Variable Type Default Applies at Description
LCCL_PROTO Name list Simple Communicator creation Protocols to allow. List LL and Simple separated by commas; a leading ^ excludes the listed ones. Case-insensitive. An empty value is treated as not set. Unknown names are ignored without an error
LCCL_ALGO Name list Ring Communicator creation Algorithms to allow. Ring is the only one, so there is no reason to change it. The format is the same as LCCL_PROTO

Be careful when you set these variables. If the allow list becomes empty (a misspelled name, or only LCCL_PROTO=LL on the LPU, which has no LL kernel), every collective fails with kLcclEnqueueNoAlgo. Values such as NCCL's tree and LL128 do not exist in this library. For the basis and scope, see Support matrix.

Transport layers

P2P and shared memory

Variable Type Default Applies at Description
LCCL_P2P_DISABLE 0 | 1 0 Communicator creation If 1, the P2P transport is not used
LCCL_P2P_LEVEL LOC | PIX | PXB | PHB | SYS Auto Communicator creation Maximum interconnect distance at which P2P is allowed. The legacy numeric form (0 and up) is also accepted. When auto, it is PXB on Intel and Zhaoxin x86 hosts and on ARM hosts, and SYS otherwise
LCCL_IGNORE_DISABLED_P2P 0 | 1 0 Communicator creation Hides the INFO message (P2P is disabled between connected LPUs) logged for each pair of LPUs for which the driver does not allow P2P. P2P for that pair is turned off regardless of this variable
LCCL_SHM_PATH Path /dev/shm When communication connections are created Directory where shared memory files are created. All ranks on a server must use the same value

P2P can be used only between ranks in the same process. In the common configuration where each rank uses its own process (torchrun, multiprocessing), the P2P transport is not selected and falls back to shared memory. LCCL_P2P_DISABLE and LCCL_P2P_LEVEL can affect how the communication path is chosen even in this configuration, where P2P is not used, so if you change them, check the result.

Shared memory is created as lccl-XXXXXX files under LCCL_SHM_PATH. When ranks on the same server run in separate processes, they communicate through this shared memory, and the capacity needed per server grows with the square of the number of ranks on that server. With the default LCCL_BUFFSIZE, 2 ranks need about 22 MiB, 4 ranks about 80 MiB, and 8 ranks about 300 MiB. The default /dev/shm of a container (64 MiB) holds at most 3 ranks, so for more ranks, increase --shm-size or point LCCL_SHM_PATH at a larger filesystem. If space or permissions are insufficient, the result is kLcclMemShmMapFail.

Network

Variable Type Default Applies at Description
LCCL_SOCKET_IFNAME Name list Auto First use in the process Network interface to use for bootstrap and for socket and ibverbs connections. eth (prefix), =eth0 (exact match), ^docker,lo (exclude). Once set, it does not fall back to automatic selection
LCCL_SOCKET_FAMILY AF_INET | AF_INET6 Auto First use in the process Filters candidate interfaces to those with an IPv4 or IPv6 address. If you specify a kind that the server does not have, no interface matches and the result is kLcclBootstrapNoNetIf. Other values are ignored
LCCL_NET IB | Socket Auto Communicator creation Pins the network backend. Case-insensitive. If that backend cannot be used, it fails with kLcclInvalidArgument

If you do not set LCCL_SOCKET_IFNAME, the first matching interface is chosen in the order ib*; anything that is not docker or lo; docker*; lo. Loopback is the last resort. If no interface matches the name you specify, it logs a warning and treats this as no interface found (kLcclBootstrapNoNetIf); it does not substitute another interface. This selection is made once per process and then stays fixed.

The network backend tries ibverbs (IB) first and uses sockets (Socket) if ibverbs cannot be used. If libibverbs is missing at runtime or there is no RDMA device, it silently falls back to sockets.

ibverbs (InfiniBand, RoCE)

These variables come from NCCL and have the same meaning as the NCCL_IB_* variables of the same name. The value is fixed when it is first read in the process. For the configurations that have been verified, see Support matrix.

Variable Default Description
LCCL_IB_DISABLE 0 If 1, ibverbs is not used and sockets are used
LCCL_IB_HCA All List the devices and ports to use, separated by commas (mlx5_0:1,mlx5_1). A leading = means exact match, and ^ means exclude. Communicator creation fails if the two ends of a connection use different kinds of port (InfiniBand and RoCE), so if both kinds are present, select ports of the same kind on every server
LCCL_IB_GID_INDEX -1 (auto) GID index to use with RoCE
LCCL_IB_ROCE_VERSION_NUM 2 RoCE version preferred in automatic selection
LCCL_IB_ADDR_FAMILY AF_INET Address family preferred in automatic GID selection (AF_INET/AF_INET6). On a RoCE network that has only IPv6 addresses, set AF_INET6
LCCL_IB_ADDR_RANGE Not set Address range preferred in automatic GID selection (for example, 10.0.0.0/16). Specify an address in the family set by LCCL_IB_ADDR_FAMILY
LCCL_IB_TIMEOUT 20 QP transmission timeout exponent
LCCL_IB_RETRY_CNT 7 QP retry count
LCCL_IB_SL, LCCL_IB_TC, LCCL_IB_FIFO_TC -1 (not set) Service level, traffic class, and traffic class of the control FIFO
LCCL_IB_QPS_PER_CONNECTION 1 Number of QPs per connection. All ranks must use the same value; otherwise communicator creation fails
LCCL_IB_SPLIT_DATA_ON_QPS 0 If 1, data is split across QPs

The other tuning variables are listed with their defaults. Do not touch them unless you need to: LCCL_IB_PKEY (0), LCCL_IB_USE_INLINE (0), LCCL_IB_PCI_RELAXED_ORDERING (2, auto), LCCL_IB_ADAPTIVE_ROUTING (-2, not set), LCCL_IB_AR_THRESHOLD (8192), LCCL_IB_MERGE_VFS (1), LCCL_IB_ECE_ENABLE (1), LCCL_IB_ROUTABLE_FLID_GID_INDEX (1), LCCL_IB_MQP_RETRY_ALL (0), LCCL_IB_MQP_RETRY_CNT (34), LCCL_IB_MQP_RETRY_SLEEP_MSEC (100).

The variables below control LPU Direct RDMA. For the conditions under which it is used, see Multi-node execution.

Variable Default Description
LCCL_NET_RDMA_LEVEL PXB Maximum distance of the PCIe path between the LPU and the RNIC for using LPU Direct RDMA. One of LOC, PIX, PXB, PHB, and SYS; LOC turns it off
LCCL_NET_RDMA_READ Not set 0 turns off LPU Direct RDMA on the send side
LCCL_GDR_FLUSH_DISABLE 0 A value other than 0 turns off LPU Direct RDMA on the receive side
LCCL_NET_FORCE_FLUSH 1 0 turns off LPU Direct RDMA on the receive side. This differs from what NCCL_NET_FORCE_FLUSH=0 means in NCCL, so do not carry that setting over
LCCL_DMABUF_POOL_SIZE_PER_CHANNEL 64 MiB (67108864) Per-DRAM-channel size, in bytes, of the buffers used for LPU Direct RDMA. Must be a positive multiple of 4096 that is 64 MiB or less; otherwise it does not revert to the default, and communicator creation fails with kLcclInvalidArgument when LPU Direct RDMA is used

When a process makes its first LPU Direct RDMA connection on an LPU, it allocates (LCCL_DMABUF_POOL_SIZE_PER_CHANNEL + 64 MiB) × the number of DRAM channels of LPU memory on that LPU (1 GiB with the default), keeps it until the process exits, and gives each connection LCCL_BUFFSIZE of it. If communicator creation fails with kLcclMemDeviceAllocFail because this memory cannot be allocated or has no room left, destroy communicators you no longer use, or turn off LPU Direct RDMA with LCCL_NET_RDMA_LEVEL=LOC.

Execution

Variable Type Default Applies at Description
LCCL_BLOCKING 0 | 1 1 Communicator creation If 0, communicator creation, LcclCommFinalize, and the submission of a group that needs connections can return early with kLcclInProgress, and you check completion with LcclCommGetAsyncError. LcclCommDestroy and LcclCommAbort finish inside the call when they are called outside a group. Values other than 0 and 1 are ignored, and when set, it overrides blocking in LcclConfig
LCCL_KERNEL_TIMEOUT_MS Milliseconds 5000 Kernel execution Maximum time to wait for one kernel to finish. On the LPU, it applies to each device kernel run, and the time includes waiting for work already queued on the calling thread's default stream and for other ranks to enter the operation. When it is exceeded, the communicator is aborted and kLcclKernelSpinTimeout is returned. A value of 0 or less turns off the watchdog
LCCL_DEVICE_LOOP_LAUNCH_BYTES Bytes 256 MiB (268435456) Operation call Maximum number of bytes one device kernel run of LcclAllReduce, LcclReduceScatter, or LcclSend/LcclRecv (including LcclAllToAll) handles on the LPU. Larger calls of these operations run in several parts. The value is chosen so that one run finishes within LCCL_KERNEL_TIMEOUT_MS. Other operations are not split
LCCL_LOCAL_REGISTER 0 | 1 1 Registration call If 0, LcclCommRegister does nothing and returns success. With the bundled network backends, registration has no effect even when this is 1
HA_TOOLCHAIN 0 | 1 Not set Kernel execution A variable shared with Runtime; on the LPU, set it to 1 (Legato). If it is not set, a warning is logged once and it behaves as 1; with 0, point-to-point operations and collectives on communicators with 2 or more ranks fail

LCCL_BLOCKING=0 is a mode in which you must check completion with LcclCommGetAsyncError. Python has no function to query asynchronous errors, so do not change it to 0 in Python.

LCCL_KERNEL_TIMEOUT_MS is a safeguard that keeps a collective from waiting forever when it was called with different arguments or a peer rank never arrives. A communicator that is aborted after the time elapses is not recovered, so destroy it and re-create it. If a normal operation is cut off in a slow environment, increase this value. On the LPU, keep it within 1 to 9999. Outside that range, the library logs a warning, and a stuck kernel can reach the driver's 12-second limit and leave the card needing a reset.

Diagnostics

Variable Type Default Applies at Description
HA_LOG_LEVEL 0 to 6 2 (Info). 3 (Warn) in release builds Library load Log level. Release builds are set up to raise levels below Warn to Warn, so lowering the level usually has no effect
HA_LOG_OUTPUT_MODE 0 | 1 0 (console) First use in the process If 1, writes the log to the HA_LOG_PATH file
HA_LOG_PATH Path ha_runtime.log Library load Log file path. Takes effect when HA_LOG_OUTPUT_MODE=1

HA_LOG_* are variables shared with Runtime. For the detailed values and the constraints of release builds, see Logging in the Runtime environment variables. Release builds do not emit logs at INFO level or below, so the diagnostic procedures in Troubleshooting are written so that you can follow them without INFO logs.


What environment variables do not control

Setting How to specify
Current device Call HaSetDevice() (hart.set_device() in Python) on the calling thread before creating the communicator
Log settings in code halogger.set_level() / set_file(), and the environment variables under Diagnostics in C++. The runtime reconfigures logging from the HA_LOG_* environment variables when it is first called, so call the halogger functions after hart.set_device()

NCCL_* variables are not read

Setting them has no effect, and no warning appears. The corresponding variables are mostly the same except that the prefix is LCCL_.


See also