Skip to content

5. Multi-Device Programming

The device is determined not by an argument of the API call but by the device set for the calling thread. That is why mem_alloc() (C++: HaMemAlloc()) and launch_kernel_with_binary_path() (C++: HaLaunchKernel()) have no device argument. You do not have to pass a device on every call, but you must set the device each thread uses yourself. This chapter covers how to do that and what to watch out for.

Responsibilities are split as follows.

  • Runtime — mem_free() (C++: HaMemFree()) and mem_cpy() (C++: HaMemCpy()), which take pointers, operate on the device the pointer belongs to. A copy between devices is also done with mem_cpy(); there is no separate peer copy API.
  • Application — Set the device each thread uses with set_device() (C++: HaSetDevice()).

When you extend single-device code to multiple devices, problems mostly arise in two places. A newly created thread does not inherit the parent thread's device setting, so it fails with an error right away if it does not call set_device(). A device reset invalidates the old handles and pointers. If a new allocation after the reset gets the same address, an old pointer silently points to the new allocation, which makes the cause hard to find. The sections below cover the cause and remedy for each.

Contents

Section Objective
5.1 Current Device Current device rules and their behavioral consequences
5.2 Two Structures for Multi-Device Programs One thread per device vs. one thread switching between devices
5.3 Virtual Device ID Management separate from physical IDs, and the mechanism that prevents index holes
5.4 Data Movement Between Devices Conditions for peer transfer and the boundary with collective communication
5.5 Reset in Multi-Device Processes Characteristics of a reset operation that affects the entire process

5.1 Current Device

Most major entry points, including mem_alloc() (C++: HaMemAlloc()) and launch_kernel_with_binary_path() (C++: HaLaunchKernel()), have no parameter that specifies the device directly. Instead, these APIs operate on the current device set beforehand through the set_device() (C++: HaSetDevice()) interface, and no default is provided.

1
2
3
4
5
6
count = hart.get_device_count()
hart.set_device(0)
buf0 = hart.mem_alloc(size)     # Allocate on device 0

hart.set_device(1)
buf1 = hart.mem_alloc(size)     # Allocate on device 1
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
int count;
HaGetDeviceCount(&count);

HaSetDevice(0);
HaDevicePtr buf0;
HaMemAlloc(&buf0, size, /*is_interleaved=*/false);   // Allocate on device 0

HaSetDevice(1);
HaDevicePtr buf1;
HaMemAlloc(&buf1, size, /*is_interleaved=*/false);   // Allocate on device 1

The key architectural consequences of this design are as follows.

  • Newly created threads have no device — A newly created thread has no default current device. Every worker thread that calls the runtime API must explicitly call set_device() (C++: HaSetDevice()) right after it starts. A thread that skips this and attempts a runtime operation fails immediately with a HaDeviceInvalidArgument error. This constraint applies without exception not only to threads the application developer creates directly, but also to background thread pools and worker threads created internally by external frameworks.
  • Device selection is independent of execution synchronization — The current device setting is not an execution synchronization lock. When a thread switches its current device to a different number, the runtime does not block and wait for asynchronous work already issued and in progress on the previous device to complete.

5.2 Two Structures for Multi-Device Programs

Structure A and Structure B compared

One thread per device (Structure A) and one thread switching devices (Structure B)

Structure A: One thread per device (recommended). Each worker thread sets its device once at startup and never changes it. This is also the natural execution model the runtime assumes. Per-device executors run completely independently of each other, and there is no state that could be mistakenly shared between threads.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import threading
import hart

def worker(device_id: int):
    hart.set_device(device_id)
    stream = hart.stream_create()
    while True:
        run_step(stream)

threads = [
    threading.Thread(target=worker, args=(i,))
    for i in range(hart.get_device_count())
]
for t in threads:
    t.start()
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
#include <thread>
#include <vector>

void Worker(int device_id) {
  HaSetDevice(device_id);
  HaStream_t stream = nullptr;
  HaStreamCreate(&stream);
  while (true) {
    RunStep(stream);
  }
}

int count;
HaGetDeviceCount(&count);
std::vector<std::thread> threads;
for (int i = 0; i < count; ++i) {
  threads.emplace_back(Worker, i);
}
for (auto &t : threads) {
  t.join();
}

Structure B: A single thread iterates over devices and switches between them. This structure serializes host-side execution, so it is not suitable for a steady-state serving loop. It is useful, however, when you need to iterate over multiple devices temporarily, such as during initialization or cleanup.

1
2
3
for device_id in range(hart.get_device_count()):
    hart.set_device(device_id)
    hart.launch_kernel_with_binary_path(kernel_path, args)
1
2
3
4
5
6
int count;
HaGetDeviceCount(&count);
for (int device_id = 0; device_id < count; ++device_id) {
  HaSetDevice(device_id);
  HaLaunchKernel(kernel_path, args);
}

5.3 Virtual Device ID

As the example code for both structures shows, the index used when iterating over devices always falls safely within the range [0, HaGetDeviceCount()). This is always a valid range because the value passed to HaSetDevice() is not the hardware's physical ID but a virtual ID assigned by the runtime.

  • Managed separately from hardware physical IDs — The runtime assigns virtual ID k to the k-th device in the order devices are found during the device scan. Physical IDs can be non-contiguous depending on the system configuration, and some devices may fail to initialize, so hardware physical IDs and software virtual IDs are kept strictly separate.
  • No index holes on initialization failure — Even if a particular device fails to initialize, the runtime does not stop and keeps running with the healthy devices. For example, on a system with 8 devices installed, if HaGetDeviceCount() returns 7, one device failed to initialize and the other 7 devices initialized normally. In this case the runtime assigns contiguous virtual IDs starting from 0 again to only the devices that initialized successfully, so no index holes appear in the virtual IDs. This is why the for iteration pattern shown above is always safe.

Virtual ID mapping that skips a device that failed to initialize

Example virtual ID mapping when physical device 1 of 8 fails to initialize

Virtual ID assignment can change

Do not assume that a virtual ID always refers to a specific physical device. Even on the same machine, the mapping can change depending on the execution environment or the device configuration. When you diagnose a hardware fault or need to identify a specific card directly, read its PCI address (PCI_DOMAIN, PCI_BUS, PCI_SLOT, PCI_FUNC) with get_device_attribute() (C++: HaGetDeviceAttribute()).

5.4 Data Movement Between Devices

Data is copied between devices by calling mem_cpy() with the direction set to DeviceToDevice (Python: hart.HaMemCpyDirection.DeviceToDevice / C++: HaMemCpyDirection::DeviceToDevice).

1
hart.mem_cpy(dst_on_device1, src_on_device0, size, hart.HaMemCpyDirection.DeviceToDevice)
1
HaMemCpy(dst_on_device1, src_on_device0, size, HaMemCpyDirection::DeviceToDevice);

The caller does not need to specify device IDs. The runtime identifies the owning devices automatically by analyzing the pointer addresses and routes accordingly; if the source and destination devices differ, it switches internally to a peer transfer.

Peer Transfer Limitations

If the devices do not support peer transfer, a device-to-device copy fails with HaMemCpyNotSupported; there is no automatic fallback through host memory. In that case, split the copy yourself into two copies, D2H and H2D.

When you copy between devices with mem_cpy_async() (C++: HaMemCpyAsync()), pass a stream of the source or the destination device. A stream that belongs to neither device is rejected with HaMemCpyNotSupported. Synchronize that stream before resetting either device, so that neither is reset while the copy is still running.

Use LCCL for Collective Communication

Point-to-point copies are suited to one-off transfers. Collective communication patterns such as all-reduce and all-gather are the job of LCCL. LCCL builds ring channels efficiently and overlaps data transfer with computation. Implementing collective communication yourself by chaining multiple mem_cpy() (C++: HaMemCpy()) calls is not recommended for performance.

5.5 Reset in Multi-Device Processes

reset_all_devices() (C++: HaResetAllDevices()) resets every device attached to the process without exception, including devices currently in use by other threads. It is a global operation that affects the entire process, not limited to a particular device or thread.

Do not reset while other threads are running

A reset invalidates the allocations, streams, and events of all devices. In a multithreaded process, stop all workers first. There is no per-thread reset, and the runtime does not stop other threads for you.


Next Steps