5. Multi-Device Programming¶
The device is determined not by an argument of the API call but by the device set for the calling thread. That is why
mem_alloc() (C++: HaMemAlloc()) and launch_kernel_with_binary_path() (C++: HaLaunchKernel()) have no
device argument. You do not have to pass a device on every call, but you must set the device each thread uses
yourself. This chapter covers how to do that and what to watch out for.
Responsibilities are split as follows.
- Runtime —
mem_free()(C++:HaMemFree()) andmem_cpy()(C++:HaMemCpy()), which take pointers, operate on the device the pointer belongs to. A copy between devices is also done withmem_cpy(); there is no separate peer copy API. - Application — Set the device each thread uses with
set_device()(C++:HaSetDevice()).
When you extend single-device code to multiple devices, problems mostly arise in two places. A newly created thread
does not inherit the parent thread's device setting, so it fails with an error right away if it does not call
set_device(). A device reset invalidates the old handles and pointers. If a new allocation after the reset gets the
same address, an old pointer silently points to the new allocation, which makes the cause hard to find. The sections
below cover the cause and remedy for each.
Contents¶
| Section | Objective |
|---|---|
| 5.1 Current Device | Current device rules and their behavioral consequences |
| 5.2 Two Structures for Multi-Device Programs | One thread per device vs. one thread switching between devices |
| 5.3 Virtual Device ID | Management separate from physical IDs, and the mechanism that prevents index holes |
| 5.4 Data Movement Between Devices | Conditions for peer transfer and the boundary with collective communication |
| 5.5 Reset in Multi-Device Processes | Characteristics of a reset operation that affects the entire process |
5.1 Current Device¶
Most major entry points, including mem_alloc() (C++: HaMemAlloc()) and launch_kernel_with_binary_path() (C++:
HaLaunchKernel()), have no parameter that specifies the device directly. Instead, these APIs
operate on the current device set beforehand through the set_device() (C++: HaSetDevice())
interface, and no
default is provided.
1 2 3 4 5 6 | |
1 2 3 4 5 6 7 8 9 10 | |
The key architectural consequences of this design are as follows.
- Newly created threads have no device — A newly created thread has no default
current device. Every worker thread that calls the runtime API
must explicitly call
set_device()(C++:HaSetDevice()) right after it starts. A thread that skips this and attempts a runtime operation fails immediately with aHaDeviceInvalidArgumenterror. This constraint applies without exception not only to threads the application developer creates directly, but also to background thread pools and worker threads created internally by external frameworks. - Device selection is independent of execution synchronization — The current device setting is not an execution synchronization lock. When a thread switches its current device to a different number, the runtime does not block and wait for asynchronous work already issued and in progress on the previous device to complete.
5.2 Two Structures for Multi-Device Programs¶
One thread per device (Structure A) and one thread switching devices (Structure B)
Structure A: One thread per device (recommended). Each worker thread sets its device once at startup and never changes it. This is also the natural execution model the runtime assumes. Per-device executors run completely independently of each other, and there is no state that could be mistakenly shared between threads.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | |
Structure B: A single thread iterates over devices and switches between them. This structure serializes host-side execution, so it is not suitable for a steady-state serving loop. It is useful, however, when you need to iterate over multiple devices temporarily, such as during initialization or cleanup.
1 2 3 | |
1 2 3 4 5 6 | |
5.3 Virtual Device ID¶
As the example code for both structures shows, the index used when iterating over devices
always falls safely within the range [0, HaGetDeviceCount()).
This is always a valid range because the value passed to HaSetDevice()
is not the hardware's physical ID but a virtual ID assigned by the runtime.
- Managed separately from hardware physical IDs — The runtime assigns virtual ID
kto thek-th device in the order devices are found during the device scan. Physical IDs can be non-contiguous depending on the system configuration, and some devices may fail to initialize, so hardware physical IDs and software virtual IDs are kept strictly separate. - No index holes on initialization failure — Even if a particular device fails to
initialize, the runtime does not stop and keeps running with the healthy
devices. For example, on a system with 8 devices installed, if
HaGetDeviceCount()returns 7, one device failed to initialize and the other 7 devices initialized normally. In this case the runtime assigns contiguous virtual IDs starting from 0 again to only the devices that initialized successfully, so no index holes appear in the virtual IDs. This is why theforiteration pattern shown above is always safe.
Example virtual ID mapping when physical device 1 of 8 fails to initialize
Virtual ID assignment can change
Do not assume that a virtual ID always refers to a specific physical device.
Even on the same machine, the mapping can change depending on the execution environment or the device
configuration. When you diagnose a hardware fault or need to identify a specific card directly,
read its PCI address (PCI_DOMAIN, PCI_BUS, PCI_SLOT, PCI_FUNC) with get_device_attribute()
(C++: HaGetDeviceAttribute()).
5.4 Data Movement Between Devices¶
Data is copied between devices by calling mem_cpy() with the direction set to DeviceToDevice (Python: hart.HaMemCpyDirection.DeviceToDevice
/ C++: HaMemCpyDirection::DeviceToDevice).
1 | |
1 | |
The caller does not need to specify device IDs. The runtime identifies the owning devices automatically by analyzing the pointer addresses and routes accordingly; if the source and destination devices differ, it switches internally to a peer transfer.
Peer Transfer Limitations¶
If the devices do not support peer transfer, a device-to-device copy fails with HaMemCpyNotSupported;
there is no automatic fallback through host memory. In that case, split the copy yourself into two
copies, D2H and H2D.
When you copy between devices with mem_cpy_async() (C++: HaMemCpyAsync()),
pass a stream of the source or the destination device. A stream that belongs to neither device is rejected
with HaMemCpyNotSupported. Synchronize that stream before resetting either device, so that neither is reset
while the copy is still running.
Use LCCL for Collective Communication¶
Point-to-point copies are suited to one-off transfers. Collective communication patterns such as all-reduce
and all-gather are the job of LCCL.
LCCL builds ring channels efficiently and overlaps data transfer
with computation. Implementing collective communication yourself by chaining
multiple mem_cpy() (C++:
HaMemCpy()) calls is not
recommended for performance.
5.5 Reset in Multi-Device Processes¶
reset_all_devices() (C++: HaResetAllDevices()) resets every device attached to the process
without exception, including devices currently in use by other
threads. It is a global operation that affects the entire process, not limited to a
particular device or thread.
Do not reset while other threads are running
A reset invalidates the allocations, streams, and events of all devices. In a multithreaded process, stop all workers first. There is no per-thread reset, and the runtime does not stop other threads for you.