1. Overview¶
This guide shows how to run PyTorch code on the HyperAccel LPU. torch_ha adds the LPU to PyTorch as the lpu
device: you move tensors and models with .to("lpu") and compile model computation with
torch.compile(backend="legato"). If you serve models with vllm_hyperaccel, you do not use torch_ha directly.
Read this chapter first. Later chapters assume the concepts defined here.
Contents¶
| Section | Objective |
|---|---|
| 1.1 What the Backend Does | The device, allocator, kernels, and distributed backend that torch_ha registers with PyTorch |
| 1.2 Design Background | Consequences of PrivateUse1 registration and import-time autoloading |
| 1.3 Key Concepts | Terms used throughout later chapters |
| 1.4 First Program | A minimal example and three rules to know from the start |
| 1.5 Inside the Backend | The components and responsibilities needed to read error messages |
1.1 What the Backend Does¶
torch_ha makes the LPU a PyTorch device. It registers the following:
- A device:
"lpu", used throughtorch.device("lpu")andtorch.accelerator. - An allocator and device guard: tensors reside in LPU memory, and
.to("lpu")works. - Eager operators: operators that move tensors and create views and slices, plus experimental eager compute
(
add,sub,mul, andmmonbfloat16). Usetorch.compile(backend="legato")for model computation. See Operators §3.1. - A distributed backend:
"lccl"fortorch.distributed. - Autocast: not used on LPU.
Operators inside
torch.autocast("lpu")raiseNotImplementedError, so cast explicitly. See Device Semantics §Autocast.
import torch registers all of these. You do not need to import torch_ha or call an initialization function.
1.2 Design Background¶
Two facts explain most of the behavior you will encounter.
The backend registers as PyTorch's PrivateUse1 device. PyTorch provides one renamable device type, PrivateUse1, for
out-of-tree backends. torch_ha claims it and renames it to lpu, so you use .to("lpu") and torch.accelerator like any
built-in device. It also means that only one out-of-tree backend can be loaded per process, because PrivateUse1
has a single slot.
Registration happens during import torch. When PyTorch is imported, it loads the installed device backends,
and torch_ha registers the LPU at that point. If registration fails, import torch still succeeds so that the rest
of PyTorch keeps working, and the error goes only to the log. The lpu device is then missing, and the first
.to("lpu") fails with an unknown device type.
1.3 Key Concepts¶
| Concept | Meaning |
|---|---|
lpu device |
The registered device string. torch.device("lpu"), tensor.to("lpu"). |
PrivateUse1 |
PyTorch's dispatch key for out-of-tree backends. Renamed to lpu when the backend loads. |
| Registered operator | An ATen operator with an LPU implementation registered on the eager path. A registered operator does not necessarily run on the device. See Operator Coverage. |
| Backend meta | Metadata that describes how a tensor's storage is laid out in device memory. |
| Tensor kind | The device layout a weight uses. nn.Linear and nn.Embedding weights are tagged automatically when moved to lpu. |
lccl backend |
The torch.distributed backend name for LPU collective communication. |
Registration failures are silent
If backend registration fails, import torch still succeeds and the failure is only logged. The symptom appears
later, when torch.device("lpu") fails with an unknown device error. The actual cause is in the import-time log.
See Troubleshooting.
1.4 First Program¶
1 2 3 4 5 6 7 8 9 10 11 | |
In eager mode, add, sub, mul, and mm on bfloat16 compute on the device, with contiguous operands. For model
computation, see
Operators §3.1.
Three rules come up repeatedly:
- Create random tensors on the CPU, then move them. Use
torch.randn(...).to("lpu")instead oftorch.randn(..., device="lpu"). - Reading a value waits for the device.
print(x),.cpu(), and.item()wait for pending work and copy the value to the host. - Select a device once per rank. A process can see multiple devices, and the current device is bound per thread.
Select it at startup with
torch.accelerator.set_device_index(), then allocate without an index. See Distributed §4.2.
1.5 Inside the Backend¶
You do not need this section to use the backend, but you need it to read errors.
torch_ha components below the dispatcher. The operator path and the distributed path meet at hart.
| Component | Responsibility |
|---|---|
| Allocator | Backs LPU tensors with device memory obtained from hart. |
| Device guard | Makes the current device follow PyTorch's device context rules. |
| Operator registration | Hand-written for operators the dispatcher requires; code-generated for compute operators. |
| Backend meta | Preserves the device layout across view operations. A reshaped or strided tensor still records how its storage is laid out. |
torch_ha.comm |
Registers lccl as a torch.distributed backend. |
In practice:
- The backend name in error messages is usually
lpu.PrivateUse1, which appears in some low-level messages, refers to the same backend. - Layout errors surface in operators that reinterpret storage (
view,as_strided,reshape), but the root cause is often how the tensor was created.
Next Steps¶
- Device Semantics: how the device behaves
- Operators: what is registered, and what happens when an operator is not
- Distributed: using multiple devices
- Paged KV Cache: mapping KV cache blocks with virtual memory
- Migrating from CUDA: differences from CUDA code
- Error Handling: how to read exceptions and logs
- Troubleshooting: index by symptom