Skip to content

1. Overview

This guide shows how to run PyTorch code on the HyperAccel LPU. torch_ha adds the LPU to PyTorch as the lpu device: you move tensors and models with .to("lpu") and compile model computation with torch.compile(backend="legato"). If you serve models with vllm_hyperaccel, you do not use torch_ha directly.

Read this chapter first. Later chapters assume the concepts defined here.

Contents

Section Objective
1.1 What the Backend Does The device, allocator, kernels, and distributed backend that torch_ha registers with PyTorch
1.2 Design Background Consequences of PrivateUse1 registration and import-time autoloading
1.3 Key Concepts Terms used throughout later chapters
1.4 First Program A minimal example and three rules to know from the start
1.5 Inside the Backend The components and responsibilities needed to read error messages

1.1 What the Backend Does

torch_ha makes the LPU a PyTorch device. It registers the following:

  • A device: "lpu", used through torch.device("lpu") and torch.accelerator.
  • An allocator and device guard: tensors reside in LPU memory, and .to("lpu") works.
  • Eager operators: operators that move tensors and create views and slices, plus experimental eager compute (add, sub, mul, and mm on bfloat16). Use torch.compile(backend="legato") for model computation. See Operators §3.1.
  • A distributed backend: "lccl" for torch.distributed.
  • Autocast: not used on LPU. Operators inside torch.autocast("lpu") raise NotImplementedError, so cast explicitly. See Device Semantics §Autocast.

import torch registers all of these. You do not need to import torch_ha or call an initialization function.

1.2 Design Background

Two facts explain most of the behavior you will encounter.

The backend registers as PyTorch's PrivateUse1 device. PyTorch provides one renamable device type, PrivateUse1, for out-of-tree backends. torch_ha claims it and renames it to lpu, so you use .to("lpu") and torch.accelerator like any built-in device. It also means that only one out-of-tree backend can be loaded per process, because PrivateUse1 has a single slot.

Registration happens during import torch. When PyTorch is imported, it loads the installed device backends, and torch_ha registers the LPU at that point. If registration fails, import torch still succeeds so that the rest of PyTorch keeps working, and the error goes only to the log. The lpu device is then missing, and the first .to("lpu") fails with an unknown device type.

1.3 Key Concepts

Concept Meaning
lpu device The registered device string. torch.device("lpu"), tensor.to("lpu").
PrivateUse1 PyTorch's dispatch key for out-of-tree backends. Renamed to lpu when the backend loads.
Registered operator An ATen operator with an LPU implementation registered on the eager path. A registered operator does not necessarily run on the device. See Operator Coverage.
Backend meta Metadata that describes how a tensor's storage is laid out in device memory.
Tensor kind The device layout a weight uses. nn.Linear and nn.Embedding weights are tagged automatically when moved to lpu.
lccl backend The torch.distributed backend name for LPU collective communication.

Registration failures are silent

If backend registration fails, import torch still succeeds and the failure is only logged. The symptom appears later, when torch.device("lpu") fails with an unknown device error. The actual cause is in the import-time log. See Troubleshooting.

1.4 First Program

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import torch

# No import torch_ha — registration happens during `import torch`.
x = torch.randn(64, 256, dtype=torch.bfloat16).to("lpu")
y = torch.randn(64, 256, dtype=torch.bfloat16).to("lpu")

z = x + y
print(z.device)          # lpu:0

# Bring the values back to look at them.
print(z.cpu())

In eager mode, add, sub, mul, and mm on bfloat16 compute on the device, with contiguous operands. For model computation, see Operators §3.1.

Three rules come up repeatedly:

  1. Create random tensors on the CPU, then move them. Use torch.randn(...).to("lpu") instead of torch.randn(..., device="lpu").
  2. Reading a value waits for the device. print(x), .cpu(), and .item() wait for pending work and copy the value to the host.
  3. Select a device once per rank. A process can see multiple devices, and the current device is bound per thread. Select it at startup with torch.accelerator.set_device_index(), then allocate without an index. See Distributed §4.2.

1.5 Inside the Backend

You do not need this section to use the backend, but you need it to read errors.

User code goes through the PyTorch dispatcher to the LPU kernels and allocator and down to hart; distributed calls go through torch_ha.comm and lccl down to hart

torch_ha components below the dispatcher. The operator path and the distributed path meet at hart.

Component Responsibility
Allocator Backs LPU tensors with device memory obtained from hart.
Device guard Makes the current device follow PyTorch's device context rules.
Operator registration Hand-written for operators the dispatcher requires; code-generated for compute operators.
Backend meta Preserves the device layout across view operations. A reshaped or strided tensor still records how its storage is laid out.
torch_ha.comm Registers lccl as a torch.distributed backend.

In practice:

  • The backend name in error messages is usually lpu. PrivateUse1, which appears in some low-level messages, refers to the same backend.
  • Layout errors surface in operators that reinterpret storage (view, as_strided, reshape), but the root cause is often how the tensor was created.

Next Steps