Skip to content

Torch Backend for HyperAccel

Torch Backend for HyperAccel (torch_ha) registers the HyperAccel LPU as a native PyTorch device through PyTorch's out-of-tree backend mechanism (PrivateUse1). Operations on a tensor moved to "lpu" dispatch to the LPU backend, and torch.distributed gains an LPU backend. Model computation is compiled with torch.compile(backend="legato") and runs on the LPU.


Where It Sits in the Stack

torch_ha is the layer users interact with first. It sits between PyTorch and the runtime, and vllm_hyperaccel is built on top of it.

Layered stack: user PyTorch code and vllm_hyperaccel go through PyTorch down to torch_ha, which reaches the LPU through the hart runtime and lccl collective communication

torch_ha sits between PyTorch and the runtime (hart) and collective communication (lccl).


At a Glance

Package torch_ha
Version 0.0.6
Device string "lpu". torch.device("lpu"), torch.accelerator
PyTorch 2.10.0
Distributed backend "lccl"
Registration Automatic on import torch
Python 3.10
Status Beta

Documentation

  • 📋 Release Notes

    Changes by version, deprecations, and known issues.


First Program

You do not need to import torch_ha. PyTorch discovers and registers the backend during import torch.

The following example moves two bfloat16 tensors to the LPU and adds them.

1
2
3
4
5
6
7
8
import torch

x = torch.randn(64, 256, dtype=torch.bfloat16).to("lpu")
y = torch.randn(64, 256, dtype=torch.bfloat16).to("lpu")
z = x + y

print(z.device)          # lpu:0
print(z.cpu())

Use torch.compile for model computation. The following example compiles the same addition. The first call includes compilation time.

1
2
3
4
5
6
7
8
@torch.compile(backend="legato", dynamic=False, options={"legato.device": "bertha"})
def add(a, b):
    return a + b

with torch.no_grad():
    z = add(x, y)

print(z.cpu())

Torch Legato Introduction covers the compile path in detail.


Before You Port a Model

torch_ha is in beta, and only a subset of PyTorch operators currently runs on the LPU. Run models by compiling them with torch.compile(backend="legato").

Four operators compute on the LPU in eager mode

These are add, sub, mul, and mm on bfloat16, for prebuilt shapes with contiguous operands. Calling other compute operators in eager mode raises NotImplementedError. multinomial and exponential_ are exceptions: they return fixed values without an error (Operator Coverage).

The other eager operators that torch_ha supports move tensors or create views and slices. They are listed in Operator Coverage.

The legato resolvers decide coverage on the torch.compile path

Model computation is supported on the torch.compile(backend="legato") path. The conditions under which operators compile follow the legato resolvers. See Compile PyTorch Model with Legato.

Operators that are not compiled run in eager mode, so any compute operator other than these four fails at run time. A passing dry run does not guarantee that execution succeeds. To check, see Operators §3.3.


Installation

torch_ha is not installed separately. It ships in the single HyperAccel SDK wheel along with the compiler, runtime, profiler, and serving integration. See the SDK installation guide.


Compatibility

This documentation is written against torch_ha 0.0.6.

Verified version
PyTorch 2.10.0
Python 3.10
Runtime (hart) 0.0.6
Collective communication (lccl) 0.0.6

The PyTorch version is exact, not a minimum

A backend that registers against PyTorch's private dispatch surface works only with the PyTorch version it was built against. See the Support Matrix.