Torch Backend for HyperAccel¶
Torch Backend for HyperAccel (torch_ha) registers the HyperAccel LPU as a native PyTorch device through PyTorch's
out-of-tree backend mechanism (PrivateUse1). Operations on a tensor moved to "lpu" dispatch to the LPU backend,
and torch.distributed gains an LPU backend. Model computation is compiled with torch.compile(backend="legato") and
runs on the LPU.
Where It Sits in the Stack¶
torch_ha is the layer users interact with first. It sits between PyTorch and the runtime, and vllm_hyperaccel is
built on top of it.
torch_ha sits between PyTorch and the runtime (hart) and collective communication (lccl).
At a Glance¶
| Package | torch_ha |
| Version | 0.0.6 |
| Device string | "lpu". torch.device("lpu"), torch.accelerator |
| PyTorch | 2.10.0 |
| Distributed backend | "lccl" |
| Registration | Automatic on import torch |
| Python | 3.10 |
| Status | Beta |
Documentation¶
Programming Guide
How the device behaves, which operators are available, and how to write PyTorch code that runs on the LPU.
Device Semantics · Operators · Distributed · Paged KV Cache · Migrating from CUDA · Troubleshooting
API Reference
Exactly what is registered.
Operator Coverage · Device API · Custom Operators · VMM API · Environment Variables · Support Matrix
Release Notes
Changes by version, deprecations, and known issues.
First Program¶
You do not need to import torch_ha. PyTorch discovers and registers the backend during import torch.
The following example moves two bfloat16 tensors to the LPU and adds them.
1 2 3 4 5 6 7 8 | |
Use torch.compile for model computation. The following example compiles the same addition. The first call includes
compilation time.
1 2 3 4 5 6 7 8 | |
Torch Legato Introduction covers the compile path in detail.
Before You Port a Model¶
torch_ha is in beta, and only a subset of PyTorch operators currently runs on the LPU. Run models by
compiling them with torch.compile(backend="legato").
Four operators compute on the LPU in eager mode
These are add, sub, mul, and mm on bfloat16, for prebuilt shapes with contiguous operands. Calling other
compute operators in eager mode raises NotImplementedError. multinomial and exponential_ are exceptions: they return fixed values without an error (Operator Coverage).
The other eager operators that torch_ha supports move tensors or create views and slices. They are listed in
Operator Coverage.
The legato resolvers decide coverage on the torch.compile path
Model computation is supported on the torch.compile(backend="legato") path. The conditions under which
operators compile follow the legato resolvers. See
Compile PyTorch Model with Legato.
Operators that are not compiled run in eager mode, so any compute operator other than these four fails at run time. A passing dry run does not guarantee that execution succeeds. To check, see Operators §3.3.
Installation¶
torch_ha is not installed separately. It ships in the single HyperAccel SDK wheel along with the compiler, runtime,
profiler, and serving integration. See the SDK installation guide.
Compatibility¶
This documentation is written against torch_ha 0.0.6.
| Verified version | |
|---|---|
| PyTorch | 2.10.0 |
| Python | 3.10 |
Runtime (hart) |
0.0.6 |
Collective communication (lccl) |
0.0.6 |
The PyTorch version is exact, not a minimum
A backend that registers against PyTorch's private dispatch surface works only with the PyTorch version it was built against. See the Support Matrix.