Skip to content

Support Matrix

This page lists what has been verified with torch_ha 0.0.6. Anything not listed here has not yet been verified.

Legend — ✅ Supported · 🟡 Limited · ❌ Not available

❌ marks items confirmed not to work. Items outside the verified scope are listed in Known Limitations.


Platform

Item Status Notes
PyTorch 2.10.0 ✅ Exact version required, not a minimum
Other PyTorch versions ❌ See the warning below
Python 3.10 ✅
Linux, x86_64 ✅
Windows, macOS ❌ No wheels are published
Runtime (hart) 0.0.6 ✅
Collective communication (lccl) 0.0.6 ✅

The PyTorch version is exact

The backend registers with the PyTorch dispatcher and structured kernel code generation, an internal interface whose ABI is not stable across releases. On any other PyTorch version, registration fails without a version mismatch message, and the device appears to be absent.

One out-of-tree backend per process

torch_ha occupies the PyTorch PrivateUse1 dispatch key. Run other packages that use the same key in a separate process.

Operators

Path Status Notes
torch.compile(backend="legato") ✅ The model compute path. Coverage is determined by the legato resolvers, not by this table — Compile PyTorch Model with Legato
eager — operators for the torch.compile path ✅ 26 operators for allocation, copy, views, slices, and similar tensor handling
eager — experimental compute 🟡 add, sub, mul, and mm on bfloat16. Runs for prebuilt shapes with contiguous operands
eager — other registered compute operators ❌ NotImplementedError
Unregistered operators ❌ Dispatch error

For the full list, see Operator Coverage.

Data Types

dtype Compute Storage Notes
bfloat16 ✅ ✅ Primary
float16 ❌ ✅ Moving a tensor without conversion prints a one-time warning and stores it as float16. Convert with .to("lpu", torch.bfloat16) before compute
float32 🟡 ✅ No eager compute. On the torch.compile path, support is determined by the legato resolvers
int32, int64 🟡 ✅ Indexing, comparison, and reduction outputs
bool 🟡 ✅ Output of comparison operators
float64 ❌ ❌ Rejected by device layout dtype conversion
int8, uint8 🟡 ✅ Quantized weights via torch.ops.lpu.quant_weight_*

Device Features

Feature Status Notes
.to("lpu") for tensors and modules ✅
torch.save / torch.load 🟡 Loading with map_location="cpu" gives incorrect values for some weights (for example nn.Linear, nn.Embedding). Load without map_location on a machine with the backend — Device Semantics §2.5
Views, slicing, reshape 🟡 The device layout propagates through views. Slices that device compute takes as is: Device Semantics §Slicing
Automatic weight layout for nn.Linear, nn.Embedding ✅ Applied when the module is moved to lpu
Multiple devices in one process ✅ torch.accelerator.set_device_index — Distributed §4.2
torch.compile with the LPU backend ✅ backend="legato"
Synchronization ✅ Reading a value, or torch.accelerator.synchronize()
Memory statistics ✅ torch.accelerator.memory_stats() and related functions
Creating tensors directly on lpu 🟡 Value factories (zeros, ones, full, arange, empty) work with device="lpu" (the current device) or an explicit index. Create random tensors on the CPU and move them
Autocast ❌ Operators inside torch.autocast("lpu") raise NotImplementedError naming the Autocastlpu backend, in eager mode and under torch.compile. Cast explicitly
Streams and events through PyTorch 🟡 Streams exist in hart, but PyTorch currently uses only the default stream (torch.accelerator.current_stream()); creating a custom stream (torch.Stream("lpu")) or recording an event (torch.Event) is not available yet.
Pinned host memory and asynchronous copies ❌ pin_memory() (including DataLoader(pin_memory=True)) raises NotImplementedError; non_blocking=True is accepted but every copy blocks
Paged KV cache (vmm_*) ✅ Used by the vLLM plugin — VMM API

Distributed

Feature Status Notes
dist.init_process_group("lccl") ✅
torchrun and standard environment variables ✅
all-reduce, all-gather ✅ Device kernels (tensor parallel hot path)
send / recv ✅ Pipeline parallelism. Copy path with no dtype restriction
dist.batch_isend_irecv ✅ Supports coalescing. Used for bidirectional exchange
torch.ops.lpu.lccl_* fast path ✅ For compiled graphs — Distributed §4.1
Other collective and reduction combinations 🟡 See the LCCL support matrix
DistributedDataParallel ❌ Inference only

Workloads

Workload Status
LLM inference (decoder-only transformers) ✅ Target
LLM serving through vLLM ✅
Vision and multimodal 🟡 Text path only, for the models brought up so far
Training and fine-tuning ❌ Inference only

Known Limitations

  • Model computation uses the torch.compile(backend="legato") path. In eager mode, the operators computed on the device are add, sub, mul, and mm on bfloat16, and every operand, including the output, must be contiguous. Call .contiguous() on a non-contiguous view first, or call .clone() on a slice that device compute does not take as is (Device Semantics §Slicing).
  • multinomial and exponential_ return fixed values without an error. There is no device random number generation; only stubs for dry runs are registered.
  • Unregistered operators fail with a dispatch error. This is by design: with a CPU fallback, a model running at host speed would be hard to notice.
  • The PyTorch version must match exactly, and a mismatch appears as a missing device rather than a version error.
  • Registration failures are logged, not raised. import torch succeeds in either case.
  • is_available() returns True if there is at least one device. It is equivalent to device_count() > 0.
  • Do not use torch.autocast("lpu"). Entering the context succeeds, but operators inside it raise NotImplementedError naming the Autocastlpu backend, in eager mode and under torch.compile. Cast explicitly.
  • PyTorch currently uses only the default stream (torch.accelerator.current_stream()). Streams exist in the runtime layer, but creating a custom stream (torch.Stream("lpu")) or recording an event (torch.Event) is not available yet.
  • The LPU is for inference only. Training, fine-tuning, and DistributedDataParallel are outside the supported scope.
  • Windows and macOS are not target platforms.
  • float16 is supported for storage only. Moving a float16 tensor to the LPU without conversion prints a one-time warning and copies it as float16. Convert it with .to("lpu", torch.bfloat16) before compute.

See Also