Support Matrix¶
This page lists what has been verified with torch_ha 0.0.6. Anything not listed here has not yet been verified.
Legend — ✅ Supported · 🟡 Limited · ❌ Not available
❌ marks items confirmed not to work. Items outside the verified scope are listed in Known Limitations.
Platform¶
| Item | Status | Notes |
|---|---|---|
| PyTorch 2.10.0 | ✅ | Exact version required, not a minimum |
| Other PyTorch versions | ❌ | See the warning below |
| Python 3.10 | ✅ | |
| Linux, x86_64 | ✅ | |
| Windows, macOS | ❌ | No wheels are published |
Runtime (hart) 0.0.6 |
✅ | |
Collective communication (lccl) 0.0.6 |
✅ |
The PyTorch version is exact
The backend registers with the PyTorch dispatcher and structured kernel code generation, an internal interface whose ABI is not stable across releases. On any other PyTorch version, registration fails without a version mismatch message, and the device appears to be absent.
One out-of-tree backend per process
torch_ha occupies the PyTorch PrivateUse1 dispatch key. Run other packages that use the same key in a
separate process.
Operators¶
| Path | Status | Notes |
|---|---|---|
torch.compile(backend="legato") |
✅ | The model compute path. Coverage is determined by the legato resolvers, not by this table — Compile PyTorch Model with Legato |
eager — operators for the torch.compile path |
✅ | 26 operators for allocation, copy, views, slices, and similar tensor handling |
| eager — experimental compute | 🟡 | add, sub, mul, and mm on bfloat16. Runs for prebuilt shapes with contiguous operands |
| eager — other registered compute operators | ❌ | NotImplementedError |
| Unregistered operators | ❌ | Dispatch error |
For the full list, see Operator Coverage.
Data Types¶
| dtype | Compute | Storage | Notes |
|---|---|---|---|
bfloat16 |
✅ | ✅ | Primary |
float16 |
❌ | ✅ | Moving a tensor without conversion prints a one-time warning and stores it as float16. Convert with .to("lpu", torch.bfloat16) before compute |
float32 |
🟡 | ✅ | No eager compute. On the torch.compile path, support is determined by the legato resolvers |
int32, int64 |
🟡 | ✅ | Indexing, comparison, and reduction outputs |
bool |
🟡 | ✅ | Output of comparison operators |
float64 |
❌ | ❌ | Rejected by device layout dtype conversion |
int8, uint8 |
🟡 | ✅ | Quantized weights via torch.ops.lpu.quant_weight_* |
Device Features¶
| Feature | Status | Notes |
|---|---|---|
.to("lpu") for tensors and modules |
✅ | |
torch.save / torch.load |
🟡 | Loading with map_location="cpu" gives incorrect values for some weights (for example nn.Linear, nn.Embedding). Load without map_location on a machine with the backend — Device Semantics §2.5 |
| Views, slicing, reshape | 🟡 | The device layout propagates through views. Slices that device compute takes as is: Device Semantics §Slicing |
Automatic weight layout for nn.Linear, nn.Embedding |
✅ | Applied when the module is moved to lpu |
| Multiple devices in one process | ✅ | torch.accelerator.set_device_index — Distributed §4.2 |
torch.compile with the LPU backend |
✅ | backend="legato" |
| Synchronization | ✅ | Reading a value, or torch.accelerator.synchronize() |
| Memory statistics | ✅ | torch.accelerator.memory_stats() and related functions |
Creating tensors directly on lpu |
🟡 | Value factories (zeros, ones, full, arange, empty) work with device="lpu" (the current device) or an explicit index. Create random tensors on the CPU and move them |
| Autocast | ❌ | Operators inside torch.autocast("lpu") raise NotImplementedError naming the Autocastlpu backend, in eager mode and under torch.compile. Cast explicitly |
| Streams and events through PyTorch | 🟡 | Streams exist in hart, but PyTorch currently uses only the default stream (torch.accelerator.current_stream()); creating a custom stream (torch.Stream("lpu")) or recording an event (torch.Event) is not available yet. |
| Pinned host memory and asynchronous copies | ❌ | pin_memory() (including DataLoader(pin_memory=True)) raises NotImplementedError; non_blocking=True is accepted but every copy blocks |
Paged KV cache (vmm_*) |
✅ | Used by the vLLM plugin — VMM API |
Distributed¶
| Feature | Status | Notes |
|---|---|---|
dist.init_process_group("lccl") |
✅ | |
torchrun and standard environment variables |
✅ | |
| all-reduce, all-gather | ✅ | Device kernels (tensor parallel hot path) |
| send / recv | ✅ | Pipeline parallelism. Copy path with no dtype restriction |
dist.batch_isend_irecv |
✅ | Supports coalescing. Used for bidirectional exchange |
torch.ops.lpu.lccl_* fast path |
✅ | For compiled graphs — Distributed §4.1 |
| Other collective and reduction combinations | 🟡 | See the LCCL support matrix |
DistributedDataParallel |
❌ | Inference only |
Workloads¶
| Workload | Status |
|---|---|
| LLM inference (decoder-only transformers) | ✅ Target |
| LLM serving through vLLM | ✅ |
| Vision and multimodal | 🟡 Text path only, for the models brought up so far |
| Training and fine-tuning | ❌ Inference only |
Known Limitations¶
- Model computation uses the
torch.compile(backend="legato")path. In eager mode, the operators computed on the device areadd,sub,mul, andmmonbfloat16, and every operand, including the output, must be contiguous. Call.contiguous()on a non-contiguous view first, or call.clone()on a slice that device compute does not take as is (Device Semantics §Slicing). multinomialandexponential_return fixed values without an error. There is no device random number generation; only stubs for dry runs are registered.- Unregistered operators fail with a dispatch error. This is by design: with a CPU fallback, a model running at host speed would be hard to notice.
- The PyTorch version must match exactly, and a mismatch appears as a missing device rather than a version error.
- Registration failures are logged, not raised.
import torchsucceeds in either case. is_available()returnsTrueif there is at least one device. It is equivalent todevice_count() > 0.- Do not use
torch.autocast("lpu"). Entering the context succeeds, but operators inside it raiseNotImplementedErrornaming theAutocastlpubackend, in eager mode and undertorch.compile. Cast explicitly. - PyTorch currently uses only the default stream (
torch.accelerator.current_stream()). Streams exist in the runtime layer, but creating a custom stream (torch.Stream("lpu")) or recording an event (torch.Event) is not available yet. - The LPU is for inference only. Training, fine-tuning, and
DistributedDataParallelare outside the supported scope. - Windows and macOS are not target platforms.
float16is supported for storage only. Moving afloat16tensor to the LPU without conversion prints a one-time warning and copies it as float16. Convert it with.to("lpu", torch.bfloat16)before compute.