Skip to content

6. Migrating from CUDA

Most PyTorch code that runs on GPUs is written for the cuda device. This chapter covers what to change, and what you can keep, when moving cuda code to lpu.

Contents

Section Objective
6.1 Name Mapping CUDA APIs and their LPU counterparts
6.2 Differences That Require Code Changes Areas that need code changes, including the compute path, operator coverage, tensor creation, and synchronization
6.3 What You Do Not Need to Change PyTorch usage patterns that work unchanged
6.4 Migration Procedure The recommended order for migration
6.5 Example The same code for CUDA and LPU, side by side

6.1 Name Mapping

CUDA LPU
torch.device("cuda") torch.device("lpu")
tensor.cuda() tensor.to("lpu")
torch.cuda.is_available() torch.accelerator.is_available()
torch.cuda.set_device(i) torch.accelerator.set_device_index(i)
torch.cuda.device_count() torch.accelerator.device_count()
torch.cuda.device(i) torch.accelerator.device_index(i)
torch.compile(model) torch.compile(model, backend="legato", options={"legato.device": "bertha"})
torch.autocast("cuda") Cast explicitly with .to(torch.bfloat16). See §6.2
dist.init_process_group("nccl") dist.init_process_group("lccl")
torch.cuda.synchronize() torch.accelerator.synchronize() or reading a value back. See §6.2
torch.cuda.Stream / Event PyTorch uses only the default stream (torch.accelerator.current_stream()). torch.Stream("lpu") and torch.Event are not available yet
torch.cuda.memory_allocated() torch.accelerator.memory_allocated()
import torch Unchanged. import torch_ha is not required

6.2 Differences That Require Code Changes

Area CUDA LPU
Model compute path Runs on the GPU in eager mode Compile with torch.compile(backend="legato") to run on the device. The default backend (inductor) cannot generate lpu kernels and fails. Operators
Operator coverage Effectively all of ATen Decided by the legato resolvers on the torch.compile path. Eager compute is supported experimentally for only four operators: add, sub, mul, and mm on bfloat16
Unregistered operators Rarely encountered Raises a dispatch error that names the lpu backend
Tensor creation torch.randn(..., device="cuda") Value factories work with device="lpu" (which resolves to the current device) or an explicit index. Create random tensors on the CPU, then move them with .to("lpu")
Explicit synchronization torch.cuda.synchronize() Read a value back (.cpu(), .item()) or call torch.accelerator.synchronize()
Multiple devices in one process torch.cuda.set_device(i) Supported. torch.accelerator.set_device_index(i). Distributed §4.2
Autocast Changes the dtype operators run in Operators inside torch.autocast("lpu") raise NotImplementedError. Cast explicitly with .to(torch.bfloat16)
Collective coverage All NCCL operations and reductions Only the collectives and reductions LCCL implements. reduce, gather, scatter, and all_to_all are not available. LCCL matrix
Training Supported Inference only

6.3 What You Do Not Need to Change

The following work as they do with CUDA:

  • .to("lpu") on tensors and modules, and .cpu() to bring results back to the host
  • torch.save / torch.load on LPU tensors (for loading with map_location="cpu", see Device Semantics §2.5)
  • torch.distributed with the standard environment variables and torchrun
  • Selecting the device once per rank and then allocating without an index
  • Views, slicing, and reshape on LPU tensors (device compute takes only some slices as is; see Device Semantics §Slicing)
  • bfloat16 module casts (float16 is storage-only)

6.4 Migration Procedure

  1. Replace device strings. cuda → lpu, nccl → lccl.
  2. Move random tensor creation off the device. Value factories (zeros, full, and so on) work with device="lpu" (which resolves to the current device) or an explicit index. Instead of torch.randn(..., device=dev), create the tensor on the CPU and move it with .to(dev).
  3. Check your model with a legato dry run on the CPU. See Operators §3.3 for the procedure. This step decides whether the model can be ported, so do it before the remaining steps.
  4. Compile the model's computation with torch.compile(backend="legato"). The default backend (inductor) cannot generate lpu kernels. See Compile PyTorch Model with Legato.
  5. Remove stream code, and switch memory statistics to torch.accelerator. Operators run on the default stream, so removing stream code does not change behavior.
  6. Replace torch.cuda.synchronize(). Call out.cpu() or torch.accelerator.synchronize() where you need to wait for completion.
  7. Before you design around collectives and reductions, check that the LCCL matrix supports them.
  8. Cast explicitly instead of relying on autocast.

Do not carry over performance assumptions

The relative cost of kernels differs from that on GPUs, so an implementation that was faster on a GPU is not necessarily faster on the LPU. Measure with haprof.

6.5 Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import torch
import torch.distributed as dist

dist.init_process_group("nccl")
torch.cuda.set_device(rank)

model = MyModel().cuda().half()
x = torch.randn(B, S, D, device="cuda", dtype=torch.float16)

with torch.autocast("cuda", dtype=torch.float16):
    out = model(x)

dist.all_reduce(out, op=dist.ReduceOp.SUM)
torch.cuda.synchronize()
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import os

import torch
import torch.distributed as dist

# The backend binds rank r to lpu:<r>.
# Select the same device first so that tensors are placed there too.
local_rank = int(os.environ["LOCAL_RANK"])
torch.accelerator.set_device_index(local_rank)

dist.init_process_group("lccl")

# Cast explicitly instead of relying on autocast.
model = MyModel().to("lpu").to(torch.bfloat16)
model = torch.compile(model, backend="legato",
                      options={"legato.device": "bertha"})
x = torch.randn(B, S, D, dtype=torch.bfloat16).to("lpu")

out = model(x)

dist.all_reduce(out, op=dist.ReduceOp.SUM)
out = out.cpu()          # this is the synchronization point

Next Steps