Most PyTorch code that runs on GPUs is written for the cuda device. This chapter covers what to change, and what
you can keep, when moving cuda code to lpu.
Compile with torch.compile(backend="legato") to run on the device. The default backend (inductor) cannot generate lpu kernels and fails. Operators
Operator coverage
Effectively all of ATen
Decided by the legato resolvers on the torch.compile path. Eager compute is supported experimentally for only four operators: add, sub, mul, and mm on bfloat16
Unregistered operators
Rarely encountered
Raises a dispatch error that names the lpu backend
Tensor creation
torch.randn(..., device="cuda")
Value factories work with device="lpu" (which resolves to the current device) or an explicit index. Create random tensors on the CPU, then move them with .to("lpu")
Explicit synchronization
torch.cuda.synchronize()
Read a value back (.cpu(), .item()) or call torch.accelerator.synchronize()
Move random tensor creation off the device. Value factories (zeros, full, and so on) work with
device="lpu" (which resolves to the current device) or an explicit index. Instead of
torch.randn(..., device=dev), create the tensor on the CPU and move it with .to(dev).
Check your model with a legato dry run on the CPU. See
Operators §3.3 for the procedure. This step decides whether the model can
be ported, so do it before the remaining steps.
Compile the model's computation with torch.compile(backend="legato"). The default backend (inductor) cannot
generate lpu kernels. See
Compile PyTorch Model with Legato.
Remove stream code, and switch memory statistics to torch.accelerator. Operators run on the default stream, so
removing stream code does not change behavior.
Replace torch.cuda.synchronize(). Call out.cpu() or torch.accelerator.synchronize() where you need to
wait for completion.
Before you design around collectives and reductions, check that the
LCCL matrix supports them.
Cast explicitly instead of relying on autocast.
Do not carry over performance assumptions
The relative cost of kernels differs from that on GPUs, so an implementation that was faster on a GPU is not
necessarily faster on the LPU. Measure with haprof.
importosimporttorchimporttorch.distributedasdist# The backend binds rank r to lpu:<r>.# Select the same device first so that tensors are placed there too.local_rank=int(os.environ["LOCAL_RANK"])torch.accelerator.set_device_index(local_rank)dist.init_process_group("lccl")# Cast explicitly instead of relying on autocast.model=MyModel().to("lpu").to(torch.bfloat16)model=torch.compile(model,backend="legato",options={"legato.device":"bertha"})x=torch.randn(B,S,D,dtype=torch.bfloat16).to("lpu")out=model(x)dist.all_reduce(out,op=dist.ReduceOp.SUM)out=out.cpu()# this is the synchronization point