Skip to content

2. Device Semantics

This chapter describes how the lpu device behaves and where it differs from what PyTorch users expect.

Contents

Section Objective
2.1 Moving Tensors .to("lpu"), .cpu(), moving modules, and factory functions
2.2 Weight Layout Tensor kinds, the rules that assign them when a tensor moves, and slicing
2.3 Data Types Supported dtypes and autocast behavior
2.4 Device Context Manager How torch.accelerator.device_index behaves and how to select a device
2.5 Serialization torch.save / torch.load and checkpoint portability
2.6 Synchronization Asynchronous execution, synchronization points, and timing
2.7 Memory When freed buffers are reclaimed, empty_cache, memory statistics, and fill paths
2.8 Current Limitations Stream and Event, and pinned host memory with asynchronous copies

2.1 Moving Tensors

1
2
3
x = torch.randn(32, 128, dtype=torch.bfloat16)  # CPU
x = x.to("lpu")                                  # transfer to the device
x = x.cpu()                                      # and back

Both .to("lpu") and .to(torch.device("lpu")) work. Modules can be moved the same way.

1
model = MyModel().to("lpu")

Factory functions

Value factories (zeros, ones, full, arange, empty) work with device="lpu" (the current device) or an explicit index, as in torch.zeros(32, 128, dtype=torch.bfloat16, device="lpu"). A with torch.device("lpu"): block also resolves to the current device.

Create random tensors (torch.randn, torch.rand) on the CPU and move them.

2.2 Weight Layout

Tensors in device memory have a layout, and the fastest layout depends on how the consuming kernel reads the tensor. The backend calls this layout the tensor kind and records it in the storage of each LPU tensor.

Kind Placement Primary use
dram_word_interleaved Interleaved across all channels in DRAM-word units. Any rank Default. Activations, intermediates, 1-D tensors
mpu_tensor The tile pattern the MPU reads (Value pattern). Rank 2 or higher Weight operand of a matrix multiply
mpu_tensor_transposed The transposed pattern of mpu_tensor (Key pattern). The hardware reads it as a transposed view. Rank 2 or higher Weights consumed as W.t() (nn.Linear)
single_pass_interleaved Each channel holds 1/N of the tensor as one contiguous chunk Tables gathered by row (nn.Embedding)
row_interleaved Interleaved across channels one row at a time Row-wise access
max_rows_interleaved Places a group of rows the size of the MAC reuse window on each channel as one unit Input activations

How Tensor Kinds Are Assigned

When a tensor moves to lpu, its kind is determined in the following order:

  1. nn.Linear weights → mpu_tensor_transposed. When the module moves to lpu or its dtype changes on lpu, the weight is reallocated with this kind. The logical shape (out, in) does not change, so state_dict round trips are unaffected. The bias moves as an ordinary tensor, and a warning about this is printed once on the first move.
  2. nn.Embedding weights → single_pass_interleaved, regardless of requires_grad.
  3. Everything else → dram_word_interleaved. Tensors created by factory functions fall here as well.

Rules 1 and 2 do not apply when you call .to("lpu") directly on a weight tensor that is not part of a module. Move whole modules.

To allocate with an explicit kind, use torch.ops.lpu.empty_with_kind. It accepts the six names above. Usage is described in the Custom Operators API. For how these kinds map to the kinds of legato kernel parameters, see the Legato documentation.

The layout follows the storage, not the tensor

View operations (view, reshape, as_strided, slicing) create a new tensor over the same storage, and the backend propagates the layout to it. An operation that reinterprets storage in a way the layout cannot express fails at that view, not at the operation that appears to be the cause.

Slicing

As in PyTorch, slicing a tensor (basic indexing such as x[i] and x[a:b], and select, narrow, split, and chunk) returns a view that shares storage with its base tensor; no data is copied.

Device compute currently assumes a contiguous base tensor. A slice can be passed to device compute directly when all of the following hold:

  • Only batch dimensions are sliced. The slice is taken along dim -3 (the third-to-last dimension) or an earlier one; the last two dimensions (-2, -1) are left whole.
  • The result is contiguous. is_contiguous() on the sliced view returns True.
  • It starts and ends on an interleave-round boundary. In dram_word_interleaved (the default kind), row_interleaved, and max_rows_interleaved, the slice must start and end on an interleave-round boundary, where an interleave round is the unit that places data once on every channel. When one batch element (x[i], M × K elements) is a multiple of a round, every x[i] and x[a:b] meets this condition. single_pass_interleaved is passed directly only as a whole tensor.

The following examples assume a contiguous x of shape (B, M, K) that meets the round condition above:

Expression Sliced dimension Device compute
x[i], x[a:b], x.select(0, i), x.narrow(0, a, n), x.split(n), x.chunk(n) -3 Passed directly
x[:, a:b], x.narrow(1, a, n) -2 Requires .contiguous()
x[..., a:b], x.narrow(2, a, n) -1 Requires .contiguous()

For shape (N, B, M, K) under the same round condition, contiguous slices such as x[i], x[a:b], and x[i, a:b] are passed directly. x[:, a:b] slices a batch dimension, but its result is not contiguous, so compute on it after .contiguous(). In the MPU kinds (mpu_tensor, mpu_tensor_transposed), the last two dimensions must fill whole tiles.

Other slices are still valid views. They can be read and written with copy_, .cpu(), and fill_, which may copy through the host. To compute on one, call .contiguous() if it is not contiguous, or .clone() if it is, to get a whole tensor of the same kind.

2.3 Data Types

dtype Status
bfloat16 Primary
float16 Storage and transfer only. It is not converted to bfloat16 automatically; moving it unconverted emits a Python UserWarning once per process. Convert with .to("lpu", torch.bfloat16) before computing
float32 Storage and transfer. The legato resolvers decide compute support on the torch.compile path
int32, int64 Supported for indexing, comparison, and reduction outputs
bool Output of comparison operators
float64 Convert to bfloat16 or float32 before moving to the LPU
int8, uint8 Storage and transfer. Quantized weights via torch.ops.lpu.quant_weight_*
int16 Storage and transfer

LLM inference on this hardware runs in bfloat16. float32 tensors can be stored and transferred, and device compute in float32 runs for the operators the legato resolvers support. When comparing results against the CPU, use bfloat16 on both sides.

Autocast

Cast explicitly

Do not use torch.autocast("lpu"). Entering the context succeeds, but operators inside it raise NotImplementedError naming the Autocastlpu backend, in eager mode and under torch.compile. Cast explicitly.

1
model = model.to("lpu").to(torch.bfloat16)

2.4 Device Context Manager

torch.accelerator.device_index(i) switches the calling thread's current device to i inside the block and restores the previous device when the block exits.

1
2
with torch.accelerator.device_index(1):
    y = x.to("lpu")           # placed on lpu:1

In the common case of setting the device once per rank, use torch.accelerator.set_device_index(). See Distributed §4.2.

2.5 Serialization

torch.save and torch.load work with LPU tensors.

Loading a checkpoint saved from the LPU without map_location fails on a machine where the backend is not registered. map_location="cpu" does not convert the device layout, so weights in a non-default kind (for example nn.Linear and nn.Embedding weights) load with incorrect values. Load LPU checkpoints without map_location on a machine with the backend, or save with .cpu() as below.

1
torch.save({k: v.cpu() for k, v in model.state_dict().items()}, "ckpt.pt")

2.6 Synchronization

Like a GPU, the LPU executes asynchronously with respect to the host. To wait for the device explicitly, call torch.accelerator.synchronize().

Existing synchronization points still work. Reading a value on the host (.cpu(), .item(), print) blocks until the value is ready.

Timing without a synchronization point measures nothing

1
2
3
start = time.time()
out = model(x)                 # returns before the device finishes
elapsed = time.time() - start  # meaningless

Synchronize before stopping the clock. Calling out.cpu() is sufficient. Alternatively, use haprof, which reads device time at the correct fence.

2.7 Memory

Memory caching is built into the hart runtime. The hart memory pool works the same way as the CUDA caching allocator: it splits segments into blocks and keeps freed blocks for reuse. torch_ha therefore uses the pool directly, with no caching layer of its own, and allocates and frees in stream order. When you use it from PyTorch, keep the following in mind.

  • Freed buffers are reused only after a synchronization. When a tensor is freed, its buffer stays on the stream as a pending free request and returns to the pool only after a synchronization completes. Unlike CUDA, the next allocation on the same stream does not reuse it right away. If an allocation fails because of insufficient memory, the allocator synchronizes the device once to reclaim these buffers, then retries.
  • torch.accelerator.empty_cache() releases pool memory back to the device. It synchronizes all devices to return buffers awaiting release to the pool, then releases segments that hold no blocks in use back to the device. Check the increased free memory with get_memory_info().
  • Read memory statistics through torch.accelerator. memory_stats(), memory_allocated(), memory_reserved(), max_memory_allocated(), reset_peak_memory_stats(), and get_memory_info() return values read from the hart pool and the driver. A freed buffer does not return to the pool until a synchronization, so it counts toward memory_allocated() until then.
  • fill_ takes a different path depending on size. fill_ and zero_ fill tensors of 1 MiB or more with a memset kernel on the device. Smaller tensors and views the kernel cannot handle use a staging buffer that is filled on the host and copied to the device. HA_FILL_KERNEL_MIN_BYTES changes the size threshold, and HA_FILL_KERNEL=0 turns the kernel off. See Environment Variables.
  • copy_ within one device runs on a device copy kernel. This applies when both tensors have the same kind, dtype, shape, and strides, and their placement meets the kernel's conditions. Other copies use the existing path.

For how the pool splits and keeps segments, see the hart memory model.

2.8 Current Limitations

Feature Status
Stream / Event (torch.Stream, torch.Event) Streams exist in the runtime layer, but PyTorch currently uses only the default stream (torch.accelerator.current_stream()); creating a custom stream (torch.Stream("lpu")) or recording an event (torch.Event) is not available yet.
Pinned host memory (pin_memory()) and asynchronous copies (non_blocking=True) pin_memory() (including DataLoader(pin_memory=True)) raises NotImplementedError; non_blocking=True is accepted but every copy blocks.

Stream and event control belongs to the runtime, one layer below. See hart.


Next Steps