2. Device Semantics¶
This chapter describes how the lpu device behaves and where it differs from what PyTorch users expect.
Contents¶
| Section | Objective |
|---|---|
| 2.1 Moving Tensors | .to("lpu"), .cpu(), moving modules, and factory functions |
| 2.2 Weight Layout | Tensor kinds, the rules that assign them when a tensor moves, and slicing |
| 2.3 Data Types | Supported dtypes and autocast behavior |
| 2.4 Device Context Manager | How torch.accelerator.device_index behaves and how to select a device |
| 2.5 Serialization | torch.save / torch.load and checkpoint portability |
| 2.6 Synchronization | Asynchronous execution, synchronization points, and timing |
| 2.7 Memory | When freed buffers are reclaimed, empty_cache, memory statistics, and fill paths |
| 2.8 Current Limitations | Stream and Event, and pinned host memory with asynchronous copies |
2.1 Moving Tensors¶
1 2 3 | |
Both .to("lpu") and .to(torch.device("lpu")) work. Modules can be moved the same way.
1 | |
Factory functions
Value factories (zeros, ones, full, arange, empty) work with device="lpu" (the current device) or an
explicit index, as in torch.zeros(32, 128, dtype=torch.bfloat16, device="lpu"). A with torch.device("lpu"):
block also resolves to the current device.
Create random tensors (torch.randn, torch.rand) on the CPU and move them.
2.2 Weight Layout¶
Tensors in device memory have a layout, and the fastest layout depends on how the consuming kernel reads the tensor. The backend calls this layout the tensor kind and records it in the storage of each LPU tensor.
| Kind | Placement | Primary use |
|---|---|---|
dram_word_interleaved |
Interleaved across all channels in DRAM-word units. Any rank | Default. Activations, intermediates, 1-D tensors |
mpu_tensor |
The tile pattern the MPU reads (Value pattern). Rank 2 or higher | Weight operand of a matrix multiply |
mpu_tensor_transposed |
The transposed pattern of mpu_tensor (Key pattern). The hardware reads it as a transposed view. Rank 2 or higher |
Weights consumed as W.t() (nn.Linear) |
single_pass_interleaved |
Each channel holds 1/N of the tensor as one contiguous chunk | Tables gathered by row (nn.Embedding) |
row_interleaved |
Interleaved across channels one row at a time | Row-wise access |
max_rows_interleaved |
Places a group of rows the size of the MAC reuse window on each channel as one unit | Input activations |
How Tensor Kinds Are Assigned¶
When a tensor moves to lpu, its kind is determined in the following order:
nn.Linearweights →mpu_tensor_transposed. When the module moves tolpuor its dtype changes onlpu, the weight is reallocated with this kind. The logical shape(out, in)does not change, sostate_dictround trips are unaffected. The bias moves as an ordinary tensor, and a warning about this is printed once on the first move.nn.Embeddingweights →single_pass_interleaved, regardless ofrequires_grad.- Everything else →
dram_word_interleaved. Tensors created by factory functions fall here as well.
Rules 1 and 2 do not apply when you call .to("lpu") directly on a weight tensor that is not part of a module. Move
whole modules.
To allocate with an explicit kind, use torch.ops.lpu.empty_with_kind. It accepts the six names above. Usage is
described in the Custom Operators API. For how these kinds map to the kinds of
legato kernel parameters, see the Legato documentation.
The layout follows the storage, not the tensor
View operations (view, reshape, as_strided, slicing) create a new tensor over the same storage, and the
backend propagates the layout to it. An operation that reinterprets storage in a way the layout cannot express
fails at that view, not at the operation that appears to be the cause.
Slicing¶
As in PyTorch, slicing a tensor (basic indexing such as x[i] and x[a:b], and select, narrow, split, and
chunk) returns a view that shares storage with its base tensor;
no data is copied.
Device compute currently assumes a contiguous base tensor. A slice can be passed to device compute directly when all of the following hold:
- Only batch dimensions are sliced. The slice is taken along dim
-3(the third-to-last dimension) or an earlier one; the last two dimensions (-2,-1) are left whole. - The result is contiguous.
is_contiguous()on the sliced view returnsTrue. - It starts and ends on an interleave-round boundary. In
dram_word_interleaved(the default kind),row_interleaved, andmax_rows_interleaved, the slice must start and end on an interleave-round boundary, where an interleave round is the unit that places data once on every channel. When one batch element (x[i],M × Kelements) is a multiple of a round, everyx[i]andx[a:b]meets this condition.single_pass_interleavedis passed directly only as a whole tensor.
The following examples assume a contiguous x of shape (B, M, K) that meets the round condition above:
| Expression | Sliced dimension | Device compute |
|---|---|---|
x[i], x[a:b], x.select(0, i), x.narrow(0, a, n), x.split(n), x.chunk(n) |
-3 |
Passed directly |
x[:, a:b], x.narrow(1, a, n) |
-2 |
Requires .contiguous() |
x[..., a:b], x.narrow(2, a, n) |
-1 |
Requires .contiguous() |
For shape (N, B, M, K) under the same round condition, contiguous slices such as x[i], x[a:b], and x[i, a:b]
are passed directly. x[:, a:b] slices a batch dimension, but its result is not contiguous, so compute on it after .contiguous(). In the
MPU kinds (mpu_tensor, mpu_tensor_transposed), the last two dimensions must fill whole tiles.
Other slices are still valid views. They can be read and written with copy_, .cpu(), and fill_, which may copy
through the host. To compute on one, call .contiguous() if it is not contiguous, or .clone() if it is, to get a whole tensor of the
same kind.
2.3 Data Types¶
| dtype | Status |
|---|---|
bfloat16 |
Primary |
float16 |
Storage and transfer only. It is not converted to bfloat16 automatically; moving it unconverted emits a Python UserWarning once per process. Convert with .to("lpu", torch.bfloat16) before computing |
float32 |
Storage and transfer. The legato resolvers decide compute support on the torch.compile path |
int32, int64 |
Supported for indexing, comparison, and reduction outputs |
bool |
Output of comparison operators |
float64 |
Convert to bfloat16 or float32 before moving to the LPU |
int8, uint8 |
Storage and transfer. Quantized weights via torch.ops.lpu.quant_weight_* |
int16 |
Storage and transfer |
LLM inference on this hardware runs in bfloat16. float32 tensors can be stored and transferred, and device compute
in float32 runs for the operators the legato resolvers support. When comparing results against the CPU, use
bfloat16 on both sides.
Autocast¶
Cast explicitly
Do not use torch.autocast("lpu"). Entering the context succeeds, but operators inside it raise
NotImplementedError naming the Autocastlpu backend, in eager mode and under torch.compile. Cast explicitly.
1 | |
2.4 Device Context Manager¶
torch.accelerator.device_index(i) switches the calling thread's current device to i inside the block and restores
the previous device when the block exits.
1 2 | |
In the common case of setting the device once per rank, use torch.accelerator.set_device_index(). See
Distributed §4.2.
2.5 Serialization¶
torch.save and torch.load work with LPU tensors.
Loading a checkpoint saved from the LPU without map_location fails on a machine where the backend is not
registered. map_location="cpu" does not convert the device layout, so weights in a non-default kind (for example
nn.Linear and nn.Embedding weights) load with incorrect values. Load LPU checkpoints without map_location on a
machine with the backend, or save with .cpu() as below.
1 | |
2.6 Synchronization¶
Like a GPU, the LPU executes asynchronously with respect to the host. To wait for the device explicitly, call
torch.accelerator.synchronize().
Existing synchronization points still work. Reading a value on the host (.cpu(), .item(), print) blocks until
the value is ready.
Timing without a synchronization point measures nothing
1 2 3 | |
Synchronize before stopping the clock. Calling out.cpu() is sufficient. Alternatively, use
haprof, which reads device time at the correct fence.
2.7 Memory¶
Memory caching is built into the hart runtime. The hart memory pool works the same way as the CUDA caching allocator:
it splits segments into blocks and keeps freed blocks for reuse. torch_ha therefore uses the pool directly, with no
caching layer of its own, and allocates and frees in stream order. When you use it from PyTorch, keep the following in
mind.
- Freed buffers are reused only after a synchronization. When a tensor is freed, its buffer stays on the stream as a pending free request and returns to the pool only after a synchronization completes. Unlike CUDA, the next allocation on the same stream does not reuse it right away. If an allocation fails because of insufficient memory, the allocator synchronizes the device once to reclaim these buffers, then retries.
torch.accelerator.empty_cache()releases pool memory back to the device. It synchronizes all devices to return buffers awaiting release to the pool, then releases segments that hold no blocks in use back to the device. Check the increased free memory withget_memory_info().- Read memory statistics through
torch.accelerator.memory_stats(),memory_allocated(),memory_reserved(),max_memory_allocated(),reset_peak_memory_stats(), andget_memory_info()return values read from the hart pool and the driver. A freed buffer does not return to the pool until a synchronization, so it counts towardmemory_allocated()until then. fill_takes a different path depending on size.fill_andzero_fill tensors of 1 MiB or more with a memset kernel on the device. Smaller tensors and views the kernel cannot handle use a staging buffer that is filled on the host and copied to the device.HA_FILL_KERNEL_MIN_BYTESchanges the size threshold, andHA_FILL_KERNEL=0turns the kernel off. See Environment Variables.copy_within one device runs on a device copy kernel. This applies when both tensors have the same kind, dtype, shape, and strides, and their placement meets the kernel's conditions. Other copies use the existing path.
For how the pool splits and keeps segments, see the hart memory model.
2.8 Current Limitations¶
| Feature | Status |
|---|---|
Stream / Event (torch.Stream, torch.Event) |
Streams exist in the runtime layer, but PyTorch currently uses only the default stream (torch.accelerator.current_stream()); creating a custom stream (torch.Stream("lpu")) or recording an event (torch.Event) is not available yet. |
Pinned host memory (pin_memory()) and asynchronous copies (non_blocking=True) |
pin_memory() (including DataLoader(pin_memory=True)) raises NotImplementedError; non_blocking=True is accepted but every copy blocks. |
Stream and event control belongs to the runtime, one layer below. See hart.
Next Steps¶
- Operators: what dispatches to the device
- Distributed: using multiple devices
- Paged KV Cache: mapping KV cache blocks with virtual memory