Skip to content

Bertha Layouts

Bertha tensors are usually declared with legato.get_model().tensor_type(dtype, shape, kind). The kind string selects both a Bertha memory object and a Bertha layout map.

This page describes the layout kinds supported by the current Bertha implementation. For generic Legato layout maps, see Layouts And Memory Kinds.

Supported Kind Strings

The following kind strings are accepted by Bertha.tensor_type (see Target generations for backend limits):

  • "shared_memory" and "shared_dram"
  • "row_major"
  • "value_dram" and "mpu_dram"
  • "key_dram" and "transposed_mpu_dram"
  • "mpu_weight"
  • "mpu_weight_sram" for MPU-format weights resident in core SRAM
  • "mpu_packed_buffer"
  • "sliced_dram"
  • "instruction_table" for host constants embedded in the instruction table
1
2
3
4
5
tensor_type = legato.get_model().tensor_type(
    legato.types.float("bfloat16"),
    [10, 128],
    "shared_dram",
)

"shared_memory" is an alias for the same shared-DRAM word-distributed layout used by "shared_dram".

Row-Major Layouts

Use "row_major" for tensors in core-local SRAM with row-major indexing. This kind replaces the older "row_major_sram" name.

Row-major layout maps logical indices to a flattened row-major offset:

1
(d0, d1, ...)[s0, s1, ...] -> row_major_offset

For a 2-D tensor this is equivalent to d0 * s1 + d1.

"row_major_dram" is no longer accepted by the Bertha constructor. Use "shared_dram" for shared DRAM tensors, or define an explicit generic layout map with legato.types.tensor(...) if a custom row-major DRAM type is required.

Shared DRAM Layouts

Use "shared_dram" or "shared_memory" for ordinary shared-DRAM tensors. These kinds use a word-distributed DRAM layout across the configured channels. The EVT0 defaults are 64-byte words and 8 DRAM channels.

The implementation computes:

  • the row-major flattened element index,
  • the element's position within a 64-byte word,
  • the DRAM channel from the word index,
  • and the local word offset inside the selected channel.

This layout is useful for tensors loaded from or stored to shared memory through standard Legato tensor movement operations.

Sliced DRAM

Use "sliced_dram" when a tensor should be split into equal contiguous slices across the configured DRAM channels (8 by default on EVT0).

Constraints:

  • shape must be available when the layout is built.
  • The total physical element count must be divisible by params.num_memory_channels.
  • The dtype bit width must divide the Bertha channel size.
  • Dynamic dimensions are allowed only when their maximum physical dimension is known, as in (-1, max_dim).
1
2
3
4
5
sliced_type = legato.get_model().tensor_type(
    legato.types.float("bfloat16"),
    [8, 1024],
    "sliced_dram",
)

Key And Value DRAM Aliases

  • "key_dram" and "transposed_mpu_dram" select the same key-style kind.
  • "value_dram" and "mpu_dram" select the same value-style kind.

Use names that communicate the tensor's role, such as key/value cache storage or matmul operands. Both kinds require rank at least 2, a dtype width of 4, 8, or 16 bits, and a shape accepted by the selected backend. Let the SDK construct the layout and check its constraints.

Channel address spacing is configured by params.dram_channel_stride_bytes (64 GiB by default on EVT0); bytes_per_channel is channel capacity. Overrides must match the selected device/runtime configuration.

Weight Layouts

Use "mpu_weight" for weights received through "mpu_buffer", or "mpu_weight_sram" for compatible weights resident in core SRAM. These kinds require rank at least 2 and a dtype width of 4, 8, or 16 bits.

Tensor kind Memory Layout family
"mpu_weight" MPU buffer mpu_weight
"mpu_weight_sram" Core SRAM mpu_weight

Construct these types with the Bertha kind so that they carry the layout-family tag used by executor assignment. MPU-format SRAM weights are supported as matmul RHS operands; they are rejected as matmul LHS operands or VPU arithmetic, reduction, or normalization inputs. The compiler can prepare compact row-major SRAM weights for the MPU while keeping the tensor in SRAM.

On EVT0, a value-style source [K, N] retains its logical shape when received into SRAM; a key-style source [N, K] is received as [K, N]. Use a Bertha tensor kind to express this contract. A custom generic layout does not necessarily satisfy the requirements for a weight operand.

1
2
3
4
5
6
7
weight = legato.tensor.receive(src, handle, "sram")
result = lhs @ weight

# Explicit typing is also supported when using receive_type.
weight_type = device.tensor_type(dtype, [128, 128], "mpu_weight_sram")
weight = legato.tensor.receive_type(weight_type, 0, None, "sram")
result = lhs @ weight

The compiler prepares compatible SRAM weights for matmul, convolution, and depthwise convolution while keeping their logical tensor shape. Applications should rely on the SDK's type checks and allocation/conversion helpers. See Weight storage requirements.

For mm_weight_quant / addmm_weight_quant, use "mpu_packed_buffer" for the opaque buffer returned by the SDK weight converter. It requires rank 1 and an i8 element type. Its shape is the converted byte count, not [C, M]. Keep the converted contents unchanged until all consumers have used them.

Memory Placement

Bertha.tensor_type maps layout kinds to memory objects as follows:

  • Shared DRAM memory: "sliced_dram", "shared_dram", "shared_memory", the key-style aliases "key_dram" and "transposed_mpu_dram", and the value-style aliases "value_dram" and "mpu_dram".
  • MPU buffer memory: "mpu_weight" and "mpu_packed_buffer".
  • Core SRAM memory: "row_major" and "mpu_weight_sram".

Memory placement and layout are selected together. If only a custom generic layout map is needed, use legato.types.tensor(...) with legato.language.types.layout.map(...) instead of a Bertha kind string.

Device Tensor Kinds (empty_with_kind)

The kind strings above are compile-time annotations. A device-resident input tensor created on the host side carries its own layout kind, stamped by torch.ops.lpu.empty_with_kind, which must match the parameter's annotation kind. For the empty_with_kind name ↔ annotation kind matching table and the full create → fill → launch workflow, see PyTorch Integration.

Dynamic Shapes

Bertha tensor kinds use the same dynamic shape syntax as generic Legato tensor types:

1
2
3
4
5
dynamic_type = legato.get_model().tensor_type(
    legato.types.float("bfloat16"),
    ((-1, 10), 128),
    "shared_dram",
)

For layouts that need physical tensor size at compile time, such as "sliced_dram", the maximum dimension is used as the physical dimension.

Physical alignment and shared loads

Constraints apply to physical element counts. On default EVT0 shared DRAM, a physical row is aligned to 64 bytes: 32 BF16 elements or 16 FP32 elements. SRAM rows use 128-byte words. A compact DRAM row may need padding/reformatting when received into SRAM; the runnable examples use widths compatible with both. An explicitly unaligned type reports LE0301. Its detailed alignment message distinguishes the dtype-specific element multiple from the byte span: for example, 64 BF16 elements or 32 FP32 elements occupy a 128-byte SRAM word. See the compile diagnostics example.

Top-side shared-DRAM loads must cover whole DRAM lines. The line size is the configured channel count times the DRAM word size: 8 × 64 bytes = 512 bytes on default EVT0. For BF16 rows of width 64, ten rows occupy 1280 bytes and do not meet this condition; twelve rows occupy 1536 bytes and do. Padding only the physical first dimension does not change the logical transfer extent. Both the request and its backing allocation must be valid. Other transfer paths can have different requirements.

A view preserves its parent's physical pitch. Matmul and softmax lowering use that pitch while restricting math to logical elements. On the host, match the compiled layout with a real padded backing tensor, then pass its logical view; see Physical storage.