Skip to content

Bertha API

This page documents Bertha-specific Python APIs from legato.model.bertha.operations.

Bertha APIs are backend-specific helpers layered on top of the generic Legato language operations. They include the Bertha compute model constructor, executor accessors, memory/type helpers, MPU/TLB operations, and Bertha synchronization operations.

Typical Backend Setup

1
2
3
4
5
6
import legato
import legato.model.bertha as bertha


def get_bertha(ctx):
    return bertha.Bertha(ctx, "bertha")

Use this backend factory with @legato.compile(backend=get_bertha).

Target generations

Bertha(ctx, "bertha") selects EVT0 defaults. Override individual parameters by keyword (num_cores=..., num_memory_channels=...), or pass a MLIRBerthaDeviceParams instance. The old positional parameter list is deprecated; it does not describe EVT1.

Use backend=bertha.evt1 or Bertha(ctx, "bertha", revision="EVT1") to select the bertha_v1 compute model and EVT1 device parameters. Parameter objects must match their generation. EVT1 has its own lowering and translator pipelines; EVT0 examples and hardware layout constants cannot be reused as an EVT1 support claim. The Python wrapper explicitly rejects EVT0-only TLB, packed-weight quantized matmul, and top_kp helpers on EVT1. device.sort also supports EVT1; it currently requires descending order and finite logical inputs. Check an individual operation through the requested stage and matching target runtime.

Verified EVT1 paths

The current EVT1 backend includes the following paths. These are compiler and simulator support statements for the listed cases, rather than a hardware reference or a claim that every EVT0 operation is available on EVT1.

Operation Current coverage and limits
Tensor -, *, / Matching-shape BF16 tensors run on the EVT1 VPU, alongside the existing + path. The numerical cases load [16, 256] inputs and store the flattened result.
SRAM to shared_dram with legato.tensor.memcpy Word-interleaved DRAM write-back is numerically covered for BF16 output ranks 1 through 6. Keep input and output storage kinds compatible.
Dynamic or padded shared-DRAM request_load / receive Runtime row counts size whole-line transfers; the other dimensions use their physical extents. Destination SRAM physical extents must still be static.
Last-axis BF16 softmax The reduce_max → sub → exp → reduce_sum → div pattern is covered for [32, 256] input. Use keepdim=True; the reduced width must be a multiple of 64 BF16 elements.
device.sort Sorting is supported in descending order for finite inputs. This does not extend top_kp to EVT1.
Instruction-table reads and embedding gather Scalar reads and integer casts are supported. tensor.gather(weight, indices) supports instruction-table integer indices with a sliced_dram table whose row count is divisible by the number of DRAM channels. Instruction-table tensor views and arbitrary SRAM-index gathers are not implied.
Runtime while and nearest-loop break Integer/index/bool carried state is supported on EVT0 and EVT1. continue, loop-else, early return from the loop body, and tensor carried state remain unsupported.
RoPE The repository's BF16 prefill and decode kernels run against CPU references on the EVT1 simulator. They stage padded cosine/sine tables; this is a verified kernel composition, not a separate rope API.

Select EVT1 explicitly with backend=bertha.evt1 and an EVT1 runtime or simulator. Whole-line receive requirements still apply; short transfers and unequal per-channel word counts are separate lowering cases, not a reason to assume arbitrary unaligned copies are valid. See Memory Operations and Arithmetic Operations.

Common Tensor Kinds

Bertha.tensor_type(dtype, shape, kind) accepts Bertha-specific kind strings. Common values include:

  • "shared_memory" and "shared_dram"
  • "row_major"
  • "value_dram" and "mpu_dram" for the same value-style DRAM layout
  • "key_dram" and "transposed_mpu_dram" for the same key-style DRAM layout
  • "mpu_weight"
  • "mpu_weight_sram" for MPU-format weights resident in core SRAM
  • "mpu_packed_buffer" for an opaque rank-1 packed-weight blob
  • "sliced_dram"

See Bertha Layouts for layout constraints and memory placement.

TLB Loads To SRAM (EVT0)

request_load_tlb and request_load_tlb_dynamic accept a trailing destination argument: "mpu_buffer" (the default) or "sram". The static API takes an interpreted core list; the dynamic API takes a compiled core bitmap.

1
2
device.request_load_tlb(cache, cores, request_id, layer, head, "sram")
device.request_load_tlb_dynamic(cache, core_bitmap, request_id, layer, head, "sram")

Receive SRAM weights with legato.tensor.receive(cache, handle, "sram"). Bertha EVT0 infers the MPU-format layout and its layout-family tag from the source, so the resulting SRAM tensor can be reused as a matmul RHS. Explicit tensor_type(..., "mpu_weight_sram") with receive_type is also supported. The same inference applies to ordinary legato.tensor.request_load(cache, core, "sram"). Omitting the request destination keeps the existing MPU-buffer behavior and uses tensor_type(..., "mpu_weight") with a "mpu_buffer" receive.

The following value-cache example loads one weight into SRAM and uses it in two matmuls with different activations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
from legato.language import arith


@legato.compile(
    backend=get_bertha,
    param_kinds={"cache": "top", "out0": "core", "out1": "core"},
)
def resident_tlb_matmul(
    lhs: legato.get_model().tensor_type(legato.types.float("bfloat16"), [4, 128], "shared_dram"),
    cache: legato.get_model().tensor_type(legato.types.float("bfloat16"), [128, 128], "value_dram"),
    out0: legato.get_model().tensor_type(legato.types.float("bfloat16"), [4, 128], "shared_dram"),
    out1: legato.get_model().tensor_type(legato.types.float("bfloat16"), [4, 128], "shared_dram"),
    request_id: legato.types.integer("int64"),
    layer: legato.types.integer("int64"),
    head: legato.types.integer("int64"),
):
    device = legato.get_context().get_device()
    with device.get_top():
        legato.tensor.request_load(lhs, device.get_core(0), "sram")
        device.request_load_tlb(
            cache,
            [device.get_core(0)],
            arith.cast(request_id, legato.types.index()),
            arith.cast(layer, legato.types.index()),
            arith.cast(head, legato.types.index()),
            "sram",
        )

    with device.get_core(0):
        dtype = legato.types.float("bfloat16")
        a = legato.tensor.receive(lhs, 0, "sram")
        # This is the second SRAM receive after the activation.
        weight = legato.tensor.receive(cache, 1, "sram")
        legato.tensor.memcpy(out0, a @ weight)
        other_a = a + arith.cast(1.0, dtype)
        legato.tensor.memcpy(out1, other_a @ weight)

Initialize the EVT0 TLB and allocate the request's key/value pages before launching the kernel; see the HART TLB API. Bind cache to bertha.top and each output to bertha.core(0) at launch. The weight above is received once and remains in SRAM for both matmuls.

A "key_dram" input with logical shape [N, K] is received with shape [K, N]; a "value_dram" input [K, N] keeps that logical shape. The resulting weight may be reused as a matmul RHS. It is not a general-purpose matmul LHS, arithmetic, reduction, or normalization input. Keep the source and receive shapes compatible with the selected tensor kind; see Bertha Layouts.

Depthwise Convolution

legato.math.dwconv_1d and legato.math.dwconv_2d operate on rank-2 and rank-3 spatial inputs respectively. The 2D defaults are stride=(1, 1) and padding=(0, 0, 0, 0).

  • 1D: input [X, C], weight [KX, C], result [OX, C].
  • 2D: input [Y, X, C], weight [KY, KX, C], result [OY, OX, C].
  • The current EVT0 implementation requires static BF16 inputs, positive C, X * Y <= 32, kernel sizes 1–8, strides 1–2, and padding 0–7.
  • Load activations into core SRAM. Weights may be loaded with request_load(..., "mpu_buffer") or into SRAM using a compatible tensor kind.
  • Bias and SiLU are separate operations; add them explicitly when needed.

Weight storage requirements

Keep the documented logical weight shape. Use the SDK's tensor kinds and allocation or conversion helpers to provide compatible storage. The compiler handles the required weight formatting and padding; applications should not construct or depend on its byte representation.

Physical capacity must satisfy the selected tensor kind. For a padded tensor, allocate and initialize the backing storage before passing a logical view. Reusable weights must remain available for every consumer; a one-use streamed weight cannot be reused as if it were a persistent SRAM tensor. See Bertha Layouts.

1
2
3
4
5
6
7
8
9
with device.get_top():
    legato.tensor.request_load(inp, device.get_core(0), "sram")
    legato.tensor.request_load(weight, device.get_core(0), "sram")

with device.get_core(0):
    x = legato.tensor.receive(inp, 0, "sram")
    w = legato.tensor.receive(weight, 0, "sram")
    y = legato.math.dwconv_2d(x, w)  # x: [Y, X, C], w: [KY, KX, C]
    y = y + bias

Convolution and Pooling

legato.math.conv_1d / conv_2d and the max_pool_* / avg_pool_* APIs support the following EVT0 input contract:

  • Layout is channels-last: 1D [N*, X, C], 2D [N*, H, W, C]. Convolution weights are [KX, Cin, Cout] or [KH, KW, Cin, Cout], with optional bias [Cout].
  • N* denotes leading batch dimensions; see Broadcasting and Broadcasting on Bertha.
  • Current EVT0 support requires BF16, kernel sizes 1–8, strides 1–2, padding 0–7, unit dilation, and padding_mode="zeros". Numeric padding is per-side: (left, right) or (top, bottom, left, right).
  • Choose reusable weight storage when the computation needs to read a weight more than once; see Weight storage requirements.
  • Average pooling divides by the full kernel area, including padded positions. Max pooling treats padded positions as -inf.
1
2
3
4
5
6
7
8
9
with device.get_top():
    legato.tensor.request_load(inp, device.get_core(0), "sram")
    legato.tensor.request_load(weight, device.get_core(0), "sram")

with device.get_core(0):
    x = legato.tensor.receive(inp, 0, "sram")
    w = legato.tensor.receive(weight, 0, "sram")
    y = legato.math.conv_1d(x, w, None, (1,), (1,), (1, 1), "zeros")
    pooled = legato.math.avg_pool_1d(y, (2,), (2,))

RMSNorm and LayerNorm

legato.math.rms_norm and legato.math.layer_norm normalize the last logical axis. Affine operands are rank-1 [D] tensors. LayerNorm supports optional gain and bias. The divisor uses the logical axis length, so padding capacity does not add elements to the mathematical reduction.

See Broadcasting and Normalization examples.

Weight-Quantized Matmul

mm_weight_quant and addmm_weight_quant accept an already-converted right-hand weight. Use hart.weight_converter and pass its opaque rank-1 i8 result through "mpu_packed_buffer". Treat that result as an SDK-produced buffer.

dtype and shape describe the pre-conversion quantized weight [C, M]. granularity is "per_output_channel" or "per_input_channel". For input-channel grouping, quant_group_size defaults to 128, must be at least 128, must be a multiple of 128, and must divide shape[0]. Output-channel-wise quantization ignores the group size.

See Bertha Layouts and the complete Weight-Quantized Matmul examples.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
with device.get_core(0):
    # One scale / zero-point per output channel.
    result = device.mm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [128, 64], "per_output_channel"
    )
    # Input-channel groups of 128 (default).
    grouped = device.mm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [128, 64], "per_input_channel"
    )
    # Input-channel groups of 256.
    grouped256 = device.mm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [256, 64], "per_input_channel", 256
    )
    biased = device.addmm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [128, 64],
        "per_input_channel", legato.types.float("float32"),
    )

Compute Model

Bertha

1
Bertha(ctx: MLIRContext, name: str, params: MLIRBerthaDeviceParams | MLIRBerthaV1DeviceParams | int | None = None, *legacy_args: object, **overrides: object)

Bertha compute model wrapper.

The revision device parameter selects the hardware backend. Parameters default to Bertha EVT0; use legato.model.bertha.evt1 (or pass revision="EVT1") to target EVT1.

A Bertha instance is normally created by the backend factory passed to @legato.compile. The active MLIR context is supplied by the compiler, and the returned compute model becomes available inside kernels through legato.get_context().get_device() or legato.get_model().

The model exposes Bertha-specific helpers for executor selection, memory lookup, tensor type construction, TLB operations, MPU matmul operations, and top/core synchronization barriers.

Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import legato
import legato.model.bertha as bertha

def get_bertha(ctx):
    # Every device parameter defaults to Bertha EVT0; override only
    # what differs.
    return bertha.Bertha(ctx, "bertha", num_cores=32)

@legato.compile(backend=get_bertha)
def kernel():
    device = legato.get_context().get_device()
    with device.get_top():
        ...

__init__

1
__init__(ctx: MLIRContext, name: str, params: MLIRBerthaDeviceParams | MLIRBerthaV1DeviceParams | int | None = None, *legacy_args: object, **overrides: object)

Initialize the Bertha compute model.

This registers the Bertha dialect/model with the active MLIR context and creates the underlying C++ wrapper: MLIRBertha for EVT0, MLIRBerthaV1 (the bertha_v1 compute model) for EVT1. Both are built from the same params object, so the registered model and this wrapper cannot describe different devices.

The revision parameter selects the hardware backend. Every device hyperparameter defaults to that revision's value, defined once on the C++ side (BerthaDeviceParams / BerthaV1DeviceParams). Pass params to supply a fully built parameter object, or keyword overrides to adjust individual fields of the defaults:

1
Bertha(ctx, "bertha", num_cores=16, shared_memory_size=64 * 1024**3)

Parameters:

  • ctx (MLIRContext): Active MLIR context supplied by the compiler.
  • name (str): Symbolic name for this Bertha model instance.
  • params (MLIRBerthaDeviceParams | MLIRBerthaV1DeviceParams | int | None): Device parameters: MLIRBerthaDeviceParams for EVT0 or MLIRBerthaV1DeviceParams for EVT1. Defaults to the description of the revision requested in overrides, else Bertha EVT0. Default: None.
  • *legacy_args (object): Deprecated. Trailing arguments of the old positional signature (num_cores, use_pim, num_memory_channels, shared_memory_size), in which case params is num_cores. Default: ().
  • **overrides (object): Individual MLIRBerthaDeviceParams fields to override. Default: {}.

Raises:

  • (AttributeError): If an override does not name a parameter field.
  • (ValueError): If revision names no known Bertha generation.
  • (RuntimeError): If the model cannot be registered with ctx.

revision

1
revision: str

Silicon revision this model targets, e.g. "EVT0".

Reaches the emitted ELF as the target-variant note, so it must match the device the binary will actually run on.

generation

1
generation: str

Silicon generation ("EVT0" or "EVT1") parsed from revision; it selects the compute model.

is_evt1

1
is_evt1: bool

Whether this model targets Bertha EVT1.

num_cores

1
num_cores: int

Number of Bertha cores exposed to the kernel.

use_pim

1
use_pim: bool

Whether PIM support is enabled for this model.

num_memory_channels

1
num_memory_channels: int

Number of DRAM memory channels.

shared_memory_size

1
shared_memory_size: int

Total shared DRAM size in bytes.

bytes_per_channel

1
bytes_per_channel: int

Physical capacity of one DRAM channel, in bytes.

This is a capacity. It is not the address-space distance between adjacent channel bases -- see dram_channel_stride_bytes, which is what tensor layouts stride by.

dram_channel_stride_bytes

1
dram_channel_stride_bytes: int

Address-space distance between adjacent DRAM channel bases, in bytes.

register_model

1
register_model(ctx: MLIRContext, *args) -> bool

Register the compute model matching the params (EVT0 or EVT1) to the context.

run_backend_passes

1
run_backend_passes(builder: MLIROpBuilder) -> bool

Run the backend passes of this revision's compute model (EVT0 or EVT1).

run_binary_backend_passes

1
run_binary_backend_passes(builder: MLIROpBuilder, output_path: str) -> bool

Run this revision's backend passes and translate to a binary.

run_translator

1
run_translator(builder: MLIROpBuilder, output_format: str, output_path: str) -> bool

Run this revision's translator ("asm" or "bin").

get_layout

1
get_layout(rank: Value, dtype: Value, kind: Value, shape: Value | None = None) -> Callable[[CodeGenerator], Value]

Return the Bertha layout object for a tensor kind.

This helper is used by tensor_type and is rarely called directly. kind must be an interpreted string. Supported kinds are:

  • "shared_dram" and "shared_memory": DRAM word-distributed layout.
  • "key_dram" and "transposed_mpu_dram": same key-style DRAM layout.
  • "value_dram" and "mpu_dram": same value-style DRAM layout.
  • "mpu_weight" and "mpu_weight_sram": MPU tensor layout.
  • "mpu_packed_buffer": row-major layout in MPU buffer memory, for an already-packed opaque rank-1 buffer (no MPU tiling applied).
  • "row_major": row-major layout.
  • "sliced_dram": sliced DRAM layout; requires shape.

Parameters:

  • rank (Value): Interpreted tensor rank.
  • dtype (Value): Interpreted tensor element type.
  • kind (Value): Interpreted layout kind string.
  • shape (Value | None): Interpreted tensor shape. Required for "sliced_dram". Default: None.

Returns:

  • (Callable[[CodeGenerator], Value]): A callable that produces the interpreted layout value during codegen.

Raises:

  • (ValueError): If kind is "sliced_dram" and shape is missing.
  • (NotImplementedError): If kind is not supported by this Bertha revision.

tensor_type

1
tensor_type(dtype: Value, shape: Value, kind: Value) -> Callable[[CodeGenerator], Value]

Create a Bertha tensor type for the given dtype, shape, and kind.

This is the preferred way to annotate tensors that live in Bertha memory spaces. It combines a generic tensor type with a Bertha memory object and a Bertha layout derived from kind.

Common kind values:

  • "shared_memory" or "shared_dram" for shared DRAM tensors.
  • "row_major" for core-local SRAM tensors.
  • "mpu_dram" as an alias of "value_dram" for the value-style MPU/TLB path.
  • "transposed_mpu_dram" as an alias of "key_dram" for the key-style MPU/TLB path.
  • "mpu_weight" for values received through the MPU weight buffer.
  • "mpu_weight_sram" for MPU-format weights received into core SRAM and reused by matmul. Uses the same layout family as "mpu_weight".
  • "mpu_packed_buffer" for an already-packed opaque rank-1 buffer (e.g. a weight-quantized weight blob) streamed straight from DRAM to the MPU buffer, with no MPU tiling applied.
  • "sliced_dram", "key_dram", and "value_dram" for specialized DRAM layout names.

Dynamic dimensions are written with (-1, max_dim) in the shape.

Example:

1
2
3
4
5
x_type = legato.get_model().tensor_type(
    legato.types.float("bfloat16"),
    ((-1, 10), 64),
    "shared_dram",
)

Parameters:

  • dtype (Value): Interpreted element type.
  • shape (Value): Interpreted tensor shape.
  • kind (Value): Interpreted Bertha tensor kind string.

Returns:

  • (Callable[[CodeGenerator], Value]): A callable that produces an interpreted tensor type during codegen.

Raises:

  • (NotImplementedError): If kind does not map to a Bertha memory kind.

get_num_cores

1
get_num_cores() -> Callable[[CodeGenerator], Value]

Return the number of Bertha cores as an interpreted value.

This is useful for static generation patterns, for example unrolling a request loop over all available cores with legato.range.

get_top

1
get_top() -> Callable[[CodeGenerator], Value]

Return the Bertha top executor.

Use this in a with statement to coordinate input requests and top/core synchronization.

Example:

1
2
3
device = legato.get_context().get_device()
with device.get_top():
    legato.tensor.request_load(x, device.get_core(0), "sram")

get_core

1
get_core(index: Value) -> Callable[[CodeGenerator], Value]

Return one Bertha core executor by static core index.

Parameters:

  • index (Value): Interpreted integer core index.

Returns:

  • (Callable[[CodeGenerator], Value]): A callable that produces a BerthaCore value.

get_cores

1
get_cores(indices: Value) -> Callable[[CodeGenerator], list[Value]]

Return a list of Bertha core executors.

Parameters:

  • indices (Value): Interpreted list of interpreted integer core indices.

Returns:

  • (Callable[[CodeGenerator], list[Value]]): A callable that produces a list of BerthaCore values.

get_all_cores

1
get_all_cores() -> Callable[[CodeGenerator], list[Value]]

Return all Bertha core executors.

Use this as a context when the region should be emitted for every core.

get_use_pim

1
get_use_pim() -> Callable[[CodeGenerator], Value]

Return whether this Bertha model was configured to use PIM.

get_shared_memory_size

1
get_shared_memory_size() -> Callable[[CodeGenerator], Value]

Return the configured shared DRAM size in bytes.

get_shared_memory

1
get_shared_memory() -> Callable[[CodeGenerator], Value]

Return the shared Bertha memory object.

Tensor kinds such as "shared_dram" and "shared_memory" use this memory object. Use "mpu_dram" / "transposed_mpu_dram" kinds for layout-specific shared pool memories.

get_instruction_table

1
get_instruction_table() -> Callable[[CodeGenerator], Value]

Return the instruction table memory object.

get_mpu_dram

1
get_mpu_dram() -> Callable[[CodeGenerator], Value]

Return the Bertha MPU value-layout shared DRAM memory.

get_mpu_transposed_dram

1
get_mpu_transposed_dram() -> Callable[[CodeGenerator], Value]

Return the Bertha MPU key-layout shared DRAM memory.

get_mpu_buffer

1
get_mpu_buffer(index: Value) -> Callable[[CodeGenerator], Value]

Return an MPU buffer memory object by index.

Parameters:

  • index (Value): Interpreted integer MPU buffer index.

Returns:

  • (Callable[[CodeGenerator], Value]): A callable that produces a Memory value for the MPU buffer.

get_core_sram

1
get_core_sram(index: Value) -> Callable[[CodeGenerator], Value]

Return a core SRAM memory object by core index.

Parameters:

  • index (Value): Interpreted integer core index.

get_core_private_dram

1
get_core_private_dram(index: Value) -> Callable[[CodeGenerator], Value]

Return a core-private DRAM memory object by core index.

Parameters:

  • index (Value): Interpreted integer core index.

write_tlb

1
write_tlb(is_key: Value, dst: Value, data: Value, request_id: Value, layer: Value, head: Value) -> Callable[[CodeGenerator], Value]

Write a key or value tensor through the Bertha TLB path.

This Bertha EVT0 operation writes data into dst using the TLB address path. is_key indicates whether the operation is in key mode or value mode. If input is in key mode, but layout turns out to be incompatible, the operation will fail. Value mode works same in vice versa.

request_id, layer, and head may be interpreted scalar constants or compiled index values; interpreted values are forwarded to compiled values before the operation is emitted.

Parameters:

  • is_key (Value): Interpreted boolean indicating whether the operation is in key mode.
  • dst (Value): Destination tensor in a supported TLB DRAM layout.
  • data (Value): Source tensor to write.
  • request_id (Value): Request id index.
  • layer (Value): Layer id index.
  • head (Value): Head id index.

Returns:

  • (Callable[[CodeGenerator], Value]): The emitted TLB write operation result.

Example:

1
2
3
with device.get_core(0):
    x = legato.tensor.receive(input_tensor, 0, "sram")
    device.write_tlb(output, x, request_id, layer, head)

mm_weight_quant

1
mm_weight_quant(lhs: Value, rhs: Value, dtype: Value, shape: Value, granularity: Value, quant_group_size: Value | None = None) -> Callable[[CodeGenerator], Value]

Perform weight-quantized matrix multiplication on the Bertha MPU.

rhs is an already weight-quantized and packed rank-1 i8 buffer received into the mpu_buffer storage (mpu_packed_buffer or mpu_weight). SRAM is rejected. Use the SDK weight converter to prepare the opaque buffer. dtype/shape describe rhs's pre-packing quantized layout, granularity its quantization grouping. Result dtype matches lhs; result shape follows the conventional matmul rule [H, C] @ [C, M] => [H, M], where [C, M] = shape.

Parameters:

  • lhs (Value): Left-hand side compiled tensor, rank 2, shape [H, C].
  • rhs (Value): Packed weight buffer in the MPU buffer, a compiled rank-1 i8 tensor.
  • dtype (Value): Interpreted pre-packing quantized weight element type.
  • shape (Value): Interpreted pre-packing quantized weight shape [C, M].
  • granularity (Value): Interpreted quantization granularity, "per_input_channel" or "per_output_channel".
  • quant_group_size (Value | None): Optional actual input-channel group size (128, 256, ...). Omitted values default to 128. Used when granularity is "per_input_channel"; ignored for "per_output_channel". Default: None.

Returns:

  • (Callable[[CodeGenerator], Value]): Result tensor.

Example:

1
2
3
4
5
6
7
8
with device.get_core(0):
    result = device.mm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [256, 512], "per_output_channel"
    )
    grouped = device.mm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [256, 512], "per_input_channel"
    )
    yield result

addmm_weight_quant

1
addmm_weight_quant(lhs: Value, rhs: Value, dtype: Value, shape: Value, granularity: Value, bias_dtype: Value, quant_group_size: Value | None = None) -> Callable[[CodeGenerator], Value]

Perform weight-quantized matrix multiplication with fused bias on the Bertha MPU.

Same as mm_weight_quant, including the requirement that rhs live in the MPU buffer, but rhs's packing additionally contains bias data (opaque to Legato). bias_dtype records the pre-packing bias element type.

Parameters:

  • lhs (Value): Left-hand side compiled tensor, rank 2, shape [H, C].
  • rhs (Value): Packed weight+bias buffer in the MPU buffer, a compiled rank-1 i8 tensor.
  • dtype (Value): Interpreted pre-packing quantized weight element type.
  • shape (Value): Interpreted pre-packing quantized weight shape [C, M].
  • granularity (Value): Interpreted quantization granularity, "per_input_channel" or "per_output_channel".
  • bias_dtype (Value): Interpreted pre-packing bias element type.
  • quant_group_size (Value | None): Optional actual input-channel group size (128, 256, ...). Omitted values default to 128. Used when granularity is "per_input_channel"; ignored for "per_output_channel". Default: None.

Returns:

  • (Callable[[CodeGenerator], Value]): Result tensor.

Example:

1
2
3
4
5
6
with device.get_core(0):
    result = device.addmm_weight_quant(
        lhs, packed_rhs, legato.types.float("float8"), [256, 512],
        "per_output_channel", legato.types.float("float32"),
    )
    yield result

request_load_tlb

1
request_load_tlb(src: Value, target_devices: Value, request_id: Value, layer: Value, head: Value, target_name: Value | None = None) -> Callable[[CodeGenerator], Value]

Request a TLB load to interpreted target devices.

This Bertha-specific operation asks the TLB path to load src for one or more target executors or memories. The target device list is known at compile time, while request metadata is emitted as compiled index values.

Parameters:

  • src (Value): Source tensor in a TLB-backed DRAM layout.
  • target_devices (Value): Interpreted Executor, Memory, or list/tuple of interpreted Executor/Memory values.
  • request_id (Value): Request id index.
  • layer (Value): Layer id index.
  • head (Value): Head id index.
  • target_name (Value | None): Interpreted destination, "mpu_buffer" (default) or "sram". For SRAM, tensor.receive(src, handle, "sram") infers the MPU weight layout for matmul RHS use. Explicit "mpu_weight_sram" with receive_type is also supported. Default: None.

Returns:

  • (Callable[[CodeGenerator], Value]): No tensor value; emits the TLB load request side effect.

Example:

1
2
with device.get_top():
    device.request_load_tlb(k_cache, [device.get_core(0)], request_id, layer, head)

request_load_tlb_dynamic

1
request_load_tlb_dynamic(src: Value, target_devices: Value, request_id: Value, layer: Value, head: Value, target_name: Value | None = None) -> Callable[[CodeGenerator], Value]

Request a TLB load with a compiled target-device bitmap.

Use this variant when the target core set is a runtime/compiled index value rather than an interpreted list of executors.

Parameters:

  • src (Value): Source tensor in a TLB-backed DRAM layout.
  • target_devices (Value): Compiled index-like device bitmap.
  • request_id (Value): Request id index.
  • layer (Value): Layer id index.
  • head (Value): Head id index.
  • target_name (Value | None): Interpreted destination, "mpu_buffer" (default) or "sram". For SRAM, tensor.receive(src, handle, "sram") infers the MPU weight layout for matmul RHS use. Explicit "mpu_weight_sram" with receive_type is also supported. Default: None.

Returns:

  • (Callable[[CodeGenerator], Value]): No tensor value; emits the dynamic TLB load request side effect.

sort

1
sort(logit: Value, descending: bool = True) -> Callable[[CodeGenerator], tuple[Value, Value]]

Sort a logit vector on the selected backend.

The caller must supply only finite logical values: NaN and infinities are unsupported. Values are not checked at runtime. For static shapes, results retain the logical input length; physical capacity is the next power of two at or above the input's physical length.

Parameters:

  • logit (Value): 1-D tensor in SRAM to sort.
  • descending (bool): Sort order. Only descending (True) is currently supported. Default: True.

Returns:

  • (Callable[[CodeGenerator], tuple[Value, Value]]): Tuple containing:
    • sorted_logit: 1-D sorted logit tensor in SRAM
    • sorted_index: 1-D sorted index tensor in SRAM

top_kp

1
top_kp(sorted_logit: Value, sorted_index: Value, params: Value) -> Callable[[CodeGenerator], tuple[Value, Value]]

Apply top-k/top-p filtering and sample a token on the selected backend.

Fused operation: applies top-k/top-p condition and returns the selected logit-index pair. Parameters (top-k limit, top-p probability, random weight) are referenced by the index/address value given in params.

Parameters:

  • sorted_logit (Value): 1-D sorted logit tensor in SRAM (output of sort).
  • sorted_index (Value): 1-D sorted index tensor in SRAM (output of sort).
  • params (Value): Index/address value referencing the prepared sampling-parameter block.

Returns:

  • (Callable[[CodeGenerator], tuple[Value, Value]]): Tuple containing:
    • sampled_logit: sampled logit in SRAM scalar tensor
    • sampled_index: sampled index in SRAM scalar tensor

Device-agnostic sync barriers live in legato.sync (see Synchronization). The legacy top_consume / top_produce / core_consume / core_produce stubs were removed.

Executors

BerthaTop

1
BerthaTop(core: MLIRExecutor)

Bertha top executor wrapper.

Values of this type are produced by Bertha.get_top() and are used in with contexts to emit top-level Bertha work such as load requests and top/core synchronization.

__init__

1
__init__(core: MLIRExecutor)

Initialize a Bertha top executor wrapper.

Users should not instantiate this class directly. Instances are created by the Bertha backend bindings and returned from Bertha.get_top().

Parameters:

  • core (MLIRExecutor): Underlying MLIR executor handle.

BerthaCore

1
BerthaCore(core: MLIRExecutor, core_id: int | None = None)

Bertha core executor wrapper.

Values of this type are produced by Bertha.get_core(), Bertha.get_cores(), and Bertha.get_all_cores(). Use them in with contexts to emit work for one core or a group of cores.

__init__

1
__init__(core: MLIRExecutor, core_id: int | None = None)

Initialize a Bertha core executor wrapper.

Users should not instantiate this class directly. Instances are created by the Bertha backend bindings and returned from Bertha executor accessors.

Parameters:

  • core (MLIRExecutor): Underlying MLIR executor handle.
  • core_id (int | None): Static core index this executor was created with, if known. Set by Bertha.get_core()/get_cores()/get_all_cores(), since no core-id accessor is exposed on the underlying MLIR executor binding. Default: None.