Bertha API¶
This page documents Bertha-specific Python APIs from
legato.model.bertha.operations.
Bertha APIs are backend-specific helpers layered on top of the generic Legato language operations. They include the Bertha compute model constructor, executor accessors, memory/type helpers, MPU/TLB operations, and Bertha synchronization operations.
Typical Backend Setup¶
1 2 3 4 5 6 | |
Use this backend factory with @legato.compile(backend=get_bertha).
Target generations¶
Bertha(ctx, "bertha") selects EVT0 defaults. Override individual parameters by
keyword (num_cores=..., num_memory_channels=...), or pass a
MLIRBerthaDeviceParams instance. The old positional parameter list is
deprecated; it does not describe EVT1.
Use backend=bertha.evt1 or Bertha(ctx, "bertha", revision="EVT1") to select
the bertha_v1 compute model and EVT1 device parameters. Parameter objects must
match their generation. EVT1 has its own lowering and translator pipelines;
EVT0 examples and hardware layout constants cannot be reused as an EVT1 support
claim. The Python wrapper explicitly rejects EVT0-only TLB, packed-weight
quantized matmul, and top_kp helpers on EVT1. device.sort also supports EVT1;
it currently requires descending order and finite logical inputs. Check an individual
operation through the requested stage and matching target runtime.
Verified EVT1 paths¶
The current EVT1 backend includes the following paths. These are compiler and simulator support statements for the listed cases, rather than a hardware reference or a claim that every EVT0 operation is available on EVT1.
| Operation | Current coverage and limits |
|---|---|
Tensor -, *, / |
Matching-shape BF16 tensors run on the EVT1 VPU, alongside the existing + path. The numerical cases load [16, 256] inputs and store the flattened result. |
SRAM to shared_dram with legato.tensor.memcpy |
Word-interleaved DRAM write-back is numerically covered for BF16 output ranks 1 through 6. Keep input and output storage kinds compatible. |
Dynamic or padded shared-DRAM request_load / receive |
Runtime row counts size whole-line transfers; the other dimensions use their physical extents. Destination SRAM physical extents must still be static. |
| Last-axis BF16 softmax | The reduce_max → sub → exp → reduce_sum → div pattern is covered for [32, 256] input. Use keepdim=True; the reduced width must be a multiple of 64 BF16 elements. |
device.sort |
Sorting is supported in descending order for finite inputs. This does not extend top_kp to EVT1. |
| Instruction-table reads and embedding gather | Scalar reads and integer casts are supported. tensor.gather(weight, indices) supports instruction-table integer indices with a sliced_dram table whose row count is divisible by the number of DRAM channels. Instruction-table tensor views and arbitrary SRAM-index gathers are not implied. |
Runtime while and nearest-loop break |
Integer/index/bool carried state is supported on EVT0 and EVT1. continue, loop-else, early return from the loop body, and tensor carried state remain unsupported. |
| RoPE | The repository's BF16 prefill and decode kernels run against CPU references on the EVT1 simulator. They stage padded cosine/sine tables; this is a verified kernel composition, not a separate rope API. |
Select EVT1 explicitly with backend=bertha.evt1 and an EVT1 runtime or
simulator. Whole-line receive requirements still apply; short transfers and
unequal per-channel word counts are separate lowering cases, not a reason to
assume arbitrary unaligned copies are valid. See
Memory Operations and
Arithmetic Operations.
Common Tensor Kinds¶
Bertha.tensor_type(dtype, shape, kind) accepts Bertha-specific kind strings.
Common values include:
"shared_memory"and"shared_dram""row_major""value_dram"and"mpu_dram"for the same value-style DRAM layout"key_dram"and"transposed_mpu_dram"for the same key-style DRAM layout"mpu_weight""mpu_weight_sram"for MPU-format weights resident in core SRAM"mpu_packed_buffer"for an opaque rank-1 packed-weight blob"sliced_dram"
See Bertha Layouts for layout constraints and memory placement.
TLB Loads To SRAM (EVT0)¶
request_load_tlb and request_load_tlb_dynamic accept a trailing destination
argument: "mpu_buffer" (the default) or "sram". The static API takes an
interpreted core list; the dynamic API takes a compiled core bitmap.
1 2 | |
Receive SRAM weights with legato.tensor.receive(cache, handle, "sram").
Bertha EVT0 infers the MPU-format layout and its layout-family tag from
the source, so the resulting SRAM tensor can be reused as a matmul RHS.
Explicit tensor_type(..., "mpu_weight_sram") with receive_type is also
supported. The same inference applies to ordinary
legato.tensor.request_load(cache, core, "sram").
Omitting the request destination keeps the existing MPU-buffer behavior and
uses tensor_type(..., "mpu_weight") with a "mpu_buffer" receive.
The following value-cache example loads one weight into SRAM and uses it in two matmuls with different activations:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | |
Initialize the EVT0 TLB and allocate the request's key/value pages before
launching the kernel; see the HART TLB API.
Bind cache to bertha.top and each output to bertha.core(0) at launch.
The weight above is received once and remains in SRAM for both matmuls.
A "key_dram" input with logical shape [N, K] is received with shape
[K, N]; a "value_dram" input [K, N] keeps that logical shape.
The resulting weight may be reused as a matmul RHS. It is not a general-purpose
matmul LHS, arithmetic, reduction, or normalization input.
Keep the source and receive shapes compatible with the selected tensor kind;
see Bertha Layouts.
Depthwise Convolution¶
legato.math.dwconv_1d and legato.math.dwconv_2d operate on rank-2 and
rank-3 spatial inputs respectively. The 2D defaults are stride=(1, 1)
and padding=(0, 0, 0, 0).
- 1D: input
[X, C], weight[KX, C], result[OX, C]. - 2D: input
[Y, X, C], weight[KY, KX, C], result[OY, OX, C]. - The current EVT0 implementation requires static BF16 inputs, positive
C,X * Y <= 32, kernel sizes 1–8, strides 1–2, and padding 0–7. - Load activations into core SRAM. Weights may be loaded with
request_load(..., "mpu_buffer")or into SRAM using a compatible tensor kind. - Bias and SiLU are separate operations; add them explicitly when needed.
Weight storage requirements¶
Keep the documented logical weight shape. Use the SDK's tensor kinds and allocation or conversion helpers to provide compatible storage. The compiler handles the required weight formatting and padding; applications should not construct or depend on its byte representation.
Physical capacity must satisfy the selected tensor kind. For a padded tensor, allocate and initialize the backing storage before passing a logical view. Reusable weights must remain available for every consumer; a one-use streamed weight cannot be reused as if it were a persistent SRAM tensor. See Bertha Layouts.
1 2 3 4 5 6 7 8 9 | |
Convolution and Pooling¶
legato.math.conv_1d / conv_2d and the max_pool_* / avg_pool_* APIs
support the following EVT0 input contract:
- Layout is channels-last: 1D
[N*, X, C], 2D[N*, H, W, C]. Convolution weights are[KX, Cin, Cout]or[KH, KW, Cin, Cout], with optional bias[Cout]. N*denotes leading batch dimensions; see Broadcasting and Broadcasting on Bertha.- Current EVT0 support requires BF16, kernel sizes 1–8, strides 1–2,
padding 0–7, unit dilation, and
padding_mode="zeros". Numeric padding is per-side:(left, right)or(top, bottom, left, right). - Choose reusable weight storage when the computation needs to read a weight more than once; see Weight storage requirements.
- Average pooling divides by the full kernel area, including padded positions.
Max pooling treats padded positions as
-inf.
1 2 3 4 5 6 7 8 9 | |
RMSNorm and LayerNorm¶
legato.math.rms_norm and legato.math.layer_norm normalize the last logical
axis. Affine operands are rank-1 [D] tensors. LayerNorm supports optional
gain and bias. The divisor uses the logical axis length, so padding capacity
does not add elements to the mathematical reduction.
See Broadcasting and Normalization examples.
Weight-Quantized Matmul¶
mm_weight_quant and addmm_weight_quant accept an already-converted right-hand
weight. Use hart.weight_converter and pass its opaque rank-1 i8 result
through "mpu_packed_buffer". Treat that result as an SDK-produced buffer.
dtype and shape describe the pre-conversion quantized weight [C, M].
granularity is "per_output_channel" or "per_input_channel".
For input-channel grouping, quant_group_size defaults to 128, must be at least
128, must be a multiple of 128, and must divide shape[0].
Output-channel-wise quantization ignores the group size.
See Bertha Layouts and the complete Weight-Quantized Matmul examples.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | |
Compute Model¶
Bertha¶
1 | |
Bertha compute model wrapper.
The revision device parameter selects the hardware backend. Parameters
default to Bertha EVT0; use legato.model.bertha.evt1 (or pass
revision="EVT1") to target EVT1.
A Bertha instance is normally created by the backend factory passed to
@legato.compile. The active MLIR context is supplied by the compiler, and
the returned compute model becomes available inside kernels through
legato.get_context().get_device() or legato.get_model().
The model exposes Bertha-specific helpers for executor selection, memory lookup, tensor type construction, TLB operations, MPU matmul operations, and top/core synchronization barriers.
Example:
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
__init__¶
1 | |
Initialize the Bertha compute model.
This registers the Bertha dialect/model with the active MLIR context and
creates the underlying C++ wrapper: MLIRBertha for EVT0, MLIRBerthaV1
(the bertha_v1 compute model) for EVT1. Both are built from the same
params object, so the registered model and this wrapper cannot
describe different devices.
The revision parameter selects the hardware backend. Every device
hyperparameter defaults to that revision's value, defined once on the
C++ side (BerthaDeviceParams / BerthaV1DeviceParams). Pass params
to supply a fully built parameter object, or keyword overrides to
adjust individual fields of the defaults:
1 | |
Parameters:
ctx(MLIRContext): Active MLIR context supplied by the compiler.name(str): Symbolic name for this Bertha model instance.params(MLIRBerthaDeviceParams | MLIRBerthaV1DeviceParams | int | None): Device parameters:MLIRBerthaDeviceParamsfor EVT0 orMLIRBerthaV1DeviceParamsfor EVT1. Defaults to the description of the revision requested inoverrides, else Bertha EVT0. Default:None.*legacy_args(object): Deprecated. Trailing arguments of the old positional signature(num_cores, use_pim, num_memory_channels, shared_memory_size), in which caseparamsisnum_cores. Default:().**overrides(object): IndividualMLIRBerthaDeviceParamsfields to override. Default:{}.
Raises:
- (
AttributeError): If an override does not name a parameter field. - (
ValueError): Ifrevisionnames no known Bertha generation. - (
RuntimeError): If the model cannot be registered withctx.
revision¶
1 | |
Silicon revision this model targets, e.g. "EVT0".
Reaches the emitted ELF as the target-variant note, so it must match the device the binary will actually run on.
generation¶
1 | |
Silicon generation ("EVT0" or "EVT1") parsed from revision; it selects the compute model.
is_evt1¶
1 | |
Whether this model targets Bertha EVT1.
num_cores¶
1 | |
Number of Bertha cores exposed to the kernel.
use_pim¶
1 | |
Whether PIM support is enabled for this model.
num_memory_channels¶
1 | |
Number of DRAM memory channels.
shared_memory_size¶
1 | |
Total shared DRAM size in bytes.
bytes_per_channel¶
1 | |
Physical capacity of one DRAM channel, in bytes.
This is a capacity. It is not the address-space distance between
adjacent channel bases -- see dram_channel_stride_bytes, which is what
tensor layouts stride by.
dram_channel_stride_bytes¶
1 | |
Address-space distance between adjacent DRAM channel bases, in bytes.
register_model¶
1 | |
Register the compute model matching the params (EVT0 or EVT1) to the context.
run_backend_passes¶
1 | |
Run the backend passes of this revision's compute model (EVT0 or EVT1).
run_binary_backend_passes¶
1 | |
Run this revision's backend passes and translate to a binary.
run_translator¶
1 | |
Run this revision's translator ("asm" or "bin").
get_layout¶
1 | |
Return the Bertha layout object for a tensor kind.
This helper is used by tensor_type and is rarely called directly.
kind must be an interpreted string. Supported kinds are:
"shared_dram"and"shared_memory": DRAM word-distributed layout."key_dram"and"transposed_mpu_dram": same key-style DRAM layout."value_dram"and"mpu_dram": same value-style DRAM layout."mpu_weight"and"mpu_weight_sram": MPU tensor layout."mpu_packed_buffer": row-major layout in MPU buffer memory, for an already-packed opaque rank-1 buffer (no MPU tiling applied)."row_major": row-major layout."sliced_dram": sliced DRAM layout; requiresshape.
Parameters:
rank(Value): Interpreted tensor rank.dtype(Value): Interpreted tensor element type.kind(Value): Interpreted layout kind string.shape(Value | None): Interpreted tensor shape. Required for"sliced_dram". Default:None.
Returns:
- (
Callable[[CodeGenerator], Value]): A callable that produces the interpreted layout value during codegen.
Raises:
- (
ValueError): Ifkindis"sliced_dram"andshapeis missing. - (
NotImplementedError): Ifkindis not supported by this Bertha revision.
tensor_type¶
1 | |
Create a Bertha tensor type for the given dtype, shape, and kind.
This is the preferred way to annotate tensors that live in Bertha memory
spaces. It combines a generic tensor type with a Bertha memory object and
a Bertha layout derived from kind.
Common kind values:
"shared_memory"or"shared_dram"for shared DRAM tensors."row_major"for core-local SRAM tensors."mpu_dram"as an alias of"value_dram"for the value-style MPU/TLB path."transposed_mpu_dram"as an alias of"key_dram"for the key-style MPU/TLB path."mpu_weight"for values received through the MPU weight buffer."mpu_weight_sram"for MPU-format weights received into core SRAM and reused by matmul. Uses the same layout family as"mpu_weight"."mpu_packed_buffer"for an already-packed opaque rank-1 buffer (e.g. a weight-quantized weight blob) streamed straight from DRAM to the MPU buffer, with no MPU tiling applied."sliced_dram","key_dram", and"value_dram"for specialized DRAM layout names.
Dynamic dimensions are written with (-1, max_dim) in the shape.
Example:
1 2 3 4 5 | |
Parameters:
dtype(Value): Interpreted element type.shape(Value): Interpreted tensor shape.kind(Value): Interpreted Bertha tensor kind string.
Returns:
- (
Callable[[CodeGenerator], Value]): A callable that produces an interpreted tensor type during codegen.
Raises:
- (
NotImplementedError): Ifkinddoes not map to a Bertha memory kind.
get_num_cores¶
1 | |
Return the number of Bertha cores as an interpreted value.
This is useful for static generation patterns, for example unrolling a
request loop over all available cores with legato.range.
get_top¶
1 | |
Return the Bertha top executor.
Use this in a with statement to coordinate input requests and top/core synchronization.
Example:
1 2 3 | |
get_core¶
1 | |
Return one Bertha core executor by static core index.
Parameters:
index(Value): Interpreted integer core index.
Returns:
- (
Callable[[CodeGenerator], Value]): A callable that produces aBerthaCorevalue.
get_cores¶
1 | |
Return a list of Bertha core executors.
Parameters:
indices(Value): Interpreted list of interpreted integer core indices.
Returns:
- (
Callable[[CodeGenerator], list[Value]]): A callable that produces a list ofBerthaCorevalues.
get_all_cores¶
1 | |
Return all Bertha core executors.
Use this as a context when the region should be emitted for every core.
get_use_pim¶
1 | |
Return whether this Bertha model was configured to use PIM.
get_shared_memory_size¶
1 | |
Return the configured shared DRAM size in bytes.
get_shared_memory¶
1 | |
Return the shared Bertha memory object.
Tensor kinds such as "shared_dram" and "shared_memory" use this
memory object. Use "mpu_dram" / "transposed_mpu_dram" kinds for
layout-specific shared pool memories.
get_instruction_table¶
1 | |
Return the instruction table memory object.
get_mpu_dram¶
1 | |
Return the Bertha MPU value-layout shared DRAM memory.
get_mpu_transposed_dram¶
1 | |
Return the Bertha MPU key-layout shared DRAM memory.
get_mpu_buffer¶
1 | |
Return an MPU buffer memory object by index.
Parameters:
index(Value): Interpreted integer MPU buffer index.
Returns:
- (
Callable[[CodeGenerator], Value]): A callable that produces aMemoryvalue for the MPU buffer.
get_core_sram¶
1 | |
Return a core SRAM memory object by core index.
Parameters:
index(Value): Interpreted integer core index.
get_core_private_dram¶
1 | |
Return a core-private DRAM memory object by core index.
Parameters:
index(Value): Interpreted integer core index.
write_tlb¶
1 | |
Write a key or value tensor through the Bertha TLB path.
This Bertha EVT0 operation writes data into dst using the TLB
address path. is_key indicates whether the operation is in key mode or value mode.
If input is in key mode, but layout turns out to be incompatible, the operation will fail.
Value mode works same in vice versa.
request_id, layer, and head may be interpreted scalar constants or
compiled index values; interpreted values are forwarded to compiled
values before the operation is emitted.
Parameters:
is_key(Value): Interpreted boolean indicating whether the operation is in key mode.dst(Value): Destination tensor in a supported TLB DRAM layout.data(Value): Source tensor to write.request_id(Value): Request id index.layer(Value): Layer id index.head(Value): Head id index.
Returns:
- (
Callable[[CodeGenerator], Value]): The emitted TLB write operation result.
Example:
1 2 3 | |
mm_weight_quant¶
1 | |
Perform weight-quantized matrix multiplication on the Bertha MPU.
rhs is an already weight-quantized and packed rank-1 i8 buffer
received into the mpu_buffer storage (mpu_packed_buffer or mpu_weight).
SRAM is rejected. Use the SDK weight converter to prepare the opaque buffer.
dtype/shape describe rhs's pre-packing quantized layout,
granularity its quantization grouping. Result dtype matches lhs;
result shape follows the conventional matmul rule
[H, C] @ [C, M] => [H, M], where [C, M] = shape.
Parameters:
lhs(Value): Left-hand side compiled tensor, rank 2, shape [H, C].rhs(Value): Packed weight buffer in the MPU buffer, a compiled rank-1 i8 tensor.dtype(Value): Interpreted pre-packing quantized weight element type.shape(Value): Interpreted pre-packing quantized weight shape [C, M].granularity(Value): Interpreted quantization granularity,"per_input_channel"or"per_output_channel".quant_group_size(Value | None): Optional actual input-channel group size (128, 256, ...). Omitted values default to 128. Used whengranularityis"per_input_channel"; ignored for"per_output_channel". Default:None.
Returns:
- (
Callable[[CodeGenerator], Value]): Result tensor.
Example:
1 2 3 4 5 6 7 8 | |
addmm_weight_quant¶
1 | |
Perform weight-quantized matrix multiplication with fused bias on the Bertha MPU.
Same as mm_weight_quant, including the requirement that rhs live
in the MPU buffer, but rhs's packing additionally contains bias
data (opaque to Legato). bias_dtype records the pre-packing bias
element type.
Parameters:
lhs(Value): Left-hand side compiled tensor, rank 2, shape [H, C].rhs(Value): Packed weight+bias buffer in the MPU buffer, a compiled rank-1 i8 tensor.dtype(Value): Interpreted pre-packing quantized weight element type.shape(Value): Interpreted pre-packing quantized weight shape [C, M].granularity(Value): Interpreted quantization granularity,"per_input_channel"or"per_output_channel".bias_dtype(Value): Interpreted pre-packing bias element type.quant_group_size(Value | None): Optional actual input-channel group size (128, 256, ...). Omitted values default to 128. Used whengranularityis"per_input_channel"; ignored for"per_output_channel". Default:None.
Returns:
- (
Callable[[CodeGenerator], Value]): Result tensor.
Example:
1 2 3 4 5 6 | |
request_load_tlb¶
1 | |
Request a TLB load to interpreted target devices.
This Bertha-specific operation asks the TLB path to load src for one
or more target executors or memories. The target device list is known at
compile time, while request metadata is emitted as compiled index values.
Parameters:
src(Value): Source tensor in a TLB-backed DRAM layout.target_devices(Value): InterpretedExecutor,Memory, or list/tuple of interpretedExecutor/Memoryvalues.request_id(Value): Request id index.layer(Value): Layer id index.head(Value): Head id index.target_name(Value | None): Interpreted destination, "mpu_buffer" (default) or "sram". For SRAM,tensor.receive(src, handle, "sram")infers the MPU weight layout for matmul RHS use. Explicit"mpu_weight_sram"withreceive_typeis also supported. Default:None.
Returns:
- (
Callable[[CodeGenerator], Value]): No tensor value; emits the TLB load request side effect.
Example:
1 2 | |
request_load_tlb_dynamic¶
1 | |
Request a TLB load with a compiled target-device bitmap.
Use this variant when the target core set is a runtime/compiled index value rather than an interpreted list of executors.
Parameters:
src(Value): Source tensor in a TLB-backed DRAM layout.target_devices(Value): Compiled index-like device bitmap.request_id(Value): Request id index.layer(Value): Layer id index.head(Value): Head id index.target_name(Value | None): Interpreted destination, "mpu_buffer" (default) or "sram". For SRAM,tensor.receive(src, handle, "sram")infers the MPU weight layout for matmul RHS use. Explicit"mpu_weight_sram"withreceive_typeis also supported. Default:None.
Returns:
- (
Callable[[CodeGenerator], Value]): No tensor value; emits the dynamic TLB load request side effect.
sort¶
1 | |
Sort a logit vector on the selected backend.
The caller must supply only finite logical values: NaN and infinities are unsupported. Values are not checked at runtime. For static shapes, results retain the logical input length; physical capacity is the next power of two at or above the input's physical length.
Parameters:
logit(Value): 1-D tensor in SRAM to sort.descending(bool): Sort order. Only descending (True) is currently supported. Default:True.
Returns:
- (
Callable[[CodeGenerator], tuple[Value, Value]]): Tuple containing:- sorted_logit: 1-D sorted logit tensor in SRAM
- sorted_index: 1-D sorted index tensor in SRAM
top_kp¶
1 | |
Apply top-k/top-p filtering and sample a token on the selected backend.
Fused operation: applies top-k/top-p condition and returns the selected
logit-index pair. Parameters (top-k limit, top-p probability, random weight)
are referenced by the index/address value given in params.
Parameters:
sorted_logit(Value): 1-D sorted logit tensor in SRAM (output ofsort).sorted_index(Value): 1-D sorted index tensor in SRAM (output ofsort).params(Value): Index/address value referencing the prepared sampling-parameter block.
Returns:
- (
Callable[[CodeGenerator], tuple[Value, Value]]): Tuple containing:- sampled_logit: sampled logit in SRAM scalar tensor
- sampled_index: sampled index in SRAM scalar tensor
Device-agnostic sync barriers live in legato.sync (see
Synchronization).
The legacy top_consume / top_produce / core_consume / core_produce
stubs were removed.
Executors¶
BerthaTop¶
1 | |
Bertha top executor wrapper.
Values of this type are produced by Bertha.get_top() and are used in
with contexts to emit top-level Bertha work such as load requests and
top/core synchronization.
__init__¶
1 | |
Initialize a Bertha top executor wrapper.
Users should not instantiate this class directly. Instances are created
by the Bertha backend bindings and returned from Bertha.get_top().
Parameters:
core(MLIRExecutor): Underlying MLIR executor handle.
BerthaCore¶
1 | |
Bertha core executor wrapper.
Values of this type are produced by Bertha.get_core(),
Bertha.get_cores(), and Bertha.get_all_cores(). Use them in with
contexts to emit work for one core or a group of cores.
__init__¶
1 | |
Initialize a Bertha core executor wrapper.
Users should not instantiate this class directly. Instances are created by the Bertha backend bindings and returned from Bertha executor accessors.
Parameters:
core(MLIRExecutor): Underlying MLIR executor handle.core_id(int | None): Static core index this executor was created with, if known. Set byBertha.get_core()/get_cores()/get_all_cores(), since no core-id accessor is exposed on the underlying MLIR executor binding. Default:None.