Instruction Table ABI
Overview
The Instruction Table is a binary data structure used to pass arguments when calling Legato kernels. It provides a low-level interface for marshaling function parameters across the host-device boundary. The table layout follows strict 64-bit alignment rules to ensure correct memory access on hardware.
InstructionTable maintains two logical tables:
- Top instruction table
- Core instruction table
When adding an argument, you can route it to:
- top: inserted only into the top table
- core: inserted only into the core table
- universal: inserted into both tables
The binary ABI layout rules in this document apply identically to each table.
Each emitted table starts with a fixed shared memory-region header (6 x 64-bit fields):
1. standard region base address
2. standard region size
3. cache region base address
4. cache region size
5. reserved region base address
6. reserved region size
This 48-byte header is always emitted before argument payload, and uses the same
values for top and core tables.
Core instruction table has one additional mandatory 64-bit metadata field
immediately after the shared header:
7. core id (raw core id, not bitmask)
Therefore:
- top metadata prefix size: 48 bytes (6 x 64-bit)
- core metadata prefix size: 56 bytes (7 x 64-bit)
Purpose
When a Legato kernel is invoked, parameters are packed into one or both instruction tables depending on parameter kind. Each emitted table is a contiguous memory region and contains:
- Shared memory-region header (6 x 64-bit values)
- Core table only: core id metadata (1 x 64-bit value)
- Tensor metadata (base address and dynamic dimensions)
- Scalar values
- Tuple structures (flattened into their element types)
Data Types
The instruction table supports three fundamental parameter types.
Note: Data-type and alignment rules below describe argument payload layout that
starts immediately after the metadata prefix:
- top table payload starts after 48 bytes
- core table payload starts after 56 bytes
1. Tensor Type
Format: tensor := <base_address (64-bit)> <offset (64-bit)> <stride (32-bit unsigned)>* <dim (32-bit unsigned)>*
Size: 16 + 4 * <rank> + 4 * <num_dynamic_dims> bytes
Tensors are represented as:
- Base address (8 bytes): For DRAM tensors, a pointer to the tensor data in memory. For instruction-table tensors, an instruction-table-relative byte offset to the element data stored within the same instruction table buffer.
- Offset (8 bytes): Element offset from the base address. Always zero for instruction-table tensors.
- Strides (4 bytes each): One stride for every tensor rank, stored in rank order. Strides are always present and default to 1 when not explicitly specified.
- Dynamic dimensions (4 bytes each): Size of each dynamic dimension, stored after strides in order
- No tensor-local trailing padding: After the tensor payload, the next argument follows the normal scalar/tensor alignment rules
2. Scalar Type
Format: <data (type-dependent size)>
Size: Maximum 8 bytes
Scalars are stored directly in the instruction table with their native size:
- int8: 1 byte
- int16: 2 bytes
- int32: 4 bytes
- int64: 8 bytes
- float32: 4 bytes
- float64: 8 bytes
3. Tuple Type
Format: tuple := <flattened dtype>*
Size: Sum of all bytes required for each flattened dtype, plus padding from alignment rules
Tuples are flattened in-order, then each element is laid out using the tensor and scalar rules.
Alignment Rules
Rule 1: 64-bit Alignment for Tensors and 64-bit Scalars
All tensors start at 8-byte boundaries. Any 64-bit scalar (int64, float64, pointers) is also 8-byte aligned.
Rule 2: Tensor Layout (Base Address + Offset + Strides + Dynamic Dimensions)
Tensor arguments are laid out as:
| <base_address (64-bit)> <offset (64-bit)> <stride0 (32-bit)> ... <strideR (32-bit)> <dim0 (32-bit)> ... <dimN (32-bit)>
|
- Offset is stored immediately after the base address.
- Strides are stored in increasing rank order and are always emitted, even for tensors whose strides were not provided by the caller.
- Dynamic dimensions are stored in increasing order (
dim0, dim1, ...).
- In a 64-bit word, adjacent 32-bit stride or dimension values are packed little-endian: the earlier value occupies the low 32 bits and the next value occupies the high 32 bits.
- If a tensor ends on a 32-bit boundary, the next argument starts immediately after it unless that next argument itself requires stronger alignment.
Example (rank 3, 3 dynamic dims, default strides):
| Memory (bytes, little-endian):
| 64-bit base addr | 64-bit offset | stride0 | stride1 | stride2 | dim0 | dim1 | dim2 |
64-bit word contents (little-endian):
Word0: <base addr>
Word1: <offset>
Word2: <stride1> <stride0>
Word3: <dim0> <stride2>
Word4: <dim2> <dim1>
|
Rule 3: Scalar Packing Within 64-bit Slots
Scalars are packed in argument order into 64-bit slots.
- Write scalars sequentially from the current 64-bit aligned address.
- If a scalar would cross a 64-bit boundary, pad the remaining bytes in the current slot and start the scalar at the next 64-bit aligned address.
- If there is remaining space in a slot, the next scalar fills it (no premature padding).
Key principle: arg0 is placed at the least significant bytes of the first 64-bit slot (little-endian).
Rule 4: Tuple Flattening
Tuples are flattened in-order, then packed using the tensor and scalar rules above.
Example tuple:
| tuple<scalar, tuple<scalar, tensor>, scalar>
|
Flattened order:
| scalar, scalar, tensor, scalar
|
Memory Layout Examples
Example 1: Simple Scalar Arguments (int64)
| Offset | Size | Content
-------|------|--------
0x00 | 8 | arg0 (int64)
0x08 | 8 | arg1 (int64)
-------|------|--------
Total: 16 bytes
|
Example 2: Scalar Packing with Boundary Padding
Arguments in order: arg0: int32, arg1: int16, arg2: int32
| Memory (bytes, little-endian):
| arg0 (32) | arg1 (16) | padding (16) | arg2 (32) |
64-bit word contents:
Word0: <padding (16)> <arg1 (16)> <arg0 (32)>
Word1: <padding (32)> <arg2 (32)>
|
Example 3: Scalar Packing with Fill
Arguments in order: arg0: int32, arg1: int16, arg2: int8, arg3: int32
| Memory (bytes, little-endian):
| arg0 (32) | arg1 (16) | arg2 (8) | padding (8) | arg3 (32) |
64-bit word contents:
Word0: <padding (8)> <arg2 (8)> <arg1 (16)> <arg0 (32)>
Word1: <padding (32)> <arg3 (32)>
|
Example 4: Dynamic Tensor Argument (One Dynamic Dim)
| Offset | Size | Content
-------|------|--------
0x00 | 8 | Base address of tensor
0x08 | 8 | Offset
0x10 | 4 | stride0 (defaults to 1)
0x14 | 4 | dim0 (dynamic)
-------|------|--------
Total: 24 bytes payload
|
Example 5: Tuple Flattening
Tuple: tuple<scalar, tuple<scalar, tensor>, scalar>
Flattened order:
| scalar, scalar, tensor, scalar
|
Each element is then laid out using Rule 2 (tensor) and Rule 3 (scalar), in that order.
Using the InstructionTable API
The InstructionTable class provides a C++ API for constructing instruction tables:
Creating Arguments
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45 | // Set shared memory regions (required before setupInstructionTable)
instructionTable.setMemoryRegions(
standardRegionBaseAddress,
standardRegionSize,
cacheRegionBaseAddress,
cacheRegionSize,
reservedRegionBaseAddress,
reservedRegionSize);
// Create a scalar argument
instructionTable.createAndPushArgument<ScalarType>(value);
// Create a top-only scalar argument
instructionTable.createAndPushArgument<ScalarType>(
InstructionTable::ArgumentTarget::Top, value);
// Create a tensor argument
std::vector<std::uint32_t> dynamicDims = {dim0, dim1, ...};
instructionTable.createAndPushArgument<TensorType>(basePtr, dynamicDims);
// Create a tensor argument with explicit offset and strides
std::uint64_t offset = 0;
std::vector<std::uint32_t> strides = {stride0, stride1, ...};
instructionTable.createAndPushArgument<TensorType>(
basePtr, offset, strides, dynamicDims);
// Create a core-only tensor argument
instructionTable.createAndPushArgument<TensorType>(
InstructionTable::ArgumentTarget::Core, basePtr, dynamicDims);
// Create a tuple argument
std::vector<Argument *> elements = {...};
instructionTable.createAndPushArgument<TupleType>(elements);
// Create an instruction-table tensor (ArrayType) argument.
// The stack fields (heap address, offset, strides, dynamic dims) follow the
// same layout as TensorType, but the raw element bytes are appended to the
// heap region of the instruction table. heapAddr is computed automatically
// by setupInstructionTable(); offset is always zero for ArrayType.
std::vector<std::uint32_t> arrayStrides = {stride0, stride1, ...};
std::vector<std::uint32_t> arrayDynamicDims = {dim0, dim1, ...};
std::vector<std::uint8_t> rawData = {...}; // raw element bytes
std::uint64_t allocSizeByte = rawData.size();
instructionTable.createAndPushArgument<ArrayType>(
arrayStrides, arrayDynamicDims, allocSizeByte, rawData);
|
Setting Up the Table
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 | // Allocate memory for top and core instruction tables
void *topTable = malloc(topTableSize);
void *coreTable = malloc(coreTableSize);
// setupInstructionTable returns nullptr if memory regions were not set.
// Emit top instruction table
void *setupTopTable = instructionTable.setupInstructionTable(
topTable, topTableSize, InstructionTable::TableTarget::Top);
// Emit core instruction table
std::uint64_t coreId = 3; // raw core id
void *setupCoreTable = instructionTable.setupInstructionTable(
coreTable, coreTableSize, InstructionTable::TableTarget::Core, coreId);
// Use the table for kernel dispatch
// ...
// Clean up
free(topTable);
free(coreTable);
|
Best Practices
- Pre-calculate table size: Determine the maximum instruction table size at compile time based on parameter types.
- Respect alignment: Always follow alignment rules to ensure correct behavior and optimal performance.
- Minimize dynamic dimensions: Use static shapes when possible to reduce instruction table size and improve cache locality.
- Pack small scalars: Group small scalar arguments together to benefit from automatic packing into 64-bit slots.
- Validate at compile time: The compiler should verify instruction table layout constraints during kernel compilation.
Implementation Notes
- The instruction table is encoded in little-endian order
- All pointers are 64-bit regardless of the host architecture
- Dynamic dimension counts must be known at compile time (though their values are runtime)
- The maximum instruction table size can be computed statically from function signatures
- Padding bytes are undefined and should not be relied upon
Memory Layout Examples
Example 1: Simple Scalar Arguments
Legato Kernel:
| @legato.compile(backend=prepare_bertha)
def add_scalars(a: legato.types.integer("int64"),
b: legato.types.integer("int32")):
return a + b
|
C++ Usage Example:
| InstructionTable instructionTable;
instructionTable.createAndPushArgument<ScalarType>(
static_cast<std::int64_t>(100)); // 8 bytes
instructionTable.createAndPushArgument<ScalarType>(
static_cast<std::int32_t>(200)); // 4 bytes
|
Instruction Table Layout:
| Offset | Size | Content
-------|------|--------
0x00 | 8 | arg0 (int64) value (100)
0x08 | 4 | arg1 (int32) value (200)
-------|------|--------
Total: 16 bytes
|
| Offset | Byte 0 | Byte 1 | Byte 2 | Byte 3 | Byte 4 | Byte 5 | Byte 6 | Byte 7
-------|--------|--------|--------|--------|--------|--------|--------|--------
0x00 | 0x64 | 0x00 | 0x00 | 0x00 | 0xC8 | 0x00 | 0x00 | 0x00
| arg0(L)| arg0(L)| arg0(H)| arg1(L)| arg1(L)| arg1(H)|
| int64 | int64 | int64 | int64 | | |
| 100 | 0 | 0 | 0 | | |
-------|--------|--------|--------|--------|--------|--------|--------|--------|
Total: 8 bytes (4 bytes data, 4 bytes padding)
Note: Arguments fill from the START of the 64-bit, where internal byte ordering follows little-endian.
|
Example 2: Static Tensor Argument
Legato Kernel:
| @legato.compile(backend=prepare_bertha)
def copy_tensor(x: legato.types.tensor(legato.types.float("float32"), [10, 10])):
device = legato.get_context().get_device()
with device.get_all_cores():
y = legato.tensor.copy(x, None)
yield y
|
Instruction Table Layout:
| Offset | Size | Content
-------|------|--------
0x00 | 8 | Base address of tensor x (pointer)
-------|------|--------
Total: 8 bytes
Note: Static dimensions [10, 10] are not stored in the instruction
table as they are known at compile time.
|
Example 3: Dynamic Tensor Argument
Legato Kernel:
| @legato.compile(backend=prepare_bertha)
def process_dynamic_tensor(
x: legato.types.tensor(legato.types.float("float32"), [10, -1])
):
device = legato.get_context().get_device()
with device.get_top():
y = legato.tensor.request_load(
x, [device.get_core(0)],
(arith.cast(10, legato.types.index()), )
)
yield y
|
Instruction Table Layout (for runtime shape [10 (static), 15 (dynamic)]):
| Offset | Size | Content
-------|------|--------
0x00 | 8 | Base address of tensor x (pointer)
0x08 | 4 | dim1 = 15 (dynamic dim data)
0x0C | 4 | padding (to align to 64-bit boundary)
-------|------|--------
Total: 16 bytes
Note: The dynamic dimension is written immediately after the base address.
If the next argument requires 64-bit alignment, that next argument adds the padding.
|
Example 8: Complete Real-World Example (from InstTableExample.cpp)
Equivalent Legato Kernel:
This C++ example corresponds to a Legato kernel with the following signature:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40 | import legato
import torch
@legato.compile(backend=prepare_bertha)
def example_kernel(
arg0: legato.types.integer("int32"), # Scalar: int32
arg1: legato.types.integer("int32"), # Scalar: int32
arg2: legato.types.integer("int32"), # Scalar: int32
arg3: legato.types.integer("int16"), # Scalar: int16
arg4: legato.types.integer("int8"), # Scalar: int8
tensor_arg: legato.types.tensor( # Tensor with 10 dynamic dimensions
legato.types.float("float32"),
[(-1, 100), (-1, 100), (-1, 100), (-1, 100), (-1, 100),
(-1, 100), (-1, 100), (-1, 100), (-1, 100), (-1, 100)]
),
arg5: legato.types.integer("int16"), # Scalar: int16
arg6: legato.types.integer("int32"), # Scalar: int32
tuple_arg: legato.types.tuple([ # Tuple with two int8 values
legato.types.integer("int8"),
legato.types.integer("int8")
])
):
"""Example kernel demonstrating instruction table layout."""
device = legato.get_context().get_device()
with device.get_all_cores():
# Kernel body (implementation details)
pass
# Calling the kernel with actual values:
example_kernel(
1, # arg0: int32 = 1
2, # arg1: int32 = 2
3, # arg2: int32 = 3
4, # arg3: int16 = 4
5, # arg4: int8 = 5
torch.randn(1, 2, 3, 4, 5, 6, 7, 8, 9, 10), # 10-dimensional tensor
7, # arg5: int16 = 7
8, # arg6: int32 = 8
(-1, -2) # tuple_arg: (int8, int8)
)
|
Argument Mapping to Instruction Table:
| Argument |
Type |
Value |
Instruction Table Position |
Bytes |
Notes |
arg0 |
int32 |
1 |
Slot 0, bytes 0-3 |
4 |
First scalar, placed at LSB |
arg1 |
int32 |
2 |
Slot 0, bytes 4-7 |
4 |
Packed after arg0 |
arg2 |
int32 |
3 |
Slot 1, bytes 0-3 |
4 |
Packed before smaller scalars |
arg3 |
int16 |
4 |
Slot 1, bytes 4-5 |
2 |
Packed after arg2 |
arg4 |
int8 |
5 |
Slot 1, byte 6 |
1 |
Packed after arg3 |
tensor_arg |
tensor<?x?x?x?x?x?x?x?x?x?xf32> |
(rank 10, 10 dims) |
Bytes 0x10-0x6F |
96 |
8-byte ptr + 8-byte offset + 40 bytes (10×4) for strides + 40 bytes (10×4) for dims |
arg5 |
int16 |
7 |
Slot at 0x70, bytes 0-1 |
2 |
Packed before arg6 |
arg6 |
int32 |
8 |
Slot at 0x70, bytes 2-5 |
4 |
Packed after arg5 |
tuple_arg |
tuple |
(-1, -2) |
Bytes 0x76-0x77 |
2 |
Tuple elements stored in-order |
Detailed Argument Breakdown:
-
Scalar Arguments (arg0-arg2): Three consecutive int32 values that span two 64-bit slots
arg0=1 and arg1=2 share the first slot (0x00-0x07)
- Packed in argument order:
[arg0][arg1]
-
Mixed Scalar Arguments (arg2-arg4): Different-sized scalars packed efficiently
arg2=3 (int32, 4 bytes)
arg3=4 (int16, 2 bytes)
arg4=5 (int8, 1 byte)
- Plus 1 byte padding at 0x0F to align the following tensor
- Packed in order:
[arg2][arg3][arg4][pad]
-
Tensor Argument: Multi-dimensional tensor with dynamic dimensions
- Base address:
0x0123456789ABCD (8 bytes at offset 0x10)
- Offset:
0 (8 bytes at offset 0x18)
- 10 strides, each 4 bytes, stored at offsets 0x20-0x47
- Strides default to
1 when not provided explicitly
- 10 dynamic dimensions, each 4 bytes
- Dimensions stored as pairs in 64-bit slots: (dim0, dim1), (dim2, dim3), ..., (dim8, dim9)
- In each pair, the earlier value is the low 32 bits and the next value is the high 32 bits of the 64-bit slot
- Total: 96 bytes (8 for pointer + 8 for offset + 40 for strides + 40 for dimensions)
-
Post-Tensor Scalars (arg5-arg6): More scalars after the tensor
arg5=7 (int16, 2 bytes)
arg6=8 (int32, 4 bytes)
- Packed in order:
[arg5][arg6]
-
Tuple Argument: Compound type containing two int8 values
tuple_arg[0] = -1 (int8)
tuple_arg[1] = -2 (int8)
- Stored in-order, immediately after
arg6
C++ Usage Example:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40 | InstructionTable instructionTable;
// Three 32-bit integers
instructionTable.createAndPushArgument<ScalarType>(1); // arg0: int32
instructionTable.createAndPushArgument<ScalarType>(2); // arg1: int32
instructionTable.createAndPushArgument<ScalarType>(3); // arg2: int32
// Smaller scalars that will be packed
instructionTable.createAndPushArgument<ScalarType>(
static_cast<std::int16_t>(4) // arg3: int16
);
instructionTable.createAndPushArgument<ScalarType>(
static_cast<std::int8_t>(5) // arg4: int8
);
// Tensor with dynamic dimensions
std::vector<std::uint32_t> dynamicDims = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
instructionTable.createAndPushArgument<TensorType>(
reinterpret_cast<void *>(0x0123456789ABCD), // Example address
dynamicDims
);
// More scalars after tensor
instructionTable.createAndPushArgument<ScalarType>(
static_cast<std::int16_t>(7) // arg5: int16
);
instructionTable.createAndPushArgument<ScalarType>(
static_cast<std::int32_t>(8) // arg6: int32
);
// Tuple at the end
std::vector<Argument *> tupleArgs = {
instructionTable.createArgument<ScalarType>(
static_cast<std::int8_t>(-1) // tuple.elem0
),
instructionTable.createArgument<ScalarType>(
static_cast<std::int8_t>(-2) // tuple.elem1
),
};
instructionTable.createAndPushArgument<TupleType>(tupleArgs);
|
Actual Memory Layout
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52 | Offset | Bytes (little-endian) | Interpretation
-------|--------------------------|------------------------------------------
| 64-bit slot 0: Two int32 values (in order)
0x00 | 0x01 0x00 0x00 0x00 | arg0: int32 = 1
0x04 | 0x02 0x00 0x00 0x00 | arg1: int32 = 2
-------|--------------------------|------------------------------------------
| 64-bit slot 1: One int32, one int16, one int8 (in order)
0x08 | 0x03 0x00 0x00 0x00 | arg2: int32 = 3
0x0C | 0x04 0x00 | arg3: int16 = 4
0x0E | 0x05 | arg4: int8 = 5
0x0F | 0x00 | padding for tensor alignment
-------|--------------------------|------------------------------------------
| Tensor base address (8 bytes, little-endian)
0x10 | 0xcd 0xab 0x89 0x67 | Low 4 bytes of address
0x14 | 0x45 0x23 0x01 0x00 | High 4 bytes: 0x00 0x01 0x23 0x45
| | Full address: 0x0000_0123_4567_89AB_CD
-------|--------------------------|------------------------------------------
| Tensor offset (8 bytes, little-endian)
0x18 | 0x00 0x00 0x00 0x00 | Low 4 bytes of offset
0x1C | 0x00 0x00 0x00 0x00 | High 4 bytes of offset
-------|--------------------------|------------------------------------------
| Tensor strides (rank order, defaulted to 1)
0x20 | 0x01 0x00 0x00 0x00 | stride0 = 1
0x24 | 0x01 0x00 0x00 0x00 | stride1 = 1
0x28 | 0x01 0x00 0x00 0x00 | stride2 = 1
0x2C | 0x01 0x00 0x00 0x00 | stride3 = 1
0x30 | 0x01 0x00 0x00 0x00 | stride4 = 1
0x34 | 0x01 0x00 0x00 0x00 | stride5 = 1
0x38 | 0x01 0x00 0x00 0x00 | stride6 = 1
0x3C | 0x01 0x00 0x00 0x00 | stride7 = 1
0x40 | 0x01 0x00 0x00 0x00 | stride8 = 1
0x44 | 0x01 0x00 0x00 0x00 | stride9 = 1
-------|--------------------------|------------------------------------------
| Tensor dimensions (logical order: dim0, dim1, dim2, dim3, ...)
0x48 | 0x01 0x00 0x00 0x00 | dim0 = 1 (stored in low 32 bits of pair0)
0x4C | 0x02 0x00 0x00 0x00 | dim1 = 2 (stored in high 32 bits of pair0)
0x50 | 0x03 0x00 0x00 0x00 | dim2 = 3 (stored in low 32 bits of pair1)
0x54 | 0x04 0x00 0x00 0x00 | dim3 = 4 (stored in high 32 bits of pair1)
0x58 | 0x05 0x00 0x00 0x00 | dim4 = 5 (stored in low 32 bits of pair2)
0x5C | 0x06 0x00 0x00 0x00 | dim5 = 6 (stored in high 32 bits of pair2)
0x60 | 0x07 0x00 0x00 0x00 | dim6 = 7 (stored in low 32 bits of pair3)
0x64 | 0x08 0x00 0x00 0x00 | dim7 = 8 (stored in high 32 bits of pair3)
0x68 | 0x09 0x00 0x00 0x00 | dim8 = 9 (stored in low 32 bits of pair4)
0x6C | 0x0a 0x00 0x00 0x00 | dim9 = 10 (stored in high 32 bits of pair4)
-------|--------------------------|------------------------------------------
| 64-bit slot: int16, int32, tuple<int8,int8> (in order)
0x70 | 0x07 0x00 | arg5: int16 = 7
0x72 | 0x08 0x00 0x00 0x00 | arg6: int32 = 8
0x76 | 0xff | tuple.elem0: int8 = -1 (0xff = -1 in two's complement)
0x77 | 0xfe | tuple.elem1: int8 = -2 (0xfe = -2 in two's complement)
-------|--------------------------|------------------------------------------
Total: 120 bytes
|
Implementation Notes
- The instruction table is encoded in little-endian order
- All pointers are 64-bit regardless of the host architecture
- Dynamic dimension counts must be known at compile time (though their values are runtime)
- The maximum instruction table size can be computed statically from function signatures
- Padding bytes are undefined and should not be relied upon