Skip to content

Custom Operators API

The custom operators that torch_ha registers in the torch.ops.lpu namespace. The vmm_* operators for the paged KV cache are in the VMM API. Signatures use the registered operator schema notation.


Layout and Memory

Function Signature Notes
empty_with_kind (SymInt[] size, str kind, SymInt[]? stride=None, *, ScalarType dtype, Device device, int? alignment=None) -> Tensor Allocates a tensor with the given device layout (kind). If alignment is given (in bytes, a power of two no less than 4096), aligns the storage start address to a multiple of it
set_storage_meta_ (Tensor t, SymInt[] storage_sizes, SymInt[] storage_strides, SymInt? kind=None) -> () Records storage layout metadata. Takes chip information from the current device. Accepts lpu tensors only
get_storage_meta (Tensor t) -> (SymInt[], SymInt[], str, int?) Returns (storage_sizes, storage_strides, chip, kind), or ([], [], "", kind) if no metadata is recorded

set_storage_meta_ and get_storage_meta operate on layout metadata that torch.compile does not trace. They are not captured in the graph and cause a graph break.

set_storage_meta_ is for backend development; use it with care

It overwrites only the layout information recorded by the backend and leaves the data unchanged. If the values do not match the actual placement of the storage, subsequent kernels read the buffer with incorrect strides.

Tensor Kinds

The kind strings accepted by empty_with_kind, matched case-insensitively. An unlisted name raises an error that lists the supported values. The number in parentheses is the HaTensorKind value, which set_storage_meta_ and get_storage_meta use as kind.

kind hart.HaTensorKind Placement Applied automatically
dram_word_interleaved DRAM_WORD_INTERLEAVED (5) Interleaves data across channels in DRAM-word units Default. All other tensors and vmm_create_kv_pool
mpu_tensor MPU_TENSOR (6) Matrix layout for MPU operations —
mpu_tensor_transposed MPU_TENSOR_TRANSPOSED (7) Transposed form of mpu_tensor. Read transposed by the hardware; used for the right-hand operand of matmul nn.Linear weights
single_pass_interleaved SINGLE_PASS_INTERLEAVED (8) Places 1/N of the tensor on each channel as one contiguous chunk nn.Embedding weights
row_interleaved ROW_INTERLEAVED (3) Interleaves data across channels one row at a time —
max_rows_interleaved MAX_ROWS_INTERLEAVED (4) Groups several rows into one chunk and interleaves them across channels —

"Applied automatically" lists the tensor kind assigned when a tensor is moved to or created on lpu without an explicit kind. See Device Semantics §2.2.


Quantized Weights

These operators call the weight converter. Quantized weights are handled as rank-1 int8/uint8 tensors (blobs). quant_weight_stamp_ records the quantization scheme, logical shape, and logical dtype of a blob.

Function Signature Notes
quant_weight_pack (Tensor weight, *, Tensor? scale=None, Tensor? weight_zero_point=None, Tensor? bias=None, Tensor? mx_block_scale=None, Tensor? output_zero_point=None, int[] weight_shape, ScalarType weight_dtype, ScalarType output_dtype, str quantization_type="output_channel", int input_channel_group_size=0, bool channel_wise_packing=True, str? architecture=None) -> Tensor Converts the weights on the host and returns a CPU uint8 blob
quant_weight_converted_size (int[] weight_shape, ScalarType weight_dtype, *, bool has_bias, ScalarType output_dtype, str quantization_type="output_channel", int input_channel_group_size=0, str? architecture=None) -> int Returns the size in bytes of the quant_weight_pack output. Use it to size an allocation in advance
quant_weight_stamp_ (Tensor(a!) self, str quantization_type, int[] logical_shape, ScalarType logical_dtype) -> Tensor(a!) Records the quantization policy, logical shape, and logical dtype on the blob. Does not change data placement
quant_weight_meta (Tensor self) -> (str, int[], str) Returns the recorded (quantization_type, logical_shape, logical_dtype). logical_dtype is a ScalarType name such as "UInt4". Raises an error if nothing is recorded
Item Value
quantization_type (pack, converted_size) "output_channel", "none"
quantization_type (stamp_) "none", "output_channel", "input_channel_wise", "token_wise", "mxfp"
architecture Omit it when a device is present; the current device is used, and a different value raises an error
Blob placement Allocated as dram_word_interleaved with channel_wise_packing=False, or as single_pass_interleaved with True
logical_shape One of positive [K, N], [E, K, N], [r, s, c, m]
logical_dtype uint4, int4, uint8, int8, float8_*, float4_e2m1fn_x2

quant_weight_pack treats scale and weight_zero_point as one value per output channel, so it rejects the input_channel_wise, token_wise, and mxfp policies. To use these policies, call the weight converter directly.


Debugging

Not a stable API

This is a diagnostic operator. Its signature and return values may change between releases, so use it only to check the status or print the reason, and do not base the behavior of production code on it.

Function Signature Notes
eager_status () -> (bool ready, str reason, str[] kernels, str legato, bool built) ready says whether the eager kernels are usable, and reason says why not (empty when they are). kernels lists the kernels that can run, legato is the legato version the kernels were built with, and built says whether the build includes eager kernels. Even when ready is True, a dtype or shape the kernels do not take is declined

For the eager kernel path, see Operators §3.1.


Collective Fast Path

These operators capture collectives inside a torch.compile graph without a graph break. The lccl_* operators that take tensors have Meta kernels, so they can be traced with FakeTensor. They use the same communicator as the c10d "lccl" backend; see Distributed §4.1.

Function Signature Notes
lccl_comm_handle (str group_name) -> int Returns the communicator handle of the process group. Raises an error if the group does not use the lccl backend
lccl_all_reduce (Tensor input, str reduce_op, int comm_handle) -> Tensor out-of-place
lccl_all_reduce_ (Tensor(a!) input, str reduce_op, int comm_handle) -> Tensor(a!) in-place
lccl_broadcast_ (Tensor(a!) tensor, int root, int comm_handle) -> Tensor(a!) in-place
lccl_all_gather (Tensor input, int world_size, int comm_handle) -> Tensor Output dim 0 is world_size times that of the input
lccl_reduce_scatter (Tensor input, str reduce_op, int world_size, int comm_handle) -> Tensor Output dim 0 is 1/world_size of that of the input
lccl_reduce (Tensor input, str reduce_op, int root, int comm_handle) -> Tensor Result is valid only on root
lccl_gather (Tensor input, int root, int world_size, int comm_handle) -> Tensor Gathers on root, concatenated along dim 0
lccl_scatter (Tensor input, int root, int world_size, int comm_handle) -> Tensor Returns the chunk for this rank
lccl_all_to_all (Tensor input, int comm_handle) -> Tensor Preserves shape and dtype
lccl_send (Tensor input, int peer, int comm_handle) -> () Point-to-point send
lccl_recv (Tensor(a!) output, int peer, int comm_handle) -> Tensor(a!) Fills output
lccl_group_start () -> () Starts collecting subsequent point-to-point operations
lccl_group_end () -> () Runs the collected operations together

reduce_op accepts "sum", "prod", "max", "min", and "avg". For the supported combinations of collectives and reductions, see the LCCL Support Matrix.


Examples

eager_status

Prints the reason when the eager kernel path cannot be used. In that case import torch also logs the warning torch_ha eager kernels are unusable; eager ops will raise: <reason>.

Example
1
2
3
4
5
import torch

ready, reason = torch.ops.lpu.eager_status()[:2]
if not ready:
    print(reason)  # e.g. torch_ha and hart are from different releases; install matching versions (err=…)

empty_with_kind

Allocates a tensor with the given kind, then fills it with copy_.

Example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import torch

w = torch.ops.lpu.empty_with_kind(
    [1024, 1024], "mpu_tensor_transposed", dtype=torch.bfloat16, device="lpu:0"
)
w.copy_(torch.randn(1024, 1024, dtype=torch.bfloat16))

aligned = torch.ops.lpu.empty_with_kind(
    [8192], "dram_word_interleaved", dtype=torch.bfloat16, device="lpu:0",
    alignment=4096,
)

get_storage_meta

Reads the recorded layout metadata.

Example
1
2
3
4
5
import torch

sizes, strides, chip, kind = torch.ops.lpu.get_storage_meta(t)
if not chip:
    print("no recorded metadata")

lccl_all_gather

Concatenates the tensors from all ranks along dim 0.

Example
1
2
3
4
5
import torch
import torch.distributed as dist

handle = torch.ops.lpu.lccl_comm_handle(dist.group.WORLD.group_name)
out = torch.ops.lpu.lccl_all_gather(x, dist.get_world_size(), handle)

lccl_all_reduce

Returns the result as a new tensor.

Example
1
2
3
4
5
import torch
import torch.distributed as dist

handle = torch.ops.lpu.lccl_comm_handle(dist.group.WORLD.group_name)
out = torch.ops.lpu.lccl_all_reduce(x, "sum", handle)

lccl_all_reduce_

Overwrites the input tensor with the result.

Example
1
2
3
import torch

torch.ops.lpu.lccl_all_reduce_(x, "sum", handle)

lccl_all_to_all

Exchanges chunks between ranks, preserving shape and dtype.

Example
1
2
3
import torch

out = torch.ops.lpu.lccl_all_to_all(x, handle)

lccl_broadcast_

Overwrites the tensor on every rank with the value from root.

Example
1
2
3
import torch

torch.ops.lpu.lccl_broadcast_(x, 0, handle)

lccl_comm_handle

Obtains a handle from the process group name for use with the lccl_* operators.

Example
1
2
3
4
5
6
import torch
import torch.distributed as dist

dist.init_process_group("lccl")
group_name = dist.distributed_c10d._get_default_group().group_name
handle = torch.ops.lpu.lccl_comm_handle(group_name)

lccl_gather

Gathers the tensors from all ranks on root along dim 0.

Example
1
2
3
4
import torch
import torch.distributed as dist

out = torch.ops.lpu.lccl_gather(x, 0, dist.get_world_size(), handle)

lccl_group_end

Runs the point-to-point operations collected since lccl_group_start.

Example
1
2
3
4
5
6
import torch

torch.ops.lpu.lccl_group_start()
torch.ops.lpu.lccl_send(x, peer, handle)
torch.ops.lpu.lccl_recv(y, peer, handle)
torch.ops.lpu.lccl_group_end()

lccl_group_start

Wrap a bidirectional exchange in a group to avoid deadlock.

Example
1
2
3
4
5
6
import torch

torch.ops.lpu.lccl_group_start()
torch.ops.lpu.lccl_send(x, peer, handle)
torch.ops.lpu.lccl_recv(y, peer, handle)
torch.ops.lpu.lccl_group_end()

lccl_recv

Receives into a preallocated output.

Example
1
2
3
4
import torch

y = torch.empty(4, 8, dtype=torch.bfloat16).to("lpu")
torch.ops.lpu.lccl_recv(y, src_rank, handle)

lccl_reduce

The result is undefined on ranks other than root.

Example
1
2
3
4
5
6
import torch
import torch.distributed as dist

out = torch.ops.lpu.lccl_reduce(x, "sum", 0, handle)
if dist.get_rank() == 0:
    print(out.cpu())

lccl_reduce_scatter

Returns only the chunk of the reduction result for this rank.

Example
1
2
3
4
import torch
import torch.distributed as dist

out = torch.ops.lpu.lccl_reduce_scatter(x, "sum", dist.get_world_size(), handle)

lccl_scatter

Splits the buffer on root and returns this rank's chunk.

Example
1
2
3
4
import torch
import torch.distributed as dist

out = torch.ops.lpu.lccl_scatter(x, 0, dist.get_world_size(), handle)

lccl_send

Sends the whole tensor to peer.

Example
1
2
3
import torch

torch.ops.lpu.lccl_send(x, dst_rank, handle)

quant_weight_converted_size

Computes the required size in bytes before the blob is allocated.

Example
1
2
3
4
5
import torch

nbytes = torch.ops.lpu.quant_weight_converted_size(
    [K, N], torch.uint4, has_bias=True, output_dtype=torch.bfloat16
)

quant_weight_meta

Reads the recorded quantization information.

Example
1
2
3
4
import torch

policy, shape, dtype = torch.ops.lpu.quant_weight_meta(w)
# ("output_channel", [K, N], "UInt4")

quant_weight_pack

Converts quantized weights into a blob on the host.

Example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import torch

blob = torch.ops.lpu.quant_weight_pack(
    packed_uint4,                 # weights with two uint4 values packed into each byte
    scale=scale,
    weight_zero_point=zero_point,
    bias=bias,
    weight_shape=[K, N],
    weight_dtype=torch.uint4,
    output_dtype=torch.bfloat16,
    channel_wise_packing=False,
)

quant_weight_stamp_

Copies the blob to the device with the appropriate kind, then records the quantization information.

Example
1
2
3
4
5
6
7
import torch

w = torch.ops.lpu.empty_with_kind(
    [blob.numel()], "dram_word_interleaved", dtype=torch.int8, device="lpu:0"
)
w.copy_(blob.view(torch.int8))
torch.ops.lpu.quant_weight_stamp_(w, "output_channel", [K, N], torch.uint4)

set_storage_meta_

Records storage layout metadata directly. Intended for backend development.

Example
1
2
3
4
5
6
import hart
import torch

torch.ops.lpu.set_storage_meta_(
    t, [8, 512], [512, 1], kind=hart.HaTensorKind.DRAM_WORD_INTERLEAVED.value
)

See Also