Custom Operators API¶
The custom operators that torch_ha registers in the torch.ops.lpu namespace. The vmm_* operators for the paged
KV cache are in the VMM API. Signatures use the registered operator schema notation.
Layout and Memory¶
| Function | Signature | Notes |
|---|---|---|
empty_with_kind |
(SymInt[] size, str kind, SymInt[]? stride=None, *, ScalarType dtype, Device device, int? alignment=None) -> Tensor |
Allocates a tensor with the given device layout (kind). If alignment is given (in bytes, a power of two no less than 4096), aligns the storage start address to a multiple of it |
set_storage_meta_ |
(Tensor t, SymInt[] storage_sizes, SymInt[] storage_strides, SymInt? kind=None) -> () |
Records storage layout metadata. Takes chip information from the current device. Accepts lpu tensors only |
get_storage_meta |
(Tensor t) -> (SymInt[], SymInt[], str, int?) |
Returns (storage_sizes, storage_strides, chip, kind), or ([], [], "", kind) if no metadata is recorded |
set_storage_meta_ and get_storage_meta operate on layout metadata that torch.compile does not trace. They are
not captured in the graph and cause a graph break.
set_storage_meta_ is for backend development; use it with care
It overwrites only the layout information recorded by the backend and leaves the data unchanged. If the values do not match the actual placement of the storage, subsequent kernels read the buffer with incorrect strides.
Tensor Kinds¶
The kind strings accepted by empty_with_kind, matched case-insensitively. An unlisted name raises an error that
lists the supported values. The number in parentheses is the HaTensorKind value, which set_storage_meta_ and
get_storage_meta use as kind.
kind |
hart.HaTensorKind |
Placement | Applied automatically |
|---|---|---|---|
dram_word_interleaved |
DRAM_WORD_INTERLEAVED (5) |
Interleaves data across channels in DRAM-word units | Default. All other tensors and vmm_create_kv_pool |
mpu_tensor |
MPU_TENSOR (6) |
Matrix layout for MPU operations | — |
mpu_tensor_transposed |
MPU_TENSOR_TRANSPOSED (7) |
Transposed form of mpu_tensor. Read transposed by the hardware; used for the right-hand operand of matmul |
nn.Linear weights |
single_pass_interleaved |
SINGLE_PASS_INTERLEAVED (8) |
Places 1/N of the tensor on each channel as one contiguous chunk |
nn.Embedding weights |
row_interleaved |
ROW_INTERLEAVED (3) |
Interleaves data across channels one row at a time | — |
max_rows_interleaved |
MAX_ROWS_INTERLEAVED (4) |
Groups several rows into one chunk and interleaves them across channels | — |
"Applied automatically" lists the tensor kind assigned when a tensor is moved to or created on lpu without an explicit
kind.
See Device Semantics §2.2.
Quantized Weights¶
These operators call the weight converter. Quantized weights are handled as rank-1 int8/uint8 tensors (blobs).
quant_weight_stamp_ records the quantization scheme, logical shape, and logical dtype of a blob.
| Function | Signature | Notes |
|---|---|---|
quant_weight_pack |
(Tensor weight, *, Tensor? scale=None, Tensor? weight_zero_point=None, Tensor? bias=None, Tensor? mx_block_scale=None, Tensor? output_zero_point=None, int[] weight_shape, ScalarType weight_dtype, ScalarType output_dtype, str quantization_type="output_channel", int input_channel_group_size=0, bool channel_wise_packing=True, str? architecture=None) -> Tensor |
Converts the weights on the host and returns a CPU uint8 blob |
quant_weight_converted_size |
(int[] weight_shape, ScalarType weight_dtype, *, bool has_bias, ScalarType output_dtype, str quantization_type="output_channel", int input_channel_group_size=0, str? architecture=None) -> int |
Returns the size in bytes of the quant_weight_pack output. Use it to size an allocation in advance |
quant_weight_stamp_ |
(Tensor(a!) self, str quantization_type, int[] logical_shape, ScalarType logical_dtype) -> Tensor(a!) |
Records the quantization policy, logical shape, and logical dtype on the blob. Does not change data placement |
quant_weight_meta |
(Tensor self) -> (str, int[], str) |
Returns the recorded (quantization_type, logical_shape, logical_dtype). logical_dtype is a ScalarType name such as "UInt4". Raises an error if nothing is recorded |
| Item | Value |
|---|---|
quantization_type (pack, converted_size) |
"output_channel", "none" |
quantization_type (stamp_) |
"none", "output_channel", "input_channel_wise", "token_wise", "mxfp" |
architecture |
Omit it when a device is present; the current device is used, and a different value raises an error |
| Blob placement | Allocated as dram_word_interleaved with channel_wise_packing=False, or as single_pass_interleaved with True |
logical_shape |
One of positive [K, N], [E, K, N], [r, s, c, m] |
logical_dtype |
uint4, int4, uint8, int8, float8_*, float4_e2m1fn_x2 |
quant_weight_pack treats scale and weight_zero_point as one value per output channel, so it rejects the
input_channel_wise, token_wise, and mxfp policies. To use these policies, call the weight converter directly.
Debugging¶
Not a stable API
This is a diagnostic operator. Its signature and return values may change between releases, so use it only to check the status or print the reason, and do not base the behavior of production code on it.
| Function | Signature | Notes |
|---|---|---|
eager_status |
() -> (bool ready, str reason, str[] kernels, str legato, bool built) |
ready says whether the eager kernels are usable, and reason says why not (empty when they are). kernels lists the kernels that can run, legato is the legato version the kernels were built with, and built says whether the build includes eager kernels. Even when ready is True, a dtype or shape the kernels do not take is declined |
For the eager kernel path, see Operators §3.1.
Collective Fast Path¶
These operators capture collectives inside a torch.compile graph without a graph break. The lccl_* operators that
take tensors have Meta kernels, so they can be traced with FakeTensor. They use the same communicator as the c10d "lccl" backend; see
Distributed §4.1.
| Function | Signature | Notes |
|---|---|---|
lccl_comm_handle |
(str group_name) -> int |
Returns the communicator handle of the process group. Raises an error if the group does not use the lccl backend |
lccl_all_reduce |
(Tensor input, str reduce_op, int comm_handle) -> Tensor |
out-of-place |
lccl_all_reduce_ |
(Tensor(a!) input, str reduce_op, int comm_handle) -> Tensor(a!) |
in-place |
lccl_broadcast_ |
(Tensor(a!) tensor, int root, int comm_handle) -> Tensor(a!) |
in-place |
lccl_all_gather |
(Tensor input, int world_size, int comm_handle) -> Tensor |
Output dim 0 is world_size times that of the input |
lccl_reduce_scatter |
(Tensor input, str reduce_op, int world_size, int comm_handle) -> Tensor |
Output dim 0 is 1/world_size of that of the input |
lccl_reduce |
(Tensor input, str reduce_op, int root, int comm_handle) -> Tensor |
Result is valid only on root |
lccl_gather |
(Tensor input, int root, int world_size, int comm_handle) -> Tensor |
Gathers on root, concatenated along dim 0 |
lccl_scatter |
(Tensor input, int root, int world_size, int comm_handle) -> Tensor |
Returns the chunk for this rank |
lccl_all_to_all |
(Tensor input, int comm_handle) -> Tensor |
Preserves shape and dtype |
lccl_send |
(Tensor input, int peer, int comm_handle) -> () |
Point-to-point send |
lccl_recv |
(Tensor(a!) output, int peer, int comm_handle) -> Tensor(a!) |
Fills output |
lccl_group_start |
() -> () |
Starts collecting subsequent point-to-point operations |
lccl_group_end |
() -> () |
Runs the collected operations together |
reduce_op accepts "sum", "prod", "max", "min", and "avg". For the supported combinations of collectives
and reductions, see the
LCCL Support Matrix.
Examples¶
eager_status¶
Prints the reason when the eager kernel path cannot be used. In that case import torch also logs the warning
torch_ha eager kernels are unusable; eager ops will raise: <reason>.
Example
1 2 3 4 5 | |
empty_with_kind¶
Allocates a tensor with the given kind, then fills it with copy_.
Example
1 2 3 4 5 6 7 8 9 10 11 | |
get_storage_meta¶
Reads the recorded layout metadata.
Example
1 2 3 4 5 | |
lccl_all_gather¶
Concatenates the tensors from all ranks along dim 0.
Example
1 2 3 4 5 | |
lccl_all_reduce¶
Returns the result as a new tensor.
Example
1 2 3 4 5 | |
lccl_all_reduce_¶
Overwrites the input tensor with the result.
Example
1 2 3 | |
lccl_all_to_all¶
Exchanges chunks between ranks, preserving shape and dtype.
Example
1 2 3 | |
lccl_broadcast_¶
Overwrites the tensor on every rank with the value from root.
Example
1 2 3 | |
lccl_comm_handle¶
Obtains a handle from the process group name for use with the lccl_* operators.
Example
1 2 3 4 5 6 | |
lccl_gather¶
Gathers the tensors from all ranks on root along dim 0.
Example
1 2 3 4 | |
lccl_group_end¶
Runs the point-to-point operations collected since lccl_group_start.
Example
1 2 3 4 5 6 | |
lccl_group_start¶
Wrap a bidirectional exchange in a group to avoid deadlock.
Example
1 2 3 4 5 6 | |
lccl_recv¶
Receives into a preallocated output.
Example
1 2 3 4 | |
lccl_reduce¶
The result is undefined on ranks other than root.
Example
1 2 3 4 5 6 | |
lccl_reduce_scatter¶
Returns only the chunk of the reduction result for this rank.
Example
1 2 3 4 | |
lccl_scatter¶
Splits the buffer on root and returns this rank's chunk.
Example
1 2 3 4 | |
lccl_send¶
Sends the whole tensor to peer.
Example
1 2 3 | |
quant_weight_converted_size¶
Computes the required size in bytes before the blob is allocated.
Example
1 2 3 4 5 | |
quant_weight_meta¶
Reads the recorded quantization information.
Example
1 2 3 4 | |
quant_weight_pack¶
Converts quantized weights into a blob on the host.
Example
1 2 3 4 5 6 7 8 9 10 11 12 | |
quant_weight_stamp_¶
Copies the blob to the device with the appropriate kind, then records the quantization information.
Example
1 2 3 4 5 6 7 | |
set_storage_meta_¶
Records storage layout metadata directly. Intended for backend development.
Example
1 2 3 4 5 6 | |