Skip to content

8. Weight Conversion

LPU kernels do not read checkpoint weights as is. Weights must be converted to the byte layout defined by the target architecture before they are placed in device memory, and the Weight Converter API performs this conversion. This chapter covers why conversion is needed, the conventions for preparing inputs, and how the converted result is moved to the device.

Contents

Section Objective
8.1 Why Conversion Is Needed Differences between the checkpoint layout and the device layout
8.2 Usage Paths When you need to call the API directly and when you do not
8.3 Conversion Flow From querying the size to placing the result on the device
8.4 Input Tensor Conventions Shape, data layout, metadata tensors
8.5 Quantization Methods and Architectures Required tensors per method and support per architecture
8.6 Memory and Error Handling Host memory usage and failure categories

8.1 Why Conversion Is Needed

LPU matrix kernels read weights grouped by output channel and aligned to DRAM lines. Quantization metadata such as bias, scale, and zero-point must also sit next to each group in a fixed order. The group size, alignment unit, and metadata placement differ by architecture.

The conversion result is an opaque byte buffer that only kernels for that architecture can interpret. Do not read or modify the buffer contents directly, and do not reuse a buffer converted for a different architecture. If you specify the wrong architecture, the size and offsets look correct, but kernels silently produce wrong results.

8.2 Usage Paths

Most users do not call the Weight Converter API directly.

Environment Conversion path
vLLM HyperAccel The checkpoint conversion tool (python -m vllm_hyperaccel.tools.convert_checkpoint) converts and saves quantized weights ahead of time (see Quantization). No direct call is needed.
PyTorch (torch_ha) torch.ops.lpu.quant_weight_pack() takes PyTorch tensors and converts them. If the target architecture is omitted, it converts for the current device.
Direct runtime integration Call C++ HaWeightConvert() or Python hart.weight_converter.convert().

The rest of this chapter covers the third case: integrating directly with the runtime API.

8.3 Conversion Flow

Conversion runs only on the host and does not involve a device. There are no handles or state, so each call is independent.

  1. Prepare input tensors — Wrap the weight and each required metadata tensor in a HaWeightTensor (see Input Tensor Conventions).
  2. Query the size — Get the size in bytes of the conversion result with HaWeightGetConvertedSize().
  3. Convert — Allocate a host buffer of at least that size and call HaWeightConvert(). The converter writes directly into this buffer and makes no intermediate copy.
  4. Place on the device — Copy the converted buffer to device memory. Kernels then take this device buffer as the weight argument.

Weight conversion on the host and placement on the device

Flow of converting on the host and copying to the device (C++ API)

In Python, convert() performs steps 2 and 3 at once and returns the result as a NumPy array.

channel_wise_packing determines whether the conversion result is pre-split per DRAM channel. The layout kind of the device tensor that holds the result must match this value.

channel_wise_packing Device tensor layout
true (Python default) HaTensorKind::SinglePassInterleaved
false HaTensorKind::DramWordInterleaved

For layout kinds, see HaTensorKind in the Types API and Memory Model.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from hart.weight_converter import (
    WeightConverterConfig, convert,
    ArchitectureType, QuantType, DataType, Shape, Tensor, TensorType,
)

# w: [in_features, out_features] row-major uint8 array
weight = Tensor(w.reshape(-1), Shape(1, 1, in_features, out_features), DataType.UINT8, TensorType.Weight)
per_out = Shape(1, 1, 1, out_features)
scale = Tensor(scale_f32, per_out, DataType.FP32, TensorType.Scale)
wzp = Tensor(wzp_u8, per_out, DataType.UINT8, TensorType.WZP)
ozp = Tensor(ozp_u8, per_out, DataType.UINT8, TensorType.OZP)

cfg = WeightConverterConfig(ArchitectureType.EVT1, QuantType.OutputChannelQuant)
blob = convert(cfg, weight, None, scale, None, wzp, ozp, output_data_type=DataType.UINT8)
# blob: 1-D uint8 array, ready to copy to the device
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#include "ha_weight_converter.h"

// w: [in_features, out_features] row-major uint8 buffer
HaWeightTensor weight(w, HaWeightShape(1, 1, in_features, out_features), HaWeightDataType::kUint8,
                      HaWeightTensorType::kWeight);
HaWeightShape per_out(1, 1, 1, out_features);
HaWeightTensor scale(scale_f32, per_out, HaWeightDataType::kFp32, HaWeightTensorType::kScale);
HaWeightTensor wzp(wzp_u8, per_out, HaWeightDataType::kUint8, HaWeightTensorType::kWzp);
HaWeightTensor ozp(ozp_u8, per_out, HaWeightDataType::kUint8, HaWeightTensorType::kOzp);

HaWeightConfig cfg{HaWeightArchitectureType::kEvt1, HaWeightQuantType::kOutputChannelQuant};

uint64_t size = 0;
HaWeightGetConvertedSize(&cfg, &weight, nullptr, HaWeightDataType::kUint8, &size);

std::vector<uint8_t> blob(size);
HaError err = HaWeightConvert(&cfg, &weight, nullptr, &scale, nullptr, &wzp, &ozp, HaWeightDataType::kUint8,
                              /*channel_wise_packing=*/true, blob.data(), blob.size());

8.4 Input Tensor Conventions

Weight Shape and Data Layout

The weight shape is described in four dimensions as HaWeightShape(r, s, c, m), where c is the number of input channels and m the number of output channels. For a 2D weight, such as that of a Linear layer, specify (1, 1, in_features, out_features).

The converter reads the weight data as a contiguous row-major buffer of this shape. A 2D weight must therefore be in [in_features, out_features] order.

Checkpoint weights may need to be transposed

Many checkpoints, including PyTorch nn.Linear, store weights as [out_features, in_features]. If you pass them in this order, the element count matches so conversion succeeds without an error, but the result is the layout of a different matrix. Transpose the weight into a contiguous buffer before conversion.

For 4-bit types (kUint4, kSint4, kFp4, kMxfp4), pass a packed buffer with two elements per byte.

Metadata Tensors

Metadata tensor shapes are based on the output channel count m of the weight.

Tensor Shape Data type
bias (1, 1, 1, m) kFp32
scale, WZP (kOutputChannelQuant) (1, 1, 1, m) scale kFp32; WZP uses the same integer type as the weight
scale, WZP (kInputChannelWiseQuant) (1, 1, groups, m) Same as above
OZP (1, 1, 1, m) Same integer type as the output (kUint8 or kSint8)

groups is the number of input channel groups: r * s * c divided by input_channel_group_size, rounded up.

Each metadata tensor means the following.

  • scale is not just the weight scale but the combined value weight scale × input scale / output scale. If the input or output is quantized, include its scale in the value you pass.
  • WZP (weight zero-point) is meaningful only when the weight is an integer type, and is passed at the same precision as the weight. It is not needed for floating-point weights such as FP8 or FP4.
  • OZP (output zero-point) is required only when the output is an integer type, and is passed at the same precision as the output.

Metadata buffer lengths are not checked

The converter determines the length of a metadata tensor only from its declared shape and does not compare it with the actual buffer size. If the buffer is shorter than the shape, the converter reads past its end. Before calling, make sure the shape matches the actual buffer length.

Output Data Type

output_data_type is the data type of the output produced by the matrix operation that uses the converted weight. It is not the type of the converted weight. The supported output types are kFp32, kBf16, kFp8, kUint8, and kSint8. If the quantization method is not kNone and the output is an integer type, an OZP tensor is required.

8.5 Quantization Methods and Architectures

Quantization method Required tensors Cloud EVT0 Cloud EVT1 Edge EVT0
kNone Weight ✅ ✅ ✅
kOutputChannelQuant Weight, scale, WZP (integer weights) ✅ ✅ ✅
kInputChannelWiseQuant Weight, scale, WZP (integer weights) ✅ ✅ ✅
kMxfpQuant Weight, scale, MX scale ❌ ✅ ✅
kTokenWiseQuant — ❌ ❌ ❌

Integer outputs also require OZP, which is not shown in the table (see Input Tensor Conventions). Bias is optional for every method.

The weight type determines the quantization method. 16-bit (kBf16) weights are converted with kNone and no quantization information, 8-bit and 4-bit weights with kOutputChannelQuant or kInputChannelWiseQuant, and MXFP weights (kMxfp8, kMxfp4) with kMxfpQuant.

For kInputChannelWiseQuant, input_channel_group_size must be a multiple of 128 (Cloud EVT0, EVT1) or 64 (Edge EVT0). kMxfpQuant ignores this value and uses the group size defined by the architecture.

8.6 Memory and Error Handling

Part of the host memory used for conversion is allocated from the runtime's host memory manager and counts toward the HA_HOST_MEM_MAX_BYTES limit.

  • Input copies held by HaWeightTensor::CopyOf() and the Python Tensor
  • Result arrays returned by Python convert()

When converting a large model in one process, release input tensors and result arrays as soon as you are done with them. For the limit, see Environment Variables.

Like other runtime APIs, the C++ conversion functions return HaError. In Python, the same failures are raised as RuntimeError.

Error code Common causes
HaInvalidArgument Missing required tensor, insufficient buffer size, input_channel_group_size constraint violation
HaNotSupported Quantization method not supported by the target architecture
HaInternalError Other conversion failures, such as an unsupported data type combination

The HaWeightTensor constructor is an exception. If an argument is invalid (null pointer, zero dimension, data type that does not fit the tensor role), it throws std::runtime_error instead of returning HaError, so create tensors before calling the conversion functions.


Next Steps