8. Weight Conversion¶
LPU kernels do not read checkpoint weights as is. Weights must be converted to the byte layout defined by the target architecture before they are placed in device memory, and the Weight Converter API performs this conversion. This chapter covers why conversion is needed, the conventions for preparing inputs, and how the converted result is moved to the device.
Contents¶
| Section | Objective |
|---|---|
| 8.1 Why Conversion Is Needed | Differences between the checkpoint layout and the device layout |
| 8.2 Usage Paths | When you need to call the API directly and when you do not |
| 8.3 Conversion Flow | From querying the size to placing the result on the device |
| 8.4 Input Tensor Conventions | Shape, data layout, metadata tensors |
| 8.5 Quantization Methods and Architectures | Required tensors per method and support per architecture |
| 8.6 Memory and Error Handling | Host memory usage and failure categories |
8.1 Why Conversion Is Needed¶
LPU matrix kernels read weights grouped by output channel and aligned to DRAM lines. Quantization metadata such as bias, scale, and zero-point must also sit next to each group in a fixed order. The group size, alignment unit, and metadata placement differ by architecture.
The conversion result is an opaque byte buffer that only kernels for that architecture can interpret. Do not read or modify the buffer contents directly, and do not reuse a buffer converted for a different architecture. If you specify the wrong architecture, the size and offsets look correct, but kernels silently produce wrong results.
8.2 Usage Paths¶
Most users do not call the Weight Converter API directly.
| Environment | Conversion path |
|---|---|
| vLLM HyperAccel | The checkpoint conversion tool (python -m vllm_hyperaccel.tools.convert_checkpoint) converts and saves quantized weights ahead of time (see Quantization). No direct call is needed. |
PyTorch (torch_ha) |
torch.ops.lpu.quant_weight_pack() takes PyTorch tensors and converts them. If the target architecture is omitted, it converts for the current device. |
| Direct runtime integration | Call C++ HaWeightConvert() or Python hart.weight_converter.convert(). |
The rest of this chapter covers the third case: integrating directly with the runtime API.
8.3 Conversion Flow¶
Conversion runs only on the host and does not involve a device. There are no handles or state, so each call is independent.
- Prepare input tensors — Wrap the weight and each required metadata tensor in a
HaWeightTensor(see Input Tensor Conventions). - Query the size — Get the size in bytes of the conversion result with
HaWeightGetConvertedSize(). - Convert — Allocate a host buffer of at least that size and call
HaWeightConvert(). The converter writes directly into this buffer and makes no intermediate copy. - Place on the device — Copy the converted buffer to device memory. Kernels then take this device buffer as the weight argument.
Flow of converting on the host and copying to the device (C++ API)
In Python, convert() performs steps 2 and 3 at once and returns the result as a NumPy array.
channel_wise_packing determines whether the conversion result is pre-split per DRAM channel. The
layout kind of the device tensor that holds the result must match this value.
channel_wise_packing |
Device tensor layout |
|---|---|
true (Python default) |
HaTensorKind::SinglePassInterleaved |
false |
HaTensorKind::DramWordInterleaved |
For layout kinds, see HaTensorKind in the Types API and
Memory Model.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | |
8.4 Input Tensor Conventions¶
Weight Shape and Data Layout¶
The weight shape is described in four dimensions as HaWeightShape(r, s, c, m), where c is the number of input channels and m the number of output channels.
For a 2D weight, such as that of a Linear layer, specify (1, 1, in_features, out_features).
The converter reads the weight data as a contiguous row-major buffer of this shape. A 2D weight must therefore be in
[in_features, out_features] order.
Checkpoint weights may need to be transposed
Many checkpoints, including PyTorch nn.Linear, store weights as [out_features, in_features]. If you pass them in this
order, the element count matches so conversion succeeds without an error, but the result is the layout of a different matrix. Transpose
the weight into a contiguous buffer before conversion.
For 4-bit types (kUint4, kSint4, kFp4, kMxfp4), pass a packed buffer with two elements per byte.
Metadata Tensors¶
Metadata tensor shapes are based on the output channel count m of the weight.
| Tensor | Shape | Data type |
|---|---|---|
| bias | (1, 1, 1, m) |
kFp32 |
scale, WZP (kOutputChannelQuant) |
(1, 1, 1, m) |
scale kFp32; WZP uses the same integer type as the weight |
scale, WZP (kInputChannelWiseQuant) |
(1, 1, groups, m) |
Same as above |
| OZP | (1, 1, 1, m) |
Same integer type as the output (kUint8 or kSint8) |
groups is the number of input channel groups: r * s * c divided by input_channel_group_size, rounded up.
Each metadata tensor means the following.
- scale is not just the weight scale but the combined value
weight scale × input scale / output scale. If the input or output is quantized, include its scale in the value you pass. - WZP (weight zero-point) is meaningful only when the weight is an integer type, and is passed at the same precision as the weight. It is not needed for floating-point weights such as FP8 or FP4.
- OZP (output zero-point) is required only when the output is an integer type, and is passed at the same precision as the output.
Metadata buffer lengths are not checked
The converter determines the length of a metadata tensor only from its declared shape and does not compare it with the actual buffer size. If the buffer is shorter than the shape, the converter reads past its end. Before calling, make sure the shape matches the actual buffer length.
Output Data Type¶
output_data_type is the data type of the output produced by the matrix operation that uses the converted weight. It is not the
type of the converted weight. The supported output types are kFp32, kBf16, kFp8, kUint8, and kSint8. If the quantization method
is not kNone and the output is an integer type, an OZP tensor is required.
8.5 Quantization Methods and Architectures¶
| Quantization method | Required tensors | Cloud EVT0 | Cloud EVT1 | Edge EVT0 |
|---|---|---|---|---|
kNone |
Weight | ✅ | ✅ | ✅ |
kOutputChannelQuant |
Weight, scale, WZP (integer weights) | ✅ | ✅ | ✅ |
kInputChannelWiseQuant |
Weight, scale, WZP (integer weights) | ✅ | ✅ | ✅ |
kMxfpQuant |
Weight, scale, MX scale | ❌ | ✅ | ✅ |
kTokenWiseQuant |
— | ❌ | ❌ | ❌ |
Integer outputs also require OZP, which is not shown in the table (see Input Tensor Conventions). Bias is optional for every method.
The weight type determines the quantization method. 16-bit (kBf16) weights are converted with kNone and no quantization information,
8-bit and 4-bit weights with kOutputChannelQuant or kInputChannelWiseQuant, and MXFP weights (kMxfp8, kMxfp4)
with kMxfpQuant.
For kInputChannelWiseQuant, input_channel_group_size must be a multiple of 128 (Cloud EVT0, EVT1) or 64 (Edge EVT0).
kMxfpQuant ignores this value and uses the group size defined by the architecture.
8.6 Memory and Error Handling¶
Part of the host memory used for conversion is allocated from the runtime's host memory manager and
counts toward the HA_HOST_MEM_MAX_BYTES limit.
- Input copies held by
HaWeightTensor::CopyOf()and the PythonTensor - Result arrays returned by Python
convert()
When converting a large model in one process, release input tensors and result arrays as soon as you are done with them. For the limit, see Environment Variables.
Like other runtime APIs, the C++ conversion functions return HaError. In Python, the same failures
are raised as RuntimeError.
| Error code | Common causes |
|---|---|
HaInvalidArgument |
Missing required tensor, insufficient buffer size, input_channel_group_size constraint violation |
HaNotSupported |
Quantization method not supported by the target architecture |
HaInternalError |
Other conversion failures, such as an unsupported data type combination |
The HaWeightTensor constructor is an exception. If an argument is invalid (null pointer, zero dimension, data
type that does not fit the tensor role), it throws std::runtime_error instead of returning HaError, so create tensors
before calling the conversion functions.
Next Steps¶
- Weight Converter API (C++) — exact signatures
- Weight Converter API (Python)
- Memory Model — device tensor layout and copies