Quantization¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.6 · Last checked: 2026-11-13
Overview¶
The LPU reads quantized weights in an internal LPU format: vLLM's compressed-tensors method with LPU packing fields
added. When an official quantized checkpoint is served by its Hugging Face repository ID, vllm serve converts it to
this format on the first run and caches it; later runs read the cache. Conversion preserves the quantized values and
scales as they are; nothing is requantized.
| Model | Scheme | Source | Source layout |
|---|---|---|---|
| EXAONE-4.5 33B | INT4 | LGAI-EXAONE/EXAONE-4.5-33B-AWQ |
compressed-tensors pack-quantized |
| Qwen3-8B | INT4 | Qwen/Qwen3-8B-AWQ |
AWQ |
| Qwen3-30B-A3B | INT4 | Qwen/Qwen3-30B-A3B-GPTQ-Int4 |
GPTQ |
| Qwen3-8B | FP8 | Qwen/Qwen3-8B-FP8 |
quant_method: fp8 |
| Qwen3-30B-A3B | FP8 | Qwen/Qwen3-30B-A3B-FP8 |
quant_method: fp8 |
All of them are converted automatically on the first vllm serve. No quantization flag is needed: the method is read
from the checkpoint's configuration. Activations and the KV cache stay in bfloat16. There is no path that quantizes
a BF16 source, so the source must already be a quantized checkpoint.
First-Run Conversion¶
1 | |
On the first run the source is downloaded, converted to the LPU's internal format, and stored in
<--download-dir or the Hugging Face cache>/hyperaccel/bertha-<int4|fp8>/<org>--<model>/<revision>-<hash>/.
Later runs read that cache directly. The cache is keyed by the source commit SHA and the HART version, so a new source
or HART version triggers a new conversion. If several processes start the same model at once, a file lock ensures the
conversion runs only once.
The decoder's Q/K/V and gate/up projections are stored fused, and an MoE model's routed experts are packed with them. The LM head, embeddings, norms, and the vision tower keep their original dtype. Only tensor parallel size 1 is supported.
Sources That Can Be Converted¶
quantization_config in the source config.json names the source layout, and the layout determines the scheme
(INT4 or FP8). Only sources whose values can be carried over without requantization are converted; any other
source is rejected, with the reason, before the weights are downloaded.
| Source layout | Scheme | Conditions | Models |
|---|---|---|---|
| AWQ | INT4 | 4-bit, group size 128, version=gemm, zero_point=true |
exaone4.5, qwen3, qwen3-moe |
| GPTQ | INT4 | 4-bit, group size 128, desc_act=false, sym=true |
Same as above |
compressed-tensors pack-quantized |
INT4 | 4-bit int, group size 128, symmetric, weight-only | Same as above |
quant_method: fp8 (official Qwen3 FP8) |
FP8 | E4M3, weight_block_size=[128, 128], activation_scheme=dynamic |
qwen3, qwen3-moe |
compressed-tensors float-quantized |
FP8 | 8-bit float (E4M3), group size 128, symmetric, weight-only | Same as above |
For example, a source with a group size other than 128, asymmetric or activation-ordered quantization, a quantized
lm_head, or a compressed-tensors source that also quantizes activations (such as W8A8) is rejected. A BF16 source and
a checkpoint already converted to the LPU format are loaded as they are, without conversion.
Local Directory Sources¶
With HF_HUB_OFFLINE=1, vLLM replaces the repository ID with the snapshot path in the HF cache. That path contains the
repository ID and the commit SHA, so it is converted as usual. Any other local directory source has no known commit
SHA and is not converted automatically. In that case, convert it yourself with the
checkpoint conversion CLI below and serve the result.
1 2 3 4 | |
EXAONE-4.5 INT4¶
Serve LGAI-EXAONE/EXAONE-4.5-33B-AWQ (compressed-tensors pack-quantized, 4-bit, group size 128, symmetric) by its
Hugging Face repository ID as is.
1 | |
The supported scope is BF16 activations, tensor parallel size 1, one request at a time, and a context of up to 4,096 tokens.
Qwen3 INT4¶
Serve the official INT4 checkpoints Qwen/Qwen3-8B-AWQ (AWQ) and Qwen/Qwen3-30B-A3B-GPTQ-Int4 (GPTQ) by their
repository IDs as they are.
1 2 | |
The LPU device tests compare against a CPU reference with 64 input + 64 output tokens, one request at a time.
Qwen3 FP8¶
Serve the official FP8 checkpoints Qwen/Qwen3-8B-FP8 and Qwen/Qwen3-30B-A3B-FP8 by their repository IDs as they
are.
1 2 | |
1 2 3 | |
FP8 on the LPU keeps one scale per output channel for each group of 128 input channels. The 128x128 block scale of the
official checkpoints is that group scale repeated with the same value over 128 output channels, so the FP8 values are
converted unchanged. The official checkpoints declare dynamic per-token FP8 quantization of activations but store no
tensors for it, so the LPU computes activations in bfloat16.
The attention and feed-forward (expert, for MoE) matrix weights are FP8; the other parameters remain bfloat16. The
LPU device tests compare against a CPU reference with 64 input + 64 output tokens, one request at a time.
Earlier FP8 checkpoints
The earlier FP8 checkpoints with one scale per output channel (a BF16 source quantized with LLM Compressor,
hyper-accel/Qwen3-30B-A3B-FP8-OC) are no longer loaded. Serve the official Qwen/Qwen3-30B-A3B-FP8 instead.
Checkpoint Conversion CLI¶
python -m vllm_hyperaccel.tools.convert_checkpoint runs the same converter as the first-run conversion. Use it to
convert a local directory source, or to write the result to a directory of your choice instead of the cache.
--quant-method selects the scheme. If --model is omitted, the model is detected from architectures in the source
config.json.
1 2 3 4 5 6 7 | |
When conversion finishes, serve the output directory as is.
1 | |
--model |
--quant-method |
architectures |
|---|---|---|
exaone4.5 |
int4 |
Exaone4_5_ForConditionalGeneration |
qwen3 |
int4, fp8 |
Qwen3ForCausalLM |
qwen3-moe |
int4, fp8 |
Qwen3MoeForCausalLM |
The source must be one of the quantized checkpoints in the Sources That Can Be Converted table.
| Option | Meaning |
|---|---|
output |
Directory for the converted checkpoint; must be empty or not exist |
--quant-method |
Quantization scheme to produce (int4, fp8). Required |
--model |
Model family. When omitted, detected from architectures in the source config.json |
--source |
Hugging Face repository ID or local directory. Required |
--source-revision |
Source commit SHA. Required for a local directory source |
--source-model |
Repository ID recorded for provenance. Required for a local directory source; defaults to --source for a repository ID |
--max-shard-size |
Safetensors shard size in bytes |
An unsupported model or scheme, a --model that does not match the source architectures, or a source that cannot be
converted fails with exit code 2 before the weights are downloaded.
1 2 | |