Skip to content

Quantization

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.6 · Last checked: 2026-11-13

Overview

The LPU reads quantized weights in an internal LPU format: vLLM's compressed-tensors method with LPU packing fields added. When an official quantized checkpoint is served by its Hugging Face repository ID, vllm serve converts it to this format on the first run and caches it; later runs read the cache. Conversion preserves the quantized values and scales as they are; nothing is requantized.

Model Scheme Source Source layout
EXAONE-4.5 33B INT4 LGAI-EXAONE/EXAONE-4.5-33B-AWQ compressed-tensors pack-quantized
Qwen3-8B INT4 Qwen/Qwen3-8B-AWQ AWQ
Qwen3-30B-A3B INT4 Qwen/Qwen3-30B-A3B-GPTQ-Int4 GPTQ
Qwen3-8B FP8 Qwen/Qwen3-8B-FP8 quant_method: fp8
Qwen3-30B-A3B FP8 Qwen/Qwen3-30B-A3B-FP8 quant_method: fp8

All of them are converted automatically on the first vllm serve. No quantization flag is needed: the method is read from the checkpoint's configuration. Activations and the KV cache stay in bfloat16. There is no path that quantizes a BF16 source, so the source must already be a quantized checkpoint.

First-Run Conversion

1
vllm serve Qwen/Qwen3-30B-A3B-FP8 --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling

On the first run the source is downloaded, converted to the LPU's internal format, and stored in <--download-dir or the Hugging Face cache>/hyperaccel/bertha-<int4|fp8>/<org>--<model>/<revision>-<hash>/. Later runs read that cache directly. The cache is keyed by the source commit SHA and the HART version, so a new source or HART version triggers a new conversion. If several processes start the same model at once, a file lock ensures the conversion runs only once.

The decoder's Q/K/V and gate/up projections are stored fused, and an MoE model's routed experts are packed with them. The LM head, embeddings, norms, and the vision tower keep their original dtype. Only tensor parallel size 1 is supported.

Sources That Can Be Converted

quantization_config in the source config.json names the source layout, and the layout determines the scheme (INT4 or FP8). Only sources whose values can be carried over without requantization are converted; any other source is rejected, with the reason, before the weights are downloaded.

Source layout Scheme Conditions Models
AWQ INT4 4-bit, group size 128, version=gemm, zero_point=true exaone4.5, qwen3, qwen3-moe
GPTQ INT4 4-bit, group size 128, desc_act=false, sym=true Same as above
compressed-tensors pack-quantized INT4 4-bit int, group size 128, symmetric, weight-only Same as above
quant_method: fp8 (official Qwen3 FP8) FP8 E4M3, weight_block_size=[128, 128], activation_scheme=dynamic qwen3, qwen3-moe
compressed-tensors float-quantized FP8 8-bit float (E4M3), group size 128, symmetric, weight-only Same as above

For example, a source with a group size other than 128, asymmetric or activation-ordered quantization, a quantized lm_head, or a compressed-tensors source that also quantizes activations (such as W8A8) is rejected. A BF16 source and a checkpoint already converted to the LPU format are loaded as they are, without conversion.

Local Directory Sources

With HF_HUB_OFFLINE=1, vLLM replaces the repository ID with the snapshot path in the HF cache. That path contains the repository ID and the commit SHA, so it is converted as usual. Any other local directory source has no known commit SHA and is not converted automatically. In that case, convert it yourself with the checkpoint conversion CLI below and serve the result.

1
2
3
4
python -m vllm_hyperaccel.tools.convert_checkpoint <output-dir> --quant-method int4 \
  --source <source-dir> --source-revision <source commit SHA> \
  --source-model LGAI-EXAONE/EXAONE-4.5-33B-AWQ
vllm serve <output-dir> --language-model-only --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling

EXAONE-4.5 INT4

Serve LGAI-EXAONE/EXAONE-4.5-33B-AWQ (compressed-tensors pack-quantized, 4-bit, group size 128, symmetric) by its Hugging Face repository ID as is.

1
vllm serve LGAI-EXAONE/EXAONE-4.5-33B-AWQ --language-model-only --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling

The supported scope is BF16 activations, tensor parallel size 1, one request at a time, and a context of up to 4,096 tokens.

Qwen3 INT4

Serve the official INT4 checkpoints Qwen/Qwen3-8B-AWQ (AWQ) and Qwen/Qwen3-30B-A3B-GPTQ-Int4 (GPTQ) by their repository IDs as they are.

1
2
vllm serve Qwen/Qwen3-8B-AWQ --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling
vllm serve Qwen/Qwen3-30B-A3B-GPTQ-Int4 --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling

The LPU device tests compare against a CPU reference with 64 input + 64 output tokens, one request at a time.

Qwen3 FP8

Serve the official FP8 checkpoints Qwen/Qwen3-8B-FP8 and Qwen/Qwen3-30B-A3B-FP8 by their repository IDs as they are.

1
2
vllm serve Qwen/Qwen3-8B-FP8 --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling
vllm serve Qwen/Qwen3-30B-A3B-FP8 --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling
1
2
3
from vllm import LLM

llm = LLM(model="Qwen/Qwen3-30B-A3B-FP8", max_model_len=2048, dtype="bfloat16", enable_prefix_caching=False, async_scheduling=False)

FP8 on the LPU keeps one scale per output channel for each group of 128 input channels. The 128x128 block scale of the official checkpoints is that group scale repeated with the same value over 128 output channels, so the FP8 values are converted unchanged. The official checkpoints declare dynamic per-token FP8 quantization of activations but store no tensors for it, so the LPU computes activations in bfloat16.

The attention and feed-forward (expert, for MoE) matrix weights are FP8; the other parameters remain bfloat16. The LPU device tests compare against a CPU reference with 64 input + 64 output tokens, one request at a time.

Earlier FP8 checkpoints

The earlier FP8 checkpoints with one scale per output channel (a BF16 source quantized with LLM Compressor, hyper-accel/Qwen3-30B-A3B-FP8-OC) are no longer loaded. Serve the official Qwen/Qwen3-30B-A3B-FP8 instead.

Checkpoint Conversion CLI

python -m vllm_hyperaccel.tools.convert_checkpoint runs the same converter as the first-run conversion. Use it to convert a local directory source, or to write the result to a directory of your choice instead of the cache. --quant-method selects the scheme. If --model is omitted, the model is detected from architectures in the source config.json.

1
2
3
4
5
6
7
# Repository ID source: model detected automatically
python -m vllm_hyperaccel.tools.convert_checkpoint <output-dir> --quant-method int4 \
  --source LGAI-EXAONE/EXAONE-4.5-33B-AWQ

# The model can also be given explicitly (case and -/_ are ignored: qwen3MoE == qwen3-moe)
python -m vllm_hyperaccel.tools.convert_checkpoint <output-dir> --model qwen3-moe --quant-method fp8 \
  --source Qwen/Qwen3-30B-A3B-FP8

When conversion finishes, serve the output directory as is.

1
vllm serve <output-dir> --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling
--model --quant-method architectures
exaone4.5 int4 Exaone4_5_ForConditionalGeneration
qwen3 int4, fp8 Qwen3ForCausalLM
qwen3-moe int4, fp8 Qwen3MoeForCausalLM

The source must be one of the quantized checkpoints in the Sources That Can Be Converted table.

Option Meaning
output Directory for the converted checkpoint; must be empty or not exist
--quant-method Quantization scheme to produce (int4, fp8). Required
--model Model family. When omitted, detected from architectures in the source config.json
--source Hugging Face repository ID or local directory. Required
--source-revision Source commit SHA. Required for a local directory source
--source-model Repository ID recorded for provenance. Required for a local directory source; defaults to --source for a repository ID
--max-shard-size Safetensors shard size in bytes

An unsupported model or scheme, a --model that does not match the source architectures, or a source that cannot be converted fails with exit code 2 before the weights are downloaded.

1
2
--model llama3 --quant-method int4    # error: argument --model: unsupported model 'llama3'; supported: exaone4.5, qwen3, qwen3-moe
--model exaone4.5 --quant-method fp8  # error: quant method 'fp8' is not supported for model 'exaone4.5'; supported: int4

Next