Qwen3 30B-A3B (MoE)¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview¶
This tutorial runs the mixture-of-experts model Qwen/Qwen3-30B-A3B in bfloat16 and with quantized weights (FP8, INT4).
In an MoE model, each layer's feed-forward block is replaced by many small "expert" networks, of which only a few run per token: Qwen3-30B-A3B has 128 experts and uses 8 per token. On the LPU, the experts chosen for a batch run together as a single kernel.
Validated on the LPU nightly, checked token by token against a CPU reference.
Setup¶
Start from an environment where Installation is complete.
BF16¶
1. Start the server¶
1 | |
2. Send a request with cURL¶
1 2 3 4 5 6 7 8 | |
Offline inference¶
1 2 3 4 5 6 | |
Quantized Weights¶
Serve an official quantized checkpoint by its repository ID as is. The first run converts it to the LPU format automatically and stores it in a cache; later runs read the cache. No quantization flag is needed.
1 2 3 4 5 | |
1 2 3 | |
The attention and expert matrix weights are quantized; the other parameters remain bfloat16. The conversion cache
and the sources that can be converted are covered in
Quantization.
Notes¶
- Architecture — the MoE kernel is built for Qwen3-30B-A3B's expert layout (128 experts, top-8).
- FP8, INT4 — the LPU device tests compare against a CPU reference with 64 input + 64 output tokens, one request at a time.
- Multiple LPUs — PP and DP have so far been validated only with a dense model (Qwen3-8B). See Using Multiple LPUs.