Skip to content

Qwen3 30B-A3B (MoE)

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

This tutorial runs the mixture-of-experts model Qwen/Qwen3-30B-A3B in bfloat16 and with quantized weights (FP8, INT4).

In an MoE model, each layer's feed-forward block is replaced by many small "expert" networks, of which only a few run per token: Qwen3-30B-A3B has 128 experts and uses 8 per token. On the LPU, the experts chosen for a batch run together as a single kernel.

Validated on the LPU nightly, checked token by token against a CPU reference.

Setup

Start from an environment where Installation is complete.

BF16

1. Start the server

1
vllm serve Qwen/Qwen3-30B-A3B --max-model-len 2048 --dtype bfloat16 --no-enable-prefix-caching --no-async-scheduling

2. Send a request with cURL

1
2
3
4
5
6
7
8
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-30B-A3B",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 64,
    "temperature": 0
  }'

Offline inference

1
2
3
4
5
6
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-30B-A3B", max_model_len=2048, dtype="bfloat16", enable_prefix_caching=False, async_scheduling=False)

outputs = llm.generate("The capital of France is", SamplingParams(temperature=0, max_tokens=16))
print(outputs[0].outputs[0].text)

Quantized Weights

Serve an official quantized checkpoint by its repository ID as is. The first run converts it to the LPU format automatically and stores it in a cache; later runs read the cache. No quantization flag is needed.

1
2
3
4
5
# FP8
vllm serve Qwen/Qwen3-30B-A3B-FP8 --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling

# INT4
vllm serve Qwen/Qwen3-30B-A3B-GPTQ-Int4 --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling
1
2
3
from vllm import LLM

llm = LLM(model="Qwen/Qwen3-30B-A3B-FP8", max_model_len=2048, dtype="bfloat16", enable_prefix_caching=False, async_scheduling=False)

The attention and expert matrix weights are quantized; the other parameters remain bfloat16. The conversion cache and the sources that can be converted are covered in Quantization.

Notes

  • Architecture — the MoE kernel is built for Qwen3-30B-A3B's expert layout (128 experts, top-8).
  • FP8, INT4 — the LPU device tests compare against a CPU reference with 64 input + 64 output tokens, one request at a time.
  • Multiple LPUs — PP and DP have so far been validated only with a dense model (Qwen3-8B). See Using Multiple LPUs.

Next