Qwen3 8B¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview¶
This tutorial runs Qwen/Qwen3-8B with vLLM HyperAccel.
Qwen3-8B is the most broadly validated model on the LPU: nightly generation tests, serving at its full 16,384-token context, long-context runs at 4K / 8K / 16K, pipeline parallelism on 2 and 4 LPUs, and data parallelism on 2 LPUs.
Setup¶
Start from an environment where Installation is complete. The model is downloaded from Hugging Face on first run.
Online Serving¶
1. Start the server¶
1 2 3 4 5 | |
This is the configuration the hardware serving test runs. The server is ready when the log shows
Application startup complete. Weight loading and model compilation run first, so the first start takes a while.
2. Send a request with cURL¶
From a new terminal:
1 2 3 4 5 6 7 8 9 | |
The response is a standard OpenAI chat.completion object: the generated text is in choices[0].message.content,
and usage reports token counts. enable_thinking is the Qwen3 chat-template switch for thinking mode.
3. Send a request with the OpenAI Python SDK¶
1 2 3 4 5 6 7 8 9 10 11 12 | |
Offline Inference¶
1 2 3 4 5 6 7 8 9 10 11 12 | |
The prompts passed to generate are processed together as one batch.
Running on Two LPUs¶
One replica per LPU, behind a single server (data parallelism):
1 | |
One model split across two LPUs (pipeline parallelism):
1 | |
See Using Multiple LPUs for details.
Notes¶
- dtype — the LPU computes in
bfloat16. - Context — up to 16,384 tokens. A smaller
--max-model-lenreserves less KV cache per request and allows more concurrent requests. See Context and KV Cache. - Sampling — greedy and
temperature/top_k/top_prun on the LPU. See LPU Sampling.