Skip to content

Qwen3 8B

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

This tutorial runs Qwen/Qwen3-8B with vLLM HyperAccel.

Qwen3-8B is the most broadly validated model on the LPU: nightly generation tests, serving at its full 16,384-token context, long-context runs at 4K / 8K / 16K, pipeline parallelism on 2 and 4 LPUs, and data parallelism on 2 LPUs.

Setup

Start from an environment where Installation is complete. The model is downloaded from Hugging Face on first run.

Online Serving

1. Start the server

1
2
3
4
5
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --dtype bfloat16 \
  --no-enable-prefix-caching \
  --no-async-scheduling

This is the configuration the hardware serving test runs. The server is ready when the log shows Application startup complete. Weight loading and model compilation run first, so the first start takes a while.

2. Send a request with cURL

From a new terminal:

1
2
3
4
5
6
7
8
9
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 64,
    "temperature": 0,
    "chat_template_kwargs": {"enable_thinking": false}
  }'

The response is a standard OpenAI chat.completion object: the generated text is in choices[0].message.content, and usage reports token counts. enable_thinking is the Qwen3 chat-template switch for thinking mode.

3. Send a request with the OpenAI Python SDK

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

chat_completion = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Summarize what an LPU is in one sentence."}],
    max_tokens=64,
    temperature=0.7,
    top_p=0.8,
)
print(chat_completion.choices[0].message.content)

Offline Inference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-8B", max_model_len=4096, enable_prefix_caching=False, async_scheduling=False)

prompts = [
    "The capital of France is",
    "The largest planet in the solar system is",
]
outputs = llm.generate(prompts, SamplingParams(temperature=0, max_tokens=16))

for output in outputs:
    print(output.prompt, "->", output.outputs[0].text)

The prompts passed to generate are processed together as one batch.

Running on Two LPUs

One replica per LPU, behind a single server (data parallelism):

1
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --data-parallel-size 2 --no-enable-prefix-caching --no-async-scheduling

One model split across two LPUs (pipeline parallelism):

1
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --pipeline-parallel-size 2 --no-enable-prefix-caching --no-async-scheduling

See Using Multiple LPUs for details.

Notes

  • dtype — the LPU computes in bfloat16.
  • Context — up to 16,384 tokens. A smaller --max-model-len reserves less KV cache per request and allows more concurrent requests. See Context and KV Cache.
  • Sampling — greedy and temperature / top_k / top_p run on the LPU. See LPU Sampling.

Next