OpenAI-Compatible Server
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview
vLLM provides an HTTP server that implements OpenAI's Completions API and Chat API. Because the LPU is integrated as
a platform plugin, a server started with vllm serve exposes the OpenAI-compatible API unchanged. For details on the
server itself, see the vLLM documentation.
Starting the Server
| vllm serve Qwen/Qwen3-8B --max-model-len 16384 --no-enable-prefix-caching --no-async-scheduling
|
The server is ready once the log shows that the platform plugin bertha is activated and ends with
Application startup complete. The default address is http://localhost:8000/v1. Use --host 0.0.0.0 for external
access and --port for a different port.
To start it in a container, see Installation.
Checking the Server
| curl http://localhost:8000/health # health
curl http://localhost:8000/v1/models # served models
curl http://localhost:8000/metrics # Prometheus metrics
|
Requests with cURL
Completions API
| curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"prompt": "Hello! How is the weather today?",
"max_tokens": 100,
"temperature": 0.7,
"top_p": 0.9
}'
|
Chat Completions API
| curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"messages": [
{"role": "system", "content": "You are a helpful AI assistant."},
{"role": "user", "content": "Please tell me how to sort a list in Python."}
],
"max_tokens": 200,
"temperature": 0.5
}'
|
Streaming
| curl -N http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-8B",
"prompt": "Write a short poem about the sea.",
"max_tokens": 100,
"stream": true
}'
|
Requests with the OpenAI Python SDK
| import openai
client = openai.OpenAI(
base_url="http://localhost:8000/v1",
api_key="EMPTY", # vLLM does not validate API keys unless --api-key is set
)
print([m.id for m in client.models.list().data])
|
Completions
| response = client.completions.create(
model="Qwen/Qwen3-8B",
prompt="Hello, my name is",
max_tokens=150,
temperature=0.7,
)
print(response.choices[0].text)
print(f"Tokens used: {response.usage.total_tokens}")
|
Chat Completions
| response = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[
{"role": "system", "content": "You are a friendly coding tutor."},
{"role": "user", "content": "Please write code to print numbers from 1 to 10 using a loop."},
],
max_tokens=200,
temperature=0.5,
)
print(response.choices[0].message.content)
|
Streaming
| stream = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[{"role": "user", "content": "Tell me about the LPU."}],
max_tokens=200,
stream=True,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
|
The hyperaccel-sdk repository also ships a ready-to-run client, vllm/examples/openai_completion_client.py (add
--stream for streaming).
Batching
When several requests arrive at once, the scheduler processes them as one batch, removing finished requests and
adding new ones while it runs (continuous batching). No batching option or API is needed. The input tokens plus
max_tokens must not exceed --max-model-len.
LPU Metrics
/metrics provides vLLM's built-in metrics plus the LPU metrics exported by the plugin.
| Metric |
Meaning |
vllm:lpu_prefill_requests |
Requests currently in the prefill phase |
vllm:lpu_decode_requests |
Requests currently in the decode phase |
vllm:lpu_scheduled_tokens |
Tokens scheduled in the current step |
vllm:lpu_batch_utilization |
Scheduled tokens relative to the LPU token budget (0.0 – 1.0) |
vllm:lpu_step_duration_seconds |
Duration of a single model execution step |
Per-request metrics such as TTFT and TPOT are part of vLLM's built-in metrics. To look deeper, at the kernel level,
see Profiling.
Next