Skip to content

OpenAI-Compatible Server

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

vLLM provides an HTTP server that implements OpenAI's Completions API and Chat API. Because the LPU is integrated as a platform plugin, a server started with vllm serve exposes the OpenAI-compatible API unchanged. For details on the server itself, see the vLLM documentation.

Starting the Server

1
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --no-enable-prefix-caching --no-async-scheduling

The server is ready once the log shows that the platform plugin bertha is activated and ends with Application startup complete. The default address is http://localhost:8000/v1. Use --host 0.0.0.0 for external access and --port for a different port.

To start it in a container, see Installation.

Checking the Server

1
2
3
curl http://localhost:8000/health      # health
curl http://localhost:8000/v1/models   # served models
curl http://localhost:8000/metrics     # Prometheus metrics

Requests with cURL

Completions API

1
2
3
4
5
6
7
8
9
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "prompt": "Hello! How is the weather today?",
    "max_tokens": 100,
    "temperature": 0.7,
    "top_p": 0.9
  }'

Chat Completions API

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant."},
      {"role": "user", "content": "Please tell me how to sort a list in Python."}
    ],
    "max_tokens": 200,
    "temperature": 0.5
  }'

Streaming

1
2
3
4
5
6
7
8
curl -N http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "prompt": "Write a short poem about the sea.",
    "max_tokens": 100,
    "stream": true
  }'

Requests with the OpenAI Python SDK

1
2
3
4
5
6
7
import openai

client = openai.OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",  # vLLM does not validate API keys unless --api-key is set
)
print([m.id for m in client.models.list().data])

Completions

1
2
3
4
5
6
7
8
response = client.completions.create(
    model="Qwen/Qwen3-8B",
    prompt="Hello, my name is",
    max_tokens=150,
    temperature=0.7,
)
print(response.choices[0].text)
print(f"Tokens used: {response.usage.total_tokens}")

Chat Completions

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "system", "content": "You are a friendly coding tutor."},
        {"role": "user", "content": "Please write code to print numbers from 1 to 10 using a loop."},
    ],
    max_tokens=200,
    temperature=0.5,
)
print(response.choices[0].message.content)

Streaming

1
2
3
4
5
6
7
8
9
stream = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Tell me about the LPU."}],
    max_tokens=200,
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

The hyperaccel-sdk repository also ships a ready-to-run client, vllm/examples/openai_completion_client.py (add --stream for streaming).

Batching

When several requests arrive at once, the scheduler processes them as one batch, removing finished requests and adding new ones while it runs (continuous batching). No batching option or API is needed. The input tokens plus max_tokens must not exceed --max-model-len.

LPU Metrics

/metrics provides vLLM's built-in metrics plus the LPU metrics exported by the plugin.

Metric Meaning
vllm:lpu_prefill_requests Requests currently in the prefill phase
vllm:lpu_decode_requests Requests currently in the decode phase
vllm:lpu_scheduled_tokens Tokens scheduled in the current step
vllm:lpu_batch_utilization Scheduled tokens relative to the LPU token budget (0.0 – 1.0)
vllm:lpu_step_duration_seconds Duration of a single model execution step

Per-request metrics such as TTFT and TPOT are part of vLLM's built-in metrics. To look deeper, at the kernel level, see Profiling.

Next