Skip to content

Quickstart

On a server that has been through Quick install, try four things: move a tensor to the card, run a kernel on the card, generate text with vLLM, and serve the model and call it with the OpenAI SDK.

Move a tensor to the card

lpu is a PyTorch device name, in the same place as cuda.

tensor.py
1
2
3
4
5
6
import torch

x = torch.randn(4, 256, dtype=torch.bfloat16)
y = x.to("lpu")          # copy to the card; with several cards, pick one by number: "lpu:1"
print(y.device)          # lpu:0
print(torch.equal(y.cpu(), x))   # bring it back to the CPU and compare: True

Run it in the SDK image. No import torch_ha is needed: PyTorch finds the lpu device at startup.

1
2
docker run --rm --device hyperaccel.ai/lpu=all -v $PWD:/work -w /work --entrypoint python3 \
  public.hyperaccel.net/hyperaccel-public/hyperaccel-sdk:v0.0.6 tensor.py

To run model computation on the card, see Torch.

Run a kernel on the card

The SDK image ships a matrix multiply written in Legato. It compiles it, runs it on the card and compares the result with the CPU.

1
2
docker run --rm --device hyperaccel.ai/lpu=all --entrypoint python3 \
  public.hyperaccel.net/hyperaccel-public/hyperaccel-sdk:v0.0.6 /opt/legato-examples/legato_example.py

The last line should read legato DSL matmul OK. To write a kernel yourself, see Your first kernel.

Generate text with vLLM

Use vLLM's Python API as is. The plugin picks the LPU settings, so the code never names a device.

generate.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-8B", max_model_len=2048, enable_prefix_caching=False, async_scheduling=False)
params = SamplingParams(temperature=0, max_tokens=64)

prompts = [
    "The capital of South Korea is",
    "In LLM inference, the KV cache",
]
for out in llm.generate(prompts, params):
    print(out.prompt + out.outputs[0].text)

max_model_len is the number of tokens one request may use, input and output together. Pass enable_prefix_caching=False and async_scheduling=False for now: vLLM turns both on by default, and the LPU needs them off. The prompts passed to generate are processed together as one batch.

Run it in the serving image:

1
2
3
docker run --rm --device hyperaccel.ai/lpu=all -v $PWD:/work -w /work \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  public.hyperaccel.net/hyperaccel-public/vllm-hyperaccel:0.0.6 python3 generate.py

Mounting the Hugging Face cache means the model is downloaded once.

Serve it and call it with the OpenAI SDK

Start the same model as an OpenAI-compatible server:

1
2
3
4
5
6
docker run -d --name qwen3 --device hyperaccel.ai/lpu=all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  public.hyperaccel.net/hyperaccel-public/vllm-hyperaccel:0.0.6 \
  vllm serve Qwen/Qwen3-8B --max-model-len 2048 --no-enable-prefix-caching --no-async-scheduling

docker logs -f qwen3   # ready once you see Application startup complete

With the OpenAI Python SDK (pip install openai), only the address changes:

chat.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "How long does the KTX take from Seoul to Busan?"}],
    max_tokens=128,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

With enable_thinking off, Qwen3 does not write out a long reasoning trace before answering. Pass stream=True to receive tokens as they are generated. OpenAI SDKs in other languages connect the same way.

Next

Examples for each component are collected under Tutorials.