Quickstart¶
On a server that has been through Quick install, try four things: move a tensor to the card, run a kernel on the card, generate text with vLLM, and serve the model and call it with the OpenAI SDK.
Move a tensor to the card¶
lpu is a PyTorch device name, in the same place as cuda.
| tensor.py | |
|---|---|
1 2 3 4 5 6 | |
Run it in the SDK image. No import torch_ha is needed: PyTorch finds the lpu device at startup.
1 2 | |
To run model computation on the card, see Torch.
Run a kernel on the card¶
The SDK image ships a matrix multiply written in Legato. It compiles it, runs it on the card and compares the result with the CPU.
1 2 | |
The last line should read legato DSL matmul OK. To write a kernel yourself, see Your first kernel.
Generate text with vLLM¶
Use vLLM's Python API as is. The plugin picks the LPU settings, so the code never names a device.
| generate.py | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 | |
max_model_len is the number of tokens one request may use, input and output together.
Pass enable_prefix_caching=False and async_scheduling=False for now: vLLM turns both on by default, and the LPU needs them off.
The prompts passed to generate are processed together as one batch.
Run it in the serving image:
1 2 3 | |
Mounting the Hugging Face cache means the model is downloaded once.
Serve it and call it with the OpenAI SDK¶
Start the same model as an OpenAI-compatible server:
1 2 3 4 5 6 | |
With the OpenAI Python SDK (pip install openai), only the address changes:
| chat.py | |
|---|---|
1 2 3 4 5 6 7 8 9 10 11 | |
With enable_thinking off, Qwen3 does not write out a long reasoning trace before answering.
Pass stream=True to receive tokens as they are generated. OpenAI SDKs in other languages connect the same way.
Next¶
Examples for each component are collected under Tutorials.