vLLM HyperAccel: A vLLM Plugin for the LPU¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.6 (vLLM 0.28.0, PyTorch 2.10.0, Python 3.10) · Last checked: 2026-10-15
vLLM HyperAccel (vllm-hyperaccel) is the hardware plugin
for running vLLM on the HyperAccel LPU (LLM Processing Unit).
vLLM is not forked. Following vLLM's
hardware-pluggable design (RFC), the LPU is integrated through
vLLM's plugin interface, so you use the same vllm serve command and LLM class you would use on any other vLLM
platform.
Installation¶
Requirements¶
| Component | Version |
|---|---|
| Python | 3.10 |
| vLLM | 0.28.0 |
| PyTorch | 2.10.0 |
| vLLM HyperAccel | 0.0.6 |
Running on an LPU card also requires the HyperAccel kernel driver and compatible firmware on the host, with an LPU
device node (for example /dev/ha0) visible. Driver installation is covered in
Quick install.
Container¶
The vLLM serving image contains Python, PyTorch, vLLM, the plugin, and the SDK components the plugin depends on. Run it on the Docker host of the machine with the LPU.
Log in first with the account HyperAccel issued: docker login public.hyperaccel.net.
1 2 3 4 5 6 7 8 9 10 11 | |
--device hyperaccel.ai/lpu=allpasses the cards through CDI. Without CDI, pass device nodes directly, such as--device /dev/ha0.--privilegedis not needed.- Mounting the Hugging Face cache means a model is downloaded only once. For gated models, also pass
-e HF_TOKEN=<token>. - For Kubernetes deployment, see Kubernetes support.
Source install¶
The plugin lives in the vllm/ directory of the hyperaccel-sdk repository and depends on other SDK packages in the
same repository (torch_ha, hart, legato, legato_kernels, torch_legato, lccl, hapti). Dependencies are
managed with uv.
1 2 3 4 5 6 7 8 | |
Verify the installation¶
1 | |
The plugin registers itself through vLLM's entry points, so there is nothing to enable (VLLM_PLUGINS=bertha is not
needed). When vLLM starts and its log reports that the platform plugin bertha is activated, installation is
complete.
Design Overview¶
The plugin does not reimplement vLLM. Everything a user touches — the OpenAI-compatible API server, the LLM class,
tokenization, the engine, request scheduling — is upstream vLLM. The plugin takes over at the point where vLLM hands
work to a device, and replaces only the parts that have to know about the LPU.
1 2 3 4 5 6 7 8 9 10 11 12 | |
Installing the vllm_hyperaccel package registers three entry points that vLLM discovers on its own.
| Entry point | What it gives vLLM |
|---|---|
Platform (bertha) |
LPUPlatform, which describes the lpu device and chooses the LPU-specific scheduler, worker, attention backend, and compiler |
Models (bertha_model) |
Subclasses of vLLM's own Llama, Qwen3, and Gemma-4 classes that override only what differs on the LPU |
Ops (bertha_ops) |
LPU implementations of linear, embedding, RMSNorm, activations, rotary embedding, and fused MoE |
What happens when the server starts¶
- The configuration is fitted to the LPU.
LPUPlatforminspects the configuration built from your command line and applies chunked prefill, a per-step token budget derived from the LPU core count,bfloat16, and compiled execution. See the Configuration Guide for the exact values. - The worker claims an LPU. vLLM starts each engine worker in its own process, and
HAWorkerselects the LPU for its process. - The weights are loaded. vLLM's standard model loader reads the original Hugging Face checkpoint. Each weight is allocated once, directly in the memory layout the LPU's matrix unit needs, rather than being loaded one way and converted.
- The model is compiled. vLLM traces the model with
torch.compileand splits the graph at every attention operation. Each piece between attention operations (normalization, projections, feed-forward, residuals) is lowered byLegatoCompilerinto a single LPU binary, and attention runs as a dedicated HyperAccel attention kernel. The LM head is compiled separately by the model runner.
Compilation happens once, at startup. That is why a server takes a while to report ready, and why later requests do not pay for it. The device memory left after the weights is reserved for the KV cache.
What happens on every step¶
- Validation — sampling options are checked before a request is accepted.
- Scheduling —
EVT0Schedulerdecides which requests run this step and how many tokens each gets, within the LPU token budget, along with the KV cache pages each request needs. - KV cache mapping — pages for newly scheduled tokens are mapped and pages of finished requests are released.
- Forward pass — the compiled pieces and attention kernels run in sequence on the LPU.
- Logits — only the last position of each request, the one needed to choose its next token, goes through the LM head.
- Sampling — greedy and
temperature/top_k/top_psampling run on the LPU, and only token ids return to the host. See LPU Sampling. - Back to vLLM — the engine detokenizes the tokens and streams the text to the client.
Three design principles¶
- A token budget tied to the hardware. Each LPU core handles up to 32 tokens per step. With 32 cores, prefill and decode share 1,024 tokens per step. A prompt longer than the budget is read over several steps (chunked prefill).
- A KV cache addressed per request. Each request gets one contiguous region reached through the hardware address translation table, and memory is committed page by page as the request grows. See Context and KV Cache.
- Fail clearly, never quietly. A configuration or request option the LPU cannot run is rejected with an error that names it. A different computation is never substituted behind your back.
Relationship to the SDK¶
Compilation, LPU kernels, kernel launches, and device memory management belong to the HyperAccel SDK, which does not depend on vLLM. The plugin is a thin layer that translates vLLM's contracts into SDK calls, which lets it follow new vLLM releases closely. The layers below are covered in Torch, Runtime, and Compiler.
Tutorials & Features¶
- Model tutorials — step-by-step examples running vLLM HyperAccel with representative models - Qwen3 8B - Qwen3 30B-A3B (MoE) - Gemma-4 31B - EXAONE-4.5 33B - Llama-2 7B
- Feature descriptions — the key capabilities of vLLM HyperAccel - Supported Features - Context and KV Cache - LPU Sampling - Using Multiple LPUs - Quantization - OpenAI-Compatible Server - Profiling
- Configuration Guide — engine arguments, values the plugin sets, environment variables
- Troubleshooting
Supported Models¶
The models supported by vLLM HyperAccel. Every model is validated on LPU hardware against a CPU reference.
Text Generation Models¶
| Architecture | Example model | Validated configuration | Tutorial |
|---|---|---|---|
Qwen3ForCausalLM |
Qwen/Qwen3-8B |
BF16 · 16K context · concurrent requests · pipeline parallel (2, 4 LPUs) · data parallel | Qwen3 8B |
Qwen3MoeForCausalLM |
Qwen/Qwen3-30B-A3B |
BF16 · FP8 weights · 2K context | Qwen3 MoE |
Gemma4ForConditionalGeneration |
google/gemma-4-31B-it |
BF16 · 16K context · concurrent requests · sliding-window attention | Gemma-4 |
Exaone4_5_ForConditionalGeneration |
LGAI-EXAONE/EXAONE-4.5-33B |
BF16 · INT4 weights · 4K context · one request at a time | EXAONE-4.5 |
LlamaForCausalLM |
meta-llama/Llama-2-7b-hf |
BF16 (float16 checkpoint run as bfloat16) |
Llama-2 |
Gemma-4 and EXAONE-4.5 are published as multimodal checkpoints. On the LPU they are served as text models from the
original checkpoint with --language-model-only; no converted checkpoint is needed.
Quantized Models¶
| Source | Scheme | Conversion | Validated configuration | Doc |
|---|---|---|---|---|
LGAI-EXAONE/EXAONE-4.5-33B-AWQ |
INT4 | Automatic, on the first vllm serve |
4K context, one request at a time | Quantization |
Qwen/Qwen3-8B-AWQ |
INT4 | Automatic, on the first vllm serve |
64 input + 64 output tokens, one request at a time | Quantization |
Qwen/Qwen3-30B-A3B-GPTQ-Int4 |
INT4 | Automatic, on the first vllm serve |
64 input + 64 output tokens, one request at a time | Quantization |
Qwen/Qwen3-8B-FP8 |
FP8 | Automatic, on the first vllm serve |
64 input + 64 output tokens, one request at a time | Quantization |
Qwen/Qwen3-30B-A3B-FP8 |
FP8 | Automatic, on the first vllm serve |
64 input + 64 output tokens, one request at a time | Quantization |
How models are validated¶
| Model | Nightly on the LPU | On demand on the LPU |
|---|---|---|
| Qwen3-8B | Three concurrent requests | Serving at 16K; long context at 4K / 8K / 16K; pipeline parallel on 2 and 4 LPUs; data parallel on 2 LPUs; KV cache preemption |
| Qwen3-30B-A3B | Greedy generation | FP8 at a 2K context |
| Gemma-4 31B | 64 input + 64 output tokens | KV cache preemption with 8 / 16 / 32 requests; contexts beyond the sliding window |
| EXAONE-4.5 33B | Short prompts; a full 4K context; INT4 | |
| Llama-2 7B | Short prompts |
The LPU computes in bfloat16, so output is not judged by comparing final text. Instead, each token the LPU chooses
is checked against the choice the CPU reference makes at that step, with a small tolerance for near-ties.