Skip to content

vLLM HyperAccel: A vLLM Plugin for the LPU

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.6 (vLLM 0.28.0, PyTorch 2.10.0, Python 3.10) · Last checked: 2026-10-15

vLLM HyperAccel (vllm-hyperaccel) is the hardware plugin for running vLLM on the HyperAccel LPU (LLM Processing Unit).

vLLM is not forked. Following vLLM's hardware-pluggable design (RFC), the LPU is integrated through vLLM's plugin interface, so you use the same vllm serve command and LLM class you would use on any other vLLM platform.

Installation

Requirements

Component Version
Python 3.10
vLLM 0.28.0
PyTorch 2.10.0
vLLM HyperAccel 0.0.6

Running on an LPU card also requires the HyperAccel kernel driver and compatible firmware on the host, with an LPU device node (for example /dev/ha0) visible. Driver installation is covered in Quick install.

Container

The vLLM serving image contains Python, PyTorch, vLLM, the plugin, and the SDK components the plugin depends on. Run it on the Docker host of the machine with the LPU.

Log in first with the account HyperAccel issued: docker login public.hyperaccel.net.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
docker run -d --name vllm-hyperaccel \
  --device hyperaccel.ai/lpu=all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  public.hyperaccel.net/hyperaccel-public/vllm-hyperaccel:0.0.6 \
  vllm serve Qwen/Qwen3-8B \
    --host 0.0.0.0 --port 8000 \
    --max-model-len 16384 \
    --no-enable-prefix-caching --no-async-scheduling

docker logs -f vllm-hyperaccel   # ready once "Application startup complete." appears
  • --device hyperaccel.ai/lpu=all passes the cards through CDI. Without CDI, pass device nodes directly, such as --device /dev/ha0. --privileged is not needed.
  • Mounting the Hugging Face cache means a model is downloaded only once. For gated models, also pass -e HF_TOKEN=<token>.
  • For Kubernetes deployment, see Kubernetes support.

Source install

The plugin lives in the vllm/ directory of the hyperaccel-sdk repository and depends on other SDK packages in the same repository (torch_ha, hart, legato, legato_kernels, torch_legato, lccl, hapti). Dependencies are managed with uv.

1
2
3
4
5
6
7
8
# From the hyperaccel-sdk repository root
git submodule update --init --recursive
export HART_BUILD_BACKEND=hw     # build for the card; sim for the simulator
uv sync --project vllm
source .venv/bin/activate

export HA_TOOLCHAIN=1
export BERTHA_ATTN_BACKEND="hyperaccel"

Verify the installation

1
python -c "import vllm_hyperaccel; print(vllm_hyperaccel.__version__)"

The plugin registers itself through vLLM's entry points, so there is nothing to enable (VLLM_PLUGINS=bertha is not needed). When vLLM starts and its log reports that the platform plugin bertha is activated, installation is complete.

Design Overview

The plugin does not reimplement vLLM. Everything a user touches — the OpenAI-compatible API server, the LLM class, tokenization, the engine, request scheduling — is upstream vLLM. The plugin takes over at the point where vLLM hands work to a device, and replaces only the parts that have to know about the LPU.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
Client                OpenAI-compatible API or the LLM class
   │
Upstream vLLM         API server · engine (tokenization, request lifecycle, detokenization)   ← unchanged
   │
vLLM HyperAccel       LPUPlatform        configuration and component selection
                      EVT0Scheduler      token budget and KV cache pages
                      HAWorker / HAModelRunner   load · compile · execute · sample
                      compiled model segments · HyperAccel attention · LPU sampling
   │
HyperAccel SDK        Legato compiler and kernels · runtime · device memory
   │
LPU

Installing the vllm_hyperaccel package registers three entry points that vLLM discovers on its own.

Entry point What it gives vLLM
Platform (bertha) LPUPlatform, which describes the lpu device and chooses the LPU-specific scheduler, worker, attention backend, and compiler
Models (bertha_model) Subclasses of vLLM's own Llama, Qwen3, and Gemma-4 classes that override only what differs on the LPU
Ops (bertha_ops) LPU implementations of linear, embedding, RMSNorm, activations, rotary embedding, and fused MoE

What happens when the server starts

  1. The configuration is fitted to the LPU. LPUPlatform inspects the configuration built from your command line and applies chunked prefill, a per-step token budget derived from the LPU core count, bfloat16, and compiled execution. See the Configuration Guide for the exact values.
  2. The worker claims an LPU. vLLM starts each engine worker in its own process, and HAWorker selects the LPU for its process.
  3. The weights are loaded. vLLM's standard model loader reads the original Hugging Face checkpoint. Each weight is allocated once, directly in the memory layout the LPU's matrix unit needs, rather than being loaded one way and converted.
  4. The model is compiled. vLLM traces the model with torch.compile and splits the graph at every attention operation. Each piece between attention operations (normalization, projections, feed-forward, residuals) is lowered by LegatoCompiler into a single LPU binary, and attention runs as a dedicated HyperAccel attention kernel. The LM head is compiled separately by the model runner.

Compilation happens once, at startup. That is why a server takes a while to report ready, and why later requests do not pay for it. The device memory left after the weights is reserved for the KV cache.

What happens on every step

  1. Validation — sampling options are checked before a request is accepted.
  2. Scheduling — EVT0Scheduler decides which requests run this step and how many tokens each gets, within the LPU token budget, along with the KV cache pages each request needs.
  3. KV cache mapping — pages for newly scheduled tokens are mapped and pages of finished requests are released.
  4. Forward pass — the compiled pieces and attention kernels run in sequence on the LPU.
  5. Logits — only the last position of each request, the one needed to choose its next token, goes through the LM head.
  6. Sampling — greedy and temperature / top_k / top_p sampling run on the LPU, and only token ids return to the host. See LPU Sampling.
  7. Back to vLLM — the engine detokenizes the tokens and streams the text to the client.

Three design principles

  • A token budget tied to the hardware. Each LPU core handles up to 32 tokens per step. With 32 cores, prefill and decode share 1,024 tokens per step. A prompt longer than the budget is read over several steps (chunked prefill).
  • A KV cache addressed per request. Each request gets one contiguous region reached through the hardware address translation table, and memory is committed page by page as the request grows. See Context and KV Cache.
  • Fail clearly, never quietly. A configuration or request option the LPU cannot run is rejected with an error that names it. A different computation is never substituted behind your back.

Relationship to the SDK

Compilation, LPU kernels, kernel launches, and device memory management belong to the HyperAccel SDK, which does not depend on vLLM. The plugin is a thin layer that translates vLLM's contracts into SDK calls, which lets it follow new vLLM releases closely. The layers below are covered in Torch, Runtime, and Compiler.

Tutorials & Features

Supported Models

The models supported by vLLM HyperAccel. Every model is validated on LPU hardware against a CPU reference.

Text Generation Models

Architecture Example model Validated configuration Tutorial
Qwen3ForCausalLM Qwen/Qwen3-8B BF16 · 16K context · concurrent requests · pipeline parallel (2, 4 LPUs) · data parallel Qwen3 8B
Qwen3MoeForCausalLM Qwen/Qwen3-30B-A3B BF16 · FP8 weights · 2K context Qwen3 MoE
Gemma4ForConditionalGeneration google/gemma-4-31B-it BF16 · 16K context · concurrent requests · sliding-window attention Gemma-4
Exaone4_5_ForConditionalGeneration LGAI-EXAONE/EXAONE-4.5-33B BF16 · INT4 weights · 4K context · one request at a time EXAONE-4.5
LlamaForCausalLM meta-llama/Llama-2-7b-hf BF16 (float16 checkpoint run as bfloat16) Llama-2

Gemma-4 and EXAONE-4.5 are published as multimodal checkpoints. On the LPU they are served as text models from the original checkpoint with --language-model-only; no converted checkpoint is needed.

Quantized Models

Source Scheme Conversion Validated configuration Doc
LGAI-EXAONE/EXAONE-4.5-33B-AWQ INT4 Automatic, on the first vllm serve 4K context, one request at a time Quantization
Qwen/Qwen3-8B-AWQ INT4 Automatic, on the first vllm serve 64 input + 64 output tokens, one request at a time Quantization
Qwen/Qwen3-30B-A3B-GPTQ-Int4 INT4 Automatic, on the first vllm serve 64 input + 64 output tokens, one request at a time Quantization
Qwen/Qwen3-8B-FP8 FP8 Automatic, on the first vllm serve 64 input + 64 output tokens, one request at a time Quantization
Qwen/Qwen3-30B-A3B-FP8 FP8 Automatic, on the first vllm serve 64 input + 64 output tokens, one request at a time Quantization

How models are validated

Model Nightly on the LPU On demand on the LPU
Qwen3-8B Three concurrent requests Serving at 16K; long context at 4K / 8K / 16K; pipeline parallel on 2 and 4 LPUs; data parallel on 2 LPUs; KV cache preemption
Qwen3-30B-A3B Greedy generation FP8 at a 2K context
Gemma-4 31B 64 input + 64 output tokens KV cache preemption with 8 / 16 / 32 requests; contexts beyond the sliding window
EXAONE-4.5 33B Short prompts; a full 4K context; INT4
Llama-2 7B Short prompts

The LPU computes in bfloat16, so output is not judged by comparing final text. Instead, each token the LPU chooses is checked against the choice the CPU reference makes at that step, with a small tolerance for near-ties.