Skip to content

Configuration Guide

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

vLLM HyperAccel uses the same entry points as upstream vLLM — LLM(...), vllm serve, and EngineArgs. Most arguments work the same way, with a few LPU-specific constraints. Use vllm serve --help for the full list.

LPU-specific settings live in two places:

  • Values the plugin sets — fixed by the plugin when the engine configuration is built
  • Environment variables — HA_*, BERTHA_*

Engine Arguments

The arguments below matter on the LPU or carry LPU constraints.

max_model_len (--max-model-len)

  • Maximum: 16,384. 4,096 for EXAONE-4.5.
  • The longest context a single request can have. KV is read in 64-token units up to each request's current context, and memory follows what is used, so a large value does not slow down short requests. See Context and KV Cache.
  • For a model with sliding-window layers that the LPU has a kernel for (Gemma-4), the context may exceed the window, up to 16,384. For other sliding-window models, max_model_len is reduced to the window size with a warning so results stay correct.

max_num_seqs (--max-num-seqs)

  • Upper bound on concurrent requests. The scheduler further limits concurrency to what fits in the KV cache.

gpu_memory_utilization (--gpu-memory-utilization)

  • The fraction of device memory the plugin uses, which sets the size of the KV cache pool.

pipeline_parallel_size (--pipeline-parallel-size)

  • 1, 2, or 4. Splits one model across several LPUs, using vLLM's default multiprocessing executor. See Using Multiple LPUs.

data_parallel_size (--data-parallel-size)

  • One independent replica per LPU.

tensor_parallel_size (fixed at 1)

  • Only 1 is accepted on the LPU. Use pipeline or data parallelism for multiple LPUs.

language_model_only (--language-model-only)

  • Required for Gemma-4 and EXAONE-4.5. Serves the multimodal checkpoint as a text model.

chat_template (--chat-template)

  • Needed to use the Chat Completions API with models that ship no chat template (for example the Llama-2 base model).

enable_prefix_caching (--no-enable-prefix-caching)

  • vLLM enables prefix caching by default. For LPU serving, turn it off with --no-enable-prefix-caching.

async_scheduling (--no-async-scheduling)

  • Pass --no-async-scheduling to run the same configuration as the hardware serving tests.
1
2
3
4
5
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --dtype bfloat16 \
  --no-enable-prefix-caching \
  --no-async-scheduling
1
2
3
4
5
6
7
8
9
from vllm import LLM

llm = LLM(
    model="Qwen/Qwen3-8B",
    max_model_len=16384,
    dtype="bfloat16",
    enable_prefix_caching=False,
    async_scheduling=False,
)

Values the Plugin Sets

When vLLM builds its configuration, the plugin adjusts the following. You do not need to set them; values you pass that conflict are handled as described.

Setting Value If you pass something else
Device lpu Any other device is rejected
Chunked prefill Enabled Always enabled
max_num_batched_tokens LPU cores × 32 (1,024 with 32 cores) Overridden
KV cache block size 1 Overridden, with a warning if --block-size was given
Model dtype bfloat16 float16 is switched to bfloat16 with a warning
enforce_eager False True is reset with a warning; models always run compiled
Compilation vLLM piecewise compilation + the Legato compiler Disable only with BERTHA_DISABLE_PIECEWISE=1
Scheduler EVT0Scheduler Set by the plugin
Worker HAWorker Set by the plugin
Parallelism tensor_parallel_size=1, pipeline_parallel_size ∈ Other combinations are rejected
Quantization compressed-tensors Read from the checkpoint's configuration

KV cache block size

Earlier releases used a block size of 128. The KV cache is now paged one token per block, so the block size is fixed at 1 and --block-size has no effect. --block-size 1 and --num-gpu-blocks-override can be removed from existing commands.

Environment Variables

Set these on the process that launches the engine. For upstream vLLM environment variables, see the vLLM environment variables guide.

Runtime

HA_TOOLCHAIN

  • Default: unset.
  • When 1, the SDK runtime uses the Legato toolchain for kernel launches. Required for the plugin. Already set in the serving image.

HA_BACKEND

  • Default: follows the build.
  • SDK runtime backend: hw (LPU card) or sim (functional simulator). The available backends are decided at build time by HART_BUILD_BACKEND.

BERTHA_ATTN_BACKEND

  • Default: hyperaccel.
  • Attention backend selector. The native HyperAccel backend is used.
1
2
export HA_TOOLCHAIN=1
export BERTHA_ATTN_BACKEND="hyperaccel"

Plugin behavior

BERTHA_CORE_NUM

  • Default: 32.
  • Number of LPU cores to use. Sets the per-step token budget (cores × 32). The default matches a chip with all cores functional.

BERTHA_FUSE_KV_CACHE_WRITE

  • Default: 1.
  • Writes the KV cache inside the compiled segment that produces K/V, where the kernel supports it. Set to 0 to write it inside the attention launch for every layer.

BERTHA_DISABLE_PIECEWISE

  • Default: 0.
  • Set to 1 to disable piecewise compilation and run the model eagerly. Intended for debugging.

No longer needed

Variable Status
VLLM_PLUGINS=bertha Not needed. The plugin is loaded automatically through its entry point
BERTHA_CORE_NUM=32 Not needed. 32 is the default

Next