Configuration Guide¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview¶
vLLM HyperAccel uses the same entry points as upstream vLLM — LLM(...), vllm serve, and EngineArgs. Most
arguments work the same way, with a few LPU-specific constraints. Use vllm serve --help for the full list.
LPU-specific settings live in two places:
- Values the plugin sets — fixed by the plugin when the engine configuration is built
- Environment variables —
HA_*,BERTHA_*
Engine Arguments¶
The arguments below matter on the LPU or carry LPU constraints.
max_model_len (--max-model-len)
- Maximum: 16,384. 4,096 for EXAONE-4.5.
- The longest context a single request can have. KV is read in 64-token units up to each request's current context, and memory follows what is used, so a large value does not slow down short requests. See Context and KV Cache.
- For a model with sliding-window layers that the LPU has a kernel for (Gemma-4), the context may exceed the window,
up to 16,384. For other sliding-window models,
max_model_lenis reduced to the window size with a warning so results stay correct.
max_num_seqs (--max-num-seqs)
- Upper bound on concurrent requests. The scheduler further limits concurrency to what fits in the KV cache.
gpu_memory_utilization (--gpu-memory-utilization)
- The fraction of device memory the plugin uses, which sets the size of the KV cache pool.
pipeline_parallel_size (--pipeline-parallel-size)
- 1, 2, or 4. Splits one model across several LPUs, using vLLM's default multiprocessing executor. See Using Multiple LPUs.
data_parallel_size (--data-parallel-size)
- One independent replica per LPU.
tensor_parallel_size (fixed at 1)
- Only 1 is accepted on the LPU. Use pipeline or data parallelism for multiple LPUs.
language_model_only (--language-model-only)
- Required for Gemma-4 and EXAONE-4.5. Serves the multimodal checkpoint as a text model.
chat_template (--chat-template)
- Needed to use the Chat Completions API with models that ship no chat template (for example the Llama-2 base model).
enable_prefix_caching (--no-enable-prefix-caching)
- vLLM enables prefix caching by default. For LPU serving, turn it off with
--no-enable-prefix-caching.
async_scheduling (--no-async-scheduling)
- Pass
--no-async-schedulingto run the same configuration as the hardware serving tests.
Recommended invocation¶
1 2 3 4 5 | |
1 2 3 4 5 6 7 8 9 | |
Values the Plugin Sets¶
When vLLM builds its configuration, the plugin adjusts the following. You do not need to set them; values you pass that conflict are handled as described.
| Setting | Value | If you pass something else |
|---|---|---|
| Device | lpu |
Any other device is rejected |
| Chunked prefill | Enabled | Always enabled |
max_num_batched_tokens |
LPU cores × 32 (1,024 with 32 cores) | Overridden |
| KV cache block size | 1 |
Overridden, with a warning if --block-size was given |
| Model dtype | bfloat16 |
float16 is switched to bfloat16 with a warning |
enforce_eager |
False |
True is reset with a warning; models always run compiled |
| Compilation | vLLM piecewise compilation + the Legato compiler | Disable only with BERTHA_DISABLE_PIECEWISE=1 |
| Scheduler | EVT0Scheduler |
Set by the plugin |
| Worker | HAWorker |
Set by the plugin |
| Parallelism | tensor_parallel_size=1, pipeline_parallel_size ∈ |
Other combinations are rejected |
| Quantization | compressed-tensors |
Read from the checkpoint's configuration |
KV cache block size
Earlier releases used a block size of 128. The KV cache is now paged one token per block, so the block size is
fixed at 1 and --block-size has no effect. --block-size 1 and --num-gpu-blocks-override can be removed from
existing commands.
Environment Variables¶
Set these on the process that launches the engine. For upstream vLLM environment variables, see the vLLM environment variables guide.
Runtime¶
HA_TOOLCHAIN
- Default: unset.
- When
1, the SDK runtime uses the Legato toolchain for kernel launches. Required for the plugin. Already set in the serving image.
HA_BACKEND
- Default: follows the build.
- SDK runtime backend:
hw(LPU card) orsim(functional simulator). The available backends are decided at build time byHART_BUILD_BACKEND.
BERTHA_ATTN_BACKEND
- Default:
hyperaccel. - Attention backend selector. The native HyperAccel backend is used.
1 2 | |
Plugin behavior¶
BERTHA_CORE_NUM
- Default:
32. - Number of LPU cores to use. Sets the per-step token budget (cores × 32). The default matches a chip with all cores functional.
BERTHA_FUSE_KV_CACHE_WRITE
- Default:
1. - Writes the KV cache inside the compiled segment that produces K/V, where the kernel supports it. Set to
0to write it inside the attention launch for every layer.
BERTHA_DISABLE_PIECEWISE
- Default:
0. - Set to
1to disable piecewise compilation and run the model eagerly. Intended for debugging.
No longer needed¶
| Variable | Status |
|---|---|
VLLM_PLUGINS=bertha |
Not needed. The plugin is loaded automatically through its entry point |
BERTHA_CORE_NUM=32 |
Not needed. 32 is the default |