Gemma-4 31B¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview¶
This tutorial runs google/gemma-4-31B-it with vLLM HyperAccel.
Gemma-4 is published as a multimodal checkpoint. On the LPU it is served as a text model from that original
checkpoint with --language-model-only; no converted checkpoint is needed.
Validated on the LPU nightly (64 input + 64 output tokens, checked against a CPU reference), and under memory pressure with 8, 16, and 32 concurrent requests.
Setup¶
Start from an environment where Installation is complete.
Online Serving¶
1. Start the server¶
1 | |
2. Send a request with cURL¶
1 2 3 4 5 6 7 8 | |
Offline Inference¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
Attention in Gemma-4¶
Gemma-4 mixes two kinds of layers: local layers that attend over a 1,024-token sliding window, and global layers that attend over the whole context. Both run on the LPU, for contexts up to 16,384 tokens.
- There is no switch for sliding-window attention. Which layers are local is decided by the model's own configuration, which the plugin leaves untouched.
- Keep
disable_sliding_windowatFalse. - Do not enlarge
sliding_windowthroughhf_overrides. The model's own 1,024-token window is used. - If the whole context fits inside the window (
max_model_len <= 1024), full attention is used automatically, since the two are mathematically identical there.
1 | |
Notes¶
- Prompt format — Gemma-4 needs its BOS token. The chat API adds it for you. When sending raw text to the
Completions API or
llm.generate, keep special tokens enabled; otherwise the model degenerates into repeating the prompt. - Memory — Gemma-4's layers use two different attention head widths. The plugin stores the KV cache at the narrower width and lets wide layers span several slots, which uses about 7/12 of the memory that padding every layer to the widest would.
- Tied embeddings — the output head shares the embedding weights on the device rather than holding a second copy.
Next¶
- EXAONE-4.5 33B — another model served with
--language-model-only - Context and KV Cache
- Supported Features