Skip to content

Gemma-4 31B

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

This tutorial runs google/gemma-4-31B-it with vLLM HyperAccel.

Gemma-4 is published as a multimodal checkpoint. On the LPU it is served as a text model from that original checkpoint with --language-model-only; no converted checkpoint is needed.

Validated on the LPU nightly (64 input + 64 output tokens, checked against a CPU reference), and under memory pressure with 8, 16, and 32 concurrent requests.

Setup

Start from an environment where Installation is complete.

Online Serving

1. Start the server

1
vllm serve google/gemma-4-31B-it --language-model-only --max-model-len 4096 --no-enable-prefix-caching --no-async-scheduling

2. Send a request with cURL

1
2
3
4
5
6
7
8
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 64,
    "temperature": 0
  }'

Offline Inference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from vllm import LLM, SamplingParams

llm = LLM(
    model="google/gemma-4-31B-it",
    language_model_only=True,
    max_model_len=4096,
    dtype="bfloat16",
    enable_prefix_caching=False,
    async_scheduling=False,
)

messages = [{"role": "user", "content": "What is the capital of France?"}]
outputs = llm.chat(messages, SamplingParams(temperature=0, max_tokens=64))
print(outputs[0].outputs[0].text)

Attention in Gemma-4

Gemma-4 mixes two kinds of layers: local layers that attend over a 1,024-token sliding window, and global layers that attend over the whole context. Both run on the LPU, for contexts up to 16,384 tokens.

  • There is no switch for sliding-window attention. Which layers are local is decided by the model's own configuration, which the plugin leaves untouched.
  • Keep disable_sliding_window at False.
  • Do not enlarge sliding_window through hf_overrides. The model's own 1,024-token window is used.
  • If the whole context fits inside the window (max_model_len <= 1024), full attention is used automatically, since the two are mathematically identical there.
1
llm = LLM(model="google/gemma-4-31B-it", language_model_only=True, max_model_len=1024, enable_prefix_caching=False, async_scheduling=False)

Notes

  • Prompt format — Gemma-4 needs its BOS token. The chat API adds it for you. When sending raw text to the Completions API or llm.generate, keep special tokens enabled; otherwise the model degenerates into repeating the prompt.
  • Memory — Gemma-4's layers use two different attention head widths. The plugin stores the KV cache at the narrower width and lets wide layers span several slots, which uses about 7/12 of the memory that padding every layer to the widest would.
  • Tied embeddings — the output head shares the embedding weights on the device rather than holding a second copy.

Next