Skip to content

Context and KV Cache

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-16

Overview

This page explains how the LPU's KV cache is laid out, how that determines the number of requests that can run at once, and what happens when it fills up.

How the KV Cache Is Organized

On a GPU, vLLM scatters the KV cache across fixed-size blocks and follows a block table to find them. The LPU instead gives each running request one contiguous region, reached through the hardware's address translation table. Two numbers describe it:

  • The region size — an address range that can hold up to max_model_len tokens. It only reserves addresses and takes no memory by itself.
  • The memory actually used — grows page by page as the request's context grows. A request that has only read 300 tokens holds far less memory than a full max_model_len.

The device memory left after the model weights is the KV cache pool those pages come from. The KV cache block size is fixed at one token, so --block-size does not need to be set.

Choosing --max-model-len

--max-model-len is the longest context (prompt plus generated tokens) a single request can have. Set it to the longest context you actually need. 16,384 is the maximum. EXAONE-4.5 is limited to 4,096.

A large value does not slow down short requests or make them use more memory.

  • KV is read in 64-token units. At every step, attention reads a request's KV cache in 64-token pieces, only up to that request's current context length rounded up to a multiple of 64. A request with a 300-token context reads 320 tokens, whatever --max-model-len is.
  • Memory follows what is used. As described in How the KV Cache Is Organized, pages are added as the context grows. --max-model-len is only used at startup to check that the KV cache pool can hold at least one request of that length.
1
2
3
4
5
# Chat workloads: a 4K context is enough
vllm serve Qwen/Qwen3-8B --max-model-len 4096 --no-enable-prefix-caching --no-async-scheduling

# Long documents: full context
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --no-enable-prefix-caching --no-async-scheduling

Long contexts are validated on the LPU with Qwen3-8B at the 4K, 8K, and 16K boundaries, including a 15,000-token prompt followed by 1,000 generated tokens.

Concurrency

Three things bound how many requests run in the same step:

  1. --max-num-seqs — the upper limit you set.
  2. KV cache memory — the scheduler only runs requests whose pages fit in the pool.
  3. The per-step token budget — LPU cores × 32 tokens (1,024 with 32 cores), shared between prompts being read in (prefill) and requests generating (decode).

--gpu-memory-utilization controls what fraction of device memory the plugin plans to use, and therefore the size of the KV cache pool.

When the KV Cache Fills Up

Because memory is committed as requests grow, a set of requests that fit when they started can outgrow the pool while generating. When that happens, the scheduler preempts a request: its pages are released, the remaining requests continue, and the preempted request is resumed later and recomputed.

This is automatic and does not change results. It is validated on the LPU with Qwen3-8B and Gemma-4 at 8, 16, and 32 concurrent requests, sized so that the batch fits at the start and not at its peak; every preempted request resumes and all outputs match a CPU reference.

Preemption costs time, since a resumed request recomputes its context. If you see it often, lower --max-num-seqs or --max-model-len.

Checking Memory

hyperaccel-smi reports total device allocations in GiB, including runtime reservations and the KV cache. This differs from checkpoint weight bytes in decimal GB. Compare readings during weight loading separately from readings after KV cache initialization.

Next