Context and KV Cache¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-16
Overview¶
This page explains how the LPU's KV cache is laid out, how that determines the number of requests that can run at once, and what happens when it fills up.
How the KV Cache Is Organized¶
On a GPU, vLLM scatters the KV cache across fixed-size blocks and follows a block table to find them. The LPU instead gives each running request one contiguous region, reached through the hardware's address translation table. Two numbers describe it:
- The region size — an address range that can hold up to
max_model_lentokens. It only reserves addresses and takes no memory by itself. - The memory actually used — grows page by page as the request's context grows. A request that has only read 300
tokens holds far less memory than a full
max_model_len.
The device memory left after the model weights is the KV cache pool those pages come from. The KV cache block size
is fixed at one token, so --block-size does not need to be set.
Choosing --max-model-len¶
--max-model-len is the longest context (prompt plus generated tokens) a single request can have. Set it to the
longest context you actually need. 16,384 is the maximum. EXAONE-4.5 is limited to 4,096.
A large value does not slow down short requests or make them use more memory.
- KV is read in 64-token units. At every step, attention reads a request's KV cache in 64-token pieces, only up
to that request's current context length rounded up to a multiple of 64. A request with a 300-token context reads
320 tokens, whatever
--max-model-lenis. - Memory follows what is used. As described in How the KV Cache Is Organized,
pages are added as the context grows.
--max-model-lenis only used at startup to check that the KV cache pool can hold at least one request of that length.
1 2 3 4 5 | |
Long contexts are validated on the LPU with Qwen3-8B at the 4K, 8K, and 16K boundaries, including a 15,000-token prompt followed by 1,000 generated tokens.
Concurrency¶
Three things bound how many requests run in the same step:
--max-num-seqs— the upper limit you set.- KV cache memory — the scheduler only runs requests whose pages fit in the pool.
- The per-step token budget — LPU cores × 32 tokens (1,024 with 32 cores), shared between prompts being read in (prefill) and requests generating (decode).
--gpu-memory-utilization controls what fraction of device memory the plugin plans to use, and therefore the size of
the KV cache pool.
When the KV Cache Fills Up¶
Because memory is committed as requests grow, a set of requests that fit when they started can outgrow the pool while generating. When that happens, the scheduler preempts a request: its pages are released, the remaining requests continue, and the preempted request is resumed later and recomputed.
This is automatic and does not change results. It is validated on the LPU with Qwen3-8B and Gemma-4 at 8, 16, and 32 concurrent requests, sized so that the batch fits at the start and not at its peak; every preempted request resumes and all outputs match a CPU reference.
Preemption costs time, since a resumed request recomputes its context. If you see it often, lower --max-num-seqs or
--max-model-len.
Checking Memory¶
hyperaccel-smi reports total device allocations in GiB, including runtime reservations and the KV cache. This
differs from checkpoint weight bytes in decimal GB. Compare readings during weight loading separately from readings
after KV cache initialization.
Next¶
- Configuration Guide
- Using Multiple LPUs — each LPU has its own KV cache