Skip to content

LPU Sampling

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

As vocabularies grow, choosing the next token takes a growing share of inference time. Copying the full logits to the host to choose there means a vocabulary-sized transfer every step.

vLLM HyperAccel runs greedy decoding and temperature / top_k / top_p sampling on the LPU, and only the chosen token ids return to the host.

Enabling It

It is on by default and there is nothing to configure. Use the standard vLLM SamplingParams or the OpenAI API request fields as usual.

1
2
3
4
5
6
7
8
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen3-8B", max_model_len=4096, enable_prefix_caching=False, async_scheduling=False)

greedy = SamplingParams(temperature=0, max_tokens=50)
random = SamplingParams(temperature=0.7, top_p=0.8, top_k=20, seed=0, max_tokens=50)

outputs = llm.generate(["The capital of France is", "Write a haiku about the sea."], greedy)

Batches that mix greedy and random-sampling requests are processed together on the LPU.

Supported Parameters

Parameter Where it runs Description
temperature LPU 0 selects greedy decoding
top_k LPU
top_p LPU
seed LPU Same seed, same result
stop, max_tokens, n vLLM Handled by the vLLM engine

The platform checks sampling options before accepting a request. A request that uses an option outside this list does not fail deep inside the engine; it is returned at submission with an error naming the option.

Greedy Ties

When several tokens share the highest logit, the LPU may pick a different one of them than torch.argmax would. The LPU computes in bfloat16, so when two tokens score almost the same and the other one is chosen, the text can diverge from there. Outputs are otherwise the same as a CPU reference.

Next