LPU Sampling¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview¶
As vocabularies grow, choosing the next token takes a growing share of inference time. Copying the full logits to the host to choose there means a vocabulary-sized transfer every step.
vLLM HyperAccel runs greedy decoding and temperature / top_k / top_p sampling on the LPU, and only the
chosen token ids return to the host.
Enabling It¶
It is on by default and there is nothing to configure. Use the standard vLLM SamplingParams or the OpenAI API
request fields as usual.
1 2 3 4 5 6 7 8 | |
Batches that mix greedy and random-sampling requests are processed together on the LPU.
Supported Parameters¶
| Parameter | Where it runs | Description |
|---|---|---|
temperature |
LPU | 0 selects greedy decoding |
top_k |
LPU | |
top_p |
LPU | |
seed |
LPU | Same seed, same result |
stop, max_tokens, n |
vLLM | Handled by the vLLM engine |
The platform checks sampling options before accepting a request. A request that uses an option outside this list does not fail deep inside the engine; it is returned at submission with an error naming the option.
Greedy Ties¶
When several tokens share the highest logit, the LPU may pick a different one of them than torch.argmax would.
The LPU computes in bfloat16, so when two tokens score almost the same and the other one is chosen, the text can
diverge from there. Outputs are otherwise the same as a CPU reference.