Skip to content

Using Multiple LPUs

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

vLLM starts each engine worker in its own process, and every vLLM HyperAccel worker drives exactly one LPU. Scaling to several LPUs therefore uses vLLM's standard parallelism options.

Pipeline parallelism (PP) Data parallelism (DP)
What it does Splits one model's layers across LPUs Runs one full copy of the model per LPU
Use it when The model does not fit on one LPU You need more throughput
Sizes 2 or 4 As many LPUs as you have
Validated with Qwen3-8B on 2 and 4 LPUs Qwen3-8B, 2 replicas
Option --pipeline-parallel-size --data-parallel-size

Pipeline Parallelism (PP)

The model's layers are divided into consecutive stages, one per LPU. Each step, a stage runs its layers and sends its activations to the next stage; the last stage computes the logits and samples. Activations travel between LPUs through the HyperAccel collective communication library, LCCL.

1
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --pipeline-parallel-size 2 --no-enable-prefix-caching --no-async-scheduling
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from vllm import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3-8B",
    max_model_len=16384,
    pipeline_parallel_size=2,   # or 4
    dtype="bfloat16",
    enable_prefix_caching=False,
    async_scheduling=False,
)
outputs = llm.generate("The capital of France is", SamplingParams(temperature=0, max_tokens=8))
print(outputs[0].outputs[0].text)

Requirements:

  • pipeline_parallel_size is 2 or 4, and tensor_parallel_size is 1.
  • vLLM's default multiprocessing executor is used.
  • Stage r runs on lpu:<r>, so the node needs at least as many LPUs as stages.

Validated with Qwen3-8B on 2 and on 4 LPUs, with output checked against a CPU reference.

Data Parallelism (DP)

Each replica is a complete, independent engine with its own scheduler and its own KV cache, bound to its own LPU: replica r runs on lpu:<r>. vLLM's front end distributes incoming requests across the replicas.

1
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --data-parallel-size 2 --no-enable-prefix-caching --no-async-scheduling
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from vllm import AsyncEngineArgs
from vllm.v1.engine.async_llm import AsyncLLM

engine = AsyncLLM.from_engine_args(
    AsyncEngineArgs(
        model="Qwen/Qwen3-8B",
        data_parallel_size=2,
        dtype="bfloat16",
        enable_prefix_caching=False,
    )
)

Notes:

  • Replicas do not share a KV cache, so a request stays on the replica it was assigned to.
  • The node needs one LPU per replica.

Validated with Qwen3-8B as two replicas on two LPUs, with output checked against a CPU reference.

Device Binding

Each worker checks at startup that the LPU it selected is the one the runtime bound. If they differ, or if the requested LPU index does not exist, it stops with an error naming both.

Next