Skip to content

Llama-2 7B

Document status

Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15

Overview

This tutorial runs meta-llama/Llama-2-7b-hf with vLLM HyperAccel.

It is a base (non-chat) model, so the examples use the Completions API.

Setup

Start from an environment where Installation is complete.

meta-llama/Llama-2-7b-hf is a gated model. Accept its license on Hugging Face, then log in or set HF_TOKEN.

1
hf auth login

Online Serving

1. Start the server

1
vllm serve meta-llama/Llama-2-7b-hf --no-enable-prefix-caching --no-async-scheduling

The server is ready when the log shows Application startup complete.

2. Send a request with cURL

1
2
3
4
5
6
7
8
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-2-7b-hf",
    "prompt": "The capital of France is",
    "max_tokens": 16,
    "temperature": 0
  }'

The response is a standard OpenAI text_completion object; the generated text is in choices[0].text.

3. Send a request with the OpenAI Python SDK

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

model = client.models.list().data[0].id

completion = client.completions.create(
    model=model,
    prompt="A robot may not injure a human being",
    max_tokens=10,
    temperature=0.7,
    n=2,  # generate 2 responses
)
for choice in completion.choices:
    print(choice.text)

Offline Inference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_prefix_caching=False, async_scheduling=False)

prompts = [
    "Hello world!",
    "How are you?",
    "The capital of France is",
]
outputs = llm.generate(prompts, SamplingParams(temperature=0, max_tokens=16))

for output in outputs:
    print(output.prompt, "->", output.outputs[0].text)

Notes

  • dtype — Llama-2 checkpoints are stored as float16. The LPU computes in bfloat16, so the plugin switches the model dtype to bfloat16 automatically and logs a warning.
  • Chat — to use the Chat Completions API with a base model, pass a template with --chat-template.

Next