Llama-2 7B¶
Document status
Owner: LLMOps · Basis: vLLM HyperAccel 0.0.5 · Last checked: 2026-10-15
Overview¶
This tutorial runs meta-llama/Llama-2-7b-hf with vLLM HyperAccel.
It is a base (non-chat) model, so the examples use the Completions API.
Setup¶
Start from an environment where Installation is complete.
meta-llama/Llama-2-7b-hf is a gated model. Accept its license on Hugging Face, then log in or set HF_TOKEN.
1 | |
Online Serving¶
1. Start the server¶
1 | |
The server is ready when the log shows Application startup complete.
2. Send a request with cURL¶
1 2 3 4 5 6 7 8 | |
The response is a standard OpenAI text_completion object; the generated text is in choices[0].text.
3. Send a request with the OpenAI Python SDK¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
Offline Inference¶
1 2 3 4 5 6 7 8 9 10 11 12 13 | |
Notes¶
- dtype — Llama-2 checkpoints are stored as
float16. The LPU computes inbfloat16, so the plugin switches the model dtype tobfloat16automatically and logs a warning. - Chat — to use the Chat Completions API with a base model, pass a template with
--chat-template.