vLLM OpenAI-Compatible Server: Drop-In API Replacement (2026)

The vLLM OpenAI-compatible server is so good at being a drop-in replacement for OpenAI’s API that you can often forget it’s not actually OpenAI.

Let’s see it in action. Imagine you have a Python script that talks to OpenAI’s chat completion endpoint. Here’s a simplified version:

import openai

openai.api_key = "YOUR_OPENAI_API_KEY" # This will be replaced later

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ]
)

print(response.choices[0].message.content)

Now, let’s say you want to switch to vLLM for faster inference, potentially on your own hardware. You’d first start the vLLM server. Assuming you have a model like Llama-2 7B downloaded, you’d run:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --host 0.0.0.0 \
    --port 8000

This command spins up a web server on http://0.0.0.0:8000 that mimics the OpenAI API. It’s now listening for requests.

To point your Python script to this local server instead of OpenAI, you make two small changes:

1. Set the openai.api_base: This tells the openai library where to send requests.
2. Remove or set a dummy openai.api_key: The vLLM server doesn’t authenticate requests by default, so the key isn’t used.

Here’s the modified Python script:

import openai

# Point to your local vLLM server
openai.api_base = "http://localhost:8000/v1"
# vLLM server doesn't require a key by default, but the library might expect one
openai.api_key = "EMPTY"

response = openai.ChatCompletion.create(
    model="meta-llama/Llama-2-7b-chat-hf", # Use the model name vLLM is serving
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ]
)

print(response.choices[0].message.content)

When you run this script, it sends the request to your vLLM server. vLLM receives it, processes it using the Llama-2 model, and returns a response that’s formatted exactly like OpenAI’s. The script then prints the answer.

The problem vLLM solves is the latency and cost associated with calling large language models over the network, especially for high-throughput applications. By running the model locally or on your own infrastructure, you gain control over performance, privacy, and operational costs. The OpenAI compatibility means you can switch without rewriting significant portions of your application code that already uses the openai Python library or any other client that targets the OpenAI API specification.

Internally, vLLM uses a technique called PagedAttention. Traditional LLM serving systems process attention computations sequentially, which can lead to memory fragmentation and inefficient GPU utilization. PagedAttention, inspired by operating system virtual memory management, breaks down the attention key-value (KV) cache into fixed-size blocks. These blocks can be dynamically allocated and shared among different sequences, dramatically improving memory efficiency and throughput. This allows vLLM to handle significantly more concurrent requests than other serving systems.

The specific levers you control are primarily in how you launch the api_server. The --model argument is crucial, specifying which Hugging Face model identifier vLLM should load. Beyond that, --tensor-parallel-size allows you to distribute a single large model across multiple GPUs, essential for models that don’t fit on one card. --dtype (e.g., float16, bfloat16) lets you control the precision, impacting memory usage and speed. For fine-tuned models, you might use --served-model-name to give it a different alias than its Hugging Face identifier.

When you specify a model like meta-llama/Llama-2-7b-chat-hf to vLLM, it doesn’t just load the weights. It also needs to understand the model’s tokenizer and configuration. If you’re using a model from Hugging Face Hub, vLLM typically handles downloading these automatically. However, if you have a custom model or a model that requires specific configurations not immediately apparent from the name, you might need to ensure the model’s tokenizer.json, config.json, and generation_config.json files are correctly placed or referenced. vLLM’s ChatCompletion endpoint specifically relies on the model’s chat template, which is usually defined in tokenizer_config.json or directly in the tokenizer’s code, to correctly format system, user, and assistant messages into the token sequence the model understands. Without a proper chat template, the model might interpret your structured messages as plain text, leading to nonsensical outputs.

The next concept you’ll likely encounter is handling streaming responses, which vLLM also supports seamlessly.