vLLM’s CUDA graph optimization is a technique to significantly reduce the overhead associated with launching kernels on the GPU, particularly for repetitive operations like those found in LLM inference.
Here’s how it works in practice. Imagine a typical LLM forward pass: a sequence of operations, each involving a kernel launch. Without CUDA graphs, each kernel launch incurs a small but cumulative cost from the CUDA driver and runtime. vLLM can capture a sequence of these kernels into a single CUDA graph. When this graph is executed, the GPU launches the entire sequence as a single, optimized unit, bypassing much of the per-kernel launch overhead. This is especially impactful for shorter sequences or when processing many small batches, where the launch overhead can become a substantial portion of the total execution time.
Let’s look at some actual configuration. To enable CUDA graph optimization in vLLM, you typically set the enable_cuda_graph flag to True during the LLMEngine initialization.
from vllm import LLM, SamplingParams
# Initialize the LLM engine with CUDA graph optimization enabled
llm = LLM(model="meta-llama/Llama-2-7b-hf", enable_cuda_graph=True)
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
# Example prompts
prompts = [
"Hello, my name is",
"The capital of France is",
"The future of AI is",
]
# Generate output
outputs = llm.generate(prompts, sampling_params)
# Print the outputs
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
The enable_cuda_graph=True parameter tells the vLLM engine to attempt to capture and optimize kernel sequences. When the engine processes the first request for a given sequence length and batch size, it will record the sequence of kernels required for that computation into a CUDA graph. Subsequent requests with the same characteristics can then execute this pre-compiled graph, leading to faster inference.
The core problem this solves is the latency introduced by the CPU-to-GPU communication and kernel dispatch process. For many small inference requests, or when generating short sequences, this overhead can dominate the actual computation time. CUDA graphs allow the GPU to manage the execution of the entire kernel sequence more autonomously, reducing the back-and-forth between the CPU and GPU.
Internally, vLLM uses the CUDA Driver API (cudaLaunchKernel, cudaStreamCaptureBegin, cudaStreamCaptureEnd, cudaGraphInstantiate, cudaGraphLaunch) to achieve this. When enable_cuda_graph is true, vLLM enters a capture mode. As kernels are launched for the forward pass, their parameters and execution configurations are recorded. Once the sequence is complete, the captured operations are turned into an executable graph. This graph is then instantiated and can be launched repeatedly. The graph is dynamic, meaning vLLM can handle variations in sequence length (e.g., by padding or truncating) and batch size, although performance is often best when these parameters are consistent.
The most surprising aspect of CUDA graph optimization is how it can make small, frequent inference requests faster than larger, less frequent ones, even if the total amount of computation is the same. This is because the per-request overhead becomes a much smaller fraction of the total time for smaller requests, allowing the graph optimization to shine. A common misconception is that it only benefits massive batch sizes, but its strength lies in amortizing launch costs across any repeated computation pattern.
The next concept you’ll likely encounter is how vLLM manages different CUDA graph instances for varying sequence lengths and batch sizes, and the memory implications of maintaining these cached graphs.