serverAppConfigs/docs/VLLM Overview.md

VLLM Overview

What Is VLLM

VLLM is a fast and easy library for large language model inference and serving. You can use VLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. VLLM supports over 200 model architectures from Hugging Face.

VLLM is fast with these features:

• PagedAttention for efficient key and value memory management
• Continuous batching of incoming requests
• Chunked prefill and prefix caching
• FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
• Tensor, pipeline, data, expert, and context parallelism
• Speculative decoding with n-gram, suffix, EAGLE/EAGLE3, DFlash, DSpark, and MTP
• Automatic kernel generation with torch.compile
• Disaggregated prefill, decode, and encode

VLLM is flexible with these features:

• Seamless integration with Hugging Face models
• High-throughput serving with parallel sampling and beam search
• Streaming outputs
• Structured outputs with xgrammar or guidance
• Tool calling and reasoning parsers
• OpenAI-compatible API server
• Anthropic Messages API and gRPC support
• Efficient multi-LoRA support

Installation

You can install VLLM with uv (recommended) or pip:

uv pip install vllm

Or with pip for NVIDIA GPUs:

pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

For AMD GPUs with ROCm:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

For CPU:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/

Quick Start

Offline Batched Inference

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")

prompts = [
    "Hello, my name is",
    "The capital of France is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    generated_text = output.outputs[0].text
    print(generated_text)

Online Serving

Start the OpenAI-compatible API server:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Query the server:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 7,
        "temperature": 0
    }'

Chat completions:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "user", "content": "Hello!"}
        ]
    }'

Key Arguments

Model Arguments

Argument	Default	Description
model	required	Model name or path
tokenizer	same as model	Tokenizer name or path
tokenizer_mode	auto	Tokenizer mode
trust_remote_code	false	Trust remote code from Hugging Face
dtype	auto	Data type (auto, float16, bfloat16, float32)
max_model_len	auto	Maximum context length
quantization	null	Quantization method (awq, gptq, fp8, compressed-tensors)
load_format	auto	Weight loading format

GPU Arguments

Argument	Default	Description
tensor_parallel_size	1	Number of GPUs for tensor parallelism
pipeline_parallel_size	1	Number of GPUs for pipeline parallelism
gpu_memory_utilization	0.92	Fraction of GPU memory for KV cache
swap_space	removed	Removed in current vLLM; use --kv-offloading-size
enforce_eager	false	Use eager mode (no CUDA graphs)
max_num_batched_tokens	2048	Max tokens per batch
max_num_seqs	128	Max sequences per batch
block_size	16	Token block size
num_gpu_blocks_override	null	Override number of GPU blocks
max_model_len_per_seq	null	Max length per sequence

Optimization Arguments

Argument	Default	Description
disable_log_stats	false	Disable logging
disable_log_requests	false	Disable request logging
max_log_len	null	Max log length
disable_custom_all_reduce	false	Disable custom all-reduce
tokenizer_pool_size	0	Tokenizer pool size
tokenizer_pool_type	none	Tokenizer pool type
enable_chunked_prefill	auto	Enable chunked prefill
max_num_batched_tokens	2048	Max tokens per batch
attention_backend	auto	Attention backend (flash_attn, flashinfer)

Memory Arguments

Argument	Default	Description
gpu_memory_utilization	0.92	Fraction of GPU memory for KV cache
swap_space	removed	Removed in current vLLM; use --kv-offloading-size
kv_cache_dtype	auto	KV cache data type (auto, fp8, fp8_e5m2)
max_model_len	auto	Maximum context length
num_gpu_blocks	auto	Number of GPU blocks
num_cpu_blocks	auto	Number of CPU blocks

Device Arguments

Argument	Default	Description
device	auto	Device type (cuda, rocm, xpu, cpu, metal)
distributed_executor_backend	auto	Distributed executor backend
huggingface_tokenizer_kwargs	null	Hugging Face tokenizer kwargs

Architecture

VLLM uses a multi-process architecture:

• API server process handles HTTP requests, tokenization, and input processing
• Engine core process runs the scheduler and coordinates GPU workers
• One GPU worker process per GPU executes model forward passes

VLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.

VLLM supports multiple attention backends:

• FlashAttention for NVIDIA GPUs
• FlashInfer for NVIDIA GPUs
• Triton attention for AMD ROCm
• ROCm attention for AMD ROCm

Supported Models

VLLM supports over 200 model architectures including:

• Decoder-only LLMs (Llama, Qwen, Gemma)
• Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
• Hybrid attention and state-space models (Mamba, Qwen3.5)
• Multi-modal models (LLaVA, Qwen-VL, Pixtral)
• Embedding and retrieval models (E5-Mistral, GTE, ColBERT)
• Reward and classification models (Qwen-Math)

Quantization

VLLM supports these quantization methods:

• FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
• MXFP8/MXFP4 for Hopper GPUs
• NVFP4 for Hopper and Blackwell GPUs
• INT8 for most GPUs
• INT4 for most GPUs
• GPTQ for most GPUs
• AWQ for Turing and newer GPUs (NVIDIA; native support varies on ROCm)
• GGUF for CPU and GPU
• Compressed-tensors for custom quantization
• ModelOpt for NVIDIA Model Optimizer
• TorchAO for PyTorch AO

AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not native tensor cores. AWQ and GPTQ support in vLLM on AMD ROCm is limited; use GGUF or AMD Quark.

Parallelism

VLLM supports multiple parallelism strategies:

• Tensor Parallelism (TP) shards model parameters across GPUs
• Pipeline Parallelism (PP) distributes model layers across GPUs
• Data Parallelism (DP) replicates the model across GPU sets
• Expert Parallelism (EP) distributes expert networks in MoE models
• Context Parallelism (CP) splits context across GPUs

See Also

• VLLM Index
• RAM CPU Offloading in VLLM
• Hybrid CPU GPU Processing Dankserver
• Shared Memory Architecture Dankclaw
• Running Qwen3.6 35B A3B FP8 on Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw