serverAppConfigs/serverAppConfigs-docs/LLM/VLLM Overview.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	vLLM is a fast and easy library for large language model inference and serving.	llm vllm	2026-08-28

vLLM Overview

What Is vLLM

vLLM is a fast and easy library for large language model inference and serving. You can use vLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. vLLM supports over 200 model architectures from Hugging Face.

vLLM is fast with these features:

• PagedAttention for efficient key and value memory management
• Continuous batching of incoming requests
• Chunked prefill and prefix caching
• FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
• Tensor, pipeline, data, expert, and context parallelism
• Speculative decoding with n-gram, suffix, EAGLE/EAGLE3, DFlash, DSpark, and MTP
• Automatic kernel generation with torch.compile
• Disaggregated prefill, decode, and encode

vLLM is flexible with these features:

• Seamless integration with Hugging Face models
• High-throughput serving with parallel sampling and beam search
• Streaming outputs
• Structured outputs with xgrammar or guidance
• Tool calling and reasoning parsers
• OpenAI-compatible API server
• Anthropic Messages API and gRPC support
• Efficient multi-LoRA support

Installation

You can install vLLM with uv (recommended) or pip:

uv pip install vllm

Or with pip for NVIDIA GPUs:

pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

For AMD GPUs with ROCm:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

For CPU:

uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/

Quick Start

Offline Batched Inference

from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")

prompts = [
    "Hello, my name is",
    "The capital of France is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    generated_text = output.outputs[0].text
    print(generated_text)

Online Serving

Start the OpenAI-compatible API server:

vllm serve Qwen/Qwen2.5-1.5B-Instruct

Query the server:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 7,
        "temperature": 0
    }'

Chat completions:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "user", "content": "Hello!"}
        ]
    }'

Key Arguments

Model Arguments

Argument	Default	Description
model	required	Model name or path
tokenizer	same as model	Tokenizer name or path
tokenizer_mode	auto	Tokenizer mode
trust_remote_code	false	Trust remote code from Hugging Face
dtype	auto	Data type (auto, float16, bfloat16, float32)
max_model_len	auto	Maximum context length
quantization	null	Quantization method (awq, gptq, fp8, compressed-tensors)
load_format	auto	Weight loading format

GPU Arguments

Argument	Default	Description
tensor_parallel_size	1	Number of GPUs for tensor parallelism
pipeline_parallel_size	1	Number of GPUs for pipeline parallelism
gpu_memory_utilization	0.92	Fraction of GPU memory for KV cache
swap_space	—	retired; the live CPU-KV-offload knob pair is in RAM CPU Offloading in VLLM
enforce_eager	false	Use eager mode (no CUDA graphs)
max_num_batched_tokens	2048	Max tokens per batch
max_num_seqs	128	Max sequences per batch
block_size	16	Token block size
num_gpu_blocks_override	null	Override number of GPU blocks
max_model_len_per_seq	null	Max length per sequence

Optimization Arguments

Argument	Default	Description
disable_log_stats	false	Disable logging
disable_log_requests	false	Disable request logging
max_log_len	null	Max log length
disable_custom_all_reduce	false	Disable custom all-reduce
tokenizer_pool_size	0	Tokenizer pool size
tokenizer_pool_type	none	Tokenizer pool type
enable_chunked_prefill	auto	Enable chunked prefill
max_num_batched_tokens	2048	Max tokens per batch
attention_backend	auto	Attention backend (flash_attn, flashinfer)

Memory Arguments

Argument	Default	Description
gpu_memory_utilization	0.92	Fraction of GPU memory for KV cache
swap_space	—	retired; the live CPU-KV-offload knob pair is in RAM CPU Offloading in VLLM
kv_cache_dtype	auto	KV cache data type (auto, fp8, fp8_e5m2)
max_model_len	auto	Maximum context length
num_gpu_blocks	auto	Number of GPU blocks
num_cpu_blocks	auto	Number of CPU blocks

Device Arguments

Argument	Default	Description
device	auto	Device type (cuda, rocm, xpu, cpu, metal)
distributed_executor_backend	auto	Distributed executor backend
huggingface_tokenizer_kwargs	null	Hugging Face tokenizer kwargs

Architecture

vLLM uses a multi-process architecture:

• API server process handles HTTP requests, tokenization, and input processing
• Engine core process runs the scheduler and coordinates GPU workers
• One GPU worker process per GPU executes model forward passes

vLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.

vLLM supports multiple attention backends:

• FlashAttention for NVIDIA GPUs
• FlashInfer for NVIDIA GPUs
• Triton attention for AMD ROCm
• ROCm attention for AMD ROCm

Supported Models

vLLM supports over 200 model architectures including:

• Decoder-only LLMs (Llama, Qwen, Gemma)
• Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
• Hybrid attention and state-space models (Mamba, Qwen3.5)
• Multi-modal models (LLaVA, Qwen-VL, Pixtral)
• Embedding and retrieval models (E5-Mistral, GTE, ColBERT)
• Reward and classification models (Qwen-Math)

Quantization

vLLM supports these quantization methods:

• FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
• MXFP8/MXFP4 for Hopper GPUs
• NVFP4 for Hopper and Blackwell GPUs
• INT8 for most GPUs
• INT4 for most GPUs
• GPTQ for most GPUs
• AWQ for Turing and newer GPUs (NVIDIA; native support varies on ROCm)
• GGUF for CPU and GPU
• Compressed-tensors for custom quantization
• ModelOpt for NVIDIA Model Optimizer
• TorchAO for PyTorch AO

AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not native tensor cores. AWQ and GPTQ support in vLLM on AMD ROCm is limited; use GGUF or AMD Quark.

Parallelism

vLLM supports multiple parallelism strategies:

• Tensor Parallelism (TP) shards model parameters across GPUs
• Pipeline Parallelism (PP) distributes model layers across GPUs
• Data Parallelism (DP) replicates the model across GPU sets
• Expert Parallelism (EP) distributes expert networks in MoE models
• Context Parallelism (CP) splits context across GPUs

Chunked Prefill — the flag surface, and what it actually does

YES, the runtime exposes chunked-prefill options, and the running shape already uses them. The tcclaviger fork is a vLLM fork, so the standard scheduler flags are present in the shipped image (verified by reading the vllm.engine.arg_utils and vllm.config.scheduler source in the live container frank-archon-qwen, not from a stub):

Flag	Default	What it controls
--enable-chunked-prefill	True in SchedulerConfig; the CLI arg is an auto-flag ("default": None)	whether prefill may be split into chunks
--max-num-batched-tokens (MNBT)	2048 (DEFAULT_MAX_NUM_BATCHED_TOKENS)	tokens processed per prefill iteration
--prefill-schedule-interval	0	gap between prefill iterations

The running shape's argv carries --max-num-batched-tokens 2048 alongside --max-model-len 200704, so chunked prefill is ON and already active: a 200704-token prompt is prefilled in ~98 iterations of 2048 tokens, not in one shot.

It is a TTFT lever, not a throughput lever. MEASURED 2026-09-29 on the ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):

• MNBT 8192 vs the default 2048: neutral on decode at every depth.
• Small values re-chunk prefill only. They hurt TTFT (each chunk is its own iteration, so the first token waits on more iterations) and they never touch decode: a decode step is max_num_seqs x (spec+1) tokens, independent of MNBT.

There is no --ubatch-size equivalent in vLLM. --ubatch-size is a llama.cpp compute-buffer concept (--batch-size/--ubatch-size); the tcclaviger fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-zealot was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no counterpart here, and the two backends should not be reasoned about by analogy.

MNBT is not a lever on this shape either. The deployed 2048 is the fork's default, so a change is a deliberate departure from the shipped value, not a tune. The one thing MNBT does cost: every change re-enters the compile RANGE path, so FRANK_ARCHON_CACHE_DIR pays one recompilation.

See Also

• VLLM Index
• RAM CPU Offloading in VLLM
• Hybrid CPU GPU Processing Dankserver
• Shared Memory Architecture Dankclaw
• Running Qwen3.6 35B A3B FP8 on Dankserver
• Running Qwen3.6 27B FP8 on Dankclaw