serverAppConfigs/serverAppConfigs-docs/LLM/VLLM Overview.md

---
type: reference
description: "vLLM is a fast and easy library for large language model inference and serving."
tags:
  - llm
  - vllm
timestamp: 2026-08-28
---

# vLLM Overview

## What Is vLLM

vLLM is a fast and easy library for large language model inference and serving. You can use vLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. vLLM supports over 200 model architectures from Hugging Face.

vLLM is fast with these features:

- PagedAttention for efficient key and value memory management
- Continuous batching of incoming requests
- Chunked prefill and prefix caching
- FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
- Tensor, pipeline, data, expert, and context parallelism
- Speculative decoding with n-gram, suffix, EAGLE/EAGLE3, DFlash, DSpark, and MTP
- Automatic kernel generation with torch.compile
- Disaggregated prefill, decode, and encode

vLLM is flexible with these features:

- Seamless integration with Hugging Face models
- High-throughput serving with parallel sampling and beam search
- Streaming outputs
- Structured outputs with xgrammar or guidance
- Tool calling and reasoning parsers
- OpenAI-compatible API server
- Anthropic Messages API and gRPC support
- Efficient multi-LoRA support

## Installation

You can install vLLM with uv (recommended) or pip:

```
uv pip install vllm
```

Or with pip for NVIDIA GPUs:

```
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
```

For AMD GPUs with ROCm:

```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

For CPU:

```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
```

## Quick Start

### Offline Batched Inference

```python
from vllm import LLM, SamplingParams

llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")

prompts = [
    "Hello, my name is",
    "The capital of France is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    generated_text = output.outputs[0].text
    print(generated_text)
```

### Online Serving

Start the OpenAI-compatible API server:

```
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```

Query the server:

```
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 7,
        "temperature": 0
    }'
```

Chat completions:

```
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "messages": [
            {"role": "user", "content": "Hello!"}
        ]
    }'
```

## Key Arguments

### Model Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `model` | required | Model name or path |
| `tokenizer` | same as model | Tokenizer name or path |
| `tokenizer_mode` | auto | Tokenizer mode |
| `trust_remote_code` | false | Trust remote code from Hugging Face |
| `dtype` | auto | Data type (auto, float16, bfloat16, float32) |
| `max_model_len` | auto | Maximum context length |
| `quantization` | null | Quantization method (awq, gptq, fp8, compressed-tensors) |
| `load_format` | auto | Weight loading format |

### GPU Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `tensor_parallel_size` | 1 | Number of GPUs for tensor parallelism |
| `pipeline_parallel_size` | 1 | Number of GPUs for pipeline parallelism |
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
| `swap_space` | — | retired; the live CPU-KV-offload knob pair is in [[RAM CPU Offloading in VLLM]] |
| `enforce_eager` | false | Use eager mode (no CUDA graphs) |
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
| `max_num_seqs` | 128 | Max sequences per batch |
| `block_size` | 16 | Token block size |
| `num_gpu_blocks_override` | null | Override number of GPU blocks |
| `max_model_len_per_seq` | null | Max length per sequence |

### Optimization Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `disable_log_stats` | false | Disable logging |
| `disable_log_requests` | false | Disable request logging |
| `max_log_len` | null | Max log length |
| `disable_custom_all_reduce` | false | Disable custom all-reduce |
| `tokenizer_pool_size` | 0 | Tokenizer pool size |
| `tokenizer_pool_type` | none | Tokenizer pool type |
| `enable_chunked_prefill` | auto | Enable chunked prefill |
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
| `attention_backend` | auto | Attention backend (flash_attn, flashinfer) |

### Memory Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
| `swap_space` | — | retired; the live CPU-KV-offload knob pair is in [[RAM CPU Offloading in VLLM]] |
| `kv_cache_dtype` | auto | KV cache data type (auto, fp8, fp8_e5m2) |
| `max_model_len` | auto | Maximum context length |
| `num_gpu_blocks` | auto | Number of GPU blocks |
| `num_cpu_blocks` | auto | Number of CPU blocks |

### Device Arguments

| Argument | Default | Description |
|----------|---------|-------------|
| `device` | auto | Device type (cuda, rocm, xpu, cpu, metal) |
| `distributed_executor_backend` | auto | Distributed executor backend |
| `huggingface_tokenizer_kwargs` | null | Hugging Face tokenizer kwargs |

## Architecture

vLLM uses a multi-process architecture:

- API server process handles HTTP requests, tokenization, and input processing
- Engine core process runs the scheduler and coordinates GPU workers
- One GPU worker process per GPU executes model forward passes

vLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.

vLLM supports multiple attention backends:

- FlashAttention for NVIDIA GPUs
- FlashInfer for NVIDIA GPUs
- Triton attention for AMD ROCm
- ROCm attention for AMD ROCm

## Supported Models

vLLM supports over 200 model architectures including:

- Decoder-only LLMs (Llama, Qwen, Gemma)
- Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
- Hybrid attention and state-space models (Mamba, Qwen3.5)
- Multi-modal models (LLaVA, Qwen-VL, Pixtral)
- Embedding and retrieval models (E5-Mistral, GTE, ColBERT)
- Reward and classification models (Qwen-Math)

## Quantization

vLLM supports these quantization methods:

- FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
- MXFP8/MXFP4 for Hopper GPUs
- NVFP4 for Hopper and Blackwell GPUs
- INT8 for most GPUs
- INT4 for most GPUs
- GPTQ for most GPUs
- AWQ for Turing and newer GPUs (NVIDIA; native support varies on ROCm)
- GGUF for CPU and GPU
- Compressed-tensors for custom quantization
- ModelOpt for NVIDIA Model Optimizer
- TorchAO for PyTorch AO

AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not native tensor cores. AWQ and GPTQ support in vLLM on AMD ROCm is limited; use GGUF or AMD Quark.

## Parallelism

vLLM supports multiple parallelism strategies:

- Tensor Parallelism (TP) shards model parameters across GPUs
- Pipeline Parallelism (PP) distributes model layers across GPUs
- Data Parallelism (DP) replicates the model across GPU sets
- Expert Parallelism (EP) distributes expert networks in MoE models
- Context Parallelism (CP) splits context across GPUs


## Chunked Prefill — the flag surface, and what it actually does

**YES, the runtime exposes chunked-prefill options, and the running shape
already uses them.** The tcclaviger fork is a vLLM fork, so the standard
scheduler flags are present in the shipped image (verified by reading the
`vllm.engine.arg_utils` and `vllm.config.scheduler` source in the live
container `frank-archon-qwen`, not from a stub):

| Flag | Default | What it controls |
|---|---|---|
| `--enable-chunked-prefill` | `True` in `SchedulerConfig`; the CLI arg is an auto-flag (`"default": None`) | whether prefill may be split into chunks |
| `--max-num-batched-tokens` (MNBT) | 2048 (`DEFAULT_MAX_NUM_BATCHED_TOKENS`) | tokens processed per prefill iteration |
| `--prefill-schedule-interval` | 0 | gap between prefill iterations |

The running shape's argv carries `--max-num-batched-tokens 2048` alongside
`--max-model-len 200704`, so chunked prefill is **ON and already active**: a
200704-token prompt is prefilled in ~98 iterations of 2048 tokens, not in one
shot.

**It is a TTFT lever, not a throughput lever.** MEASURED 2026-09-29 on the
ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):

- MNBT 8192 vs the default 2048: **neutral** on decode at every depth.
- Small values re-chunk prefill only. They hurt TTFT (each chunk is its own
  iteration, so the first token waits on more iterations) and they never touch
  decode: a decode step is `max_num_seqs x (spec+1)` tokens, independent of
  MNBT.

**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-zealot
was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
counterpart here, and the two backends should not be reasoned about by analogy.

**MNBT is not a lever on this shape either.** The deployed 2048 is the fork's
default, so a change is a deliberate departure from the shipped value, not a
tune. The one thing MNBT does cost: every change re-enters the compile RANGE
path, so `FRANK_ARCHON_CACHE_DIR` pays one recompilation.

## See Also

- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]