serverAppConfigs/docs/RAM CPU Offloading in VLLM.md

RAM CPU Offloading in VLLM

Overview

VLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. VLLM does not offload model weights to CPU. Instead, VLLM moves KV cache data between GPU and CPU.

How CPU Offloading Works

VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.

The KV cache swap mechanism works as follows:

1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space

Key Parameters

gpu_memory_utilization

This parameter controls the fraction of GPU memory used for KV cache.

Value	Description
0.7	Uses 70% of GPU memory for KV cache
0.8	Uses 80% of GPU memory for KV cache
0.9	Uses 90% of GPU memory for KV cache (default)
0.95	Uses 95% of GPU memory for KV cache

Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.

swap_space

This parameter sets the CPU swap space in GiB.

Value	Description
0	No CPU swap space
4	4 GiB swap space (default)
8	8 GiB swap space
16	16 GiB swap space
32	32 GiB swap space
64	64 GiB swap space

The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.

max_model_len

This parameter sets the maximum context length.

Value	Description
auto	VLLM calculates optimal length
2048	2048 tokens
4096	4096 tokens
8192	8192 tokens
32768	32768 tokens

Smaller values reduce memory usage but limit context length.

max_num_batched_tokens

This parameter sets the maximum number of tokens per batch.

Value	Description
2048	Default for online serving
4096	Moderate batch size
8192	Large batch size
16384	Very large batch size

max_num_seqs

This parameter sets the maximum number of sequences per batch.

Value	Description
128	Default for online serving
256	Default for offline inference
512	Large batch

kv_cache_dtype

This parameter sets the KV cache data type.

Value	Description
auto	Uses model default dtype
fp8	Uses FP8 for KV cache (reduces memory by 50%)
fp8_e5m2	Uses FP8 E5M2 format for KV cache

Memory Configuration Examples

Small Model on Single GPU

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
    --gpu-memory-utilization 0.9 \
    --swap-space 4 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 128

Large Model with CPU Offload

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.8 \
    --swap-space 16 \
    --max-model-len 8192

Memory-Constrained GPU

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 4096 \
    --max-num-batched-tokens 2048

Preemption Modes

VLLM uses two preemption modes when GPU memory is insufficient:

Recompute Mode (Default in V1)

VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.

Swap Mode

VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.

Monitoring

Watch for preemption warnings:

WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

If you see frequent preemptions:

1. Increase gpu_memory_utilization
2. Increase swap_space
3. Decrease max_num_seqs
4. Increase tensor_parallel_size
5. Increase max_model_len to match actual usage

Memory Conservation Tips

• Use quantization (FP8, INT8) to reduce model size
• Reduce max_model_len to match actual needs
• Reduce max_num_seqs to limit concurrent requests
• Disable CUDA graphs with --enforce-eager to save memory
• Set compilation_config.cudagraph_capture_sizes to smaller values
• Use --kv-cache-memory to skip memory profiling
• Limit multi-modal inputs with limit_mm_per_prompt

Environment Variables

Variable	Description	Default
VLLM_CPU_KVCACHE_SPACE	KV cache space in GiB for CPU backend	4
VLLM_CPU_OMP_THREADS_BIND	CPU thread binding for OpenMP	auto
VLLM_CPU_NUM_OF_RESERVED_CPU	Reserved CPU cores per rank	0-1
VLLM_CACHE_ROOT	torch.compile cache directory	~/.cache/vllm

See Also

• VLLM Overview
• VLLM Index
• Hybrid CPU GPU Processing Dankserver
• Shared Memory Architecture Dankclaw