serverAppConfigs/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md

type, description, tags, timestamp

type	description	tags	timestamp
reference	vLLM uses CPU RAM and swap space to extend GPU memory capacity.	llm vllm	2026-08-28

RAM CPU Offloading in vLLM

Overview

Stock vLLM offloads KV pages, not weights: the scheduler moves KV cache between GPU and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the tcclaviger/R4D fork ran a real weight offload (--enable-expert-offload, tiered expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a 246 GiB box: RDNA4 MoE Expert Offload VRAM Budget and the tcclaviger era row of frank-archon. First-hand in-repo evidence outranks a generic upstream claim about our own hosts (Model Swap and Rollback §10).

Verified 2026-10-19 against vllm/config/cache.py on main: there is no swap_space field — --swap-space is gone from current engine args. CPU KV offload is --kv-offloading-size, and it activates only together with --kv-offloading-backend (default native); both fields live in that one file. This page is the fleet's owner of that knob contract — other pages link here instead of restating it.

How CPU Offloading Works

VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.

The KV cache swap mechanism works as follows:

1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space

Key Parameters

gpu_memory_utilization

This parameter controls the fraction of GPU memory used for KV cache.

Value	Description
0.7	Uses 70% of GPU memory for KV cache
0.8	Uses 80% of GPU memory for KV cache
0.92	Uses 92% of GPU memory for KV cache (default)
0.95	Uses 95% of GPU memory for KV cache

Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.

swap_space (removed) / kv-offloading-size + kv-offloading-backend (current)

--swap-space no longer exists in current engine args (verified above); the live pair is --kv-offloading-size with --kv-offloading-backend, and offload stays off until the size is set. The table below is the historical --swap-space sizing guidance, kept as a sizing reference only.

Value	Description
0	No CPU swap space
4	4 GiB swap space (historical default)
8	8 GiB swap space
16	16 GiB swap space
32	32 GiB swap space
64	64 GiB swap space

The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.

max_model_len

This parameter sets the maximum context length.

Value	Description
auto	VLLM calculates optimal length
2048	2048 tokens
4096	4096 tokens
8192	8192 tokens
32768	32768 tokens

Smaller values reduce memory usage but limit context length.

max_num_batched_tokens

This parameter sets the maximum number of tokens per batch.

Value	Description
2048	Default for online serving
4096	Moderate batch size
8192	Large batch size
16384	Very large batch size

max_num_seqs

This parameter sets the maximum number of sequences per batch.

Value	Description
128	Default for online serving
256	Large batch

kv_cache_dtype

This parameter sets the KV cache data type.

Value	Description
auto	Uses model default dtype
fp8	Uses FP8 for KV cache (reduces memory by 50%)
fp8_e5m2	Uses FP8 E5M2 format for KV cache

Memory Configuration Examples

All three examples carry the retired --swap-space syntax and are kept as sizing references; the live knob pair is named above.

Small Model on Single GPU

vllm serve Qwen/Qwen2.5-1.5B-Instruct \
    --gpu-memory-utilization 0.9 \
    --swap-space 4 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 128

Large Model with CPU Offload

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.8 \
    --swap-space 16 \
    --max-model-len 8192

Memory-Constrained GPU

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 4096 \
    --max-num-batched-tokens 2048

Preemption Modes

VLLM uses two preemption modes when GPU memory is insufficient:

Recompute Mode (Default in V1)

VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.

Swap Mode

VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.

Monitoring

Watch for preemption warnings:

WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.

If you see frequent preemptions:

1. Increase gpu_memory_utilization
2. Raise the KV offload budget (--kv-offloading-size, with --kv-offloading-backend)
3. Decrease max_num_seqs
4. Increase tensor_parallel_size
5. Increase max_model_len to match actual usage

Memory Conservation Tips

• Use quantization (FP8, INT8) to reduce model size
• Reduce max_model_len to match actual needs
• Reduce max_num_seqs to limit concurrent requests
• Disable CUDA graphs with --enforce-eager to save memory
• Set compilation_config.cudagraph_capture_sizes to smaller values
• Set --kv-cache-memory-bytes to pin the KV budget instead of letting the profiler guess (the flag our own scripts/patches/gpu_worker-kv-cap.patch reads)
• Limit multi-modal inputs with limit_mm_per_prompt

Environment Variables

Variable	Description	Default
vLLM_CPU_KVCACHE_SPACE	KV cache space in GiB for CPU backend	4
vLLM_CPU_OMP_THREADS_BIND	CPU thread binding for OpenMP	auto
vLLM_CPU_NUM_OF_RESERVED_CPU	Reserved CPU cores per rank	0-1
vLLM_CACHE_ROOT	torch.compile cache directory	~/.cache/vllm

See Also

• VLLM Overview
• VLLM Index
• Hybrid CPU GPU Processing Dankserver
• Shared Memory Architecture Dankclaw