serverAppConfigs/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md

---
type: reference
description: "vLLM uses CPU RAM and swap space to extend GPU memory capacity."
tags:
  - llm
  - vllm
timestamp: 2026-08-28
---

# RAM CPU Offloading in vLLM

## Overview

Stock vLLM offloads **KV pages**, not weights: the scheduler moves KV cache between GPU
and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the
tcclaviger/R4D fork ran a real **weight** offload (`--enable-expert-offload`, tiered
expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a
246 GiB box: [[RDNA4 MoE Expert Offload VRAM Budget]] and the tcclaviger era row of
[[frank-archon]]. First-hand in-repo evidence outranks a generic upstream claim about our
own hosts ([[Model Swap and Rollback]] §10).

> **Verified 2026-10-19 against `vllm/config/cache.py` on `main`:** there is no
> `swap_space` field — `--swap-space` is gone from current engine args. CPU KV offload is
> `--kv-offloading-size`, and it activates **only** together with `--kv-offloading-backend`
> (default `native`); both fields live in that one file. This page is the fleet's owner of
> that knob contract — other pages link here instead of restating it.

## How CPU Offloading Works

VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.

The KV cache swap mechanism works as follows:

1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space

## Key Parameters

### gpu_memory_utilization

This parameter controls the fraction of GPU memory used for KV cache.

| Value | Description |
|-------|-------------|
| 0.7 | Uses 70% of GPU memory for KV cache |
| 0.8 | Uses 80% of GPU memory for KV cache |
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
| 0.95 | Uses 95% of GPU memory for KV cache |

Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.

### swap_space (removed) / kv-offloading-size + kv-offloading-backend (current)

`--swap-space` no longer exists in current engine args (verified above); the live pair is
`--kv-offloading-size` with `--kv-offloading-backend`, and offload stays off until the size
is set. The table below is the historical `--swap-space` sizing guidance, kept as a sizing
reference only.

| Value | Description |
|-------|-------------|
| 0 | No CPU swap space |
| 4 | 4 GiB swap space (historical default) |
| 8 | 8 GiB swap space |
| 16 | 16 GiB swap space |
| 32 | 32 GiB swap space |
| 64 | 64 GiB swap space |

The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.

### max_model_len

This parameter sets the maximum context length.

| Value | Description |
|-------|-------------|
| auto | VLLM calculates optimal length |
| 2048 | 2048 tokens |
| 4096 | 4096 tokens |
| 8192 | 8192 tokens |
| 32768 | 32768 tokens |

Smaller values reduce memory usage but limit context length.

### max_num_batched_tokens

This parameter sets the maximum number of tokens per batch.

| Value | Description |
|-------|-------------|
| 2048 | Default for online serving |
| 4096 | Moderate batch size |
| 8192 | Large batch size |
| 16384 | Very large batch size |

### max_num_seqs

This parameter sets the maximum number of sequences per batch.

| Value | Description |
|-------|-------------|
| 128 | Default for online serving |
| 256 | Large batch |

### kv_cache_dtype

This parameter sets the KV cache data type.

| Value | Description |
|-------|-------------|
| auto | Uses model default dtype |
| fp8 | Uses FP8 for KV cache (reduces memory by 50%) |
| fp8_e5m2 | Uses FP8 E5M2 format for KV cache |

## Memory Configuration Examples

All three examples carry the retired `--swap-space` syntax and are kept as sizing
references; the live knob pair is named above.

### Small Model on Single GPU

```
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
    --gpu-memory-utilization 0.9 \
    --swap-space 4 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 128
```

### Large Model with CPU Offload

```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.8 \
    --swap-space 16 \
    --max-model-len 8192
```

### Memory-Constrained GPU

```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 4096 \
    --max-num-batched-tokens 2048
```

## Preemption Modes

VLLM uses two preemption modes when GPU memory is insufficient:

### Recompute Mode (Default in V1)

VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.

### Swap Mode

VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.

## Monitoring

Watch for preemption warnings:

```
WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.
```

If you see frequent preemptions:

1. Increase `gpu_memory_utilization`
2. Raise the KV offload budget (`--kv-offloading-size`, with `--kv-offloading-backend`)
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage

## Memory Conservation Tips

- Use quantization (FP8, INT8) to reduce model size
- Reduce `max_model_len` to match actual needs
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Set `--kv-cache-memory-bytes` to pin the KV budget instead of letting the profiler guess
  (the flag our own `scripts/patches/gpu_worker-kv-cap.patch` reads)
- Limit multi-modal inputs with `limit_mm_per_prompt`

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| vLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| vLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| vLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| vLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |

## See Also

- [[VLLM Overview]]
- [[VLLM Index]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]