serverAppConfigs/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md

# RAM CPU Offloading in vLLM

## Overview

vLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. vLLM does not offload model weights to CPU. Instead, vLLM moves KV cache data between GPU and CPU.

> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.

## How CPU Offloading Works

VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.

The KV cache swap mechanism works as follows:

1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space

## Key Parameters

### gpu_memory_utilization

This parameter controls the fraction of GPU memory used for KV cache.

| Value | Description |
|-------|-------------|
| 0.7 | Uses 70% of GPU memory for KV cache |
| 0.8 | Uses 80% of GPU memory for KV cache |
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
| 0.95 | Uses 95% of GPU memory for KV cache |

Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.

### swap_space (removed) / kv-offloading-size (current)

The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference.

| Value | Description |
|-------|-------------|
| 0 | No CPU swap space |
| 4 | 4 GiB swap space (historical default) |
| 8 | 8 GiB swap space |
| 16 | 16 GiB swap space |
| 32 | 32 GiB swap space |
| 64 | 64 GiB swap space |

The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.

### max_model_len

This parameter sets the maximum context length.

| Value | Description |
|-------|-------------|
| auto | VLLM calculates optimal length |
| 2048 | 2048 tokens |
| 4096 | 4096 tokens |
| 8192 | 8192 tokens |
| 32768 | 32768 tokens |

Smaller values reduce memory usage but limit context length.

### max_num_batched_tokens

This parameter sets the maximum number of tokens per batch.

| Value | Description |
|-------|-------------|
| 2048 | Default for online serving |
| 4096 | Moderate batch size |
| 8192 | Large batch size |
| 16384 | Very large batch size |

### max_num_seqs

This parameter sets the maximum number of sequences per batch.

| Value | Description |
|-------|-------------|
| 128 | Default for online serving |
| 256 | Large batch |

### kv_cache_dtype

This parameter sets the KV cache data type.

| Value | Description |
|-------|-------------|
| auto | Uses model default dtype |
| fp8 | Uses FP8 for KV cache (reduces memory by 50%) |
| fp8_e5m2 | Uses FP8 E5M2 format for KV cache |

## Memory Configuration Examples

### Small Model on Single GPU

```
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
    --gpu-memory-utilization 0.9 \
    --swap-space 4 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 128
```

### Large Model with CPU Offload

```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.8 \
    --swap-space 16 \
    --max-model-len 8192
```

### Memory-Constrained GPU

```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.7 \
    --swap-space 32 \
    --max-model-len 4096 \
    --max-num-batched-tokens 2048
```

## Preemption Modes

VLLM uses two preemption modes when GPU memory is insufficient:

### Recompute Mode (Default in V1)

VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.

### Swap Mode

VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.

## Monitoring

Watch for preemption warnings:

```
WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.
```

If you see frequent preemptions:

1. Increase `gpu_memory_utilization`
2. Increase `swap_space`
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage

## Memory Conservation Tips

- Use quantization (FP8, INT8) to reduce model size
- Reduce `max_model_len` to match actual needs
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Use `--kv-cache-memory` to skip memory profiling
- Limit multi-modal inputs with `limit_mm_per_prompt`

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| vLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| vLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| vLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| vLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |

## See Also

- [[VLLM Overview]]
- [[VLLM Index]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]