serverAppConfigs/docs/LLM/Hybrid CPU GPU Processing Dankserver.md

Hybrid CPU GPU Processing Dankserver

Overview

Dankserver uses NVIDIA GPU with system RAM for hybrid processing. vLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.

Architecture

Model Weights

Model weights stay on GPU. vLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.

KV Cache

KV cache moves between GPU and CPU:

1. Active KV cache pages stay in GPU VRAM
2. Inactive pages swap to CPU RAM
3. Swapped pages write to disk swap space if RAM is full
4. Pages reload to GPU when needed

Compute Distribution

• GPU handles all matrix multiplication and attention computation
• CPU handles tokenization, detokenization, and input processing
• CPU handles KV cache swap operations
• GPU worker processes run on CPU cores via multiprocessing

Optimal Settings for Dankserver

GPU Memory Utilization

Set --gpu-memory-utilization based on your GPU:

GPU	Recommended Value
24 GB (RTX 3090/4090)	0.7-0.8
40 GB (A100)	0.8-0.9
80 GB (A100 80GB)	0.85-0.95
80 GB (H100)	0.9-0.95

Lower values leave more GPU memory for KV cache. This reduces swap frequency.

Swap Space / KV Offload

Recent vLLM releases removed the --swap-space engine argument. Use --kv-offloading-size to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical --swap-space guidance as a sizing reference.

System RAM	Recommended Swap
32 GB	8-16 GB
64 GB	16-32 GB
128 GB	32-64 GB
256 GB	64-128 GB

More swap space allows longer contexts and more concurrent requests.

Tensor Parallelism

Use tensor parallelism for large models:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.85 \
    --swap-space 16

KV Cache Data Type

Use FP8 KV cache for Ampere and newer GPUs:

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.9 \
    --swap-space 16

FP8 KV cache reduces memory usage by 50% compared to BF16.

NUMA Optimization

Dankserver may have multiple NUMA nodes. Optimize with:

vllm serve model \
    --tensor-parallel-size 4 \
    --numa-bind

This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.

For explicit NUMA binding:

vllm serve model \
    --tensor-parallel-size 4 \
    --numa-bind \
    --numa-bind-nodes 0 0 1 1

CPU Resource Requirements

VLLM needs CPU cores for:

• 1 API server process
• 1 engine core process
• N GPU worker processes (one per GPU)

Minimum physical cores = 2 + N (where N is GPU count).

For Dankserver with 1 GPU:

• Minimum 3 physical CPU cores
• Recommended 8+ physical CPU cores

Performance Tuning

Chunked Prefill

Chunked prefill is enabled by default. Tune with:

vllm serve model \
    --max-num-batched-tokens 8192

Higher values improve throughput. Lower values improve latency.

CUDA Graphs

CUDA graphs improve throughput. Disable for faster startup:

vllm serve model \
    --enforce-eager

Compilation Cache

Reuse torch.compile artifacts:

export VLLM_CACHE_ROOT=~/.cache/vllm
vllm serve model

Copy cache between boots for faster startup.

Skip Memory Profiling

Skip memory profiling on repeated boots:

vllm serve model \
    --kv-cache-memory <value_from_first_boot_log>

Monitoring

Check GPU memory:

nvidia-smi

Check for preemption warnings in VLLM logs.

Check CPU utilization:

htop

Example Commands

Single GPU, Medium Model

vllm serve Qwen/Qwen3.6-35B-A3B \
    --gpu-memory-utilization 0.85 \
    --swap-space 16 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 128

Single GPU, FP8 Model

vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.9 \
    --swap-space 16 \
    --max-num-batched-tokens 8192

Multi-GPU, Large Model

vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.9 \
    --swap-space 32 \
    --max-model-len 8192

See Also

• VLLM Overview
• VLLM Index
• RAM CPU Offloading in VLLM
• Shared Memory Architecture Dankclaw
• Running Qwen3.6 35B A3B FP8 on Dankserver