--- type: reference description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing." tags: - llm timestamp: 2026-08-28 --- # Hybrid CPU GPU Processing Dankserver ## Overview Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]). ## Architecture ### Model Weights In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2). ### KV Cache KV cache moves between GPU and CPU: 1. Active KV cache pages stay in GPU VRAM 2. Inactive pages swap to CPU RAM 3. Swapped pages write to disk swap space if RAM is full 4. Pages reload to GPU when needed ### Compute Distribution - GPU handles all matrix multiplication and attention computation - CPU handles tokenization, detokenization, and input processing - CPU handles KV cache swap operations - GPU worker processes run on CPU cores via multiprocessing ## Optimal Settings for Dankserver ### GPU Memory Utilization Set `--gpu-memory-utilization` based on your GPU: | GPU | Recommended Value | |-----|-------------------| | 24 GB (RTX 3090/4090) | 0.7-0.8 | | 40 GB (A100) | 0.8-0.9 | | 80 GB (A100 80GB) | 0.85-0.95 | | 80 GB (H100) | 0.9-0.95 | Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s. The live engine does not use this knob at all — it claims an explicit per-card pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`, `scripts/frank-archon.env:230-231`). Lower values leave more GPU memory for KV cache. This reduces swap frequency. ### Swap Space / KV Offload Recent vLLM releases removed the `--swap-space` engine argument — `CacheConfig` on `main` has no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference. | System RAM | Recommended Swap | |------------|------------------| | 32 GB | 8-16 GB | | 64 GB | 16-32 GB | | 128 GB | 32-64 GB | | 256 GB | 64-128 GB | More swap space allows longer contexts and more concurrent requests. ### Tensor Parallelism Use tensor parallelism for large models: ``` vllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.85 ``` ### KV Cache Data Type Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache). ``` vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.9 ``` FP8 KV cache reduces memory usage by 50% compared to BF16. ## NUMA Optimization Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies: ``` vllm serve model \ --tensor-parallel-size 4 \ --numa-bind ``` This pins each GPU worker to its nearest NUMA node. This reduces memory access latency. For explicit NUMA binding: ``` vllm serve model \ --tensor-parallel-size 4 \ --numa-bind \ --numa-bind-nodes 0 0 1 1 ``` ## CPU Resource Requirements VLLM needs CPU cores for: - 1 API server process - 1 engine core process - N GPU worker processes (one per GPU) Minimum physical cores = 2 + N (where N is GPU count). For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read 2026-10-18): - Minimum 4 physical CPU cores by the 2 + N rule - Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off) ## Performance Tuning ### Chunked Prefill Chunked prefill is enabled by default. Tune with: ``` vllm serve model \ --max-num-batched-tokens 8192 ``` Higher values improve throughput. Lower values improve latency. ### CUDA Graphs CUDA graphs improve throughput. Disable for faster startup: ``` vllm serve model \ --enforce-eager ``` ### Compilation Cache Reuse torch.compile artifacts: ``` export VLLM_CACHE_ROOT=~/.cache/vllm vllm serve model ``` Copy cache between boots for faster startup. ### Skip Memory Profiling Skip memory profiling on repeated boots: ``` vllm serve model \ --kv-cache-memory ``` ## Monitoring Check GPU memory: ``` rocm-smi ``` Check for preemption warnings in VLLM logs. Check CPU utilization: ``` htop ``` ## Example Commands ### Single GPU, Medium Model ``` vllm serve Qwen/Qwen3.6-35B-A3B \ --gpu-memory-utilization 0.85 \ --max-num-batched-tokens 4096 \ --max-num-seqs 128 ``` ### Single GPU, FP8 Model ``` vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ --kv-cache-dtype fp8 \ --gpu-memory-utilization 0.9 \ --max-num-batched-tokens 8192 ``` ### Multi-GPU, Large Model ``` vllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --max-model-len 8192 ``` ## See Also - [[VLLM Overview]] - [[VLLM Index]] - [[RAM CPU Offloading in VLLM]] - [[Shared Memory Architecture Dankclaw]] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]]