Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
223 lines
6.1 KiB
Markdown
223 lines
6.1 KiB
Markdown
---
|
||
type: reference
|
||
description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing."
|
||
tags:
|
||
- llm
|
||
timestamp: 2026-08-28
|
||
---
|
||
|
||
# Hybrid CPU GPU Processing Dankserver
|
||
|
||
## Overview
|
||
|
||
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]).
|
||
|
||
## Architecture
|
||
|
||
### Model Weights
|
||
|
||
In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2).
|
||
|
||
### KV Cache
|
||
|
||
KV cache moves between GPU and CPU:
|
||
|
||
1. Active KV cache pages stay in GPU VRAM
|
||
2. Inactive pages swap to CPU RAM
|
||
3. Swapped pages write to disk swap space if RAM is full
|
||
4. Pages reload to GPU when needed
|
||
|
||
### Compute Distribution
|
||
|
||
- GPU handles all matrix multiplication and attention computation
|
||
- CPU handles tokenization, detokenization, and input processing
|
||
- CPU handles KV cache swap operations
|
||
- GPU worker processes run on CPU cores via multiprocessing
|
||
|
||
## Optimal Settings for Dankserver
|
||
|
||
### GPU Memory Utilization
|
||
|
||
Set `--gpu-memory-utilization` based on your GPU:
|
||
|
||
| GPU | Recommended Value |
|
||
|-----|-------------------|
|
||
| 24 GB (RTX 3090/4090) | 0.7-0.8 |
|
||
| 40 GB (A100) | 0.8-0.9 |
|
||
| 80 GB (A100 80GB) | 0.85-0.95 |
|
||
| 80 GB (H100) | 0.9-0.95 |
|
||
|
||
Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s.
|
||
The live engine does not use this knob at all — it claims an explicit per-card
|
||
pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`,
|
||
`scripts/frank-archon.env:230-231`).
|
||
|
||
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
|
||
|
||
### Swap Space / KV Offload
|
||
|
||
Recent vLLM releases removed the `--swap-space` engine argument — `CacheConfig` on `main` has no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||
|
||
| System RAM | Recommended Swap |
|
||
|------------|------------------|
|
||
| 32 GB | 8-16 GB |
|
||
| 64 GB | 16-32 GB |
|
||
| 128 GB | 32-64 GB |
|
||
| 256 GB | 64-128 GB |
|
||
|
||
More swap space allows longer contexts and more concurrent requests.
|
||
|
||
### Tensor Parallelism
|
||
|
||
Use tensor parallelism for large models:
|
||
|
||
```
|
||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||
--tensor-parallel-size 4 \
|
||
--gpu-memory-utilization 0.85
|
||
```
|
||
|
||
### KV Cache Data Type
|
||
|
||
Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache).
|
||
|
||
```
|
||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||
--kv-cache-dtype fp8 \
|
||
--gpu-memory-utilization 0.9
|
||
```
|
||
|
||
FP8 KV cache reduces memory usage by 50% compared to BF16.
|
||
|
||
## NUMA Optimization
|
||
|
||
Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies:
|
||
|
||
```
|
||
vllm serve model \
|
||
--tensor-parallel-size 4 \
|
||
--numa-bind
|
||
```
|
||
|
||
This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.
|
||
|
||
For explicit NUMA binding:
|
||
|
||
```
|
||
vllm serve model \
|
||
--tensor-parallel-size 4 \
|
||
--numa-bind \
|
||
--numa-bind-nodes 0 0 1 1
|
||
```
|
||
|
||
## CPU Resource Requirements
|
||
|
||
VLLM needs CPU cores for:
|
||
|
||
- 1 API server process
|
||
- 1 engine core process
|
||
- N GPU worker processes (one per GPU)
|
||
|
||
Minimum physical cores = 2 + N (where N is GPU count).
|
||
|
||
For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read
|
||
2026-10-18):
|
||
|
||
- Minimum 4 physical CPU cores by the 2 + N rule
|
||
- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off)
|
||
|
||
## Performance Tuning
|
||
|
||
### Chunked Prefill
|
||
|
||
Chunked prefill is enabled by default. Tune with:
|
||
|
||
```
|
||
vllm serve model \
|
||
--max-num-batched-tokens 8192
|
||
```
|
||
|
||
Higher values improve throughput. Lower values improve latency.
|
||
|
||
### CUDA Graphs
|
||
|
||
CUDA graphs improve throughput. Disable for faster startup:
|
||
|
||
```
|
||
vllm serve model \
|
||
--enforce-eager
|
||
```
|
||
|
||
### Compilation Cache
|
||
|
||
Reuse torch.compile artifacts:
|
||
|
||
```
|
||
export VLLM_CACHE_ROOT=~/.cache/vllm
|
||
vllm serve model
|
||
```
|
||
|
||
Copy cache between boots for faster startup.
|
||
|
||
### Skip Memory Profiling
|
||
|
||
Skip memory profiling on repeated boots:
|
||
|
||
```
|
||
vllm serve model \
|
||
--kv-cache-memory <value_from_first_boot_log>
|
||
```
|
||
|
||
## Monitoring
|
||
|
||
Check GPU memory:
|
||
|
||
```
|
||
rocm-smi
|
||
```
|
||
|
||
Check for preemption warnings in VLLM logs.
|
||
|
||
Check CPU utilization:
|
||
|
||
```
|
||
htop
|
||
```
|
||
|
||
## Example Commands
|
||
|
||
### Single GPU, Medium Model
|
||
|
||
```
|
||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||
--gpu-memory-utilization 0.85 \
|
||
--max-num-batched-tokens 4096 \
|
||
--max-num-seqs 128
|
||
```
|
||
|
||
### Single GPU, FP8 Model
|
||
|
||
```
|
||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||
--kv-cache-dtype fp8 \
|
||
--gpu-memory-utilization 0.9 \
|
||
--max-num-batched-tokens 8192
|
||
```
|
||
|
||
### Multi-GPU, Large Model
|
||
|
||
```
|
||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||
--tensor-parallel-size 4 \
|
||
--gpu-memory-utilization 0.9 \
|
||
--max-model-len 8192
|
||
```
|
||
|
||
## See Also
|
||
|
||
- [[VLLM Overview]]
|
||
- [[VLLM Index]]
|
||
- [[RAM CPU Offloading in VLLM]]
|
||
- [[Shared Memory Architecture Dankclaw]]
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|