docs(vault): fleet-wide fact re-verification pass (67 pages)
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
This commit is contained in:
1 parent
466fd477d8
commit
e7f0d93c86
67 files changed
+2619
-1152
No files matched your search
@@ -10,13 +10,13 @@ timestamp: 2026-08-28
|
||||
|
||||
## Overview
|
||||
|
||||
Dankserver uses NVIDIA GPU with system RAM for hybrid processing. vLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.
|
||||
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]).
|
||||
|
||||
## Architecture
|
||||
|
||||
### Model Weights
|
||||
|
||||
Model weights stay on GPU. vLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.
|
||||
In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2).
|
||||
|
||||
### KV Cache
|
||||
|
||||
@@ -47,11 +47,16 @@ Set `--gpu-memory-utilization` based on your GPU:
|
||||
| 80 GB (A100 80GB) | 0.85-0.95 |
|
||||
| 80 GB (H100) | 0.9-0.95 |
|
||||
|
||||
Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s.
|
||||
The live engine does not use this knob at all — it claims an explicit per-card
|
||||
pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`,
|
||||
`scripts/frank-archon.env:230-231`).
|
||||
|
||||
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
|
||||
|
||||
### Swap Space / KV Offload
|
||||
|
||||
Recent vLLM releases removed the `--swap-space` engine argument. Use `--kv-offloading-size` to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||||
Recent vLLM releases removed the `--swap-space` engine argument — `CacheConfig` on `main` has no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||||
|
||||
| System RAM | Recommended Swap |
|
||||
|------------|------------------|
|
||||
@@ -69,26 +74,24 @@ Use tensor parallelism for large models:
|
||||
```
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 16
|
||||
--gpu-memory-utilization 0.85
|
||||
```
|
||||
|
||||
### KV Cache Data Type
|
||||
|
||||
Use FP8 KV cache for Ampere and newer GPUs:
|
||||
Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache).
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 16
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
FP8 KV cache reduces memory usage by 50% compared to BF16.
|
||||
|
||||
## NUMA Optimization
|
||||
|
||||
Dankserver may have multiple NUMA nodes. Optimize with:
|
||||
Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
@@ -117,10 +120,11 @@ VLLM needs CPU cores for:
|
||||
|
||||
Minimum physical cores = 2 + N (where N is GPU count).
|
||||
|
||||
For Dankserver with 1 GPU:
|
||||
For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read
|
||||
2026-10-18):
|
||||
|
||||
- Minimum 3 physical CPU cores
|
||||
- Recommended 8+ physical CPU cores
|
||||
- Minimum 4 physical CPU cores by the 2 + N rule
|
||||
- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off)
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
@@ -169,7 +173,7 @@ vllm serve model \
|
||||
Check GPU memory:
|
||||
|
||||
```
|
||||
nvidia-smi
|
||||
rocm-smi
|
||||
```
|
||||
|
||||
Check for preemption warnings in VLLM logs.
|
||||
@@ -187,7 +191,6 @@ htop
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 16 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--max-num-seqs 128
|
||||
```
|
||||
@@ -198,7 +201,6 @@ vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 16 \
|
||||
--max-num-batched-tokens 8192
|
||||
```
|
||||
|
||||
@@ -208,7 +210,6 @@ vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 32 \
|
||||
--max-model-len 8192
|
||||
```
|
||||
|
||||
|
||||
Reference in new issue
Block a user