docs: fact-check knowledge base against live machines and vLLM source

Correct stale and incorrect facts after exhaustive verification:
- Dankclaw: NUMA node 0 = 31213 MB; disk 661G/1.1T/38%; vLLM 0.26.0 in
  /home/dank/vllm-rocm-venv (not vllm-env); clarify 96 GiB GPU-visible is a
  driver overcommit ceiling over 30 GiB physical (confirmed via sysfs
  mem_info_vram_total=103079215104)
- Dankserver: disk 729G/140G free/84%; /mnt/entropy 910G free; active service
  is vLLM EngineCore (22 GB), not llama-server; CPU model Ryzen 7 7700X
- index.md: Dankclaw 30GiB phys / 96GiB GPU-visible (was '61 GB unified');
  add deployment topology mermaid diagram
- vLLM defaults (from source): gpu_memory_utilization=0.92, max_num_seqs=128,
  swap_space removed (now --kv-offloading-size) across Overview/Index/RAM/Hybrid
- Quantization: FP8 not native on Ampere (RTX 3090) or RDNA 3.5 (gfx1151);
  add per-model hardware quantization notes; AWQ/GPTQ limited on ROCm
- Speculative decoding: add DFlash, DSpark, MTP, EAGLE3 to feature list
- Confirm Qwen3.6-27B and Qwen3.6-35B-A3B specs are correct (HF-verified)
- Production note: Dankclaw primary llama.cpp service runs ROCmFPX
  Ling-3.0-flash RocmFP4 MTP, not vLLM Qwen3.6-27B
This commit is contained in:
Frank committed 2026-08-10 20:59:33 -07:00
1 parent 841d872aa7
commit 3a64c8f3dd
10 files changed
+86 -49

No files matched your search

+7 -6
View File
@@ -4,6 +4,8 @@
VLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. VLLM does not offload model weights to CPU. Instead, VLLM moves KV cache data between GPU and CPU.
> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.
## How CPU Offloading Works
VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.
@@ -26,19 +28,19 @@ This parameter controls the fraction of GPU memory used for KV cache.
|-------|-------------|
| 0.7 | Uses 70% of GPU memory for KV cache |
| 0.8 | Uses 80% of GPU memory for KV cache |
| 0.9 | Uses 90% of GPU memory for KV cache (default) |
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
| 0.95 | Uses 95% of GPU memory for KV cache |
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
### swap_space
### swap_space (removed) / kv-offloading-size (current)
This parameter sets the CPU swap space in GiB.
The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference.
| Value | Description |
|-------|-------------|
| 0 | No CPU swap space |
| 4 | 4 GiB swap space (default) |
| 4 | 4 GiB swap space (historical default) |
| 8 | 8 GiB swap space |
| 16 | 16 GiB swap space |
| 32 | 32 GiB swap space |
@@ -78,8 +80,7 @@ This parameter sets the maximum number of sequences per batch.
| Value | Description |
|-------|-------------|
| 128 | Default for online serving |
| 256 | Default for offline inference |
| 512 | Large batch |
| 256 | Large batch |
### kv_cache_dtype