docs(NPU page): era-tag host-RAM and PP-rate restatements

One fact, one owner (de-dup campaign C4):

- `FastFlowLM NPU Inference.md` carried the host RAM figures three times
  (29 GiB budget input kept as era data; the 2026-10-18 `free -g` re-reads
  of 121/246 GiB cut -> [[Dankclaw System Specs]]/[[Dankserver System Specs]]).
- The 14-18 t/s PP-split decode is this page's own solo-held era measurement:
  kept, tagged to the 2026-08 era with [[frank-redo-2026-08-17]].
- Promotion section: `frank-zealot` "now runs the NPU endpoint" era-tagged
  (the name has been reused twice since; the NPU unit is `frank-npu`, masked).
- Repaired the Test Runbook splice introduced by the range edit above.

Refs: vault de-dup campaign, cluster C4.
This commit is contained in:
Frank committed 2026-10-19 01:58:03 -07:00
1 parent a518fc7c22
commit e32df1b04f
2 files changed
+46 -17

No files matched your search

@@ -11,9 +11,19 @@ timestamp: 2026-08-28
## Overview
vLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. vLLM does not offload model weights to CPU. Instead, vLLM moves KV cache data between GPU and CPU.
Stock vLLM offloads **KV pages**, not weights: the scheduler moves KV cache between GPU
and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the
tcclaviger/R4D fork ran a real **weight** offload (`--enable-expert-offload`, tiered
expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a
246 GiB box: [[RDNA4 MoE Expert Offload VRAM Budget]] and the tcclaviger era row of
[[frank-archon]]. First-hand in-repo evidence outranks a generic upstream claim about our
own hosts ([[Model Swap and Rollback]] §10).
> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.
> **Verified 2026-10-19 against `vllm/config/cache.py` on `main`:** there is no
> `swap_space` field — `--swap-space` is gone from current engine args. CPU KV offload is
> `--kv-offloading-size`, and it activates **only** together with `--kv-offloading-backend`
> (default `native`); both fields live in that one file. This page is the fleet's owner of
> that knob contract — other pages link here instead of restating it.
## How CPU Offloading Works
@@ -42,9 +52,12 @@ This parameter controls the fraction of GPU memory used for KV cache.
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
### swap_space (removed) / kv-offloading-size (current)
### swap_space (removed) / kv-offloading-size + kv-offloading-backend (current)
The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference.
`--swap-space` no longer exists in current engine args (verified above); the live pair is
`--kv-offloading-size` with `--kv-offloading-backend`, and offload stays off until the size
is set. The table below is the historical `--swap-space` sizing guidance, kept as a sizing
reference only.
| Value | Description |
|-------|-------------|
@@ -103,6 +116,9 @@ This parameter sets the KV cache data type.
## Memory Configuration Examples
All three examples carry the retired `--swap-space` syntax and are kept as sizing
references; the live knob pair is named above.
### Small Model on Single GPU
```
@@ -157,7 +173,7 @@ WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE
If you see frequent preemptions:
1. Increase `gpu_memory_utilization`
2. Increase `swap_space`
2. Raise the KV offload budget (`--kv-offloading-size`, with `--kv-offloading-backend`)
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage
@@ -169,7 +185,8 @@ If you see frequent preemptions:
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Use `--kv-cache-memory` to skip memory profiling
- Set `--kv-cache-memory-bytes` to pin the KV budget instead of letting the profiler guess
(the flag our own `scripts/patches/gpu_worker-kv-cap.patch` reads)
- Limit multi-modal inputs with `limit_mm_per_prompt`
## Environment Variables