From e32df1b04f68690bcb0174ab0eea738ed28a5cda Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 01:58:03 -0700 Subject: [PATCH] docs(NPU page): era-tag host-RAM and PP-rate restatements One fact, one owner (de-dup campaign C4): - `FastFlowLM NPU Inference.md` carried the host RAM figures three times (29 GiB budget input kept as era data; the 2026-10-18 `free -g` re-reads of 121/246 GiB cut -> [[Dankclaw System Specs]]/[[Dankserver System Specs]]). - The 14-18 t/s PP-split decode is this page's own solo-held era measurement: kept, tagged to the 2026-08 era with [[frank-redo-2026-08-17]]. - Promotion section: `frank-zealot` "now runs the NPU endpoint" era-tagged (the name has been reused twice since; the NPU unit is `frank-npu`, masked). - Repaired the Test Runbook splice introduced by the range edit above. Refs: vault de-dup campaign, cluster C4. --- .../LLM/FastFlowLM NPU Inference.md | 34 +++++++++++++------ .../LLM/RAM CPU Offloading in VLLM.md | 29 ++++++++++++---- 2 files changed, 46 insertions(+), 17 deletions(-) diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md index b0bb6c0..df516ab 100644 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md @@ -57,7 +57,10 @@ Key properties, per the project README and docs: ## NPU Hardware on Dankclaw -Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS (vendor spec: "NPU TOPS — Up to 50 TOPS", [amd.com product page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)). It sits beside the Radeon 8060S iGPU and shares the same unified memory pool — 30 GiB exposed to the OS in the 2026-08 reading below; the box presents **121 GiB** today (`free -g` total 121, read 2026-10-18). +Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU +with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory +via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at +4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]]. Live verification on Dankclaw, 2026-08-16: @@ -89,10 +92,9 @@ Budget math for this fleet: | Dankserver (Ryzen 7 7700X) | 62 GB | no NPU | | FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB | -2026-10-18 re-read: Dankclaw's OS now reports **121 GiB** (`free -g`) and -Dankserver **246 GiB** (`free -g`), so the half-of-RAM rule would price the NPU -budget at ~60 GiB and the 7700X box still has no NPU. The 30 GiB / 62 GB rows -are the 2026-08-16 state; the table is moot while no NPU endpoint is deployed. +The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]] +and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate` +(it owns the current number). Weight sizes (approximate, Q4-class): @@ -159,7 +161,10 @@ Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**: | Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 | | Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 | -Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology. +- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current + PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at + 14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade + the worker's model. The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it. @@ -219,7 +224,12 @@ Notes: ## Promotion (2026-08-16) -`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch. +`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused +twice after this window - see the banner - and the NPU unit today is `frank-npu`, +masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), +wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. +`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap +`FLM_MODEL` in the env to switch. ### Boot policy (2026-08-17) @@ -256,11 +266,13 @@ The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-repo Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots. ## Test Runbook - This runbook targets a retired endpoint; keep it as the procedure if the NPU is -ever revived. Do not run large model loads while the production service holds -most of the unified pool — 30 GiB exposed to the OS in the 2026-08 reading, -**121 GiB** today (`free -g`, read 2026-10-18). Check free RAM first, or stop +ever revived. + +**Host RAM availability.** The gate is half of system RAM by default - a relic of the +install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's +box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM +first, or stop the service the way the bench policy requires (`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would otherwise restart it). The NPU shares the unified pool with everything else on diff --git a/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md b/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md index e4f9482..d5df203 100644 --- a/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md +++ b/serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md @@ -11,9 +11,19 @@ timestamp: 2026-08-28 ## Overview -vLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. vLLM does not offload model weights to CPU. Instead, vLLM moves KV cache data between GPU and CPU. +Stock vLLM offloads **KV pages**, not weights: the scheduler moves KV cache between GPU +and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the +tcclaviger/R4D fork ran a real **weight** offload (`--enable-expert-offload`, tiered +expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a +246 GiB box: [[RDNA4 MoE Expert Offload VRAM Budget]] and the tcclaviger era row of +[[frank-archon]]. First-hand in-repo evidence outranks a generic upstream claim about our +own hosts ([[Model Swap and Rollback]] §10). -> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details. +> **Verified 2026-10-19 against `vllm/config/cache.py` on `main`:** there is no +> `swap_space` field — `--swap-space` is gone from current engine args. CPU KV offload is +> `--kv-offloading-size`, and it activates **only** together with `--kv-offloading-backend` +> (default `native`); both fields live in that one file. This page is the fleet's owner of +> that knob contract — other pages link here instead of restating it. ## How CPU Offloading Works @@ -42,9 +52,12 @@ This parameter controls the fraction of GPU memory used for KV cache. Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency. -### swap_space (removed) / kv-offloading-size (current) +### swap_space (removed) / kv-offloading-size + kv-offloading-backend (current) -The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference. +`--swap-space` no longer exists in current engine args (verified above); the live pair is +`--kv-offloading-size` with `--kv-offloading-backend`, and offload stays off until the size +is set. The table below is the historical `--swap-space` sizing guidance, kept as a sizing +reference only. | Value | Description | |-------|-------------| @@ -103,6 +116,9 @@ This parameter sets the KV cache data type. ## Memory Configuration Examples +All three examples carry the retired `--swap-space` syntax and are kept as sizing +references; the live knob pair is named above. + ### Small Model on Single GPU ``` @@ -157,7 +173,7 @@ WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE If you see frequent preemptions: 1. Increase `gpu_memory_utilization` -2. Increase `swap_space` +2. Raise the KV offload budget (`--kv-offloading-size`, with `--kv-offloading-backend`) 3. Decrease `max_num_seqs` 4. Increase `tensor_parallel_size` 5. Increase `max_model_len` to match actual usage @@ -169,7 +185,8 @@ If you see frequent preemptions: - Reduce `max_num_seqs` to limit concurrent requests - Disable CUDA graphs with `--enforce-eager` to save memory - Set `compilation_config.cudagraph_capture_sizes` to smaller values -- Use `--kv-cache-memory` to skip memory profiling +- Set `--kv-cache-memory-bytes` to pin the KV budget instead of letting the profiler guess + (the flag our own `scripts/patches/gpu_worker-kv-cap.patch` reads) - Limit multi-modal inputs with `limit_mm_per_prompt` ## Environment Variables