dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(NPU page): era-tag host-RAM and PP-rate restatements

Browse Source 
One fact, one owner (de-dup campaign C4):

- `FastFlowLM NPU Inference.md` carried the host RAM figures three times
  (29 GiB budget input kept as era data; the 2026-10-18 `free -g` re-reads
  of 121/246 GiB cut -> [[Dankclaw System Specs]]/[[Dankserver System Specs]]).
- The 14-18 t/s PP-split decode is this page's own solo-held era measurement:
  kept, tagged to the 2026-08 era with [[frank-redo-2026-08-17]].
- Promotion section: `frank-zealot` "now runs the NPU endpoint" era-tagged
  (the name has been reused twice since; the NPU unit is `frank-npu`, masked).
- Repaired the Test Runbook splice introduced by the range edit above.

Refs: vault de-dup campaign, cluster C4.
This commit is contained in:
Frank
2026-10-19 01:58:03 -07:00
parent a518fc7c22
commit e32df1b04f
2 changed files with 46 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
+23 -11
View File
@@ -57,7 +57,10 @@ Key properties, per the project README and docs:
## NPU Hardware on Dankclaw
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS (vendor spec: "NPU TOPS — Up to 50 TOPS", [amd.com product page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)). It sits beside the Radeon 8060S iGPU and shares the same unified memory pool — 30 GiB exposed to the OS in the 2026-08 reading below; the box presents **121 GiB** today (`free -g` total 121, read 2026-10-18).
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU
with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory
via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at
4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]].
Live verification on Dankclaw, 2026-08-16:
@@ -89,10 +92,9 @@ Budget math for this fleet:
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
2026-10-18 re-read: Dankclaw's OS now reports **121 GiB** (`free -g`) and
Dankserver **246 GiB** (`free -g`), so the half-of-RAM rule would price the NPU
budget at ~60 GiB and the 7700X box still has no NPU. The 30 GiB / 62 GB rows
are the 2026-08-16 state; the table is moot while no NPU endpoint is deployed.
The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]]
and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate`
(it owns the current number).
Weight sizes (approximate, Q4-class):
@@ -159,7 +161,10 @@ Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**:
| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 |
| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 |
Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology.
- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current
PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at
14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade
the worker's model.
The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.
@@ -219,7 +224,12 @@ Notes:
## Promotion (2026-08-16)
`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused
twice after this window - see the banner - and the NPU unit today is `frank-npu`,
masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`),
wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`.
`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap
`FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
@@ -256,11 +266,13 @@ The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-repo
Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.
## Test Runbook
This runbook targets a retired endpoint; keep it as the procedure if the NPU is
ever revived. Do not run large model loads while the production service holds
most of the unified pool — 30 GiB exposed to the OS in the 2026-08 reading,
**121 GiB** today (`free -g`, read 2026-10-18). Check free RAM first, or stop
ever revived.
**Host RAM availability.** The gate is half of system RAM by default - a relic of the
install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's
box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM
first, or stop
the service the way the bench policy requires (`sudo systemctl stop
frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
otherwise restart it). The NPU shares the unified pool with everything else on
serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md
+23 -6
View File
@@ -11,9 +11,19 @@ timestamp: 2026-08-28
## Overview
vLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. vLLM does not offload model weights to CPU. Instead, vLLM moves KV cache data between GPU and CPU.
Stock vLLM offloads **KV pages**, not weights: the scheduler moves KV cache between GPU
and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the
tcclaviger/R4D fork ran a real **weight** offload (`--enable-expert-offload`, tiered
expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a
246 GiB box: [[RDNA4 MoE Expert Offload VRAM Budget]] and the tcclaviger era row of
[[frank-archon]]. First-hand in-repo evidence outranks a generic upstream claim about our
own hosts ([[Model Swap and Rollback]] §10).
> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.
> **Verified 2026-10-19 against `vllm/config/cache.py` on `main`:** there is no
> `swap_space` field — `--swap-space` is gone from current engine args. CPU KV offload is
> `--kv-offloading-size`, and it activates **only** together with `--kv-offloading-backend`
> (default `native`); both fields live in that one file. This page is the fleet's owner of
> that knob contract — other pages link here instead of restating it.
## How CPU Offloading Works
@@ -42,9 +52,12 @@ This parameter controls the fraction of GPU memory used for KV cache.
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
### swap_space (removed) / kv-offloading-size (current)
### swap_space (removed) / kv-offloading-size + kv-offloading-backend (current)
The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference.
`--swap-space` no longer exists in current engine args (verified above); the live pair is
`--kv-offloading-size` with `--kv-offloading-backend`, and offload stays off until the size
is set. The table below is the historical `--swap-space` sizing guidance, kept as a sizing
reference only.
| Value | Description |
|-------|-------------|
@@ -103,6 +116,9 @@ This parameter sets the KV cache data type.
## Memory Configuration Examples
All three examples carry the retired `--swap-space` syntax and are kept as sizing
references; the live knob pair is named above.
### Small Model on Single GPU
```
@@ -157,7 +173,7 @@ WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE
If you see frequent preemptions:
1. Increase `gpu_memory_utilization`
2. Increase `swap_space`
2. Raise the KV offload budget (`--kv-offloading-size`, with `--kv-offloading-backend`)
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage
@@ -169,7 +185,8 @@ If you see frequent preemptions:
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Use `--kv-cache-memory` to skip memory profiling
- Set `--kv-cache-memory-bytes` to pin the KV budget instead of letting the profiler guess
(the flag our own `scripts/patches/gpu_worker-kv-cap.patch` reads)
- Limit multi-modal inputs with `limit_mm_per_prompt`
## Environment Variables