From d0ade28f6d94bd1831ecc3b2f64610da3433e646 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 16 Aug 2026 14:22:13 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20fleet=20topology=20update=20=E2=80=94?= =?UTF-8?q?=20PP=20split=20renamed=20frank-default-large,=20NPU=20endpoint?= =?UTF-8?q?=20promoted=20to=20frank-default-small,=204-model=20NPU=20bakeo?= =?UTF-8?q?ff=20matrix=20(2026-08-16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AGENTS.md | 11 ++++---- docs/FastFlowLM NPU Inference.md | 26 ++++++++++++++++++- .../Qwen3.8 PP Deployment.md | 6 ++--- docs/index.md | 24 ++++++++++------- 4 files changed, 48 insertions(+), 19 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6008176..b20ebe2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ when the bench finishes. The users will call if they need the model. Rules: 1. Stop the service before any benchmark: `systemctl stop frank-default-large` - (Dankclaw) or `systemctl stop frank-default-small` (Dankserver). Use the + (Dankclaw) or `systemctl stop frank-default-small` (Dankclaw NPU). Use the passwordless helper or the password `Assface.3241` with `sudo -S`. 2. Run the bench on a scratch port (never the service port 7770). 3. Restart the service when the bench completes: `systemctl start `. @@ -41,8 +41,8 @@ All script edits must flow through git. Do not edit files on the remote machine Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint. ``` -./scripts/deploy.sh dankclaw # vLLM (frank-default-large, port 7770) -./scripts/deploy.sh dankserver # llama.cpp (frank-default-small, port 7770) +./scripts/deploy.sh dankclaw # PP head (frank-default-large, port 7770) +./scripts/deploy.sh dankserver # PP worker (frank-pp-worker, RPC 50052) ./scripts/deploy.sh dankclaw --dry-run # preview without changes ``` @@ -61,8 +61,9 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | llama.cpp (UD-Q8_K_XL, `--spec-default` ngram-mod, Vulkan0, 2 slots) | frank-default-large | 7770 | -| Dankserver | dank@dankserver | llama.cpp (Q6_K_P + ngram-mod, MoE offload) | frank-default-small | 7770 | +| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, gemma4-it:e4b; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | +| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 | ## Runtime & Troubleshooting diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md index dd1155c..eae1876 100644 --- a/docs/FastFlowLM NPU Inference.md +++ b/docs/FastFlowLM NPU Inference.md @@ -161,7 +161,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization, - Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed. - `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`. - `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot. -- The `llama-server` (frank-pp-head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16. +- The `llama-server` (frank-default-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16. ## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU) @@ -177,6 +177,30 @@ Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_con Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested. +## Model Bakeoff (measured 2026-08-16, 4 models) + +All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-default-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom). + +| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) | +|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------| +| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 | +| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 | +| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** | +| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 | + +Notes: + +- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder. +- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load. +- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s. +- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it. + +## Promotion (2026-08-16) + +`frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch. + +The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`). + ## Parallel Streams (measured 2026-08-16) The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency. diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md index 67937cf..69b4c70 100644 --- a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -33,9 +33,9 @@ llama-server b10433 ggml-rpc-server b10433 |------|---------|---------| | `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) | | `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build | -| `scripts/frank-pp-head.env` | claw | head tunables (split, ctx, spec, KV, I/O) | +| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename | | `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") | -| `scripts/frank-pp-head.service` / `frank-pp-worker.service` | both | optional systemd units (manual install) | +| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` | | `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) | | `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` | @@ -89,7 +89,7 @@ FRANK_SUDO_PW= ./scripts/pp-test.sh stop-pp FRANK_SUDO_PW= ./scripts/pp-test.sh restore ``` -Config variants: edit `scripts/frank-pp-head.env` on the claw (git commit + +Config variants: edit `scripts/frank-default-large.env` on the claw (git commit + pull if the bench runs from the repo), then re-run `start` + `bench`: - No spec baseline: `SPEC_NONE=1` diff --git a/docs/index.md b/docs/index.md index f731be5..95511a4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,24 +8,28 @@ Documentation for running vLLM inference on two local machines. graph LR subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04] A[RTX 3090 24GB + 62GB RAM] - B[llama.cpp ngram-mod -frank-default-small] - C[Qwen3.6-35B-A3B uncensored Q6_K_P -MoE experts on CPU, port 7770] - A --> B --> C + B[ggml-rpc worker +frank-pp-worker, port 50052] + A --> B end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] - E[llama.cpp --spec-default ngram-mod -frank-default-large] + E[llama.cpp PP head +frank-default-large, port 7770] F[Qwen3.8-27B UD-Q8_K_XL -Vulkan0 backend, port 7770, PARALLEL=2] +split 0.40/0.60 to worker, MTP+ngram-mod] D --> E --> F + G[FastFlowLM NPU XDNA2 50 TOPS +frank-default-small, port 52625] + H[gemma4-it:e4b +12.2 t/s, 32k ctx, vision] + D --> G --> H end - G[LLM Clients / API] -->|HTTP :7770| B - G -->|HTTP :7770| E + I[LLM Clients / API] -->|HTTP :7770| E + I -->|HTTP :52625| G + E -->|RPC layers 0.40| B ``` ## Servers