docs: fleet topology update — PP split renamed frank-default-large, NPU endpoint promoted to frank-default-small, 4-model NPU bakeoff matrix (2026-08-16)

2026-08-16 14:22:13 -07:00
parent 3f42210922
commit d0ade28f6d
4 changed files with 48 additions and 19 deletions
@@ -161,7 +161,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
 - Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
 - `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
 - `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-pp-head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
+- The `llama-server` (frank-default-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.

 ## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)

@@ -177,6 +177,30 @@ Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_con

 Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.

+## Model Bakeoff (measured 2026-08-16, 4 models)
+
+All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-default-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
+
+| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
+|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
+| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 |
+| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 |
+| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** |
+| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 |
+
+Notes:
+
+- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
+- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
+- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
+- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.
+
+## Promotion (2026-08-16)
+
+`frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
+
+The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`).
+
 ## Parallel Streams (measured 2026-08-16)

 The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
@@ -33,9 +33,9 @@ llama-server b10433                      ggml-rpc-server b10433
 |------|---------|---------|
 | `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
 | `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
-| `scripts/frank-pp-head.env` | claw | head tunables (split, ctx, spec, KV, I/O) |
+| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
 | `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
-| `scripts/frank-pp-head.service` / `frank-pp-worker.service` | both | optional systemd units (manual install) |
+| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
 | `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
 | `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |

@@ -89,7 +89,7 @@ FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
 FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
 ```

-Config variants: edit `scripts/frank-pp-head.env` on the claw (git commit +
+Config variants: edit `scripts/frank-default-large.env` on the claw (git commit +
 pull if the bench runs from the repo), then re-run `start` + `bench`:

 - No spec baseline: `SPEC_NONE=1`
@@ -8,24 +8,28 @@ Documentation for running vLLM inference on two local machines.
 graph LR
    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
        A[RTX 3090 24GB + 62GB RAM]
-        B[llama.cpp ngram-mod
-frank-default-small]
-        C[Qwen3.6-35B-A3B uncensored Q6_K_P
-MoE experts on CPU, port 7770]
-        A --> B --> C
+        B[ggml-rpc worker
+frank-pp-worker, port 50052]
+        A --> B
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
-        E[llama.cpp --spec-default ngram-mod
-frank-default-large]
+        E[llama.cpp PP head
+frank-default-large, port 7770]
        F[Qwen3.8-27B UD-Q8_K_XL
-Vulkan0 backend, port 7770, PARALLEL=2]
+split 0.40/0.60 to worker, MTP+ngram-mod]
        D --> E --> F
+        G[FastFlowLM NPU XDNA2 50 TOPS
+frank-default-small, port 52625]
+        H[gemma4-it:e4b
+12.2 t/s, 32k ctx, vision]
+        D --> G --> H
    end

-    G[LLM Clients / API] -->|HTTP :7770| B
-    G -->|HTTP :7770| E
+    I[LLM Clients / API] -->|HTTP :7770| E
+    I -->|HTTP :52625| G
+    E -->|RPC layers 0.40| B
 ```

 ## Servers