PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)

- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py - head design verified in b10433 source + live 4B smoke test: * no --device with --rpc (RPC device silently dropped otherwise) * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan) * default split proportional to reported free mem = ~20/80 here, pin 1,1 * -dio required for large model over RPC - docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers, quant landscape, 2026-08-16 test matrix), PP deployment runbook, index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs) - worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC) - second head build build-vulkan-rpc (HIP off) for backend A/B
2026-08-16 05:40:27 -07:00
parent c775077016
commit 9544c36afc
13 changed files with 1037 additions and 5 deletions
@@ -6,7 +6,23 @@ inference runtime tuning, and server/OS/hardware settings. Each item lists a
 priority and the exact action.

 Dates: researched 2026-08. Hardware is verified on the live machine unless
-marked "verify first".
+marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
+
+## Corrections (re-verified 2026-08-15/16)
+
+- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
+  (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
+  Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
+  302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
+  hold on the 8060S. See [[frank-default-large-benchmark]].
+- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
+  does not exist on this machine (checked 2026-08-15). GPU DPM is already
+  `performance`, forced level `auto`. This item is N/A.
+- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
+- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
+  (ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
+- **TTM**: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still
+  pending a reboot.

 ## Verified Memory Picture (measured 2026-08-11)

@@ -47,8 +63,8 @@ Verified against the running service and llama.cpp source.

 | Change | Recommendation | Priority |
 | ------ | -------------- | -------- |
-| Backend | Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU. | Done |
-| KV cache | Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed. | Keep |
+| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
+| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
 | Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
 | Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
 | Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
@@ -62,7 +78,7 @@ All require root and a reboot for the modprobe change.
 | Setting | Action | Effect | Priority |
 | ------- | ------ | ------ | -------- |
 | TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High |
-| Power profile | `echo performance > /sys/class/drm/card0/device/power_profile` | Sustained inference speed; apply at boot via a udev/systemd unit | Medium |
+| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
 | NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
 | BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
 | Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
@@ -0,0 +1,137 @@
+# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
+
+Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
+3090 (worker) using llama.cpp RPC. This page is the design record and the
+morning-of runbook. Research background:
+[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
+
+## Topology
+
+```
+Dankclaw (head, 192.168.0.143)          Dankserver (worker, 192.168.0.235)
+llama-server b10433                      ggml-rpc-server b10433
+  --rpc 192.168.0.235:50052                 --host 192.168.0.235 --port 50052
+  model file (31.5 GB)                     -d CUDA0 -c (tensor cache)
+  [RPC0, ROCm0|Vulkan0]  layers            RTX 3090: layers 0..~31
+  Strix:    layers ~32..64 + lm_head
+  hidden state ~20 KiB crosses the link once per token, each direction
+```
+
+- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
+  the worker, the rest on the head, in order. One hidden-state handoff per
+  token step. RTT-bound, not bandwidth-bound.
+- **Why the head is on Dankclaw:** the model file is already there; the
+  worker streams its tensor share at load (no 31.5 GB copy to the server).
+- **Both sides must run the identical build** (b10433 / 9b05354ec). The
+  worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
+  so the production b10121 build stays untouched. A version skew hangs the
+  RPC handshake with no error.
+
+## Files
+
+| File | Machine | Purpose |
+|------|---------|---------|
+| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
+| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
+| `scripts/frank-pp-head.env` | claw | head tunables (split, ctx, spec, KV, I/O) |
+| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
+| `scripts/frank-pp-head.service` / `frank-pp-worker.service` | both | optional systemd units (manual install) |
+| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
+| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
+
+## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
+
+1. **No `--device` on the head.** Setting `params.devices` makes
+   `llama_prepare_model_devices` use exactly that list; the RPC device is
+   silently dropped and `--rpc` becomes a no-op. Omit it; default selection
+   puts RPC first.
+2. **The iGPU side is ROCm by default.** Backends register in fixed order
+   (HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
+   Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
+   head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
+   `HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
+   decode (measured on this box).
+3. **Pin `--tensor-split 1,1`.** The default split is proportional to
+   *reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
+   3090's 24 GB, so the default lands ~20/80 on the slow device (verified
+   with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
+4. **`-dio` on the head.** Large model + RPC hangs at load without direct
+   I/O.
+5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
+   0.0.0.0-equivalent by design (`--host 192.168.0.235`).
+6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
+   (HIP runtime reads them; the 2026-08-13 `TEMP` incident).
+7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
+   the head before load (warm-cache loads crash fast; cold loads survive).
+
+## Runbook (morning of 2026-08-16)
+
+Run from any box with ssh access to both (repo path on claw:
+`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
+
+```
+# 0. one-time: pull the kit on both boxes
+ssh dank@dankclaw   'cd /home/dank/vllmScripts  && git pull'
+ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
+
+# 1. checks (read-only)
+./scripts/pp-test.sh preflight
+
+# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
+
+# or step by step:
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
+./scripts/pp-test.sh verify          # layer split + worker VRAM
+./scripts/pp-test.sh bench 4 256 2   # ~4k prompt, 256 out, 2 reps
+./scripts/pp-test.sh bench 16 256 1
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
+```
+
+Config variants: edit `scripts/frank-pp-head.env` on the claw (git commit +
+pull if the bench runs from the repo), then re-run `start` + `bench`:
+
+- No spec baseline: `SPEC_NONE=1`
+- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
+- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
+- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
+- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
+
+Verification checklist after each `start`:
+
+1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
+2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
+   1,1 split.
+3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
+   (production small service + worker share).
+4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
+   (`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
+
+## Expected numbers
+
+Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
+step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
+
+- Production reference: 25 t/s @16k (Vulkan, spec, single node).
+- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
+- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
+  raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
+
+## Rollback
+
+`pp-test.sh restore` starts both production services. If a PP test wedges a
+box: `systemctl restart <unit>` after killing the stray processes
+(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
+with a pattern matching your own ssh command line).
+
+## Open questions for the test day
+
+1. Does the MTP head (which sits in the last layer group, iGPU side) draft
+   cleanly with the verify pass crossing the RPC boundary? Watch acceptance
+   in the head log (`draft acceptance = X`).
+2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
+   serialized per compute — expect the 2-slot speedup to be weaker than
+   single-node. Measure, do not assume.
+3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
+   the ubatch crossing the RPC boundary is the suspect.
@@ -6,7 +6,7 @@ tags:
  - pipeline-parallelism
  - distributed
  - index
-timestamp: 2026-08-14
+timestamp: 2026-08-15
 ---

 # Pipeline Parallelism
@@ -58,6 +58,14 @@ graph LR
 - [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
  across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.

+## Deployment
+
+- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
+  (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
+  `scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
+  `--rpc`, default split is proportional to reported free memory (pin 1,1),
+  the iGPU side defaults to the ROCm backend.
+
 ## Sample Scripts

 - [[Sample Script Set]] — a two-node llama.cpp RPC split of
@@ -53,6 +53,7 @@ Vulkan0 backend, port 7770, PARALLEL=2]
 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
 - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
 - [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15)
+- [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
 - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
 - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
 - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
@@ -71,3 +72,4 @@ Vulkan0 backend, port 7770, PARALLEL=2]
  - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
  - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
  - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
+  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — production design + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
@@ -0,0 +1,247 @@
+# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
+
+This page answers one question: **how do we make Qwen3.8-27B serve faster on
+Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
+the model's real architecture, the levers that are available, their evidence,
+and the test matrix planned for 2026-08-16 morning.
+
+Related pages:
+- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
+- [Backend and spec benchmark](frank-default-large-benchmark.md)
+- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
+- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
+
+## TL;DR — ranked levers
+
+| # | Lever | Expected gain | Cost | Status |
+|---|-------|--------------|------|--------|
+| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
+| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
+| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
+| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
+| 5 | TTM `pages_limit` raise (15.2 GiB → 90+) | removes a hard VRAM ceiling, stable 96 GiB budget | reboot | pending |
+| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
+
+Everything else (quantization downgrade, NVFP4, larger drafts) is documented
+below for completeness but is out of scope per the "no lowering" constraint.
+
+## What the model actually is
+
+Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
+dumped 2026-08-15) and the official model card.
+
+- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
+- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
+- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
+  layers, in a 3:1 pattern (`full_attention_interval = 4`).
+- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
+- Native context 262144 (extendable per the card; production runs 262k/slot).
+- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
+  per token). Only the 16 full-attention layers grow with context.
+- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
+  256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
+  262144 ctx ≈ 8 GiB.
+- Sampling defaults (card, thinking mode on by default): temperature 1.0,
+  top-k 20, top-p 0.95, min-p 0.0. Production already matches.
+
+Consequences of the hybrid design:
+
+1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
+   the 48 GDN layers are O(1) per token. Long-context decode degrades far
+   slower than a dense transformer.
+2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
+   cannot be prefix-cached the way attention KV can. Every agentic turn
+   re-prefills the whole conversation. **Prefill is the hidden cost of
+   agentic use**, and it is the dominant cost we have not yet optimized.
+3. **MTP is native and strong.** The model ships a nextn head; our matrix
+   measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan.
+
+## Current production state (baseline)
+
+`frank-default-large` (Dankclaw, Vulkan0, 2026-08-15):
+
+- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
+- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
+- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256,
+  threads 12, `-dio` off, `--fit off`.
+- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k
+  (Vulkan) vs 410 t/s @64k (ROCm, same build).
+- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.**
+
+## The levers, in detail
+
+### 1. Pipeline-parallel to the RTX 3090 (biggest win)
+
+The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with
+~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device
+runs its layer range, hidden states cross the 2.5 GbE ethernet once per
+token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both
+directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split
+is latency-bound, not bandwidth-bound (see
+[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)).
+
+Why this is a real speedup and not just "two GPUs": the 3090 does
+matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
+memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
+each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
+of all-64-layers-on-Strix.
+
+Design decisions (verified in b10433 source, 2026-08-15):
+
+- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays
+  on the head; the worker receives its tensor share over the wire at load
+  (with `-c` it caches the tensors locally after the first load).
+- **Head must NOT pass `--device`.** When `params.devices` is set,
+  `llama_prepare_model_devices` builds the device list from exactly those
+  devices and the RPC device is silently dropped (`--rpc` becomes a no-op).
+  Default device selection inserts RPC servers **at the front** of the list.
+- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks
+  registered backends in order; HIP registers before Vulkan, and the iGPU
+  dedup rule (PR #23897 workaround) keeps the first backend's device.
+  Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never
+  listed. Consequence: the Strix half runs on the ROCm backend — better
+  prefill, ~3% slower decode than Vulkan0 (measured on this box).
+  A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off,
+  Vulkan on, RPC on) was built the same night for an A/B; with it the
+  device list becomes `[RPC0, Vulkan0]`.
+- **Default split is proportional to *reported free memory*, which is wrong
+  here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB,
+  so the default lands ~20/80 in favor of the slow device (verified with a
+  4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1`
+  (50/50) explicitly.
+- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without
+  it (visorcraft Strix finding; matches our 28 GiB load-stall history).
+- **Version skew hangs the handshake.** Worker built at the identical
+  commit (9b05354ec / b10433) in a worktree
+  `/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build
+  untouched.
+
+Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched):
+worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on
+the tiny model. RTT on the link: ~0.3–0.5 ms.
+
+Full design, scripts, and morning runbook:
+[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md).
+
+### 2. Speculative-decoding tuning (same bits, free tokens)
+
+- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix
+  (Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec;
+  the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs
+  28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix
+  re-tests 3/4/5 on the real split.
+- **Q4_0 draft head.** The published Strix card measured a standalone
+  Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
+  head is a 248k-vocab output projection; cheaper dequant, same accuracy).
+  The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
+  `ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
+  an A/B via `--spec-draft-model`.
+- **Ngram-mod fallback stays.** It is the only spec type that survived
+  parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot).
+
+### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
+
+Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
+`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
+more ALUs per element; on a memory-bound GPU the difference is usually
+small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
+ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
+zero quality cost.
+
+### 4. Prefill is the agentic bottleneck — route it right
+
+Every agentic turn re-prefills the whole conversation (no prefix caching for
+hybrid state, upstream). Levers:
+
+- **Backend choice per workload.** Measured @64k on this box (b10433):
+  ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s
+  decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm.
+  The PP head (ROCm side) gets this for free.
+- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
+  (512/1024) fill the iGPU better during prefill; `--threads-batch` for
+  the CPU-bound tail. Untested at prefill scale on this box — matrix item.
+- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
+  at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
+  the safe default.
+
+### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
+
+- **TTM `pages_limit` is still 3,995,281 pages ≈ 15.24 GiB** — verified
+  today, unchanged. The 96 GiB GTT budget only materializes if the TTM cap
+  is raised (kernel parameter, needs reboot). Until then the effective
+  ceiling is lower and the 28 GiB load-stall risk remains. **Highest-value
+  single OS change; schedule with the next reboot.**
+- `iommu=pt` — verified present on the kernel cmdline. Done.
+- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
+  or loader-layer env is set. Pinning the radeon ICD and disabling loader
+  layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
+  hardening; testable as service env vars, zero risk.
+- **Power** — GPU DPM is already `performance`, forced level `auto`. The
+  `power_profile` sysfs file assumed in the Improvements doc does not exist
+  on this machine (checked 2026-08-15); that item is N/A here.
+- Threads: 12 of 16 cores in production. Community reports 12–16 are
+  equivalent; leave at 12.
+
+### 6. llama.cpp version watch
+
+b10433 (2026-08-15) is adequate. Through b10451 there are no
+spec-decoding fixes that help us; DFlash's multi-slot corruption is still
+open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
+re-testable candidates (DSpark was the best single-machine draft before the
+ROCm crash). No rebuild planned until then.
+
+## Quantization landscape (reference)
+
+The constraint is to keep UD-Q8_K_XL. This section records what exists so
+future decisions are informed.
+
+| Format | Size | Bits/weight | Notes |
+|--------|------|-------------|-------|
+| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
+| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
+| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
+| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
+| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
+| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
+
+If the 8-bit PP result (lever 1) is disappointing, the decision order is:
+push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
+replacement) → revisit the constraint with Frank.
+
+## Morning test matrix (2026-08-16)
+
+Policy: stop both production services, run tests, restore. Each config =
+one head restart (~10 min load incl. weight stream; ~2 min with the worker
+tensor cache warm).
+
+| # | Config | Measures |
+|---|--------|----------|
+| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
+| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
+| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
+| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
+| E | A with n_max 3 and 5 | n_max curve on the split |
+| F | A, plain Q8_0 model file | same-bit A/B |
+| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
+| Ref | production frank-default-large (Vulkan, 2 slots) | the bar to beat |
+
+Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
+(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
+acceptance, worker VRAM, head GTT usage.
+
+Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s
+production @16k) with the same 8-bit model.
+
+## Sources
+
+- GGUF header dump of the production model (this machine, 2026-08-15).
+- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
+  `unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
+- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
+  Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
+- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
+  `-dio` for RPC).
+- llama.cpp b10433 source: `src/llama.cpp` (device selection),
+  `common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
+- Local benchmarks: [frank-default-large-benchmark](frank-default-large-benchmark.md),
+  [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
@@ -0,0 +1,76 @@
+# Head tunables — Qwen3.8-27B pipeline-parallel head (Dankclaw).
+#
+# The head is the LOCAL side of the split. It runs llama-server (b10433,
+# HIP+Vulkan+RPC build in /home/dank/llama.cpp/build) with --rpc pointing at
+# the worker on Dankserver. The model file lives on this machine only; the
+# worker receives its tensor share over the network at load time.
+#
+# CRITICAL: this script must NOT pass --device. When --device is set,
+# llama.cpp uses exactly that device list and silently drops the RPC device
+# (src/llama.cpp llama_prepare_model_devices). Default device selection
+# inserts RPC servers at the FRONT of the device list, so the split becomes
+# [RPC0, iGPU] and --tensor-split index 0 is the worker.
+#
+# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE
+# (HIP runtime collision, see the 2026-08-13 TEMP incident).
+
+MODEL_DIR=/home/dank/models
+MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf
+HEAD_BIN=/home/dank/llama.cpp/build/bin/llama-server
+ALIAS=frank-pp-large
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
+
+# Comma-separated RPC endpoints (host:port). One worker here.
+RPC_SERVERS=192.168.0.235:50052
+
+# Model split fractions, one per device in order [RPC0, iGPU].
+# 1,1 = 50/50: the 3090 gets the first ~32 layers, the Strix the rest.
+# Leave empty for the default proportional-to-free-memory split; note the
+# Strix reports ~96 GiB (GTT ceiling) vs the 3090's ~24 GB, so the default
+# lands around 20% of layers on the 3090 and 80% on the Strix.
+TENSOR_SPLIT=1,1
+
+# Context pool. 131072 single slot keeps the KV small (~4.5 GB total at
+# q8_0) on both sides; raise for high-context tests (see docs).
+CTX_SIZE=131072
+PARALLEL=1
+
+# Speculative decoding. Matches production frank-default-large: MTP (bundled
+# head, n_max 4 per the published tuning curve) with the ngram-mod fallback.
+# Set SPEC_NONE=1 for a raw baseline with no speculation.
+SPEC_NONE=0
+SPEC_DEFAULT=1
+MTP=1
+SPEC_DRAFT_N_MAX=4
+# Optional standalone MTP draft head (e.g. the Q4_0 ggml-org file). When set,
+# it replaces the bundled in-model head via --spec-draft-model.
+SPEC_DRAFT_MODEL=
+SPEC_NGRAM_MOD_N_MIN=48
+SPEC_NGRAM_MOD_N_MAX=64
+SPEC_NGRAM_MOD_N_MATCH=24
+
+# KV on GPU, quantized. Keep K == V so Flash Attention stays enabled.
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=q8_0
+
+# Batch sizing: production-verified values (2026-08-14).
+BATCH_SIZE=8192
+UBATCH_SIZE=256
+
+# Sampling — Qwen3.8 thinking-mode defaults.
+TEMPERATURE=1.0
+TOP_K=20
+TOP_P=0.95
+MIN_P=0.0
+
+THREADS=12
+THREADS_HTTP=1
+
+# 10 (debug): prints per-layer device assignment so the split is verifiable
+# in the log. Drop to 4 for quiet production use.
+VERBOSITY=10
+
+# 1: --direct-io model load. Required in practice for large models loaded
+# over RPC (load hangs otherwise; visorcraft Strix RPC finding).
+DIRECT_IO=1
@@ -0,0 +1,16 @@
+[Unit]
+Description=llama.cpp Qwen3.8-27B pipeline-parallel head (Strix + RPC to Dankserver)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+Environment="FRANK_PP_HEAD_ENV_FILE=/home/dank/vllmScripts/scripts/frank-pp-head.env"
+ExecStart=/home/dank/vllmScripts/scripts/serve-pp-head.sh
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,30 @@
+# Worker tunables — Qwen3.8-27B pipeline-parallel RPC worker (Dankserver).
+#
+# The worker is the REMOTE side of the split. It exposes the RTX 3090 to the
+# head on Dankclaw as an RPC device. It runs from the b10433 CUDA+RPC build
+# in /home/dank/llama-cpp-rpc (worktree of /home/dank/llama.cpp).
+#
+# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE.
+# The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
+# incident in docs/deployment.md).
+
+# Bind address and port for ggml-rpc-server. Must be the LAN address the head
+# can reach; the default bind (127.0.0.1) refuses remote clients.
+RPC_BIND=192.168.0.235
+RPC_PORT=50052
+
+# Device to expose. CUDA0 = the RTX 3090.
+DEVICE=CUDA0
+
+# Threads for the worker's CPU device (server has 8 cores; leave 1 for the OS
+# and the running frank-default-small service).
+THREADS=7
+
+# 1: enable the local tensor file cache (~/.cache/llama.cpp/rpc/). The head
+# streams the 31.5 GB model to the worker on first load; with the cache, later
+# loads skip the transfer.
+CACHE=1
+
+# Worker binary. Must be the SAME llama.cpp build as the head (b10433 /
+# 9b05354ec). Version skew hangs the RPC handshake.
+WORKER_BIN=/home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server
@@ -0,0 +1,15 @@
+[Unit]
+Description=llama.cpp Qwen3.8-27B pipeline-parallel RPC worker (RTX 3090)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+ExecStart=/home/dank/vLLM/scripts/serve-pp-worker.sh
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,103 @@
+#!/usr/bin/env python3
+"""pp-bench.py — drive the PP head API and report server-side timings.
+
+Runs on the HEAD machine (Dankclaw). Targets the running PP head
+(serve-pp-head.sh, port from argument). Builds a ~Nk-token prose prompt,
+sends one warmup completion, then REPS measured completions, and reads the
+authoritative `slot print_timing` lines from the head log.
+
+Usage:
+    python3 pp-bench.py <prompt_k> <out_tokens> <reps> <port> <head_log>
+"""
+import json
+import re
+import sys
+import time
+import urllib.request
+
+prompt_k, out_tokens, reps, port, head_log = sys.argv[1:6]
+out_tokens = int(out_tokens)
+reps = int(reps)
+target = int(prompt_k) * 1000
+
+# ~38 tokens per 160-char prose unit; size the filler to hit the target.
+unit = ("The careful study of natural language reveals patterns of meaning "
+        "that emerge from the deliberate arrangement of words and phrases. ")
+approx = max(1, target // 38)
+prompt = unit * approx + (
+    "\n\nNow, in a detailed and flowing paragraph, "
+    "continue this essay about the philosophy of "
+    "language and its influence on modern computation. "
+)
+
+
+def req(n):
+    body = json.dumps({"prompt": prompt, "max_tokens": n,
+                       "temperature": 0.0, "stream": False}).encode()
+    r = urllib.request.Request(
+        f"http://127.0.0.1:{port}/v1/completions",
+        data=body, headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(r, timeout=7200) as resp:
+        return json.load(resp)
+
+
+def log_timings():
+    try:
+        lines = open(head_log, encoding="utf-8", errors="ignore").read().splitlines()
+    except OSError:
+        return []
+    out = []
+    for l in lines:
+        if re.search(r"prompt eval time|eval time|draft acceptance", l):
+            out.append(l.strip())
+    return out
+
+
+GEN_RE = re.compile(
+    r"eval time =\s+([\d.]+) ms /\s+(\d+) tokens.*?([\d.]+) tokens per second")
+PRE_RE = re.compile(
+    r"prompt eval time =\s+([\d.]+) ms /\s+(\d+) tokens.*?([\d.]+) tokens per second")
+ACC_RE = re.compile(r"draft acceptance = ([\d.]+)")
+
+# warmup (first run pays graph-compile / JIT costs)
+warm = req(16)
+print(f"warmup: prompt_tokens={warm['usage']['prompt_tokens']} "
+      f"completion_tokens={warm['usage']['completion_tokens']}")
+
+gen_rates, prefill_rates, accepts = [], [], []
+for i in range(reps):
+    t0 = time.time()
+    out = req(out_tokens)
+    dt = time.time() - t0
+    u = out["usage"]
+    pt, ct = u["prompt_tokens"], u["completion_tokens"]
+    gen = pre = acc = None
+    for l in log_timings():
+        if "prompt eval" not in l:
+            m = GEN_RE.search(l)
+            if m:
+                gen = float(m.group(3))
+        m = PRE_RE.search(l)
+        if m:
+            pre = float(m.group(3))
+        m = ACC_RE.search(l)
+        if m:
+            acc = float(m.group(1))
+    gen_rates.append(gen if gen is not None else ct / dt)
+    if pre is not None:
+        prefill_rates.append(pre)
+    if acc is not None:
+        accepts.append(acc)
+    print(f"rep {i + 1}: prompt={pt} gen={ct} wall={dt:.1f}s "
+          f"gen={gen_rates[-1]:.2f} t/s "
+          f"prefill={f'{pre:.1f}' if pre is not None else 'n/a'} t/s "
+          f"acceptance={f'{acc:.3f}' if acc is not None else 'n/a'}")
+
+print("== RESULT pp ==")
+print(f"avg gen tok/s   = {sum(gen_rates) / len(gen_rates):.2f}")
+if prefill_rates:
+    print(f"avg prefill t/s = {sum(prefill_rates) / len(prefill_rates):.1f}")
+if accepts:
+    print(f"avg acceptance  = {sum(accepts) / len(accepts):.3f}")
+for l in log_timings()[-3:]:
+    print("  " + l)
@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# pp-test.sh — Qwen3.8-27B pipeline-parallel (llama.cpp RPC) test orchestrator.
+#
+# Runs Qwen3.8-27B UD-Q8_K_XL split across Dankclaw (head, Strix iGPU) and
+# Dankserver (worker, RTX 3090) over the 2.5 GbE LAN. Both boxes run the same
+# llama.cpp build (b10433 / 9b05354ec); the worker build lives in
+# /home/dank/llama-cpp-rpc on Dankserver.
+#
+# BENCHMARKING POLICY (AGENTS.md): this test STOPS both production services
+# and restores them when done. Do not run it while users are mid-stream.
+#
+# Usage (from any box with ssh access to both targets, repo checked out):
+#   ./scripts/pp-test.sh preflight
+#   ./scripts/pp-test.sh start                # stop prod, start PP head+worker
+#   ./scripts/pp-test.sh verify               # show the layer split + VRAM
+#   ./scripts/pp-test.sh bench [prompt_k] [out_tokens] [reps]
+#   ./scripts/pp-test.sh stop-pp              # stop PP head+worker only
+#   ./scripts/pp-test.sh restore              # start production services
+#   ./scripts/pp-test.sh all                  # start + verify + bench + restore
+#
+# Sudo: set FRANK_SUDO_PW for headless runs (stop/start of system services).
+#   FRANK_SUDO_PW=... ./scripts/pp-test.sh start
+set -euo pipefail
+
+CLAW="dank@dankclaw"
+SERVER="dank@dankserver"
+CLAW_REPO="/home/dank/vllmScripts"
+SERVER_REPO="/home/dank/vLLM"
+HEAD_PORT=7770
+HEAD_LOG="/tmp/pp-head.log"
+WORKER_LOG="/tmp/pp-worker.log"
+RPC_PORT=50052
+
+SUDO="sudo"
+if [ -n "${FRANK_SUDO_PW:-}" ]; then
+    SUDO="printf '%s\n' '$FRANK_SUDO_PW' | sudo -S"
+fi
+
+log()  { echo "[pp-test] $*"; }
+die()  { echo "[pp-test] FAIL: $*" >&2; exit 1; }
+
+ssh_claw()   { ssh -o BatchMode=yes "$CLAW"   "$@"; }
+ssh_server() { ssh -o BatchMode=yes "$SERVER" "$@"; }
+
+# ---------------------------------------------------------------------------
+# preflight
+# ---------------------------------------------------------------------------
+do_preflight() {
+    log "preflight: build identity"
+    local h_v w_v
+    h_v=$(ssh_claw '/home/dank/llama.cpp/build/bin/llama-server --version 2>&1 | head -1')
+    w_v=$(ssh_server '/home/dank/llama-cpp-rpc/build/bin/llama-server --version 2>&1 | head -1')
+    log "  head:   $h_v"
+    log "  worker: $w_v"
+    [[ "$h_v" == *9b05354ec* && "$w_v" == *9b05354ec* ]] || die "build skew (need identical commit 9b05354ec on both sides)"
+
+    log "preflight: binaries"
+    ssh_claw   "[ -x $CLAW_REPO/scripts/serve-pp-head.sh ]"   || die "missing $CLAW_REPO/scripts/serve-pp-head.sh on claw (git pull?)"
+    ssh_server "[ -x $SERVER_REPO/scripts/serve-pp-worker.sh ]" || die "missing $SERVER_REPO/scripts/serve-pp-worker.sh on server (git pull?)"
+    ssh_server "[ -x /home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server ]" || die "missing worker RPC binary on server"
+    ssh_claw   "[ -f /home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf ]" || die "missing model on claw"
+
+    log "preflight: network (claw -> server:50052 listener check needs the worker up; testing route + link)"
+    ssh_claw "ip -4 addr show eno1 | grep -q 192.168.0.143" || die "unexpected claw address"
+    ssh_server "ip -4 addr show eno1 | grep -q 192.168.0.235" || die "unexpected server address"
+    ssh_claw "ping -c 2 -W 2 192.168.0.235 >/dev/null" || die "claw cannot reach server"
+    local rtt
+    rtt=$(ssh_claw "ping -c 5 -W 2 192.168.0.235 | tail -1" | grep -oE '[0-9.]+/[0-9.]+ ms' | head -1 || true)
+    log "  rtt: ${rtt:-unknown}"
+
+    log "preflight: production services"
+    ssh_claw   "$SUDO systemctl is-active frank-default-large"  2>/dev/null || true
+    ssh_server "$SUDO systemctl is-active frank-default-small" 2>/dev/null || true
+    log "preflight OK"
+}
+
+# ---------------------------------------------------------------------------
+# start
+# ---------------------------------------------------------------------------
+do_start() {
+    log "stopping production services (both boxes)"
+    ssh_claw   "$SUDO systemctl stop frank-default-large"  || die "could not stop frank-default-large"
+    ssh_server "$SUDO systemctl stop frank-default-small"  || die "could not stop frank-default-small"
+    sleep 5
+
+    log "dropping page caches on claw (reliable model load)"
+    ssh_claw "$SUDO sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'" || log "WARN: cache drop failed"
+
+    log "starting RPC worker on server"
+    ssh_server "cd $SERVER_REPO && nohup bash scripts/serve-pp-worker.sh > $WORKER_LOG 2>&1 & echo worker_pid=\$!"
+    for i in $(seq 1 30); do
+        if ssh_server "ss -tln | grep -q :$RPC_PORT"; then
+            log "worker listening on :$RPC_PORT"
+            break
+        fi
+        [ "$i" = "30" ] && { ssh_server "tail -20 $WORKER_LOG"; die "worker did not bind :$RPC_PORT"; }
+        sleep 1
+    done
+
+    log "starting PP head on claw (model load + weight stream: allow ~10 min)"
+    local head_pid
+    head_pid=$(ssh_claw "cd $CLAW_REPO && nohup bash scripts/serve-pp-head.sh > $HEAD_LOG 2>&1 & echo \$!")
+    log "head pid on claw: $head_pid"
+    for i in $(seq 1 360); do
+        local st
+        st=$(ssh_claw "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$HEAD_PORT/health 2>/dev/null || true")
+        if [ "$st" = "200" ]; then
+            log "head healthy after ~$((i * 5))s"
+            return 0
+        fi
+        if ! ssh_claw "kill -0 $head_pid 2>/dev/null"; then
+            ssh_claw "tail -30 $HEAD_LOG"
+            die "head process died during load"
+        fi
+        [ $((i % 12)) = 0 ] && log "  still loading... ($((i * 5))s)"
+        sleep 5
+    done
+    ssh_claw "tail -30 $HEAD_LOG"
+    die "head did not become healthy in 30 min"
+}
+
+# ---------------------------------------------------------------------------
+# verify
+# ---------------------------------------------------------------------------
+do_verify() {
+    log "health + device split (from head log)"
+    ssh_claw "curl -s http://127.0.0.1:$HEAD_PORT/health; echo"
+    ssh_claw "grep -aE 'using device|assigned to device|MiB free' $HEAD_LOG | head -20 || true"
+    log "worker VRAM on server:"
+    ssh_server "nvidia-smi --query-gpu=memory.used --format=csv,noheader"
+    log "worker log tail:"
+    ssh_server "tail -5 $WORKER_LOG"
+}
+
+# ---------------------------------------------------------------------------
+# bench
+# ---------------------------------------------------------------------------
+do_bench() {
+    local PROMPT_K="${1:-4}"
+    local OUT_TOKENS="${2:-256}"
+    local REPS="${3:-2}"
+    log "bench: prompt ~${PROMPT_K}k tokens, out=${OUT_TOKENS}, reps=${REPS} (on claw, via :$HEAD_PORT)"
+    ssh_claw "python3 $CLAW_REPO/scripts/pp-bench.py $PROMPT_K $OUT_TOKENS $REPS $HEAD_PORT $HEAD_LOG"
+}
+
+# ---------------------------------------------------------------------------
+# stop / restore
+# ---------------------------------------------------------------------------
+do_stop_pp() {
+    log "stopping PP head (claw) and worker (server)"
+    ssh_claw   "pkill -f 'serve-pp-head.sh' 2>/dev/null; pkill -f 'llama-server.*frank-pp-large' 2>/dev/null; true"
+    ssh_server "pkill -x ggml-rpc-server 2>/dev/null; true"
+    sleep 3
+    ssh_server "nvidia-smi --query-gpu=memory.used --format=csv,noheader"
+}
+
+do_restore() {
+    log "starting production services"
+    ssh_server "$SUDO systemctl start frank-default-small" || die "could not start frank-default-small"
+    ssh_claw   "$SUDO systemctl start frank-default-large" || die "could not start frank-default-large"
+    log "waiting for health (model load takes minutes; up to ~17 min cold on claw)"
+    for i in $(seq 1 300); do
+        local a b
+        a=$(ssh_claw   "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:7770/health 2>/dev/null || true")
+        b=$(ssh_server "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:7770/health 2>/dev/null || true")
+        if [ "$a" = "200" ] && [ "$b" = "200" ]; then
+            log "both production services healthy after ~$((i * 4))s"
+            return 0
+        fi
+        [ $((i % 15)) = 0 ] && log "  claw=$a server=$b ... ($((i * 4))s)"
+        sleep 4
+    done
+    die "production services did not both become healthy"
+}
+
+# ---------------------------------------------------------------------------
+case "${1:-}" in
+    preflight) do_preflight ;;
+    start)     do_start ;;
+    verify)    do_verify ;;
+    bench)     shift; do_bench "$@" ;;
+    stop-pp)   do_stop_pp ;;
+    restore)   do_restore ;;
+    all)
+        do_preflight
+        do_start
+        do_verify
+        do_bench
+        do_restore
+        ;;
+    *)
+        echo "Usage: $0 {preflight|start|verify|bench [prompt_k] [out] [reps]|stop-pp|restore|all}"
+        exit 1
+        ;;
+esac
@@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+# Serve the Qwen3.8-27B pipeline-parallel head on Dankclaw.
+#
+# Runs llama-server (b10433) with --rpc pointing at the worker on
+# Dankserver. The model is split pipeline-style across [RPC0, iGPU]:
+# layers run in order, hidden states cross the ethernet boundary once per
+# token step.
+#
+# IMPORTANT: no --device flag. The RPC device only joins the model's device
+# list through default device selection; an explicit --device silently drops
+# it. See the header of frank-pp-head.env.
+#
+# Tunables come from frank-pp-head.env in this directory unless the
+# FRANK_PP_HEAD_ENV_FILE environment variable overrides that path.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_PP_HEAD_ENV_FILE:-$SCRIPT_DIR/frank-pp-head.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${MODEL_DIR:=/home/dank/models}"
+: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
+: "${HEAD_BIN:=/home/dank/llama.cpp/build/bin/llama-server}"
+: "${ALIAS:=frank-pp-large}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7770}"
+: "${RPC_SERVERS:=192.168.0.235:50052}"
+: "${TENSOR_SPLIT:=1,1}"
+: "${CTX_SIZE:=131072}"
+: "${PARALLEL:=1}"
+: "${SPEC_NONE:=0}"
+: "${SPEC_DEFAULT:=1}"
+: "${MTP:=0}"
+: "${SPEC_DRAFT_N_MAX:=4}"
+: "${SPEC_DRAFT_MODEL:=}"
+: "${SPEC_NGRAM_MOD_N_MIN:=48}"
+: "${SPEC_NGRAM_MOD_N_MAX:=64}"
+: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
+: "${CACHE_TYPE_K:=q8_0}"
+: "${CACHE_TYPE_V:=q8_0}"
+: "${BATCH_SIZE:=8192}"
+: "${UBATCH_SIZE:=256}"
+: "${TEMPERATURE:=1.0}"
+: "${TOP_K:=20}"
+: "${TOP_P:=0.95}"
+: "${MIN_P:=0.0}"
+: "${THREADS:=12}"
+: "${THREADS_HTTP:=1}"
+: "${VERBOSITY:=4}"
+: "${DIRECT_IO:=1}"
+
+MODEL="$MODEL_DIR/$MODEL_FILE"
+
+if [[ ! -f "$MODEL" ]]; then
+    echo "missing required file: $MODEL" >&2
+    exit 1
+fi
+if [[ ! -x "$HEAD_BIN" ]]; then
+    echo "missing head binary: $HEAD_BIN" >&2
+    exit 1
+fi
+
+# --- RPC endpoints -----------------------------------------------------------
+RPC_ARGS=()
+for s in ${RPC_SERVERS//,/ }; do
+    RPC_ARGS+=(--rpc "$s")
+done
+
+# --- split fractions ---------------------------------------------------------
+SPLIT_ARGS=()
+if [[ -n "${TENSOR_SPLIT}" ]]; then
+    SPLIT_ARGS+=(--tensor-split "${TENSOR_SPLIT}")
+fi
+
+# --- speculative decoding -----------------------------------------------------
+SPEC_ARGS=()
+if [[ "${SPEC_NONE}" == "1" ]]; then
+    SPEC_ARGS+=(--spec-type none)
+else
+    if [[ "${SPEC_DEFAULT}" == "1" ]]; then
+        if [[ "${MTP}" == "1" ]]; then
+            # MTP first (bundled in-model head by default, or a standalone
+            # draft file when SPEC_DRAFT_MODEL is set), then the ngram-mod
+            # fallback. Order matters: --spec-type appends to the impl chain.
+            SPEC_ARGS+=(--spec-type draft-mtp)
+            SPEC_ARGS+=(--spec-draft-n-max "${SPEC_DRAFT_N_MAX}")
+            if [[ -n "${SPEC_DRAFT_MODEL}" ]]; then
+                if [[ ! -f "${SPEC_DRAFT_MODEL}" ]]; then
+                    echo "SPEC_DRAFT_MODEL file missing: ${SPEC_DRAFT_MODEL}" >&2
+                    exit 1
+                fi
+                SPEC_ARGS+=(--spec-draft-model "${SPEC_DRAFT_MODEL}")
+            fi
+        fi
+        SPEC_ARGS+=(--spec-default)
+        SPEC_ARGS+=(--spec-ngram-mod-n-min "${SPEC_NGRAM_MOD_N_MIN}")
+        SPEC_ARGS+=(--spec-ngram-mod-n-max "${SPEC_NGRAM_MOD_N_MAX}")
+        SPEC_ARGS+=(--spec-ngram-mod-n-match "${SPEC_NGRAM_MOD_N_MATCH}")
+    else
+        echo "SPEC_NONE=0 requires SPEC_DEFAULT=1" >&2
+        exit 1
+    fi
+fi
+
+# --- load mode ----------------------------------------------------------------
+LOAD_ARGS=()
+if [[ "${DIRECT_IO}" == "1" ]]; then
+    LOAD_ARGS+=(--direct-io)
+fi
+
+exec "$HEAD_BIN" \
+    --model "$MODEL" \
+    "${RPC_ARGS[@]}" \
+    "${SPLIT_ARGS[@]}" \
+    "${SPEC_ARGS[@]}" \
+    "${LOAD_ARGS[@]}" \
+    --gpu-layers all \
+    --fit off \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --ctx-size "$CTX_SIZE" \
+    --parallel "$PARALLEL" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --temp "$TEMPERATURE" \
+    --top-k "$TOP_K" \
+    --top-p "$TOP_P" \
+    --min-p "$MIN_P" \
+    --threads "$THREADS" \
+    --threads-http "$THREADS_HTTP" \
+    --alias "$ALIAS" \
+    -lv "$VERBOSITY" \
+    --host "$BIND_ADDR" \
+    --port "$LISTEN_PORT"
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# Serve the Qwen3.8-27B pipeline-parallel RPC worker on Dankserver.
+#
+# Runs ggml-rpc-server from the b10433 CUDA+RPC build and exposes the RTX
+# 3090 as a remote device for the head on Dankclaw.
+#
+# Tunables come from frank-pp-worker.env in this directory unless the
+# FRANK_PP_WORKER_ENV_FILE environment variable overrides that path.
+#
+# Security note (upstream): the RPC server is "fragile and insecure". No
+# authentication, no encryption. Keep it on the trusted LAN only.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_PP_WORKER_ENV_FILE:-$SCRIPT_DIR/frank-pp-worker.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${RPC_BIND:=192.168.0.235}"
+: "${RPC_PORT:=50052}"
+: "${DEVICE:=CUDA0}"
+: "${THREADS:=7}"
+: "${CACHE:=1}"
+: "${WORKER_BIN:=/home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server}"
+
+if [[ ! -x "$WORKER_BIN" ]]; then
+    echo "missing worker binary: $WORKER_BIN" >&2
+    echo "Build it: cd /home/dank/llama-cpp-rpc && cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release -- -j 8" >&2
+    exit 1
+fi
+
+ARGS=()
+if [[ "${CACHE}" == "1" ]]; then
+    ARGS+=(--cache)
+fi
+
+exec "$WORKER_BIN" \
+    --host "$RPC_BIND" \
+    --port "$RPC_PORT" \
+    --device "$DEVICE" \
+    --threads "$THREADS" \
+    "${ARGS[@]}"