diff --git a/docs/Dankclaw Improvements.md b/docs/Dankclaw Improvements.md index 68aa224..7346e58 100644 --- a/docs/Dankclaw Improvements.md +++ b/docs/Dankclaw Improvements.md @@ -6,7 +6,23 @@ inference runtime tuning, and server/OS/hardware settings. Each item lists a priority and the exact action. Dates: researched 2026-08. Hardware is verified on the live machine unless -marked "verify first". +marked "verify first". Re-verified 2026-08-15/16; corrections marked below. + +## Corrections (re-verified 2026-08-15/16) + +- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box + (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx + Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs + 302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not + hold on the 8060S. See [[frank-default-large-benchmark]]. +- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file + does not exist on this machine (checked 2026-08-15). GPU DPM is already + `performance`, forced level `auto`. This item is N/A. +- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized). +- **Spec**: DFlash/DSpark are retired. Production is `--spec-default` + (ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. +- **TTM**: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still + pending a reboot. ## Verified Memory Picture (measured 2026-08-11) @@ -47,8 +63,8 @@ Verified against the running service and llama.cpp source. | Change | Recommendation | Priority | | ------ | -------------- | -------- | -| Backend | Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU. | Done | -| KV cache | Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed. | Keep | +| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done | +| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep | | Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep | | Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional | | Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium | @@ -62,7 +78,7 @@ All require root and a reboot for the modprobe change. | Setting | Action | Effect | Priority | | ------- | ------ | ------ | -------- | | TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High | -| Power profile | `echo performance > /sys/class/drm/card0/device/power_profile` | Sustained inference speed; apply at boot via a udev/systemd unit | Medium | +| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ | | NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done | | BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify | | Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify | diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md new file mode 100644 index 0000000..67937cf --- /dev/null +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -0,0 +1,137 @@ +# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver) + +Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX +3090 (worker) using llama.cpp RPC. This page is the design record and the +morning-of runbook. Research background: +[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md). + +## Topology + +``` +Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235) +llama-server b10433 ggml-rpc-server b10433 + --rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052 + model file (31.5 GB) -d CUDA0 -c (tensor cache) + [RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0..~31 + Strix: layers ~32..64 + lm_head + hidden state ~20 KiB crosses the link once per token, each direction +``` + +- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on + the worker, the rest on the head, in order. One hidden-state handoff per + token step. RTT-bound, not bandwidth-bound. +- **Why the head is on Dankclaw:** the model file is already there; the + worker streams its tensor share at load (no 31.5 GB copy to the server). +- **Both sides must run the identical build** (b10433 / 9b05354ec). The + worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver + so the production b10121 build stays untouched. A version skew hangs the + RPC handshake with no error. + +## Files + +| File | Machine | Purpose | +|------|---------|---------| +| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) | +| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build | +| `scripts/frank-pp-head.env` | claw | head tunables (split, ctx, spec, KV, I/O) | +| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") | +| `scripts/frank-pp-head.service` / `frank-pp-worker.service` | both | optional systemd units (manual install) | +| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) | +| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` | + +## Gotchas (all verified in b10433 source / live tests, 2026-08-15) + +1. **No `--device` on the head.** Setting `params.devices` makes + `llama_prepare_model_devices` use exactly that list; the RPC device is + silently dropped and `--rpc` becomes a no-op. Omit it; default selection + puts RPC first. +2. **The iGPU side is ROCm by default.** Backends register in fixed order + (HIP before Vulkan); the iGPU dedup rule keeps the first backend's view. + Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second + head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set + `HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower + decode (measured on this box). +3. **Pin `--tensor-split 1,1`.** The default split is proportional to + *reported free memory*: the Strix reports up to 96 GiB (GTT) vs the + 3090's 24 GB, so the default lands ~20/80 on the slow device (verified + with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency. +4. **`-dio` on the head.** Large model + RPC hangs at load without direct + I/O. +5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds + 0.0.0.0-equivalent by design (`--host 192.168.0.235`). +6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE` + (HIP runtime reads them; the 2026-08-13 `TEMP` incident). +7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on + the head before load (warm-cache loads crash fast; cold loads survive). + +## Runbook (morning of 2026-08-16) + +Run from any box with ssh access to both (repo path on claw: +`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`). + +``` +# 0. one-time: pull the kit on both boxes +ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull' +ssh dank@dankserver 'cd /home/dank/vLLM && git pull' + +# 1. checks (read-only) +./scripts/pp-test.sh preflight + +# 2. full run: stop prod -> start PP -> verify split -> bench -> restore +FRANK_SUDO_PW= ./scripts/pp-test.sh all + +# or step by step: +FRANK_SUDO_PW= ./scripts/pp-test.sh start +./scripts/pp-test.sh verify # layer split + worker VRAM +./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps +./scripts/pp-test.sh bench 16 256 1 +FRANK_SUDO_PW= ./scripts/pp-test.sh stop-pp +FRANK_SUDO_PW= ./scripts/pp-test.sh restore +``` + +Config variants: edit `scripts/frank-pp-head.env` on the claw (git commit + +pull if the bench runs from the repo), then re-run `start` + `bench`: + +- No spec baseline: `SPEC_NONE=1` +- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server` +- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below) +- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5` +- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw) + +Verification checklist after each `start`: + +1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`. +2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a + 1,1 split. +3. `nvidia-smi` on the server shows ~15–16 GB used by the worker + (production small service + worker share). +4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices + (`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue). + +## Expected numbers + +Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token +step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT. + +- Production reference: 25 t/s @16k (Vulkan, spec, single node). +- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec. +- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate; + raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold. + +## Rollback + +`pp-test.sh restore` starts both production services. If a PP test wedges a +box: `systemctl restart ` after killing the stray processes +(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f` +with a pattern matching your own ssh command line). + +## Open questions for the test day + +1. Does the MTP head (which sits in the last layer group, iGPU side) draft + cleanly with the verify pass crossing the RPC boundary? Watch acceptance + in the head log (`draft acceptance = X`). +2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is + serialized per compute — expect the 2-slot speedup to be weaker than + single-node. Measure, do not assume. +3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not, + the ubatch crossing the RPC boundary is the suspect. diff --git a/docs/Pipeline Parallelism/index.md b/docs/Pipeline Parallelism/index.md index 1f6a9cf..83f93f0 100644 --- a/docs/Pipeline Parallelism/index.md +++ b/docs/Pipeline Parallelism/index.md @@ -6,7 +6,7 @@ tags: - pipeline-parallelism - distributed - index -timestamp: 2026-08-14 +timestamp: 2026-08-15 --- # Pipeline Parallelism @@ -58,6 +58,14 @@ graph LR - [[Expected Performance]] — estimated tokens per second for a 27B Q8 split across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE. +## Deployment + +- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw + (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in + `scripts/`, morning-of test runbook. Verified gotchas: no `--device` with + `--rpc`, default split is proportional to reported free memory (pin 1,1), + the iGPU side defaults to the ROCm backend. + ## Sample Scripts - [[Sample Script Set]] — a two-node llama.cpp RPC split of diff --git a/docs/index.md b/docs/index.md index 113cbbf..158d2da 100644 --- a/docs/index.md +++ b/docs/index.md @@ -53,6 +53,7 @@ Vulkan0 backend, port 7770, PARALLEL=2] - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) - [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15) +- [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings @@ -71,3 +72,4 @@ Vulkan0 backend, port 7770, PARALLEL=2] - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split + - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — production design + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/ diff --git a/docs/qwen3.8-strix-optimizations.md b/docs/qwen3.8-strix-optimizations.md new file mode 100644 index 0000000..a87ed8d --- /dev/null +++ b/docs/qwen3.8-strix-optimizations.md @@ -0,0 +1,247 @@ +# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant + +This page answers one question: **how do we make Qwen3.8-27B serve faster on +Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records +the model's real architecture, the levers that are available, their evidence, +and the test matrix planned for 2026-08-16 morning. + +Related pages: +- [Speculative-decoding matrix](qwen3.8-spec-matrix.md) +- [Backend and spec benchmark](frank-default-large-benchmark.md) +- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items) +- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md) + +## TL;DR — ranked levers + +| # | Lever | Expected gain | Cost | Status | +|---|-------|--------------|------|--------| +| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 | +| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 | +| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 | +| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data | +| 5 | TTM `pages_limit` raise (15.2 GiB → 90+) | removes a hard VRAM ceiling, stable 96 GiB budget | reboot | pending | +| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 | + +Everything else (quantization downgrade, NVFP4, larger drafts) is documented +below for completeness but is out of scope per the "no lowering" constraint. + +## What the model actually is + +Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`, +dumped 2026-08-15) and the official model card. + +- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**. +- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks. +- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention + layers, in a 3:1 pattern (`full_attention_interval = 4`). +- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256. +- Native context 262144 (extendable per the card; production runs 262k/slot). +- GDN layers keep a **fixed-size** recurrent state (constant per layer, not + per token). Only the 16 full-attention layers grow with context. +- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads × + 256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at + 262144 ctx ≈ 8 GiB. +- Sampling defaults (card, thinking mode on by default): temperature 1.0, + top-k 20, top-p 0.95, min-p 0.0. Production already matches. + +Consequences of the hybrid design: + +1. **Decode is cheap in memory.** Attention cost scales with only 16 layers; + the 48 GDN layers are O(1) per token. Long-context decode degrades far + slower than a dense transformer. +2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state + cannot be prefix-cached the way attention KV can. Every agentic turn + re-prefills the whole conversation. **Prefill is the hidden cost of + agentic use**, and it is the dominant cost we have not yet optimized. +3. **MTP is native and strong.** The model ships a nextn head; our matrix + measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan. + +## Current production state (baseline) + +`frank-default-large` (Dankclaw, Vulkan0, 2026-08-15): + +- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head. +- `--spec-default` (ngram-mod) with draft-mtp first in the chain. +- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256, + threads 12, `-dio` off, `--fit off`. +- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k + (Vulkan) vs 410 t/s @64k (ROCm, same build). +- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.** + +## The levers, in detail + +### 1. Pipeline-parallel to the RTX 3090 (biggest win) + +The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with +~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device +runs its layer range, hidden states cross the 2.5 GbE ethernet once per +token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both +directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split +is latency-bound, not bandwidth-bound (see +[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)). + +Why this is a real speedup and not just "two GPUs": the 3090 does +matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s +memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so +each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead +of all-64-layers-on-Strix. + +Design decisions (verified in b10433 source, 2026-08-15): + +- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays + on the head; the worker receives its tensor share over the wire at load + (with `-c` it caches the tensors locally after the first load). +- **Head must NOT pass `--device`.** When `params.devices` is set, + `llama_prepare_model_devices` builds the device list from exactly those + devices and the RPC device is silently dropped (`--rpc` becomes a no-op). + Default device selection inserts RPC servers **at the front** of the list. +- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks + registered backends in order; HIP registers before Vulkan, and the iGPU + dedup rule (PR #23897 workaround) keeps the first backend's device. + Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never + listed. Consequence: the Strix half runs on the ROCm backend — better + prefill, ~3% slower decode than Vulkan0 (measured on this box). + A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off, + Vulkan on, RPC on) was built the same night for an A/B; with it the + device list becomes `[RPC0, Vulkan0]`. +- **Default split is proportional to *reported free memory*, which is wrong + here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB, + so the default lands ~20/80 in favor of the slow device (verified with a + 4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1` + (50/50) explicitly. +- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without + it (visorcraft Strix finding; matches our 28 GiB load-stall history). +- **Version skew hangs the handshake.** Worker built at the identical + commit (9b05354ec / b10433) in a worktree + `/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build + untouched. + +Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched): +worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on +the tiny model. RTT on the link: ~0.3–0.5 ms. + +Full design, scripts, and morning runbook: +[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md). + +### 2. Speculative-decoding tuning (same bits, free tokens) + +- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix + (Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec; + the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs + 28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix + re-tests 3/4/5 on the real split. +- **Q4_0 draft head.** The published Strix card measured a standalone + Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft + head is a 248k-vocab output projection; cheaper dequant, same accuracy). + The bundled head inside UD-Q8_K_XL is Q8-class; the standalone + `ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for + an A/B via `--spec-draft-model`. +- **Ngram-mod fallback stays.** It is the only spec type that survived + parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot). + +### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0 + +Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain +`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs +more ALUs per element; on a memory-bound GPU the difference is usually +small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF` +ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at +zero quality cost. + +### 4. Prefill is the agentic bottleneck — route it right + +Every agentic turn re-prefills the whole conversation (no prefix caching for +hybrid state, upstream). Levers: + +- **Backend choice per workload.** Measured @64k on this box (b10433): + ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s + decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm. + The PP head (ROCm side) gets this for free. +- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches + (512/1024) fill the iGPU better during prefill; `--threads-batch` for + the CPU-bound tail. Untested at prefill scale on this box — matrix item. +- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB + at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is + the safe default. + +### 5. OS level (from the Improvements doc, re-verified 2026-08-15) + +- **TTM `pages_limit` is still 3,995,281 pages ≈ 15.24 GiB** — verified + today, unchanged. The 96 GiB GTT budget only materializes if the TTM cap + is raised (kernel parameter, needs reboot). Until then the effective + ceiling is lower and the 28 GiB load-stall risk remains. **Highest-value + single OS change; schedule with the next reboot.** +- `iommu=pt` — verified present on the kernel cmdline. Done. +- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES` + or loader-layer env is set. Pinning the radeon ICD and disabling loader + layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix + hardening; testable as service env vars, zero risk. +- **Power** — GPU DPM is already `performance`, forced level `auto`. The + `power_profile` sysfs file assumed in the Improvements doc does not exist + on this machine (checked 2026-08-15); that item is N/A here. +- Threads: 12 of 16 cores in production. Community reports 12–16 are + equivalent; leave at 12. + +### 6. llama.cpp version watch + +b10433 (2026-08-15) is adequate. Through b10451 there are no +spec-decoding fixes that help us; DFlash's multi-slot corruption is still +open upstream (fix #27117 pending). If that lands, DFlash/DSpark become +re-testable candidates (DSpark was the best single-machine draft before the +ROCm crash). No rebuild planned until then. + +## Quantization landscape (reference) + +The constraint is to keep UD-Q8_K_XL. This section records what exists so +future decisions are informed. + +| Format | Size | Bits/weight | Notes | +|--------|------|-------------|-------| +| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. | +| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). | +| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. | +| Q6_K | ~21.5 GB | 6.6 | Same tier as above. | +| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. | +| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. | + +If the 8-bit PP result (lever 1) is disappointing, the decision order is: +push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a +replacement) → revisit the constraint with Frank. + +## Morning test matrix (2026-08-16) + +Policy: stop both production services, run tests, restore. Each config = +one head restart (~10 min load incl. weight stream; ~2 min with the worker +tensor cache warm). + +| # | Config | Measures | +|---|--------|----------| +| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP | +| B | A + `SPEC_NONE=1` | raw (no spec) PP speed | +| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B | +| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B | +| E | A with n_max 3 and 5 | n_max curve on the split | +| F | A, plain Q8_0 model file | same-bit A/B | +| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap | +| Ref | production frank-default-large (Vulkan, 2 slots) | the bar to beat | + +Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16` +(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP +acceptance, worker VRAM, head GTT usage. + +Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s +production @16k) with the same 8-bit model. + +## Sources + +- GGUF header dump of the production model (this machine, 2026-08-15). +- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` / + `unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face). +- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve, + Q4_0 vs Q8_0 draft head, ROCmFP4 numbers). +- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning, + `-dio` for RPC). +- llama.cpp b10433 source: `src/llama.cpp` (device selection), + `common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol). +- Local benchmarks: [frank-default-large-benchmark](frank-default-large-benchmark.md), + [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md). diff --git a/scripts/frank-pp-head.env b/scripts/frank-pp-head.env new file mode 100644 index 0000000..fe5ecca --- /dev/null +++ b/scripts/frank-pp-head.env @@ -0,0 +1,76 @@ +# Head tunables — Qwen3.8-27B pipeline-parallel head (Dankclaw). +# +# The head is the LOCAL side of the split. It runs llama-server (b10433, +# HIP+Vulkan+RPC build in /home/dank/llama.cpp/build) with --rpc pointing at +# the worker on Dankserver. The model file lives on this machine only; the +# worker receives its tensor share over the network at load time. +# +# CRITICAL: this script must NOT pass --device. When --device is set, +# llama.cpp uses exactly that device list and silently drops the RPC device +# (src/llama.cpp llama_prepare_model_devices). Default device selection +# inserts RPC servers at the FRONT of the device list, so the split becomes +# [RPC0, iGPU] and --tensor-split index 0 is the worker. +# +# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE +# (HIP runtime collision, see the 2026-08-13 TEMP incident). + +MODEL_DIR=/home/dank/models +MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf +HEAD_BIN=/home/dank/llama.cpp/build/bin/llama-server +ALIAS=frank-pp-large +BIND_ADDR=0.0.0.0 +LISTEN_PORT=7770 + +# Comma-separated RPC endpoints (host:port). One worker here. +RPC_SERVERS=192.168.0.235:50052 + +# Model split fractions, one per device in order [RPC0, iGPU]. +# 1,1 = 50/50: the 3090 gets the first ~32 layers, the Strix the rest. +# Leave empty for the default proportional-to-free-memory split; note the +# Strix reports ~96 GiB (GTT ceiling) vs the 3090's ~24 GB, so the default +# lands around 20% of layers on the 3090 and 80% on the Strix. +TENSOR_SPLIT=1,1 + +# Context pool. 131072 single slot keeps the KV small (~4.5 GB total at +# q8_0) on both sides; raise for high-context tests (see docs). +CTX_SIZE=131072 +PARALLEL=1 + +# Speculative decoding. Matches production frank-default-large: MTP (bundled +# head, n_max 4 per the published tuning curve) with the ngram-mod fallback. +# Set SPEC_NONE=1 for a raw baseline with no speculation. +SPEC_NONE=0 +SPEC_DEFAULT=1 +MTP=1 +SPEC_DRAFT_N_MAX=4 +# Optional standalone MTP draft head (e.g. the Q4_0 ggml-org file). When set, +# it replaces the bundled in-model head via --spec-draft-model. +SPEC_DRAFT_MODEL= +SPEC_NGRAM_MOD_N_MIN=48 +SPEC_NGRAM_MOD_N_MAX=64 +SPEC_NGRAM_MOD_N_MATCH=24 + +# KV on GPU, quantized. Keep K == V so Flash Attention stays enabled. +CACHE_TYPE_K=q8_0 +CACHE_TYPE_V=q8_0 + +# Batch sizing: production-verified values (2026-08-14). +BATCH_SIZE=8192 +UBATCH_SIZE=256 + +# Sampling — Qwen3.8 thinking-mode defaults. +TEMPERATURE=1.0 +TOP_K=20 +TOP_P=0.95 +MIN_P=0.0 + +THREADS=12 +THREADS_HTTP=1 + +# 10 (debug): prints per-layer device assignment so the split is verifiable +# in the log. Drop to 4 for quiet production use. +VERBOSITY=10 + +# 1: --direct-io model load. Required in practice for large models loaded +# over RPC (load hangs otherwise; visorcraft Strix RPC finding). +DIRECT_IO=1 diff --git a/scripts/frank-pp-head.service b/scripts/frank-pp-head.service new file mode 100644 index 0000000..411618f --- /dev/null +++ b/scripts/frank-pp-head.service @@ -0,0 +1,16 @@ +[Unit] +Description=llama.cpp Qwen3.8-27B pipeline-parallel head (Strix + RPC to Dankserver) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=dank +Group=dank +Environment="FRANK_PP_HEAD_ENV_FILE=/home/dank/vllmScripts/scripts/frank-pp-head.env" +ExecStart=/home/dank/vllmScripts/scripts/serve-pp-head.sh +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/frank-pp-worker.env b/scripts/frank-pp-worker.env new file mode 100644 index 0000000..d9914e4 --- /dev/null +++ b/scripts/frank-pp-worker.env @@ -0,0 +1,30 @@ +# Worker tunables — Qwen3.8-27B pipeline-parallel RPC worker (Dankserver). +# +# The worker is the REMOTE side of the split. It exposes the RTX 3090 to the +# head on Dankclaw as an RPC device. It runs from the b10433 CUDA+RPC build +# in /home/dank/llama-cpp-rpc (worktree of /home/dank/llama.cpp). +# +# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE. +# The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP +# incident in docs/deployment.md). + +# Bind address and port for ggml-rpc-server. Must be the LAN address the head +# can reach; the default bind (127.0.0.1) refuses remote clients. +RPC_BIND=192.168.0.235 +RPC_PORT=50052 + +# Device to expose. CUDA0 = the RTX 3090. +DEVICE=CUDA0 + +# Threads for the worker's CPU device (server has 8 cores; leave 1 for the OS +# and the running frank-default-small service). +THREADS=7 + +# 1: enable the local tensor file cache (~/.cache/llama.cpp/rpc/). The head +# streams the 31.5 GB model to the worker on first load; with the cache, later +# loads skip the transfer. +CACHE=1 + +# Worker binary. Must be the SAME llama.cpp build as the head (b10433 / +# 9b05354ec). Version skew hangs the RPC handshake. +WORKER_BIN=/home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server diff --git a/scripts/frank-pp-worker.service b/scripts/frank-pp-worker.service new file mode 100644 index 0000000..a748e67 --- /dev/null +++ b/scripts/frank-pp-worker.service @@ -0,0 +1,15 @@ +[Unit] +Description=llama.cpp Qwen3.8-27B pipeline-parallel RPC worker (RTX 3090) +After=network-online.target +Wants=network-online.target + +[Service] +Type=simple +User=dank +Group=dank +ExecStart=/home/dank/vLLM/scripts/serve-pp-worker.sh +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=multi-user.target diff --git a/scripts/pp-bench.py b/scripts/pp-bench.py new file mode 100755 index 0000000..74ceffa --- /dev/null +++ b/scripts/pp-bench.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +"""pp-bench.py — drive the PP head API and report server-side timings. + +Runs on the HEAD machine (Dankclaw). Targets the running PP head +(serve-pp-head.sh, port from argument). Builds a ~Nk-token prose prompt, +sends one warmup completion, then REPS measured completions, and reads the +authoritative `slot print_timing` lines from the head log. + +Usage: + python3 pp-bench.py +""" +import json +import re +import sys +import time +import urllib.request + +prompt_k, out_tokens, reps, port, head_log = sys.argv[1:6] +out_tokens = int(out_tokens) +reps = int(reps) +target = int(prompt_k) * 1000 + +# ~38 tokens per 160-char prose unit; size the filler to hit the target. +unit = ("The careful study of natural language reveals patterns of meaning " + "that emerge from the deliberate arrangement of words and phrases. ") +approx = max(1, target // 38) +prompt = unit * approx + ( + "\n\nNow, in a detailed and flowing paragraph, " + "continue this essay about the philosophy of " + "language and its influence on modern computation. " +) + + +def req(n): + body = json.dumps({"prompt": prompt, "max_tokens": n, + "temperature": 0.0, "stream": False}).encode() + r = urllib.request.Request( + f"http://127.0.0.1:{port}/v1/completions", + data=body, headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(r, timeout=7200) as resp: + return json.load(resp) + + +def log_timings(): + try: + lines = open(head_log, encoding="utf-8", errors="ignore").read().splitlines() + except OSError: + return [] + out = [] + for l in lines: + if re.search(r"prompt eval time|eval time|draft acceptance", l): + out.append(l.strip()) + return out + + +GEN_RE = re.compile( + r"eval time =\s+([\d.]+) ms /\s+(\d+) tokens.*?([\d.]+) tokens per second") +PRE_RE = re.compile( + r"prompt eval time =\s+([\d.]+) ms /\s+(\d+) tokens.*?([\d.]+) tokens per second") +ACC_RE = re.compile(r"draft acceptance = ([\d.]+)") + +# warmup (first run pays graph-compile / JIT costs) +warm = req(16) +print(f"warmup: prompt_tokens={warm['usage']['prompt_tokens']} " + f"completion_tokens={warm['usage']['completion_tokens']}") + +gen_rates, prefill_rates, accepts = [], [], [] +for i in range(reps): + t0 = time.time() + out = req(out_tokens) + dt = time.time() - t0 + u = out["usage"] + pt, ct = u["prompt_tokens"], u["completion_tokens"] + gen = pre = acc = None + for l in log_timings(): + if "prompt eval" not in l: + m = GEN_RE.search(l) + if m: + gen = float(m.group(3)) + m = PRE_RE.search(l) + if m: + pre = float(m.group(3)) + m = ACC_RE.search(l) + if m: + acc = float(m.group(1)) + gen_rates.append(gen if gen is not None else ct / dt) + if pre is not None: + prefill_rates.append(pre) + if acc is not None: + accepts.append(acc) + print(f"rep {i + 1}: prompt={pt} gen={ct} wall={dt:.1f}s " + f"gen={gen_rates[-1]:.2f} t/s " + f"prefill={f'{pre:.1f}' if pre is not None else 'n/a'} t/s " + f"acceptance={f'{acc:.3f}' if acc is not None else 'n/a'}") + +print("== RESULT pp ==") +print(f"avg gen tok/s = {sum(gen_rates) / len(gen_rates):.2f}") +if prefill_rates: + print(f"avg prefill t/s = {sum(prefill_rates) / len(prefill_rates):.1f}") +if accepts: + print(f"avg acceptance = {sum(accepts) / len(accepts):.3f}") +for l in log_timings()[-3:]: + print(" " + l) diff --git a/scripts/pp-test.sh b/scripts/pp-test.sh new file mode 100755 index 0000000..5c87889 --- /dev/null +++ b/scripts/pp-test.sh @@ -0,0 +1,195 @@ +#!/usr/bin/env bash +# pp-test.sh — Qwen3.8-27B pipeline-parallel (llama.cpp RPC) test orchestrator. +# +# Runs Qwen3.8-27B UD-Q8_K_XL split across Dankclaw (head, Strix iGPU) and +# Dankserver (worker, RTX 3090) over the 2.5 GbE LAN. Both boxes run the same +# llama.cpp build (b10433 / 9b05354ec); the worker build lives in +# /home/dank/llama-cpp-rpc on Dankserver. +# +# BENCHMARKING POLICY (AGENTS.md): this test STOPS both production services +# and restores them when done. Do not run it while users are mid-stream. +# +# Usage (from any box with ssh access to both targets, repo checked out): +# ./scripts/pp-test.sh preflight +# ./scripts/pp-test.sh start # stop prod, start PP head+worker +# ./scripts/pp-test.sh verify # show the layer split + VRAM +# ./scripts/pp-test.sh bench [prompt_k] [out_tokens] [reps] +# ./scripts/pp-test.sh stop-pp # stop PP head+worker only +# ./scripts/pp-test.sh restore # start production services +# ./scripts/pp-test.sh all # start + verify + bench + restore +# +# Sudo: set FRANK_SUDO_PW for headless runs (stop/start of system services). +# FRANK_SUDO_PW=... ./scripts/pp-test.sh start +set -euo pipefail + +CLAW="dank@dankclaw" +SERVER="dank@dankserver" +CLAW_REPO="/home/dank/vllmScripts" +SERVER_REPO="/home/dank/vLLM" +HEAD_PORT=7770 +HEAD_LOG="/tmp/pp-head.log" +WORKER_LOG="/tmp/pp-worker.log" +RPC_PORT=50052 + +SUDO="sudo" +if [ -n "${FRANK_SUDO_PW:-}" ]; then + SUDO="printf '%s\n' '$FRANK_SUDO_PW' | sudo -S" +fi + +log() { echo "[pp-test] $*"; } +die() { echo "[pp-test] FAIL: $*" >&2; exit 1; } + +ssh_claw() { ssh -o BatchMode=yes "$CLAW" "$@"; } +ssh_server() { ssh -o BatchMode=yes "$SERVER" "$@"; } + +# --------------------------------------------------------------------------- +# preflight +# --------------------------------------------------------------------------- +do_preflight() { + log "preflight: build identity" + local h_v w_v + h_v=$(ssh_claw '/home/dank/llama.cpp/build/bin/llama-server --version 2>&1 | head -1') + w_v=$(ssh_server '/home/dank/llama-cpp-rpc/build/bin/llama-server --version 2>&1 | head -1') + log " head: $h_v" + log " worker: $w_v" + [[ "$h_v" == *9b05354ec* && "$w_v" == *9b05354ec* ]] || die "build skew (need identical commit 9b05354ec on both sides)" + + log "preflight: binaries" + ssh_claw "[ -x $CLAW_REPO/scripts/serve-pp-head.sh ]" || die "missing $CLAW_REPO/scripts/serve-pp-head.sh on claw (git pull?)" + ssh_server "[ -x $SERVER_REPO/scripts/serve-pp-worker.sh ]" || die "missing $SERVER_REPO/scripts/serve-pp-worker.sh on server (git pull?)" + ssh_server "[ -x /home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server ]" || die "missing worker RPC binary on server" + ssh_claw "[ -f /home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf ]" || die "missing model on claw" + + log "preflight: network (claw -> server:50052 listener check needs the worker up; testing route + link)" + ssh_claw "ip -4 addr show eno1 | grep -q 192.168.0.143" || die "unexpected claw address" + ssh_server "ip -4 addr show eno1 | grep -q 192.168.0.235" || die "unexpected server address" + ssh_claw "ping -c 2 -W 2 192.168.0.235 >/dev/null" || die "claw cannot reach server" + local rtt + rtt=$(ssh_claw "ping -c 5 -W 2 192.168.0.235 | tail -1" | grep -oE '[0-9.]+/[0-9.]+ ms' | head -1 || true) + log " rtt: ${rtt:-unknown}" + + log "preflight: production services" + ssh_claw "$SUDO systemctl is-active frank-default-large" 2>/dev/null || true + ssh_server "$SUDO systemctl is-active frank-default-small" 2>/dev/null || true + log "preflight OK" +} + +# --------------------------------------------------------------------------- +# start +# --------------------------------------------------------------------------- +do_start() { + log "stopping production services (both boxes)" + ssh_claw "$SUDO systemctl stop frank-default-large" || die "could not stop frank-default-large" + ssh_server "$SUDO systemctl stop frank-default-small" || die "could not stop frank-default-small" + sleep 5 + + log "dropping page caches on claw (reliable model load)" + ssh_claw "$SUDO sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'" || log "WARN: cache drop failed" + + log "starting RPC worker on server" + ssh_server "cd $SERVER_REPO && nohup bash scripts/serve-pp-worker.sh > $WORKER_LOG 2>&1 & echo worker_pid=\$!" + for i in $(seq 1 30); do + if ssh_server "ss -tln | grep -q :$RPC_PORT"; then + log "worker listening on :$RPC_PORT" + break + fi + [ "$i" = "30" ] && { ssh_server "tail -20 $WORKER_LOG"; die "worker did not bind :$RPC_PORT"; } + sleep 1 + done + + log "starting PP head on claw (model load + weight stream: allow ~10 min)" + local head_pid + head_pid=$(ssh_claw "cd $CLAW_REPO && nohup bash scripts/serve-pp-head.sh > $HEAD_LOG 2>&1 & echo \$!") + log "head pid on claw: $head_pid" + for i in $(seq 1 360); do + local st + st=$(ssh_claw "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$HEAD_PORT/health 2>/dev/null || true") + if [ "$st" = "200" ]; then + log "head healthy after ~$((i * 5))s" + return 0 + fi + if ! ssh_claw "kill -0 $head_pid 2>/dev/null"; then + ssh_claw "tail -30 $HEAD_LOG" + die "head process died during load" + fi + [ $((i % 12)) = 0 ] && log " still loading... ($((i * 5))s)" + sleep 5 + done + ssh_claw "tail -30 $HEAD_LOG" + die "head did not become healthy in 30 min" +} + +# --------------------------------------------------------------------------- +# verify +# --------------------------------------------------------------------------- +do_verify() { + log "health + device split (from head log)" + ssh_claw "curl -s http://127.0.0.1:$HEAD_PORT/health; echo" + ssh_claw "grep -aE 'using device|assigned to device|MiB free' $HEAD_LOG | head -20 || true" + log "worker VRAM on server:" + ssh_server "nvidia-smi --query-gpu=memory.used --format=csv,noheader" + log "worker log tail:" + ssh_server "tail -5 $WORKER_LOG" +} + +# --------------------------------------------------------------------------- +# bench +# --------------------------------------------------------------------------- +do_bench() { + local PROMPT_K="${1:-4}" + local OUT_TOKENS="${2:-256}" + local REPS="${3:-2}" + log "bench: prompt ~${PROMPT_K}k tokens, out=${OUT_TOKENS}, reps=${REPS} (on claw, via :$HEAD_PORT)" + ssh_claw "python3 $CLAW_REPO/scripts/pp-bench.py $PROMPT_K $OUT_TOKENS $REPS $HEAD_PORT $HEAD_LOG" +} + +# --------------------------------------------------------------------------- +# stop / restore +# --------------------------------------------------------------------------- +do_stop_pp() { + log "stopping PP head (claw) and worker (server)" + ssh_claw "pkill -f 'serve-pp-head.sh' 2>/dev/null; pkill -f 'llama-server.*frank-pp-large' 2>/dev/null; true" + ssh_server "pkill -x ggml-rpc-server 2>/dev/null; true" + sleep 3 + ssh_server "nvidia-smi --query-gpu=memory.used --format=csv,noheader" +} + +do_restore() { + log "starting production services" + ssh_server "$SUDO systemctl start frank-default-small" || die "could not start frank-default-small" + ssh_claw "$SUDO systemctl start frank-default-large" || die "could not start frank-default-large" + log "waiting for health (model load takes minutes; up to ~17 min cold on claw)" + for i in $(seq 1 300); do + local a b + a=$(ssh_claw "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:7770/health 2>/dev/null || true") + b=$(ssh_server "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:7770/health 2>/dev/null || true") + if [ "$a" = "200" ] && [ "$b" = "200" ]; then + log "both production services healthy after ~$((i * 4))s" + return 0 + fi + [ $((i % 15)) = 0 ] && log " claw=$a server=$b ... ($((i * 4))s)" + sleep 4 + done + die "production services did not both become healthy" +} + +# --------------------------------------------------------------------------- +case "${1:-}" in + preflight) do_preflight ;; + start) do_start ;; + verify) do_verify ;; + bench) shift; do_bench "$@" ;; + stop-pp) do_stop_pp ;; + restore) do_restore ;; + all) + do_preflight + do_start + do_verify + do_bench + do_restore + ;; + *) + echo "Usage: $0 {preflight|start|verify|bench [prompt_k] [out] [reps]|stop-pp|restore|all}" + exit 1 + ;; +esac diff --git a/scripts/serve-pp-head.sh b/scripts/serve-pp-head.sh new file mode 100755 index 0000000..ebac11d --- /dev/null +++ b/scripts/serve-pp-head.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# Serve the Qwen3.8-27B pipeline-parallel head on Dankclaw. +# +# Runs llama-server (b10433) with --rpc pointing at the worker on +# Dankserver. The model is split pipeline-style across [RPC0, iGPU]: +# layers run in order, hidden states cross the ethernet boundary once per +# token step. +# +# IMPORTANT: no --device flag. The RPC device only joins the model's device +# list through default device selection; an explicit --device silently drops +# it. See the header of frank-pp-head.env. +# +# Tunables come from frank-pp-head.env in this directory unless the +# FRANK_PP_HEAD_ENV_FILE environment variable overrides that path. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${FRANK_PP_HEAD_ENV_FILE:-$SCRIPT_DIR/frank-pp-head.env}" +if [[ -f "$ENV_FILE" ]]; then + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a +fi + +# --- defaults (used when the env file does not set them) ------------------- +: "${MODEL_DIR:=/home/dank/models}" +: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}" +: "${HEAD_BIN:=/home/dank/llama.cpp/build/bin/llama-server}" +: "${ALIAS:=frank-pp-large}" +: "${BIND_ADDR:=0.0.0.0}" +: "${LISTEN_PORT:=7770}" +: "${RPC_SERVERS:=192.168.0.235:50052}" +: "${TENSOR_SPLIT:=1,1}" +: "${CTX_SIZE:=131072}" +: "${PARALLEL:=1}" +: "${SPEC_NONE:=0}" +: "${SPEC_DEFAULT:=1}" +: "${MTP:=0}" +: "${SPEC_DRAFT_N_MAX:=4}" +: "${SPEC_DRAFT_MODEL:=}" +: "${SPEC_NGRAM_MOD_N_MIN:=48}" +: "${SPEC_NGRAM_MOD_N_MAX:=64}" +: "${SPEC_NGRAM_MOD_N_MATCH:=24}" +: "${CACHE_TYPE_K:=q8_0}" +: "${CACHE_TYPE_V:=q8_0}" +: "${BATCH_SIZE:=8192}" +: "${UBATCH_SIZE:=256}" +: "${TEMPERATURE:=1.0}" +: "${TOP_K:=20}" +: "${TOP_P:=0.95}" +: "${MIN_P:=0.0}" +: "${THREADS:=12}" +: "${THREADS_HTTP:=1}" +: "${VERBOSITY:=4}" +: "${DIRECT_IO:=1}" + +MODEL="$MODEL_DIR/$MODEL_FILE" + +if [[ ! -f "$MODEL" ]]; then + echo "missing required file: $MODEL" >&2 + exit 1 +fi +if [[ ! -x "$HEAD_BIN" ]]; then + echo "missing head binary: $HEAD_BIN" >&2 + exit 1 +fi + +# --- RPC endpoints ----------------------------------------------------------- +RPC_ARGS=() +for s in ${RPC_SERVERS//,/ }; do + RPC_ARGS+=(--rpc "$s") +done + +# --- split fractions --------------------------------------------------------- +SPLIT_ARGS=() +if [[ -n "${TENSOR_SPLIT}" ]]; then + SPLIT_ARGS+=(--tensor-split "${TENSOR_SPLIT}") +fi + +# --- speculative decoding ----------------------------------------------------- +SPEC_ARGS=() +if [[ "${SPEC_NONE}" == "1" ]]; then + SPEC_ARGS+=(--spec-type none) +else + if [[ "${SPEC_DEFAULT}" == "1" ]]; then + if [[ "${MTP}" == "1" ]]; then + # MTP first (bundled in-model head by default, or a standalone + # draft file when SPEC_DRAFT_MODEL is set), then the ngram-mod + # fallback. Order matters: --spec-type appends to the impl chain. + SPEC_ARGS+=(--spec-type draft-mtp) + SPEC_ARGS+=(--spec-draft-n-max "${SPEC_DRAFT_N_MAX}") + if [[ -n "${SPEC_DRAFT_MODEL}" ]]; then + if [[ ! -f "${SPEC_DRAFT_MODEL}" ]]; then + echo "SPEC_DRAFT_MODEL file missing: ${SPEC_DRAFT_MODEL}" >&2 + exit 1 + fi + SPEC_ARGS+=(--spec-draft-model "${SPEC_DRAFT_MODEL}") + fi + fi + SPEC_ARGS+=(--spec-default) + SPEC_ARGS+=(--spec-ngram-mod-n-min "${SPEC_NGRAM_MOD_N_MIN}") + SPEC_ARGS+=(--spec-ngram-mod-n-max "${SPEC_NGRAM_MOD_N_MAX}") + SPEC_ARGS+=(--spec-ngram-mod-n-match "${SPEC_NGRAM_MOD_N_MATCH}") + else + echo "SPEC_NONE=0 requires SPEC_DEFAULT=1" >&2 + exit 1 + fi +fi + +# --- load mode ---------------------------------------------------------------- +LOAD_ARGS=() +if [[ "${DIRECT_IO}" == "1" ]]; then + LOAD_ARGS+=(--direct-io) +fi + +exec "$HEAD_BIN" \ + --model "$MODEL" \ + "${RPC_ARGS[@]}" \ + "${SPLIT_ARGS[@]}" \ + "${SPEC_ARGS[@]}" \ + "${LOAD_ARGS[@]}" \ + --gpu-layers all \ + --fit off \ + --cache-type-k "$CACHE_TYPE_K" \ + --cache-type-v "$CACHE_TYPE_V" \ + --ctx-size "$CTX_SIZE" \ + --parallel "$PARALLEL" \ + --batch-size "$BATCH_SIZE" \ + --ubatch-size "$UBATCH_SIZE" \ + --temp "$TEMPERATURE" \ + --top-k "$TOP_K" \ + --top-p "$TOP_P" \ + --min-p "$MIN_P" \ + --threads "$THREADS" \ + --threads-http "$THREADS_HTTP" \ + --alias "$ALIAS" \ + -lv "$VERBOSITY" \ + --host "$BIND_ADDR" \ + --port "$LISTEN_PORT" diff --git a/scripts/serve-pp-worker.sh b/scripts/serve-pp-worker.sh new file mode 100755 index 0000000..aae8600 --- /dev/null +++ b/scripts/serve-pp-worker.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# Serve the Qwen3.8-27B pipeline-parallel RPC worker on Dankserver. +# +# Runs ggml-rpc-server from the b10433 CUDA+RPC build and exposes the RTX +# 3090 as a remote device for the head on Dankclaw. +# +# Tunables come from frank-pp-worker.env in this directory unless the +# FRANK_PP_WORKER_ENV_FILE environment variable overrides that path. +# +# Security note (upstream): the RPC server is "fragile and insecure". No +# authentication, no encryption. Keep it on the trusted LAN only. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${FRANK_PP_WORKER_ENV_FILE:-$SCRIPT_DIR/frank-pp-worker.env}" +if [[ -f "$ENV_FILE" ]]; then + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a +fi + +# --- defaults (used when the env file does not set them) ------------------- +: "${RPC_BIND:=192.168.0.235}" +: "${RPC_PORT:=50052}" +: "${DEVICE:=CUDA0}" +: "${THREADS:=7}" +: "${CACHE:=1}" +: "${WORKER_BIN:=/home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server}" + +if [[ ! -x "$WORKER_BIN" ]]; then + echo "missing worker binary: $WORKER_BIN" >&2 + echo "Build it: cd /home/dank/llama-cpp-rpc && cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release -- -j 8" >&2 + exit 1 +fi + +ARGS=() +if [[ "${CACHE}" == "1" ]]; then + ARGS+=(--cache) +fi + +exec "$WORKER_BIN" \ + --host "$RPC_BIND" \ + --port "$RPC_PORT" \ + --device "$DEVICE" \ + --threads "$THREADS" \ + "${ARGS[@]}"