Frank
aa1941d592
- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53 (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the KV-per-device rounding rule (round(f x 66)), and why 35 blocks is unreachable (9-FA flip at f~0.538). - PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache entries -> corrupt weights -> degenerate output; fix + signature. - deployment.md + AGENTS.md + index.md: numbers brought current. - scripts/prune-rpc-cache.sh: cron guard capping the worker cache at 40 GiB (oldest first) to prevent the disk-full recurrence.
202 lines
9.9 KiB
Markdown
202 lines
9.9 KiB
Markdown
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||
|
||
Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||
3090 (worker) using llama.cpp RPC. This page is the design record and the
|
||
morning-of runbook. Research background:
|
||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||
|
||
## Production state (2026-08-17: single 262k slot, split 0.53)
|
||
|
||
This split is the production `frank-default-large` service. Current
|
||
state, verified against the live journal and output smoke test
|
||
(2026-08-17, load 01:21):
|
||
|
||
- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
|
||
provably-safe value; see "Split limits" below for the boundary
|
||
analysis and the failed 0.54/0.55 attempts.
|
||
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
|
||
- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
|
||
fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
|
||
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
|
||
Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
|
||
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
|
||
- Unit `frank-default-large.service`, launched through
|
||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||
|
||
## Split limits (measured 2026-08-17)
|
||
|
||
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
|
||
With the pool shrunk to 262144, the KV per full-attention layer drops to
|
||
exactly 1024 MiB, freeing headroom. Measurements:
|
||
|
||
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
|
||
free. Healthy (2026-08-17 00:29 load).
|
||
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
|
||
cudaMalloc OOM by ~740 MiB.
|
||
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
|
||
The KV-per-device count rounds the split fraction over 66 layers
|
||
(round(f x 66)), independent of the weight boundary; the 9-FA flip
|
||
sits at f ~= 0.538, just below the weight threshold for block 35, so
|
||
the 35-block config is unreachable.
|
||
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
|
||
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
|
||
|
||
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
|
||
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
|
||
includes the MTP head for weights and token_embd for KV — the two
|
||
boundaries diverge, which is why 0.54 split them apart.
|
||
|
||
## Cache incident (2026-08-17): garbage output from a full worker disk
|
||
|
||
Symptom: after a split change, the service loaded "healthy" but every
|
||
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
|
||
content, degenerate logits). Root cause chain:
|
||
|
||
1. The worker RPC tensor cache keys per session. Each model load writes
|
||
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
|
||
4434 files / 96 GB.
|
||
2. The Dankserver root disk filled to 100%. Cache writes started
|
||
failing, leaving 0-byte entries. A subsequent reload hit corrupt
|
||
cache entries and the worker served garbage weights for the
|
||
newly-needed layers without validation.
|
||
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
|
||
worker, free the disk, reload the head (full re-stream ~2.5 min).
|
||
A clean cache and a healthy disk produced correct output immediately
|
||
(acceptance 0.89 on the first smoke test).
|
||
|
||
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
|
||
Never let the cache directory grow unchecked; a full disk silently
|
||
corrupts it. Signature to recognize: healthy load + degenerate repeated-
|
||
token output.
|
||
|
||
## Topology
|
||
|
||
```
|
||
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
|
||
llama-server b10433 ggml-rpc-server b10433
|
||
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
|
||
model file (31.5 GB) -d CUDA0 -c (tensor cache)
|
||
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0..~31
|
||
Strix: layers ~32..64 + lm_head
|
||
hidden state ~20 KiB crosses the link once per token, each direction
|
||
```
|
||
|
||
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
|
||
the worker, the rest on the head, in order. One hidden-state handoff per
|
||
token step. RTT-bound, not bandwidth-bound.
|
||
- **Why the head is on Dankclaw:** the model file is already there; the
|
||
worker streams its tensor share at load (no 31.5 GB copy to the server).
|
||
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
|
||
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
|
||
so the production b10121 build stays untouched. A version skew hangs the
|
||
RPC handshake with no error.
|
||
|
||
## Files
|
||
|
||
| File | Machine | Purpose |
|
||
|------|---------|---------|
|
||
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
|
||
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
|
||
| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
|
||
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
|
||
| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
|
||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||
|
||
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
|
||
|
||
1. **No `--device` on the head.** Setting `params.devices` makes
|
||
`llama_prepare_model_devices` use exactly that list; the RPC device is
|
||
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
|
||
puts RPC first.
|
||
2. **The iGPU side is ROCm by default.** Backends register in fixed order
|
||
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
|
||
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
|
||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
|
||
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
|
||
decode (measured on this box).
|
||
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
|
||
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
|
||
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
|
||
with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
|
||
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
|
||
I/O.
|
||
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
|
||
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
|
||
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
|
||
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
|
||
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
|
||
the head before load (warm-cache loads crash fast; cold loads survive).
|
||
|
||
## Runbook (morning of 2026-08-16)
|
||
|
||
Run from any box with ssh access to both (repo path on claw:
|
||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
|
||
|
||
```
|
||
# 0. one-time: pull the kit on both boxes
|
||
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
|
||
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
|
||
|
||
# 1. checks (read-only)
|
||
./scripts/pp-test.sh preflight
|
||
|
||
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
|
||
|
||
# or step by step:
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
|
||
./scripts/pp-test.sh verify # layer split + worker VRAM
|
||
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
|
||
./scripts/pp-test.sh bench 16 256 1
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
|
||
```
|
||
|
||
Config variants: edit `scripts/frank-default-large.env` on the claw (git commit +
|
||
pull if the bench runs from the repo), then re-run `start` + `bench`:
|
||
|
||
- No spec baseline: `SPEC_NONE=1`
|
||
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
|
||
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
|
||
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
|
||
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
|
||
|
||
Verification checklist after each `start`:
|
||
|
||
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
|
||
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
|
||
1,1 split.
|
||
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
|
||
(production small service + worker share).
|
||
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
|
||
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
|
||
|
||
## Expected numbers
|
||
|
||
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
|
||
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
|
||
|
||
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
|
||
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
|
||
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
|
||
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
|
||
|
||
## Rollback
|
||
|
||
`pp-test.sh restore` starts both production services. If a PP test wedges a
|
||
box: `systemctl restart <unit>` after killing the stray processes
|
||
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
|
||
with a pattern matching your own ssh command line).
|
||
|
||
## Open questions for the test day
|
||
|
||
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
|
||
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
|
||
in the head log (`draft acceptance = X`).
|
||
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
|
||
serialized per compute — expect the 2-slot speedup to be weaker than
|
||
single-node. Measure, do not assume.
|
||
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
|
||
the ubatch crossing the RPC boundary is the suspect.
|