PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py - head design verified in b10433 source + live 4B smoke test: * no --device with --rpc (RPC device silently dropped otherwise) * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan) * default split proportional to reported free mem = ~20/80 here, pin 1,1 * -dio required for large model over RPC - docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers, quant landscape, 2026-08-16 test matrix), PP deployment runbook, index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs) - worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC) - second head build build-vulkan-rpc (HIP off) for backend A/B
This commit is contained in:
Frank committed
1 parent
c775077016
commit
9544c36afc
13 files changed
+1037
-5
No files matched your search
@@ -0,0 +1,137 @@
|
||||
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||||
|
||||
Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||||
3090 (worker) using llama.cpp RPC. This page is the design record and the
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
|
||||
llama-server b10433 ggml-rpc-server b10433
|
||||
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
|
||||
model file (31.5 GB) -d CUDA0 -c (tensor cache)
|
||||
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0..~31
|
||||
Strix: layers ~32..64 + lm_head
|
||||
hidden state ~20 KiB crosses the link once per token, each direction
|
||||
```
|
||||
|
||||
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
|
||||
the worker, the rest on the head, in order. One hidden-state handoff per
|
||||
token step. RTT-bound, not bandwidth-bound.
|
||||
- **Why the head is on Dankclaw:** the model file is already there; the
|
||||
worker streams its tensor share at load (no 31.5 GB copy to the server).
|
||||
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
|
||||
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
|
||||
so the production b10121 build stays untouched. A version skew hangs the
|
||||
RPC handshake with no error.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Machine | Purpose |
|
||||
|------|---------|---------|
|
||||
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
|
||||
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
|
||||
| `scripts/frank-pp-head.env` | claw | head tunables (split, ctx, spec, KV, I/O) |
|
||||
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
|
||||
| `scripts/frank-pp-head.service` / `frank-pp-worker.service` | both | optional systemd units (manual install) |
|
||||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||||
|
||||
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
|
||||
|
||||
1. **No `--device` on the head.** Setting `params.devices` makes
|
||||
`llama_prepare_model_devices` use exactly that list; the RPC device is
|
||||
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
|
||||
puts RPC first.
|
||||
2. **The iGPU side is ROCm by default.** Backends register in fixed order
|
||||
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
|
||||
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
|
||||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
|
||||
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
|
||||
decode (measured on this box).
|
||||
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
|
||||
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
|
||||
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
|
||||
with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
|
||||
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
|
||||
I/O.
|
||||
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
|
||||
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
|
||||
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
|
||||
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
|
||||
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
|
||||
the head before load (warm-cache loads crash fast; cold loads survive).
|
||||
|
||||
## Runbook (morning of 2026-08-16)
|
||||
|
||||
Run from any box with ssh access to both (repo path on claw:
|
||||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
|
||||
|
||||
```
|
||||
# 0. one-time: pull the kit on both boxes
|
||||
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
|
||||
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
|
||||
|
||||
# 1. checks (read-only)
|
||||
./scripts/pp-test.sh preflight
|
||||
|
||||
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
|
||||
|
||||
# or step by step:
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
|
||||
./scripts/pp-test.sh verify # layer split + worker VRAM
|
||||
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
|
||||
./scripts/pp-test.sh bench 16 256 1
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
|
||||
```
|
||||
|
||||
Config variants: edit `scripts/frank-pp-head.env` on the claw (git commit +
|
||||
pull if the bench runs from the repo), then re-run `start` + `bench`:
|
||||
|
||||
- No spec baseline: `SPEC_NONE=1`
|
||||
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
|
||||
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
|
||||
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
|
||||
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
|
||||
|
||||
Verification checklist after each `start`:
|
||||
|
||||
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
|
||||
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
|
||||
1,1 split.
|
||||
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
|
||||
(production small service + worker share).
|
||||
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
|
||||
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
|
||||
|
||||
## Expected numbers
|
||||
|
||||
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
|
||||
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
|
||||
|
||||
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
|
||||
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
|
||||
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
|
||||
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
|
||||
|
||||
## Rollback
|
||||
|
||||
`pp-test.sh restore` starts both production services. If a PP test wedges a
|
||||
box: `systemctl restart <unit>` after killing the stray processes
|
||||
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
|
||||
with a pattern matching your own ssh command line).
|
||||
|
||||
## Open questions for the test day
|
||||
|
||||
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
|
||||
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
|
||||
in the head log (`draft acceptance = X`).
|
||||
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
|
||||
serialized per compute — expect the 2-slot speedup to be weaker than
|
||||
single-node. Measure, do not assume.
|
||||
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
|
||||
the ubatch crossing the RPC boundary is the suspect.
|
||||
Reference in new issue
Block a user