138 lines
6.8 KiB
Markdown
138 lines
6.8 KiB
Markdown
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||
|
||
Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||
3090 (worker) using llama.cpp RPC. This page is the design record and the
|
||
morning-of runbook. Research background:
|
||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||
|
||
## Topology
|
||
|
||
```
|
||
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
|
||
llama-server b10433 ggml-rpc-server b10433
|
||
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
|
||
model file (31.5 GB) -d CUDA0 -c (tensor cache)
|
||
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0..~31
|
||
Strix: layers ~32..64 + lm_head
|
||
hidden state ~20 KiB crosses the link once per token, each direction
|
||
```
|
||
|
||
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
|
||
the worker, the rest on the head, in order. One hidden-state handoff per
|
||
token step. RTT-bound, not bandwidth-bound.
|
||
- **Why the head is on Dankclaw:** the model file is already there; the
|
||
worker streams its tensor share at load (no 31.5 GB copy to the server).
|
||
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
|
||
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
|
||
so the production b10121 build stays untouched. A version skew hangs the
|
||
RPC handshake with no error.
|
||
|
||
## Files
|
||
|
||
| File | Machine | Purpose |
|
||
|------|---------|---------|
|
||
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
|
||
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
|
||
| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
|
||
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
|
||
| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
|
||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||
|
||
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
|
||
|
||
1. **No `--device` on the head.** Setting `params.devices` makes
|
||
`llama_prepare_model_devices` use exactly that list; the RPC device is
|
||
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
|
||
puts RPC first.
|
||
2. **The iGPU side is ROCm by default.** Backends register in fixed order
|
||
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
|
||
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
|
||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
|
||
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
|
||
decode (measured on this box).
|
||
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
|
||
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
|
||
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
|
||
with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
|
||
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
|
||
I/O.
|
||
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
|
||
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
|
||
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
|
||
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
|
||
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
|
||
the head before load (warm-cache loads crash fast; cold loads survive).
|
||
|
||
## Runbook (morning of 2026-08-16)
|
||
|
||
Run from any box with ssh access to both (repo path on claw:
|
||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
|
||
|
||
```
|
||
# 0. one-time: pull the kit on both boxes
|
||
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
|
||
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
|
||
|
||
# 1. checks (read-only)
|
||
./scripts/pp-test.sh preflight
|
||
|
||
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
|
||
|
||
# or step by step:
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
|
||
./scripts/pp-test.sh verify # layer split + worker VRAM
|
||
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
|
||
./scripts/pp-test.sh bench 16 256 1
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
|
||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
|
||
```
|
||
|
||
Config variants: edit `scripts/frank-default-large.env` on the claw (git commit +
|
||
pull if the bench runs from the repo), then re-run `start` + `bench`:
|
||
|
||
- No spec baseline: `SPEC_NONE=1`
|
||
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
|
||
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
|
||
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
|
||
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
|
||
|
||
Verification checklist after each `start`:
|
||
|
||
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
|
||
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
|
||
1,1 split.
|
||
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
|
||
(production small service + worker share).
|
||
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
|
||
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
|
||
|
||
## Expected numbers
|
||
|
||
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
|
||
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
|
||
|
||
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
|
||
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
|
||
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
|
||
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
|
||
|
||
## Rollback
|
||
|
||
`pp-test.sh restore` starts both production services. If a PP test wedges a
|
||
box: `systemctl restart <unit>` after killing the stray processes
|
||
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
|
||
with a pattern matching your own ssh command line).
|
||
|
||
## Open questions for the test day
|
||
|
||
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
|
||
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
|
||
in the head log (`draft acceptance = X`).
|
||
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
|
||
serialized per compute — expect the 2-slot speedup to be weaker than
|
||
single-node. Measure, do not assume.
|
||
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
|
||
the ubatch crossing the RPC boundary is the suspect.
|