serverAppConfigs/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md

Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)

Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX 3090 (worker) using llama.cpp RPC. This page is the design record and the morning-of runbook. Research background: Qwen3.8 Strix optimizations.

Production state (2026-08-17: single 262k slot, split 0.53)

This split is the production frank-default-large service. Current state, verified against the live journal and output smoke test (2026-08-17, load 01:21):

• Split 0.53,0.47 = 34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe value; see "Split limits" below for the boundary analysis and the failed 0.54/0.55 attempts.
• Pool 262144, PARALLEL=1, live n_ctx_slot = 262144.
• MTP enabled: --spec-type draft-mtp --spec-draft-n-max 4 + ngram-mod fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
• Head GPU side: ROCm0 (no --device flag; HIP registers before Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
• Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
• Unit frank-default-large.service, launched through serve-pp-head.sh + frank-default-large.env.

Split limits (measured 2026-08-17)

The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks. With the pool shrunk to 262144, the KV per full-attention layer drops to exactly 1024 MiB, freeing headroom. Measurements:

• 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB free. Healthy (2026-08-17 00:29 load).
• 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB -> cudaMalloc OOM by ~740 MiB.
• 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM. The KV-per-device count rounds the split fraction over 66 layers (round(f x 66)), independent of the weight boundary; the 9-FA flip sits at f ~= 0.538, just below the weight threshold for block 35, so the 35-block config is unreachable.
• 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.

GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each), linear blocks 431.3 MB; total layer weight 27720 MB. The split pool includes the MTP head for weights and token_embd for KV — the two boundaries diverge, which is why 0.54 split them apart.

Cache incident (2026-08-17): garbage output from a full worker disk

Symptom: after a split change, the service loaded "healthy" but every completion degenerated to a repeated junk token (e.g. 48 x "/", empty content, degenerate logits). Root cause chain:

1. The worker RPC tensor cache keys per session. Each model load writes a fresh full copy (~15 GB); over ~5 sessions of two models it grew to 4434 files / 96 GB.
2. The Dankserver root disk filled to 100%. Cache writes started failing, leaving 0-byte entries. A subsequent reload hit corrupt cache entries and the worker served garbage weights for the newly-needed layers without validation.
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the worker, free the disk, reload the head (full re-stream ~2.5 min). A clean cache and a healthy disk produced correct output immediately (acceptance 0.89 on the first smoke test).

Guard: scripts/prune-rpc-cache.sh on Dankserver (cron, 40 GiB cap). Never let the cache directory grow unchecked; a full disk silently corrupts it. Signature to recognize: healthy load + degenerate repeated- token output.

Topology

Dankclaw (head, 192.168.0.143)          Dankserver (worker, 192.168.0.235)
llama-server b10433                      ggml-rpc-server b10433
  --rpc 192.168.0.235:50052                 --host 192.168.0.235 --port 50052
  model file (31.5 GB)                     -d CUDA0 -c (tensor cache)
  [RPC0, ROCm0|Vulkan0]  layers            RTX 3090: layers 0..~31
  Strix:    layers ~32..64 + lm_head
  hidden state ~20 KiB crosses the link once per token, each direction

• Layer-sequential split (true pipeline parallelism): layers 0..N run on the worker, the rest on the head, in order. One hidden-state handoff per token step. RTT-bound, not bandwidth-bound.
• Why the head is on Dankclaw: the model file is already there; the worker streams its tensor share at load (no 31.5 GB copy to the server).
• Both sides must run the identical build (b10433 / 9b05354ec). The worker build lives in a worktree /home/dank/llama-cpp-rpc on Dankserver so the production b10121 build stays untouched. A version skew hangs the RPC handshake with no error.

Files

File	Machine	Purpose
scripts/frank-pp-worker.env	server	worker tunables (bind, port, device, cache)
scripts/serve-pp-worker.sh	server	runs ggml-rpc-server from the b10433 worktree build
scripts/frank-default-large.env	claw	head tunables (split, ctx, spec, KV, I/O); was frank-pp-head.env before the 2026-08-16 rename
scripts/serve-pp-head.sh	claw	runs llama-server --rpc ... (see "Gotchas")
scripts/frank-default-large.service / frank-pp-worker.service	both	optional systemd units (manual install); the head unit was renamed from frank-pp-head.service on 2026-08-16 and is symlinked from the repo into /etc/systemd/system
scripts/pp-test.sh	any box with ssh to both	test orchestrator (preflight/start/verify/bench/stop-pp/restore)
scripts/pp-bench.py	claw	API driver; reports server-side print_timing

Gotchas (all verified in b10433 source / live tests, 2026-08-15)

1. No --device on the head. Setting params.devices makes llama_prepare_model_devices use exactly that list; the RPC device is silently dropped and --rpc becomes a no-op. Omit it; default selection puts RPC first.
2. The iGPU side is ROCm by default. Backends register in fixed order (HIP before Vulkan); the iGPU dedup rule keeps the first backend's view. Live check showed [RPC0, ROCm0]. For a Vulkan side, use the second head build /home/dank/llama.cpp/build-vulkan-rpc (HIP off) — set HEAD_BIN in the env. Trade: ROCm side = faster prefill, ~3% slower decode (measured on this box).
3. Pin --tensor-split 1,1. The default split is proportional to reported free memory: the Strix reports up to 96 GiB (GTT) vs the 3090's 24 GB, so the default lands ~20/80 on the slow device (verified with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
4. -dio on the head. Large model + RPC hangs at load without direct I/O.
5. RPC is unauthenticated and unencrypted. LAN only. The worker binds 0.0.0.0-equivalent by design (--host 192.168.0.235).
6. Env var names. Never TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads them; the 2026-08-13 TEMP incident).
7. Page-cache state confounds loads. pp-test.sh start drops caches on the head before load (warm-cache loads crash fast; cold loads survive).

Runbook (morning of 2026-08-16)

Run from any box with ssh access to both (repo path on claw: /home/dank/vllmScripts, on server: /home/dank/vLLM).

# 0. one-time: pull the kit on both boxes
ssh dank@dankclaw   'cd /home/dank/vllmScripts  && git pull'
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'

# 1. checks (read-only)
./scripts/pp-test.sh preflight

# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all

# or step by step:
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
./scripts/pp-test.sh verify          # layer split + worker VRAM
./scripts/pp-test.sh bench 4 256 2   # ~4k prompt, 256 out, 2 reps
./scripts/pp-test.sh bench 16 256 1
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore

Config variants: edit scripts/frank-default-large.env on the claw (git commit + pull if the bench runs from the repo), then re-run start + bench:

• No spec baseline: SPEC_NONE=1
• Vulkan head side: HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server
• Q4_0 MTP draft head: add --spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf to the serve script's SPEC block (env hook: see SPEC_DRAFT_MODEL below)
• n_max sweep: SPEC_DRAFT_N_MAX=3|4|5
• Plain Q8_0: MODEL_FILE=Qwen3.8-27B-Q8_0.gguf (file must be on the claw)

Verification checklist after each start:

1. verify shows using device RPC0 ... using device ROCm0|Vulkan0.
2. Layer assignment lines (-lv 10): worker gets the first ~32 layers at a 1,1 split.
3. nvidia-smi on the server shows ~15–16 GB used by the worker (production small service + worker share).
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices (--gpu-layers all offloads everything; CPU holds embeddings/lm_head glue).

Expected numbers

Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.

• Production reference: 25 t/s @16k (Vulkan, spec, single node).
• Target: ≥ 30 t/s @4k, ≥ 20 t/s @16k with spec.
• Decode floor: with ngram/mod acceptance, spec multiplies the raw rate; raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.

Rollback

pp-test.sh restore starts both production services. If a PP test wedges a box: systemctl restart <unit> after killing the stray processes (pkill -x llama-server / pkill -x ggml-rpc-server — never pkill -f with a pattern matching your own ssh command line).

Open questions for the test day

1. Does the MTP head (which sits in the last layer group, iGPU side) draft cleanly with the verify pass crossing the RPC boundary? Watch acceptance in the head log (draft acceptance = X).
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is serialized per compute — expect the 2-slot speedup to be weaker than single-node. Measure, do not assume.
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not, the ubatch crossing the RPC boundary is the suspect.