serverAppConfigs/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md

---
type: reference
description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX"
tags:
  - llm
  - pipeline
timestamp: 2026-09-03
---

> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
> [deployment.md](../deployment.md) for the active configuration.

# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)

Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
3090 (worker) using llama.cpp RPC. This page is the design record and the
morning-of runbook. Research background:
[Qwen3.8 Strix optimizations](../qwen3.8-strix-optimizations.md).

## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)

This split is the production `frank-dragoon` service. Current
state, verified against the live journal and output smoke tests
(2026-08-17, load 02:06):

- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
  bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
  (3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
  zero worker impact — measured, not assumed).
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
  layer weight. Maximum for this config; see "Split limits" below.
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
  MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
  the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
  GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len
  4.0-4.3, per-position ~0.86/0.71/0.71/0.71.
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
  Vulkan).
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
  stable; all allocations are upfront).
- Unit `frank-dragoon.service`, launched through
  `serve-pp-head.sh` + `frank-dragoon.env`.

## Service enablement (2026-08-17)

The PP pair is the only inference service enabled for boot. Both units
(`frank-dragoon` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-zealot` on Dankclaw) and all retired
units are disabled. Restart order: worker first, then head. The Dankserver
helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-zealot` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. `deploy.sh dankserver` is broken until retargeted. See
[[deployment]] for the full boot-policy table.

## FA-all-quants rebuild (2026-08-17)

Flash Attention with mismatched K/V cache types requires the
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
worker (CUDA sm_86) compile the same sources, so the same CMake flag
covers both:

    cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
    cmake --build build -j --target llama-server      # dankclaw
    cmake --build build -j --target ggml-rpc-server   # dankserver

Both hosts at commit 9b05354ec (b10433). The flag was already cached
ON in the worker build (Aug 16 objects carry it); the head required a
full ggml-hip rebuild. Keep both binaries on the same commit — version
skew hangs the RPC handshake.

## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)

KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).

- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
  free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
  (f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
  9408 -> OOM by ~0.85 GiB.

The KV-per-device count = round(f x 66) layers, independent of the
weight boundary (same rule as the Q8_K_XL config). FA blocks at
3,7,...,63.

## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded

The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
With the pool shrunk to 262144, the KV per full-attention layer drops to
exactly 1024 MiB, freeing headroom. Measurements:

- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
  free. Healthy (2026-08-17 00:29 load).
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
  cudaMalloc OOM by ~740 MiB.
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
  The KV-per-device count rounds the split fraction over 66 layers
  (round(f x 66)), independent of the weight boundary; the 9-FA flip
  sits at f ~= 0.538, just below the weight threshold for block 35, so
  the 35-block config is unreachable.
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
  rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.

GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
includes the MTP head for weights and token_embd for KV — the two
boundaries diverge, which is why 0.54 split them apart.

## Cache incident (2026-08-17): garbage output from a full worker disk

Symptom: after a split change, the service loaded "healthy" but every
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
content, degenerate logits). Root cause chain:

1. The worker RPC tensor cache keys per session. Each model load writes
   a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
   4434 files / 96 GB.
2. The Dankserver root disk filled to 100%. Cache writes started
   failing, leaving 0-byte entries. A subsequent reload hit corrupt
   cache entries and the worker served garbage weights for the
   newly-needed layers without validation.
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
   worker, free the disk, reload the head (full re-stream ~2.5 min).
   A clean cache and a healthy disk produced correct output immediately
   (acceptance 0.89 on the first smoke test).

Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
Never let the cache directory grow unchecked; a full disk silently
corrupts it. Signature to recognize: healthy load + degenerate repeated-
token output.

## Topology

```
Dankclaw (head, 192.168.0.143)          Dankserver (worker, 192.168.0.235)
llama-server b10433                      ggml-rpc-server b10433
  --rpc 192.168.0.235:50052                 --host 192.168.0.235 --port 50052
  model file (31.5 GB)                     -d CUDA0 -c (tensor cache)
  [RPC0, ROCm0|Vulkan0]  layers            RTX 3090: layers 0-44 (45 of 65 blocks)
  Strix:    layers 45-64 + lm_head
  hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
  measured wire bytes ~40 KB/position/direction (serialized graph
  dominates); 20.5 MB per ubatch both directions. Details:
  [[Ethernet Bandwidth Math]] section 0.
```

- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
  the worker, the rest on the head, in order. One hidden-state handoff per
  token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5%
  of step time). PREFILL and the prompt-cache KV snapshot ARE
  bandwidth-bound: the link runs at line rate for the wire burst of every
  ubatch and ~100% during the snapshot copy. Measured numbers:
  [[Ethernet Bandwidth Math]] section 0.
- **Why the head is on Dankclaw:** the model file is already there; the
  worker streams its tensor share at load (no 31.5 GB copy to the server).
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
  worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
  so the production b10121 build stays untouched. A version skew hangs the
  RPC handshake with no error.

## Files

| File | Machine | Purpose |
|------|---------|---------|
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
| `scripts/frank-dragoon.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
| `scripts/frank-dragoon.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |

## Gotchas (all verified in b10433 source / live tests, 2026-08-15)

1. **No `--device` on the head.** Setting `params.devices` makes
   `llama_prepare_model_devices` use exactly that list; the RPC device is
   silently dropped and `--rpc` becomes a no-op. Omit it; default selection
   puts RPC first.
2. **The iGPU side is ROCm by default.** Backends register in fixed order
   (HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
   Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
   head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
   `HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
   decode (measured on this box).
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
   *reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
   3090's 24 GB, so the default lands ~20/80 on the slow device (verified
   with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
   I/O.
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
   0.0.0.0-equivalent by design (`--host 192.168.0.235`).
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
   (HIP runtime reads them; the 2026-08-13 `TEMP` incident).
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
   the head before load (warm-cache loads crash fast; cold loads survive).

## Runbook (morning of 2026-08-16)

Run from any box with ssh access to both (repo path on claw:
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).

```
# 0. one-time: pull the kit on both boxes
ssh dank@dankclaw   'cd /home/dank/vllmScripts  && git pull'
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'

# 1. checks (read-only)
./scripts/pp-test.sh preflight

# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all

# or step by step:
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
./scripts/pp-test.sh verify          # layer split + worker VRAM
./scripts/pp-test.sh bench 4 256 2   # ~4k prompt, 256 out, 2 reps
./scripts/pp-test.sh bench 16 256 1
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
```

Config variants: edit `scripts/frank-dragoon.env` on the claw (git commit +
pull if the bench runs from the repo), then re-run `start` + `bench`:

- No spec baseline: `SPEC_NONE=1`
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)

Verification checklist after each `start`:

1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
   1,1 split.
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
   (production small service + worker share).
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
   (`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).

## Expected numbers

Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.

- Production reference: 25 t/s @16k (Vulkan, spec, single node).
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
  raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.

## Rollback

`pp-test.sh restore` starts both production services. If a PP test wedges a
box: `systemctl restart <unit>` after killing the stray processes
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
with a pattern matching your own ssh command line).

## Open questions for the test day

1. Does the MTP head (which sits in the last layer group, iGPU side) draft
   cleanly with the verify pass crossing the RPC boundary? Watch acceptance
   in the head log (`draft acceptance = X`).
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
   serialized per compute — expect the 2-slot speedup to be weaker than
   single-node. Measure, do not assume.
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
   the ubatch crossing the RPC boundary is the suspect.