dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8775e64ed06a4ccd62a45b10b458d993ee46ddad
serverAppConfigs/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

16 KiB
Raw Blame History

type, description, tags, timestamp
type description tags timestamp
reference The real split deployment: which node is head and which is worker, the flags, and the verified gotchas.
llm
pipeline
2026-09-03

SUPERSEDED 2026-08-17. The Frank redo retired this topology. See frank-redo-2026-08-17.md and deployment.md for the active configuration.

2026-10-18: nothing pipeline-parallel runs on the fleet. No RPC listener exists on either host (ss -ltn shows no :50052), every artifact named below is gone (§Files), and the worker hardware left the box: the RTX 3090 was removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism inside one process (Dual-R9700-Cooperation, frank-archon).

Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)

Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX 3090 (worker) using llama.cpp RPC. This page is the design record and the morning-of runbook. Research background: Qwen3.8 Strix optimizations.

Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)

This split is the production frank-dragoon service. Current state, verified against the live journal and output smoke tests (2026-08-17, load 02:06):

  • Model: Qwen3.8-27B-Uncensored-Q6_K.gguf (22.43 GB, imatrix, no bundled MTP head). Draft: Qwen3.8-27B-Uncensored-draft-Q8_0.gguf (3.16 GB) via --spec-draft-model, running on ROCm0 only (1718 MiB; zero worker impact — measured, not assumed).
  • Split 0.68,0.32 = 45 of 65 blocks (0-44) on the 3090 — 69% of the layer weight. Maximum for this config; see "Split limits" below.
  • KV: K q8_0 + V f16 (mismatched), 784 MiB per FA layer: RPC0 8624 MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite the mismatch (resolve_fused_ops: Flash Attention enabled) via the GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
  • Pool 262144, PARALLEL=1, live n_ctx_slot = 262144.
  • MTP: standalone draft head, acceptance 0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 — stronger at every position than the old bundled head, which measured 0.739/0.554/0.427/0.340.
  • Spec chain as deployed: --spec-draft-n-max 4 on the standalone draft, with the --spec-default ngram-mod fallback at n_match 24 / n_min 48 / n_max 64 (the preset that survived the DSpark/DFlash retirements — qwen3.8-spec-matrix).
  • Batch 8192 / ubatch 256, --direct-io, --fit off, --gpu-layers all. The pi models.json declared 262144 (contextWindow = maxTokens, the house convention of that era).
  • Head GPU side: ROCm0 (no --device flag; HIP registers before Vulkan).
  • Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but stable; all allocations are upfront).
  • Unit frank-dragoon.service, launched through serve-pp-head.sh + frank-dragoon.env.

Service enablement (2026-08-17)

The PP pair is the only inference service enabled for boot. Both units (frank-dragoon on Dankclaw, frank-pp-worker on Dankserver) are enabled. The NPU unit (frank-zealot on Dankclaw) and all retired units are disabled. Restart order: worker first, then head. The Dankserver helper frank-restart-small.sh is obsolete (it restarts the removed frank-zealot unit); restart the worker with sudo systemctl restart frank-pp-worker. The Dankclaw helper frank-restart-large.sh (sync + cache drop + restart) is correct for the head. The "deploy.sh dankserver is broken until retargeted" note is long fixed: ./scripts/deploy.sh dankserver --service frank-archon is the documented invocation today (scripts/deploy.sh:38, dispatch case dankserver:frank-archon at scripts/deploy.sh:149). See deployment for the full boot-policy table.

FA-all-quants rebuild (2026-08-17)

Flash Attention with mismatched K/V cache types requires the GGML_CUDA_FA_ALL_QUANTS compile definition. Without it, ggml/src/ggml-cuda/fattn.cu gates FA off when K->type != V->type (#ifndef GGML_CUDA_FA_ALL_QUANTS). With it, the K==V check is skipped and all quantized-K combos are compiled. The head (HIP gfx1151) and the worker (CUDA sm_86) compile the same sources, so the same CMake flag covers both:

cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j --target llama-server      # dankclaw
cmake --build build -j --target ggml-rpc-server   # dankserver

Both hosts at commit 9b05354ec (b10433). The flag was already cached ON in the worker build (Aug 16 objects carry it); the head required a full ggml-hip rebuild. Keep both binaries on the same commit — version skew hangs the RPC handshake.

Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)

KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048 B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side (RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).

  • f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
  • f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks (f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV 9408 -> OOM by ~0.85 GiB.

The KV-per-device count = round(f x 66) layers, independent of the weight boundary (same rule as the Q8_K_XL config). FA blocks at 3,7,...,63.

Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded

The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks. With the pool shrunk to 262144, the KV per full-attention layer drops to exactly 1024 MiB, freeing headroom. Measurements:

  • 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB free. Healthy (2026-08-17 00:29 load).
  • 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB -> cudaMalloc OOM by ~740 MiB.
  • 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM. The KV-per-device count rounds the split fraction over 66 layers (round(f x 66)), independent of the weight boundary; the 9-FA flip sits at f ~= 0.538, just below the weight threshold for block 35, so the 35-block config is unreachable.
  • 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.

GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each), linear blocks 431.3 MB; total layer weight 27720 MB. The split pool includes the MTP head for weights and token_embd for KV — the two boundaries diverge, which is why 0.54 split them apart.

Cache incident (2026-08-17): garbage output from a full worker disk

Symptom: after a split change, the service loaded "healthy" but every completion degenerated to a repeated junk token (e.g. 48 x "/", empty content, degenerate logits). Root cause chain:

  1. The worker RPC tensor cache keys per session. Each model load writes a fresh full copy (~15 GB); over ~5 sessions of two models it grew to 4434 files / 96 GB.
  2. The Dankserver root disk filled to 100%. Cache writes started failing, leaving 0-byte entries. A subsequent reload hit corrupt cache entries and the worker served garbage weights for the newly-needed layers without validation.
  3. Fix: move the cache to .AGENTTRASH (then empty it), restart the worker, free the disk, reload the head (full re-stream ~2.5 min). A clean cache and a healthy disk produced correct output immediately (acceptance 0.89 on the first smoke test).

Guard was scripts/prune-rpc-cache.sh on Dankserver (cron, 40 GiB cap). That script is gone from the repo and from the host, yet the crontab line still runs every 10 minutes against it (*/10 * * * * /home/dank/vLLM/scripts/prune-rpc-cache.sh, crontab -l, read 2026-10-18 — the target does not exist, so it fails silently). Never let the cache directory grow unchecked; a full disk silently corrupts it. Signature to recognize: healthy load + degenerate repeated-token output.

Topology

Dankclaw (head, 192.168.0.143)          Dankserver (worker, 192.168.0.235)
llama-server b10433                      ggml-rpc-server b10433
  --rpc 192.168.0.235:50052                 --host 192.168.0.235 --port 50052
  model file (31.5 GB)                     -d CUDA0 -c (tensor cache)
  [RPC0, ROCm0|Vulkan0]  layers            RTX 3090: layers 0-44 (45 of 65 blocks)
  Strix:    layers 45-64 + lm_head
  hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
  measured wire bytes ~40 KB/position/direction (serialized graph
  dominates); 20.5 MB per ubatch both directions. Details:
  [[Ethernet Bandwidth Math]] section 0.
  • Layer-sequential split (true pipeline parallelism): layers 0..N run on the worker, the rest on the head, in order. One hidden-state handoff per token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5% of step time). PREFILL and the prompt-cache KV snapshot ARE bandwidth-bound: the link runs at line rate for the wire burst of every ubatch and ~100% during the snapshot copy. Measured numbers: Ethernet Bandwidth Math section 0.
  • Why the head is on Dankclaw: the model file is already there; the worker streams its tensor share at load (no 31.5 GB copy to the server).
  • Both sides must run the identical build (b10433 / 9b05354ec). The worker build lived in a worktree /home/dank/llama-cpp-rpc on Dankserver so the production b10121 build stayed untouched (that path no longer exists, read 2026-10-18). A version skew hangs the RPC handshake with no error.

Files

File Machine Purpose
scripts/frank-pp-worker.env server worker tunables (bind, port, device, cache)
scripts/serve-pp-worker.sh server runs ggml-rpc-server from the b10433 worktree build
scripts/frank-dragoon.env claw head tunables (split, ctx, spec, KV, I/O); was frank-pp-head.env before the 2026-08-16 rename
scripts/serve-pp-head.sh claw runs llama-server --rpc ... (see "Gotchas")
scripts/frank-dragoon.service / frank-pp-worker.service both optional systemd units (manual install); the head unit was renamed from frank-pp-head.service on 2026-08-16 and is symlinked from the repo into /etc/systemd/system
scripts/pp-test.sh any box with ssh to both test orchestrator (preflight/start/verify/bench/stop-pp/restore)
scripts/pp-bench.py claw API driver; reports server-side print_timing

Every PP file in that table except scripts/frank-dragoon.env no longer exists (checkout read 2026-10-18: frank-pp-worker.env, serve-pp-worker.sh, serve-pp-head.sh, frank-pp-worker.service, pp-test.sh, pp-bench.py and prune-rpc-cache.sh are all absent), and scripts/frank-dragoon.env is today the halogen-flash-server env, not a PP head env (frank-dragoon).

Gotchas (all verified in b10433 source / live tests, 2026-08-15)

  1. No --device on the head. Setting params.devices makes llama_prepare_model_devices use exactly that list; the RPC device is silently dropped and --rpc becomes a no-op. Omit it; default selection puts RPC first.
  2. The iGPU side is ROCm by default. Backends register in fixed order (HIP before Vulkan); the iGPU dedup rule keeps the first backend's view. Live check showed [RPC0, ROCm0]. For a Vulkan side, use the second head build /home/dank/llama.cpp/build-vulkan-rpc (HIP off; removed — the path is gone on 2026-10-18) — set HEAD_BIN in the env. Trade: ROCm side = faster prefill, ~3% slower decode (measured on this box).
  3. Pin --tensor-split 1,1. The default split is proportional to reported free memory: the Strix reports up to 96 GiB (GTT) vs the 3090's 24 GB, so the default lands ~20/80 on the slow device. 1,1 = 50/50 = balanced per-layer latency. The 2026-08-15 4B smoke test that proved it (production untouched): at a 0.2/0.8 split only 4 of 36 layers reached the 3090, and at a real split worker VRAM rose ~819 MiB with 95 t/s prefill / 58.8 t/s decode on the tiny model; link RTT ~0.3–0.5 ms.
  4. -dio on the head. Large model + RPC hangs at load without direct I/O.
  5. RPC is unauthenticated and unencrypted. LAN only. The worker binds 0.0.0.0-equivalent by design (--host 192.168.0.235).
  6. Env var names. Never TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP runtime reads them; the 2026-08-13 TEMP incident).
  7. Page-cache state confounds loads. pp-test.sh start drops caches on the head before load (warm-cache loads crash fast; cold loads survive).

Runbook (morning of 2026-08-16)

Run from any box with ssh access to both (repo path on claw: /home/dank/vllmScripts, on server: /home/dank/vLLM — both still exist as stale checkouts; the deployment checkout today is /home/dank/serverAppConfigs on both hosts).

# 0. one-time: pull the kit on both boxes
ssh dank@dankclaw   'cd /home/dank/vllmScripts  && git pull'
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'

# 1. checks (read-only)
./scripts/pp-test.sh preflight

# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all

# or step by step:
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
./scripts/pp-test.sh verify          # layer split + worker VRAM
./scripts/pp-test.sh bench 4 256 2   # ~4k prompt, 256 out, 2 reps
./scripts/pp-test.sh bench 16 256 1
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore

Config variants: edit scripts/frank-dragoon.env on the claw (git commit + pull if the bench runs from the repo), then re-run start + bench:

  • No spec baseline: SPEC_NONE=1
  • Vulkan head side: HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server
  • Q4_0 MTP draft head: add --spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf to the serve script's SPEC block (env hook: see SPEC_DRAFT_MODEL below)
  • n_max sweep: SPEC_DRAFT_N_MAX=3|4|5
  • Plain Q8_0: MODEL_FILE=Qwen3.8-27B-Q8_0.gguf (file must be on the claw)

Verification checklist after each start:

  1. verify shows using device RPC0 ... using device ROCm0|Vulkan0.
  2. Layer assignment lines (-lv 10): worker gets the first ~32 layers at a 1,1 split.
  3. nvidia-smi on the server showed ~15–16 GB used by the worker (production small service + worker share). No NVIDIA card is in that host today — the equivalent check is rocm-smi --showmeminfo vram.
  4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices (--gpu-layers all offloads everything; CPU holds embeddings/lm_head glue).

Expected numbers

Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.

  • Production reference: 25 t/s @16k (Vulkan, spec, single node).
  • Target: ≥ 30 t/s @4k, ≥ 20 t/s @16k with spec.
  • Decode floor: with ngram/mod acceptance, spec multiplies the raw rate; raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.

Rollback

pp-test.sh restore starts both production services. If a PP test wedges a box: systemctl restart <unit> after killing the stray processes (pkill -x llama-server / pkill -x ggml-rpc-server — never pkill -f with a pattern matching your own ssh command line).

Open questions for the test day

  1. Does the MTP head (which sits in the last layer group, iGPU side) draft cleanly with the verify pass crossing the RPC boundary? Watch acceptance in the head log (draft acceptance = X).
  2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is serialized per compute — expect the 2-slot speedup to be weaker than single-node. Measure, do not assume.
  3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not, the ubatch crossing the RPC boundary is the suspect.
Reference in New Issue View Git Blame Copy Permalink