docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild

- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current
  production = Uncensored Q6_K + standalone MTP draft, mismatched
  q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at
  9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free.
- PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both
  hosts) + new-model split-limits measurements; old Q8_K_XL limits
  marked superseded.
- serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16
  (env already correct; defaults now match production).
This commit is contained in:
Frank committed 2026-08-17 02:08:08 -07:00
1 parent 11f37dafd8
commit 3f88b8e1ed
6 files changed
+112 -50

No files matched your search

@@ -5,25 +5,68 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
morning-of runbook. Research background:
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
## Production state (2026-08-17: single 262k slot, split 0.53)
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
This split is the production `frank-default-large` service. Current
state, verified against the live journal and output smoke test
(2026-08-17, load 01:21):
state, verified against the live journal and output smoke tests
(2026-08-17, load 02:06):
- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
provably-safe value; see "Split limits" below for the boundary
analysis and the failed 0.54/0.55 attempts.
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
(3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
zero worker impact — measured, not assumed).
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
layer weight. Maximum for this config; see "Split limits" below.
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len
4.0-4.3, per-position ~0.86/0.71/0.71/0.71.
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
Vulkan).
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
stable; all allocations are upfront).
- Unit `frank-default-large.service`, launched through
`serve-pp-head.sh` + `frank-default-large.env`.
## Split limits (measured 2026-08-17)
## FA-all-quants rebuild (2026-08-17)
Flash Attention with mismatched K/V cache types requires the
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
worker (CUDA sm_86) compile the same sources, so the same CMake flag
covers both:
cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j --target llama-server # dankclaw
cmake --build build -j --target ggml-rpc-server # dankserver
Both hosts at commit 9b05354ec (b10433). The flag was already cached
ON in the worker build (Aug 16 objects carry it); the head required a
full ggml-hip rebuild. Keep both binaries on the same commit — version
skew hangs the RPC handshake.
## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
(f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
9408 -> OOM by ~0.85 GiB.
The KV-per-device count = round(f x 66) layers, independent of the
weight boundary (same rule as the Q8_K_XL config). FA blocks at
3,7,...,63.
## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
With the pool shrunk to 262144, the KV per full-attention layer drops to