docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild
- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current production = Uncensored Q6_K + standalone MTP draft, mismatched q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at 9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free. - PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both hosts) + new-model split-limits measurements; old Q8_K_XL limits marked superseded. - serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16 (env already correct; defaults now match production).
This commit is contained in:
1 parent
11f37dafd8
commit
3f88b8e1ed
6 files changed
+112
-50
No files matched your search
@@ -5,25 +5,68 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||||
|
||||
## Production state (2026-08-17: single 262k slot, split 0.53)
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
This split is the production `frank-default-large` service. Current
|
||||
state, verified against the live journal and output smoke test
|
||||
(2026-08-17, load 01:21):
|
||||
state, verified against the live journal and output smoke tests
|
||||
(2026-08-17, load 02:06):
|
||||
|
||||
- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
|
||||
provably-safe value; see "Split limits" below for the boundary
|
||||
analysis and the failed 0.54/0.55 attempts.
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
|
||||
bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
|
||||
zero worker impact — measured, not assumed).
|
||||
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
|
||||
layer weight. Maximum for this config; see "Split limits" below.
|
||||
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
|
||||
MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
|
||||
the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
|
||||
GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
|
||||
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
|
||||
- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
|
||||
fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
|
||||
- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len
|
||||
4.0-4.3, per-position ~0.86/0.71/0.71/0.71.
|
||||
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
|
||||
Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
|
||||
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
|
||||
Vulkan).
|
||||
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
|
||||
stable; all allocations are upfront).
|
||||
- Unit `frank-default-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||||
|
||||
## Split limits (measured 2026-08-17)
|
||||
## FA-all-quants rebuild (2026-08-17)
|
||||
|
||||
Flash Attention with mismatched K/V cache types requires the
|
||||
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
|
||||
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
|
||||
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
|
||||
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
|
||||
worker (CUDA sm_86) compile the same sources, so the same CMake flag
|
||||
covers both:
|
||||
|
||||
cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
cmake --build build -j --target llama-server # dankclaw
|
||||
cmake --build build -j --target ggml-rpc-server # dankserver
|
||||
|
||||
Both hosts at commit 9b05354ec (b10433). The flag was already cached
|
||||
ON in the worker build (Aug 16 objects carry it); the head required a
|
||||
full ggml-hip rebuild. Keep both binaries on the same commit — version
|
||||
skew hangs the RPC handshake.
|
||||
|
||||
## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
|
||||
|
||||
KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
|
||||
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
|
||||
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
|
||||
|
||||
- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
|
||||
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
|
||||
free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
|
||||
(f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
|
||||
9408 -> OOM by ~0.85 GiB.
|
||||
|
||||
The KV-per-device count = round(f x 66) layers, independent of the
|
||||
weight boundary (same rule as the Q8_K_XL config). FA blocks at
|
||||
3,7,...,63.
|
||||
|
||||
## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded
|
||||
|
||||
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
|
||||
With the pool shrunk to 262144, the KV per full-attention layer drops to
|
||||
|
||||
Reference in new issue
Block a user