docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild

- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current
  production = Uncensored Q6_K + standalone MTP draft, mismatched
  q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at
  9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free.
- PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both
  hosts) + new-model split-limits measurements; old Q8_K_XL limits
  marked superseded.
- serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16
  (env already correct; defaults now match production).
This commit is contained in:
Frank committed 2026-08-17 02:08:08 -07:00
1 parent 11f37dafd8
commit 3f88b8e1ed
6 files changed
+112 -50

No files matched your search

+25 -17
View File
@@ -5,36 +5,44 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
worker.
## Production state (2026-08-17: single 262k slot, split 0.53)
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
Current production, verified against the live process and journal
(2026-08-17, load 01:21, output smoke test). Topology detail:
(2026-08-17, load 02:06 + smoke tests). Topology detail:
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
passes no `--device` flag. Default selection puts RPC first and picks
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
an option via `HEAD_BIN=.../build-vulkan-rpc`.
- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the
3090, about 52% of the layer weight. This is the maximum provably-safe
value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention
layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction
over 66 layers, independent of the weight boundary; the 9-FA flip sits
at frac ~= 0.538. Boundary analysis: the PP deployment doc.
- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
(3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
(1718 MiB, zero worker impact).
- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
`--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
stronger than the old bundled head (0.739/0.554/0.427/0.340).
- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
enabled with the mismatch** (`resolve_fused_ops: Flash Attention
enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
at 9b05354ec — see the PP doc.
- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
~0.85 GiB. Boundary math: the PP doc.
- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
`models.json` declares 262144 (contextWindow = maxTokens, house
convention).
- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled
head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke
acceptance 0.83 (50/60, 2026-08-17); full production-window stats in
the superseded section below.
- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device
(8 full-attention layers each; FA at blocks 3,7,...,63).
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
(tight but stable; all allocations happen at load).
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
`--gpu-layers all`.
## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)
## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53)
Superseded for topology on 2026-08-16, when the PP split became
production. The ngram-mod preset survives as the fallback layer under