docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild
- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current production = Uncensored Q6_K + standalone MTP draft, mismatched q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at 9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free. - PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both hosts) + new-model split-limits measurements; old Q8_K_XL limits marked superseded. - serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16 (env already correct; defaults now match production).
This commit is contained in:
1 parent
11f37dafd8
commit
3f88b8e1ed
6 files changed
+112
-50
No files matched your search
+25
-17
@@ -5,36 +5,44 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
|
||||
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
|
||||
worker.
|
||||
|
||||
## Production state (2026-08-17: single 262k slot, split 0.53)
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
Current production, verified against the live process and journal
|
||||
(2026-08-17, load 01:21, output smoke test). Topology detail:
|
||||
(2026-08-17, load 02:06 + smoke tests). Topology detail:
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
||||
|
||||
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
||||
passes no `--device` flag. Default selection puts RPC first and picks
|
||||
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
||||
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
||||
- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the
|
||||
3090, about 52% of the layer weight. This is the maximum provably-safe
|
||||
value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention
|
||||
layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction
|
||||
over 66 layers, independent of the weight boundary; the 9-FA flip sits
|
||||
at frac ~= 0.538. Boundary analysis: the PP deployment doc.
|
||||
- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
|
||||
imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
|
||||
standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
|
||||
(1718 MiB, zero worker impact).
|
||||
- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
|
||||
`--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
|
||||
0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
|
||||
stronger than the old bundled head (0.739/0.554/0.427/0.340).
|
||||
- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
|
||||
784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
|
||||
3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
|
||||
enabled with the mismatch** (`resolve_fused_ops: Flash Attention
|
||||
enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
|
||||
at 9b05354ec — see the PP doc.
|
||||
- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
|
||||
3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
|
||||
margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
|
||||
~0.85 GiB. Boundary math: the PP doc.
|
||||
- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
|
||||
`models.json` declares 262144 (contextWindow = maxTokens, house
|
||||
convention).
|
||||
- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled
|
||||
head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke
|
||||
acceptance 0.83 (50/60, 2026-08-17); full production-window stats in
|
||||
the superseded section below.
|
||||
- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device
|
||||
(8 full-attention layers each; FA at blocks 3,7,...,63).
|
||||
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
|
||||
- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
|
||||
(tight but stable; all allocations happen at load).
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)
|
||||
## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53)
|
||||
|
||||
Superseded for topology on 2026-08-16, when the PP split became
|
||||
production. The ngram-mod preset survives as the fallback layer under
|
||||
|
||||
Reference in new issue
Block a user