docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild

- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current production = Uncensored Q6_K + standalone MTP draft, mismatched q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at 9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free. - PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both hosts) + new-model split-limits measurements; old Q8_K_XL limits marked superseded. - serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16 (env already correct; defaults now match production).
2026-08-17 02:08:08 -07:00
parent 11f37dafd8
commit 3f88b8e1ed
6 changed files with 112 additions and 50 deletions
@@ -3,12 +3,14 @@
 ## Overview

 This project manages deployment scripts for the LLM inference fleet.
-Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth
-UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the
-`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool
-262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX
-3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port
-7770.
+Dankclaw runs a llama.cpp pipeline-parallel head serving
+Qwen3.8-27B-Uncensored Q6_K (imatrix): MTP draft (standalone
+`-draft-Q8_0` sidecar, `--spec-draft-n-max 4`) over the `--spec-default`
+ngram-mod fallback, 1 slot at 262144 context (pool 262144), tensors
+split 0.68/0.32 (45 of 65 blocks) to the Dankserver RTX 3090 `ggml-rpc`
+worker, KV q8_0/f16 (mismatched, FA retained via the
+GGML_CUDA_FA_ALL_QUANTS rebuild), head GPU side on ROCm0 (no `--device`
+flag), port 7770.
 The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
 serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
 `qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously
@@ -5,25 +5,68 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
 morning-of runbook. Research background:
 [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).

-## Production state (2026-08-17: single 262k slot, split 0.53)
+## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)

 This split is the production `frank-default-large` service. Current
-state, verified against the live journal and output smoke test
-(2026-08-17, load 01:21):
+state, verified against the live journal and output smoke tests
+(2026-08-17, load 02:06):

- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum
-  provably-safe value; see "Split limits" below for the boundary
-  analysis and the failed 0.54/0.55 attempts.
+- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
+  bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
+  (3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
+  zero worker impact — measured, not assumed).
+- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
+  layer weight. Maximum for this config; see "Split limits" below.
+- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
+  MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
+  the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
+  GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
 - Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod
-  fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17).
+- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len
+  4.0-4.3, per-position ~0.86/0.71/0.71/0.71.
 - Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
-  Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each).
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
+  Vulkan).
+- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
+  stable; all allocations are upfront).
 - Unit `frank-default-large.service`, launched through
  `serve-pp-head.sh` + `frank-default-large.env`.

-## Split limits (measured 2026-08-17)
+## FA-all-quants rebuild (2026-08-17)
+
+Flash Attention with mismatched K/V cache types requires the
+`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
+`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
+(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
+and all quantized-K combos are compiled. The head (HIP gfx1151) and the
+worker (CUDA sm_86) compile the same sources, so the same CMake flag
+covers both:
+
+    cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
+    cmake --build build -j --target llama-server      # dankclaw
+    cmake --build build -j --target ggml-rpc-server   # dankserver
+
+Both hosts at commit 9b05354ec (b10433). The flag was already cached
+ON in the worker build (Aug 16 objects carry it); the head required a
+full ggml-hip rebuild. Keep both binaries on the same commit — version
+skew hangs the RPC handshake.
+
+## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
+
+KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
+B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
+(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
+
+- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
+- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
+  free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
+  (f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
+  9408 -> OOM by ~0.85 GiB.
+
+The KV-per-device count = round(f x 66) layers, independent of the
+weight boundary (same rule as the Q8_K_XL config). FA blocks at
+3,7,...,63.
+
+## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded

 The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
 With the pool shrunk to 262144, the KV per full-attention layer drops to
@@ -141,22 +141,31 @@ curl -sf http://127.0.0.1:7770/health
 ### Dankclaw — frank-default-large

 The active service on Dankclaw is the llama.cpp **pipeline-parallel
-head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the
-alias `frank-default-large` on port 7770. The head splits the tensors
-`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052):
-34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split
-(measured 2026-08-17; see the PP doc for the boundary analysis).
-Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`)
-with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side
-runs **ROCm0** (no `--device` flag; default device selection puts RPC
-first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool
-262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per
-device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192
-/ ubatch 256, `--direct-io`. The unit file is
+head**. It serves `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti,
+imatrix) under the alias `frank-default-large` on port 7770. The head
+splits the tensors `0.68,0.32` to the `ggml-rpc` worker on Dankserver
+(RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the
+maximum for this config (measured 2026-08-17; see the PP doc).
+Speculative decoding runs **MTP** with a standalone draft head
+(`--spec-draft-model`, `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`, n_max
+4) plus the `--spec-default` ngram-mod fallback (48/64/24). The head GPU
+side runs **ROCm0** (no `--device` flag; default device selection puts
+RPC first and picks ROCm for the iGPU). **1 slot** at 262144 context
+(pool 262144, `n_ctx_slot = 262144`), KV cache **K q8_0 + V f16**
+(mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with
+Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full
+layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
+256, `--direct-io`. The unit file is
 `scripts/frank-default-large.service`. It launches through
 `scripts/serve-pp-head.sh`, which reads tunables from
 `scripts/frank-default-large.env`.

+> Note: 2026-08-17 — the Q6_K swap: model + standalone draft head,
+> mismatched q8_0/f16 KV with FA (both binaries rebuilt with
+> GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the
+> Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance
+> 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head.
+
 > Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
 > (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
 > landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
@@ -58,7 +58,7 @@ frank-default-small, port 52625]
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
 - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 1 slot @ 262144, split 0.53 (2026-08-17)
+- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history + current production state: Uncensored Q6_K + standalone MTP draft, q8_0/f16 KV with FA (FA_ALL_QUANTS rebuild), 1 slot @ 262144, split 0.68 (2026-08-17)
 - [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
 - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
 - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
@@ -79,7 +79,7 @@ frank-default-small, port 52625]
  - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
  - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
  - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
-  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits + cache incident measured 2026-08-17
+  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits, FA-all-quants rebuild, cache incident (2026-08-17)

 ## Sibling projects

@@ -5,36 +5,44 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
 (`frank-default-large`), now a pipeline-parallel head with a Dankserver
 worker.

-## Production state (2026-08-17: single 262k slot, split 0.53)
+## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)

 Current production, verified against the live process and journal
-(2026-08-17, load 01:21, output smoke test). Topology detail:
+(2026-08-17, load 02:06 + smoke tests). Topology detail:
 [[Pipeline Parallelism/Qwen3.8 PP Deployment]].

 - Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
  passes no `--device` flag. Default selection puts RPC first and picks
  ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
  an option via `HEAD_BIN=.../build-vulkan-rpc`.
- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the
-  3090, about 52% of the layer weight. This is the maximum provably-safe
-  value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention
-  layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction
-  over 66 layers, independent of the weight boundary; the 9-FA flip sits
-  at frac ~= 0.538. Boundary analysis: the PP deployment doc.
- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi
+- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
+  imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
+  standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
+  (3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
+  (1718 MiB, zero worker impact).
+- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
+  `--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
+  0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
+  stronger than the old bundled head (0.739/0.554/0.427/0.340).
+- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
+  784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
+  3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
+  enabled with the mismatch** (`resolve_fused_ops: Flash Attention
+  enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
+  at 9b05354ec — see the PP doc.
+- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
+  3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
+  margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
+  ~0.85 GiB. Boundary math: the PP doc.
+- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
  `models.json` declares 262144 (contextWindow = maxTokens, house
  convention).
- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled
-  head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke
-  acceptance 0.83 (50/60, 2026-08-17); full production-window stats in
-  the superseded section below.
- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device
-  (8 full-attention layers each; FA at blocks 3,7,...,63).
- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free.
+- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
+  (tight but stable; all allocations happen at load).
 - Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
  `--gpu-layers all`.

-## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40)
+## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53)

 Superseded for topology on 2026-08-16, when the PP split became
 production. The ngram-mod preset survives as the fallback layer under
@@ -25,7 +25,7 @@ fi

 # --- defaults (used when the env file does not set them) -------------------
 : "${MODEL_DIR:=/home/dank/models}"
-: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
+: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_K.gguf}"
 : "${HEAD_BIN:=/home/dank/llama.cpp/build/bin/llama-server}"
 : "${ALIAS:=frank-pp-large}"
 : "${BIND_ADDR:=0.0.0.0}"
@@ -43,7 +43,7 @@ fi
 : "${SPEC_NGRAM_MOD_N_MAX:=64}"
 : "${SPEC_NGRAM_MOD_N_MATCH:=24}"
 : "${CACHE_TYPE_K:=q8_0}"
-: "${CACHE_TYPE_V:=q8_0}"
+: "${CACHE_TYPE_V:=f16}"
 : "${BATCH_SIZE:=8192}"
 : "${UBATCH_SIZE:=256}"
 : "${TEMPERATURE:=1.0}"