From 3f88b8e1ed93ee6002739bcf86157ba53d41e8cf Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 17 Aug 2026 02:08:08 -0700 Subject: [PATCH] docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild - spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current production = Uncensored Q6_K + standalone MTP draft, mismatched q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at 9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free. - PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both hosts) + new-model split-limits measurements; old Q8_K_XL limits marked superseded. - serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16 (env already correct; defaults now match production). --- AGENTS.md | 14 ++-- .../Qwen3.8 PP Deployment.md | 65 +++++++++++++++---- docs/deployment.md | 33 ++++++---- docs/index.md | 4 +- docs/qwen3.8-spec-matrix.md | 42 +++++++----- scripts/serve-pp-head.sh | 4 +- 6 files changed, 112 insertions(+), 50 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index dd91d40..0867f22 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,12 +3,14 @@ ## Overview This project manages deployment scripts for the LLM inference fleet. -Dankclaw runs a llama.cpp pipeline-parallel head for Qwen3.8-27B (unsloth -UD-Q8_K_XL): MTP draft (bundled head, `--spec-draft-n-max 4`) over the -`--spec-default` ngram-mod fallback, 1 slot at 262144 context (pool -262144), tensors split 0.53/0.47 (34 of 65 blocks) to the Dankserver RTX -3090 `ggml-rpc` worker, head GPU side on ROCm0 (no `--device` flag), port -7770. +Dankclaw runs a llama.cpp pipeline-parallel head serving +Qwen3.8-27B-Uncensored Q6_K (imatrix): MTP draft (standalone +`-draft-Q8_0` sidecar, `--spec-draft-n-max 4`) over the `--spec-default` +ngram-mod fallback, 1 slot at 262144 context (pool 262144), tensors +split 0.68/0.32 (45 of 65 blocks) to the Dankserver RTX 3090 `ggml-rpc` +worker, KV q8_0/f16 (mismatched, FA retained via the +GGML_CUDA_FA_ALL_QUANTS rebuild), head GPU side on ROCm0 (no `--device` +flag), port 7770. The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag `qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md index 0fe5177..ff0e9b4 100644 --- a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -5,25 +5,68 @@ Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX morning-of runbook. Research background: [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md). -## Production state (2026-08-17: single 262k slot, split 0.53) +## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68) This split is the production `frank-default-large` service. Current -state, verified against the live journal and output smoke test -(2026-08-17, load 01:21): +state, verified against the live journal and output smoke tests +(2026-08-17, load 02:06): -- Split `0.53,0.47` = 34 of 65 blocks (0-33) on the 3090 — the maximum - provably-safe value; see "Split limits" below for the boundary - analysis and the failed 0.54/0.55 attempts. +- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no + bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf` + (3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB; + zero worker impact — measured, not assumed). +- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the + layer weight. Maximum for this config; see "Split limits" below. +- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624 + MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite + the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the + GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below. - Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`. -- MTP enabled: `--spec-type draft-mtp --spec-draft-n-max 4` + ngram-mod - fallback (48/64/24). Smoke acceptance 0.83 (50/60, 2026-08-17). +- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len + 4.0-4.3, per-position ~0.86/0.71/0.71/0.71. - Head GPU side: **ROCm0** (no `--device` flag; HIP registers before - Vulkan). KV at load: 8192 MiB ROCm0 + 8192 MiB RPC0 (8 FA each). -- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free. + Vulkan). +- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but + stable; all allocations are upfront). - Unit `frank-default-large.service`, launched through `serve-pp-head.sh` + `frank-default-large.env`. -## Split limits (measured 2026-08-17) +## FA-all-quants rebuild (2026-08-17) + +Flash Attention with mismatched K/V cache types requires the +`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it, +`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type` +(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped +and all quantized-K combos are compiled. The head (HIP gfx1151) and the +worker (CUDA sm_86) compile the same sources, so the same CMake flag +covers both: + + cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON + cmake --build build -j --target llama-server # dankclaw + cmake --build build -j --target ggml-rpc-server # dankserver + +Both hosts at commit 9b05354ec (b10433). The flag was already cached +ON in the worker build (Aug 16 objects carry it); the head required a +full ggml-hip rebuild. Keep both binaries on the same commit — version +skew hangs the RPC handshake. + +## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV) + +KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048 +B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side +(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd). + +- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1. +- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB + free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks + (f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV + 9408 -> OOM by ~0.85 GiB. + +The KV-per-device count = round(f x 66) layers, independent of the +weight boundary (same rule as the Q8_K_XL config). FA blocks at +3,7,...,63. + +## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks. With the pool shrunk to 262144, the KV per full-attention layer drops to diff --git a/docs/deployment.md b/docs/deployment.md index 0076dfd..020b561 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -141,22 +141,31 @@ curl -sf http://127.0.0.1:7770/health ### Dankclaw — frank-default-large The active service on Dankclaw is the llama.cpp **pipeline-parallel -head**. It serves the unsloth Qwen3.8-27B `UD-Q8_K_XL` model under the -alias `frank-default-large` on port 7770. The head splits the tensors -`0.53,0.47` to the `ggml-rpc` worker on Dankserver (RTX 3090, port 50052): -34 of 65 blocks (0-33) on the 3090 — the maximum provably-safe split -(measured 2026-08-17; see the PP doc for the boundary analysis). -Speculative decoding runs **MTP** (bundled head, `--spec-draft-n-max 4`) -with the `--spec-default` ngram-mod fallback (48/64/24). The head GPU side -runs **ROCm0** (no `--device` flag; default device selection puts RPC -first and picks ROCm for the iGPU). **1 slot** at 262144 context (pool -262144, `n_ctx_slot = 262144`), KV cache `f16` K / `f16` V (8192 MiB per -device), full layer offload (`--gpu-layers all`, `--fit off`), batch 8192 -/ ubatch 256, `--direct-io`. The unit file is +head**. It serves `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, +imatrix) under the alias `frank-default-large` on port 7770. The head +splits the tensors `0.68,0.32` to the `ggml-rpc` worker on Dankserver +(RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the +maximum for this config (measured 2026-08-17; see the PP doc). +Speculative decoding runs **MTP** with a standalone draft head +(`--spec-draft-model`, `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`, n_max +4) plus the `--spec-default` ngram-mod fallback (48/64/24). The head GPU +side runs **ROCm0** (no `--device` flag; default device selection puts +RPC first and picks ROCm for the iGPU). **1 slot** at 262144 context +(pool 262144, `n_ctx_slot = 262144`), KV cache **K q8_0 + V f16** +(mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with +Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full +layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch +256, `--direct-io`. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-pp-head.sh`, which reads tunables from `scripts/frank-default-large.env`. +> Note: 2026-08-17 — the Q6_K swap: model + standalone draft head, +> mismatched q8_0/f16 KV with FA (both binaries rebuilt with +> GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the +> Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance +> 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head. + > Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144 > (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090 > landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full diff --git a/docs/index.md b/docs/index.md index 89cc24c..a172883 100644 --- a/docs/index.md +++ b/docs/index.md @@ -58,7 +58,7 @@ frank-default-small, port 52625] - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug) -- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec history + current production state: PP head, MTP (n_max 4) over the ngram-mod fallback, 1 slot @ 262144, split 0.53 (2026-08-17) +- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history + current production state: Uncensored Q6_K + standalone MTP draft, q8_0/f16 KV with FA (FA_ALL_QUANTS rebuild), 1 slot @ 262144, split 0.68 (2026-08-17) - [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small @@ -79,7 +79,7 @@ frank-default-small, port 52625] - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split - - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits + cache incident measured 2026-08-17 + - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits, FA-all-quants rebuild, cache incident (2026-08-17) ## Sibling projects diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md index f407fe4..e2d1eb3 100644 --- a/docs/qwen3.8-spec-matrix.md +++ b/docs/qwen3.8-spec-matrix.md @@ -5,36 +5,44 @@ tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service (`frank-default-large`), now a pipeline-parallel head with a Dankserver worker. -## Production state (2026-08-17: single 262k slot, split 0.53) +## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68) Current production, verified against the live process and journal -(2026-08-17, load 01:21, output smoke test). Topology detail: +(2026-08-17, load 02:06 + smoke tests). Topology detail: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]. - Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head passes no `--device` flag. Default selection puts RPC first and picks ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays an option via `HEAD_BIN=.../build-vulkan-rpc`. -- Split: `--tensor-split 0.53,0.47` — 34 of 65 blocks (0-33) on the - 3090, about 52% of the layer weight. This is the maximum provably-safe - value (measured 2026-08-17): 0.54 jumped the KV to 9 full-attention - layers -> cudaMalloc OOM. The KV-per-device count rounds the fraction - over 66 layers, independent of the weight boundary; the 9-FA flip sits - at frac ~= 0.538. Boundary analysis: the PP deployment doc. -- Context: pool 262144, 1 slot. `n_ctx_slot = 262144` exactly. The pi +- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB, + imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's + standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf` + (3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only + (1718 MiB, zero worker impact). +- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the + `--spec-default` ngram-mod fallback (48/64/24). Measured acceptance + 0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 — + stronger than the old bundled head (0.739/0.554/0.427/0.340). +- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer = + 784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0 + 3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays + enabled with the mismatch** (`resolve_fused_ops: Flash Attention + enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries + at 9b05354ec — see the PP doc. +- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the + 3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB + margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by + ~0.85 GiB. Boundary math: the PP doc. +- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi `models.json` declares 262144 (contextWindow = maxTokens, house convention). -- Spec: **MTP** — `--spec-type draft-mtp --spec-draft-n-max 4` (bundled - head) with the `--spec-default` ngram-mod fallback (48/64/24). Smoke - acceptance 0.83 (50/60, 2026-08-17); full production-window stats in - the superseded section below. -- KV: `f16` K / `f16` V. At the 262144 pool: 8192 MiB per device - (8 full-attention layers each; FA at blocks 3,7,...,63). -- Worker steady state: 23460/24576 MiB used on the 3090, ~667 MiB free. +- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free + (tight but stable; all allocations happen at load). - Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, `--gpu-layers all`. -## Result (superseded 2026-08-17: 2 slots x 200704, split 0.40) +## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53) Superseded for topology on 2026-08-16, when the PP split became production. The ngram-mod preset survives as the fallback layer under diff --git a/scripts/serve-pp-head.sh b/scripts/serve-pp-head.sh index 5d31b97..cd72c25 100755 --- a/scripts/serve-pp-head.sh +++ b/scripts/serve-pp-head.sh @@ -25,7 +25,7 @@ fi # --- defaults (used when the env file does not set them) ------------------- : "${MODEL_DIR:=/home/dank/models}" -: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}" +: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_K.gguf}" : "${HEAD_BIN:=/home/dank/llama.cpp/build/bin/llama-server}" : "${ALIAS:=frank-pp-large}" : "${BIND_ADDR:=0.0.0.0}" @@ -43,7 +43,7 @@ fi : "${SPEC_NGRAM_MOD_N_MAX:=64}" : "${SPEC_NGRAM_MOD_N_MATCH:=24}" : "${CACHE_TYPE_K:=q8_0}" -: "${CACHE_TYPE_V:=q8_0}" +: "${CACHE_TYPE_V:=f16}" : "${BATCH_SIZE:=8192}" : "${UBATCH_SIZE:=256}" : "${TEMPERATURE:=1.0}"