From b51e078e36f3ba676a1de6ec372c9e8e47260e65 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 7 Oct 2026 13:58:30 -0700 Subject: [PATCH] frank-secondary: record the deployed f16-K boot ledger, correct the k_eq_v billing law MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Verbosity 4 paid for itself: the engine's own llama_kv_cache lines give the exact accounting — the cache bills the K+V pair TWICE under attention_k_eq_v (6.125 B/elem at f16 K + q8_0 V), reproducing 3,001.25 MiB (5 full layers at n_head_kv 2) + 306.25 MiB (25 SWA rings at n_head_kv 8, window 1024) to the byte. The earlier "bill K-only" note was wrong; full layers have 2 KV heads, not 8. Deployed boot: 21,868 used / 2,257 free — the f16-K raise (+2,161 MiB KV) is funded by the ubatch 2048 -> 1024 cut (~616 MiB of compute buffer) plus variance, so exact K came essentially free. Measured decode 139.9 t/s temp 0; vision probe returns Red at prompt_tokens 151. --- AGENTS.md | 32 ++++++----- scripts/frank-secondary.env | 31 +++++----- ...rank-secondary-llama-gemma4-q4km-200704.md | 56 ++++++++++++------- 3 files changed, 72 insertions(+), 47 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9ca1b8f..21bc822 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 | -| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Gemma4-26B-A4B-QAT-Uncensored **Q4_K_M** (25.2 B-param MoE, ~4 B active, 16,780,192,888 B reported by the engine; 30 layers = 25 sliding-window + 5 full-attention) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **ONE 200,704-token slot** (`--parallel 1`; two 200,704 slots do not fit 24 GiB), **f16 K + q8_0 V** under flash attention (the file declares `attention_k_eq_v` and this build honours it — the pool bills K-only), batch 4096 / **ubatch 1024**, `--log-verbosity 4`, **MTP OFF** (the assistant is a SEPARATE file and its draft KV is built against the target's full window — 23,806 / 319 MiB free with it loaded, no margin beside the f16-K raise). Measured q8_0-K boot: **21,890 used / 2,235 free**. Cutover 2026-10-07 from Swift-1.5 RCO-IQ3_XXS 2×114,688. All three endpoints route harness **compaction** here. | 7771 | +| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Gemma4-26B-A4B-QAT-Uncensored **Q4_K_M** (25.2 B-param MoE, ~4 B active, 16,780,192,888 B reported by the engine; 30 layers = 25 sliding-window (`n_head_kv` 8, window 1024) + 5 full-attention (indices 5/11/17/23/29, `n_head_kv` 2)) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **ONE 200,704-token slot** (`--parallel 1`; two 200,704 slots do not fit 24 GiB), **f16 K + q8_0 V** under flash attention, batch 4096 / **ubatch 1024**, `--log-verbosity 4`, **MTP OFF** (the assistant is a SEPARATE file and its draft KV is built against the target's full window — 23,806 / 319 MiB free with it loaded). Measured boots: **q8_0 K/ubatch 2048 = 21,890 used / 2,235 free; f16 K/ubatch 1024 = 21,868 / 2,257** (the +2,161 MiB of f16-K KV is paid by the ubatch cut). `attention_k_eq_v` is a compute-graph mapping only — the cache bills the K+V pair **twice** (6.125 B/elem at f16 K + q8_0 V; verified exactly against the engine's own `llama_kv_cache` lines). Cutover 2026-10-07 from Swift-1.5 RCO-IQ3_XXS 2×114,688. All three endpoints route harness **compaction** here. | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -428,14 +428,16 @@ from the Swift-1.5 RCO-IQ3_XXS 2×114,688 shape. The MTP assistant (`mtp-gemma-4-26B-A4B-it.gguf`, 251,937,728 B) is on disk, **not loaded**. **Architecture: gemma4 MoE hybrid** — 30 layers, 128 experts (8 active), -n_embd 2816, native 262,144 window. Attention is hybrid: **25 -sliding-window layers (window 1024, a fixed ring each) + 5 full-attention -layers (5, 11, 17, 23, 29)**, 8 KV heads × 256 dim — so the KV pool scales -with the 5 full layers ONLY: ~10.6 KiB/token at q8_0, ~20.5 KiB/token at -f16. `attention_k_eq_v: True` — `gemma4.cpp` maps the absent `wv` to -`Vcur = Kcur`, and this build's KV cache honours it (no V slots allocated: -the q8_0 pool measured ~2.05 GiB at 200,704, the K-only figure). **Bill -K-only.** +n_embd 2816, native 262,144 window. Attention is hybrid per the file's +`sliding_window_pattern`: **25 sliding-window layers (window 1024, +`n_head_kv = 8`) + 5 full-attention layers (indices 5, 11, 17, 23, 29, +`n_head_kv = 2`)**, head_dim 256. Two KV caches: the full-context cache +scales with the 5 layers, the SWA cache is a fixed 1024-token ring per +sliding layer. `attention_k_eq_v: True` is a **compute-graph** mapping +(`gemma4.cpp` maps the absent `wv` to `Vcur = Kcur`) — the cache reserves K +and V and, with `k_eq_v`, reserves the pair TWICE: 6.125 B/elem at f16 K + +q8_0 V, 2.125 B/elem at q8_0/q8_0. The engine's own `llama_kv_cache` buffer +sizes reproduce that rate exactly, so bill the pair, never K alone. **The shape (2026-10-07):** `--ctx-size 200704 --parallel 1` = ONE conversation (Frank's 200k cap; the harness `contextWindow` quotes @@ -450,14 +452,16 @@ against the target's FULL window — measured cost **~1.9 GiB** (23,806 used / 319 free with the assistant loaded, vs 21,890 / 2,235 without). The assistant also warns `n_ctx_seq (200704) > n_ctx_train (131072)` — the draft runs 53 % past its trained window. The q8_0-K boot leaves 2,235 MiB, -which the f16-K raise (+2.05 GiB) spends; MTP and exact-K cannot coexist on -this card. +which the f16-K raise (+2,161 MiB of KV) spends together with the ubatch +cut; MTP and exact-K cannot coexist on this card. **Measured boots** (nvidia-smi, the referee): q8_0 K + q8_0 V, ubatch 2048, MTP off: **21,890 used / 2,235 free**; same shape with the MTP assistant: -**23,806 / 319**. The f16-K + ubatch-1024 boot: see the runbook's ledger -(projected ~23 GiB). Decode is expected ubatch-invariant (Swift-era 3090 -law; unmeasured on Gemma4). +**23,806 / 319**; **f16 K + q8_0 V, ubatch 1024, MTP off (deployed): +21,868 / 2,257**. The verbosity-4 ledger for the deployed boot: weights +16,002.91 MiB CUDA0 + 577.50 MiB CPU-mapped, KV 3,001.25 (5 full layers) + +306.25 (25 SWA rings) MiB, compute buffers 926.39 + 150.63 MiB. Decode is +expected ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4). **Vision**: the publisher's own BF16 mmproj via `--mmproj`. Probe traps carry over from the Swift era: ≥256 px probe images or the model answers diff --git a/scripts/frank-secondary.env b/scripts/frank-secondary.env index df0a6e8..360a57e 100644 --- a/scripts/frank-secondary.env +++ b/scripts/frank-secondary.env @@ -3,17 +3,19 @@ # (Q4_K_M) on the RTX 3090, :7771. # # 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to -# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, q8_0 KV (FP8 class), -# MTP DISABLED (SPEC_TYPE=none). +# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V, +# MTP DISABLED (SPEC_TYPE=none), log verbosity 4. # -# Architecture: gemma4 MoE — 30 layers, 128 experts (8 active), -# hybrid attention: 25 sliding_window + 5 full_attention -# (full at layers [5, 11, 17, 23, 29]), sliding_window 1024, -# 8 KV heads, head_dim 256, attention_k_eq_v (V = K), -# native 262,144 window. +# Architecture (read from the file's own header at load): gemma4 MoE — 30 +# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding +# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29, +# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window. # -# KV: q8_0 K + q8_0 V (FP8 class, 1.0625 B/elem). The build has no -# true fp8 KV dtype; q8_0 is the FP8 class. +# KV rate law (measured, exact): the cache bills the K+V pair TWICE under +# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at +# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x +# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the +# FP8 class. # # MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model # (spec flags move as a set). The assistant file @@ -35,9 +37,10 @@ CACHE_TYPE_V=q8_0 SPEC_TYPE=none NGL=99 -# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph capture), -# ubatch 1024, K raised to f16 (V stays q8_0). f16 K costs +2.05 GiB over q8_0 -# (5 full-attn layers x 200,704 tok x 8 heads x 256 dim x 2 B); the MTP-off -# boot had 2,235 MiB free, so this is a deliberate tight fit — read the boot -# ledger, not the arithmetic. +# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph +# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of +# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) + +# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63 +# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The +# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235. VERBOSITY=4 diff --git a/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md b/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md index 32e0ca8..7b53eb1 100644 --- a/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md +++ b/serverAppConfigs-docs/LLM/frank-secondary-llama-gemma4-q4km-200704.md @@ -10,7 +10,7 @@ RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP). |File|Path|Bytes| |---|---|---| -|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,780,192,888| +|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,796,015,520 on disk; the engine reports 16,780,192,888| |Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|~1.195 GiB| |MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|~0.252 GiB| @@ -38,28 +38,46 @@ morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON context costs ~1.9 GiB because its KV is built against the target's full window ([[Llama Cpp Serving Sizing]]). -Same evening (Frank): **K raised to f16, ubatch cut 2048 → 1024, verbosity -4**. f16 K doubles the K side (20.5 KiB/token over the 5 full layers ≈ 4.1 -GiB at 200,704, +2.05 GiB over q8_0); the ubatch cut claws back part of it -(the Swift-era 3090 measurement put the 2048 rung at ~746 MiB of compute -buffers). Projected usage ~23 GiB — **the boot ledger is the referee, not -this arithmetic**: on a 24,125 MiB card a failed graph capture is the failure -mode. Decode is ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4 — -inference). +**Deployed shape (2026-10-07 evening, Frank): K = f16, ubatch 1024, verbosity +4. Measured boot: 21,868 MiB used / 2,257 MiB free** — i.e. the f16-K raise +(+2,161 MiB of KV, see the accounting below) was paid for by the ubatch cut +(2048 → 1024 returns ~616 MiB of compute buffer) plus slot-geometry variance. +Exact K came essentially free. The verbosity-4 ledger (journal, `llama_kv_cache` +lines): model 16,002.91 MiB (CUDA0) + 577.50 MiB (CPU-mapped), **KV 3,001.25 +MiB (5 full layers) + 306.25 MiB (25 SWA rings)**, compute buffers 926.39 + +150.63 MiB, output buffer 1.00 MiB. + +**Measured decode: 139.9 tok/s** (`/completion` timings, temp 0, 256 tokens, +deployed shape) — the ~4 B-active MoE is at home on the 3090. The model +thinks: a 256-token budget on a short prompt can be consumed entirely by +`reasoning_content` with empty `content` and `finish_reason: length`, so +harness requests must budget for thinking (same law as the uncensored Swift +payload). Vision probe on this build: solid-red 512 px PNG via the OpenAI +`image_url` part → `Red`, `prompt_tokens` 151 (the image reached the model). + +Decode is expected ubatch-invariant (Swift-era 3090 law; the 139.9 t/s above +is at ubatch 1024 — one rung measured, not a ladder). ## Architecture facts Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144 -native window, n_vocab 262,144. Hybrid attention: **25 sliding-window layers -(window 1024) + 5 full-attention layers (5, 11, 17, 23, 29)** — the KV pool -scales with the 5 full layers only (8 KV heads × 256 dim): q8_0 ≈ 10.6 -KiB/token, f16 ≈ 20.5 KiB/token. A 200,704 slot therefore costs ~2.05 GiB at -q8_0 — far less than the Swift 16-layer shape (34 KiB/token). -`attention_k_eq_v: True` — in `gemma4.cpp` the absent `wv` tensor makes -`Vcur = Kcur` in the compute graph, and in this build (5fc4f3c) the KV cache -**honours it: no V slots are allocated**. Measured: the q8_0 pool landed at -~2.05 GiB, exactly the K-only figure. **Bill K-only.** The `--cache-type-v` -flag is kept explicit for contract visibility; with `k_eq_v` it is inert. +native window, n_vocab 262,144. Attention is hybrid per +`gemma4.attention.sliding_window_pattern` (read from the file at load): **25 +sliding-window layers (window 1024, `n_head_kv = 8`) + 5 full-attention layers +(indices 5, 11, 17, 23, 29, `n_head_kv = 2`)**, head_dim 256 for K and V in +both classes. The KV pool is TWO caches: the full-context cache scales with +the 5 layers at the full 200,704 window; the SWA cache is a fixed 1024-token +ring per sliding layer. + +**The rate law (measured, exact):** bytes/token = `n_kv_heads × 256 × 6.125 B` +per layer at f16 K + q8_0 V — `attention_k_eq_v` does **NOT** shrink the +allocation: `Vcur = Kcur` is a compute-graph mapping, the cache reserves K and +V slots and, with `k_eq_v`, reserves the pair **twice** (6.125 = 2 × (2 B f16 + +1.0625 B q8_0)). Verified against the boot: 5 × 2 × 256 × 6.125 = 15,680 +B/token × 200,704 = **3,001.25 MiB** ✓ and 25 × 1024 × 8 × 256 × 6.125 = +**306.25 MiB** ✓. At q8_0 K + q8_0 V the rate is 2.125 B/elem: the same shape +costs 1,041 + 106 = **1,147 MiB**, so the f16-K raise is **+2,161 MiB**. The +`--cache-type-v` flag is honoured, not inert. The llama.cpp master build at commit `5fc4f3c` supports the arch (`src/models/gemma4.cpp`, `LLM_ARCH_GEMMA4 = "gemma4"`). Note: `grep -c`