frank-secondary: record the deployed f16-K boot ledger, correct the k_eq_v billing law
Verbosity 4 paid for itself: the engine's own llama_kv_cache lines give the exact accounting — the cache bills the K+V pair TWICE under attention_k_eq_v (6.125 B/elem at f16 K + q8_0 V), reproducing 3,001.25 MiB (5 full layers at n_head_kv 2) + 306.25 MiB (25 SWA rings at n_head_kv 8, window 1024) to the byte. The earlier "bill K-only" note was wrong; full layers have 2 KV heads, not 8. Deployed boot: 21,868 used / 2,257 free — the f16-K raise (+2,161 MiB KV) is funded by the ubatch 2048 -> 1024 cut (~616 MiB of compute buffer) plus variance, so exact K came essentially free. Measured decode 139.9 t/s temp 0; vision probe returns Red at prompt_tokens 151.
This commit is contained in:
Frank
@@ -27,7 +27,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
|
||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Gemma4-26B-A4B-QAT-Uncensored **Q4_K_M** (25.2 B-param MoE, ~4 B active, 16,780,192,888 B reported by the engine; 30 layers = 25 sliding-window + 5 full-attention) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **ONE 200,704-token slot** (`--parallel 1`; two 200,704 slots do not fit 24 GiB), **f16 K + q8_0 V** under flash attention (the file declares `attention_k_eq_v` and this build honours it — the pool bills K-only), batch 4096 / **ubatch 1024**, `--log-verbosity 4`, **MTP OFF** (the assistant is a SEPARATE file and its draft KV is built against the target's full window — 23,806 / 319 MiB free with it loaded, no margin beside the f16-K raise). Measured q8_0-K boot: **21,890 used / 2,235 free**. Cutover 2026-10-07 from Swift-1.5 RCO-IQ3_XXS 2×114,688. All three endpoints route harness **compaction** here. | 7771 |
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Gemma4-26B-A4B-QAT-Uncensored **Q4_K_M** (25.2 B-param MoE, ~4 B active, 16,780,192,888 B reported by the engine; 30 layers = 25 sliding-window (`n_head_kv` 8, window 1024) + 5 full-attention (indices 5/11/17/23/29, `n_head_kv` 2)) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **ONE 200,704-token slot** (`--parallel 1`; two 200,704 slots do not fit 24 GiB), **f16 K + q8_0 V** under flash attention, batch 4096 / **ubatch 1024**, `--log-verbosity 4`, **MTP OFF** (the assistant is a SEPARATE file and its draft KV is built against the target's full window — 23,806 / 319 MiB free with it loaded). Measured boots: **q8_0 K/ubatch 2048 = 21,890 used / 2,235 free; f16 K/ubatch 1024 = 21,868 / 2,257** (the +2,161 MiB of f16-K KV is paid by the ubatch cut). `attention_k_eq_v` is a compute-graph mapping only — the cache bills the K+V pair **twice** (6.125 B/elem at f16 K + q8_0 V; verified exactly against the engine's own `llama_kv_cache` lines). Cutover 2026-10-07 from Swift-1.5 RCO-IQ3_XXS 2×114,688. All three endpoints route harness **compaction** here. | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -428,14 +428,16 @@ from the Swift-1.5 RCO-IQ3_XXS 2×114,688 shape. The MTP assistant
|
||||
(`mtp-gemma-4-26B-A4B-it.gguf`, 251,937,728 B) is on disk, **not loaded**.
|
||||
|
||||
**Architecture: gemma4 MoE hybrid** — 30 layers, 128 experts (8 active),
|
||||
n_embd 2816, native 262,144 window. Attention is hybrid: **25
|
||||
sliding-window layers (window 1024, a fixed ring each) + 5 full-attention
|
||||
layers (5, 11, 17, 23, 29)**, 8 KV heads × 256 dim — so the KV pool scales
|
||||
with the 5 full layers ONLY: ~10.6 KiB/token at q8_0, ~20.5 KiB/token at
|
||||
f16. `attention_k_eq_v: True` — `gemma4.cpp` maps the absent `wv` to
|
||||
`Vcur = Kcur`, and this build's KV cache honours it (no V slots allocated:
|
||||
the q8_0 pool measured ~2.05 GiB at 200,704, the K-only figure). **Bill
|
||||
K-only.**
|
||||
n_embd 2816, native 262,144 window. Attention is hybrid per the file's
|
||||
`sliding_window_pattern`: **25 sliding-window layers (window 1024,
|
||||
`n_head_kv = 8`) + 5 full-attention layers (indices 5, 11, 17, 23, 29,
|
||||
`n_head_kv = 2`)**, head_dim 256. Two KV caches: the full-context cache
|
||||
scales with the 5 layers, the SWA cache is a fixed 1024-token ring per
|
||||
sliding layer. `attention_k_eq_v: True` is a **compute-graph** mapping
|
||||
(`gemma4.cpp` maps the absent `wv` to `Vcur = Kcur`) — the cache reserves K
|
||||
and V and, with `k_eq_v`, reserves the pair TWICE: 6.125 B/elem at f16 K +
|
||||
q8_0 V, 2.125 B/elem at q8_0/q8_0. The engine's own `llama_kv_cache` buffer
|
||||
sizes reproduce that rate exactly, so bill the pair, never K alone.
|
||||
|
||||
**The shape (2026-10-07):** `--ctx-size 200704 --parallel 1` = ONE
|
||||
conversation (Frank's 200k cap; the harness `contextWindow` quotes
|
||||
@@ -450,14 +452,16 @@ against the target's FULL window — measured cost **~1.9 GiB** (23,806 used
|
||||
/ 319 free with the assistant loaded, vs 21,890 / 2,235 without). The
|
||||
assistant also warns `n_ctx_seq (200704) > n_ctx_train (131072)` — the
|
||||
draft runs 53 % past its trained window. The q8_0-K boot leaves 2,235 MiB,
|
||||
which the f16-K raise (+2.05 GiB) spends; MTP and exact-K cannot coexist on
|
||||
this card.
|
||||
which the f16-K raise (+2,161 MiB of KV) spends together with the ubatch
|
||||
cut; MTP and exact-K cannot coexist on this card.
|
||||
|
||||
**Measured boots** (nvidia-smi, the referee): q8_0 K + q8_0 V, ubatch 2048,
|
||||
MTP off: **21,890 used / 2,235 free**; same shape with the MTP assistant:
|
||||
**23,806 / 319**. The f16-K + ubatch-1024 boot: see the runbook's ledger
|
||||
(projected ~23 GiB). Decode is expected ubatch-invariant (Swift-era 3090
|
||||
law; unmeasured on Gemma4).
|
||||
**23,806 / 319**; **f16 K + q8_0 V, ubatch 1024, MTP off (deployed):
|
||||
21,868 / 2,257**. The verbosity-4 ledger for the deployed boot: weights
|
||||
16,002.91 MiB CUDA0 + 577.50 MiB CPU-mapped, KV 3,001.25 (5 full layers) +
|
||||
306.25 (25 SWA rings) MiB, compute buffers 926.39 + 150.63 MiB. Decode is
|
||||
expected ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4).
|
||||
|
||||
**Vision**: the publisher's own BF16 mmproj via `--mmproj`. Probe traps
|
||||
carry over from the Swift era: ≥256 px probe images or the model answers
|
||||
|
||||
+17
-14
@@ -3,17 +3,19 @@
|
||||
# (Q4_K_M) on the RTX 3090, :7771.
|
||||
#
|
||||
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
|
||||
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, q8_0 KV (FP8 class),
|
||||
# MTP DISABLED (SPEC_TYPE=none).
|
||||
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V,
|
||||
# MTP DISABLED (SPEC_TYPE=none), log verbosity 4.
|
||||
#
|
||||
# Architecture: gemma4 MoE — 30 layers, 128 experts (8 active),
|
||||
# hybrid attention: 25 sliding_window + 5 full_attention
|
||||
# (full at layers [5, 11, 17, 23, 29]), sliding_window 1024,
|
||||
# 8 KV heads, head_dim 256, attention_k_eq_v (V = K),
|
||||
# native 262,144 window.
|
||||
# Architecture (read from the file's own header at load): gemma4 MoE — 30
|
||||
# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding
|
||||
# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29,
|
||||
# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window.
|
||||
#
|
||||
# KV: q8_0 K + q8_0 V (FP8 class, 1.0625 B/elem). The build has no
|
||||
# true fp8 KV dtype; q8_0 is the FP8 class.
|
||||
# KV rate law (measured, exact): the cache bills the K+V pair TWICE under
|
||||
# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at
|
||||
# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x
|
||||
# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the
|
||||
# FP8 class.
|
||||
#
|
||||
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
|
||||
# (spec flags move as a set). The assistant file
|
||||
@@ -35,9 +37,10 @@ CACHE_TYPE_V=q8_0
|
||||
SPEC_TYPE=none
|
||||
NGL=99
|
||||
|
||||
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph capture),
|
||||
# ubatch 1024, K raised to f16 (V stays q8_0). f16 K costs +2.05 GiB over q8_0
|
||||
# (5 full-attn layers x 200,704 tok x 8 heads x 256 dim x 2 B); the MTP-off
|
||||
# boot had 2,235 MiB free, so this is a deliberate tight fit — read the boot
|
||||
# ledger, not the arithmetic.
|
||||
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph
|
||||
# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of
|
||||
# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) +
|
||||
# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63
|
||||
# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The
|
||||
# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235.
|
||||
VERBOSITY=4
|
||||
|
||||
@@ -10,7 +10,7 @@ RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP).
|
||||
|
||||
|File|Path|Bytes|
|
||||
|---|---|---|
|
||||
|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,780,192,888|
|
||||
|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,796,015,520 on disk; the engine reports 16,780,192,888|
|
||||
|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|~1.195 GiB|
|
||||
|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|~0.252 GiB|
|
||||
|
||||
@@ -38,28 +38,46 @@ morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON
|
||||
context costs ~1.9 GiB because its KV is built against the target's full
|
||||
window ([[Llama Cpp Serving Sizing]]).
|
||||
|
||||
Same evening (Frank): **K raised to f16, ubatch cut 2048 → 1024, verbosity
|
||||
4**. f16 K doubles the K side (20.5 KiB/token over the 5 full layers ≈ 4.1
|
||||
GiB at 200,704, +2.05 GiB over q8_0); the ubatch cut claws back part of it
|
||||
(the Swift-era 3090 measurement put the 2048 rung at ~746 MiB of compute
|
||||
buffers). Projected usage ~23 GiB — **the boot ledger is the referee, not
|
||||
this arithmetic**: on a 24,125 MiB card a failed graph capture is the failure
|
||||
mode. Decode is ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4 —
|
||||
inference).
|
||||
**Deployed shape (2026-10-07 evening, Frank): K = f16, ubatch 1024, verbosity
|
||||
4. Measured boot: 21,868 MiB used / 2,257 MiB free** — i.e. the f16-K raise
|
||||
(+2,161 MiB of KV, see the accounting below) was paid for by the ubatch cut
|
||||
(2048 → 1024 returns ~616 MiB of compute buffer) plus slot-geometry variance.
|
||||
Exact K came essentially free. The verbosity-4 ledger (journal, `llama_kv_cache`
|
||||
lines): model 16,002.91 MiB (CUDA0) + 577.50 MiB (CPU-mapped), **KV 3,001.25
|
||||
MiB (5 full layers) + 306.25 MiB (25 SWA rings)**, compute buffers 926.39 +
|
||||
150.63 MiB, output buffer 1.00 MiB.
|
||||
|
||||
**Measured decode: 139.9 tok/s** (`/completion` timings, temp 0, 256 tokens,
|
||||
deployed shape) — the ~4 B-active MoE is at home on the 3090. The model
|
||||
thinks: a 256-token budget on a short prompt can be consumed entirely by
|
||||
`reasoning_content` with empty `content` and `finish_reason: length`, so
|
||||
harness requests must budget for thinking (same law as the uncensored Swift
|
||||
payload). Vision probe on this build: solid-red 512 px PNG via the OpenAI
|
||||
`image_url` part → `Red`, `prompt_tokens` 151 (the image reached the model).
|
||||
|
||||
Decode is expected ubatch-invariant (Swift-era 3090 law; the 139.9 t/s above
|
||||
is at ubatch 1024 — one rung measured, not a ladder).
|
||||
|
||||
## Architecture facts
|
||||
|
||||
Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144
|
||||
native window, n_vocab 262,144. Hybrid attention: **25 sliding-window layers
|
||||
(window 1024) + 5 full-attention layers (5, 11, 17, 23, 29)** — the KV pool
|
||||
scales with the 5 full layers only (8 KV heads × 256 dim): q8_0 ≈ 10.6
|
||||
KiB/token, f16 ≈ 20.5 KiB/token. A 200,704 slot therefore costs ~2.05 GiB at
|
||||
q8_0 — far less than the Swift 16-layer shape (34 KiB/token).
|
||||
`attention_k_eq_v: True` — in `gemma4.cpp` the absent `wv` tensor makes
|
||||
`Vcur = Kcur` in the compute graph, and in this build (5fc4f3c) the KV cache
|
||||
**honours it: no V slots are allocated**. Measured: the q8_0 pool landed at
|
||||
~2.05 GiB, exactly the K-only figure. **Bill K-only.** The `--cache-type-v`
|
||||
flag is kept explicit for contract visibility; with `k_eq_v` it is inert.
|
||||
native window, n_vocab 262,144. Attention is hybrid per
|
||||
`gemma4.attention.sliding_window_pattern` (read from the file at load): **25
|
||||
sliding-window layers (window 1024, `n_head_kv = 8`) + 5 full-attention layers
|
||||
(indices 5, 11, 17, 23, 29, `n_head_kv = 2`)**, head_dim 256 for K and V in
|
||||
both classes. The KV pool is TWO caches: the full-context cache scales with
|
||||
the 5 layers at the full 200,704 window; the SWA cache is a fixed 1024-token
|
||||
ring per sliding layer.
|
||||
|
||||
**The rate law (measured, exact):** bytes/token = `n_kv_heads × 256 × 6.125 B`
|
||||
per layer at f16 K + q8_0 V — `attention_k_eq_v` does **NOT** shrink the
|
||||
allocation: `Vcur = Kcur` is a compute-graph mapping, the cache reserves K and
|
||||
V slots and, with `k_eq_v`, reserves the pair **twice** (6.125 = 2 × (2 B f16 +
|
||||
1.0625 B q8_0)). Verified against the boot: 5 × 2 × 256 × 6.125 = 15,680
|
||||
B/token × 200,704 = **3,001.25 MiB** ✓ and 25 × 1024 × 8 × 256 × 6.125 =
|
||||
**306.25 MiB** ✓. At q8_0 K + q8_0 V the rate is 2.125 B/elem: the same shape
|
||||
costs 1,041 + 106 = **1,147 MiB**, so the f16-K raise is **+2,161 MiB**. The
|
||||
`--cache-type-v` flag is honoured, not inert.
|
||||
|
||||
The llama.cpp master build at commit `5fc4f3c` supports the arch
|
||||
(`src/models/gemma4.cpp`, `LLM_ARCH_GEMMA4 = "gemma4"`). Note: `grep -c`
|
||||
|
||||
Reference in New Issue
Block a user