frank-secondary: record the deployed f16-K boot ledger, correct the k_eq_v billing law

Verbosity 4 paid for itself: the engine's own llama_kv_cache lines give the
exact accounting — the cache bills the K+V pair TWICE under attention_k_eq_v
(6.125 B/elem at f16 K + q8_0 V), reproducing 3,001.25 MiB (5 full layers at
n_head_kv 2) + 306.25 MiB (25 SWA rings at n_head_kv 8, window 1024) to the
byte. The earlier "bill K-only" note was wrong; full layers have 2 KV heads,
not 8.

Deployed boot: 21,868 used / 2,257 free — the f16-K raise (+2,161 MiB KV) is
funded by the ubatch 2048 -> 1024 cut (~616 MiB of compute buffer) plus
variance, so exact K came essentially free. Measured decode 139.9 t/s temp 0;
vision probe returns Red at prompt_tokens 151.
This commit is contained in:
Frank
2026-10-07 13:58:30 -07:00
parent 68a525740d
commit b51e078e36
3 changed files with 72 additions and 47 deletions
+17 -14
View File
@@ -3,17 +3,19 @@
# (Q4_K_M) on the RTX 3090, :7771.
#
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, q8_0 KV (FP8 class),
# MTP DISABLED (SPEC_TYPE=none).
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V,
# MTP DISABLED (SPEC_TYPE=none), log verbosity 4.
#
# Architecture: gemma4 MoE — 30 layers, 128 experts (8 active),
# hybrid attention: 25 sliding_window + 5 full_attention
# (full at layers [5, 11, 17, 23, 29]), sliding_window 1024,
# 8 KV heads, head_dim 256, attention_k_eq_v (V = K),
# native 262,144 window.
# Architecture (read from the file's own header at load): gemma4 MoE — 30
# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding
# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29,
# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window.
#
# KV: q8_0 K + q8_0 V (FP8 class, 1.0625 B/elem). The build has no
# true fp8 KV dtype; q8_0 is the FP8 class.
# KV rate law (measured, exact): the cache bills the K+V pair TWICE under
# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at
# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x
# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the
# FP8 class.
#
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
# (spec flags move as a set). The assistant file
@@ -35,9 +37,10 @@ CACHE_TYPE_V=q8_0
SPEC_TYPE=none
NGL=99
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph capture),
# ubatch 1024, K raised to f16 (V stays q8_0). f16 K costs +2.05 GiB over q8_0
# (5 full-attn layers x 200,704 tok x 8 heads x 256 dim x 2 B); the MTP-off
# boot had 2,235 MiB free, so this is a deliberate tight fit — read the boot
# ledger, not the arithmetic.
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph
# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of
# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) +
# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63
# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The
# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235.
VERBOSITY=4