Frankandomp
1d062f06a3
Capacity trade: the 9B dense-hybrid (qwen35: 32 layers = 8 full-attention + 24 Gated DeltaNet, 4 KV heads x 256 dim, native 262,144, MTP-less, natively multimodal) costs ~7 GiB of weights where the 26B MoE cost ~16.6 GiB. The freed room buys back the second conversation slot and exact KV: --ctx-size 401408 --parallel 2 (two 200,704 windows), f16 K + f16 V, MTP off (the file carries no nextn/mtp tensors - flag-only enable is impossible), vision via the publisher's BF16 mmproj, ubatch 1024 (the CUDA decode-optimal value), verbosity 4, publisher sampler recipe (0.6/0.95/20/0) in models.yml. Hosts with the MiMo-V2.6-Distill Q6_K boots at this geometry: 21,276 MiB used / 2,849 free at ubatch 1024, 40,960 tokens of fp16 KV at ubatch 2048. Compaction stays routed here; window math is unchanged (maxTokens 16,384 keeps the worst-case harness request inside the slot). Gemma4 payload (Q4_K_M + mmproj + MTP assistant) stays on disk as the rollback; its runbook is retired in place, keeping the attention_k_eq_v pair-billing law. Co-Authored-By: omp <omp@franksdank.cloud>
112 lines
5.6 KiB
Markdown
112 lines
5.6 KiB
Markdown
# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
|
||
|
||
**RETIRED 2026-10-07** — same day it went live. Superseded by
|
||
[[frank-secondary-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
|
||
2 × 200,704 with exact KV); the payload stays at
|
||
`/home/dank/models/Gemma4-26B-A4B/` as the rollback. This page keeps the
|
||
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
|
||
the drafter-cost boots, both of which outlive the model.
|
||
|
||
## Payload
|
||
|
||
|File|Path|Bytes|
|
||
|---|---|---|
|
||
|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,796,015,520 on disk; the engine reports 16,780,192,888|
|
||
|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|~1.195 GiB|
|
||
|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|~0.252 GiB|
|
||
|
||
Repo: `HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP`
|
||
(base `google/gemma-4-26B-A4B-it`). Downloaded 2026-10-07 by plain
|
||
`curl -fL` (no token needed, ungated).
|
||
|
||
## Shape
|
||
|
||
```
|
||
--ctx-size 200704 --parallel 1 # ONE conversation, Frank's 200k cap
|
||
--flash-attn on
|
||
--cache-type-k f16 --cache-type-v q8_0 # exact K, FP8-class V (q8_0)
|
||
--batch-size 4096 --ubatch-size 1024
|
||
--n-gpu-layers 99
|
||
--log-verbosity 4 # debug: KV layout, buffers, graph capture
|
||
--spec-type none # MTP OFF (see below)
|
||
--mmproj <BF16 projector>
|
||
--jinja --reasoning auto --reasoning-format deepseek
|
||
```
|
||
|
||
Measured boot at **q8_0 K + q8_0 V, ubatch 2048, MTP off** (2026-10-07
|
||
morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON
|
||
(the assistant loaded) the same shape was **23,806 / 319** — the drafter's
|
||
context costs ~1.9 GiB because its KV is built against the target's full
|
||
window ([[Llama Cpp Serving Sizing]]).
|
||
|
||
**Deployed shape (2026-10-07 evening, Frank): K = f16, ubatch 1024, verbosity
|
||
4. Measured boot: 21,868 MiB used / 2,257 MiB free** — i.e. the f16-K raise
|
||
(+2,161 MiB of KV, see the accounting below) was paid for by the ubatch cut
|
||
(2048 → 1024 returns ~616 MiB of compute buffer) plus slot-geometry variance.
|
||
Exact K came essentially free. The verbosity-4 ledger (journal, `llama_kv_cache`
|
||
lines): model 16,002.91 MiB (CUDA0) + 577.50 MiB (CPU-mapped), **KV 3,001.25
|
||
MiB (5 full layers) + 306.25 MiB (25 SWA rings)**, compute buffers 926.39 +
|
||
150.63 MiB, output buffer 1.00 MiB.
|
||
|
||
**Measured decode: 139.9 tok/s** (`/completion` timings, temp 0, 256 tokens,
|
||
deployed shape) — the ~4 B-active MoE is at home on the 3090. The model
|
||
thinks: a 256-token budget on a short prompt can be consumed entirely by
|
||
`reasoning_content` with empty `content` and `finish_reason: length`, so
|
||
harness requests must budget for thinking (same law as the uncensored Swift
|
||
payload). Vision probe on this build: solid-red 512 px PNG via the OpenAI
|
||
`image_url` part → `Red`, `prompt_tokens` 151 (the image reached the model).
|
||
|
||
Decode is expected ubatch-invariant (Swift-era 3090 law; the 139.9 t/s above
|
||
is at ubatch 1024 — one rung measured, not a ladder).
|
||
|
||
## Architecture facts
|
||
|
||
Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144
|
||
native window, n_vocab 262,144. Attention is hybrid per
|
||
`gemma4.attention.sliding_window_pattern` (read from the file at load): **25
|
||
sliding-window layers (window 1024, `n_head_kv = 8`) + 5 full-attention layers
|
||
(indices 5, 11, 17, 23, 29, `n_head_kv = 2`)**, head_dim 256 for K and V in
|
||
both classes. The KV pool is TWO caches: the full-context cache scales with
|
||
the 5 layers at the full 200,704 window; the SWA cache is a fixed 1024-token
|
||
ring per sliding layer.
|
||
|
||
**The rate law (measured, exact):** bytes/token = `n_kv_heads × 256 × 6.125 B`
|
||
per layer at f16 K + q8_0 V — `attention_k_eq_v` does **NOT** shrink the
|
||
allocation: `Vcur = Kcur` is a compute-graph mapping, the cache reserves K and
|
||
V slots and, with `k_eq_v`, reserves the pair **twice** (6.125 = 2 × (2 B f16 +
|
||
1.0625 B q8_0)). Verified against the boot: 5 × 2 × 256 × 6.125 = 15,680
|
||
B/token × 200,704 = **3,001.25 MiB** ✓ and 25 × 1024 × 8 × 256 × 6.125 =
|
||
**306.25 MiB** ✓. At q8_0 K + q8_0 V the rate is 2.125 B/elem: the same shape
|
||
costs 1,041 + 106 = **1,147 MiB**, so the f16-K raise is **+2,161 MiB**. The
|
||
`--cache-type-v` flag is honoured, not inert.
|
||
|
||
The llama.cpp master build at commit `5fc4f3c` supports the arch
|
||
(`src/models/gemma4.cpp`, `LLM_ARCH_GEMMA4 = "gemma4"`). Note: `grep -c`
|
||
arch names on the packed binary returns 0 for EVERY arch (including the
|
||
`qwen3_5` it serves) — check the source tree, not the binary.
|
||
|
||
## MTP law
|
||
|
||
MTP for Gemma4 is a SEPARATE file (`mtp-gemma-4-26B-A4B-it.gguf`, arch
|
||
`gemma4-assistant`), loaded via `--spec-draft-model` beside
|
||
`--spec-type draft-mtp` — unlike Swift, whose MTP head ships inside the main
|
||
GGUF. The serve script gained `SPEC_DRAFT_FILE` for this. 2026-10-07 (Frank):
|
||
**MTP disabled** — `SPEC_TYPE=none` strips both spec flags (flags move as a
|
||
set). The assistant payload stays on disk; setting `SPEC_TYPE=draft-mtp` +
|
||
`SPEC_DRAFT_FILE` re-enables it at the cost of ~1.9 GiB (boots at 319 MiB
|
||
free).
|
||
|
||
## Verification
|
||
|
||
- `curl http://192.168.0.143:7771/health` → `{"status":"ok"}`
|
||
- `/v1/models` reports `n_ctx: 200704`, `ftype: Q4_K - Medium`
|
||
- Vision probe: solid-color PNG ≥ 256 px via OpenAI `image_url` part,
|
||
expect the exact color word.
|
||
|
||
## Rollback
|
||
|
||
Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
|
||
`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
|
||
payload remains on disk. See
|
||
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]].
|