# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)

**ACTIVE since 2026-10-07.** RTX 3090, port 7771, root systemd unit
`frank-secondary.service`, served by `scripts/serve-frank-secondary.sh` +
`scripts/frank-secondary.env`. Supersedes
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] (Swift-1.5-Qwen3.8-27B
RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP).

## Payload

|File|Path|Bytes|
|---|---|---|
|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,780,192,888|
|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|~1.195 GiB|
|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|~0.252 GiB|

Repo: `HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP`
(base `google/gemma-4-26B-A4B-it`). Downloaded 2026-10-07 by plain
`curl -fL` (no token needed, ungated).

## Shape

```
--ctx-size 200704 --parallel 1        # ONE conversation, Frank's 200k cap
--flash-attn on
--cache-type-k f16 --cache-type-v q8_0   # exact K, FP8-class V (q8_0)
--batch-size 4096 --ubatch-size 1024
--n-gpu-layers 99
--log-verbosity 4                     # debug: KV layout, buffers, graph capture
--spec-type none                      # MTP OFF (see below)
--mmproj <BF16 projector>
--jinja --reasoning auto --reasoning-format deepseek
```

Measured boot at **q8_0 K + q8_0 V, ubatch 2048, MTP off** (2026-10-07
morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON
(the assistant loaded) the same shape was **23,806 / 319** — the drafter's
context costs ~1.9 GiB because its KV is built against the target's full
window ([[Llama Cpp Serving Sizing]]).

Same evening (Frank): **K raised to f16, ubatch cut 2048 → 1024, verbosity
4**. f16 K doubles the K side (20.5 KiB/token over the 5 full layers ≈ 4.1
GiB at 200,704, +2.05 GiB over q8_0); the ubatch cut claws back part of it
(the Swift-era 3090 measurement put the 2048 rung at ~746 MiB of compute
buffers). Projected usage ~23 GiB — **the boot ledger is the referee, not
this arithmetic**: on a 24,125 MiB card a failed graph capture is the failure
mode. Decode is ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4 —
inference).

## Architecture facts

Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144
native window, n_vocab 262,144. Hybrid attention: **25 sliding-window layers
(window 1024) + 5 full-attention layers (5, 11, 17, 23, 29)** — the KV pool
scales with the 5 full layers only (8 KV heads × 256 dim): q8_0 ≈ 10.6
KiB/token, f16 ≈ 20.5 KiB/token. A 200,704 slot therefore costs ~2.05 GiB at
q8_0 — far less than the Swift 16-layer shape (34 KiB/token).
`attention_k_eq_v: True` — in `gemma4.cpp` the absent `wv` tensor makes
`Vcur = Kcur` in the compute graph, and in this build (5fc4f3c) the KV cache
**honours it: no V slots are allocated**. Measured: the q8_0 pool landed at
~2.05 GiB, exactly the K-only figure. **Bill K-only.** The `--cache-type-v`
flag is kept explicit for contract visibility; with `k_eq_v` it is inert.

The llama.cpp master build at commit `5fc4f3c` supports the arch
(`src/models/gemma4.cpp`, `LLM_ARCH_GEMMA4 = "gemma4"`). Note: `grep -c`
arch names on the packed binary returns 0 for EVERY arch (including the
`qwen3_5` it serves) — check the source tree, not the binary.

## MTP law

MTP for Gemma4 is a SEPARATE file (`mtp-gemma-4-26B-A4B-it.gguf`, arch
`gemma4-assistant`), loaded via `--spec-draft-model` beside
`--spec-type draft-mtp` — unlike Swift, whose MTP head ships inside the main
GGUF. The serve script gained `SPEC_DRAFT_FILE` for this. 2026-10-07 (Frank):
**MTP disabled** — `SPEC_TYPE=none` strips both spec flags (flags move as a
set). The assistant payload stays on disk; setting `SPEC_TYPE=draft-mtp` +
`SPEC_DRAFT_FILE` re-enables it at the cost of ~1.9 GiB (boots at 319 MiB
free).

## Verification

- `curl http://192.168.0.143:7771/health` → `{"status":"ok"}`
- `/v1/models` reports `n_ctx: 200704`, `ftype: Q4_K - Medium`
- Vision probe: solid-color PNG ≥ 256 px via OpenAI `image_url` part,
  expect the exact color word.

## Rollback

Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
payload remains on disk. See
[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]].