frank-secondary: swap Swift-1.5 RCO-IQ3_XXS -> Gemma4-26B-A4B QAT Q4_K_M

Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B) + the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/. Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the 24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch 1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk, not loaded: its draft KV is built against the target's full window, measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise). Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off = 21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319. Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5 full-attention layers) so KV scales with 5 layers only, and attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the measured q8_0 pool = 2.05 GiB = the K-only figure). Serve script gained VERBOSITY (+ --log-verbosity in the argv), the SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4. deploy-remote.sh payload gates follow the env. The frank-secondary bats seam is rewritten for the new contract (argv-based) and fixes a pre-existing vacuity: serve_argv regenerates the test env, so mutations made before it were silently undone - tests that mutate now exec through run_stage. frank-secondary is the fleet compaction hub (200,704 is the only window that fits a worst-case compaction input).
2026-10-07 13:48:54 -07:00
parent 5b7b3ccb90
commit 68a525740d
8 changed files with 335 additions and 222 deletions
@@ -0,0 +1,92 @@
+# frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
+
+**ACTIVE since 2026-10-07.** RTX 3090, port 7771, root systemd unit
+`frank-secondary.service`, served by `scripts/serve-frank-secondary.sh` +
+`scripts/frank-secondary.env`. Supersedes
+[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] (Swift-1.5-Qwen3.8-27B
+RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP).
+
+## Payload
+
+|File|Path|Bytes|
+|---|---|---|
+|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,780,192,888|
+|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|~1.195 GiB|
+|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|~0.252 GiB|
+
+Repo: `HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP`
+(base `google/gemma-4-26B-A4B-it`). Downloaded 2026-10-07 by plain
+`curl -fL` (no token needed, ungated).
+
+## Shape
+
+```
+--ctx-size 200704 --parallel 1        # ONE conversation, Frank's 200k cap
+--flash-attn on
+--cache-type-k f16 --cache-type-v q8_0   # exact K, FP8-class V (q8_0)
+--batch-size 4096 --ubatch-size 1024
+--n-gpu-layers 99
+--log-verbosity 4                     # debug: KV layout, buffers, graph capture
+--spec-type none                      # MTP OFF (see below)
+--mmproj <BF16 projector>
+--jinja --reasoning auto --reasoning-format deepseek
+```
+
+Measured boot at **q8_0 K + q8_0 V, ubatch 2048, MTP off** (2026-10-07
+morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON
+(the assistant loaded) the same shape was **23,806 / 319** — the drafter's
+context costs ~1.9 GiB because its KV is built against the target's full
+window ([[Llama Cpp Serving Sizing]]).
+
+Same evening (Frank): **K raised to f16, ubatch cut 2048 → 1024, verbosity
+4**. f16 K doubles the K side (20.5 KiB/token over the 5 full layers ≈ 4.1
+GiB at 200,704, +2.05 GiB over q8_0); the ubatch cut claws back part of it
+(the Swift-era 3090 measurement put the 2048 rung at ~746 MiB of compute
+buffers). Projected usage ~23 GiB — **the boot ledger is the referee, not
+this arithmetic**: on a 24,125 MiB card a failed graph capture is the failure
+mode. Decode is ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4 —
+inference).
+
+## Architecture facts
+
+Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144
+native window, n_vocab 262,144. Hybrid attention: **25 sliding-window layers
+(window 1024) + 5 full-attention layers (5, 11, 17, 23, 29)** — the KV pool
+scales with the 5 full layers only (8 KV heads × 256 dim): q8_0 ≈ 10.6
+KiB/token, f16 ≈ 20.5 KiB/token. A 200,704 slot therefore costs ~2.05 GiB at
+q8_0 — far less than the Swift 16-layer shape (34 KiB/token).
+`attention_k_eq_v: True` — in `gemma4.cpp` the absent `wv` tensor makes
+`Vcur = Kcur` in the compute graph, and in this build (5fc4f3c) the KV cache
+**honours it: no V slots are allocated**. Measured: the q8_0 pool landed at
+~2.05 GiB, exactly the K-only figure. **Bill K-only.** The `--cache-type-v`
+flag is kept explicit for contract visibility; with `k_eq_v` it is inert.
+
+The llama.cpp master build at commit `5fc4f3c` supports the arch
+(`src/models/gemma4.cpp`, `LLM_ARCH_GEMMA4 = "gemma4"`). Note: `grep -c`
+arch names on the packed binary returns 0 for EVERY arch (including the
+`qwen3_5` it serves) — check the source tree, not the binary.
+
+## MTP law
+
+MTP for Gemma4 is a SEPARATE file (`mtp-gemma-4-26B-A4B-it.gguf`, arch
+`gemma4-assistant`), loaded via `--spec-draft-model` beside
+`--spec-type draft-mtp` — unlike Swift, whose MTP head ships inside the main
+GGUF. The serve script gained `SPEC_DRAFT_FILE` for this. 2026-10-07 (Frank):
+**MTP disabled** — `SPEC_TYPE=none` strips both spec flags (flags move as a
+set). The assistant payload stays on disk; setting `SPEC_TYPE=draft-mtp` +
+`SPEC_DRAFT_FILE` re-enables it at the cost of ~1.9 GiB (boots at 319 MiB
+free).
+
+## Verification
+
+- `curl http://192.168.0.143:7771/health` → `{"status":"ok"}`
+- `/v1/models` reports `n_ctx: 200704`, `ftype: Q4_K - Medium`
+- Vision probe: solid-color PNG ≥ 256 px via OpenAI `image_url` part,
+  expect the exact color word.
+
+## Rollback
+
+Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
+`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
+payload remains on disk. See
+[[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]].
@@ -16,10 +16,9 @@ digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
 Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
 MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
 ctx, vision ON)** and
-**llama.cpp CUDA (frank-secondary, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
-Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file
-IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's
-128k cap (the tightest fit in the endpoint's history — 489 MiB free),
+**llama.cpp CUDA (frank-secondary, port 7771: HauhauCS Gemma4-26B-A4B QAT
+Uncensored Q4_K_M GGUF — 26B MoE hybrid (25 sliding-window + 5 full-attention
+layers), ONE 200,704-token conversation, MTP disabled,
 q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
 mmproj, on the RTX 3090)**.

@@ -52,8 +51,8 @@ frank-large, port 7770]
 2 x 200192 ctx, MTP always-on, image input]
        G[llama.cpp master CUDA sm_86
 frank-secondary, port 7771]
-        H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF
-2 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
+        H[Gemma4-26B-A4B QAT Uncensored Q4_K_M GGUF
+1 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
        E2 --> E --> F
        E2 --> G --> H
    end
@@ -103,8 +102,17 @@ frank-secondary, port 7771]
  (`slow`/`advisor` → frank-primary) carried forward. Superseded by the
  MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
  engine moved with the model
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **ACTIVE since
-  2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
+- [[frank-secondary-llama-gemma4-q4km-200704]] — **ACTIVE since 2026-10-07**
+  (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
+  HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP **Q4_K_M**
+  on the RTX 3090 — ONE 200,704-token conversation, q8_0 K + q8_0 V,
+  vision via the publisher's mmproj. MTP is OFF (the gemma4-assistant
+  drafter costs ~1.9 GiB: 23,806 used/319 free with it, 21,890/2,235
+  without). Supersedes the Swift-1.5 era (below).
+- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
+  2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
+  ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
+  MTP back on since
  2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
  114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
  systemd unit, port
@@ -372,8 +372,16 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — the ACTIVE
-  frank-secondary runbook since 2026-10-05 (evening; **UNCECERSED RCO-IQ3_XXS
+- [[frank-secondary-llama-gemma4-q4km-200704]] — the **ACTIVE** frank-secondary
+  runbook since 2026-10-07: llama.cpp master (CUDA sm_86) serving
+  HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
+  200,704-token conversation, q8_0 K + q8_0 V, MTP **disabled** (the
+  `gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
+  with it, 21,890/2,235 without), vision through the publisher's mmproj
+- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
+  (superseded by gemma4; payload on disk as the rollback): the
+  frank-secondary runbook 2026-10-05 (evening) → 2026-10-07
+  (**UNCECERSED RCO-IQ3_XXS
  with MTP back on since 2026-10-06 evening; renamed from frank-small and
  reshaped to TWO 114,688-token conversations at Frank's 112k cap with
  the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA