0 Fork 0

Files

T

Frank 68a525740d frank-secondary: swap Swift-1.5 RCO-IQ3_XXS -> Gemma4-26B-A4B QAT Q4_K_M

Payload: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored Q4_K_M (16,796,015,520 B)
+ the publisher's BF16 mmproj, /home/dank/models/Gemma4-26B-A4B/.

Shape: ONE 200,704-token slot (--parallel 1; 2x200,704 does not fit the
24 GiB card), f16 K + q8_0 V under flash attention, batch 4096 / ubatch
1024, --log-verbosity 4, MTP OFF (the gemma4-assistant file is on disk,
not loaded: its draft KV is built against the target's full window,
measured +1.9 GiB -> 23,806/319 free, no margin beside the f16-K raise).

Measured boots (nvidia-smi): q8_0 K + q8_0 V / ubatch 2048 / MTP off =
21,890 used / 2,235 free; with the MTP assistant = 23,806 / 319.

Engine facts recorded: gemma4 hybrid attention (25 sliding-window + 5
full-attention layers) so KV scales with 5 layers only, and
attention_k_eq_v is honoured by build 5fc4f3c (pool bills K-only; the
measured q8_0 pool = 2.05 GiB = the K-only figure).

Serve script gained VERBOSITY (+ --log-verbosity in the argv), the
SPEC_DRAFT_FILE drafter path, and defaults/lore retargeted to Gemma4.
deploy-remote.sh payload gates follow the env. The frank-secondary bats
seam is rewritten for the new contract (argv-based) and fixes a
pre-existing vacuity: serve_argv regenerates the test env, so
mutations made before it were silently undone - tests that mutate now
exec through run_stage. frank-secondary is the fleet compaction hub
(200,704 is the only window that fits a worst-case compaction input).

2026-10-07 13:48:54 -07:00

4.4 KiB

Raw Blame History

frank-secondary — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)

ACTIVE since 2026-10-07. RTX 3090, port 7771, root systemd unit frank-secondary.service, served by scripts/serve-frank-secondary.sh + scripts/frank-secondary.env. Supersedes frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688 (Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS-mtp, 2×114,688, in-file MTP).

Payload

File	Path	Bytes
Main (Q4_K_M)	`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`	16,780,192,888
Vision (mmproj BF16)	`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`	~1.195 GiB
MTP assistant (on disk, NOT loaded)	`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`	~0.252 GiB

Repo: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP (base google/gemma-4-26B-A4B-it). Downloaded 2026-10-07 by plain curl -fL (no token needed, ungated).

Shape

--ctx-size 200704 --parallel 1        # ONE conversation, Frank's 200k cap
--flash-attn on
--cache-type-k f16 --cache-type-v q8_0   # exact K, FP8-class V (q8_0)
--batch-size 4096 --ubatch-size 1024
--n-gpu-layers 99
--log-verbosity 4                     # debug: KV layout, buffers, graph capture
--spec-type none                      # MTP OFF (see below)
--mmproj <BF16 projector>
--jinja --reasoning auto --reasoning-format deepseek

Measured boot at q8_0 K + q8_0 V, ubatch 2048, MTP off (2026-10-07 morning): 21,890 MiB used / 2,235 MiB free (nvidia-smi). With MTP ON (the assistant loaded) the same shape was 23,806 / 319 — the drafter's context costs ~1.9 GiB because its KV is built against the target's full window (Llama Cpp Serving Sizing).

Same evening (Frank): K raised to f16, ubatch cut 2048 → 1024, verbosity 4. f16 K doubles the K side (20.5 KiB/token over the 5 full layers ≈ 4.1 GiB at 200,704, +2.05 GiB over q8_0); the ubatch cut claws back part of it (the Swift-era 3090 measurement put the 2048 rung at ~746 MiB of compute buffers). Projected usage ~23 GiB — the boot ledger is the referee, not this arithmetic: on a 24,125 MiB card a failed graph capture is the failure mode. Decode is ubatch-invariant (Swift-era 3090 law; unmeasured on Gemma4 — inference).

Architecture facts

Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144 native window, n_vocab 262,144. Hybrid attention: 25 sliding-window layers (window 1024) + 5 full-attention layers (5, 11, 17, 23, 29) — the KV pool scales with the 5 full layers only (8 KV heads × 256 dim): q8_0 ≈ 10.6 KiB/token, f16 ≈ 20.5 KiB/token. A 200,704 slot therefore costs ~2.05 GiB at q8_0 — far less than the Swift 16-layer shape (34 KiB/token). attention_k_eq_v: True — in gemma4.cpp the absent wv tensor makes Vcur = Kcur in the compute graph, and in this build (5fc4f3c) the KV cache honours it: no V slots are allocated. Measured: the q8_0 pool landed at ~2.05 GiB, exactly the K-only figure. Bill K-only. The --cache-type-v flag is kept explicit for contract visibility; with k_eq_v it is inert.

The llama.cpp master build at commit 5fc4f3c supports the arch (src/models/gemma4.cpp, LLM_ARCH_GEMMA4 = "gemma4"). Note: grep -c arch names on the packed binary returns 0 for EVERY arch (including the qwen3_5 it serves) — check the source tree, not the binary.

MTP law

MTP for Gemma4 is a SEPARATE file (mtp-gemma-4-26B-A4B-it.gguf, arch gemma4-assistant), loaded via --spec-draft-model beside --spec-type draft-mtp — unlike Swift, whose MTP head ships inside the main GGUF. The serve script gained SPEC_DRAFT_FILE for this. 2026-10-07 (Frank): MTP disabled — SPEC_TYPE=none strips both spec flags (flags move as a set). The assistant payload stays on disk; setting SPEC_TYPE=draft-mtp + SPEC_DRAFT_FILE re-enables it at the cost of ~1.9 GiB (boots at 319 MiB free).

Verification

curl http://192.168.0.143:7771/health → {"status":"ok"}
/v1/models reports n_ctx: 200704, ftype: Q4_K - Medium
Vision probe: solid-color PNG ≥ 256 px via OpenAI image_url part, expect the exact color word.

Rollback

Revert to the Swift trio: MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…, CTX_SIZE=229376, PARALLEL=2, in-file SPEC_TYPE=draft-mtp — the Swift payload remains on disk. See frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.

4.4 KiB Raw Blame History Unescape Escape