docs: frank-small is Gemma4-12B QAT now — new runbook, Qwen retired as rollback target
AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no longer why), and the runbook index. New frank-small-gemma4-12b runbook: payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and the fit-probe error that looks like a fault at boot. The Qwen runbook and both indexes carry the retirement; work log updated.
This commit is contained in:
Frank
@@ -0,0 +1,148 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M
|
||||
description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves.
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
- dankclaw
|
||||
- llama.cpp
|
||||
- rtx3090
|
||||
- gemma
|
||||
- mtp
|
||||
- vision
|
||||
timestamp: 2026-09-17
|
||||
---
|
||||
|
||||
# frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT)
|
||||
|
||||
> **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]]
|
||||
> (one-day rollback target). Same unit, same port, same client contract —
|
||||
> the model, the drafter shape and the KV type changed.
|
||||
|
||||
## Identity
|
||||
|
||||
| what | value |
|
||||
|---|---|
|
||||
| runtime | mainline llama.cpp master `37b53fd`, CUDA build **with `-DGGML_CUDA_FA_ALL_QUANTS=ON`** (rebuilt 2026-09-17 for q8 KV) |
|
||||
| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
|
||||
| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
|
||||
| model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M |
|
||||
| model dir | `/home/dank/models/Gemma4-12B-QAT/` |
|
||||
| port | 7771 (alias `frank-small`; `frank-large` keeps 7770) |
|
||||
| window | `--fit on` → **262,144** ctx, 1 slot; q8_0 K and V |
|
||||
| vision | fleet image path via BF16 mmproj (`input: [text, image]`) |
|
||||
|
||||
## Payload (all sha256-verified against the Hub)
|
||||
|
||||
| file | bytes | sha256 (head…) |
|
||||
|---|---|---|
|
||||
| `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf` | 7,381,381,760 | `59656d74…` |
|
||||
| `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf` | 175,115,264 | `b59e8154…` |
|
||||
| `mtp-gemma-4-12B-it.gguf` | 253,707,328 | `c50c91c3…` |
|
||||
|
||||
`sha256.txt` beside them on the host carries the full digests.
|
||||
|
||||
## What the preflight found (GGUF headers, read over HTTP Range)
|
||||
|
||||
- **Target** (`arch gemma4`, 48 blocks): ctx 262144, MHA — 16 kv heads x
|
||||
512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30,
|
||||
**sliding_window 1024**, `swa_type = STANDARD`, per-layer token embeddings
|
||||
(PLE) via `LLM_KV_EMBEDDING_LENGTH_PER_LAYER`. 12 full-attention layers
|
||||
every 4th (`[3,7,…,47]` implied by `full_attention_interval = 4`).
|
||||
48 blocks maps to `LLM_TYPE_UNKNOWN` in the loader's size switch —
|
||||
cosmetic only.
|
||||
- **mmproj**: clip, vision 224/16 patches, **also carries an audio
|
||||
encoder** (`has_audio_encoder=True`) — vision is the fleet contract,
|
||||
audio is inert bonus.
|
||||
- **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure
|
||||
assistant (no trunk). Its KV **shares** the target's last layers — the
|
||||
share lambda at `llama-model.cpp:2662` references
|
||||
`llama_model_n_layer(model_other) - 2` for SWA layers. Adding the
|
||||
drafter therefore costs no second KV pool.
|
||||
|
||||
## KV arithmetic — why 262k fits
|
||||
|
||||
Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) =
|
||||
**16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the
|
||||
sliding layers cache only the window; the full-attention layers cache the
|
||||
full context. The engine's own `--fit` measured the combination and took
|
||||
the whole 262,144-token window: total footprint at idle is **11,996 MiB**
|
||||
of the card's 24,576 MiB, weights included.
|
||||
|
||||
This is the practical payoff of a sliding-window architecture: the naive
|
||||
"all-global" worst case was 32 GiB at q8/262k — impossible; the real
|
||||
pattern makes the full window the cheapest possible answer.
|
||||
|
||||
## Spec-decode chain
|
||||
|
||||
`--spec-type draft-mtp,ngram-mod` with `--model-draft
|
||||
$MODEL_DIR/mtp-gemma-4-12B-it.gguf`:
|
||||
|
||||
- `draft-mtp` is context-backed and reads the assistant sidecar. Two
|
||||
context-backed drafters cannot share one process, so the chain carries
|
||||
exactly one.
|
||||
- `ngram-mod` is draftless and attaches unconditionally, catching
|
||||
prompt-echo traffic MTP would miss. It prints **no startup banner** —
|
||||
verify behaviorally via the per-task `draft acceptance` line, not the
|
||||
load log.
|
||||
- Measured at deploy: `draft acceptance = 0.38–0.46, mean len ≈ 2.2`
|
||||
(lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a
|
||||
much smaller model — the point of the swap was the VRAM and the window,
|
||||
not acceptance).
|
||||
|
||||
## Deployed flags
|
||||
|
||||
Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`.
|
||||
Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`,
|
||||
`MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`,
|
||||
`DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`,
|
||||
`SPEC_TYPE=draft-mtp,ngram-mod`, `CTX_SIZE` **unset** (→ `--fit on`),
|
||||
`PARALLEL=1`, `NGL=999`, `FLASH_ATTN=on`, `--jinja`, `--metrics`,
|
||||
greedy `--temp 0.0` server default. The serve script refuses to start on
|
||||
any missing file.
|
||||
|
||||
## Client contract
|
||||
|
||||
- `reasoning_content` carries thinking; `preserve_thinking: true` is sent
|
||||
by the harness to keep it across turns.
|
||||
- **Every `reasoning_effort` level is accepted** (all seven probed
|
||||
2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's
|
||||
`reasoningEffortMap` in `~/.omp/agent/models.yml` is therefore inert
|
||||
but kept as a safety net.
|
||||
- Tools work through the `--jinja` template (`tool_calls` verified).
|
||||
- Images work through the mmproj (verified with a live JPEG).
|
||||
- One slot: a second concurrent request queues behind the first.
|
||||
- Harness: `contextWindow: 262144`, `maxTokens: 32768`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`llama_init_from_model: failed to initialize the context:
|
||||
Gemma4Assistant requires ctx_other to be set` printed at boot** — this
|
||||
is the `--fit` probe, not a fault. The engine cannot build the assistant
|
||||
context without the target during measurement, logs the (self-described
|
||||
normal) error, fits without the extra model, and loads the draft
|
||||
against the real context seconds later. Do not restart over it.
|
||||
- **Draft looks unengaged**: ngram-mod is silent at load; read the
|
||||
`draft acceptance = …` line per task from
|
||||
`journalctl -u frank-small` (the server must not be restarted just to
|
||||
see this).
|
||||
- **Quantized KV regression**: if someone rebuilds `build-cuda` without
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON`, prefill collapses ~12x with q8 KV (see
|
||||
the CUDA lesson in AGENTS.md). Check
|
||||
`grep FA_ALL_QUANTS build-cuda/CMakeCache.txt` after any rebuild.
|
||||
- Logs: `journalctl -u frank-small` (root unit, not quadlet).
|
||||
|
||||
## Rollback
|
||||
|
||||
Redeploying the prior model means repointing `MODEL_DIR`/`MODEL_FILE`/
|
||||
`DRAFT_FILE` back to the Qwen trio and `SPEC_TYPE=draft-mtp` — see
|
||||
[[frank-small-qwen38-gsq-rco]] for the exact deployed values. The Qwen
|
||||
weights were NOT deleted from the host.
|
||||
|
||||
## References
|
||||
|
||||
- `scripts/serve-frank-small.sh` — the actual exec line
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED predecessor (rollback)
|
||||
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
|
||||
- [[x4-pair-experiment]] — why the 3090 serves its own model, not the trunk
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S
|
||||
description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves.
|
||||
description: The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves.
|
||||
tags:
|
||||
- llm
|
||||
- frank-small
|
||||
@@ -16,10 +16,11 @@ timestamp: 2026-09-16
|
||||
|
||||
# frank-small — llama.cpp on the RTX 3090
|
||||
|
||||
> **ACTIVE since 2026-09-16.** Created when Dankclaw's inference was re-split:
|
||||
> [[halogen-flash-server]] took the gfx1151 iGPU back for `frank-large`, which
|
||||
> freed the RTX 3090 for this endpoint. Two endpoints, one host: `frank-large`
|
||||
> on `:7770`, `frank-small` on `:7771`.
|
||||
> **RETIRED 2026-09-17** — replaced by [[frank-small-gemma4-12b]]. Kept as
|
||||
> the rollback target and the record of the deployment it succeeded.
|
||||
> Originally ACTIVE from 2026-09-16, created when Dankclaw's inference was
|
||||
> re-split: [[halogen-flash-server]] took the gfx1151 iGPU back for
|
||||
> `frank-large`, which freed the RTX 3090 for this endpoint.
|
||||
|
||||
## Identity
|
||||
|
||||
|
||||
@@ -87,11 +87,15 @@ frank-small, port 7771]
|
||||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting
|
||||
- [[frank-small-qwen38-gsq-rco]] — ACTIVE frank-small runbook since 2026-09-16
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO IQ3_S on the RTX 3090, with the model's own
|
||||
Multi-Token Prediction head for speculative decoding and the BF16 projector
|
||||
for the fleet's image path
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB
|
||||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
f16 KV at 131k
|
||||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||||
|
||||
Reference in New Issue
Block a user