docs: frank-small is Gemma4-12B QAT now — new runbook, Qwen retired as rollback target
AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no longer why), and the runbook index. New frank-small-gemma4-12b runbook: payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and the fit-probe error that looks like a fault at boot. The Qwen runbook and both indexes carry the retirement; work log updated.
This commit is contained in:
@@ -793,3 +793,30 @@ timestamp: 2026-09-14
|
||||
contiguous 2 MiB blocks left (240 before the refresh).
|
||||
- Runbook: [[halogen-flash-server]] gains a "Refreshing the weights" note; the
|
||||
quant comparison is appended to [[x4-pair-experiment]].
|
||||
|
||||
## 2026-09-17 — frank-small swapped to Gemma4-12B QAT (Q4_K_M)
|
||||
|
||||
- **Cutover:** `frank-small` (Dankclaw RTX 3090, :7771) replaced
|
||||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with
|
||||
`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M —
|
||||
7.38 GiB main + 175 MiB BF16 mmproj (with audio encoder) + 254 MiB
|
||||
`mtp-gemma-4-12B-it.gguf` assistant sidecar, all sha256-verified.
|
||||
- **Spec decode:** the MTP head is now a separate `gemma4-assistant`
|
||||
file loaded via `--model-draft` (shares the target's last layers' KV,
|
||||
`llama-model.cpp:2662`), chained behind draftless `ngram-mod`:
|
||||
`--spec-type draft-mtp,ngram-mod`. Measured acceptance 0.38–0.46,
|
||||
mean draft len ~2.2.
|
||||
- **KV:** q8_0 K and V; `build-cuda` rebuilt with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON` (required, else ~12x prefill collapse).
|
||||
`CTX_SIZE` unset → `--fit on` chose the full 262,144-token window;
|
||||
idle card usage 11,996 / 24,576 MiB (was 22,362).
|
||||
- **Template behavior:** Gemma4 accepts every `reasoning_effort` level
|
||||
(all seven probed: 200) — the Qwen3.8 gate is gone; harness map kept
|
||||
as a safety net.
|
||||
- **Verified live:** health 200, chat + `reasoning_content`, `tool_calls`,
|
||||
vision (live JPEG), deploy e2e gate passed (31 passed, 1 flaky crawl).
|
||||
- **Harness:** `models.yml` frank-small `contextWindow: 262144`.
|
||||
- **Note:** the boot log's `Gemma4Assistant requires ctx_other to be
|
||||
set` error is `--fit`'s own probe — normal, self-recovers seconds later.
|
||||
- Runbook: [[frank-small-gemma4-12b]] (active);
|
||||
[[frank-small-qwen38-gsq-rco]] retired 2026-09-17 as rollback target.
|
||||
|
||||
Reference in New Issue
Block a user