dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

Browse Source 
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM
- Restructure log.md to OKF (## YYYY-MM-DD, newest-first)
- Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change
- Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe)
- Write active runbook bodies from ground truth (frank-large, frank-small)
- Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026)
- Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs
- All 336 tests pass; 136 pages, 0 unresolved wikilinks
This commit is contained in:
Frank committed 2026-09-22 02:09:41 -07:00
1 parent 24f7a7772d
commit 93d1181fee
103 files changed
+2047 -4329
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md
+81 -146
View File
@@ -1,7 +1,7 @@
---
type: reference
title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M
description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves.
title: "frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
description: "The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-small
@@ -14,172 +14,107 @@ tags:
timestamp: 2026-09-17
---
# frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT)
# frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
> **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]]
> (one-day rollback target). Same unit, same port, same client contract —
> the model, the drafter shape and the KV type changed.
> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
> [[frank-small-vllm-gemma4]] (the same-day vLLM experiment, also the
> rollback target). Same port (7771), same alias (`frank-small`), same
> client contract surface.
## Identity
| what | value |
|---|---|
| runtime | mainline llama.cpp master `37b53fd`, CUDA build **with `-DGGML_CUDA_FA_ALL_QUANTS=ON`** (rebuilt 2026-09-17 for q8 KV) |
| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
| model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M |
| model dir | `/home/dank/models/Gemma4-12B-QAT/` |
| port | 7771 (alias `frank-small`; `frank-large` keeps 7770) |
| window | **4 slots x 262,144** (`CTX_SIZE=1048576`, `PARALLEL=4`); q8_0 K and V |
| vision + audio | fleet image path via BF16 mmproj (`input: [text, image]`); `input_audio` transcribes verbatim (verified) |
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
collapses prefill ~12x on CUDA.
- **Service shape:** root systemd unit `scripts/frank-small.service`
(`User=dank`), launched by `scripts/serve-frank-small.sh` with tunables from
`scripts/frank-small.env`.
## Payload (all sha256-verified against the Hub)
## Shape
| file | bytes | sha256 (head…) |
|---|---|---|
| `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf` | 7,381,381,760 | `59656d74…` |
| `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf` | 175,115,264 | `b59e8154…` |
| `mtp-gemma-4-12B-it.gguf` | 253,707,328 | `c50c91c3…` |
- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
token embeddings (PLE) inside the file, and a sliding-window attention
pattern (`attention.sliding_window = 1024`) — most layers cache only a
1024-position window, which is why the KV pool is a fraction of the
full-ctx x all-layers number.
- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
`PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
maximises the TOTAL pool, which would shrink each slot below the full window
once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
the card. 4 is the measured max.
- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
only tokenizes and samples (its CPU belongs to the halogen engine serving
frank-large).
`sha256.txt` beside them on the host carries the full digests.
## Spec method
## What the preflight found (GGUF headers, read over HTTP Range)
The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
with `--spec-type draft-mtp`. The same `--spec-type` value chains the
draftless **ngram-mod** matcher, which attaches unconditionally; two
context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
depth 4 + poll 100 decoded WORSE at long context — the longer verification
round costs more attention re-read than the extra accepted token earns).
- **Target** (`arch gemma4`, 48 blocks): ctx 262144, MHA — 16 kv heads x
512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30,
**sliding_window 1024**, `swa_type = STANDARD`, per-layer token embeddings
(PLE) via `LLM_KV_EMBEDDING_LENGTH_PER_LAYER`. 12 full-attention layers
every 4th (`[3,7,…,47]` implied by `full_attention_interval = 4`).
48 blocks maps to `LLM_TYPE_UNKNOWN` in the loader's size switch —
cosmetic only.
- **mmproj**: clip, vision 224/16 patches, **also carries an audio
encoder** (`has_audio_encoder=True`) — vision is the fleet contract,
and audio works: a JFK WAV sent as an `input_audio` content part was
transcribed verbatim (2026-09-17).
- **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure
assistant (no trunk). Its KV **shares** the target's last layers — the
share lambda at `llama-model.cpp:2662` references
`llama_model_n_layer(model_other) - 2` for SWA layers. Adding the
drafter therefore costs no second KV pool.
## KV
## KV arithmetic — why four full windows fit
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
the >12.8 GiB floor does not apply.
Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) =
**16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the
sliding layers cache only the window; the full-attention layers cache the
full context — the pattern that makes the window cheap (the naive
"all-global" worst case was 32 GiB at q8/262k, impossible).
## Vision + audio
Measured on the 3090 (24,576 MiB): one full-context slot idles at
11,996 MiB, four at 22,346 MiB — 3,450 MiB per slot, ~8,546 MiB fixed.
A fifth slot projects past the card, so **4 x 262,144 is the ceiling**.
`CTX_SIZE` is set explicitly to `PARALLEL x 262144` rather than left for
`--fit`: fit maximises the TOTAL pool and would divide a smaller window
per slot at `parallel > 1`. The slot-grid seam in
`tests/frank_small_unit.bats` pins both the full-window division and the
budget inequality so an oversized bump fails pre-deploy, not in boot.
`--mmproj` loads the BF16 projector
(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
52M params), which also carries an audio encoder. The fleet's image path
lives here. Audio works: `input_audio` content parts transcribe verbatim
(verified 2026-09-17 with a 16k mono WAV).
## Spec-decode chain
## Payload
`--spec-type draft-mtp,ngram-mod` with `--model-draft
$MODEL_DIR/mtp-gemma-4-12B-it.gguf`:
- `draft-mtp` is context-backed and reads the assistant sidecar. Two
context-backed drafters cannot share one process, so the chain carries
exactly one.
- `ngram-mod` is draftless and attaches unconditionally, catching
prompt-echo traffic MTP would miss. It prints **no startup banner** —
verify behaviorally via the per-task `draft acceptance` line, not the
load log.
- Measured at deploy: `draft acceptance = 0.38–0.46, mean len ≈ 2.2`
(lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a
much smaller model — the point of the swap was the VRAM and the window,
not acceptance).
## Throughput tuning (measured 2026-09-17)
Salted API probes at the deployed geometry (4 slots, q8 KV, flash attn),
prefill ~254k tokens / decode at ~253k context:
| batch (b/ub) | draft n-max / poll | pp t/s | tg@250k t/s | idle MiB |
|---|---|---|---|---|
| 2048/512 (old) | 3 / 50 | 912–924 | 90*–204 | 22,346 |
| **8192/1024 (deployed)** | **3 / 50** | **962–965** | **226–302** | **23,300** |
| 8192/1024 | 4 / 100 | 961 | 230–257 | 23,300 |
| 4096/1024 | 4 / 100 | 959 | 205–277 | 23,300 |
| 8192/2048 | — | — | — | OOM |
- **ubatch 1024 is the winner**: +5% max-length prefill, and decode at
250k clearly above the 512 baseline.
- **ubatch 2048 cannot run**: the MTP draft context needs ~3.3 GiB of pp
compute buffer at that ubatch against ~2.2 GiB headroom — it dies in
`common_speculative_init_result` after the main model loads.
- **Deeper draft and busy-wait HURT long-ctx decode**: draft n-max 4 with
poll 100 verified a longer round per token, and at 250k the extra
full-attention re-read outweighs the accepted-token gain (230–257 vs
257–302). Keep 3 / 50.
- **All ubatch-1024 variants idle identically** (23,300 MiB) — compute
buffers are reserved at max graph size regardless of the logical batch,
so 8192 costs nothing over 4096.
- **Concurrency caveat**: a long prefill monopolises the GPU — a decode
running beside a 254k-token prefill drops from ~240 to ~100–135 t/s.
Four slots share one engine with no per-slot QoS; that is inherent, not
a config fault.
- **No CUDA graphs in this build** — `--help` shows zero graph flags; the
decode-launch-overhead lever does not exist here.
## Deployed flags
Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`.
Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`,
`MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`,
`DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`, `CACHE_RAM=2048`,
`SPEC_TYPE=draft-mtp,ngram-mod`, `SPEC_DRAFT_N_MAX=3`, `POLL_LEVEL=50`,
`CTX_SIZE=1048576` with `PARALLEL=4` (four full 262,144-token slots),
`BATCH_SIZE=8192`, `UBATCH_SIZE=1024`, `NGL=999`, `FLASH_ATTN=on`,
`--jinja`, `--metrics`, greedy `--temp 0.0` server default. The serve
script refuses to start on any missing file.
The main GGUF, the BF16 mmproj, and the MTP sidecar live under
`/home/dank/models/Gemma4-12B-QAT`. The serve script refuses to start if any
is missing (a missing file is an unfinished download, and the unit's restart
policy would otherwise turn it into a crash loop that reads like a runtime
fault).
## Client contract
- `reasoning_content` carries thinking; `preserve_thinking: true` is sent
by the harness to keep it across turns.
- **Every `reasoning_effort` level is accepted** (all seven probed
2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's
`reasoningEffortMap` in `~/.omp/agent/models.yml` is therefore inert
but kept as a safety net.
- Tools work through the `--jinja` template (`tool_calls` verified).
- Images work through the mmproj (verified with a live JPEG).
- Four concurrent conversations (one per slot). No per-slot QoS: a long
prefill in one slot slows decodes in the others — see the caveat above.
- Harness: `contextWindow: 262144`, `maxTokens: 32768`.
- **Alias:** `frank-small` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned even though this build defaults
it on, so a future build that flips the default cannot silently take tools
away).
- **Vision + audio:** supported.
- **Sampling:** greedy by default (fleet contract); a client that sends a
temperature gets its own, and the harness applies per-model overrides from
`models.yml`.
## Troubleshooting
## Measured (max length)
- **`Gemma4Assistant requires ctx_other to be set` printed at boot** —
the `--fit` probe, not a fault (fit cannot build the assistant context
during measurement). With `CTX_SIZE` set explicitly, as deployed, this
line does not appear at all. Do not restart over it either way.
- **Draft looks unengaged**: ngram-mod is silent at load; read the
`draft acceptance = …` line per task from
`journalctl -u frank-small` (the server must not be restarted just to
see this).
- **Quantized KV regression**: if someone rebuilds `build-cuda` without
`GGML_CUDA_FA_ALL_QUANTS=ON`, prefill collapses ~12x with q8 KV (see
the CUDA lesson in AGENTS.md). Check
`grep FA_ALL_QUANTS build-cuda/CMakeCache.txt` after any rebuild.
- Logs: `journalctl -u frank-small` (root unit, not quadlet).
At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
(2048) OOM the MTP draft context's pp compute buffer and must not be tried.
## Rollback
Redeploying the prior model means repointing `MODEL_DIR`/`MODEL_FILE`/
`DRAFT_FILE` back to the Qwen trio and `SPEC_TYPE=draft-mtp` — see
[[frank-small-qwen38-gsq-rco]] for the exact deployed values. The Qwen
weights were NOT deleted from the host.
Payload and runtime for [[frank-small-qwen38-gsq-rco]] (ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-small-vllm-gemma4]]
is the same-day vLLM experiment (also a rollback target).
## References
# Citations
- `scripts/serve-frank-small.sh` — the actual exec line
- [[frank-small-qwen38-gsq-rco]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[x4-pair-experiment]] — why the 3090 serves its own model, not the trunk
- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
- llama.cpp: https://github.com/ggml-org/llama.cpp
- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`