docs: frank-small is Gemma4-12B QAT now — new runbook, Qwen retired as rollback target
AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no longer why), and the runbook index. New frank-small-gemma4-12b runbook: payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and the fit-probe error that looks like a fault at boot. The Qwen runbook and both indexes carry the retirement; work log updated.
This commit is contained in:
Frank committed
1 parent
0a25351caa
commit
87caca4894
6 files changed
+226
-29
No files matched your search
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 327680 ctx over 2 slots of 163840, batch 6144x512, vision (BF16 mmproj) | 7770 |
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, `--fit` → 262,144 ctx / 1 slot, q8_0 KV, the fleet's vision endpoint | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -299,20 +299,31 @@ Rollback: [[frank-large-ik-uncensored]].
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
Mainline llama.cpp master `37b53fd` as a **root systemd unit** (repo
|
||||
Mainline llama.cpp master `37b53fd` (CUDA build with
|
||||
`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo
|
||||
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
|
||||
serving `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` at IQ3_S — the `-mtp` build, so
|
||||
the model's own Multi-Token Prediction head drives speculative decoding — with
|
||||
the BF16 mmproj, on port 7771, 131,072 ctx / 1 slot, f16 KV, greedy default.
|
||||
Deployed flags are in `scripts/frank-small.env`; the serve script refuses to
|
||||
start if the weights, the projector or the binary are missing.
|
||||
serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE` unset so `--fit` sizes the pool, 1 slot,
|
||||
greedy default. Deployed flags are in `scripts/frank-small.env`; the serve
|
||||
script refuses to start if the weights, the projector, the sidecar or the
|
||||
binary are missing.
|
||||
|
||||
Verified at deploy (2026-09-16): `/v1/models` reports `frank-small`; a chat
|
||||
completion returns content plus `reasoning_content`; a tool call returns
|
||||
`tool_calls`; the slot timing lines show `draft acceptance = 0.66–0.94, mean
|
||||
len ≈ 3` (MTP engaged); an image request is answered from the projector while
|
||||
the halogen side refuses images with a named reason; 22,362 MiB of the card's
|
||||
24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`.
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; the slot timing lines show `draft acceptance =
|
||||
0.38–0.46, mean len ≈ 2.2` (MTP engaged); an image request is answered
|
||||
from the projector. `--fit` took the FULL window — `n_slots = 1,
|
||||
n_ctx_slot = 262144` — because Gemma4's sliding-window pattern keeps the
|
||||
q8 KV pool cheap: 11,996 MiB of the card's 24,576 MiB in use. The
|
||||
`Gemma4Assistant requires ctx_other to be set` error that prints at boot
|
||||
is the fit's own probe (the log calls it normal); the sidecar then loads
|
||||
against the real context. Gemma4's template accepts every
|
||||
`reasoning_effort` level (all seven probed: 200).
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
|
||||
|
||||
@@ -594,7 +605,9 @@ linked runbooks.
|
||||
returns `tool_calls`. Check tool support before routing a role at an endpoint.
|
||||
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
|
||||
option and the same ~12x prefill collapse described in the ROCm note above.
|
||||
f16 KV needs nothing, which is why frank-small runs f16.
|
||||
f16 KV needs nothing; frank-small ran f16 for that reason until
|
||||
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
|
||||
endpoint could take q8_0 KV.
|
||||
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
|
||||
`-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
|
||||
configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
|
||||
@@ -685,7 +698,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
|
||||
| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
|
||||
| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
@@ -696,6 +709,7 @@ Retired (rollback targets and history):
|
||||
|---------|------|--------|---------|
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
|
||||
Reference in new issue
Block a user