dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-small Q6_K cutover runbook + measured post-boot numbers

Browse Source 
- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used /
  2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on
  Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card),
  512px red-PNG vision verified, and the new probe-size law (>=256 px: a
  10x10 image undersupplies image tokens and the model answers 'video'),
  nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides)
- frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and
  becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured)
- AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index
  (active row + two retired rows); LLM + root index entries
- Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical
  to Q5_K_M - the vision probe scare was probe size, not payload
This commit is contained in:
Frank
2026-10-03 11:20:33 -07:00
parent ff8c0ee264
commit 50459e4e31
5 changed files with 206 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/index.md
+14 -10
View File
@@ -288,16 +288,20 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
2026-10-02, resliced 2026-10-03: a CUDA (sm_86) build of llama.cpp master
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's
RTX 3090, port 7771 — TWO conversations at 200,704 tokens each with fully
unquantized f16 KV, vision through the repo's mmproj, and the boot-pair
arithmetic that sizes the pool; 104.8 tok/s decode, 4,046 MiB free
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
- [[frank-small-llama-mimo-q6k-2x200704]] — the ACTIVE frank-small runbook
since 2026-10-03: llama.cpp master (CUDA sm_86) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on Dankclaw's RTX 3090,
port 7771 — TWO conversations at 200,704 tokens each with fully
unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung
costs a measured −10 % vs Q5_K_M — weight-bandwidth), 2,849 MiB free, and
the ≥256 px image-probe trap (tiny images answer "video")
- [[frank-small-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
target, payload on disk; owns the measured boot-pair table): same runtime
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
[[frank-small-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
SGLang in a self-contained python3.12 venv serving
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5