docs: frank-small Q6_K cutover runbook + measured post-boot numbers

- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used /
  2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on
  Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card),
  512px red-PNG vision verified, and the new probe-size law (>=256 px: a
  10x10 image undersupplies image tokens and the model answers 'video'),
  nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides)
- frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and
  becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured)
- AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index
  (active row + two retired rows); LLM + root index entries
- Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical
  to Q5_K_M - the vision probe scare was probe size, not payload
This commit is contained in:
Frank committed 2026-10-03 11:20:33 -07:00
1 parent ff8c0ee264
commit 50459e4e31
5 files changed
+206 -73

No files matched your search

+17 -10
View File
@@ -19,7 +19,7 @@ kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can
read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE
session at 262,144, f16 KV, vision via the BF16 projector)** and
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
> 2026-08-22: the former `vllmScripts` repository is merged into the
@@ -81,13 +81,18 @@ frank-small, port 7771]
ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the
hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on
disk
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02, resliced
2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
TWO slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K
+ f16 V) and the mmproj for vision, 104.8 tok/s decode, 4,046 MiB of the
card left free. The boot-pair pool arithmetic, the slot-window contract,
rollback to 3 x 180,224 or Q6_K
- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
measured -10 % vs Q5_K_M: weight-bandwidth-bound), 2,849 MiB of the card
left free. Probe-size trap (>=256 px images or the model says "video")
documented. Rollback: one MODEL_FILE line back to Q5_K_M
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
target; payload on disk; owns the measured boot-pair table)**: the same
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
104.8 tok/s, 3,571 MiB true free
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
@@ -252,8 +257,10 @@ frank-small, port 7771]
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract,
troubleshooting (2026-09-16 → 09-19)
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
target)**: llama.cpp (master, CUDA sm_86 build) serving
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
[[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
build) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the