docs: frank-small Q6_K cutover runbook + measured post-boot numbers
- new active runbook frank-small-llama-mimo-q6k-2x200704.md: 21,276 used / 2,849 true free (inside the 2-3 GiB floor), decode 94.44 t/s vs 104.84 on Q5_K_M - the rung costs a measured -10 % (weight-bandwidth-bound card), 512px red-PNG vision verified, and the new probe-size law (>=256 px: a 10x10 image undersupplies image tokens and the model answers 'video'), nvidia-smi free-field law (driver holds ~450 MiB the subtraction hides) - frank-small-llama-mimo-q5km.md retired; it keeps the boot-pair table and becomes the Q5 rollback reference (3,571 free, 104.84 t/s measured) - AGENTS.md fleet row, endpoint section, --parallel lesson, runbook index (active row + two retired rows); LLM + root index entries - Q6_K GGUF-vs-GGUF diff proven: template and all 46 KV keys byte-identical to Q5_K_M - the vision probe scare was probe size, not payload
This commit is contained in:
1 parent
ff8c0ee264
commit
50459e4e31
5 files changed
+206
-73
No files matched your search
@@ -19,7 +19,7 @@ kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
|
||||
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can
|
||||
read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE
|
||||
session at 262,144, f16 KV, vision via the BF16 projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
@@ -81,13 +81,18 @@ frank-small, port 7771]
|
||||
ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the
|
||||
hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on
|
||||
disk
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02, resliced
|
||||
2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
TWO slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K
|
||||
+ f16 V) and the mmproj for vision, 104.8 tok/s decode, 4,046 MiB of the
|
||||
card left free. The boot-pair pool arithmetic, the slot-window contract,
|
||||
rollback to 3 x 180,224 or Q6_K
|
||||
- [[frank-small-llama-mimo-q6k-2x200704]] — **ACTIVE since 2026-10-03**
|
||||
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090, TWO
|
||||
slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K +
|
||||
f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a
|
||||
measured -10 % vs Q5_K_M: weight-bandwidth-bound), 2,849 MiB of the card
|
||||
left free. Probe-size trap (>=256 px images or the model says "video")
|
||||
documented. Rollback: one MODEL_FILE line back to Q5_K_M
|
||||
- [[frank-small-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
@@ -252,8 +257,10 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02 (rollback
|
||||
target)**: llama.cpp (master, CUDA sm_86 build) serving
|
||||
- [[frank-small-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||||
[[frank-small-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
|
||||
Reference in new issue
Block a user