docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it
This commit is contained in:
Frank committed 2026-10-02 03:46:50 -07:00
1 parent 7fe881de01
commit 934285c2b2
7 files changed
+245 -67

No files matched your search

+3 -2
View File
@@ -285,8 +285,9 @@ The LLM inference fleet on the two hosts. Full index:
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract
(f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode
3090, port 7771 — THREE conversations at 180,224 tokens each, the
asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
and the two VRAM measurements that set the pool size; ~100 tok/s decode
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
llama.cpp + GGUF shape for this endpoint, Q6_K at the same