docs(frank-small): record the 3 x 180,224 + vision shape and what it cost
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new Key Lessons, one stale SGLang label in the bench list) and the bundle log now describe the deployed shape: pool 540,672, 180,224 tokens per conversation, f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. The claims that mattered: - three contexts buy 180,224 tokens per conversation, not the native 262,144, and a longer prompt is a hard 400 - so models.yml contextWindow is 180224 and frank-primary's 183,501-token compaction request no longer fits here (forward pointer added to the retired SGLang runbook, which asserted it) - the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting two measured boots, not by a formula - /v1/chat/completions ignores the legacy top-level 'images' field: the model invents a color; only the image_url content part reaches it
This commit is contained in:
1 parent
7fe881de01
commit
934285c2b2
7 files changed
+245
-67
No files matched your search
@@ -285,8 +285,9 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
|
||||
2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
|
||||
3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract
|
||||
(f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode
|
||||
3090, port 7771 — THREE conversations at 180,224 tokens each, the
|
||||
asymmetric KV contract (f16 K + q8_0 V), vision through the repo's mmproj,
|
||||
and the two VRAM measurements that set the pool size; ~100 tok/s decode
|
||||
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
|
||||
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
|
||||
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
|
||||
|
||||
Reference in new issue
Block a user