docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it
This commit is contained in:
Frank
2026-10-02 03:46:50 -07:00
parent 7fe881de01
commit 934285c2b2
7 changed files with 245 additions and 67 deletions
+3 -2
View File
@@ -76,8 +76,9 @@ frank-small, port 7771]
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork