docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it
This commit is contained in:
Frank
2026-10-02 03:46:50 -07:00
parent 7fe881de01
commit 934285c2b2
7 changed files with 245 additions and 67 deletions
+3 -2
View File
@@ -13,8 +13,9 @@
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB used,
# ~2.7 GiB free, the middle of what was asked for.
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
#
# KV arithmetic: attention KV exists only in the 8 full-attention layers
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token