docs(frank-small): record the 3 x 180,224 + vision shape and what it cost
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new Key Lessons, one stale SGLang label in the bench list) and the bundle log now describe the deployed shape: pool 540,672, 180,224 tokens per conversation, f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. The claims that mattered: - three contexts buy 180,224 tokens per conversation, not the native 262,144, and a longer prompt is a hard 400 - so models.yml contextWindow is 180224 and frank-primary's 183,501-token compaction request no longer fits here (forward pointer added to the retired SGLang runbook, which asserted it) - the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting two measured boots, not by a formula - /v1/chat/completions ignores the legacy top-level 'images' field: the model invents a color; only the image_url content part reaches it
This commit is contained in:
@@ -13,8 +13,9 @@
|
||||
# KV = 14,113 MiB => non-KV WITH the vision tower = 8,859 MiB
|
||||
# The tower costs ~1.4 GiB, not the ~1.0 its file size suggested, so 196,608
|
||||
# per slot left only 1.1 GiB free — outside the band. The pool therefore is
|
||||
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB used,
|
||||
# ~2.7 GiB free, the middle of what was asked for.
|
||||
# 3 x 180,224 = 540,672 tokens: KV ~12,936 + non-KV 8,859 = ~21,800 MiB predicted.
|
||||
# MEASURED at this shape: 21,684 MiB used / 2,441 MiB free — inside the band, and
|
||||
# the number the bats budget gate reproduces (it prices non-KV at 9 GiB).
|
||||
#
|
||||
# KV arithmetic: attention KV exists only in the 8 full-attention layers
|
||||
# (4 KV heads x 256 head_dim). K f16 = 16 KiB/token, V q8_0 = 8.5 KiB/token
|
||||
|
||||
Reference in New Issue
Block a user