docs(frank-small): record the 3 x 180,224 + vision shape and what it cost
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new Key Lessons, one stale SGLang label in the bench list) and the bundle log now describe the deployed shape: pool 540,672, 180,224 tokens per conversation, f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. The claims that mattered: - three contexts buy 180,224 tokens per conversation, not the native 262,144, and a longer prompt is a hard 400 - so models.yml contextWindow is 180224 and frank-primary's 183,501-token compaction request no longer fits here (forward pointer added to the retired SGLang runbook, which asserted it) - the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting two measured boots, not by a formula - /v1/chat/completions ignores the legacy top-level 'images' field: the model invents a color; only the image_url content part reaches it
This commit is contained in:
@@ -76,8 +76,9 @@ frank-small, port 7771]
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
|
||||
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
|
||||
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
Reference in New Issue
Block a user