docs(frank-small): record the 3 x 180,224 + vision shape and what it cost

Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new
Key Lessons, one stale SGLang label in the bench list) and the bundle log now
describe the deployed shape: pool 540,672, 180,224 tokens per conversation,
f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8
tok/s decode, 3,814 t/s prefill.

The claims that mattered:
- three contexts buy 180,224 tokens per conversation, not the native 262,144,
  and a longer prompt is a hard 400 - so models.yml contextWindow is 180224
  and frank-primary's 183,501-token compaction request no longer fits here
  (forward pointer added to the retired SGLang runbook, which asserted it)
- the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting
  two measured boots, not by a formula
- /v1/chat/completions ignores the legacy top-level 'images' field: the model
  invents a color; only the image_url content part reaches it
This commit is contained in:
Frank committed 2026-10-02 03:46:50 -07:00
1 parent 7fe881de01
commit 934285c2b2
7 files changed
+245 -67

No files matched your search

+68
View File
@@ -925,3 +925,71 @@ any repo service.
`~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
drafter, not a remnant).
## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
Frank's order was the highest quality of
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
llama.cpp, gufo and every other loader on the fleet cannot read — only the
kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
(n-max 4, `--spec-draft-device ROCm0`).
- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
(+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
routing variant was chosen over the card's plain Q8_0 because it keeps 187
output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
0.953 vs 0.911.
- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
fetchers moved to `old/`. Runbook
[[frank-large-rocmfpx-q8agent-mtp]].
## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
with two contexts ("whatever context size fits the 3090"), then **three
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
the 3090", then **"frank-small should have vision"**.
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
**21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
- **The pool came from two boots, not a formula.** 2 x 262,144 with no
projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
booted healthy but left only 1,153 MiB free, which is how the band got
violated and why the seam now prices non-KV at the measured 9 GiB and
REJECTS that shape.
- **Three contexts cost window, and that is the tradeoff Frank was told:**
180,224 tokens per conversation, not the native 262,144. A longer prompt is
a hard `400 exceed_context_size_error`, so `models.yml` sets
`contextWindow: 180224`. Consequence: frank-primary's 183,501-token
compaction request no longer fits here with summary room; the two-slot
rollback is the way back.
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
`reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
top-level `images` field — that request answered "Brown." for the same red
image, i.e. it generated from text alone. Only the `image_url` content part
reaches the model. `/completion` accepts `images` but this build still
requires `messages`.
- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
empty) in the env file is the documented text-only opt-out and returns ~1.4
GiB to KV.
- Reclaim staged, **not purged, awaiting approval**:
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].