docs(frank-small): record the 3 x 180,224 + vision shape and what it cost
Runbook, both indexes, AGENTS.md (topology row, endpoint section, three new Key Lessons, one stale SGLang label in the bench list) and the bundle log now describe the deployed shape: pool 540,672, 180,224 tokens per conversation, f16 K + q8_0 V, mmproj vision, 21,684 used / 2,441 free of 24,576 MiB, 106.8 tok/s decode, 3,814 t/s prefill. The claims that mattered: - three contexts buy 180,224 tokens per conversation, not the native 262,144, and a longer prompt is a hard 400 - so models.yml contextWindow is 180224 and frank-primary's 183,501-token compaction request no longer fits here (forward pointer added to the retired SGLang runbook, which asserted it) - the vision tower bills ~1.4 GiB for a 918 MB file, established by subtracting two measured boots, not by a formula - /v1/chat/completions ignores the legacy top-level 'images' field: the model invents a color; only the image_url content part reaches it
This commit is contained in:
1 parent
7fe881de01
commit
934285c2b2
7 files changed
+245
-67
No files matched your search
@@ -925,3 +925,71 @@ any repo service.
|
||||
`~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
|
||||
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
|
||||
drafter, not a remnant).
|
||||
|
||||
## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
|
||||
|
||||
Frank's order was the highest quality of
|
||||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
|
||||
runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
|
||||
carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
|
||||
llama.cpp, gufo and every other loader on the fleet cannot read — only the
|
||||
kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
|
||||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
|
||||
`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
|
||||
bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
|
||||
(n-max 4, `--spec-draft-device ROCm0`).
|
||||
|
||||
- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
|
||||
(+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
|
||||
mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
|
||||
routing variant was chosen over the card's plain Q8_0 because it keeps 187
|
||||
output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
|
||||
0.953 vs 0.911.
|
||||
- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
|
||||
- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
|
||||
fetchers moved to `old/`. Runbook
|
||||
[[frank-large-rocmfpx-q8agent-mtp]].
|
||||
|
||||
## 2026-10-02 — frank-small: Q5_K_M, then three contexts + vision, sized by two boots
|
||||
|
||||
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
|
||||
with two contexts ("whatever context size fits the 3090"), then **three
|
||||
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
|
||||
the 3090", then **"frank-small should have vision"**.
|
||||
|
||||
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
|
||||
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
|
||||
**21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
|
||||
- **The pool came from two boots, not a formula.** 2 x 262,144 with no
|
||||
projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
|
||||
with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
|
||||
difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
|
||||
encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
|
||||
booted healthy but left only 1,153 MiB free, which is how the band got
|
||||
violated and why the seam now prices non-KV at the measured 9 GiB and
|
||||
REJECTS that shape.
|
||||
- **Three contexts cost window, and that is the tradeoff Frank was told:**
|
||||
180,224 tokens per conversation, not the native 262,144. A longer prompt is
|
||||
a hard `400 exceed_context_size_error`, so `models.yml` sets
|
||||
`contextWindow: 180224`. Consequence: frank-primary's 183,501-token
|
||||
compaction request no longer fits here with summary room; the two-slot
|
||||
rollback is the way back.
|
||||
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
|
||||
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
|
||||
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
|
||||
queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
|
||||
`reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
|
||||
- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
|
||||
top-level `images` field — that request answered "Brown." for the same red
|
||||
image, i.e. it generated from text alone. Only the `image_url` content part
|
||||
reaches the model. `/completion` accepts `images` but this build still
|
||||
requires `messages`.
|
||||
- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
|
||||
mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
|
||||
empty) in the env file is the documented text-only opt-out and returns ~1.4
|
||||
GiB to KV.
|
||||
- Reclaim staged, **not purged, awaiting approval**:
|
||||
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
|
||||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||||
gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]].
|
||||
Reference in new issue
Block a user