docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M

frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115
tensor-type lock-in that forced the fork, the pinned build (set_rows history),
the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live
client contract (reasoning_content, effort fold, vision restored) and rollback.
frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired
(rollback target). gufo runbook marked retired (rollback target). Both indexes,
the fleet topology table, the vision/models.yml notes, the benchmarking law and
three new Key Lessons updated.
This commit is contained in:
Frank
2026-10-02 01:51:11 -07:00
parent 125d04967f
commit 72bd4c967e
7 changed files with 522 additions and 147 deletions
+27 -11
View File
@@ -213,20 +213,27 @@ The LLM inference fleet on the two hosts. Full index:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record)
- [[frank-large-vllm-mimo-v26-bf16]] — the ACTIVE frank-large runbook
since 2026-10-01 (root systemd unit, port 7770): the digest-pinned
- [[frank-large-rocmfpx-q8agent-mtp]] — the ACTIVE frank-large runbook
since 2026-10-02 (root systemd unit, port 7770): the pinned kingjones30/
ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF **Q8_0_ROCMFPX_AGENT** — ftype 115,
311 tensors of GGML_TYPE_Q8_0_ROCMFPX, a format no other loader on the
host reads — with the repo's Q4_0 MTP head drafting 4 tokens and its BF16
projector, one session at the native 262,144 window
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (rollback
target, payload and `scripts/build-gufo.sh` both on disk): gufo v0.1.1
built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
drafter on the gfx1151 iGPU — the two from-source build workarounds, the
template-hash allowlist, the measured prefill/decode trade
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
replaced by the gufo rollback, then by
[[frank-large-rocmfpx-q8agent-mtp]]): the digest-pinned
kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
the `reasoning` field change, troubleshooting
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback via tag
frank-large-gufo-q4kxl-20261001): gufo v0.1.1 built from source serving
Qwen3.8-27B GGUF + the z-lab DFlash2 drafter on the gfx1151 iGPU — the
two from-source build workarounds, the client-contract delta, the
measured prefill/decode trade
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
@@ -275,12 +282,21 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-small-sglang-mimo-v26]] — the ACTIVE frank-small runbook since
2026-10-01: SGLang in a self-contained python3.12 venv serving
- [[frank-small-llama-mimo-q5km]] — the ACTIVE frank-small runbook since
2026-10-02: a CUDA (sm_86) build of llama.cpp master serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q5_K_M** on Dankclaw's RTX
3090, port 7771 — 2 x 262,144 native windows, the asymmetric KV contract
(f16 K + q8_0 V) and the VRAM arithmetic behind it, ~100 tok/s decode
- [[frank-small-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; replaced
by [[frank-small-llama-mimo-q5km]], payload still on disk): the first
llama.cpp + GGUF shape for this endpoint, Q6_K at the same
2 x 262,144 / f16-K + q8_0-V contract, 20.9 GiB measured
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
SGLang in a self-contained python3.12 venv serving
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
4 x 151,552 slots, bf16 mamba states, eager prefill
2 x 262,144 slots, bf16 mamba states, eager prefill
- [[frank-small-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M