docs: frank-large vLLM MiMo-BF16 runbook; retire the gufo runbook
New runbook carries the roof math, the measured three-arm spec table, the greedy-text numerics trap, the reasoning-field contract change and the rollback tag. LLM index and AGENTS.md fleet table, bench policy and runbook index follow; the frank-small index entries catch up to the SGLang cutover (entry was missed on 2026-10-01). Work log records the cutover.
This commit is contained in:
Frank
@@ -10,12 +10,12 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints —
|
||||
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
|
||||
unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter
|
||||
on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small,
|
||||
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
|
||||
port 7771).
|
||||
runs two endpoints on the same arch —
|
||||
vLLM (digest-pinned kyuz0 TheRock gfx1151 image; frank-large,
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 with cross-model MTP
|
||||
spec decode from the Qwen3.5-9B head on the gfx1151 iGPU, 4 x 262144 ctx,
|
||||
port 7770) and SGLang (frank-small, the same model as an INT8 W8A16
|
||||
AutoRound conversion on the RTX 3090, 2 x 262144 ctx, port 7771).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -39,14 +39,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[stock llama.cpp (unsloth b11030, gfx1151)
|
||||
E[vLLM TheRock gfx1151, digest-pinned
|
||||
frank-large, port 7770]
|
||||
F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
|
||||
262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
|
||||
G[mainline llama.cpp 37b53fd
|
||||
F[MiMo-V2.6-Distill-Qwen-9B BF16 + Qwen3.5-9B MTP head
|
||||
4 x 262144 ctx, spec 2 tok, qwen3/qwen3_coder parsers]
|
||||
G[SGLang venv cu130
|
||||
frank-small, port 7771]
|
||||
H[Gemma4-12B QAT Q4_K_M
|
||||
4 x 262144 ctx, MTP + vision + audio]
|
||||
H[MiMo-V2.6 INT8 W8A16 AutoRound
|
||||
2 x 262144 ctx, pinned pool 606208, in-checkpoint vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -162,14 +162,23 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
|
||||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
|
||||
Strix Halo inference engine, built from source on Dankclaw, serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
|
||||
Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
|
||||
262,144-token conversation, greedy, text-only, the two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode trade
|
||||
against llama.cpp, troubleshooting
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — ACTIVE frank-large runbook since
|
||||
2026-10-01 (root systemd unit, port 7770): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B
|
||||
(the fleet's first; 71%/44% per-position acceptance measured) —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s decode), the greedy-text numerics
|
||||
trap, the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
|
||||
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback target via git
|
||||
tag frank-large-gufo-q4kxl-20261001, engine + payload + drafter still on
|
||||
the host): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine,
|
||||
built from source on Dankclaw, serving Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL
|
||||
with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — the two
|
||||
from-source build workarounds, the client-contract delta, the measured
|
||||
prefill/decode trade against llama.cpp
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -203,12 +212,18 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||||
- [[frank-small-sglang-mimo-v26]] — ACTIVE frank-small runbook since
|
||||
2026-10-01 (root systemd unit, port 7771): SGLang in a self-contained
|
||||
python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-
|
||||
AutoRound on the RTX 3090 — the same architecture frank-large now serves
|
||||
in BF16 on the iGPU; 2 x 262144 ctx on a pinned 606,208-token fp8 pool,
|
||||
NO spec decode (zero mtp.* tensors shipped), in-checkpoint vision.
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]]; rollback target, payload and binary
|
||||
intact on disk): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
|
||||
vLLM detour (2026-09-19) was reverted the same day; see below.
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
|
||||
Reference in New Issue
Block a user