docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M

frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115
tensor-type lock-in that forced the fork, the pinned build (set_rows history),
the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live
client contract (reasoning_content, effort fold, vision restored) and rollback.
frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired
(rollback target). gufo runbook marked retired (rollback target). Both indexes,
the fleet topology table, the vision/models.yml notes, the benchmarking law and
three new Key Lessons updated.
This commit is contained in:
Frank
2026-10-02 01:51:11 -07:00
parent 125d04967f
commit 72bd4c967e
7 changed files with 522 additions and 147 deletions
+23 -9
View File
@@ -1,6 +1,6 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
Documentation for the LLM inference fleet on the two hosts (2026-10-02):
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
@@ -10,10 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B
UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU,
rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
session at 262,144, f16 KV, vision via the bundled projector)** and
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
> 2026-08-22: the former `vllmScripts` repository is merged into the
@@ -38,13 +39,13 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 114 GiB GTT]
E[gufo v0.1.1 built from source
E[ROCmFPX llama.cpp gfx1151, pinned fork build
frank-large, port 7770]
F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter
1 x 262144 ctx, f16 KV, greedy, adaptive draft 7]
F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
G[llama.cpp master CUDA sm_86
frank-small, port 7771]
H[MiMo-V2.6 Q6_K GGUF
H[MiMo-V2.6 Q5_K_M GGUF
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
E2 --> E --> F
E2 --> G --> H
@@ -64,6 +65,19 @@ frank-small, port 7771]
## Active Deployment
- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
(root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
(26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
runtime is not interchangeable, the pinned-build set_rows history, the
measured envelope, rollback to gufo
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork