docs: record the 2026-10-02 swaps — frank-large on ROCmFPX, frank-small on Q5_K_M
frank-large: new runbook frank-large-rocmfpx-q8agent-mtp.md — the ftype-115 tensor-type lock-in that forced the fork, the pinned build (set_rows history), the measured MTP A/B (17.0 vs 7.6 t/s decode, acceptance 0.462), the live client contract (reasoning_content, effort fold, vision restored) and rollback. frank-small: new runbook frank-small-llama-mimo-q5km.md; q6k marked retired (rollback target). gufo runbook marked retired (rollback target). Both indexes, the fleet topology table, the vision/models.yml notes, the benchmarking law and three new Key Lessons updated.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-01):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
@@ -10,10 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B
|
||||
UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU,
|
||||
rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
|
||||
file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
|
||||
session at 262,144, f16 KV, vision via the bundled projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
@@ -38,13 +39,13 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[gufo v0.1.1 built from source
|
||||
E[ROCmFPX llama.cpp gfx1151, pinned fork build
|
||||
frank-large, port 7770]
|
||||
F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter
|
||||
1 x 262144 ctx, f16 KV, greedy, adaptive draft 7]
|
||||
F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
|
||||
1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q6_K GGUF
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
@@ -64,6 +65,19 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
|
||||
(root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
|
||||
the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
|
||||
(26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
|
||||
tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
|
||||
tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
|
||||
runtime is not interchangeable, the pinned-build set_rows history, the
|
||||
measured envelope, rollback to gufo
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
|
||||
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
2 x 262,144 with K f16 + V q8_0, 100.4-101.8 tok/s decode, 20,164 MiB
|
||||
measured. The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
|
||||
Reference in New Issue
Block a user