docs: record the frank-small llama.cpp Q6_K cutover and the frank-large gufo rollback
This commit is contained in:
Frank
@@ -10,12 +10,11 @@ headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, uti
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
runs two endpoints on the same arch —
|
||||
vLLM (digest-pinned kyuz0 TheRock gfx1151 image; frank-large,
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 with cross-model MTP
|
||||
spec decode from the Qwen3.5-9B head on the gfx1151 iGPU, 4 x 262144 ctx,
|
||||
port 7770) and SGLang (frank-small, the same model as an INT8 W8A16
|
||||
AutoRound conversion on the RTX 3090, 2 x 262144 ctx, port 7771).
|
||||
runs two endpoints — **gufo v0.1.1 (frank-large, port 7770: Qwen3.8-27B
|
||||
UD-Q4_K_XL + DFlash2, ONE session at 262,144, f16 KV on the gfx1151 iGPU,
|
||||
rolled back from the one-day vLLM MiMo experiment 2026-10-01 night)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q6_K
|
||||
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -39,14 +38,14 @@ fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[vLLM TheRock gfx1151, digest-pinned
|
||||
E[gufo v0.1.1 built from source
|
||||
frank-large, port 7770]
|
||||
F[MiMo-V2.6-Distill-Qwen-9B BF16 + Qwen3.5-9B MTP head
|
||||
4 x 262144 ctx, spec 2 tok, qwen3/qwen3_coder parsers]
|
||||
G[SGLang venv cu130
|
||||
F[Qwen3.8-27B UD-Q4_K_XL + DFlash2 drafter
|
||||
1 x 262144 ctx, f16 KV, greedy, adaptive draft 7]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 INT8 W8A16 AutoRound
|
||||
2 x 262144 ctx, pinned pool 606208, in-checkpoint vision]
|
||||
H[MiMo-V2.6 Q6_K GGUF
|
||||
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -65,7 +64,9 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-primary-glm53-flash-w4a16]] — the ACTIVE frank-primary runbook
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||||
@@ -162,23 +163,24 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — ACTIVE frank-large runbook since
|
||||
2026-10-01 (root systemd unit, port 7770): the digest-pinned
|
||||
- [[frank-large-gufo-qwen38-27b]] — **the ACTIVE frank-large runbook
|
||||
again since 2026-10-01 night** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||||
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode
|
||||
trade against llama.cpp
|
||||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
|
||||
4 x 262144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B
|
||||
(the fleet's first; 71%/44% per-position acceptance measured) —
|
||||
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
|
||||
MTP-vs-DFlash table (19.8 vs 17.4 t/s decode), the greedy-text numerics
|
||||
trap, the `reasoning` field change, troubleshooting
|
||||
- [[frank-large-gufo-qwen38-27b]] — RETIRED 2026-09-28 → 2026-10-01
|
||||
(replaced by [[frank-large-vllm-mimo-v26-bf16]]; rollback target via git
|
||||
tag frank-large-gufo-q4kxl-20261001, engine + payload + drafter still on
|
||||
the host): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine,
|
||||
built from source on Dankclaw, serving Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL
|
||||
with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU — the two
|
||||
from-source build workarounds, the client-contract delta, the measured
|
||||
prefill/decode trade against llama.cpp
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||||
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
|
||||
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
|
||||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
@@ -212,14 +214,22 @@ frank-small, port 7771]
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-small-sglang-mimo-v26]] — ACTIVE frank-small runbook since
|
||||
2026-10-01 (root systemd unit, port 7771): SGLang in a self-contained
|
||||
python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-
|
||||
AutoRound on the RTX 3090 — the same architecture frank-large now serves
|
||||
in BF16 on the iGPU; 2 x 262144 ctx on a pinned 606,208-token fp8 pool,
|
||||
NO spec decode (zero mtp.* tensors shipped), in-checkpoint vision.
|
||||
- [[frank-small-llama-mimo-q6k]] — **the ACTIVE frank-small runbook
|
||||
since 2026-10-01 night** (root systemd unit, port 7771): llama.cpp
|
||||
(master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill-
|
||||
Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO native 262,144-token
|
||||
sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces
|
||||
flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no
|
||||
vision), reasoning_content, no spec decode by default
|
||||
- [[frank-small-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-small-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv + INT8 payload on disk as rollback): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-small-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-small-sglang-mimo-v26]]; rollback target, payload and binary
|
||||
[[frank-small-sglang-mimo-v26]] → [[frank-small-llama-mimo-q6k]];
|
||||
rollback target, payload and binary intact on disk): mainline llama.cpp serving
|
||||
intact on disk): mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
|
||||
Reference in New Issue
Block a user