frank-default-large: swap to agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix + own MTP draft

Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.

Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).

Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.

The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
This commit is contained in:
Frank committed 2026-08-29 20:41:12 -07:00
1 parent e22631d892
commit 3c2a5d2fe7
6 files changed
+313 -140

No files matched your search

+11 -7
View File
@@ -1,10 +1,10 @@
# LLM Fleet — Knowledge Base
Documentation for the LLM inference fleet on two local machines
(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small,
Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
services run the Qwen-Sharp chat template.
(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST
imatrix + MTP draft). Both services run the Qwen-Sharp chat template.
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -26,10 +26,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
E2[gfx1151 iGPU 64 GiB unified memory]
E[charlie12345/ROCmFPX build-kairic (v81)
E[charlie12345/ROCmFPX build-qwen38
frank-default-large, port 7770]
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
F[agentionai Qwen3.8-Flash-Next
ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot]
E2 --> E --> F
end
@@ -53,7 +53,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[kingjones-strix-rocmfp4]] — active frank-default-large runbook:
- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head
(ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29)
- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook:
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results