frank-default-large: swap to agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix + own MTP draft
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF — fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST (2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3). Vision tower (f16) loaded via mmproj/. Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) — the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1 deployment (ciru-ai runtime). Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0, presence 1.5) mixed with the validated build-qwen38 serving profile: 262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx checkpoints (no --checkpoint-min-step — unsupported on this binary), --fit off (auto-fit segfaults on qwen4exp hyper-connection config), --spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured worse per author card), p-min 0.75. Qwen-Sharp chat template kept; reasoning stays in DeepSeek format. The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN model was a cross-vendor pairing problem: the head carries blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
This commit is contained in:
1 parent
e22631d892
commit
3c2a5d2fe7
6 files changed
+313
-140
No files matched your search
@@ -1,10 +1,10 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST
|
||||
imatrix + MTP draft). Both services run the Qwen-Sharp chat template.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -26,10 +26,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
E[charlie12345/ROCmFPX build-qwen38
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
F[agentionai Qwen3.8-Flash-Next
|
||||
ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
@@ -53,7 +53,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[kingjones-strix-rocmfp4]] — active frank-default-large runbook:
|
||||
- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook:
|
||||
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
|
||||
Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head
|
||||
(ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29)
|
||||
- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
|
||||
Reference in new issue
Block a user