dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: swap to agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix + own MTP draft

Browse Source 
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.

Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).

Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.

The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
This commit is contained in:
Frank
2026-08-29 20:41:12 -07:00
parent e22631d892
commit 3c2a5d2fe7
6 changed files with 313 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+25 -16
View File
@@ -27,7 +27,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head (ROCmFP4-FAST), 262K context / 1 slot, reasoning on | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -274,27 +274,36 @@ When you rename a service or move a model to a new host, update
## Runtime & Troubleshooting
### Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN llama.cpp (Dankclaw, ACTIVE)
### Qwen3.8-Flash-Next ROCmFP4-FAST imatrix llama.cpp (Dankclaw, ACTIVE)
The large service serves
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF` (public, 98.5 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN/`
with a charlie12345/ROCmFPX `llama-server` built at
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (public,
87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated) from
`/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/` with a
charlie12345/ROCmFPX `llama-server` built at
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture
merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
hyper-connections (count=4, low_rank=320), PLE (layer 1). Q4_0_ROCMFP4 quant.
It runs f16 KV cache, 131072 context / 1 slot, `--fit off` (auto-fit segfaults
on qwen4exp hyper-connection config), and reasoning on (Sharp template, budget
-1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` +
`preserve_reasoning`). Env `scripts/frank-default-large.env`, serve
`scripts/serve-frank-large.sh`.
hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter
n-gram table is split per head INSIDE the single GGUF — fully VRAM-resident,
no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST
quant matched to target, ~66% draft acceptance) drives speculative decoding
at depth 3; vision tower (f16) ships as mmproj/. It runs 262144 context / 1
slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit
segfaults on qwen4exp hyper-connection config), and reasoning on (Sharp
template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
`preserve_thinking` + `preserve_reasoning`). Env
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`.
**Serving status (2026-09-01):** deployed the Flash-Next 56B MoE model. The
previous kingjones Q6_0 AGENT model (27B, standalone MTP, 262144 context) is
retired. The Q8_0 AGENT experiment is documented in
`serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
`ggml_compute_forward_set_rows` and did not provide a working service.
**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix
model with its own matched MTP draft head, replacing the jcbtc CIRU v1.1
deployment (ciru-ai runtime). The 2026-08-28 failure of this head on the
kingjones STRIX_LEAN model (`missing tensor blk.0.hc_attn_norm.weight`) was a
cross-vendor pairing problem — the head carries the hyper-connection norm
tensor (`blk.48.hc_attn_norm.weight`) and works against its own main file.
Runbook:
[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).
### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE)