- runtime: charlie build-qwen38 -> kingjones30/ROCmFPX (card recipe
build, LLAMA_BUILD_WEBUI=OFF since npm/HF-bucket provisioning fails;
all 35 of our serving flags verified present in the tree)
- model: agentionai FAST imatrix -> kingjones777 STRIX single GGUF
(on-host copy sha256 == HF LFS oid c6770d74...49fa verified)
- args unchanged from our validated profile (401408x2, b4096/ub512,
t16/tb16, f16 KV, fa on, fit off); mmap stays default per card
(PLE streams via page cache; --no-mmap = silent cgroup OOM)
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
Runtime: ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp at tag v1.1, built
with the vendor build-linux-amd.sh flag set plus GGML_CUDA_FA +
GGML_CUDA_FA_ALL_QUANTS (the Q4_1_MOE_FORCE_J path is live in the hipified
mmq kernels). Model: jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 (Q4_1 IU4 experts,
MTP Q8_0 draft, CIRUPLE1 PLE sidecar) with checksums verified at download.
Vendor production profile verbatim on first deployment: 262144 ctx x 1 slot,
b 2048 / ub 512, t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache +
8 GiB cache-ram + idle slots, 32 ctx checkpoints (min step 8192), MTP depth
3 with p-split 0.10, PLE pager env knobs (Q41_MOE_FORCE_J=32,
PLE_WORKERS=16, PLE_STRICT_SHA=0, HIPBLASLT=1). Qwen-Sharp chat template
kept deliberately; reasoning stays in DeepSeek format.
Both the EasiiX Q8_0 and agentionai ROCmFP4-FAST MTP draft heads fail
with 'missing tensor blk.0.hc_attn_norm.weight' against the STRIX_LEAN
Uncensored model. The qwen4exp binary expects this hyper-connection
attn norm tensor which neither draft GGUF carries. Using ngram-mod
only until a compatible MTP head is found.
The EasiiX Q8_0 draft was incompatible with STRIX_LEAN (missing
blk.0.hc_attn_norm.weight). Switched to agentionai/Qwen3.8-Flash-Next-
MTP-ROCmFP4-FAST-GGUF (2.28 GiB, ROCmFP4-FAST quant matched to target).
n_max 3 per the HF card (n_max 4 measured worse). SPEC_TYPE=draft-mtp,ngram-mod.
The env file had the old STRIX paths (directory doesn't exist on disk).
Updated to the actual running STRIX_LEAN (Uncensored) variant:
- MODEL_DIR → kingjones-Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN
- MODEL_FILE → 3-shard Q4_0 (00001-of-00003)
- MMPROJ_PATH → mmproj-Qwen3.8-Flash-Next-Uncensored-BF16.gguf
- serve-frank-large.sh: add --mmproj $MMPROJ_PATH
Reconciles the uncommitted local changes that were on Dankclaw.
frank-default-large now serves the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
model with a standalone MTP draft head (Q4_0), f16 K + TurboQuant V KV cache,
and the Sharp chat template. Context is maxed to 262144 (one slot); reasoning
is on with preserve_thinking + preserve_reasoning in the chat kwargs.
- frank-default-large.env: Q8_0 AGENT model paths, CTX_SIZE=262144, REASONING=on,
REASONING_FORMAT=none, CHAT_TEMPLATE_KWARGS {enable_thinking, preserve_thinking,
preserve_reasoning}
- serve-frank-large.sh: standalone draft-mtp profile (already switched off the
Kairic/PromptForge path)
- download-q8agent.sh: reproducible HF fetch of the three payload files
- docs/LLM/kingjones-q8agent-rocmfpx.md: Q8_0 AGENT runbook
- AGENTS.md: topology + ACTIVE runtime section + TTM/GTT load note
- qwen3.8-strix-optimizations, Dankclaw Improvements: TTM pages_limit now 114 GiB
(applied 2026-08-25), full 262k KV + Q8 payload fit in one slot
- frank_large_config.bats: assert 262144 context, reasoning + preserve_* enabled
The TTM/GTT was raised to 114 GiB (29,883,536 pages) to hold the full 262k KV.
Replace the retired kingjones ROCmFP4 STRIX MTP payload on Dankclaw with
cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF (eemin namespace, public), served by the
charlie12345/ROCmFPX llama.cpp build (ROCmI4 W4A4, c49ebdbd) on the Strix Halo
iGPU. Q4_0_ROCMI4 with an embedded NextN/MTP head (no separate draft file);
strict-Qwen MTP n16/p0.60/backend sampling on ROCm0. Binary
/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server; model dir
/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP. Updated serve script + env,
ubatch bench, large-config bats test, docs, AGENTS.md.
- frank-default-small (Dankserver): Ornith-1.5-35B-A3B-CRACK Q8_0, bundled
native draft-mtp (n_max 4), MoE expert FFN tensors pinned to CPU via an
explicit --override-tensor \.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU
(replaces --n-cpu-moe), Qwen-Sharp chat template.
- frank-default-large (Dankclaw): kingjones Qwen3.8-27B-Uncensored-Q6_0
ROCmFP4 AGENT dense GGUF, standalone mtp-Qwen3.8-27B-Q4_0 draft head
(--model-draft, draft-mtp n_max 4), Qwen-Sharp chat template. Drop the
retired PromptForge .pfs sidecar lanes.
- Track scripts/chat-template.jinja (v22.4.0 Sharp) and apply via
--chat-template-file on both services.
- Docs: new kingjones-strix-mtp and dealign-ornith-crack runbook pages;
update index, deployment, and both system specs; models.json names.
Smoke-tested both models on the real hardware on scratch ports before
commit: model loads, MTP draft attaches, Sharp template applies, chat
completion returns.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
The LLM fleet serve scripts, systemd units, env tunables, and bench
harnesses join the deploy scripts. The vllmScripts deploy.sh is
removed here; its functionality is folded into the unified
scripts/deploy.sh in a follow-up commit.
bench-spec-stack.sh gains the executable bit (required by the
scripts_syntax test seam).
Frank