diff --git a/AGENTS.md b/AGENTS.md index f1f59b6..14489d6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head (ROCmFP4-FAST), 262K context / 1 slot, reasoning on | 7770 | +| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 262K context / 1 slot, reasoning on | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -285,23 +285,29 @@ charlie12345/ROCmFPX `llama-server` built at merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn + 36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048), hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter -n-gram table is split per head INSIDE the single GGUF — fully VRAM-resident, -no sidecar, no `--ngram-on-disk`. The model's own MTP draft head +n-gram table is a single joined tensor in the `v2/` GGUF we serve (the root +`-ple16` per-head split requires the LaurentZuijdwijk Vulkan fork and aborts +this loader's `GGML_ASSERT(qwen4exp is missing the PLE n-gram table)`) — fully +VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST -quant matched to target, ~66% draft acceptance) drives speculative decoding -at depth 3; vision tower (f16) ships as mmproj/. It runs 262144 context / 1 +quant matched to target) is on disk but does **not** load on this binary +(loader requires a full per-layer `blk..hc_attn_norm.weight` set; the head +carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision +tower (f16) ships as mmproj/. It runs 262144 context / 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit -segfaults on qwen4exp hyper-connection config), and reasoning on (Sharp +segfaults on qwen4exp hyper-connection config), `--flash-attn off` (required: +fa on aborts graph build on the QSA indexer path — F32 `ggml_fill` on the F16 +kq mask, ggml.c:5371), and reasoning on (Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` + `preserve_reasoning`). Env `scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`. **Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix -model with its own matched MTP draft head, replacing the jcbtc CIRU v1.1 -deployment (ciru-ai runtime). The 2026-08-28 failure of this head on the -kingjones STRIX_LEAN model (`missing tensor blk.0.hc_attn_norm.weight`) was a -cross-vendor pairing problem — the head carries the hyper-connection norm -tensor (`blk.48.hc_attn_norm.weight`) and works against its own main file. +model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1 +deployments. The own-matched MTP draft head does NOT load on build-qwen38 — +verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`), +third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The +head stays on disk for a future compatible binary. Runbook: [serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md). diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index d33115a..d9d315f 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -10,25 +10,20 @@ # # Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, # 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE -# at layer 1) quantized Q4_0_ROCMFP4_FAST imatrix-calibrated, 87.06 GiB, -# 4.23 bpw. The n-gram PLE table is split per head inside the single GGUF -# (fully VRAM-resident — no ple sidecar, no --ngram-on-disk). Vision tower -# (f16, lossless) ships as mmproj/. The model's own MTP head (2.28 GiB, -# matched ROCmFP4-FAST quant, ~66% draft acceptance) drives spec decoding. +# KNOWN CONSTRAINTS on the charlie12345/ROCmFPX build-qwen38 runtime. +# 1. FA must stay OFF. --flash-attn on aborts graph build: the QSA +# indexer path calls ggml_fill on the F16-cast kq mask (ggml_fill +# asserts GGML_TYPE_F32 at ggml.c:5371). The v2 main loads and serves +# fine with --flash-attn off (verified live: model loaded, health 200, +# real completion). +# 2. The MTP draft head does NOT load on this binary. The qwen4exp loader +# requires blk.0.hc_attn_norm.weight per layer; the agentionai FAST +# head carries only blk.48 (verified 3x: EasiiX Q8_0 + FAST against +# kingjones STRIX_LEAN, FAST against this v2 main). Speculative +# decoding therefore runs ngram-mod only. The 2.28 GiB head stays on +# disk in case a compatible binary/head pair appears. # -# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0, -# presence 1.5 — non-thinking instruct) mixed with the validated -# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096, -# t 16 / tb 16 (threads-batch 16 measured best for prefill), f16 KV, fa on, -# cont-batching, prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx -# checkpoints, --fit off (auto-fit segfaults on qwen4exp -# hyper-connection/indexer config). MTP depth 3 (n-max 3 measured best on -# this head; adaptive p-split measured worse, left off). The Qwen-Sharp -# chat template is kept on purpose (chat-template.jinja): the service keeps -# emitting reasoning in DeepSeek format for its clients. -# -# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The -# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP +# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP # incident). llama.cpp reads --host/--port/--temp as CLI flags, never as # env vars. @@ -37,11 +32,14 @@ LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server # Model package from # https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF -# Layout: single main GGUF (per-head PLE table, fully VRAM-resident) + -# mmproj/ vision tower. The MTP draft head lives in -# /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST (already on the host). -MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST -MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf +# Layout: v2/ joined-layout GGUF (PLE n-gram table as a single +# per_layer_token_embd.weight tensor — the layout the charlie12345/ROCmFPX +# qwen4exp loader requires; the root -ple16 file splits the table per head +# and only loads in the LaurentZuijdwijk Vulkan fork). mmproj/ vision tower. +# The MTP draft head lives in /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST +# (already on the host). +MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2 +MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf # Qwen-Sharp chat template (tracked in the repo, staged on the host). The @@ -72,9 +70,8 @@ CACHE_TYPE_V=f16 CTX_CHECKPOINTS=16 CACHE_RAM=8192 SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state -SPLIT_MODE=none -FIT=off -NGL=999 + +FLASH_ATTN=off # Sampling (card instruct table, non-thinking). TEMPERATURE=0.7 @@ -95,14 +92,16 @@ HSA_OVERRIDE_GFX_VERSION=11.5.1 VERBOSITY=4 -# Speculative decoding: the model's own MTP head (agentionai -# Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, ROCmFP4-FAST quant -# matched to the target) at depth 3 stacked on ngram-mod. n-max 3 measured -# best (32.4 t/s, 0.612 acceptance); adaptive p-split measured worse, so it -# stays off (p-min stays at the 0.75 default). Set SPEC_TYPE=none to disable. +# Speculative decoding: ngram-mod only. The MTP draft head is retained +# (agentionai Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB) but does +# NOT load on build-qwen38 (loader requires blk.0.hc_attn_norm.weight the +# head lacks — see KNOWN CONSTRAINTS). If a compatible binary appears, set +# SPEC_TYPE=draft-mtp,ngram-mod (n-max 3 measured best at 32.4 t/s / 0.612 +# acceptance; adaptive p-split measured worse, stays off). SPEC_TYPE=none +# disables speculative decoding entirely. DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf -SPEC_TYPE=draft-mtp,ngram-mod +SPEC_TYPE=ngram-mod SPEC_DRAFT_N_MAX=3 SPEC_DRAFT_N_MIN=0 SPEC_DRAFT_P_MIN=0.75 diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 50baeb3..68df18f 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -12,18 +12,18 @@ # Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, # 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE # at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB. -# The n-gram PLE table is split per head inside the single GGUF — fully -# VRAM-resident, no sidecar, no --ngram-on-disk. The model's own MTP -# head (2.28 GiB, matched ROCmFP4-FAST quant) drives speculative decoding -# at depth 3 (~66% draft acceptance). +# The n-gram PLE table is a single per_layer_token_embd.weight tensor — +# fully VRAM-resident, no sidecar, no --ngram-on-disk. # # Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0, # presence 1.5 — non-thinking instruct) mixed with the validated # build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096, -# t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB -# cache-ram + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit -# segfaults on qwen4exp hyper-connection config). MTP depth 3 (n-max 3 -# measured best; adaptive p-split measured worse, left off). +# t 16 / tb 16, f16 KV, fa OFF (see KNOWN CONSTRAINTS in the env file: +# flash-attn on aborts graph build on the QSA indexer path), cont-batching, +# prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx checkpoints, +# --fit off (auto-fit segfaults on qwen4exp hyper-connection config). +# Speculative decoding is ngram-mod only; the MTP head does not load on +# build-qwen38 (loader requires blk.0.hc_attn_norm.weight it lacks). set -euo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" @@ -37,9 +37,9 @@ fi # --- defaults (used when the env file does not set them) ------------------- : "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST}" -: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf}" -: "${MMPROJ_PATH:=$MODEL_DIR/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}" +: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2}" +: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf}" +: "${MMPROJ_PATH:=$MODEL_DIR/../mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}" : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" : "${CHAT_TEMPLATE_KWARGS:={}}" : "${ALIAS:=frank-default-large}" @@ -57,17 +57,15 @@ fi : "${CACHE_RAM:=8192}" : "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}" : "${SPLIT_MODE:=none}" -: "${FIT:=off}" -: "${NGL:=999}" +: "${FLASH_ATTN:=off}" : "${TEMPERATURE:=0.7}" : "${TOP_K:=20}" : "${TOP_P:=0.8}" : "${MIN_P:=0.0}" : "${PRESENCE_PENALTY:=1.5}" : "${REASONING:=on}" -: "${REASONING_FORMAT:=deepseek}" : "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}" -: "${SPEC_TYPE:=draft-mtp,ngram-mod}" +: "${SPEC_TYPE:=ngram-mod}" : "${DRAFT_MODEL_DIR:=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST}" : "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf}" : "${SPEC_DRAFT_N_MAX:=3}" @@ -154,7 +152,7 @@ exec /usr/bin/env \ --batch-size "$BATCH_SIZE" \ --ubatch-size "$UBATCH_SIZE" \ --fit "$FIT" \ - --flash-attn on \ + --flash-attn "$FLASH_ATTN" \ --cache-type-k "$CACHE_TYPE_K" \ --cache-type-v "$CACHE_TYPE_V" \ --threads "$NUM_THREADS" \ diff --git a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md index b60b57d..63e7909 100644 --- a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md +++ b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md @@ -1,7 +1,7 @@ --- type: reference title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw -description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large. +description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large. tags: - llama.cpp - rocm @@ -17,29 +17,33 @@ timestamp: 2026-08-29 # agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw -> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime) -> deployment on the same slot. +> **Active since 2026-08-29.** Replaced the kingjones STRIX_LEAN and jcbtc +> CIRU v1.1 deployments on the same slot. +> **MTP draft disabled** — the author's own head does not load on +> build-qwen38 (see Known issues #3). Speculative decoding is ngram-mod only. `frank-default-large` (port 7770) serves `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main, -`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's -own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the +`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout) +with the author's own MTP draft head +(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the charlie12345/ROCmFPX `build-qwen38` runtime. ## Why this model - 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory - box, including the 51.2 B-parameter n-gram (PLE) table, which is split per - head inside the single GGUF (`v2/` is the joined-table layout; root is the - VRAM-resident one and is what we run). + box, including the 51.2 B-parameter n-gram (PLE) table. REPO `v2/` layout + is used (joined single `per_layer_token_embd.weight` tensor) because the + charlie12345/ROCmFPX loader requires it; the root `-ple16` file's per-head + split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2). - imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2 (better than AesSedai IQ4_XS / IQ3_S at smaller size). - 18-25% faster prompt processing than a mainline quant of similar size (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth). -- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head - (~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3). +- Ships the model's own f16 vision tower (`mmproj/`). - The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0 - draft measured worse on this head family (acceptance + throughput). + draft measured worse on this head family — but it does not load on + build-qwen38 (Known issues #3). ## Runtime @@ -61,7 +65,8 @@ author's own matched MTP head on this binary. | Item | Path on Dankclaw | Size | |---|---|---| -| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) | +| Main (v2 joined) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf` | 93,484,242,816 B (87.06 GiB) | +| Main (root ple16, unused) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) | | Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B | | Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B | @@ -79,11 +84,11 @@ build-qwen38 profile: ``` llama-server \ - --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \ + --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \ --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \ --host 0.0.0.0 --port 7770 --jinja \ --device ROCm0 --gpu-layers 999 \ - --split-mode none --fit off --flash-attn on \ + --split-mode none --fit off --flash-attn off \ --ctx-size 262144 --parallel 1 \ --batch-size 4096 --ubatch-size 4096 \ --cache-type-k f16 --cache-type-v f16 \ @@ -94,11 +99,7 @@ llama-server \ --reasoning on --reasoning-format deepseek --reasoning-budget -1 \ --chat-template-file chat-template.jinja \ --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \ - --spec-type draft-mtp,ngram-mod \ - --spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \ - --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \ - --spec-draft-ngl 999 --spec-draft-device ROCm0 \ - --spec-draft-type-k f16 --spec-draft-type-v f16 + --spec-type ngram-mod ``` Key choices and deviations from the CIRU deployment: @@ -107,27 +108,45 @@ Key choices and deviations from the CIRU deployment: (same as every Flash-Next deployment). - **`--checkpoint-min-step` removed** — build-qwen38 does not implement it (verified via `--help`); only `--ctx-checkpoints` is passed. -- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split), - not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the +- **No PLE sidecar args** — the n-gram table is in-GGUF (joined + `per_layer_token_embd.weight` from the `v2/` layout), not a `ple/` + CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the `GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU- runtime-only and are gone. - **`--mmproj` active** — vision tower included (f16, lossless). -- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s / - 0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468). - `p-min` stays at the 0.75 default. +- **`--flash-attn off` is mandatory.** With fa on, the QSA indexer path + (layers with attention compression) calls `ggml_fill` on the F16-cast kq + mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`, + ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this + v2 main — loads and serves fine with fa off. - Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's quick-start is 32768 but the full window is fine on SVM). +- MTP is disabled (ngram-mod only) — see the MTP section below. -## MTP draft head +## MTP draft head — INCOMPATIBLE on build-qwen38 -The draft is agentionai's own head for this exact model: -`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture -`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as -the CIRU draft that works). The earlier 2026-08-28 failure of this head on -the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`) -was a cross-vendor pairing problem, not a defect of the head itself. The -draft carries a 512-expert MoE, so adaptive/long drafting costs real forward -passes — keep n-max 3. +The author's own matched head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4- +FAST-GGUF`, 2.28 GiB, `qwen4exp` arch) does **not** load on this binary: + +``` +error loading model: missing tensor 'blk.0.hc_attn_norm.weight' +``` + +The build-qwen38 qwen4exp loader requires a full per-layer +`blk..hc_attn_norm.weight` set in the draft (the graph reads all 48 +layers); the head only carries `blk.48.hc_attn_norm.weight` (count 1 — all +three Flash-Next-era heads examined on this fork have that same partial +layout). Verified three times, not a cross-vendor artifact: + +1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed. +2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed. +3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment) + — failed identically. + +The 2.28 GiB head stays on disk; the moment a binary or head appears that +carries the full per-layer norm set, re-enable with +`SPEC_TYPE=draft-mtp,ngram-mod`, `--spec-draft-n-max 3` (author-measured +best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse). ## Config files @@ -152,12 +171,21 @@ loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it. 1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU runtime reject these files. -2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a - good link. -3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are - incompatible with this binary's draft expectations for Flash-Next - (hyper-connection norm tensor). -4. **`SVM mapping failed` kernel spam** is a documented symptom, not root +2. **Use the `v2/` joined layout, NOT the root `-ple16` split.** This binary's + qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single + `per_layer_token_embd.weight` tensor and GGML_ASSERTs "qwen4exp is missing + the PLE n-gram table" against the per-head split. The root file only loads + in the LaurentZuijdwijk Vulkan fork (`vulkan-qwen4exp-rocmfpx`). The v2 + file is byte-identical in weights, ~5 KB larger header, loads fully into + SVM unified memory (no `--ngram-on-disk` needed — kingjones STRIX ran a + 50 GiB joined PLE table the same way). +3. **No MTP draft head loads on build-qwen38.** The qwen4exp loader + requires a full per-layer `blk..hc_attn_norm.weight` set; every + Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries + only `blk.48`. Spec decoding is ngram-mod only. See the MTP section. +4. **`--flash-attn off` is mandatory.** fa on aborts graph build on the QSA + indexer path (F32 `ggml_fill` on the F16 mask) — see Serving profile. +5. **`SVM mapping failed` kernel spam** is a documented symptom, not root cause — investigate before blaming memory pressure. ## Citations