docs: record fleet double swap (large -> kingjones STRIX on pinned fork, small -> Huihui FP8 on FreeToken)

- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook: runtime build recipe (WEBUI=OFF gotcha), deployed serving profile, fa-on-works-unpatched evidence, MTP heads documented unloadable, swap procedure + 2026-08-30 validation record - agentionai-fast-imatrix marked RETIRED (rollback record kept) - freetoken-frank-default-small: model/lineage/template sections now Huihui abliterated FP8; new operational quirk: chat API 422 without explicit model field - LLM index + AGENTS fleet table/runtime sections current; both services' history chains fixed (small was still Tiel in AGENTS.md) - worklog entry added; models.json contextWindow for large corrected to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
2026-08-30 02:35:38 -07:00
parent aa9c1d065a
commit 0937cfb047
6 changed files with 268 additions and 152 deletions
@@ -22,12 +22,12 @@ prefix, reorganized in follow-up commits). The retired target checkouts
 archived. All services now run from `/home/dank/serverAppConfigs` on each
 host.

-### LLM fleet topology (2026-08-22)
+### LLM fleet topology (2026-08-30)

 | Target | Runtime | Service | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
-| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 32K context / 1 slot, reasoning on | 7770 |
+| Dankserver | FreeToken v0.1.2 | frank-default-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
+| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-default-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |

 History: the 2026-08-17 redo retired the pipeline-parallel split and the
 FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -40,7 +40,12 @@ Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
 Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp
 Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0
 AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into
-charlie's ROCmFPX fork, --fit off, 128k context). Runbooks:
+charlie's ROCmFPX fork, --fit off, 128k context). On 2026-09-02
+Dankserver moved to the FreeToken runtime serving the official
+Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again:
+Dankclaw to the kingjones STRIX model on the card's pinned
+kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated
+FP8 checkpoint on FreeToken. Runbooks:
 [serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
 Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.

@@ -274,63 +279,85 @@ When you rename a service or move a model to a new host, update

 ## Runtime & Troubleshooting

-### Qwen3.8-Flash-Next ROCmFP4-FAST imatrix llama.cpp (Dankclaw, ACTIVE)
+### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE)

 The large service serves
-`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (public,
-87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated) from
-`/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/` with a
-charlie12345/ROCmFPX `llama-server` built at
-`/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture
-merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
-36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
-hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter
-n-gram table is a single joined tensor in the `v2/` GGUF we serve (the root
-`-ple16` per-head split requires the LaurentZuijdwijk Vulkan fork and aborts
-this loader's `GGML_ASSERT(qwen4exp is missing the PLE n-gram table)`) — fully
-VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
-(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST
-quant matched to target) is on disk but does **not** load on this binary
-(loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; the head
-carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision
-tower (f16) ships as mmproj/. It runs 32768 context / 1
-slot (author quick-start; the fully GPU-resident 87 GiB model + f16 KV +
-compute buffer OOMs the SVM pool beyond that — verified live: cudaMalloc
-fails at 131072/262144), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache,
-`--fit off` (auto-fit segfaults on qwen4exp hyper-connection config),
-`--flash-attn on` (restored 2026-08-29 via the QSA F32-mask patch,
-`scripts/patches/qwen4exp-qsa-f32-mask.patch` — re-apply inside
-`/home/dank/GitPackages/ROCmFPX` before any clean rebuild), and reasoning on
-(Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
-`preserve_thinking` + `preserve_reasoning`). Env
+`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47
+GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from
+`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with
+the card's **pinned runtime**: the `kingjones30/ROCmFPX` fork
+(charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537 patch — ROCmFP4
+types and qwen4exp arch in one tree), built at
+`/home/dank/GitPackages/kingjones-ROCmFPX/build` with
+`-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON
+-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF` (WebUI
+provisioning needs npm/HF-bucket and aborts the link at 98% without
+the OFF flag). Args: our validated profile — 401408 context / 2 slots,
+b 4096 / ub 512 (ub 512 matches the card's re-measurement), t 16 / tb
+16, f16 KV, `--fit off`, `--flash-attn on` (works UNPATCHED on this
+model — the QSA F16 ggml_fill assert that needed
+`scripts/patches/qwen4exp-qsa-f32-mask.patch` is specific to the
+agentionai FAST layer set; if a future build aborts with
+`GGML_ASSERT(a-type == GGML_TYPE_F32)`, port that fix), reasoning on
+(Sharp template, deepseek format), no speculative decoding (card ships
+no spec flags; Flash-Next MTP heads do not load on this loader family).
+Two hard card constraints: **mmap must stay on** (`--no-mmap` forces
+the 50.66 GiB Q8_0 PLE table into anonymous memory → silent cgroup
+kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is
+correct here — mmap pages ARE page cache) and **never pin the PLE
+tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never
+enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file
+sha256 verified == HF LFS oid `c6770d74...49fa`. Env
 `scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`.
-
-**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix
-model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1
-deployments. The own-matched MTP draft head does NOT load on build-qwen38 —
-verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`),
-third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The
-head stays on disk for a future compatible binary. Same day, `--flash-attn on`
-was restored: the QSA F32-mask patch (vendored
-`scripts/patches/qwen4exp-qsa-f32-mask.patch`) fixes the F16-kq_mask
-`ggml_fill` abort; the rebuilt fork binary carries it, verified live at 32K.
 Runbook:
+[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md).
+
+**Serving status (2026-08-30):** swapped from the agentionai FAST
+deployment. Validated live before the systemd handoff: clean load,
+health 200, reasoning completions, and an operator session's ~111k
+token prefill at 213–232 t/s (card rung: 196 at 131k). Note llama.cpp
+clamps n_ctx_train 262144 → per-slot 131072 with parallel 2.
+
+### agentionai ROCmFP4-FAST imatrix llama.cpp (Dankclaw, prior, 2026-08-29 to 2026-08-30)
+
+Historical — rollback target. `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF`
+(87.06 GiB, Q4_0_ROCMFP4_FAST imatrix, v2/ joined-PLE layout only — the
+root `-ple16` per-head split aborts this loader family) on the charlie
+`build-qwen38` tree (`/home/dank/GitPackages/ROCmFPX`, PR #27742
+merged) at 32768/1, ub 4096. That tree REQUIRES
+`scripts/patches/qwen4exp-qsa-f32-mask.patch` for `--flash-attn on`
+(F16-kq_mask ggml_fill abort otherwise). Flash-Next MTP heads do not
+load on any current loader (require full per-layer
+`blk.<il>.hc_attn_norm.weight`; heads carry only `blk.48` — verified
+3x). Binary and payload retained. Runbook:
 [serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).

-### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE)
+### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE)

-The small service serves `peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF` (public,
-30 GB, UD-Q6_K_XL quant) from `/home/dank/models/tiel-coder-35b/` with
-upstream llama.cpp (latest master, CUDA build with
-`-DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86`). The model is a
-qwen35moe hybrid MoE: 40 layers (10 attention + 30 SSM/Mamba,
-full_attention_interval=4), 256 experts / 8 used, 2 KV heads × 256 dim. It
-runs Q8_0 KV cache, 200700 total context (66900/slot × 3 parallel), `--cpu-moe`
-for expert offload, `-ngl 999` (non-expert on GPU), and mmap. Env
-`scripts/frank-default-small.env`, serve `scripts/serve-frank-small.sh`.
+The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
+(public, ~35 GB, 26 safetensors shards) from
+`/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8/` with
+FreeToken v0.1.2 (`/home/dank/freetoken/.venv`, `ft serve`). Native
+modelopt block-fp8 (e4m3, dynamic, 128×128 weight_block_size,
+un-fused per-expert layout) — same format and loader path as the
+official Qwen3.6-35B-A3B-FP8 it replaced on 2026-08-30, so the entire
+memory/pool plan carries: experts host-RAM with 20% LRU GPU residency,
+paged KV (only the 10 full-attention layers page; 30 GDN layers use
+constant-size mamba slots), ~3.4 GiB VRAM headroom at boot, and the
+validated `--memory-num-gpu-blocks-experts 2048
+--memory-num-gpu-tokens-kv-cache 453005 --max-num-seqs 4` pool triple
+(never grow the expert pool). 262144 context, reasoning on (the
+checkpoint ships the froggeric fixed chat template — no override).
+FreeToken's API REQUIRES an explicit `"model"` field in chat requests
+(422 without it). Env `scripts/frank-default-small.env`, serve
+`scripts/serve-frank-small.sh`. Runbook:
+[serverAppConfigs-docs/LLM/freetoken-frank-default-small.md](serverAppConfigs-docs/LLM/freetoken-frank-default-small.md).

-**Serving status (2026-09-01):** deployed the Tiel-Coder-35B model. The
-previous ik_llama.cpp Ornith CRACK Q8_0 deployment is retired.
+**Serving status (2026-08-30):** swapped the model within the FreeToken
+runtime (official FP8 → Huihui abliterated FP8). ~40 tok/s verified on
+a 282-token completion post-deploy. Prior: official Qwen FP8
+(2026-09-02, runtime swap), Tiel-Coder-35B on llama.cpp (2026-09-01),
+ik_llama.cpp Ornith CRACK (retired 2026-09-02).

 ### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)

@@ -17,17 +17,21 @@ timestamp: 2026-08-29

 # agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw

-> **Active since 2026-08-29.** Replaced the kingjones STRIX_LEAN and jcbtc
-> CIRU v1.1 deployments on the same slot.
-> **MTP draft disabled** — the author's own head does not load on
-> build-qwen38 (see Known issues #3). Speculative decoding is ngram-mod only.
+> **RETIRED 2026-08-30** — replaced by the kingjones STRIX model on the
+> card's pinned kingjones30/ROCmFPX runtime; see
+> [[kingjones-strix-rocmfp4]] for the active runbook. This page stays as
+> the deployment record: the v2-joined-PLE layout requirement, the MTP
+> head incompatibility (verified 3x), and the QSA F32-mask patch
+> (`scripts/patches/qwen4exp-qsa-f32-mask.patch`) that made
+> `--flash-attn on` work on the charlie build-qwen38 tree — all still
+> true for that binary + model pair, which remains the rollback target.

-`frank-default-large` (port 7770) serves
+`frank-default-large` (port 7770) served
 `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
 `Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
 with the author's own MTP draft head
-(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the
-charlie12345/ROCmFPX `build-qwen38` runtime.
+(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`, never loaded) on the
+charlie12345/ROCmFPX `build-qwen38` runtime, 2026-08-29 → 2026-08-30.

 ## Why this model

@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: FreeToken frank-default-small Runbook
-description: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090).
+description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-default-small on Dankserver (RTX 3090).
 tags:
  - freetoken
  - dankserver
@@ -9,32 +9,40 @@ tags:
  - qwen3.6
  - moe
  - fp8
-timestamp: 2026-09-02
+timestamp: 2026-08-30
 ---

 # FreeToken frank-default-small Runbook

 The `frank-default-small` service on Dankserver is served by
 **FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
-expert weights in host RAM with an LRU GPU cache. It replaced the
-llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.
+expert weights in host RAM with an LRU GPU cache. Model history on this
+runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
+`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
+same format and pool budget). The engine itself replaced the llama.cpp
+Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.

 ## Model

 | Field | Value |
 |-------|-------|
-| HuggingFace | `Qwen/Qwen3.6-35B-A3B-FP8` |
+| HuggingFace | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` |
+| Lineage | Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp) |
 | Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
-| Quant | modelopt block-fp8 (128×128 `weight_block_size`) |
-| Size on disk | 35 GB |
-| Path | `/home/dank/models/qwen36-35b-a3b-fp8` |
+| Quant | native modelopt block-fp8 e4m3, dynamic activations, 128×128 `weight_block_size`, bf16 `weight_scale_inv` — byte-format identical to the official FP8 repo |
+| BF16 kept | lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module |
+| Experts | un-fused per-expert layout (`mlp.experts.{0..255}.*`) matching official Qwen FP8 — same FreeToken loader path |
+| Size on disk | ~35 GB (26 safetensors shards) |
+| Path | `/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8` |
 | Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
 | Experts | 256 per layer, 8 used per token |
 | KV heads | 2 × 256 dim |
 | Context | 262,144 (model max) |
+| Chat template | froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in `tokenizer_config.json`; FreeToken reads it from the checkpoint — no override) |

-**Important:** This serves base Qwen3.6-35B-A3B weights, not the
-Tiel-Coder finetune that was previously served via llama.cpp.
+**Important:** this is the Huihui *abliterated* finetune (refusals
+removed) — not the base Qwen weights served before 2026-08-30, and not
+the earlier Tiel-Coder finetune. Expect a direct-answers assistant.

 ## Runtime

@@ -71,7 +79,7 @@ ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
 - The fit check refuses overcommit: if the plan doesn't fit, the old
  cache is retained and serving continues.

-### Benchmark Results (2026-09-02)
+### Benchmark Results (2026-09-02, base FP8 checkpoint)

 See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
 across KV pool sizes 8k–700k.
@@ -110,7 +118,7 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:

 ## Limitations

- **Model identity**: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
+- **Model identity**: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era.
 - **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
  (~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
  `~/.cache/tvm-ffi`.
@@ -120,8 +128,21 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
 - **Env var collisions**: never name tunables `TEMP`, `HOST`, or
  `GPU_MAX_HEAP_SIZE` (HIP runtime reads these).

+## Operational quirks
+
+- **`model` field is required** in `/v1/chat/completions` bodies —
+  FreeToken rejects requests without it with a 422 (`Field required`),
+  unlike llama.cpp's server which defaults to the loaded model. Send
+  `"model": "frank-default-small"`.
+- Load on this swap took well under the old llama.cpp numbers:
+  weights → experts (parallel build) → CUDA graph capture → Uvicorn
+  in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
+  282-token completion.
+
 ## Citations

 - FreeToken paper: arXiv 2608.16157
 - FreeToken PyPI: `freetoken[accel]` v0.1.2
- HuggingFace model: `Qwen/Qwen3.6-35B-A3B-FP8`
+- HuggingFace model: `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
+- Upstream weights: `huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated`, `Qwen/Qwen3.6-35B-A3B-FP8`
+- Fixed chat template: `froggeric/Qwen-Fixed-Chat-Templates`
@@ -1,10 +1,10 @@
 # LLM Fleet — Knowledge Base

 Documentation for the LLM inference fleet on two local machines
-(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small,
-Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
-(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST
-imatrix + MTP draft). Both services run the Qwen-Sharp chat template.
+(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-default-small,
+Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX
+llama.cpp on Dankclaw (frank-default-large, kingjones777
+Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template.

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -19,17 +19,17 @@ graph LR
        A[RTX 3090 24GB + 60GB RAM]
        B2[FreeToken v0.1.2
 frank-default-small, port 7770]
-        C[Qwen3.6-35B-A3B-FP8
+        C[Huihui-Qwen3.6-35B-A3B-abliterated-FP8
 block-fp8, 20% MoE residency, 453k KV / 4 streams]
        A --> B2 --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[gfx1151 iGPU 64 GiB unified memory]
-        E[charlie12345/ROCmFPX build-qwen38
+        E[kingjones30/ROCmFPX build
 frank-default-large, port 7770]
-        F[agentionai Qwen3.8-Flash-Next
-ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot]
+        F[kingjones777 Qwen3.8-Flash-Next
+ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE]
        E2 --> E --> F
    end

@@ -47,19 +47,20 @@ ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot]
 ## Active Deployment

 - [[freetoken-frank-default-small]] — the active frank-default-small
-  runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver
-  (2026-09-02)
+  runbook: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-
+  abliterated-FP8 on Dankserver (swapped 2026-08-30; same format/pools)
 - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook:
-  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
-  Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head
-  (ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29)
- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook:
+- [[kingjones-strix-rocmfp4]] — ACTIVE frank-default-large runbook:
  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
-  (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier
+  (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
+  kingjones30/ROCmFPX fork (swapped 2026-08-30)
+- [[agentionai-fast-imatrix]] — retired frank-default-large runbook:
+  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB
+  Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38
+  (2026-08-29 → 2026-08-30); rollback binary + payload retained
 - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
 - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
 - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
@@ -1,7 +1,7 @@
 ---
 type: reference
 title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
-description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.
+description: ACTIVE frank-default-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
 tags:
  - llama.cpp
  - rocm
@@ -11,18 +11,23 @@ tags:
  - rocmfp4
  - strix-halo
  - qwen4exp
-timestamp: 2026-08-27
+timestamp: 2026-08-30
 ---

 # kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw

-> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same
-> build. Same GPU memory profile, larger PLE table.
+> **Active again since 2026-08-30** on the model card's **pinned runtime**:
+> the `kingjones30/ROCmFPX` fork (replaced the agentionai FAST deployment
+> which ran on charlie's `build-qwen38`). Text-only tier; the agentionai
+> f16 vision tower is paired for mmproj and loads clean. Speculative
+> decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads
+> do not load on this loader family — see MTP Draft Head).

 `frank-default-large` (port 7770) serves
-`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
-A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head
-loaded as a separate GGUF sidecar (see MTP Draft Head below).
+`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
+from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
+via `scripts/serve-frank-large.sh` + `scripts/frank-default-large.env`
+and the Qwen-Sharp chat template.

 ## Model Architecture

@@ -146,12 +151,35 @@ TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
 Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
 16` is the optimal setting for both phases.

+## Runtime build (kingjones30/ROCmFPX)
+
+The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX)
+— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so
+ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at
+`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd`
+(2026-08-27 card revision) with the card's recipe:
+
+```bash
+git clone https://github.com/kingjones30/ROCmFPX.git
+cd ROCmFPX
+cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
+cmake --build build --target llama-server llama-quantize -j16
+```
+
+- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree
+  (post-WebUI llama.cpp) provisions embedded UI assets via npm or the
+  HF llama-ui bucket; npm is absent here and the bucket download
+  half-failed, aborting the link at 98%. The API server is unaffected.
+  All other compile work succeeded with zero errors, confirming the
+  card's clean-clone claim.
+- All 35 flags our serve script passes were verified present in this
+  tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`,
+  `--reasoning-budget`, `--mmproj`, …).
+
 ## Serving Profile

-Runtime: charlie12345/ROCmFPX `llama-server` built at
-`/home/dank/GitPackages/ROCmFPX/build-qwen38`.
-
-Required flags:
+Deployed args (`scripts/frank-default-large.env` — our validated
+profile, unchanged from the FAST deployment except the binary and model):

 ```
 llama-server \
@@ -159,8 +187,8 @@ llama-server \
  --host 0.0.0.0 --port 7770 --jinja \
  --device ROCm0 --gpu-layers 999 \
  --fit off --flash-attn on \
-  --ctx-size 524288 --parallel 2 \
-  --batch-size 4096 --ubatch-size 4096 \
+  --ctx-size 401408 --parallel 2 \
+  --batch-size 4096 --ubatch-size 512 \
  --cache-type-k f16 --cache-type-v f16 \
  --threads 16 --threads-batch 16 \
  --ctx-checkpoints 16 --cache-ram 8192 \
@@ -170,59 +198,57 @@ llama-server \
  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
  --chat-template-file chat-template.jinja \
  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
-  --spec-type draft-mtp,ngram-mod \
-  --model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
-  --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
-  --spec-draft-ngl 999 --spec-draft-device ROCm0 \
-  --spec-draft-type-k f16 --spec-draft-type-v f16
+  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf
 ```

 Key choices:

- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp
+- **`--fit off`** — auto-fit segfaults on this model's qwen4exp
  hyper-connection/indexer configuration.
 - **mmap (default)** — do NOT use `--no-mmap`. See memory section.
- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
-  internally.
- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod
-  fallback. The `ngram-mod` combo is recommended by the HF card; the
-  prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s
-  ([[kingjones-strix-mtp]]). Benchmark both before committing.
- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance
-  prose loses more to rejected drafts than it gains (per the HF card).
+- **`--flash-attn on` works unpatched on this model** — verified live
+  2026-08-30 (graph built, model served). The QSA F16 `ggml_fill`
+  assert that forced [[agentionai-fast-imatrix]] onto the patched
+  charlie tree did not fire for the kingjones STRIX file; if a future
+  build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port
+  `scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree.
+- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the
+  FAST deployment; the card's own ladder proves the window is nearly
+  free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth:
+  a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
+- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096
+  show no effect; 512 is already right on this hardware.
+- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai
+  FAST package's f16 mmproj (same base model) is paired and loads
+  (one benign `warmup: CLIP graph uses unsupported operators` warning).
+- **No spec flags** — speculative decoding is off; see below.

-## MTP Draft Head
+## MTP Draft Head — not loadable on this loader family

-The MTP (multi-token-prediction) draft head from
+The previous draft of this runbook ran
 [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
-(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded
-as a separate GGUF sidecar via `--model-draft`.
+(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp
+loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set and
+every Flash-Next-era head carries only `blk.48` (verified 3x against
+multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known
+issues). The service therefore runs with no speculative decoding — the
+card's own Usage block ships none. The card's published +50% decode
+figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not
+build. Revisit only if a head with the complete hyper-connection norm
+set (or a loader that tolerates partial draft tensors) appears.

-| Field | Value |
-|-------|-------|
-| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` |
-| Size | 4.1 GB (4,137,429,088 B) |
-| Quant | Q8_0 |
-| Architecture | `qwen4exp` |
-| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF |

-**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on
-code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth.
-Speculative decoding is lossless at temp 0. Our service runs at temp 0.7,
-so expect lower acceptance; the `p_min 0.75` gate compensates.
+## Validation (2026-08-30 swap)

-**Caveats:**
-
- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is
-  the "Uncensored" (abliterated) variant. Abliteration modifies a small
-  number of weight vectors, not the full model — acceptance may be
-  marginally lower.
- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our
-  `charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp)
-  merged, so the MTP mechanism is the same upstream code. If the draft
-  load crashes, check for Strix-Halo kernel patch differences first.
- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable.
+1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`,
+   byte-identical to the HF LFS oid. No re-download needed.
+2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
+3. Generation: `/v1/chat/completions` returns correct
+   `reasoning_content` + content through the Sharp template (deepseek
+   format, `preserve_reasoning`).
+4. Real load: an operator session's ~111k-token prompt prefilled at
+   213–232 t/s under the new runtime, matching the card's 131k rung
+   (196 t/s), with checkpoints + QSA KV streaming live.

 ## Deployment

@@ -241,12 +267,20 @@ curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \

 The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.

-### Swap procedure
+### Swap procedure (as run 2026-08-30)

-1. Stop service: `sudo systemctl stop frank-default-large`
-2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN`
-3. Start service: `sudo systemctl start frank-default-large`
-4. Verify health: `curl http://192.168.0.143:7770/health`
+1. Build the runtime (see Runtime build) BEFORE touching the service.
+2. Point the model file's directory at the canonical path (a plain `mv`
+   — never delete a 113 GiB payload to re-fetch it).
+3. Edit `scripts/frank-default-large.env` (+ serve script defaults) via
+   git: commit, push, `./scripts/deploy.sh dankclaw --service
+   frank-default-large`.
+4. Verify health: `curl http://192.168.0.143:7770/health` and a real
+   `/v1/chat/completions` probe.
+
+Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`,
+`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38`
+binary and the agentionai payload remain on disk.

 ### Config files

@@ -256,8 +290,10 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.

 ## Known Issues

-1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork.
-   Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
+1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/
+   ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use
+   kingjones30/ROCmFPX, or the equivalent merged charlie tree
+   (`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits).
 2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
   curl. Browser download unreliable for 113 GiB.
 3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
@@ -271,6 +307,6 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.

 - [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
 - [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar
+- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo)
+- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
 - [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
@@ -11,6 +11,33 @@ timestamp: 2026-09-14

 # Bundle Change Log

+
+## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken
+- **Dankclaw / frank-default-large**: swapped to
+  `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the card's
+  pinned `kingjones30/ROCmFPX` runtime (built at
+  `/home/dank/GitPackages/kingjones-ROCmFPX/build`, `LLAMA_BUILD_WEBUI=OFF`
+  — npm/HF-bucket asset provisioning aborts the link at 98% otherwise).
+  On-host 113.47 GiB file verified byte-identical to the HF LFS oid
+  (no re-download). Kept our args (401408×2, ub 512, fa on, fit off);
+  fa-on works UNPATCHED on this model — the QSA F32 assert was
+  agentionai-specific. No spec decode (card ships none; MTP heads
+  unloadable). Validated live: ~225 s load, health 200, reasoning
+  completions, real 111k-token prefill at 213–232 t/s. Deployed via
+  git flow; manual-process race during validation caught and fixed
+  (unit was crash-looping on a bound port while its health check
+  passed — check MainPID, not just /health).
+- **Dankserver / frank-default-small**: swapped the model within
+  FreeToken v0.1.2 to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
+ (native block-fp8, same loader path and pool plan; 35 GB downloaded
+  via `hf`, official checkpoint retained for rollback). Learned:
+  FreeToken's chat API 422s without an explicit `model` field. ~40
+  tok/s verified post-deploy. Both `.service` Descriptions refreshed.
+- Docs: [[kingjones-strix-rocmfp4]] reactivated as the large runbook
+  (runtime build + swap procedure + validation), [[agentionai-fast-imatrix]]
+  marked RETIRED (rollback record), [[freetoken-frank-default-small]]
+  model sections updated, LLM index + AGENTS fleet tables refreshed.
+
 ## 2026-08-30 (late) — download chain proven end to end; qB credentials persisted
 - **Cracked the arr↔Prowlarr feed contract**: route is `/{indexerId}/api`
  (Prowlarr `NewznabController`; absolute `/api/v1/indexer/{id}/newznab`),