From 0937cfb0478951ab63f0ba332fdf8db4c95cf62c Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 30 Aug 2026 02:35:38 -0700 Subject: [PATCH] docs: record fleet double swap (large -> kingjones STRIX on pinned fork, small -> Huihui FP8 on FreeToken) - kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook: runtime build recipe (WEBUI=OFF gotcha), deployed serving profile, fa-on-works-unpatched evidence, MTP heads documented unloadable, swap procedure + 2026-08-30 validation record - agentionai-fast-imatrix marked RETIRED (rollback record kept) - freetoken-frank-default-small: model/lineage/template sections now Huihui abliterated FP8; new operational quirk: chat API 422 without explicit model field - LLM index + AGENTS fleet table/runtime sections current; both services' history chains fixed (small was still Tiel in AGENTS.md) - worklog entry added; models.json contextWindow for large corrected to the real per-slot 131072 (llama clamps n_ctx_train/parallel) --- AGENTS.md | 133 ++++++++------ .../LLM/agentionai-fast-imatrix.md | 18 +- .../LLM/freetoken-frank-default-small.md | 47 +++-- serverAppConfigs-docs/LLM/index.md | 33 ++-- .../LLM/kingjones-strix-rocmfp4.md | 162 +++++++++++------- serverAppConfigs-docs/log.md | 27 +++ 6 files changed, 268 insertions(+), 152 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fd9483a..170b615 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,12 +22,12 @@ prefix, reorganized in follow-up commits). The retired target checkouts archived. All services now run from `/home/dank/serverAppConfigs` on each host. -### LLM fleet topology (2026-08-22) +### LLM fleet topology (2026-08-30) | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 32K context / 1 slot, reasoning on | 7770 | +| Dankserver | FreeToken v0.1.2 | frank-default-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | +| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-default-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -40,7 +40,12 @@ Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft, Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0 AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into -charlie's ROCmFPX fork, --fit off, 128k context). Runbooks: +charlie's ROCmFPX fork, --fit off, 128k context). On 2026-09-02 +Dankserver moved to the FreeToken runtime serving the official +Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again: +Dankclaw to the kingjones STRIX model on the card's pinned +kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated +FP8 checkpoint on FreeToken. Runbooks: [serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md). Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. @@ -274,63 +279,85 @@ When you rename a service or move a model to a new host, update ## Runtime & Troubleshooting -### Qwen3.8-Flash-Next ROCmFP4-FAST imatrix llama.cpp (Dankclaw, ACTIVE) +### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE) The large service serves -`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (public, -87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated) from -`/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/` with a -charlie12345/ROCmFPX `llama-server` built at -`/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture -merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn + -36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048), -hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter -n-gram table is a single joined tensor in the `v2/` GGUF we serve (the root -`-ple16` per-head split requires the LaurentZuijdwijk Vulkan fork and aborts -this loader's `GGML_ASSERT(qwen4exp is missing the PLE n-gram table)`) — fully -VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head -(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST -quant matched to target) is on disk but does **not** load on this binary -(loader requires a full per-layer `blk..hc_attn_norm.weight` set; the head -carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision -tower (f16) ships as mmproj/. It runs 32768 context / 1 -slot (author quick-start; the fully GPU-resident 87 GiB model + f16 KV + -compute buffer OOMs the SVM pool beyond that — verified live: cudaMalloc -fails at 131072/262144), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, -`--fit off` (auto-fit segfaults on qwen4exp hyper-connection config), -`--flash-attn on` (restored 2026-08-29 via the QSA F32-mask patch, -`scripts/patches/qwen4exp-qsa-f32-mask.patch` — re-apply inside -`/home/dank/GitPackages/ROCmFPX` before any clean rebuild), and reasoning on -(Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep -`preserve_thinking` + `preserve_reasoning`). Env +`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47 +GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from +`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with +the card's **pinned runtime**: the `kingjones30/ROCmFPX` fork +(charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537 patch — ROCmFP4 +types and qwen4exp arch in one tree), built at +`/home/dank/GitPackages/kingjones-ROCmFPX/build` with +`-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON +-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF` (WebUI +provisioning needs npm/HF-bucket and aborts the link at 98% without +the OFF flag). Args: our validated profile — 401408 context / 2 slots, +b 4096 / ub 512 (ub 512 matches the card's re-measurement), t 16 / tb +16, f16 KV, `--fit off`, `--flash-attn on` (works UNPATCHED on this +model — the QSA F16 ggml_fill assert that needed +`scripts/patches/qwen4exp-qsa-f32-mask.patch` is specific to the +agentionai FAST layer set; if a future build aborts with +`GGML_ASSERT(a-type == GGML_TYPE_F32)`, port that fix), reasoning on +(Sharp template, deepseek format), no speculative decoding (card ships +no spec flags; Flash-Next MTP heads do not load on this loader family). +Two hard card constraints: **mmap must stay on** (`--no-mmap` forces +the 50.66 GiB Q8_0 PLE table into anonymous memory → silent cgroup +kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is +correct here — mmap pages ARE page cache) and **never pin the PLE +tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never +enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file +sha256 verified == HF LFS oid `c6770d74...49fa`. Env `scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`. - -**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix -model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1 -deployments. The own-matched MTP draft head does NOT load on build-qwen38 — -verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`), -third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The -head stays on disk for a future compatible binary. Same day, `--flash-attn on` -was restored: the QSA F32-mask patch (vendored -`scripts/patches/qwen4exp-qsa-f32-mask.patch`) fixes the F16-kq_mask -`ggml_fill` abort; the rebuilt fork binary carries it, verified live at 32K. Runbook: +[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md). + +**Serving status (2026-08-30):** swapped from the agentionai FAST +deployment. Validated live before the systemd handoff: clean load, +health 200, reasoning completions, and an operator session's ~111k +token prefill at 213–232 t/s (card rung: 196 at 131k). Note llama.cpp +clamps n_ctx_train 262144 → per-slot 131072 with parallel 2. + +### agentionai ROCmFP4-FAST imatrix llama.cpp (Dankclaw, prior, 2026-08-29 to 2026-08-30) + +Historical — rollback target. `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` +(87.06 GiB, Q4_0_ROCMFP4_FAST imatrix, v2/ joined-PLE layout only — the +root `-ple16` per-head split aborts this loader family) on the charlie +`build-qwen38` tree (`/home/dank/GitPackages/ROCmFPX`, PR #27742 +merged) at 32768/1, ub 4096. That tree REQUIRES +`scripts/patches/qwen4exp-qsa-f32-mask.patch` for `--flash-attn on` +(F16-kq_mask ggml_fill abort otherwise). Flash-Next MTP heads do not +load on any current loader (require full per-layer +`blk..hc_attn_norm.weight`; heads carry only `blk.48` — verified +3x). Binary and payload retained. Runbook: [serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md). -### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE) +### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE) -The small service serves `peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF` (public, -30 GB, UD-Q6_K_XL quant) from `/home/dank/models/tiel-coder-35b/` with -upstream llama.cpp (latest master, CUDA build with -`-DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86`). The model is a -qwen35moe hybrid MoE: 40 layers (10 attention + 30 SSM/Mamba, -full_attention_interval=4), 256 experts / 8 used, 2 KV heads × 256 dim. It -runs Q8_0 KV cache, 200700 total context (66900/slot × 3 parallel), `--cpu-moe` -for expert offload, `-ngl 999` (non-expert on GPU), and mmap. Env -`scripts/frank-default-small.env`, serve `scripts/serve-frank-small.sh`. +The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` +(public, ~35 GB, 26 safetensors shards) from +`/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8/` with +FreeToken v0.1.2 (`/home/dank/freetoken/.venv`, `ft serve`). Native +modelopt block-fp8 (e4m3, dynamic, 128×128 weight_block_size, +un-fused per-expert layout) — same format and loader path as the +official Qwen3.6-35B-A3B-FP8 it replaced on 2026-08-30, so the entire +memory/pool plan carries: experts host-RAM with 20% LRU GPU residency, +paged KV (only the 10 full-attention layers page; 30 GDN layers use +constant-size mamba slots), ~3.4 GiB VRAM headroom at boot, and the +validated `--memory-num-gpu-blocks-experts 2048 +--memory-num-gpu-tokens-kv-cache 453005 --max-num-seqs 4` pool triple +(never grow the expert pool). 262144 context, reasoning on (the +checkpoint ships the froggeric fixed chat template — no override). +FreeToken's API REQUIRES an explicit `"model"` field in chat requests +(422 without it). Env `scripts/frank-default-small.env`, serve +`scripts/serve-frank-small.sh`. Runbook: +[serverAppConfigs-docs/LLM/freetoken-frank-default-small.md](serverAppConfigs-docs/LLM/freetoken-frank-default-small.md). -**Serving status (2026-09-01):** deployed the Tiel-Coder-35B model. The -previous ik_llama.cpp Ornith CRACK Q8_0 deployment is retired. +**Serving status (2026-08-30):** swapped the model within the FreeToken +runtime (official FP8 → Huihui abliterated FP8). ~40 tok/s verified on +a 282-token completion post-deploy. Prior: official Qwen FP8 +(2026-09-02, runtime swap), Tiel-Coder-35B on llama.cpp (2026-09-01), +ik_llama.cpp Ornith CRACK (retired 2026-09-02). ### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25) diff --git a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md index 03a57b1..64fc8e4 100644 --- a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md +++ b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md @@ -17,17 +17,21 @@ timestamp: 2026-08-29 # agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw -> **Active since 2026-08-29.** Replaced the kingjones STRIX_LEAN and jcbtc -> CIRU v1.1 deployments on the same slot. -> **MTP draft disabled** — the author's own head does not load on -> build-qwen38 (see Known issues #3). Speculative decoding is ngram-mod only. +> **RETIRED 2026-08-30** — replaced by the kingjones STRIX model on the +> card's pinned kingjones30/ROCmFPX runtime; see +> [[kingjones-strix-rocmfp4]] for the active runbook. This page stays as +> the deployment record: the v2-joined-PLE layout requirement, the MTP +> head incompatibility (verified 3x), and the QSA F32-mask patch +> (`scripts/patches/qwen4exp-qsa-f32-mask.patch`) that made +> `--flash-attn on` work on the charlie build-qwen38 tree — all still +> true for that binary + model pair, which remains the rollback target. -`frank-default-large` (port 7770) serves +`frank-default-large` (port 7770) served `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main, `Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout) with the author's own MTP draft head -(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the -charlie12345/ROCmFPX `build-qwen38` runtime. +(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`, never loaded) on the +charlie12345/ROCmFPX `build-qwen38` runtime, 2026-08-29 → 2026-08-30. ## Why this model diff --git a/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md b/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md index ef6cfc8..212ab61 100644 --- a/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md +++ b/serverAppConfigs-docs/LLM/freetoken-frank-default-small.md @@ -1,7 +1,7 @@ --- type: configuration title: FreeToken frank-default-small Runbook -description: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090). +description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-default-small on Dankserver (RTX 3090). tags: - freetoken - dankserver @@ -9,32 +9,40 @@ tags: - qwen3.6 - moe - fp8 -timestamp: 2026-09-02 +timestamp: 2026-08-30 --- # FreeToken frank-default-small Runbook The `frank-default-small` service on Dankserver is served by **FreeToken v0.1.2** — an edge-native MoE serving engine that keeps -expert weights in host RAM with an LRU GPU cache. It replaced the -llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02. +expert weights in host RAM with an LRU GPU cache. Model history on this +runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) → +`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30, +same format and pool budget). The engine itself replaced the llama.cpp +Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment. ## Model | Field | Value | |-------|-------| -| HuggingFace | `Qwen/Qwen3.6-35B-A3B-FP8` | +| HuggingFace | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` | +| Lineage | Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp) | | Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) | -| Quant | modelopt block-fp8 (128×128 `weight_block_size`) | -| Size on disk | 35 GB | -| Path | `/home/dank/models/qwen36-35b-a3b-fp8` | +| Quant | native modelopt block-fp8 e4m3, dynamic activations, 128×128 `weight_block_size`, bf16 `weight_scale_inv` — byte-format identical to the official FP8 repo | +| BF16 kept | lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module | +| Experts | un-fused per-expert layout (`mlp.experts.{0..255}.*`) matching official Qwen FP8 — same FreeToken loader path | +| Size on disk | ~35 GB (26 safetensors shards) | +| Path | `/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8` | | Layers | 40 (10 full-attention + 30 GDN/linear-attention) | | Experts | 256 per layer, 8 used per token | | KV heads | 2 × 256 dim | | Context | 262,144 (model max) | +| Chat template | froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in `tokenizer_config.json`; FreeToken reads it from the checkpoint — no override) | -**Important:** This serves base Qwen3.6-35B-A3B weights, not the -Tiel-Coder finetune that was previously served via llama.cpp. +**Important:** this is the Huihui *abliterated* finetune (refusals +removed) — not the base Qwen weights served before 2026-08-30, and not +the earlier Tiel-Coder finetune. Expect a direct-answers assistant. ## Runtime @@ -71,7 +79,7 @@ ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300 - The fit check refuses overcommit: if the plan doesn't fit, the old cache is retained and serving continues. -### Benchmark Results (2026-09-02) +### Benchmark Results (2026-09-02, base FP8 checkpoint) See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep across KV pool sizes 8k–700k. @@ -110,7 +118,7 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`: ## Limitations -- **Model identity**: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune. +- **Model identity**: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era. - **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels (~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and `~/.cache/tvm-ffi`. @@ -120,8 +128,21 @@ FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`: - **Env var collisions**: never name tunables `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE` (HIP runtime reads these). +## Operational quirks + +- **`model` field is required** in `/v1/chat/completions` bodies — + FreeToken rejects requests without it with a 422 (`Field required`), + unlike llama.cpp's server which defaults to the loaded model. Send + `"model": "frank-default-small"`. +- Load on this swap took well under the old llama.cpp numbers: + weights → experts (parallel build) → CUDA graph capture → Uvicorn + in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a + 282-token completion. + ## Citations - FreeToken paper: arXiv 2608.16157 - FreeToken PyPI: `freetoken[accel]` v0.1.2 -- HuggingFace model: `Qwen/Qwen3.6-35B-A3B-FP8` \ No newline at end of file +- HuggingFace model: `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` +- Upstream weights: `huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated`, `Qwen/Qwen3.6-35B-A3B-FP8` +- Fixed chat template: `froggeric/Qwen-Fixed-Chat-Templates` \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 698f5ed..546ba41 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,10 +1,10 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small, -Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw -(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST -imatrix + MTP draft). Both services run the Qwen-Sharp chat template. +(2026-08-30): FreeToken v0.1.2 on Dankserver (frank-default-small, +Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and kingjones30/ROCmFPX +llama.cpp on Dankclaw (frank-default-large, kingjones777 +Qwen3.8-Flash-Next ROCmFP4-STRIX). Both services run the Qwen-Sharp chat template. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -19,17 +19,17 @@ graph LR A[RTX 3090 24GB + 60GB RAM] B2[FreeToken v0.1.2 frank-default-small, port 7770] - C[Qwen3.6-35B-A3B-FP8 + C[Huihui-Qwen3.6-35B-A3B-abliterated-FP8 block-fp8, 20% MoE residency, 453k KV / 4 streams] A --> B2 --> C end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[gfx1151 iGPU 64 GiB unified memory] - E[charlie12345/ROCmFPX build-qwen38 + E[kingjones30/ROCmFPX build frank-default-large, port 7770] - F[agentionai Qwen3.8-Flash-Next -ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot] + F[kingjones777 Qwen3.8-Flash-Next +ROCmFP4-STRIX, 401k ctx / 2 slots, mmap PLE] E2 --> E --> F end @@ -47,19 +47,20 @@ ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot] ## Active Deployment - [[freetoken-frank-default-small]] — the active frank-default-small - runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver - (2026-09-02) + runbook: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B- + abliterated-FP8 on Dankserver (swapped 2026-08-30; same format/pools) - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook: - agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB - Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head - (ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29) -- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook: +- [[kingjones-strix-rocmfp4]] — ACTIVE frank-default-large runbook: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params - (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier + (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned + kingjones30/ROCmFPX fork (swapped 2026-08-30) +- [[agentionai-fast-imatrix]] — retired frank-default-large runbook: + agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB + Q4_0_ROCMFP4_FAST imatrix, v2 joined PLE) on build-qwen38 + (2026-08-29 → 2026-08-30); rollback binary + payload retained - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model diff --git a/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md b/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md index b4284d7..4fcb009 100644 --- a/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md +++ b/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md @@ -1,7 +1,7 @@ --- type: reference title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw -description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large. +description: ACTIVE frank-default-large runbook. kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots. tags: - llama.cpp - rocm @@ -11,18 +11,23 @@ tags: - rocmfp4 - strix-halo - qwen4exp -timestamp: 2026-08-27 +timestamp: 2026-08-30 --- # kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw -> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same -> build. Same GPU memory profile, larger PLE table. +> **Active again since 2026-08-30** on the model card's **pinned runtime**: +> the `kingjones30/ROCmFPX` fork (replaced the agentionai FAST deployment +> which ran on charlie's `build-qwen38`). Text-only tier; the agentionai +> f16 vision tower is paired for mmproj and loads clean. Speculative +> decoding is OFF (the card ships no spec flags; the Flash-Next MTP heads +> do not load on this loader family — see MTP Draft Head). `frank-default-large` (port 7770) serves -`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face. -A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head -loaded as a separate GGUF sidecar (see MTP Draft Head below). +`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB) +from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` +via `scripts/serve-frank-large.sh` + `scripts/frank-default-large.env` +and the Qwen-Sharp chat template. ## Model Architecture @@ -146,12 +151,35 @@ TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s. Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch 16` is the optimal setting for both phases. +## Runtime build (kingjones30/ROCmFPX) + +The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) +— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so +ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at +`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd` +(2026-08-27 card revision) with the card's recipe: + +```bash +git clone https://github.com/kingjones30/ROCmFPX.git +cd ROCmFPX +cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF +cmake --build build --target llama-server llama-quantize -j16 +``` + +- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree + (post-WebUI llama.cpp) provisions embedded UI assets via npm or the + HF llama-ui bucket; npm is absent here and the bucket download + half-failed, aborting the link at 98%. The API server is unaffected. + All other compile work succeeded with zero errors, confirming the + card's clean-clone claim. +- All 35 flags our serve script passes were verified present in this + tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`, + `--reasoning-budget`, `--mmproj`, …). + ## Serving Profile -Runtime: charlie12345/ROCmFPX `llama-server` built at -`/home/dank/GitPackages/ROCmFPX/build-qwen38`. - -Required flags: +Deployed args (`scripts/frank-default-large.env` — our validated +profile, unchanged from the FAST deployment except the binary and model): ``` llama-server \ @@ -159,8 +187,8 @@ llama-server \ --host 0.0.0.0 --port 7770 --jinja \ --device ROCm0 --gpu-layers 999 \ --fit off --flash-attn on \ - --ctx-size 524288 --parallel 2 \ - --batch-size 4096 --ubatch-size 4096 \ + --ctx-size 401408 --parallel 2 \ + --batch-size 4096 --ubatch-size 512 \ --cache-type-k f16 --cache-type-v f16 \ --threads 16 --threads-batch 16 \ --ctx-checkpoints 16 --cache-ram 8192 \ @@ -170,59 +198,57 @@ llama-server \ --reasoning on --reasoning-format deepseek --reasoning-budget -1 \ --chat-template-file chat-template.jinja \ --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \ - --spec-type draft-mtp,ngram-mod \ - --model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \ - --spec-draft-n-max 4 --spec-draft-p-min 0.75 \ - --spec-draft-ngl 999 --spec-draft-device ROCm0 \ - --spec-draft-type-k f16 --spec-draft-type-v f16 + --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf ``` Key choices: -- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp +- **`--fit off`** — auto-fit segfaults on this model's qwen4exp hyper-connection/indexer configuration. - **mmap (default)** — do NOT use `--no-mmap`. See memory section. -- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool. -- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps - internally. -- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod - fallback. The `ngram-mod` combo is recommended by the HF card; the - prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s - ([[kingjones-strix-mtp]]). Benchmark both before committing. -- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance - prose loses more to rejected drafts than it gains (per the HF card). +- **`--flash-attn on` works unpatched on this model** — verified live + 2026-08-30 (graph built, model served). The QSA F16 `ggml_fill` + assert that forced [[agentionai-fast-imatrix]] onto the patched + charlie tree did not fire for the kingjones STRIX file; if a future + build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port + `scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree. +- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the + FAST deployment; the card's own ladder proves the window is nearly + free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth: + a 111k-token prompt prefills at ~230 t/s while a second slot decodes. +- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096 + show no effect; 512 is already right on this hardware. +- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai + FAST package's f16 mmproj (same base model) is paired and loads + (one benign `warmup: CLIP graph uses unsupported operators` warning). +- **No spec flags** — speculative decoding is off; see below. -## MTP Draft Head +## MTP Draft Head — not loadable on this loader family -The MTP (multi-token-prediction) draft head from +The previous draft of this runbook ran [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) -(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded -as a separate GGUF sidecar via `--model-draft`. +(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp +loader requires a full per-layer `blk..hc_attn_norm.weight` set and +every Flash-Next-era head carries only `blk.48` (verified 3x against +multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known +issues). The service therefore runs with no speculative decoding — the +card's own Usage block ships none. The card's published +50% decode +figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not +build. Revisit only if a head with the complete hyper-connection norm +set (or a loader that tolerates partial draft tensors) appears. -| Field | Value | -|-------|-------| -| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` | -| Size | 4.1 GB (4,137,429,088 B) | -| Quant | Q8_0 | -| Architecture | `qwen4exp` | -| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF | -**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on -code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth. -Speculative decoding is lossless at temp 0. Our service runs at temp 0.7, -so expect lower acceptance; the `p_min 0.75` gate compensates. +## Validation (2026-08-30 swap) -**Caveats:** - -- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is - the "Uncensored" (abliterated) variant. Abliteration modifies a small - number of weight vectors, not the full model — acceptance may be - marginally lower. -- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our - `charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp) - merged, so the MTP mechanism is the same upstream code. If the draft - load crashes, check for Strix-Halo kernel patch differences first. -- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable. +1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`, + byte-identical to the HF LFS oid. No re-download needed. +2. Load: systemd unit healthy; model loaded in ~225 s; health 200. +3. Generation: `/v1/chat/completions` returns correct + `reasoning_content` + content through the Sharp template (deepseek + format, `preserve_reasoning`). +4. Real load: an operator session's ~111k-token prompt prefilled at + 213–232 t/s under the new runtime, matching the card's 131k rung + (196 t/s), with checkpoints + QSA KV streaming live. ## Deployment @@ -241,12 +267,20 @@ curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network. -### Swap procedure +### Swap procedure (as run 2026-08-30) -1. Stop service: `sudo systemctl stop frank-default-large` -2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN` -3. Start service: `sudo systemctl start frank-default-large` -4. Verify health: `curl http://192.168.0.143:7770/health` +1. Build the runtime (see Runtime build) BEFORE touching the service. +2. Point the model file's directory at the canonical path (a plain `mv` + — never delete a 113 GiB payload to re-fetch it). +3. Edit `scripts/frank-default-large.env` (+ serve script defaults) via + git: commit, push, `./scripts/deploy.sh dankclaw --service + frank-default-large`. +4. Verify health: `curl http://192.168.0.143:7770/health` and a real + `/v1/chat/completions` probe. + +Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`, +`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38` +binary and the agentionai payload remain on disk. ### Config files @@ -256,8 +290,10 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network. ## Known Issues -1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork. - Stock builds reject these files (TYPE_100/101 have no number in stock ggml). +1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/ + ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use + kingjones30/ROCmFPX, or the equivalent merged charlie tree + (`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits). 2. **Single-file payload.** Not sharded. Use `hf download` or a resumable curl. Browser download unreliable for 113 GiB. 3. **Model load is slow.** A few minutes cold for the 113 GiB file with a @@ -271,6 +307,6 @@ The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network. - [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF) - [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) -- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) -- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar +- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo) +- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged - [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 58099f9..806efde 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,33 @@ timestamp: 2026-09-14 # Bundle Change Log + +## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken +- **Dankclaw / frank-default-large**: swapped to + `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the card's + pinned `kingjones30/ROCmFPX` runtime (built at + `/home/dank/GitPackages/kingjones-ROCmFPX/build`, `LLAMA_BUILD_WEBUI=OFF` + — npm/HF-bucket asset provisioning aborts the link at 98% otherwise). + On-host 113.47 GiB file verified byte-identical to the HF LFS oid + (no re-download). Kept our args (401408×2, ub 512, fa on, fit off); + fa-on works UNPATCHED on this model — the QSA F32 assert was + agentionai-specific. No spec decode (card ships none; MTP heads + unloadable). Validated live: ~225 s load, health 200, reasoning + completions, real 111k-token prefill at 213–232 t/s. Deployed via + git flow; manual-process race during validation caught and fixed + (unit was crash-looping on a bound port while its health check + passed — check MainPID, not just /health). +- **Dankserver / frank-default-small**: swapped the model within + FreeToken v0.1.2 to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` + (native block-fp8, same loader path and pool plan; 35 GB downloaded + via `hf`, official checkpoint retained for rollback). Learned: + FreeToken's chat API 422s without an explicit `model` field. ~40 + tok/s verified post-deploy. Both `.service` Descriptions refreshed. +- Docs: [[kingjones-strix-rocmfp4]] reactivated as the large runbook + (runtime build + swap procedure + validation), [[agentionai-fast-imatrix]] + marked RETIRED (rollback record), [[freetoken-frank-default-small]] + model sections updated, LLM index + AGENTS fleet tables refreshed. + ## 2026-08-30 (late) — download chain proven end to end; qB credentials persisted - **Cracked the arr↔Prowlarr feed contract**: route is `/{indexerId}/api` (Prowlarr `NewznabController`; absolute `/api/v1/indexer/{id}/newznab`),