diff --git a/AGENTS.md b/AGENTS.md index 3c3aa99..f1f59b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 | +| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head (ROCmFP4-FAST), 262K context / 1 slot, reasoning on | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -274,27 +274,36 @@ When you rename a service or move a model to a new host, update ## Runtime & Troubleshooting -### Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN llama.cpp (Dankclaw, ACTIVE) +### Qwen3.8-Flash-Next ROCmFP4-FAST imatrix llama.cpp (Dankclaw, ACTIVE) The large service serves -`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF` (public, 98.5 GiB) -from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN/` -with a charlie12345/ROCmFPX `llama-server` built at +`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (public, +87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated) from +`/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/` with a +charlie12345/ROCmFPX `llama-server` built at `/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn + 36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048), -hyper-connections (count=4, low_rank=320), PLE (layer 1). Q4_0_ROCMFP4 quant. -It runs f16 KV cache, 131072 context / 1 slot, `--fit off` (auto-fit segfaults -on qwen4exp hyper-connection config), and reasoning on (Sharp template, budget --1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` + -`preserve_reasoning`). Env `scripts/frank-default-large.env`, serve -`scripts/serve-frank-large.sh`. +hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter +n-gram table is split per head INSIDE the single GGUF — fully VRAM-resident, +no sidecar, no `--ngram-on-disk`. The model's own MTP draft head +(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST +quant matched to target, ~66% draft acceptance) drives speculative decoding +at depth 3; vision tower (f16) ships as mmproj/. It runs 262144 context / 1 +slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit +segfaults on qwen4exp hyper-connection config), and reasoning on (Sharp +template, budget -1, `--reasoning-format deepseek`; chat kwargs keep +`preserve_thinking` + `preserve_reasoning`). Env +`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`. -**Serving status (2026-09-01):** deployed the Flash-Next 56B MoE model. The -previous kingjones Q6_0 AGENT model (27B, standalone MTP, 262144 context) is -retired. The Q8_0 AGENT experiment is documented in -`serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in -`ggml_compute_forward_set_rows` and did not provide a working service. +**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix +model with its own matched MTP draft head, replacing the jcbtc CIRU v1.1 +deployment (ciru-ai runtime). The 2026-08-28 failure of this head on the +kingjones STRIX_LEAN model (`missing tensor blk.0.hc_attn_norm.weight`) was a +cross-vendor pairing problem — the head carries the hyper-connection norm +tensor (`blk.48.hc_attn_norm.weight`) and works against its own main file. +Runbook: +[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md). ### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE) diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 304e8bc..d33115a 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -1,45 +1,48 @@ -# Tunables for frank-default-large (jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 on -# the ciru-ai Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp runtime, tag v1.1, -# Dankclaw, Strix Halo gfx1151 unified memory). +# Tunables for frank-default-large (agentionai/Qwen3.8-Flash-Next-ROCmFP4- +# FAST-imatrix-GGUF on the charlie12345/ROCmFPX llama.cpp runtime, +# build-qwen38, Dankclaw, Strix Halo gfx1151 unified memory). # -# Runtime: full llama.cpp fork at -# /home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151, -# built with the vendor build-linux-amd.sh flag set (HIP on, HIP_NO_VMM on, -# HIP_GRAPHS on, HIP_MMQ_MFMA on, CUDA/VULKAN/SYCL off, gfx1151) plus -# GGML_CUDA_FA + GGML_CUDA_FA_ALL_QUANTS. The IU4 WMMA expert path and the -# CIRUPLE1 PLE pager read the GGML_* env knobs exported by the serve script. -# Replaced the kingjones STRIX_LEAN (ROCmFPX build-qwen38) deployment. +# Runtime: charlie12345/ROCmFPX llama.cpp at +# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, ROCm 7.2.4, gfx1151, +# PR #27742 qwen4exp merged). The ROCmFP4 quant types are hand-ported from +# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF. Replaced the +# jcbtc CIRU v1.1 (ciru-ai runtime) and kingjones STRIX_LEAN deployments. # -# Model: Qwen3.8-Flash-Next base (56B MoE, 512 experts / 10 used, 48 layers -# = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE at layer 1) -# in the CIRU quality-first mixed quant: 144 routed-expert tensors at Q4_1 -# on the native IU4 WMMA path, protected tensors at Q5_K/Q5_1/Q8_0/BF16/F32. -# Main GGUF ~79 GB + MTP Q8_0 draft ~4 GB + exact-FP8 PLE sidecar ~52 GB -# paged from disk (never in VRAM; 4 GiB decoded-page cache via -# --ple-cache-mib). Vendor-validated on the same silicon (gfx1151). +# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, +# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE +# at layer 1) quantized Q4_0_ROCMFP4_FAST imatrix-calibrated, 87.06 GiB, +# 4.23 bpw. The n-gram PLE table is split per head inside the single GGUF +# (fully VRAM-resident — no ple sidecar, no --ngram-on-disk). Vision tower +# (f16, lossless) ships as mmproj/. The model's own MTP head (2.28 GiB, +# matched ROCmFP4-FAST quant, ~66% draft acceptance) drives spec decoding. # -# The vendor production profile is applied verbatim on this first deployment: -# -ngl all, -sm none, --fit off, ctx 262144 x 1 slot, b 2048 / ub 512, -# t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram -# + idle-slot cache, 32 ctx checkpoints with min step 8192, MTP depth 3 -# with p-split 0.10, GGML_CUDA_Q41_MOE_FORCE_J=32, -# GGML_QWEN4EXP_PLE_WORKERS=16, GGML_QWEN4EXP_PLE_STRICT_SHA=0 (checksums -# were verified with sha256sum -c at download time), ROCBLAS_USE_HIPBLASLT=1. -# The Qwen-Sharp chat template is kept on purpose (chat-template.jinja): the -# service keeps emitting reasoning in DeepSeek format for its clients. +# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0, +# presence 1.5 — non-thinking instruct) mixed with the validated +# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096, +# t 16 / tb 16 (threads-batch 16 measured best for prefill), f16 KV, fa on, +# cont-batching, prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx +# checkpoints, --fit off (auto-fit segfaults on qwen4exp +# hyper-connection/indexer config). MTP depth 3 (n-max 3 measured best on +# this head; adaptive p-split measured worse, left off). The Qwen-Sharp +# chat template is kept on purpose (chat-template.jinja): the service keeps +# emitting reasoning in DeepSeek format for its clients. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP # incident). llama.cpp reads --host/--port/--temp as CLI flags, never as # env vars. -# CIRU v1.1 runtime binary (vendor build script, plus FA_ALL_QUANTS). -LLAMA_BIN=/home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151/bin/llama-server +# ROCmFPX (build-qwen38) runtime binary. +LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server -# Model package from https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 -# Layout: main GGUF + mtp/ Q8_0 draft + ple/ CIRUPLE1 sidecar (mandatory). -MODEL_DIR=/home/dank/models/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4 -MODEL_FILE=Qwen3.8-Flash-CIRU-STRIX-IU4.gguf +# Model package from +# https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF +# Layout: single main GGUF (per-head PLE table, fully VRAM-resident) + +# mmproj/ vision tower. The MTP draft head lives in +# /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST (already on the host). +MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST +MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf +MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf # Qwen-Sharp chat template (tracked in the repo, staged on the host). The # Sharp template drives thinking on by default (medium effort); the server @@ -53,28 +56,27 @@ ALIAS=frank-default-large BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# Vendor production profile, verbatim: 262144-token context on ONE slot -# (parallel 1), batch 2048 / ubatch 512, 8+8 threads, f16 KV, 32 context -# checkpoints with min step 8192, 8 GiB prompt cache, 4 GiB PLE page cache. -# --fit off: auto-fit segfaults on this model's hyper-connection/indexer -# config; -ngl all puts every layer on the SVM-backed GPU pool. +# Serving: 262144-token context on ONE slot (parallel 1), batch 4096 / +# ubatch 4096, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB +# prompt cache. --fit off: auto-fit segfaults on this model's +# hyper-connection/indexer config; -ngl 999 puts every layer on the +# SVM-backed GPU pool. CTX_SIZE=262144 PARALLEL=1 -BATCH_SIZE=2048 -UBATCH_SIZE=512 -NUM_THREADS=8 -THREAD_BATCH=8 +BATCH_SIZE=4096 +UBATCH_SIZE=4096 +NUM_THREADS=16 +THREAD_BATCH=16 CACHE_TYPE_K=f16 CACHE_TYPE_V=f16 -CTX_CHECKPOINTS=32 -CHECKPOINT_MIN_STEP=8192 +CTX_CHECKPOINTS=16 CACHE_RAM=8192 -PLE_CACHE_MIB=4096 SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state SPLIT_MODE=none FIT=off +NGL=999 -# Sampling (Sharp default; vendor API examples agree for non-thinking mode). +# Sampling (card instruct table, non-thinking). TEMPERATURE=0.7 TOP_K=20 TOP_P=0.8 @@ -93,26 +95,18 @@ HSA_OVERRIDE_GFX_VERSION=11.5.1 VERBOSITY=4 -# Speculative decoding: MTP draft head (Q8_0) at depth 3, p-split 0.10 — -# the vendor headline profile. ngram-mod stacking is a later experiment; -# SPEC_TYPE=none disables spec decoding entirely. -DRAFT_MODEL_DIR=/home/dank/models/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/mtp -DRAFT_MODEL_FILE=Qwen3.8-Flash-CIRU-STRIX-IU4-MTP-Q8_0.gguf -SPEC_TYPE=draft-mtp +# Speculative decoding: the model's own MTP head (agentionai +# Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, ROCmFP4-FAST quant +# matched to the target) at depth 3 stacked on ngram-mod. n-max 3 measured +# best (32.4 t/s, 0.612 acceptance); adaptive p-split measured worse, so it +# stays off (p-min stays at the 0.75 default). Set SPEC_TYPE=none to disable. +DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST +DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf +SPEC_TYPE=draft-mtp,ngram-mod SPEC_DRAFT_N_MAX=3 SPEC_DRAFT_N_MIN=0 -SPEC_DRAFT_P_MIN=0.0 -SPEC_DRAFT_P_SPLIT=0.10 -SPEC_DRAFT_NGL=all +SPEC_DRAFT_P_MIN=0.75 +SPEC_DRAFT_NGL=999 SPEC_DRAFT_DEVICE=ROCm0 -SPEC_DRAFT_TYPE_K=q8_0 -SPEC_DRAFT_TYPE_V=q8_0 - -# Vendor runtime env knobs (read by the hipified mmq + PLE pager kernels). -# GGML_CUDA_Q41_MOE_FORCE_J forces the Q4_1 MoE WMMA tile width; the PLE -# pager knobs control the CIRUPLE1 page-in workers and skip re-hashing the -# 52 GB payload at every launch (checksums were verified once, at download). -GGML_CUDA_Q41_MOE_FORCE_J=32 -GGML_QWEN4EXP_PLE_WORKERS=16 -GGML_QWEN4EXP_PLE_STRICT_SHA=0 -ROCBLAS_USE_HIPBLASLT=1 +SPEC_DRAFT_TYPE_K=f16 +SPEC_DRAFT_TYPE_V=f16 \ No newline at end of file diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 774318e..50baeb3 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,28 +1,29 @@ #!/usr/bin/env bash -# Serve jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 as frank-default-large via the -# ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp runtime (tag v1.1) on -# Dankclaw (Strix Halo gfx1151, unified memory). +# Serve agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF as +# frank-default-large via the charlie12345/ROCmFPX llama.cpp runtime +# (build-qwen38, PR #27742 qwen4exp merged) on Dankclaw (Strix Halo +# gfx1151, unified memory). # -# Runtime: ciru-ai Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp built at -# /home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151 -# with the vendor build-linux-amd.sh flag set (HIP on, HIP_NO_VMM on, -# HIP_GRAPHS on, HIP_MMQ_MFMA on, CUDA/VULKAN/SYCL off, gfx1151) PLUS -# -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON. The Q4_1_MOE_FORCE_J and -# PLE env knobs below are read by the hipified mmq/pager kernels. +# Runtime: charlie12345/ROCmFPX llama.cpp built at +# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, gfx1151, ROCm 7.2.4). +# The ROCmFP4 quant types (Q4_0_ROCMFP4_FAST) are hand-ported from +# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF. # -# Model: Qwen3.8-Flash-Next base (56B MoE, 512 experts / 10 used, 48 layers -# = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE at layer 1) -# in the CIRU quality-first mixed quant: 144 routed-expert tensors at Q4_1 -# on the native IU4 WMMA path, protected tensors at Q5_K/Q5_1/Q8_0/BF16/F32. -# The exact-FP8 PLE table is a sidecar (ple/ dir) paged from disk through the -# CIRUPLE1 pager with a ple-cache-mib decoded-page cache; it never enters -# VRAM. The MTP draft head (Q8_0) drives speculative decoding at depth 3. +# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used, +# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE +# at layer 1) quantized Q4_0_ROCMFP4_FAST, imatrix-calibrated, 87.06 GiB. +# The n-gram PLE table is split per head inside the single GGUF — fully +# VRAM-resident, no sidecar, no --ngram-on-disk. The model's own MTP +# head (2.28 GiB, matched ROCmFP4-FAST quant) drives speculative decoding +# at depth 3 (~66% draft acceptance). # -# Vendor production profile is applied verbatim on the first deployment: -# -ngl all, -sm none, --fit off, ctx 262144 x 1 slot, b 2048 / ub 512, -# t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram -# + idle-slot cache, 32 ctx checkpoints with min step 8192, MTP depth 3 -# with p-split 0.10. Deviations from the vendor launcher are marked below. +# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0, +# presence 1.5 — non-thinking instruct) mixed with the validated +# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096, +# t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB +# cache-ram + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit +# segfaults on qwen4exp hyper-connection config). MTP depth 3 (n-max 3 +# measured best; adaptive p-split measured worse, left off). set -euo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" @@ -35,9 +36,10 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${LLAMA_BIN:=/home/dank/GitPackages/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4/build-gfx1151/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/ciru-Qwen3.8-Flash-CIRU-STRIX-IU4}" -: "${MODEL_FILE:=Qwen3.8-Flash-CIRU-STRIX-IU4.gguf}" +: "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}" +: "${MODEL_DIR:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST}" +: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf}" +: "${MMPROJ_PATH:=$MODEL_DIR/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}" : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" : "${CHAT_TEMPLATE_KWARGS:={}}" : "${ALIAS:=frank-default-large}" @@ -45,19 +47,18 @@ fi : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=262144}" : "${PARALLEL:=1}" -: "${BATCH_SIZE:=2048}" -: "${UBATCH_SIZE:=512}" -: "${NUM_THREADS:=8}" -: "${THREAD_BATCH:=8}" +: "${BATCH_SIZE:=4096}" +: "${UBATCH_SIZE:=4096}" +: "${NUM_THREADS:=16}" +: "${THREAD_BATCH:=16}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" -: "${CTX_CHECKPOINTS:=32}" -: "${CHECKPOINT_MIN_STEP:=8192}" +: "${CTX_CHECKPOINTS:=16}" : "${CACHE_RAM:=8192}" -: "${PLE_CACHE_MIB:=4096}" : "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}" : "${SPLIT_MODE:=none}" : "${FIT:=off}" +: "${NGL:=999}" : "${TEMPERATURE:=0.7}" : "${TOP_K:=20}" : "${TOP_P:=0.8}" @@ -66,20 +67,19 @@ fi : "${REASONING:=on}" : "${REASONING_FORMAT:=deepseek}" : "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}" -: "${SPEC_TYPE:=draft-mtp}" -: "${DRAFT_MODEL_DIR:=$MODEL_DIR/mtp}" -: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-CIRU-STRIX-IU4-MTP-Q8_0.gguf}" +: "${SPEC_TYPE:=draft-mtp,ngram-mod}" +: "${DRAFT_MODEL_DIR:=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST}" +: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf}" : "${SPEC_DRAFT_N_MAX:=3}" : "${SPEC_DRAFT_N_MIN:=0}" -: "${SPEC_DRAFT_P_MIN:=0.0}" -: "${SPEC_DRAFT_P_SPLIT:=0.10}" -: "${SPEC_DRAFT_NGL:=all}" +: "${SPEC_DRAFT_P_MIN:=0.75}" +: "${SPEC_DRAFT_NGL:=999}" : "${SPEC_DRAFT_DEVICE:=ROCm0}" -: "${SPEC_DRAFT_TYPE_K:=q8_0}" -: "${SPEC_DRAFT_TYPE_V:=q8_0}" +: "${SPEC_DRAFT_TYPE_K:=f16}" +: "${SPEC_DRAFT_TYPE_V:=f16}" MODEL="$MODEL_DIR/$MODEL_FILE" -for f in "$LLAMA_BIN" "$MODEL"; do +for f in "$LLAMA_BIN" "$MODEL" "$CHAT_TEMPLATE_FILE"; do if [[ ! -r "$f" ]]; then echo "missing or unreadable required artifact: $f" >&2 exit 1 @@ -121,8 +121,8 @@ if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then fi fi -# The CIRU package is text-only: no mmproj sidecar. Pass --mmproj only when -# MMPROJ_PATH points at a readable file (kept for future multimodal packs). +# The agentionai FAST package includes the model's own vision tower (f16, +# lossless). Pass --mmproj only when MMPROJ_PATH points at a readable file. MMPROJ_ARGS=() if [[ -n "${MMPROJ_PATH:-}" && -r "${MMPROJ_PATH:-}" ]]; then MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH") @@ -138,10 +138,6 @@ exec /usr/bin/env \ HIP_VISIBLE_DEVICES=0 \ LD_LIBRARY_PATH="$LIBRARY_PATH" \ HSA_OVERRIDE_GFX_VERSION="$HSA_OVERRIDE_GFX_VERSION" \ - GGML_CUDA_Q41_MOE_FORCE_J="${GGML_CUDA_Q41_MOE_FORCE_J:-32}" \ - GGML_QWEN4EXP_PLE_WORKERS="${GGML_QWEN4EXP_PLE_WORKERS:-16}" \ - GGML_QWEN4EXP_PLE_STRICT_SHA="${GGML_QWEN4EXP_PLE_STRICT_SHA:-0}" \ - ROCBLAS_USE_HIPBLASLT="${ROCBLAS_USE_HIPBLASLT:-1}" \ "$LLAMA_BIN" \ --model "$MODEL" \ --alias "$ALIAS" \ @@ -151,7 +147,7 @@ exec /usr/bin/env \ --port "$LISTEN_PORT" \ --jinja \ --device ROCm0 \ - --gpu-layers "${NGL:-all}" \ + --gpu-layers "$NGL" \ --split-mode "$SPLIT_MODE" \ --parallel "$PARALLEL" \ --ctx-size "$CTX_SIZE" \ @@ -164,10 +160,7 @@ exec /usr/bin/env \ --threads "$NUM_THREADS" \ --threads-batch "$THREAD_BATCH" \ --ctx-checkpoints "$CTX_CHECKPOINTS" \ - --checkpoint-min-step "$CHECKPOINT_MIN_STEP" \ --cache-ram "$CACHE_RAM" \ - --ple-sidecar "$MODEL_DIR/ple" \ - --ple-cache-mib "$PLE_CACHE_MIB" \ --slot-save-path "$SLOT_SAVE_PATH" \ --cache-prompt \ --cache-idle-slots \ @@ -181,4 +174,4 @@ exec /usr/bin/env \ --reasoning-format "$REASONING_FORMAT" \ --reasoning-budget -1 \ "${MMPROJ_ARGS[@]}" \ - "${SPEC_ARGS[@]}" + "${SPEC_ARGS[@]}" \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md new file mode 100644 index 0000000..b60b57d --- /dev/null +++ b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md @@ -0,0 +1,170 @@ +--- +type: reference +title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw +description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix, per-head PLE in-GGUF, own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-default-large. +tags: + - llama.cpp + - rocm + - dankclaw + - gfx1151 + - agentionai + - rocmfp4 + - strix-halo + - qwen4exp + - mtp +timestamp: 2026-08-29 +--- + +# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw + +> **Active since 2026-08-29.** Replaced the jcbtc CIRU v1.1 (ciru-ai runtime) +> deployment on the same slot. + +`frank-default-large` (port 7770) serves +`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main, +`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf`) with the author's +own MTP draft head (`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`) on the +charlie12345/ROCmFPX `build-qwen38` runtime. + +## Why this model + +- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory + box, including the 51.2 B-parameter n-gram (PLE) table, which is split per + head inside the single GGUF (`v2/` is the joined-table layout; root is the + VRAM-resident one and is what we run). +- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2 + (better than AesSedai IQ4_XS / IQ3_S at smaller size). +- 18-25% faster prompt processing than a mainline quant of similar size + (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth). +- Ships the model's own f16 vision tower (`mmproj/`) and its own MTP head + (~66% draft acceptance, mean accepted length 2.97 of 3 at n-max 3). +- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0 + draft measured worse on this head family (acceptance + throughput). + +## Runtime + +charlie12345/ROCmFPX `llama-server` built at +`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151, +PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4 +quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified: + +``` +strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST +``` + +CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point +this model at the CIRU runtime. The build previously served the kingjones +STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the +author's own matched MTP head on this binary. + +## Files / download + +| Item | Path on Dankclaw | Size | +|---|---|---| +| Main | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) | +| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B | +| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B | + +Draft head sha256 (verified against HF LFS): +`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590` + +Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main` +URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/` +subdirectory of the repo. + +## Serving profile + +Card sampling table (instruct, non-thinking) mixed with the validated +build-qwen38 profile: + +``` +llama-server \ + --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf \ + --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \ + --host 0.0.0.0 --port 7770 --jinja \ + --device ROCm0 --gpu-layers 999 \ + --split-mode none --fit off --flash-attn on \ + --ctx-size 262144 --parallel 1 \ + --batch-size 4096 --ubatch-size 4096 \ + --cache-type-k f16 --cache-type-v f16 \ + --threads 16 --threads-batch 16 \ + --ctx-checkpoints 16 --cache-ram 8192 \ + --cache-prompt --cache-idle-slots --metrics \ + --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \ + --reasoning on --reasoning-format deepseek --reasoning-budget -1 \ + --chat-template-file chat-template.jinja \ + --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \ + --spec-type draft-mtp,ngram-mod \ + --spec-draft-model Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf \ + --spec-draft-n-max 3 --spec-draft-n-min 0 --spec-draft-p-min 0.75 \ + --spec-draft-ngl 999 --spec-draft-device ROCm0 \ + --spec-draft-type-k f16 --spec-draft-type-v f16 +``` + +Key choices and deviations from the CIRU deployment: + +- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config + (same as every Flash-Next deployment). +- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it + (verified via `--help`); only `--ctx-checkpoints` is passed. +- **No PLE sidecar args** — the n-gram table is in-GGUF (per-head split), + not a `ple/` CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the + `GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU- + runtime-only and are gone. +- **`--mmproj` active** — vision tower included (f16, lossless). +- **MTP n-max 3, no p-split** — author measurements: n-max 3 = 32.4 t/s / + 0.612 acceptance, adaptive p-split measured worse (28.4 t/s / 0.468). + `p-min` stays at the 0.75 default. +- Context kept at 262144 × 1 slot (same as the CIRU deployment; the author's + quick-start is 32768 but the full window is fine on SVM). + +## MTP draft head + +The draft is agentionai's own head for this exact model: +`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, architecture +`qwen4exp` with `blk.48.hc_attn_norm.weight` present (count 1, same shape as +the CIRU draft that works). The earlier 2026-08-28 failure of this head on +the *kingjones* STRIX_LEAN main model (`missing tensor blk.0.hc_attn_norm`) +was a cross-vendor pairing problem, not a defect of the head itself. The +draft carries a 512-expert MoE, so adaptive/long drafting costs real forward +passes — keep n-max 3. + +## Config files + +- Env: `scripts/frank-default-large.env` +- Serve: `scripts/serve-frank-large.sh` +- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp) + +## Swap procedure + +```bash +sudo systemctl stop frank-default-large +# edit scripts/frank-default-large.env + scripts/serve-frank-large.sh, commit, push +# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-default-large +sudo systemctl start frank-default-large # or via deploy.sh +curl http://192.168.0.143:7770/health +``` + +Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while +loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it. + +## Known issues + +1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU + runtime reject these files. +2. **Single 87 GiB file.** Use resumable curl; allow ~10-20 minutes on a + good link. +3. **MTP head must be agentionai's.** EasiiX Q8_0 and other heads are + incompatible with this binary's draft expectations for Flash-Next + (hyper-connection norm tensor). +4. **`SVM mapping failed` kernel spam** is a documented symptom, not root + cause — investigate before blaming memory pressure. + +## Citations + +- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF) +- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF) +- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) +- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types +- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream) +- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 43c9a6f..698f5ed 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,10 +1,10 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small, +(2026-08-29): FreeToken v0.1.2 on Dankserver (frank-default-small, Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw -(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both -services run the Qwen-Sharp chat template. +(frank-default-large, agentionai Qwen3.8-Flash-Next ROCmFP4-FAST +imatrix + MTP draft). Both services run the Qwen-Sharp chat template. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -26,10 +26,10 @@ block-fp8, 20% MoE residency, 453k KV / 4 streams] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] E2[gfx1151 iGPU 64 GiB unified memory] - E[charlie12345/ROCmFPX build-kairic (v81) + E[charlie12345/ROCmFPX build-qwen38 frank-default-large, port 7770] - F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT -MTP draft + Turbo4 V cache, 262K ctx / 1 slot] + F[agentionai Qwen3.8-Flash-Next +ROCmFP4-FAST imatrix + MTP draft, 262K ctx / 1 slot] E2 --> E --> F end @@ -53,7 +53,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot] (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[kingjones-strix-rocmfp4]] — active frank-default-large runbook: +- [[agentionai-fast-imatrix]] — ACTIVE frank-default-large runbook: + agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB + Q4_0_ROCMFP4_FAST, per-head PLE in GGUF) + own MTP draft head + (ROCmFP4-FAST, 2.28 GiB), build-qwen38 runtime (2026-08-29) +- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index d2683be..716b15c 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -111,7 +111,10 @@ The LLM inference fleet on the two hosts. Full index: - [[freetoken-frank-default-small]] — the active frank-default-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02) - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01) -- [[kingjones-strix-rocmfp4]] — the active frank-default-large runbook: +- [[agentionai-fast-imatrix]] — the ACTIVE frank-default-large runbook: + agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, + Q4_0_ROCMFP4_FAST) + own MTP draft head, build-qwen38 (2026-08-29) +- [[kingjones-strix-rocmfp4]] — retired frank-default-large runbook: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params, 113.47 GiB) - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw