Frank
3c2a5d2fe7
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF — fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST (2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3). Vision tower (f16) loaded via mmproj/. Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) — the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1 deployment (ciru-ai runtime). Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0, presence 1.5) mixed with the validated build-qwen38 serving profile: 262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on, cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx checkpoints (no --checkpoint-min-step — unsupported on this binary), --fit off (auto-fit segfaults on qwen4exp hyper-connection config), --spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured worse per author card), p-min 0.75. Qwen-Sharp chat template kept; reasoning stays in DeepSeek format. The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN model was a cross-vendor pairing problem: the head carries blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
112 lines
4.8 KiB
Bash
112 lines
4.8 KiB
Bash
# Tunables for frank-default-large (agentionai/Qwen3.8-Flash-Next-ROCmFP4-
|
|
# FAST-imatrix-GGUF on the charlie12345/ROCmFPX llama.cpp runtime,
|
|
# build-qwen38, Dankclaw, Strix Halo gfx1151 unified memory).
|
|
#
|
|
# Runtime: charlie12345/ROCmFPX llama.cpp at
|
|
# /home/dank/GitPackages/ROCmFPX/build-qwen38 (HIP, ROCm 7.2.4, gfx1151,
|
|
# PR #27742 qwen4exp merged). The ROCmFP4 quant types are hand-ported from
|
|
# ciru-ai/ROCmFPX; upstream llama.cpp cannot load this GGUF. Replaced the
|
|
# jcbtc CIRU v1.1 (ciru-ai runtime) and kingjones STRIX_LEAN deployments.
|
|
#
|
|
# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
|
|
# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
|
|
# at layer 1) quantized Q4_0_ROCMFP4_FAST imatrix-calibrated, 87.06 GiB,
|
|
# 4.23 bpw. The n-gram PLE table is split per head inside the single GGUF
|
|
# (fully VRAM-resident — no ple sidecar, no --ngram-on-disk). Vision tower
|
|
# (f16, lossless) ships as mmproj/. The model's own MTP head (2.28 GiB,
|
|
# matched ROCmFP4-FAST quant, ~66% draft acceptance) drives spec decoding.
|
|
#
|
|
# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
|
|
# presence 1.5 — non-thinking instruct) mixed with the validated
|
|
# build-qwen38 serving profile: ctx 262144 x 1 slot, b 4096 / ub 4096,
|
|
# t 16 / tb 16 (threads-batch 16 measured best for prefill), f16 KV, fa on,
|
|
# cont-batching, prompt cache + 8 GiB cache-ram + idle-slot cache, 16 ctx
|
|
# checkpoints, --fit off (auto-fit segfaults on qwen4exp
|
|
# hyper-connection/indexer config). MTP depth 3 (n-max 3 measured best on
|
|
# this head; adaptive p-split measured worse, left off). The Qwen-Sharp
|
|
# chat template is kept on purpose (chat-template.jinja): the service keeps
|
|
# emitting reasoning in DeepSeek format for its clients.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
|
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
|
# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
|
|
# env vars.
|
|
|
|
# ROCmFPX (build-qwen38) runtime binary.
|
|
LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
|
|
|
|
# Model package from
|
|
# https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF
|
|
# Layout: single main GGUF (per-head PLE table, fully VRAM-resident) +
|
|
# mmproj/ vision tower. The MTP draft head lives in
|
|
# /home/dank/models/mtp-Qwen3.8-Flash-Next-FAST (already on the host).
|
|
MODEL_DIR=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST
|
|
MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf
|
|
MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf
|
|
|
|
# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
|
|
# Sharp template drives thinking on by default (medium effort); the server
|
|
# emits reasoning in DeepSeek format so clients receive reasoning_content
|
|
# separately instead of literal thinking tags in content.
|
|
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
|
|
|
|
# Alias + network.
|
|
ALIAS=frank-default-large
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
|
|
# Serving: 262144-token context on ONE slot (parallel 1), batch 4096 /
|
|
# ubatch 4096, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB
|
|
# prompt cache. --fit off: auto-fit segfaults on this model's
|
|
# hyper-connection/indexer config; -ngl 999 puts every layer on the
|
|
# SVM-backed GPU pool.
|
|
CTX_SIZE=262144
|
|
PARALLEL=1
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=4096
|
|
NUM_THREADS=16
|
|
THREAD_BATCH=16
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
CTX_CHECKPOINTS=16
|
|
CACHE_RAM=8192
|
|
SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
|
|
SPLIT_MODE=none
|
|
FIT=off
|
|
NGL=999
|
|
|
|
# Sampling (card instruct table, non-thinking).
|
|
TEMPERATURE=0.7
|
|
TOP_K=20
|
|
TOP_P=0.8
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=1.5
|
|
|
|
# Reasoning on (Sharp template drives it; preserve_thinking + preserve_reasoning
|
|
# keep the reasoning trail in the conversation). DeepSeek format projects the
|
|
# reasoning into the API's reasoning_content field instead of literal thinking
|
|
# tags.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
|
|
# Strix Halo unified memory + native gfx1151.
|
|
HSA_OVERRIDE_GFX_VERSION=11.5.1
|
|
|
|
VERBOSITY=4
|
|
|
|
# Speculative decoding: the model's own MTP head (agentionai
|
|
# Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF, 2.28 GiB, ROCmFP4-FAST quant
|
|
# matched to the target) at depth 3 stacked on ngram-mod. n-max 3 measured
|
|
# best (32.4 t/s, 0.612 acceptance); adaptive p-split measured worse, so it
|
|
# stays off (p-min stays at the 0.75 default). Set SPEC_TYPE=none to disable.
|
|
DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
|
|
DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf
|
|
SPEC_TYPE=draft-mtp,ngram-mod
|
|
SPEC_DRAFT_N_MAX=3
|
|
SPEC_DRAFT_N_MIN=0
|
|
SPEC_DRAFT_P_MIN=0.75
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm0
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16 |