Frank
fc75ca6ed8
Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model). - serve-frank-large.sh: SPEC_ARGS array, draft model existence check, spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0) - frank-default-large.env: MTP tunables section, fix 'no MTP' comment - kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in serving profile, EasiiX + PR #27739 citations Set SPEC_TYPE=none in the env to disable speculative decoding.
102 lines
4.2 KiB
Bash
102 lines
4.2 KiB
Bash
# Tunables for frank-default-large (kingjones777 Qwen3.8-Flash-Next
|
|
# ROCmFP4 STRIX, qwen4exp hybrid MoE, Dankclaw, Strix Halo gfx1151).
|
|
#
|
|
# Runtime: ROCmFPX llama.cpp fork with PR #27742 (qwen4exp architecture)
|
|
# merged and built at /home/dank/GitPackages/ROCmFPX/build-qwen38.
|
|
# Build flags: -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON
|
|
# -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
|
# -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie.
|
|
# Replaced the kingjones Qwen3.8-27B Q6_0 AGENT deployment on 2026-09-01.
|
|
# Replaced the STRIX_LEAN deployment on 2026-08-27.
|
|
#
|
|
# Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
|
|
# 177B params (51.2B PLE n-gram table + 125.8B total MoE/attention).
|
|
# Qwen3.8-Flash-Next base: 56B MoE (512 experts, 10 used), 48 layers
|
|
# (12 attention + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim,
|
|
# embedding 2560, QSA indexer (4 heads, top_k=2048), hyper-connections
|
|
# (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8).
|
|
# Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file.
|
|
# PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM).
|
|
# Text-only (no mmproj). MTP draft head loaded as a separate GGUF sidecar
|
|
# (EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF, Q8_0, 4.1 GB).
|
|
#
|
|
# Context: 262144 (256k) x 2 slots. f16 KV cache (~6.4 GiB per slot,
|
|
# ~12.8 GiB total). --fit off because the auto-fit code segfaults on this
|
|
# model's hyper-connection/indexer config; -ngl 999 places all layers on
|
|
# the SVM-backed GPU pool (116 GiB unified).
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
|
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
|
# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
|
|
# env vars.
|
|
|
|
# ROCmFPX llama-server (PR #27742 merged, build-qwen38).
|
|
LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
|
|
|
|
# Model files from
|
|
# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF
|
|
MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
|
|
MODEL_FILE=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf
|
|
|
|
# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
|
|
# Sharp template drives thinking on by default (medium effort); the server
|
|
# emits reasoning in DeepSeek format so clients receive reasoning_content
|
|
# separately instead of literal thinking tags in content.
|
|
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
|
|
|
|
# Alias + network.
|
|
ALIAS=frank-default-large
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
|
|
# Serving: f16 KV, context checkpoints, prompt cache. The model is 113.47
|
|
# GiB loaded via mmap into the Strix Halo unified memory pool (116 GiB
|
|
# SVM-backed). PLE table (50.66 GiB Q8_0) is streamed from page cache; never
|
|
# enters GPU VRAM. GPU resident: ~63-69 GiB. --fit off avoids the auto-fit
|
|
# segfault on this model's qwen4exp hyper-connection/indexer configuration.
|
|
CTX_SIZE=524288
|
|
PARALLEL=2
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=4096
|
|
NUM_THREADS=16
|
|
THREAD_BATCH=16
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
CTX_CHECKPOINTS=16
|
|
CACHE_RAM=8192
|
|
FIT=off
|
|
|
|
# Sampling (Sharp default).
|
|
TEMPERATURE=0.7
|
|
TOP_K=20
|
|
TOP_P=0.8
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=1.5
|
|
|
|
# Reasoning on (Sharp template drives it; preserve_thinking + preserve_reasoning
|
|
# keep the reasoning trail in the conversation). DeepSeek format projects the
|
|
# reasoning into the API's reasoning_content field instead of content tags.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
|
|
# Strix Halo unified memory + native gfx1151.
|
|
HSA_OVERRIDE_GFX_VERSION=11.5.1
|
|
|
|
VERBOSITY=4
|
|
|
|
# MTP draft head (speculative decoding). The MTP head from
|
|
# Qwen/Qwen3.8-Flash-Next, exported as a standalone Q8_0 GGUF sidecar by
|
|
# EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF. Architecture qwen4exp,
|
|
# matching the main model. Measured +50% decode on code at temp 0 on
|
|
# Strix Halo (gfx1151). Set SPEC_TYPE=none to disable.
|
|
DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next
|
|
DRAFT_MODEL_FILE=mtp-Qwen3.8-Flash-Next-Q8_0.gguf
|
|
SPEC_TYPE=draft-mtp,ngram-mod
|
|
SPEC_DRAFT_N_MAX=4
|
|
SPEC_DRAFT_N_MIN=0
|
|
SPEC_DRAFT_P_MIN=0.75
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm0
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16 |