feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8
Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache and paged KV with radix prefix sharing. Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size). Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used. Production config: - 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM) - 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers) - 4 max-running-requests (mamba slot pool: 24 slots) - bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1 - Prefill: ~1,130 tok/s at 4k prompt Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed). Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart. Old llama.cpp scripts archived to .AGENTTRASH/. Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in docs/LLM/freetoken-benchmark-sweep.md.
This commit is contained in:
Frank
@@ -1,60 +0,0 @@
|
||||
# Tunables for frank-default-large (upstream llama.cpp PR #27342 DFlash2,
|
||||
# Vulkan backend, dankclaw, Strix Halo gfx1151).
|
||||
#
|
||||
# Runtime: llama.cpp PR #27342 built with GGML_VULKAN=ON serving
|
||||
# unsloth/Qwen3.8-27B-UD-Q8_K_XL (target) with a z-lab DFlash2 drafter
|
||||
# (Qwen3.8-27B-DFlash2-Q4_K_M). Target at 31.4 GB, DFlash2 drafter at 2.0 GB.
|
||||
# Alias frank-default-large, port 52625.
|
||||
#
|
||||
# DFlash2 runs on the Vulkan (RADV) backend. Its candidate selector needs
|
||||
# TOP_K + ARGMAX, which the HIP backend does not implement. This replaced
|
||||
# the CIRU HIP runtime on 2026-08-21. See docs/dflash2-large-vulkan.md.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). This deployment uses Vulkan, not HIP, but the rule stays for
|
||||
# portability.
|
||||
|
||||
# Upstream llama.cpp DFlash2 binary (PR #27342, Vulkan build).
|
||||
LLAMA_BIN=/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server
|
||||
|
||||
# Model files. The target (UD-Q8_K_XL) was already on disk; the DFlash2
|
||||
# drafter was added for this deployment. Both live in /home/dank/models.
|
||||
MODEL_DIR=/home/dank/models
|
||||
MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf
|
||||
DRAFT_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
|
||||
|
||||
ALIAS=frank-default-large
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=52625
|
||||
|
||||
# Serving: full native 262144 context, 1 slot. The Vulkan device reports
|
||||
# 113942 MiB; KV at 262144 x 64 KiB/cell is ~16 GiB. rocm-smi reports the
|
||||
# 64 GiB budget; trust the runtime for the addressable SVM pool.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=4096
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
CTX_CHECKPOINTS=16
|
||||
CACHE_RAM=8192
|
||||
|
||||
# DFlash2 speculative decoding. The drafter's trained block size is 8;
|
||||
# SPEC_DRAFT_N_MAX clamps to it. TOP_K selector (top 16 candidates).
|
||||
SPEC_TYPE=draft-dflash
|
||||
SPEC_DRAFT_N_MAX=8
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.0
|
||||
SPEC_DRAFT_P_SPLIT=0.10
|
||||
|
||||
# Sampling (model card defaults, thinking-general).
|
||||
TEMPERATURE=0.7
|
||||
TOP_P=0.95
|
||||
|
||||
# Thinking enabled. deepseek format moves the thinking reasoning into
|
||||
# message.reasoning_content (OpenAI-compatible).
|
||||
REASONING=on
|
||||
REASONING_FORMAT=deepseek
|
||||
|
||||
VERBOSITY=4
|
||||
@@ -0,0 +1,71 @@
|
||||
# Tunables for frank-default-small (Tiel-Coder-35B-A3B UD-Q6_K_XL,
|
||||
# qwen35moe hybrid MoE, Dankserver, RTX 3090).
|
||||
#
|
||||
# Runtime: upstream llama.cpp (latest master, CUDA build with
|
||||
# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86) on RTX 3090
|
||||
# 24 GB; Ryzen 7 7700X; 62 GB RAM. Replaced the ik_llama.cpp Ornith
|
||||
# CRACK Q8_0 deployment on 2026-09-01.
|
||||
#
|
||||
# Model: peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF, UD-Q6_K_XL quant
|
||||
# (30 GB). Architecture: qwen35moe — 40 layers, full_attention_interval=4
|
||||
# (10 attention + 30 SSM/Mamba), 256 experts / 8 used, 2 KV heads × 256
|
||||
# dim, embedding 2048, rope dimension_count 64, rope freq_base 10 000 000.
|
||||
# Text-only (no mmproj). Built-in chat template in the GGUF.
|
||||
#
|
||||
# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
|
||||
# Non-expert weights (~1.6 GB) fit entirely on the 24 GB card.
|
||||
#
|
||||
# KV cache: Q8_0 for both K and V. With CTX_SIZE=200700 (66 900 per slot
|
||||
# × 3 parallel) and 10 attention layers, the KV cache is ~2 GB — the
|
||||
# requested 2 GB VRAM buffer.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
|
||||
|
||||
# Model file (UD-Q6_K_XL, 30 GB, text-only MoE hybrid).
|
||||
MODEL_DIR=/home/dank/models/tiel-coder-35b
|
||||
MODEL_FILE=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf
|
||||
|
||||
# Upstream llama.cpp llama-server (CUDA build, latest master).
|
||||
# Built with -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86
|
||||
# using the staged CUDA 13.2 toolkit at /home/dank/cuda13-stage.
|
||||
LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
|
||||
|
||||
ALIAS=frank-default-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Total context budget across all parallel slots. 200700 / 3 = 66 900
|
||||
# per slot. The model's context_length is 262 144, so 66 900 is well
|
||||
# within range.
|
||||
CTX_SIZE=200700
|
||||
PARALLEL=3
|
||||
|
||||
# Expert offload: all MoE expert weights pinned to CPU RAM.
|
||||
CPU_MOE=1
|
||||
|
||||
# All non-expert layers on GPU (auto-fit will determine the exact count;
|
||||
# with --cpu-moe the non-expert footprint is ~1.6 GB, leaving ample room).
|
||||
N_GPU_LAYERS=999
|
||||
|
||||
# KV cache: Q8_0 for both K and V. ~2 GB total for 200700 ctx × 3 slots
|
||||
# × 10 attention layers × (2 KV heads × 256 dim × 1 byte) × 2 (K+V).
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=q8_0
|
||||
|
||||
VERBOSITY=2
|
||||
|
||||
# Prefill batch sizing.
|
||||
BATCH_SIZE=4096
|
||||
UBATCH_SIZE=1024
|
||||
|
||||
# Sampling.
|
||||
TEMPERATURE=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
MIN_P=0.0
|
||||
|
||||
# Compute threads lowered so other processes keep a core.
|
||||
THREADS=7
|
||||
THREADS_HTTP=8
|
||||
@@ -0,0 +1,16 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Tiel-Coder-35B-A3B Q6_K_XL qwen35moe hybrid MoE (dankserver, frank-default-small)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-default-small.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -1,92 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Seam: frank-default-large model, ROCmFPX runtime, MTP, KV cache, template.
|
||||
#
|
||||
# load helpers
|
||||
|
||||
load helpers
|
||||
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-default-large.env"
|
||||
SERVE_FILE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
UNIT_FILE="$REPO_ROOT/scripts/frank-default-large.service"
|
||||
TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
|
||||
|
||||
@test "frank-default-large serves the kingjones AGENT model" {
|
||||
grep -q '^MODEL_FILE=Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE"
|
||||
grep -q '^MTP_DRAFT_FILE=' "$ENV_FILE"
|
||||
grep -q '^MMPROJ_FILE=mmproj-Qwen3\.8-27B-Q8_0\.gguf$' "$ENV_FILE"
|
||||
! grep -q -- '--kairic-edge' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large uses the ROCmFPX runtime" {
|
||||
grep -q 'ROCmFPX/.*/bin/llama-server' "$ENV_FILE"
|
||||
grep -q 'build-promptforge/bin/llama-server' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large runs standalone MTP" {
|
||||
grep -q -- '--spec-type draft-mtp' "$SERVE_FILE"
|
||||
grep -qF -- '--model-draft "$MTP_DRAFT_FILE"' "$SERVE_FILE"
|
||||
grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE"
|
||||
grep -q '^MTP_N_MAX=4$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large has no obsolete PromptForge sidecar wiring" {
|
||||
! grep -qE 'KAIRIC_|PROMPTFORGE_' "$SERVE_FILE"
|
||||
! grep -qE 'KAIRIC_|PROMPTFORGE_' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large fits the model and KV cache in unified memory" {
|
||||
grep -q '^CTX_SIZE=262144$' "$ENV_FILE"
|
||||
grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large uses DeepSeek reasoning output" {
|
||||
grep -q '^REASONING_FORMAT=deepseek$' "$ENV_FILE"
|
||||
grep -q -- '--reasoning-format "$REASONING_FORMAT"' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large has reasoning and preserved reasoning/thinking enabled" {
|
||||
grep -q '^REASONING=on$' "$ENV_FILE"
|
||||
grep -q 'enable_thinking":true' "$ENV_FILE"
|
||||
grep -q 'preserve_thinking":true' "$ENV_FILE"
|
||||
grep -q 'preserve_reasoning":true' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large uses f16 for both KV cache sides" {
|
||||
grep -q '^CACHE_TYPE_K=f16$' "$ENV_FILE"
|
||||
grep -q '^CACHE_TYPE_V=f16$' "$ENV_FILE"
|
||||
grep -qF -- '--cache-type-k "$CACHE_TYPE_K"' "$SERVE_FILE"
|
||||
grep -qF -- '--cache-type-v "$CACHE_TYPE_V"' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large exposes a swappable ubatch" {
|
||||
grep -q '^UBATCH_SIZE=4096$' "$ENV_FILE"
|
||||
grep -q ': "${UBATCH_SIZE:=4096}"' "$SERVE_FILE"
|
||||
grep -q -- '--ubatch-size "$UBATCH_SIZE"' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "llama-bench ubatch harness uses the active model" {
|
||||
local bench="$REPO_ROOT/scripts/bench-ubatch.sh"
|
||||
[ -x "$bench" ]
|
||||
grep -q 'llama-bench' "$bench"
|
||||
grep -q 'Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf' "$bench"
|
||||
grep -q -- '--n-prompt "$CONTEXT"' "$bench"
|
||||
grep -q -- '--ubatch-size "$UBATCH_VALUES"' "$bench"
|
||||
grep -q -- '--batch-size "$UBATCH_VALUES"' "$bench"
|
||||
grep -q -- '--mmap 0' "$bench"
|
||||
! grep -qE '(^|[[:space:]])llama-server|(^|[[:space:]])curl|print_timing' "$bench"
|
||||
}
|
||||
|
||||
@test "frank-default-large uses the tracked Qwen-Sharp template on :7770" {
|
||||
grep -q '^CHAT_TEMPLATE_FILE=' "$ENV_FILE"
|
||||
grep -q -- '--jinja' "$SERVE_FILE"
|
||||
[ -s "$TEMPLATE_FILE" ]
|
||||
grep -q '^ALIAS=frank-default-large$' "$ENV_FILE"
|
||||
grep -q '^LISTEN_PORT=7770$' "$ENV_FILE"
|
||||
grep -q 'frank-default-large' "$UNIT_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large env avoids the HIP TEMP/HOST collision" {
|
||||
# The 2026-08-13 incident: TEMP/HOST/GPU_MAX_HEAP_SIZE are HIP runtime vars.
|
||||
! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
|
||||
grep -q '^TEMPERATURE=' "$ENV_FILE"
|
||||
}
|
||||
@@ -1,88 +0,0 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: frank-default-small model, context, tensor placement, and template.
|
||||
# This test covers only the Q6_K deployment contract.
|
||||
|
||||
load helpers
|
||||
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-default-small.env"
|
||||
SERVE_FILE="$REPO_ROOT/scripts/serve-frank-small.sh"
|
||||
UNIT_FILE="$REPO_ROOT/scripts/frank-default-small.service"
|
||||
TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
|
||||
|
||||
@test "frank-default-small selects the Ornith Q8_0 model" {
|
||||
grep -q '^MODEL_FILE=Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf$' "$ENV_FILE"
|
||||
grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small uses three 163840-slot context (491520 pool)" {
|
||||
grep -q '^CTX_SIZE=491520$' "$ENV_FILE"
|
||||
grep -q '^PARALLEL=3$' "$ENV_FILE"
|
||||
grep -q '163840 per slot' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small uses default f16 K and q6_0 V KV cache" {
|
||||
grep -q '^CACHE_TYPE_K=f16$' "$ENV_FILE"
|
||||
grep -q '^CACHE_TYPE_V=q6_0$' "$ENV_FILE"
|
||||
# K at default f16 has no Hadamard; V at q6_0 keeps Hadamard.
|
||||
grep -q '^K_CACHE_HADAMARD=0$' "$ENV_FILE"
|
||||
grep -q '^V_CACHE_HADAMARD=1$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small keeps full-attention and final blocks on CUDA" {
|
||||
source "$ENV_FILE"
|
||||
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
|
||||
# output-adjacent final blocks stay on CUDA
|
||||
[[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|37|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|38|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|39|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
|
||||
# full-attention blocks (3,7,11,15,19,23) are untouched (no exps)
|
||||
[[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|15|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|19|'* ]]
|
||||
[[ "$OVERRIDE_EXPERT" != *'|23|'* ]]
|
||||
[[ "$OVERRIDE_BUFT" == CPU ]]
|
||||
# ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
|
||||
# loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
|
||||
[[ "$N_GPU_LAYERS" == 999 ]]
|
||||
}
|
||||
|
||||
@test "frank-default-small disables speculative decoding (3 parallel slots)" {
|
||||
# ik_llama limits speculative decoding to -np 1; with PARALLEL=3 a
|
||||
# configured spec stage hard-exits, so SPEC_TYPE=none.
|
||||
grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
|
||||
grep -q '^PARALLEL=3$' "$ENV_FILE"
|
||||
! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
|
||||
! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
|
||||
! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
|
||||
! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
|
||||
! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
|
||||
grep -q -- '--spec-type' "$SERVE_FILE"
|
||||
! grep -q -- 'ngram' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small uses ubatch 1024" {
|
||||
grep -q '^BATCH_SIZE=4096$' "$ENV_FILE"
|
||||
grep -q '^UBATCH_SIZE=1024$' "$ENV_FILE"
|
||||
grep -q ': "${UBATCH_SIZE:=1024}"' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small uses the tracked Qwen-Sharp template" {
|
||||
grep -q '^CHAT_TEMPLATE_FILE=' "$ENV_FILE"
|
||||
grep -q 'peculiar-ragdoll/Qwen-Sharp-Chat-Templates' "$ENV_FILE"
|
||||
grep -q 'chat-template\.jinja' "$SERVE_FILE"
|
||||
[ -s "$TEMPLATE_FILE" ]
|
||||
grep -q 'qwen3\.8-froggeric-v22\.4\.0' "$TEMPLATE_FILE"
|
||||
grep -q 'Qwen-Sharp' "$UNIT_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-small renders the custom template with --jinja" {
|
||||
# Without --jinja, ik_llama only supports built-in non-Jinja handlers and
|
||||
# silently drops the custom Qwen-Sharp template formatting.
|
||||
grep -q '^USE_JINJA=1$' "$ENV_FILE"
|
||||
grep -q -- '--jinja' "$SERVE_FILE"
|
||||
}
|
||||
@@ -1,104 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve unsloth/Qwen3.8-27B-UD-Q8_K_XL as frank-default-large with a
|
||||
# z-lab DFlash2 drafter (Qwen3.8-27B-DFlash2-Q4_K_M) on the Vulkan backend.
|
||||
#
|
||||
# Runtime: upstream llama.cpp PR #27342 (DFlash2 support) built with
|
||||
# GGML_VULKAN=ON for the Strix Halo gfx1151 (RADV). DFlash2 selects its
|
||||
# candidates through TOP_K + ARGMAX ops that the HIP backend does not
|
||||
# implement, so the whole stack runs on Vulkan0.
|
||||
#
|
||||
# The target and the drafter share the Qwen3.8 arch. DFlash is lossless
|
||||
# under strict rejection sampling. --spec-draft-n-max clamps to the drafter's
|
||||
# trained block size (8).
|
||||
#
|
||||
# Tunables come from frank-default-large.env in this directory unless
|
||||
# FRANK_LARGE_ENV_FILE overrides that path.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -------------------
|
||||
: "${LLAMA_BIN:=/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
|
||||
: "${DRAFT_FILE:=Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=52625}"
|
||||
: "${CTX_SIZE:=262144}"
|
||||
: "${PARALLEL:=1}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=4096}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CTX_CHECKPOINTS:=16}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
: "${SPEC_TYPE:=draft-dflash}"
|
||||
: "${SPEC_DRAFT_N_MAX:=8}"
|
||||
: "${SPEC_DRAFT_N_MIN:=0}"
|
||||
: "${SPEC_DRAFT_P_MIN:=0.0}"
|
||||
: "${SPEC_DRAFT_P_SPLIT:=0.10}"
|
||||
: "${TEMPERATURE:=0.7}"
|
||||
: "${TOP_P:=0.95}"
|
||||
: "${REASONING:=on}"
|
||||
: "${REASONING_FORMAT:=deepseek}"
|
||||
: "${VERBOSITY:=4}"
|
||||
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
DRAFT="$MODEL_DIR/$DRAFT_FILE"
|
||||
|
||||
if [[ ! -f "$MODEL" ]]; then
|
||||
echo "missing required file: $MODEL" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -f "$DRAFT" ]]; then
|
||||
echo "missing DFlash2 drafter: $DRAFT" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -x "$LLAMA_BIN" ]]; then
|
||||
echo "missing llama-server binary: $LLAMA_BIN" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BIN" \
|
||||
--model "$MODEL" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT" \
|
||||
-dev Vulkan0 \
|
||||
-ngl 999 \
|
||||
-c "$CTX_SIZE" \
|
||||
-b "$BATCH_SIZE" \
|
||||
-ub "$UBATCH_SIZE" \
|
||||
-fa on \
|
||||
-ctk "$CACHE_TYPE_K" \
|
||||
-ctv "$CACHE_TYPE_V" \
|
||||
-np "$PARALLEL" \
|
||||
-ctxcp "$CTX_CHECKPOINTS" \
|
||||
--cache-ram "$CACHE_RAM" \
|
||||
--cache-prompt \
|
||||
--metrics \
|
||||
--no-mmap \
|
||||
--spec-type "$SPEC_TYPE" \
|
||||
--spec-draft-model "$DRAFT" \
|
||||
--spec-draft-device Vulkan0 \
|
||||
--spec-draft-ngl 999 \
|
||||
--spec-draft-type-k f16 \
|
||||
--spec-draft-type-v f16 \
|
||||
--spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
|
||||
--spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
|
||||
--spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
|
||||
--spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \
|
||||
--temp "$TEMPERATURE" \
|
||||
--top-p "$TOP_P" \
|
||||
--reasoning "$REASONING" \
|
||||
--reasoning-format "$REASONING_FORMAT" \
|
||||
--reasoning-budget -1 \
|
||||
-lv "$VERBOSITY"
|
||||
Executable
+87
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve peculiar-ragdoll/Tiel-Coder-35B-A3B (UD-Q6_K_XL) as
|
||||
# frank-default-small via stock llama.cpp on Dankserver (RTX 3090).
|
||||
#
|
||||
# Runtime: upstream llama.cpp (latest master, CUDA build with
|
||||
# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86).
|
||||
# The model is a 35B-total / 8-active MoE hybrid (qwen35moe arch) with
|
||||
# SSM/Mamba layers (full_attention_interval=4: 10 attention + 30 SSM
|
||||
# layers out of 40). 256 experts, 8 used per token.
|
||||
#
|
||||
# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
|
||||
# Non-expert weights (attention, SSM, shared expert, embeddings, output)
|
||||
# are ~1.6 GB and fit entirely on the 24 GB RTX 3090.
|
||||
#
|
||||
# KV cache: Q8_0 for both K and V. With CTX_SIZE=200700 (66 900 per
|
||||
# slot × 3 parallel) and 10 attention layers (2 KV heads × 256 dim),
|
||||
# the KV cache is ~2 GB — the requested 2 GB VRAM buffer.
|
||||
#
|
||||
# Model is loaded with mmap (default). The 30 GB Q6_K_XL file is
|
||||
# memory-mapped; only accessed pages enter RAM.
|
||||
#
|
||||
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
|
||||
# overrides that path.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-default-small.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- defaults (used when the env file does not set them) -----------------
|
||||
: "${MODEL_DIR:=/home/dank/models/tiel-coder-35b}"
|
||||
: "${MODEL_FILE:=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf}"
|
||||
: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
|
||||
: "${ALIAS:=frank-default-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=200700}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
: "${N_GPU_LAYERS:=999}"
|
||||
: "${BATCH_SIZE:=4096}"
|
||||
: "${UBATCH_SIZE:=1024}"
|
||||
: "${VERBOSITY:=2}"
|
||||
: "${CPU_MOE:=1}"
|
||||
: "${TEMPERATURE:=1.0}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.95}"
|
||||
: "${MIN_P:=0.0}"
|
||||
: "${THREADS:=7}"
|
||||
: "${THREADS_HTTP:=8}"
|
||||
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
|
||||
if [[ ! -f "$MODEL" ]]; then
|
||||
echo "missing required file: $MODEL" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -x "$LLAMA_SERVER" ]]; then
|
||||
echo "missing llama-server binary: $LLAMA_SERVER" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec "$LLAMA_SERVER" \
|
||||
--model "$MODEL" \
|
||||
$([ "$CPU_MOE" = "1" ] && printf -- "--cpu-moe") \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
--gpu-layers "$N_GPU_LAYERS" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--temp "$TEMPERATURE" \
|
||||
--top-k "$TOP_K" \
|
||||
--top-p "$TOP_P" \
|
||||
--min-p "$MIN_P" \
|
||||
--threads "$THREADS" \
|
||||
--threads-http "$THREADS_HTTP" \
|
||||
--alias "$ALIAS" \
|
||||
--host "$BIND_ADDR" \
|
||||
--port "$LISTEN_PORT"
|
||||
Reference in New Issue
Block a user