dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: Q8_0 model, 160k single slot, q6_0 KV + Hadamard

Browse Source 
Switch to CRACK Q8_0 model (37.8 GB, mmproj f16). Servt as a single
160000-ctx slot (PARALLEL=1); speculative decoding stays disabled. KV
cache is q6_0 + Hadamard. iq5_k is not a valid KV-cache type in ik_llama
(cache whitelist excludes iq5_*); q8_KV reproduces the ggml.c:5427 view
assert even at PARALLEL=1/no-spec, so q6_0 is the verified working 6-bit
choice. Env, serve script, tests, and KB updated. Deployment was left
down after the isolation test; this commit redeploys it with the
verified Q8_0/160k/q6_0 config.
This commit is contained in:
Frank
2026-08-26 05:01:58 -07:00
parent 23e905c712
commit 8c41c821c4
4 changed files with 67 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+18 -25
View File
@@ -1,27 +1,27 @@
# Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q6_K, MTP MoE,
# Tunables for frank-default-small (Ornith-1.5-35B-A3B CRACK Q8_0, MTP MoE,
# Dankserver, RTX 3090).
#
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp, built
# 2026-08-26 from commit b166e269) on RTX 3090 24 GB; Ryzen 7 7700X; 62 GB
# RAM. Previous runtime was upstream llama.cpp b10121. MoE expert weights
# pinned to CPU with an explicit --override-tensor regex (not --n-cpu-moe).
# KV cache quantized to q8_0 with Hadamard transforms on both K and V.
# KV cache quantized to iq5_k with Hadamard transforms on both K and V.
# Split mode none (single GPU; graph would require 2+ devices).
#
# Build flags (IK_FA note, see docs/LLM/ikllama-ornith-dankserver.md) must
# ALWAYS include: -DGGML_CUDA_FA_ALL_QUANTS=ON and -DGGML_IQK_FA_ALL_QUANTS=ON
# so the quantized + Hadamard KV cache combos compile into the CUDA/CPU FA
# kernels. Replacing the HauhauCS Qwen3.6-35B-A3B model on 2026-08-24.
# Switched from CRACK Q8_0 to Q6_K on 2026-08-25.
# Switched to CRACK Q8_0 on 2026-08-26 (was Q6_K).
#
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
# Model files (Q6_K CRACK GGUF + f16 mmproj vision encoder + MTP head bundled
# Model files (Q8_0 CRACK GGUF + f16 mmproj vision encoder + MTP head bundled
# in the GGUF). Dealign2026-08-24.
MODEL_DIR=/home/dank/models/Ornith-1.5-35B-A3B-CRACK
MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf
MODEL_FILE=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf
MMPROJ_FILE=mmproj-Ornith-1.5-35B-A3B-f16.gguf
# ik_llama.cpp llama-server (built 2026-08-26 from ikawrakow/ik_llama.cpp @
# b166e269; CUDA 13.2, -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_FA_ALL_QUANTS=ON,
@@ -43,19 +43,14 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
# cache and checkpoint headroom) = 491520 total context pool. The per-slot
# 163840 is advertised in models.json.
CTX_SIZE=491520
PARALLEL=3
# Single parallel slot with 160000 ctx. Note: with PARALLEL=1 speculative
# decoding would again be allowed by ik_llama; it stays disabled (SPEC_TYPE
# =none) unless a user asks to re-enable ngram-mod/MTP.
CTX_SIZE=160000
PARALLEL=1
# Speculative decoding: DISABLED. ik_llama limits speculative decoding
# (ngram-mod / MTP / etc.) to a single server slot (-np 1) and hard-exits if
# --parallel > 1 with a spec stage configured ("Speculative decoding is
# currently limited to a single server slot (-np 1)."). This service keeps
# 3 parallel slots (one per user), so SPEC_TYPE is disabled (none). It is a
# low-parameter MoE (3B active of 35B), so throughput is fine without
# speculation. MTP is also intentionally disabled.
# Speculative decoding: DISABLED. Spec is possible at PARALLEL=1 but stays
# off for the low-parameter MoE (3B active of 35B). MTP is also disabled.
SPEC_TYPE=none
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
@@ -76,14 +71,12 @@ OVERRIDE_BUFT=CPU
# OVERRIDE_EXPERT below.
N_GPU_LAYERS=999
# KV cache on GPU, both K and V at q8_0 plus Hadamard transforms on both
# caches to preserve quality at 8-bit. NOTE: ik_llama's fast q8_KV type
# crashes at model init on this MTP/multimodal arch with a ggml_view size
# assert (ggml.c:5427) at commit b166e269; plain q8_0 works. Docs recommend
# --k-cache-hadamard for quantized KV below Q6_0; we apply --v-cache-hadamard
# to V as well. Flash-attention is on by default.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# KV cache on GPU at q6_0 (6-bit k-quant) with Hadamard transforms on both
# caches. q8_KV was also rejected (ggml.c:5427 view assert, arch+build
# independent of PARALLEL/speculation); q6_0 + Hadamard is the fallback we
# verified works. Flash-attention is on by default.
CACHE_TYPE_K=q6_0
CACHE_TYPE_V=q6_0
K_CACHE_HADAMARD=1
V_CACHE_HADAMARD=1
scripts/serve-frank-small.sh
+11 -16
View File
@@ -1,18 +1,15 @@
#!/usr/bin/env bash
# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q6_K) as
# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q8_0) as
# frank-default-small via ik_llama.cpp on Dankserver (RTX 3090, MoE
# experts on CPU).
#
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
# Speculative decoding is disabled: ik_llama limits any spec stage to a single
# server slot (-np 1), and this service keeps 3 parallel slots.
# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled.
#
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
# The ik_llama fast q8_KV type crashes at init on this arch (ggml.c:5427);
# q8_0 is used instead.
#
# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.#
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
# split-mode graph with fewer than 2 devices (hard load error).
@@ -22,10 +19,8 @@
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# without relying on MTP speculation.
#
# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage
# is configured with --parallel > 1. MTP is also intentionally disabled.
# A low-parameter MoE (3B active) does not need speculation here.
#
# Speculative decoding: disabled (none). MTP is also intentionally disabled.
# A low-parameter MoE (3B active) does not need speculation here.#
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
# overrides that path.
set -euo pipefail
@@ -41,7 +36,7 @@ fi
# --- defaults (used when the env file does not set them) -----------------
: "${MODEL_DIR:=/home/dank/models}"
: "${MODEL_FILE:=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf}"
: "${MODEL_FILE:=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf}"
: "${MMPROJ_FILE:=mmproj-Ornith-1.5-35B-A3B-f16.gguf}"
: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
@@ -53,11 +48,11 @@ fi
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${CTX_SIZE:=160000}"
: "${PARALLEL:=1}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CACHE_TYPE_K:=q6_0}"
: "${CACHE_TYPE_V:=q6_0}"
: "${K_CACHE_HADAMARD:=0}"
: "${V_CACHE_HADAMARD:=0}"
: "${SPLIT_MODE:=none}"