dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: Q8_0 model, 160k single slot, q6_0 KV + Hadamard

Browse Source 
Switch to CRACK Q8_0 model (37.8 GB, mmproj f16). Servt as a single
160000-ctx slot (PARALLEL=1); speculative decoding stays disabled. KV
cache is q6_0 + Hadamard. iq5_k is not a valid KV-cache type in ik_llama
(cache whitelist excludes iq5_*); q8_KV reproduces the ggml.c:5427 view
assert even at PARALLEL=1/no-spec, so q6_0 is the verified working 6-bit
choice. Env, serve script, tests, and KB updated. Deployment was left
down after the isolation test; this commit redeploys it with the
verified Q8_0/160k/q6_0 config.
This commit is contained in:
Frank committed 2026-08-26 05:01:58 -07:00
1 parent 23e905c712
commit 8c41c821c4
4 files changed
+67 -75
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/serve-frank-small.sh
+11 -16
View File
@@ -1,18 +1,15 @@
#!/usr/bin/env bash
# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q6_K) as
# Serve dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK, Q8_0) as
# frank-default-small via ik_llama.cpp on Dankserver (RTX 3090, MoE
# experts on CPU).
#
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
# Speculative decoding is disabled: ik_llama limits any spec stage to a single
# server slot (-np 1), and this service keeps 3 parallel slots.
# Single slot (PARALLEL=1, CTX 160000). Speculative decoding is disabled.
#
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
# KV cache: q6_0 with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
# The ik_llama fast q8_KV type crashes at init on this arch (ggml.c:5427);
# q8_0 is used instead.
#
# q8_KV crashes at init on this arch (ggml.c:5427); q6_0 is used instead.#
# Split mode is none: Dankserver has a single RTX 3090 and ik_llama rejects
# split-mode graph with fewer than 2 devices (hard load error).
@@ -22,10 +19,8 @@
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# without relying on MTP speculation.
#
# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage
# is configured with --parallel > 1. MTP is also intentionally disabled.
# A low-parameter MoE (3B active) does not need speculation here.
#
# Speculative decoding: disabled (none). MTP is also intentionally disabled.
# A low-parameter MoE (3B active) does not need speculation here.#
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
# overrides that path.
set -euo pipefail
@@ -41,7 +36,7 @@ fi
# --- defaults (used when the env file does not set them) -----------------
: "${MODEL_DIR:=/home/dank/models}"
: "${MODEL_FILE:=Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf}"
: "${MODEL_FILE:=Ornith-1.5-35B-A3B-CRACK-Q8_0.gguf}"
: "${MMPROJ_FILE:=mmproj-Ornith-1.5-35B-A3B-f16.gguf}"
: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
@@ -53,11 +48,11 @@ fi
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${CTX_SIZE:=160000}"
: "${PARALLEL:=1}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CACHE_TYPE_K:=q6_0}"
: "${CACHE_TYPE_V:=q6_0}"
: "${K_CACHE_HADAMARD:=0}"
: "${V_CACHE_HADAMARD:=0}"
: "${SPLIT_MODE:=none}"