dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: disable speculative decoding to keep 3 slots

Browse Source 
ik_llama limits speculative decoding to a single server slot (-np 1) and
hard-exits at load if a spec stage is configured with --parallel > 1:
"Speculative decoding is currently limited to a single server slot
(-np 1)." This service keeps PARALLEL=3 (one slot per user), so set
SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B
active of 35B), so throughput is fine without it. MTP stays disabled.
KB and tests updated.
This commit is contained in:
Frank
2026-08-26 03:38:25 -07:00
parent 0e5c37f2f8
commit 4144f0f0e8
5 changed files with 44 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+8 -8
View File
@@ -45,14 +45,14 @@ LISTEN_PORT=7770
CTX_SIZE=491520
PARALLEL=3
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
# old --spec-ngram-mod-n-min (48) -> n_min
# old --spec-ngram-mod-n-max (64) -> n_max
# old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
# SPEC_TYPE is passed through to --spec-type.
SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
# Speculative decoding: DISABLED. ik_llama limits speculative decoding
# (ngram-mod / MTP / etc.) to a single server slot (-np 1) and hard-exits if
# --parallel > 1 with a spec stage configured ("Speculative decoding is
# currently limited to a single server slot (-np 1)."). This service keeps
# 3 parallel slots (one per user), so SPEC_TYPE is disabled (none). It is a
# low-parameter MoE (3B active of 35B), so throughput is fine without
# speculation. MTP is also intentionally disabled.
SPEC_TYPE=none
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep every
scripts/serve-frank-small.sh
+7 -4
View File
@@ -4,8 +4,9 @@
# experts on CPU).
#
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
# this service uses ngram-mod instead of MTP speculation.
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
# Speculative decoding is disabled: ik_llama limits any spec stage to a single
# server slot (-np 1), and this service keeps 3 parallel slots.
#
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
@@ -21,7 +22,9 @@
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# without relying on MTP speculation.
#
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage
# is configured with --parallel > 1. MTP is also intentionally disabled.
# A low-parameter MoE (3B active) does not need speculation here.
#
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
# overrides that path.
@@ -47,7 +50,7 @@ fi
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${K_CACHE_HADAMARD:=0}"