frank-default-small: disable speculative decoding to keep 3 slots
ik_llama limits speculative decoding to a single server slot (-np 1) and hard-exits at load if a spec stage is configured with --parallel > 1: "Speculative decoding is currently limited to a single server slot (-np 1)." This service keeps PARALLEL=3 (one slot per user), so set SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B active of 35B), so throughput is fine without it. MTP stays disabled. KB and tests updated.
This commit is contained in:
1 parent
0e5c37f2f8
commit
4144f0f0e8
5 files changed
+44
-34
No files matched your search
@@ -4,8 +4,9 @@
|
||||
# experts on CPU).
|
||||
#
|
||||
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
|
||||
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
|
||||
# this service uses ngram-mod instead of MTP speculation.
|
||||
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
|
||||
# Speculative decoding is disabled: ik_llama limits any spec stage to a single
|
||||
# server slot (-np 1), and this service keeps 3 parallel slots.
|
||||
#
|
||||
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
|
||||
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
|
||||
@@ -21,7 +22,9 @@
|
||||
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
|
||||
# without relying on MTP speculation.
|
||||
#
|
||||
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
|
||||
# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage
|
||||
# is configured with --parallel > 1. MTP is also intentionally disabled.
|
||||
# A low-parameter MoE (3B active) does not need speculation here.
|
||||
#
|
||||
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
|
||||
# overrides that path.
|
||||
@@ -47,7 +50,7 @@ fi
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=491520}"
|
||||
: "${PARALLEL:=3}"
|
||||
: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
|
||||
: "${SPEC_TYPE:=none}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${K_CACHE_HADAMARD:=0}"
|
||||
|
||||
Reference in new issue
Block a user