frank-default-small: disable speculative decoding to keep 3 slots
ik_llama limits speculative decoding to a single server slot (-np 1) and hard-exits at load if a spec stage is configured with --parallel > 1: "Speculative decoding is currently limited to a single server slot (-np 1)." This service keeps PARALLEL=3 (one slot per user), so set SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B active of 35B), so throughput is fine without it. MTP stays disabled. KB and tests updated.
This commit is contained in:
1 parent
0e5c37f2f8
commit
4144f0f0e8
5 files changed
+44
-34
No files matched your search
+1
-1
@@ -20,7 +20,7 @@ graph LR
|
||||
B2[ik_llama.cpp build b166e269
|
||||
frank-default-small, port 7770]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
||||
ngram-mod, q8_0 + Hadamard cache, 491520 ctx]
|
||||
no speculation, q8_0 + Hadamard cache, 491520 ctx]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
|
||||
Reference in new issue
Block a user