frank-default-small: disable speculative decoding to keep 3 slots
ik_llama limits speculative decoding to a single server slot (-np 1) and hard-exits at load if a spec stage is configured with --parallel > 1: "Speculative decoding is currently limited to a single server slot (-np 1)." This service keeps PARALLEL=3 (one slot per user), so set SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B active of 35B), so throughput is fine without it. MTP stays disabled. KB and tests updated.
This commit is contained in:
1 parent
0e5c37f2f8
commit
4144f0f0e8
5 files changed
+44
-34
No files matched your search
@@ -45,14 +45,14 @@ LISTEN_PORT=7770
|
||||
CTX_SIZE=491520
|
||||
PARALLEL=3
|
||||
|
||||
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
|
||||
# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
|
||||
# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
|
||||
# old --spec-ngram-mod-n-min (48) -> n_min
|
||||
# old --spec-ngram-mod-n-max (64) -> n_max
|
||||
# old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
|
||||
# SPEC_TYPE is passed through to --spec-type.
|
||||
SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
|
||||
# Speculative decoding: DISABLED. ik_llama limits speculative decoding
|
||||
# (ngram-mod / MTP / etc.) to a single server slot (-np 1) and hard-exits if
|
||||
# --parallel > 1 with a spec stage configured ("Speculative decoding is
|
||||
# currently limited to a single server slot (-np 1)."). This service keeps
|
||||
# 3 parallel slots (one per user), so SPEC_TYPE is disabled (none). It is a
|
||||
# low-parameter MoE (3B active of 35B), so throughput is fine without
|
||||
# speculation. MTP is also intentionally disabled.
|
||||
SPEC_TYPE=none
|
||||
|
||||
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
|
||||
# layer audit found 630 MiB of expert tensors per block. Keep every
|
||||
|
||||
Reference in new issue
Block a user