frank-default-small: disable speculative decoding to keep 3 slots

ik_llama limits speculative decoding to a single server slot (-np 1) and
hard-exits at load if a spec stage is configured with --parallel > 1:
"Speculative decoding is currently limited to a single server slot
(-np 1)." This service keeps PARALLEL=3 (one slot per user), so set
SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B
active of 35B), so throughput is fine without it. MTP stays disabled.
KB and tests updated.

docs/LLM/ikllama-ornith-dankserver.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: ik_llama.cpp on Dankserver (Ornith Q6_K)
-description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
+description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
 tags:
   - ikllama
   - llamacpp
@@ -23,8 +23,9 @@ timestamp: 2026-08-26
 
 The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
 Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
-Speculation stays ngram-only (MTP intentionally disabled). See
+Speculative decoding is **disabled** (ik_llama limits spec to a single slot;
-[[dealign-ornith-crack]] for the model runbook.
+this service keeps 3 parallel slots). The low-parameter MoE does not need
+it. See [[dealign-ornith-crack]] for the model runbook.
 
 ## Why ik_llama.cpp
 
@@ -155,20 +156,25 @@ the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
   integer (the literal `all` is rejected). 999 means "load as many tensor
   layers as possible", while the expert override still pins the exps to CPU.
 
-### Speculative decoding: canonical ngram-mod
+### Speculative decoding: disabled (3 parallel slots)
 
-ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match`
+`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
-flags. The canonical form is a repeated `--spec-type ngram-mod:<k=v>`:
+intentionally disabled because ik_llama **limits any speculative stage to a
-- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24`
+a single server slot**: with `--parallel 3` a configured spec stage
+hard-exits at load with
 
-Mapping from the old upstream config:
+> `ERR [load_model] Speculative decoding is currently limited to a single
-| legacy flag | old value | new canonical key |
+> server slot (-np 1).` / `n_parallel=3`
-|-------------|----------|-------------------|
-| `--spec-ngram-mod-n-min` | 48 | `n_min=48` |
-| `--spec-ngram-mod-n-max` | 64 | `n_max=64` |
-| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` |
 
-MTP remains intentionally disabled.
+This service keeps **3 parallel slots** (one per user family via
+`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
+disabled. The model is a low-parameter MoE (3B active of 35B) and does not
+need the speculation it would cost a slot for.
+
+> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
+> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
+> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
+> extra slots win.
 
 ### Flash-attention + sampling
 
@@ -201,8 +207,8 @@ The previous runtime `upstream llama.cpp b10121` and its binary
 (`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
 rollback to the old stack, change `LLAMA_SERVER=` in
 `scripts/frank-default-small.env` back to that path and
-`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the
+`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
-`--spec-ngram-mod-*` flags. The model file is not changed either way.
+spec / `--spec-type` flags. The model file is not changed either way.
 
 ## See Also
 

docs/LLM/index.md

@@ -20,7 +20,7 @@ graph LR
         B2[ik_llama.cpp build b166e269
 frank-default-small, port 7770]
         C[Ornith-1.5-35B-A3B-CRACK Q6_K
-ngram-mod, q8_0 + Hadamard cache, 491520 ctx]
+no speculation, q8_0 + Hadamard cache, 491520 ctx]
         A --> B2 --> C
     end
 

scripts/frank-default-small.env

@@ -45,14 +45,14 @@ LISTEN_PORT=7770
 CTX_SIZE=491520
 PARALLEL=3
 
-# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
+# Speculative decoding: DISABLED. ik_llama limits speculative decoding
-# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
+# (ngram-mod / MTP / etc.) to a single server slot (-np 1) and hard-exits if
-# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
+# --parallel > 1 with a spec stage configured ("Speculative decoding is
-#   old --spec-ngram-mod-n-min  (48) -> n_min
+# currently limited to a single server slot (-np 1)."). This service keeps
-#   old --spec-ngram-mod-n-max  (64) -> n_max
+# 3 parallel slots (one per user), so SPEC_TYPE is disabled (none). It is a
-#   old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
+# low-parameter MoE (3B active of 35B), so throughput is fine without
-# SPEC_TYPE is passed through to --spec-type.
+# speculation. MTP is also intentionally disabled.
-SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
+SPEC_TYPE=none
 
 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
 # layer audit found 630 MiB of expert tensors per block. Keep every

scripts/serve-frank-small.sh

@@ -4,8 +4,9 @@
 # experts on CPU).
 #
 # Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
-# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
+# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
-# this service uses ngram-mod instead of MTP speculation.
+# Speculative decoding is disabled: ik_llama limits any spec stage to a single
+# server slot (-np 1), and this service keeps 3 parallel slots.
 #
 # KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
 # GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
@@ -21,7 +22,9 @@
 # final output-adjacent blocks stay on CUDA with all non-expert tensors,
 # without relying on MTP speculation.
 #
-# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
+# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage
+# is configured with --parallel > 1. MTP is also intentionally disabled.
+# A low-parameter MoE (3B active) does not need speculation here.
 #
 # Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
 # overrides that path.
@@ -47,7 +50,7 @@ fi
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=491520}"
 : "${PARALLEL:=3}"
-: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
+: "${SPEC_TYPE:=none}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
 : "${K_CACHE_HADAMARD:=0}"

tests/frank_small_config.bats

@@ -40,17 +40,18 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
     [[ "$N_GPU_LAYERS" == 999 ]]
 }
 
-@test "frank-default-small uses ngram-mod without MTP" {
+@test "frank-default-small disables speculative decoding for 3 parallel slots" {
-    # ik_llama main uses the canonical --spec-type ngram-mod:<k=v> form; the
+    # ik_llama limits speculative decoding to -np 1 and hard-exits with
-    # legacy --spec-ngram-mod-n-* flags are rejected.
+    # --parallel > 1; this service keeps 3 parallel slots and runs SPEC_TYPE=none.
-    grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE"
+    grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
+    grep -q '^PARALLEL=3$' "$ENV_FILE"
     ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
     ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
     ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
     ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
     ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
     grep -q -- '--spec-type' "$SERVE_FILE"
-    ! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE"
+    ! grep -q -- 'ngram' "$SERVE_FILE"
 }
 
 @test "frank-default-small uses ubatch 1024" {