dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: disable speculative decoding to keep 3 slots

Browse Source 
ik_llama limits speculative decoding to a single server slot (-np 1) and
hard-exits at load if a spec stage is configured with --parallel > 1:
"Speculative decoding is currently limited to a single server slot
(-np 1)." This service keeps PARALLEL=3 (one slot per user), so set
SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B
active of 35B), so throughput is fine without it. MTP stays disabled.
KB and tests updated.
This commit is contained in:
Frank
2026-08-26 03:38:25 -07:00
parent 0e5c37f2f8
commit 4144f0f0e8
5 changed files with 44 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/ikllama-ornith-dankserver.md
+22 -16
View File
@@ -1,7 +1,7 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q6_K)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
@@ -23,8 +23,9 @@ timestamp: 2026-08-26
The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK
Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF.
Speculation stays ngram-only (MTP intentionally disabled). See
[[dealign-ornith-crack]] for the model runbook.
Speculative decoding is **disabled** (ik_llama limits spec to a single slot;
this service keeps 3 parallel slots). The low-parameter MoE does not need
it. See [[dealign-ornith-crack]] for the model runbook.
## Why ik_llama.cpp
@@ -155,20 +156,25 @@ the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA.
integer (the literal `all` is rejected). 999 means "load as many tensor
layers as possible", while the expert override still pins the exps to CPU.
### Speculative decoding: canonical ngram-mod
### Speculative decoding: disabled (3 parallel slots)
ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match`
flags. The canonical form is a repeated `--spec-type ngram-mod:<k=v>`:
- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24`
`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
intentionally disabled because ik_llama **limits any speculative stage to a
a single server slot**: with `--parallel 3` a configured spec stage
hard-exits at load with
Mapping from the old upstream config:
| legacy flag | old value | new canonical key |
|-------------|----------|-------------------|
| `--spec-ngram-mod-n-min` | 48 | `n_min=48` |
| `--spec-ngram-mod-n-max` | 64 | `n_max=64` |
| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` |
> `ERR [load_model] Speculative decoding is currently limited to a single
> server slot (-np 1).` / `n_parallel=3`
MTP remains intentionally disabled.
This service keeps **3 parallel slots** (one per user family via
`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
disabled. The model is a low-parameter MoE (3B active of 35B) and does not
need the speculation it would cost a slot for.
> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
> extra slots win.
### Flash-attention + sampling
@@ -201,8 +207,8 @@ The previous runtime `upstream llama.cpp b10121` and its binary
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
rollback to the old stack, change `LLAMA_SERVER=` in
`scripts/frank-default-small.env` back to that path and
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the
`--spec-ngram-mod-*` flags. The model file is not changed either way.
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
spec / `--spec-type` flags. The model file is not changed either way.
## See Also
docs/LLM/index.md
+1 -1
View File
@@ -20,7 +20,7 @@ graph LR
B2[ik_llama.cpp build b166e269
frank-default-small, port 7770]
C[Ornith-1.5-35B-A3B-CRACK Q6_K
ngram-mod, q8_0 + Hadamard cache, 491520 ctx]
no speculation, q8_0 + Hadamard cache, 491520 ctx]
A --> B2 --> C
end
scripts/frank-default-small.env
+8 -8
View File
@@ -45,14 +45,14 @@ LISTEN_PORT=7770
CTX_SIZE=491520
PARALLEL=3
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy
# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config:
# old --spec-ngram-mod-n-min (48) -> n_min
# old --spec-ngram-mod-n-max (64) -> n_max
# old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n
# SPEC_TYPE is passed through to --spec-type.
SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24
# Speculative decoding: DISABLED. ik_llama limits speculative decoding
# (ngram-mod / MTP / etc.) to a single server slot (-np 1) and hard-exits if
# --parallel > 1 with a spec stage configured ("Speculative decoding is
# currently limited to a single server slot (-np 1)."). This service keeps
# 3 parallel slots (one per user), so SPEC_TYPE is disabled (none). It is a
# low-parameter MoE (3B active of 35B), so throughput is fine without
# speculation. MTP is also intentionally disabled.
SPEC_TYPE=none
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep every
scripts/serve-frank-small.sh
+7 -4
View File
@@ -4,8 +4,9 @@
# experts on CPU).
#
# Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF;
# this service uses ngram-mod instead of MTP speculation.
# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF.
# Speculative decoding is disabled: ik_llama limits any spec stage to a single
# server slot (-np 1), and this service keeps 3 parallel slots.
#
# KV cache: q8_0 with Hadamard transforms on K and V (requires a build with
# GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file).
@@ -21,7 +22,9 @@
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# without relying on MTP speculation.
#
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage
# is configured with --parallel > 1. MTP is also intentionally disabled.
# A low-parameter MoE (3B active) does not need speculation here.
#
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
# overrides that path.
@@ -47,7 +50,7 @@ fi
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}"
: "${SPEC_TYPE:=none}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${K_CACHE_HADAMARD:=0}"
tests/frank_small_config.bats
+6 -5
View File
@@ -40,17 +40,18 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
[[ "$N_GPU_LAYERS" == 999 ]]
}
@test "frank-default-small uses ngram-mod without MTP" {
# ik_llama main uses the canonical --spec-type ngram-mod:<k=v> form; the
# legacy --spec-ngram-mod-n-* flags are rejected.
grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE"
@test "frank-default-small disables speculative decoding for 3 parallel slots" {
# ik_llama limits speculative decoding to -np 1 and hard-exits with
# --parallel > 1; this service keeps 3 parallel slots and runs SPEC_TYPE=none.
grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
grep -q '^PARALLEL=3$' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
grep -q -- '--spec-type' "$SERVE_FILE"
! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE"
! grep -q -- 'ngram' "$SERVE_FILE"
}
@test "frank-default-small uses ubatch 1024" {