diff --git a/docs/LLM/ikllama-ornith-dankserver.md b/docs/LLM/ikllama-ornith-dankserver.md index 10de131..5c99ce6 100644 --- a/docs/LLM/ikllama-ornith-dankserver.md +++ b/docs/LLM/ikllama-ornith-dankserver.md @@ -1,7 +1,7 @@ --- type: reference title: ik_llama.cpp on Dankserver (Ornith Q6_K) -description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), ngram-mod canonical args, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. +description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. q8_0 KV cache with Hadamard (the fast q8_KV type crashes on this arch at this commit), split-mode none (graph requires >=2 GPUs), speculative decoding disabled (3 parallel slots), and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. tags: - ikllama - llamacpp @@ -23,8 +23,9 @@ timestamp: 2026-08-26 The model is unchanged: `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK Q6_K), 35B-total / 3B-active MoE-dense hybrid with an MTP head in the GGUF. -Speculation stays ngram-only (MTP intentionally disabled). See -[[dealign-ornith-crack]] for the model runbook. +Speculative decoding is **disabled** (ik_llama limits spec to a single slot; +this service keeps 3 parallel slots). The low-parameter MoE does not need +it. See [[dealign-ornith-crack]] for the model runbook. ## Why ik_llama.cpp @@ -155,20 +156,25 @@ the CPU buft, leaving every full-attention block plus blocks 35-40 on CUDA. integer (the literal `all` is rejected). 999 means "load as many tensor layers as possible", while the expert override still pins the exps to CPU. -### Speculative decoding: canonical ngram-mod +### Speculative decoding: disabled (3 parallel slots) -ik_llama `main` **rejects** the legacy `--spec-ngram-mod-n-min/max/match` -flags. The canonical form is a repeated `--spec-type ngram-mod:`: -- `SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24` +`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is +intentionally disabled because ik_llama **limits any speculative stage to a +a single server slot**: with `--parallel 3` a configured spec stage +hard-exits at load with -Mapping from the old upstream config: -| legacy flag | old value | new canonical key | -|-------------|----------|-------------------| -| `--spec-ngram-mod-n-min` | 48 | `n_min=48` | -| `--spec-ngram-mod-n-max` | 64 | `n_max=64` | -| `--spec-ngram-mod-n-match` | 24 | `ngram_size_n=24` | +> `ERR [load_model] Speculative decoding is currently limited to a single +> server slot (-np 1).` / `n_parallel=3` -MTP remains intentionally disabled. +This service keeps **3 parallel slots** (one per user family via +`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays +disabled. The model is a low-parameter MoE (3B active of 35B) and does not +need the speculation it would cost a slot for. + +> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in +> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep +> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the +> extra slots win. ### Flash-attention + sampling @@ -201,8 +207,8 @@ The previous runtime `upstream llama.cpp b10121` and its binary (`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To rollback to the old stack, change `LLAMA_SERVER=` in `scripts/frank-default-small.env` back to that path and -`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-append the -`--spec-ngram-mod-*` flags. The model file is not changed either way. +`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram +spec / `--spec-type` flags. The model file is not changed either way. ## See Also diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 6c70cdd..24a86af 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -20,7 +20,7 @@ graph LR B2[ik_llama.cpp build b166e269 frank-default-small, port 7770] C[Ornith-1.5-35B-A3B-CRACK Q6_K -ngram-mod, q8_0 + Hadamard cache, 491520 ctx] +no speculation, q8_0 + Hadamard cache, 491520 ctx] A --> B2 --> C end diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index bc2af8a..f982079 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -45,14 +45,14 @@ LISTEN_PORT=7770 CTX_SIZE=491520 PARALLEL=3 -# Speculative decoding: ngram-mod only. MTP is intentionally disabled. -# ik_llama main uses canonical --spec-type ngram-mod:KEY=VAL form. Legacy -# --spec-ngram-mod-n-min/max/match are rejected. Mapping from the old config: -# old --spec-ngram-mod-n-min (48) -> n_min -# old --spec-ngram-mod-n-max (64) -> n_max -# old --spec-ngram-mod-n-match (24 -> ngram lookup length) -> ngram_size_n -# SPEC_TYPE is passed through to --spec-type. -SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24 +# Speculative decoding: DISABLED. ik_llama limits speculative decoding +# (ngram-mod / MTP / etc.) to a single server slot (-np 1) and hard-exits if +# --parallel > 1 with a spec stage configured ("Speculative decoding is +# currently limited to a single server slot (-np 1)."). This service keeps +# 3 parallel slots (one per user), so SPEC_TYPE is disabled (none). It is a +# low-parameter MoE (3B active of 35B), so throughput is fine without +# speculation. MTP is also intentionally disabled. +SPEC_TYPE=none # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K # layer audit found 630 MiB of expert tensors per block. Keep every diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 98ef9f6..942b8e6 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -4,8 +4,9 @@ # experts on CPU). # # Runtime: ik_llama.cpp llama-server (ikawrakow ik_llama.cpp). The model is a -# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; -# this service uses ngram-mod instead of MTP speculation. +# 35B-total / 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF. +# Speculative decoding is disabled: ik_llama limits any spec stage to a single +# server slot (-np 1), and this service keeps 3 parallel slots. # # KV cache: q8_0 with Hadamard transforms on K and V (requires a build with # GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON; see the env file). @@ -21,7 +22,9 @@ # final output-adjacent blocks stay on CUDA with all non-expert tensors, # without relying on MTP speculation. # -# Speculative decoding: ngram-mod only. MTP is intentionally disabled. +# Speculative decoding: disabled (none). ik_llama hard-exits if a spec stage +# is configured with --parallel > 1. MTP is also intentionally disabled. +# A low-parameter MoE (3B active) does not need speculation here. # # Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE # overrides that path. @@ -47,7 +50,7 @@ fi : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=491520}" : "${PARALLEL:=3}" -: "${SPEC_TYPE:=ngram-mod:n_min=48,n_max=64,ngram_size_n=24}" +: "${SPEC_TYPE:=none}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" : "${K_CACHE_HADAMARD:=0}" diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index ed0862e..31022e3 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -40,17 +40,18 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" [[ "$N_GPU_LAYERS" == 999 ]] } -@test "frank-default-small uses ngram-mod without MTP" { - # ik_llama main uses the canonical --spec-type ngram-mod: form; the - # legacy --spec-ngram-mod-n-* flags are rejected. - grep -q '^SPEC_TYPE=ngram-mod:n_min=48,n_max=64,ngram_size_n=24$' "$ENV_FILE" +@test "frank-default-small disables speculative decoding for 3 parallel slots" { + # ik_llama limits speculative decoding to -np 1 and hard-exits with + # --parallel > 1; this service keeps 3 parallel slots and runs SPEC_TYPE=none. + grep -q '^SPEC_TYPE=none$' "$ENV_FILE" + grep -q '^PARALLEL=3$' "$ENV_FILE" ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE" ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE" ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE" ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE" ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE" grep -q -- '--spec-type' "$SERVE_FILE" - ! grep -q -- '--spec-ngram-mod-' "$SERVE_FILE" + ! grep -q -- 'ngram' "$SERVE_FILE" } @test "frank-default-small uses ubatch 1024" {