Use ngram-mod and prioritize attention experts

docs/LLM/Dankserver System Specs.md

@@ -60,11 +60,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
 
 ### Active Services
 
-The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; full-attention blocks 27 and 31 plus output-adjacent blocks 33–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
+The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
 
 ## Hardware Relevance
 
-- 24 GB VRAM hosts the Q6_K non-expert tensors and the measured hybrid expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
+- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
 - 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare.
 - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
 - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).

docs/LLM/dealign-ornith-crack.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
-description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
+description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
 tags:
   - llamacpp
   - cuda
@@ -22,9 +22,8 @@ timestamp: 2026-08-24
 `frank-default-small` (port 7770) serves
 `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
 a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
-attention architecture, abliterated (uncensored). The MTP speculative head
+attention architecture, abliterated (uncensored). The GGUF also contains a
-is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
+bundled MTP head, but this deployment uses ngram-mod instead.
-model.
 
 ## Model Files
 
@@ -45,23 +44,22 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
 ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
 block 40. The expert portion is 630 MiB in every block.
 
-The measured hybrid configuration keeps all non-expert tensors on CUDA. It
+The selected configuration keeps all non-expert tensors on CUDA. It also
-also keeps expert tensors from full-attention blocks 27 and 31, plus the
+keeps expert tensors from every full-attention block, plus every block from 35
-output-adjacent blocks 33 through 40, on CUDA. Block 40 includes the bundled
+through 40, on CUDA. The unique GPU expert set is
-MTP head. Expert tensors from all other blocks move to CPU through this
+`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other
-explicit override:
+blocks move to CPU through this explicit override:
 
 ```
-OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
+OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 ```
 
-The measured benchmark compared three policies with a 16K-token prompt:
+The 16K benchmark found the hybrid policy fastest among the three tested
-full-attention blocks plus block 40, contiguous top blocks 33–40, and this
+placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a
-hybrid. The hybrid was fastest at approximately 33.6 generated tokens per
+stable low-memory choice. The current service uses ngram-mod only; MTP is
-second, versus 33.1 for the contiguous-top policy and 30.3 for the
+intentionally disabled. The full-attention expert set and final blocks are
-full-attention-only policy. Its first-request prompt throughput was about
+kept on CUDA.
-2711 tokens per second.
 
 To list the buffer type names on the host, run a bogus override:
 
@@ -73,20 +71,21 @@ On Dankserver the available buffer types are `CPU` and `CUDA0`.
 
 ## Speculative Decoding
 
-MoE model with a bundled MTP head -> native `draft-mtp`. The head is
+The GGUF contains a bundled MTP head, but this deployment deliberately does
-auto-detected from the target (b10121 creates the MTP draft context against
+not use it. The current service enables only `ngram-mod`, with a 48-token
-the model file). ngram under MTP hurts on this family; MTP alone was
+minimum, 64-token maximum, and 24-token lookup match:
-previously selected for the production configuration. A later 8K API
-benchmark found that this MTP path produced lower generation throughput than
-`--spec-type none` for that workload, so speculative decoding should be
-re-evaluated before treating it as optimal. `--spec-draft-n-max 4` remains the
-configured value.
 
 ```
---spec-type draft-mtp
+--spec-type ngram-mod
---spec-draft-n-max 4
+--spec-ngram-mod-n-min 48
+--spec-ngram-mod-n-max 64
+--spec-ngram-mod-n-match 24
 ```
 
+In the controlled 8K test, no speculation produced approximately 53.7–53.9
+generation tokens per second. MTP was slower for that workload, so it remains
+disabled.
+
 ## Launch Excerpt
 
 ```
@@ -94,9 +93,13 @@ llama-server \
   -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
   --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
   --chat-template-file chat-template.jinja \
-  --spec-type draft-mtp \
+  --spec-type ngram-mod \
+  --spec-ngram-mod-n-min 48 \
+  --spec-ngram-mod-n-max 64 \
+  --spec-ngram-mod-n-match 24 \
   --override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
   -ngl all -c 491520 -np 3 \
+  --ubatch-size 1024 \
   --cache-type-k f16 --cache-type-v f16 --no-mmap \
   --temp 1.0 --top-p 0.95 --top-k 20
 ```
@@ -105,12 +108,13 @@ llama-server \
 
 1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
    box with the selected expert offload.
-2. The provisional context pool contains 491520 cells. Three slots receive
+2. The context pool contains 491520 cells. Three slots receive 163840 cells
-   163840 cells each, which is 96K below the model context per stream.
+   each, which is 96K below the model context per stream.
-3. `--n-cpu-moe` remains supported. This service currently prefers the
+3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
-   explicit tensor override because the 16K benchmark found this hybrid
+   override because the 16K benchmark found the hybrid placement fastest. It
-   placement fastest. It preserves the final eight blocks plus full-attention
+   preserves every full-attention expert block and blocks 35–40 on CUDA.
-   blocks 27 and 31 on CUDA.
+4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the
+   only active speculative method.
 4. The service uses the Qwen-Sharp template from
    `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
    `scripts/chat-template.jinja`.

docs/LLM/deployment.md

@@ -125,16 +125,15 @@ The active service on Dankserver is upstream llama.cpp (b10121) serving
 `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
 on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
 binary with expert tensors selected by an explicit `--override-tensor`
-regex. Expert tensors from full-attention blocks 27 and 31 and the
+regex. Expert tensors from every full-attention block and blocks 35 through 40
-output-adjacent blocks 33 through 40 stay on CUDA; other expert tensors are
+stay on CUDA; other expert tensors are CPU-resident. The service uses 7 CPU
-CPU-resident. The service uses 7 CPU threads, native `draft-mtp` speculative
+threads, ngram-mod speculative decoding only, f16 KV, context 491520 (3 slots
-decoding (under re-evaluation after an 8K no-spec benchmark), f16 KV, context
+of 163840 each), batch
-491520 (3 slots of 163840 each), batch
+4096, ubatch 1024, and `--no-mmap`. The unit is
-4096, ubatch 2048, and `--no-mmap`. The unit is
 `scripts/frank-default-small.service`. It launches through
 `scripts/serve-frank-small.sh`, which reads tunables from
-`scripts/frank-default-small.env`. The chat template comes from
+`scripts/frank-default-small.env`. It uses `ubatch=1024` and the chat template
-`peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
+from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
 
 ### Dankclaw — frank-default-large
 

docs/LLM/index.md

@@ -20,7 +20,7 @@ graph LR
         B[upstream llama.cpp b10121
 frank-default-small, port 7770]
         C[Ornith-1.5-35B-A3B-CRACK Q6_K
-native MTP, selected experts offloaded via override, 491520 ctx]
+ngram-mod, selected experts offloaded via override, 491520 ctx]
         A --> B --> C
     end
 

scripts/frank-default-small.env

@@ -30,24 +30,23 @@ LISTEN_PORT=7770
 
 # 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
 # cache and checkpoint headroom) = 491520 total context pool. The per-slot
-# 163840 will be advertised in models.json after validation.
+# 163840 is advertised in models.json.
 CTX_SIZE=491520
 PARALLEL=3
 
-# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
+# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
-# (the knee). ngram under MTP hurts; MTP alone is the proven best for MoE.
+SPEC_TYPE=ngram-mod
-SPEC_TYPE=draft-mtp
+SPEC_NGRAM_MOD_N_MIN=48
-SPEC_DRAFT_N_MAX=4
+SPEC_NGRAM_MOD_N_MAX=64
-SPEC_DRAFT_N_MIN=0
+SPEC_NGRAM_MOD_N_MATCH=24
-SPEC_DRAFT_P_MIN=0.0
 
 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
-# layer audit found 630 MiB of expert tensors per block. Keep the selected
+# layer audit found 630 MiB of expert tensors per block. Keep every
-# full-attention blocks 27 and 31 plus output-adjacent blocks 33-40 on CUDA,
+# full-attention block plus every block 35-40 on CUDA. The unique GPU set is
-# including the bundled MTP block 40. Move all other expert tensors to CPU.
+# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
-# All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the CPU
+# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
-# buft name on this host.
+# CPU buft name on this host.
-OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
+OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 N_GPU_LAYERS=all
 
@@ -59,9 +58,10 @@ VERBOSITY=4
 # Store context checkpoints in RAM to hold peak GPU memory down.
 CTX_CHECKPOINTS=16
 
-# Big prefill batches.
+# Prefill batch sizing. ubatch 1024 is selected for reduced compute-memory
+# pressure and predictable latency under the ngram-mod deployment.
 BATCH_SIZE=4096
-UBATCH_SIZE=2048
+UBATCH_SIZE=1024
 
 # Eager RAM load (expert weights on CPU; ~31 GB RAM).
 NO_MMAP=1

scripts/frank-default-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K MTP + Qwen-Sharp template (dankserver, frank-default-small)
+Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K ngram-mod + Qwen-Sharp template (dankserver, frank-default-small)
 After=network-online.target
 Wants=network-online.target
 

scripts/serve-frank-small.sh

@@ -4,17 +4,15 @@
 # experts on CPU).
 #
 # Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total /
-# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF.
+# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this
+# service uses ngram-mod instead of MTP speculation.
 #
 # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
 # --override-tensor pattern are pinned to CPU. Selected full-attention and
 # final output-adjacent blocks stay on CUDA with all non-expert tensors,
-# including the MTP head.
+# without relying on MTP speculation.
 #
-# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is
+# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
-# auto-detected from the target model; b10121 creates the MTP draft context
-# against the model file). ngram under MTP hurts (measured on the Qwen3.8
-# family), so MTP alone is kept. n_max 4 is the knee.
 #
 # Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
 # overrides that path.
@@ -40,21 +38,21 @@ fi
 : "${LISTEN_PORT:=7770}"
 : "${CTX_SIZE:=491520}"
 : "${PARALLEL:=3}"
-: "${SPEC_TYPE:=draft-mtp}"
+: "${SPEC_TYPE:=ngram-mod}"
-: "${SPEC_DRAFT_N_MAX:=4}"
+: "${SPEC_NGRAM_MOD_N_MIN:=48}"
-: "${SPEC_DRAFT_N_MIN:=0}"
+: "${SPEC_NGRAM_MOD_N_MAX:=64}"
-: "${SPEC_DRAFT_P_MIN:=0.0}"
+: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
 : "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=f16}"
 : "${VERBOSITY:=4}"
 # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
 # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
 # MoE expert weights and leave every other tensor on the GPU.
-: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
+: "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
 : "${OVERRIDE_BUFT:=CPU}"
 : "${N_GPU_LAYERS:=all}"
 : "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=2048}"
+: "${UBATCH_SIZE:=1024}"
 : "${CTX_CHECKPOINTS:=16}"
 : "${NO_MMAP:=1}"
 : "${TEMPERATURE:=1.0}"
@@ -83,9 +81,9 @@ exec "$LLAMA_SERVER" \
     --mmproj "$MMPROJ" \
     --chat-template-file "$CHAT_TEMPLATE_FILE" \
     --spec-type "$SPEC_TYPE" \
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
+    --spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \
-    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
+    --spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \
-    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
+    --spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \
     --override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
     --cache-type-k "$CACHE_TYPE_K" \
     --cache-type-v "$CACHE_TYPE_V" \

tests/frank_small_config.bats

@@ -20,15 +20,38 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
     grep -q '^PARALLEL=3$' "$ENV_FILE"
 }
 
-@test "frank-default-small uses the measured hybrid expert placement" {
+@test "frank-default-small keeps full-attention and final blocks on CUDA" {
-    override=$(grep '^OVERRIDE_EXPERT=' "$ENV_FILE")
+    source "$ENV_FILE"
-    [[ "$override" == *'26|28|29|30|32)'* ]]
+    [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
-    [[ "$override" != *'27'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
-    [[ "$override" != *'31'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
-    [[ "$override" != *'33'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
-    [[ "$override" != *'40'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|15|'* ]]
-    grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE"
+    [[ "$OVERRIDE_EXPERT" != *'|19|'* ]]
-    grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE"
+    [[ "$OVERRIDE_EXPERT" != *'|23|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|27|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|31|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
+    [[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
+    [[ "$OVERRIDE_BUFT" == CPU ]]
+    [[ "$N_GPU_LAYERS" == all ]]
+}
+
+@test "frank-default-small uses ngram-mod without MTP" {
+    grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE"
+    grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE"
+    grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE"
+    grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE"
+    ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
+    ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
+    grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE"
+}
+
+@test "frank-default-small uses ubatch 1024" {
+    grep -q '^BATCH_SIZE=4096$' "$ENV_FILE"
+    grep -q '^UBATCH_SIZE=1024$' "$ENV_FILE"
+    grep -q ': "${UBATCH_SIZE:=1024}"' "$SERVE_FILE"
 }
 
 @test "frank-default-small uses the tracked Qwen-Sharp template" {