From cd14d7bf68c431a3d805e81e4a696bfadaf3bd82 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 25 Aug 2026 01:40:06 -0700 Subject: [PATCH] Use ngram-mod and prioritize attention experts --- docs/LLM/Dankserver System Specs.md | 4 +- docs/LLM/dealign-ornith-crack.md | 70 +++++++++++++++-------------- docs/LLM/deployment.md | 15 +++---- docs/LLM/index.md | 2 +- scripts/frank-default-small.env | 30 ++++++------- scripts/frank-default-small.service | 2 +- scripts/serve-frank-small.sh | 28 ++++++------ tests/frank_small_config.bats | 41 +++++++++++++---- 8 files changed, 108 insertions(+), 84 deletions(-) diff --git a/docs/LLM/Dankserver System Specs.md b/docs/LLM/Dankserver System Specs.md index 9ebe2d4..3816032 100644 --- a/docs/LLM/Dankserver System Specs.md +++ b/docs/LLM/Dankserver System Specs.md @@ -60,11 +60,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0 ### Active Services -The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; full-attention blocks 27 and 31 plus output-adjacent blocks 33–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. +The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. ## Hardware Relevance -- 24 GB VRAM hosts the Q6_K non-expert tensors and the measured hybrid expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. Verify actual headroom after each model restart with `nvidia-smi` and the service journal. +- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal. - 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare. - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). diff --git a/docs/LLM/dealign-ornith-crack.md b/docs/LLM/dealign-ornith-crack.md index f775908..793d17c 100644 --- a/docs/LLM/dealign-ornith-crack.md +++ b/docs/LLM/dealign-ornith-crack.md @@ -1,7 +1,7 @@ --- type: reference title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver -description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small. +description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small. tags: - llamacpp - cuda @@ -22,9 +22,8 @@ timestamp: 2026-08-24 `frank-default-small` (port 7770) serves `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) + -attention architecture, abliterated (uncensored). The MTP speculative head -is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target -model. +attention architecture, abliterated (uncensored). The GGUF also contains a +bundled MTP head, but this deployment uses ngram-mod instead. ## Model Files @@ -45,23 +44,22 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block. -The measured hybrid configuration keeps all non-expert tensors on CUDA. It -also keeps expert tensors from full-attention blocks 27 and 31, plus the -output-adjacent blocks 33 through 40, on CUDA. Block 40 includes the bundled -MTP head. Expert tensors from all other blocks move to CPU through this -explicit override: +The selected configuration keeps all non-expert tensors on CUDA. It also +keeps expert tensors from every full-attention block, plus every block from 35 +through 40, on CUDA. The unique GPU expert set is +`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other +blocks move to CPU through this explicit override: ``` -OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU ``` -The measured benchmark compared three policies with a 16K-token prompt: -full-attention blocks plus block 40, contiguous top blocks 33–40, and this -hybrid. The hybrid was fastest at approximately 33.6 generated tokens per -second, versus 33.1 for the contiguous-top policy and 30.3 for the -full-attention-only policy. Its first-request prompt throughput was about -2711 tokens per second. +The 16K benchmark found the hybrid policy fastest among the three tested +placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a +stable low-memory choice. The current service uses ngram-mod only; MTP is +intentionally disabled. The full-attention expert set and final blocks are +kept on CUDA. To list the buffer type names on the host, run a bogus override: @@ -73,20 +71,21 @@ On Dankserver the available buffer types are `CPU` and `CUDA0`. ## Speculative Decoding -MoE model with a bundled MTP head -> native `draft-mtp`. The head is -auto-detected from the target (b10121 creates the MTP draft context against -the model file). ngram under MTP hurts on this family; MTP alone was -previously selected for the production configuration. A later 8K API -benchmark found that this MTP path produced lower generation throughput than -`--spec-type none` for that workload, so speculative decoding should be -re-evaluated before treating it as optimal. `--spec-draft-n-max 4` remains the -configured value. +The GGUF contains a bundled MTP head, but this deployment deliberately does +not use it. The current service enables only `ngram-mod`, with a 48-token +minimum, 64-token maximum, and 24-token lookup match: ``` ---spec-type draft-mtp ---spec-draft-n-max 4 +--spec-type ngram-mod +--spec-ngram-mod-n-min 48 +--spec-ngram-mod-n-max 64 +--spec-ngram-mod-n-match 24 ``` +In the controlled 8K test, no speculation produced approximately 53.7–53.9 +generation tokens per second. MTP was slower for that workload, so it remains +disabled. + ## Launch Excerpt ``` @@ -94,9 +93,13 @@ llama-server \ -m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \ --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \ --chat-template-file chat-template.jinja \ - --spec-type draft-mtp \ + --spec-type ngram-mod \ + --spec-ngram-mod-n-min 48 \ + --spec-ngram-mod-n-max 64 \ + --spec-ngram-mod-n-match 24 \ --override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ -ngl all -c 491520 -np 3 \ + --ubatch-size 1024 \ --cache-type-k f16 --cache-type-v f16 --no-mmap \ --temp 1.0 --top-p 0.95 --top-k 20 ``` @@ -105,12 +108,13 @@ llama-server \ 1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM box with the selected expert offload. -2. The provisional context pool contains 491520 cells. Three slots receive - 163840 cells each, which is 96K below the model context per stream. -3. `--n-cpu-moe` remains supported. This service currently prefers the - explicit tensor override because the 16K benchmark found this hybrid - placement fastest. It preserves the final eight blocks plus full-attention - blocks 27 and 31 on CUDA. +2. The context pool contains 491520 cells. Three slots receive 163840 cells + each, which is 96K below the model context per stream. +3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor + override because the 16K benchmark found the hybrid placement fastest. It + preserves every full-attention expert block and blocks 35–40 on CUDA. +4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the + only active speculative method. 4. The service uses the Qwen-Sharp template from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through `scripts/chat-template.jinja`. diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index 2858ba1..455360b 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -125,16 +125,15 @@ The active service on Dankserver is upstream llama.cpp (b10121) serving `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small` on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server` binary with expert tensors selected by an explicit `--override-tensor` -regex. Expert tensors from full-attention blocks 27 and 31 and the -output-adjacent blocks 33 through 40 stay on CUDA; other expert tensors are -CPU-resident. The service uses 7 CPU threads, native `draft-mtp` speculative -decoding (under re-evaluation after an 8K no-spec benchmark), f16 KV, context -491520 (3 slots of 163840 each), batch -4096, ubatch 2048, and `--no-mmap`. The unit is +regex. Expert tensors from every full-attention block and blocks 35 through 40 +stay on CUDA; other expert tensors are CPU-resident. The service uses 7 CPU +threads, ngram-mod speculative decoding only, f16 KV, context 491520 (3 slots +of 163840 each), batch +4096, ubatch 1024, and `--no-mmap`. The unit is `scripts/frank-default-small.service`. It launches through `scripts/serve-frank-small.sh`, which reads tunables from -`scripts/frank-default-small.env`. The chat template comes from -`peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]]. +`scripts/frank-default-small.env`. It uses `ubatch=1024` and the chat template +from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]]. ### Dankclaw — frank-default-large diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 9c6e66a..73dbb70 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -20,7 +20,7 @@ graph LR B[upstream llama.cpp b10121 frank-default-small, port 7770] C[Ornith-1.5-35B-A3B-CRACK Q6_K -native MTP, selected experts offloaded via override, 491520 ctx] +ngram-mod, selected experts offloaded via override, 491520 ctx] A --> B --> C end diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index e0dfdf3..fd8d7c7 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -30,24 +30,23 @@ LISTEN_PORT=7770 # 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for # cache and checkpoint headroom) = 491520 total context pool. The per-slot -# 163840 will be advertised in models.json after validation. +# 163840 is advertised in models.json. CTX_SIZE=491520 PARALLEL=3 -# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4 -# (the knee). ngram under MTP hurts; MTP alone is the proven best for MoE. -SPEC_TYPE=draft-mtp -SPEC_DRAFT_N_MAX=4 -SPEC_DRAFT_N_MIN=0 -SPEC_DRAFT_P_MIN=0.0 +# Speculative decoding: ngram-mod only. MTP is intentionally disabled. +SPEC_TYPE=ngram-mod +SPEC_NGRAM_MOD_N_MIN=48 +SPEC_NGRAM_MOD_N_MAX=64 +SPEC_NGRAM_MOD_N_MATCH=24 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K -# layer audit found 630 MiB of expert tensors per block. Keep the selected -# full-attention blocks 27 and 31 plus output-adjacent blocks 33-40 on CUDA, -# including the bundled MTP block 40. Move all other expert tensors to CPU. -# All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the CPU -# buft name on this host. -OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +# layer audit found 630 MiB of expert tensors per block. Keep every +# full-attention block plus every block 35-40 on CUDA. The unique GPU set is +# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to +# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the +# CPU buft name on this host. +OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU N_GPU_LAYERS=all @@ -59,9 +58,10 @@ VERBOSITY=4 # Store context checkpoints in RAM to hold peak GPU memory down. CTX_CHECKPOINTS=16 -# Big prefill batches. +# Prefill batch sizing. ubatch 1024 is selected for reduced compute-memory +# pressure and predictable latency under the ngram-mod deployment. BATCH_SIZE=4096 -UBATCH_SIZE=2048 +UBATCH_SIZE=1024 # Eager RAM load (expert weights on CPU; ~31 GB RAM). NO_MMAP=1 diff --git a/scripts/frank-default-small.service b/scripts/frank-default-small.service index 064adf3..3f9fbb4 100644 --- a/scripts/frank-default-small.service +++ b/scripts/frank-default-small.service @@ -1,5 +1,5 @@ [Unit] -Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K MTP + Qwen-Sharp template (dankserver, frank-default-small) +Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K ngram-mod + Qwen-Sharp template (dankserver, frank-default-small) After=network-online.target Wants=network-online.target diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 6cad613..154f127 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -4,17 +4,15 @@ # experts on CPU). # # Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total / -# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF. +# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this +# service uses ngram-mod instead of MTP speculation. # # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded # --override-tensor pattern are pinned to CPU. Selected full-attention and # final output-adjacent blocks stay on CUDA with all non-expert tensors, -# including the MTP head. +# without relying on MTP speculation. # -# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is -# auto-detected from the target model; b10121 creates the MTP draft context -# against the model file). ngram under MTP hurts (measured on the Qwen3.8 -# family), so MTP alone is kept. n_max 4 is the knee. +# Speculative decoding: ngram-mod only. MTP is intentionally disabled. # # Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE # overrides that path. @@ -40,21 +38,21 @@ fi : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=491520}" : "${PARALLEL:=3}" -: "${SPEC_TYPE:=draft-mtp}" -: "${SPEC_DRAFT_N_MAX:=4}" -: "${SPEC_DRAFT_N_MIN:=0}" -: "${SPEC_DRAFT_P_MIN:=0.0}" +: "${SPEC_TYPE:=ngram-mod}" +: "${SPEC_NGRAM_MOD_N_MIN:=48}" +: "${SPEC_NGRAM_MOD_N_MAX:=64}" +: "${SPEC_NGRAM_MOD_N_MATCH:=24}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" : "${VERBOSITY:=4}" # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the # MoE expert weights and leave every other tensor on the GPU. -: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}" +: "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}" : "${OVERRIDE_BUFT:=CPU}" : "${N_GPU_LAYERS:=all}" : "${BATCH_SIZE:=4096}" -: "${UBATCH_SIZE:=2048}" +: "${UBATCH_SIZE:=1024}" : "${CTX_CHECKPOINTS:=16}" : "${NO_MMAP:=1}" : "${TEMPERATURE:=1.0}" @@ -83,9 +81,9 @@ exec "$LLAMA_SERVER" \ --mmproj "$MMPROJ" \ --chat-template-file "$CHAT_TEMPLATE_FILE" \ --spec-type "$SPEC_TYPE" \ - --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \ - --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \ - --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \ + --spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \ + --spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \ + --spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \ --override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \ --cache-type-k "$CACHE_TYPE_K" \ --cache-type-v "$CACHE_TYPE_V" \ diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index c9c35f1..bb6c52c 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -20,15 +20,38 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" grep -q '^PARALLEL=3$' "$ENV_FILE" } -@test "frank-default-small uses the measured hybrid expert placement" { - override=$(grep '^OVERRIDE_EXPERT=' "$ENV_FILE") - [[ "$override" == *'26|28|29|30|32)'* ]] - [[ "$override" != *'27'* ]] - [[ "$override" != *'31'* ]] - [[ "$override" != *'33'* ]] - [[ "$override" != *'40'* ]] - grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE" - grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE" +@test "frank-default-small keeps full-attention and final blocks on CUDA" { + source "$ENV_FILE" + [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]] + [[ "$OVERRIDE_EXPERT" != *'|3|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|7|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|11|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|15|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|19|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|23|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|27|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|31|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|35|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|36|'* ]] + [[ "$OVERRIDE_EXPERT" != *'|40|'* ]] + [[ "$OVERRIDE_BUFT" == CPU ]] + [[ "$N_GPU_LAYERS" == all ]] +} + +@test "frank-default-small uses ngram-mod without MTP" { + grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE" + grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE" + grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE" + grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE" + ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE" + ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE" + grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE" +} + +@test "frank-default-small uses ubatch 1024" { + grep -q '^BATCH_SIZE=4096$' "$ENV_FILE" + grep -q '^UBATCH_SIZE=1024$' "$ENV_FILE" + grep -q ': "${UBATCH_SIZE:=1024}"' "$SERVE_FILE" } @test "frank-default-small uses the tracked Qwen-Sharp template" {