dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Use ngram-mod and prioritize attention experts

Browse Source
This commit is contained in:
Frank
2026-08-25 01:40:06 -07:00
parent 37fb420fcd
commit cd14d7bf68
8 changed files with 108 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/Dankserver System Specs.md
+2 -2
View File
@@ -60,11 +60,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; full-attention blocks 27 and 31 plus output-adjacent blocks 33–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and the measured hybrid expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
docs/LLM/dealign-ornith-crack.md
+37 -33
View File
@@ -1,7 +1,7 @@
---
type: reference
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with the MTP head bundled; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
tags:
- llamacpp
- cuda
@@ -22,9 +22,8 @@ timestamp: 2026-08-24
`frank-default-small` (port 7770) serves
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
attention architecture, abliterated (uncensored). The MTP speculative head
is **bundled in the GGUF**, so `draft-mtp` auto-detects it from the target
model.
attention architecture, abliterated (uncensored). The GGUF also contains a
bundled MTP head, but this deployment uses ngram-mod instead.
## Model Files
@@ -45,23 +44,22 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
block 40. The expert portion is 630 MiB in every block.
The measured hybrid configuration keeps all non-expert tensors on CUDA. It
also keeps expert tensors from full-attention blocks 27 and 31, plus the
output-adjacent blocks 33 through 40, on CUDA. Block 40 includes the bundled
MTP head. Expert tensors from all other blocks move to CPU through this
explicit override:
The selected configuration keeps all non-expert tensors on CUDA. It also
keeps expert tensors from every full-attention block, plus every block from 35
through 40, on CUDA. The unique GPU expert set is
`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other
blocks move to CPU through this explicit override:
```
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
```
The measured benchmark compared three policies with a 16K-token prompt:
full-attention blocks plus block 40, contiguous top blocks 33–40, and this
hybrid. The hybrid was fastest at approximately 33.6 generated tokens per
second, versus 33.1 for the contiguous-top policy and 30.3 for the
full-attention-only policy. Its first-request prompt throughput was about
2711 tokens per second.
The 16K benchmark found the hybrid policy fastest among the three tested
placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a
stable low-memory choice. The current service uses ngram-mod only; MTP is
intentionally disabled. The full-attention expert set and final blocks are
kept on CUDA.
To list the buffer type names on the host, run a bogus override:
@@ -73,20 +71,21 @@ On Dankserver the available buffer types are `CPU` and `CUDA0`.
## Speculative Decoding
MoE model with a bundled MTP head -> native `draft-mtp`. The head is
auto-detected from the target (b10121 creates the MTP draft context against
the model file). ngram under MTP hurts on this family; MTP alone was
previously selected for the production configuration. A later 8K API
benchmark found that this MTP path produced lower generation throughput than
`--spec-type none` for that workload, so speculative decoding should be
re-evaluated before treating it as optimal. `--spec-draft-n-max 4` remains the
configured value.
The GGUF contains a bundled MTP head, but this deployment deliberately does
not use it. The current service enables only `ngram-mod`, with a 48-token
minimum, 64-token maximum, and 24-token lookup match:
```
--spec-type draft-mtp
--spec-draft-n-max 4
--spec-type ngram-mod
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64
--spec-ngram-mod-n-match 24
```
In the controlled 8K test, no speculation produced approximately 53.7–53.9
generation tokens per second. MTP was slower for that workload, so it remains
disabled.
## Launch Excerpt
```
@@ -94,9 +93,13 @@ llama-server \
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \
--spec-type draft-mtp \
--spec-type ngram-mod \
--spec-ngram-mod-n-min 48 \
--spec-ngram-mod-n-max 64 \
--spec-ngram-mod-n-match 24 \
--override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 491520 -np 3 \
--ubatch-size 1024 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20
```
@@ -105,12 +108,13 @@ llama-server \
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
box with the selected expert offload.
2. The provisional context pool contains 491520 cells. Three slots receive
163840 cells each, which is 96K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service currently prefers the
explicit tensor override because the 16K benchmark found this hybrid
placement fastest. It preserves the final eight blocks plus full-attention
blocks 27 and 31 on CUDA.
2. The context pool contains 491520 cells. Three slots receive 163840 cells
each, which is 96K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
override because the 16K benchmark found the hybrid placement fastest. It
preserves every full-attention expert block and blocks 35–40 on CUDA.
4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the
only active speculative method.
4. The service uses the Qwen-Sharp template from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
`scripts/chat-template.jinja`.
docs/LLM/deployment.md
+7 -8
View File
@@ -125,16 +125,15 @@ The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors selected by an explicit `--override-tensor`
regex. Expert tensors from full-attention blocks 27 and 31 and the
output-adjacent blocks 33 through 40 stay on CUDA; other expert tensors are
CPU-resident. The service uses 7 CPU threads, native `draft-mtp` speculative
decoding (under re-evaluation after an 8K no-spec benchmark), f16 KV, context
491520 (3 slots of 163840 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
regex. Expert tensors from every full-attention block and blocks 35 through 40
stay on CUDA; other expert tensors are CPU-resident. The service uses 7 CPU
threads, ngram-mod speculative decoding only, f16 KV, context 491520 (3 slots
of 163840 each), batch
4096, ubatch 1024, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`. The chat template comes from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
`scripts/frank-default-small.env`. It uses `ubatch=1024` and the chat template
from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
### Dankclaw — frank-default-large
docs/LLM/index.md
+1 -1
View File
@@ -20,7 +20,7 @@ graph LR
B[upstream llama.cpp b10121
frank-default-small, port 7770]
C[Ornith-1.5-35B-A3B-CRACK Q6_K
native MTP, selected experts offloaded via override, 491520 ctx]
ngram-mod, selected experts offloaded via override, 491520 ctx]
A --> B --> C
end
scripts/frank-default-small.env
+15 -15
View File
@@ -30,24 +30,23 @@ LISTEN_PORT=7770
# 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
# cache and checkpoint headroom) = 491520 total context pool. The per-slot
# 163840 will be advertised in models.json after validation.
# 163840 is advertised in models.json.
CTX_SIZE=491520
PARALLEL=3
# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
# (the knee). ngram under MTP hurts; MTP alone is the proven best for MoE.
SPEC_TYPE=draft-mtp
SPEC_DRAFT_N_MAX=4
SPEC_DRAFT_N_MIN=0
SPEC_DRAFT_P_MIN=0.0
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
SPEC_TYPE=ngram-mod
SPEC_NGRAM_MOD_N_MIN=48
SPEC_NGRAM_MOD_N_MAX=64
SPEC_NGRAM_MOD_N_MATCH=24
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the selected
# full-attention blocks 27 and 31 plus output-adjacent blocks 33-40 on CUDA,
# including the bundled MTP block 40. Move all other expert tensors to CPU.
# All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the CPU
# buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
# layer audit found 630 MiB of expert tensors per block. Keep every
# full-attention block plus every block 35-40 on CUDA. The unique GPU set is
# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
# CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all
@@ -59,9 +58,10 @@ VERBOSITY=4
# Store context checkpoints in RAM to hold peak GPU memory down.
CTX_CHECKPOINTS=16
# Big prefill batches.
# Prefill batch sizing. ubatch 1024 is selected for reduced compute-memory
# pressure and predictable latency under the ngram-mod deployment.
BATCH_SIZE=4096
UBATCH_SIZE=2048
UBATCH_SIZE=1024
# Eager RAM load (expert weights on CPU; ~31 GB RAM).
NO_MMAP=1
scripts/frank-default-small.service
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K MTP + Qwen-Sharp template (dankserver, frank-default-small)
Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K ngram-mod + Qwen-Sharp template (dankserver, frank-default-small)
After=network-online.target
Wants=network-online.target
scripts/serve-frank-small.sh
+13 -15
View File
@@ -4,17 +4,15 @@
# experts on CPU).
#
# Runtime: upstream llama.cpp llama-server (b10121). The model is a 35B-total /
# 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF.
# 3B-active MoE-dense hybrid with an MTP head bundled in the GGUF; this
# service uses ngram-mod instead of MTP speculation.
#
# Expert offload: the expert (FFN exps) tensors selected by the layer-bounded
# --override-tensor pattern are pinned to CPU. Selected full-attention and
# final output-adjacent blocks stay on CUDA with all non-expert tensors,
# including the MTP head.
# without relying on MTP speculation.
#
# Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is
# auto-detected from the target model; b10121 creates the MTP draft context
# against the model file). ngram under MTP hurts (measured on the Qwen3.8
# family), so MTP alone is kept. n_max 4 is the knee.
# Speculative decoding: ngram-mod only. MTP is intentionally disabled.
#
# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
# overrides that path.
@@ -40,21 +38,21 @@ fi
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=draft-mtp}"
: "${SPEC_DRAFT_N_MAX:=4}"
: "${SPEC_DRAFT_N_MIN:=0}"
: "${SPEC_DRAFT_P_MIN:=0.0}"
: "${SPEC_TYPE:=ngram-mod}"
: "${SPEC_NGRAM_MOD_N_MIN:=48}"
: "${SPEC_NGRAM_MOD_N_MAX:=64}"
: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${VERBOSITY:=4}"
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
# MoE expert weights and leave every other tensor on the GPU.
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_BUFT:=CPU}"
: "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=1024}"
: "${CTX_CHECKPOINTS:=16}"
: "${NO_MMAP:=1}"
: "${TEMPERATURE:=1.0}"
@@ -83,9 +81,9 @@ exec "$LLAMA_SERVER" \
--mmproj "$MMPROJ" \
--chat-template-file "$CHAT_TEMPLATE_FILE" \
--spec-type "$SPEC_TYPE" \
--spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
--spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
--spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
--spec-ngram-mod-n-min "$SPEC_NGRAM_MOD_N_MIN" \
--spec-ngram-mod-n-max "$SPEC_NGRAM_MOD_N_MAX" \
--spec-ngram-mod-n-match "$SPEC_NGRAM_MOD_N_MATCH" \
--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
tests/frank_small_config.bats
+32 -9
View File
@@ -20,15 +20,38 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -q '^PARALLEL=3$' "$ENV_FILE"
}
@test "frank-default-small uses the measured hybrid expert placement" {
override=$(grep '^OVERRIDE_EXPERT=' "$ENV_FILE")
[[ "$override" == *'26|28|29|30|32)'* ]]
[[ "$override" != *'27'* ]]
[[ "$override" != *'31'* ]]
[[ "$override" != *'33'* ]]
[[ "$override" != *'40'* ]]
grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE"
grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE"
@test "frank-default-small keeps full-attention and final blocks on CUDA" {
source "$ENV_FILE"
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
[[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|15|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|19|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|23|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|27|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|31|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
[[ "$OVERRIDE_BUFT" == CPU ]]
[[ "$N_GPU_LAYERS" == all ]]
}
@test "frank-default-small uses ngram-mod without MTP" {
grep -q '^SPEC_TYPE=ngram-mod$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_MOD_N_MIN=48$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_MOD_N_MAX=64$' "$ENV_FILE"
grep -q '^SPEC_NGRAM_MOD_N_MATCH=24$' "$ENV_FILE"
! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
grep -q -- '--spec-ngram-mod-n-max' "$SERVE_FILE"
}
@test "frank-default-small uses ubatch 1024" {
grep -q '^BATCH_SIZE=4096$' "$ENV_FILE"
grep -q '^UBATCH_SIZE=1024$' "$ENV_FILE"
grep -q ': "${UBATCH_SIZE:=1024}"' "$SERVE_FILE"
}
@test "frank-default-small uses the tracked Qwen-Sharp template" {