dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Tune small model expert placement

Browse Source
This commit is contained in:
Frank committed 2026-08-25 00:31:05 -07:00
1 parent af6ffeeb64
commit 7a17984dbc
5 files changed
+16 -16
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/LLM/Dankserver System Specs.md
+3 -3
View File
@@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services ### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 36 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance ## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 37 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal. - 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.76 GiB of CPU expert tensors with room to spare. - 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
docs/LLM/dealign-ornith-crack.md
+6 -6
View File
@@ -46,16 +46,16 @@ ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
block 40. The expert portion is 630 MiB in every block. block 40. The expert portion is 630 MiB in every block.
The service keeps all non-expert tensors on CUDA. It also keeps expert tensors The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
from blocks 37 through 40 on CUDA. Those blocks sit next to the output and from blocks 36 through 40 on CUDA. Those blocks sit next to the output and
include the bundled MTP head. Expert tensors from blocks 0 through 36 move to include the bundled MTP head. Expert tensors from blocks 0 through 35 move to
CPU through this explicit override: CPU through this explicit override:
``` ```
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU OVERRIDE_BUFT=CPU
``` ```
This places 3.08 GiB of expert weights on CUDA and 22.76 GiB on CPU. The This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The
selection leaves the output-adjacent blocks and MTP path on CUDA while the selection leaves the output-adjacent blocks and MTP path on CUDA while the
f16 context cache uses the remaining GPU budget. f16 context cache uses the remaining GPU budget.
@@ -87,7 +87,7 @@ llama-server \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \ --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \ --chat-template-file chat-template.jinja \
--spec-type draft-mtp \ --spec-type draft-mtp \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 589824 -np 3 \ -ngl all -c 589824 -np 3 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \ --cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20 --temp 1.0 --top-p 0.95 --top-k 20
@@ -100,7 +100,7 @@ llama-server \
2. The context pool contains 589824 cells. Three slots receive 196608 cells 2. The context pool contains 589824 cells. Three slots receive 196608 cells
each, which is 64K below the model context per stream. each, which is 64K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor 3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
override because it preserves the final four expert blocks on CUDA. override because it preserves the final five expert blocks on CUDA.
4. The service uses the Qwen-Sharp template from 4. The service uses the Qwen-Sharp template from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
`scripts/chat-template.jinja`. `scripts/chat-template.jinja`.
docs/LLM/deployment.md
+2 -2
View File
@@ -124,8 +124,8 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankserver is upstream llama.cpp (b10121) serving The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small` `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server` on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors from blocks 0 through 36 pinned to CPU by an binary with expert tensors from blocks 0 through 35 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 37 through 40 explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp` stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is 4096, ubatch 2048, and `--no-mmap`. The unit is
scripts/frank-default-small.env
+3 -3
View File
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four # layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 37-40 on CUDA, including the bundled MTP block 40. # output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-36 to CPU. All non-expert tensors stay on # Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host. # the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all N_GPU_LAYERS=all
tests/frank_small_config.bats
+2 -2
View File
@@ -20,9 +20,9 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -q '^PARALLEL=3$' "$ENV_FILE" grep -q '^PARALLEL=3$' "$ENV_FILE"
} }
@test "frank-default-small keeps final four expert blocks on CUDA" { @test "frank-default-small keeps final five expert blocks on CUDA" {
grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE" grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE"
grep -Fq '35|36)\.ffn_' "$ENV_FILE" grep -Fq '34|35)\.ffn_' "$ENV_FILE"
grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE" grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE"
grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE" grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE"
} }