dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Tune small model expert placement

Browse Source
This commit is contained in:
Frank
2026-08-25 00:31:05 -07:00
parent af6ffeeb64
commit 7a17984dbc
5 changed files with 16 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/Dankserver System Specs.md
+3 -3
View File
@@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 36 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 37 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.76 GiB of CPU expert tensors with room to spare.
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
docs/LLM/dealign-ornith-crack.md
+6 -6
View File
@@ -46,16 +46,16 @@ ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
block 40. The expert portion is 630 MiB in every block.
The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
from blocks 37 through 40 on CUDA. Those blocks sit next to the output and
include the bundled MTP head. Expert tensors from blocks 0 through 36 move to
from blocks 36 through 40 on CUDA. Those blocks sit next to the output and
include the bundled MTP head. Expert tensors from blocks 0 through 35 move to
CPU through this explicit override:
```
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
```
This places 3.08 GiB of expert weights on CUDA and 22.76 GiB on CPU. The
This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The
selection leaves the output-adjacent blocks and MTP path on CUDA while the
f16 context cache uses the remaining GPU budget.
@@ -87,7 +87,7 @@ llama-server \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \
--spec-type draft-mtp \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 589824 -np 3 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20
@@ -100,7 +100,7 @@ llama-server \
2. The context pool contains 589824 cells. Three slots receive 196608 cells
each, which is 64K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
override because it preserves the final four expert blocks on CUDA.
override because it preserves the final five expert blocks on CUDA.
4. The service uses the Qwen-Sharp template from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
`scripts/chat-template.jinja`.
docs/LLM/deployment.md
+2 -2
View File
@@ -124,8 +124,8 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors from blocks 0 through 36 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 37 through 40
binary with expert tensors from blocks 0 through 35 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
scripts/frank-default-small.env
+3 -3
View File
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 37-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-36 to CPU. All non-expert tensors stay on
# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all
tests/frank_small_config.bats
+2 -2
View File
@@ -20,9 +20,9 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -q '^PARALLEL=3$' "$ENV_FILE"
}
@test "frank-default-small keeps final four expert blocks on CUDA" {
@test "frank-default-small keeps final five expert blocks on CUDA" {
grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE"
grep -Fq '35|36)\.ffn_' "$ENV_FILE"
grep -Fq '34|35)\.ffn_' "$ENV_FILE"
grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE"
grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE"
}