From 7a17984dbc3bac1b20177f93ba33d81e2a185503 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 25 Aug 2026 00:31:05 -0700 Subject: [PATCH] Tune small model expert placement --- docs/LLM/Dankserver System Specs.md | 6 +++--- docs/LLM/dealign-ornith-crack.md | 12 ++++++------ docs/LLM/deployment.md | 4 ++-- scripts/frank-default-small.env | 6 +++--- tests/frank_small_config.bats | 4 ++-- 5 files changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/LLM/Dankserver System Specs.md b/docs/LLM/Dankserver System Specs.md index 3cfeebb..887d5fa 100644 --- a/docs/LLM/Dankserver System Specs.md +++ b/docs/LLM/Dankserver System Specs.md @@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0 ### Active Services -The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 36 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. +The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. ## Hardware Relevance -- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 37 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal. -- 62 GB system RAM holds the ~22.76 GiB of CPU expert tensors with room to spare. +- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal. +- 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare. - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). diff --git a/docs/LLM/dealign-ornith-crack.md b/docs/LLM/dealign-ornith-crack.md index 75822b2..939c265 100644 --- a/docs/LLM/dealign-ornith-crack.md +++ b/docs/LLM/dealign-ornith-crack.md @@ -46,16 +46,16 @@ ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block. The service keeps all non-expert tensors on CUDA. It also keeps expert tensors -from blocks 37 through 40 on CUDA. Those blocks sit next to the output and -include the bundled MTP head. Expert tensors from blocks 0 through 36 move to +from blocks 36 through 40 on CUDA. Those blocks sit next to the output and +include the bundled MTP head. Expert tensors from blocks 0 through 35 move to CPU through this explicit override: ``` -OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU ``` -This places 3.08 GiB of expert weights on CUDA and 22.76 GiB on CPU. The +This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The selection leaves the output-adjacent blocks and MTP path on CUDA while the f16 context cache uses the remaining GPU budget. @@ -87,7 +87,7 @@ llama-server \ --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \ --chat-template-file chat-template.jinja \ --spec-type draft-mtp \ - --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ + --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ -ngl all -c 589824 -np 3 \ --cache-type-k f16 --cache-type-v f16 --no-mmap \ --temp 1.0 --top-p 0.95 --top-k 20 @@ -100,7 +100,7 @@ llama-server \ 2. The context pool contains 589824 cells. Three slots receive 196608 cells each, which is 64K below the model context per stream. 3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor - override because it preserves the final four expert blocks on CUDA. + override because it preserves the final five expert blocks on CUDA. 4. The service uses the Qwen-Sharp template from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through `scripts/chat-template.jinja`. diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index 783a6f2..8a2f9de 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -124,8 +124,8 @@ curl -sf http://127.0.0.1:7770/health The active service on Dankserver is upstream llama.cpp (b10121) serving `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small` on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server` -binary with expert tensors from blocks 0 through 36 pinned to CPU by an -explicit `--override-tensor` regex. Expert tensors from blocks 37 through 40 +binary with expert tensors from blocks 0 through 35 pinned to CPU by an +explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40 stay on CUDA. The service uses 7 CPU threads, native `draft-mtp` speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch 4096, ubatch 2048, and `--no-mmap`. The unit is diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 14a9c08..2b0a3e6 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K # layer audit found 630 MiB of expert tensors per block. Keep the four -# output-adjacent blocks 37-40 on CUDA, including the bundled MTP block 40. -# Move expert tensors from blocks 0-36 to CPU. All non-expert tensors stay on +# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40. +# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on # the 24 GB card. Buffer type "CPU" is the CPU buft name on this host. -OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU N_GPU_LAYERS=all diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index ec7aae4..b2ccd94 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -20,9 +20,9 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" grep -q '^PARALLEL=3$' "$ENV_FILE" } -@test "frank-default-small keeps final four expert blocks on CUDA" { +@test "frank-default-small keeps final five expert blocks on CUDA" { grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE" - grep -Fq '35|36)\.ffn_' "$ENV_FILE" + grep -Fq '34|35)\.ffn_' "$ENV_FILE" grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE" grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE" }