Reduce small model context and rebalance GPU layers

docs/LLM/Dankserver System Specs.md

@@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
 
 ### Active Services
 
-The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
+The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 32 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
 
 ## Hardware Relevance
 
-- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
+- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 33 through 40. The provisional context pool contains 491520 f16 cells across three 163840-token slots. The configuration will be rebalanced after measuring the newly available headroom. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
-- 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare.
+- 62 GB system RAM holds approximately 19.67 GiB of CPU expert tensors with room to spare.
 - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
 - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
 

docs/LLM/dealign-ornith-crack.md

@@ -45,19 +45,21 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
 ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
 block 40. The expert portion is 630 MiB in every block.
 
-The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
+The provisional service keeps all non-expert tensors on CUDA. It also keeps
-from blocks 36 through 40 on CUDA. Those blocks sit next to the output and
+expert tensors from blocks 33 through 40 on CUDA. Those blocks sit next to the
-include the bundled MTP head. Expert tensors from blocks 0 through 35 move to
+output and include the bundled MTP head. Expert tensors from blocks 0 through
-CPU through this explicit override:
+32 move to CPU through this explicit override:
 
 ```
-OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
+OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 ```
 
-This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The
+The previous five-block configuration placed 3.69 GiB of expert weights on
-selection leaves the output-adjacent blocks and MTP path on CUDA while the
+CUDA and 22.13 GiB on CPU. This provisional eight-block configuration adds
-f16 context cache uses the remaining GPU budget.
+approximately 1.89 GiB of expert weights to CUDA while reducing the context
+pool by 98304 cells. The final placement will be selected after measuring this
+new headroom and discussing layer priorities.
 
 To list the buffer type names on the host, run a bogus override:
 
@@ -87,8 +89,8 @@ llama-server \
   --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
   --chat-template-file chat-template.jinja \
   --spec-type draft-mtp \
-  --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
+  --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-  -ngl all -c 589824 -np 3 \
+  -ngl all -c 491520 -np 3 \
   --cache-type-k f16 --cache-type-v f16 --no-mmap \
   --temp 1.0 --top-p 0.95 --top-k 20
 ```
@@ -97,10 +99,11 @@ llama-server \
 
 1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
    box with the selected expert offload.
-2. The context pool contains 589824 cells. Three slots receive 196608 cells
+2. The provisional context pool contains 491520 cells. Three slots receive
-   each, which is 64K below the model context per stream.
+   163840 cells each, which is 96K below the model context per stream.
-3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
+3. `--n-cpu-moe` remains supported. This service currently prefers the
-   override because it preserves the final five expert blocks on CUDA.
+   explicit tensor override because it preserves the final eight expert
+   blocks on CUDA pending the next layer-priority decision.
 4. The service uses the Qwen-Sharp template from
    `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
    `scripts/chat-template.jinja`.

docs/LLM/deployment.md

@@ -124,10 +124,10 @@ curl -sf http://127.0.0.1:7770/health
 The active service on Dankserver is upstream llama.cpp (b10121) serving
 `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
 on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
-binary with expert tensors from blocks 0 through 35 pinned to CPU by an
+binary with expert tensors from blocks 0 through 32 pinned to CPU by an
-explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
+explicit `--override-tensor` regex. Expert tensors from blocks 33 through 40
 stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
-speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
+speculative decoding, f16 KV, context 491520 (3 slots of 163840 each), batch
 4096, ubatch 2048, and `--no-mmap`. The unit is
 `scripts/frank-default-small.service`. It launches through
 `scripts/serve-frank-small.sh`, which reads tunables from

docs/LLM/index.md

@@ -20,7 +20,7 @@ graph LR
         B[upstream llama.cpp b10121
 frank-default-small, port 7770]
         C[Ornith-1.5-35B-A3B-CRACK Q6_K
-native MTP, selected experts offloaded via override, 589824 ctx]
+native MTP, selected experts offloaded via override, 491520 ctx]
         A --> B --> C
     end
 

scripts/frank-default-small.env

@@ -28,10 +28,10 @@ ALIAS=frank-default-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770
 
-# 3 parallel slots x 196608 ctx each (262144 lowered by 64k per stream for
+# 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
-# cache and checkpoint headroom) = 589824 total context pool. The per-slot
+# cache and checkpoint headroom) = 491520 total context pool. The per-slot
-# 196608 matches the per-slot context advertised in models.json (196608).
+# 163840 will be advertised in models.json after validation.
-CTX_SIZE=589824
+CTX_SIZE=491520
 PARALLEL=3
 
 # Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
 
 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
 # layer audit found 630 MiB of expert tensors per block. Keep the four
-# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
+# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40.
-# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
+# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on
 # the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
-OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
+OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
 OVERRIDE_BUFT=CPU
 N_GPU_LAYERS=all
 

scripts/serve-frank-small.sh

@@ -37,7 +37,7 @@ fi
 : "${ALIAS:=frank-default-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7770}"
-: "${CTX_SIZE:=786432}"
+: "${CTX_SIZE:=491520}"
 : "${PARALLEL:=3}"
 : "${SPEC_TYPE:=draft-mtp}"
 : "${SPEC_DRAFT_N_MAX:=4}"
@@ -49,7 +49,7 @@ fi
 # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
 # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
 # MoE expert weights and leave every other tensor on the GPU.
-: "${OVERRIDE_EXPERT:=\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
+: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
 : "${OVERRIDE_BUFT:=CPU}"
 : "${N_GPU_LAYERS:=all}"
 : "${BATCH_SIZE:=4096}"

tests/frank_small_config.bats

@@ -14,15 +14,15 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
     grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q6_K\.gguf' "$SERVE_FILE"
 }
 
-@test "frank-default-small uses 196608 context per slot across three slots" {
+@test "frank-default-small uses 163840 context per slot across three slots" {
-    grep -q '^CTX_SIZE=589824$' "$ENV_FILE"
+    grep -q '^CTX_SIZE=491520$' "$ENV_FILE"
-    grep -q '196608 ctx each' "$ENV_FILE"
+    grep -q '163840 ctx each' "$ENV_FILE"
     grep -q '^PARALLEL=3$' "$ENV_FILE"
 }
 
-@test "frank-default-small keeps final five expert blocks on CUDA" {
+@test "frank-default-small keeps final eight expert blocks on CUDA" {
     grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE"
-    grep -Fq '34|35)\.ffn_' "$ENV_FILE"
+    grep -Fq '31|32)\.ffn_' "$ENV_FILE"
     grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE"
     grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE"
 }