dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce small model context and rebalance GPU layers

Browse Source
This commit is contained in:
Frank
2026-08-25 00:59:41 -07:00
parent 7a17984dbc
commit 339325719e
7 changed files with 38 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/Dankserver System Specs.md
+3 -3
View File
@@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 32 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare.
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 33 through 40. The provisional context pool contains 491520 f16 cells across three 163840-token slots. The configuration will be rebalanced after measuring the newly available headroom. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds approximately 19.67 GiB of CPU expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
docs/LLM/dealign-ornith-crack.md
+17 -14
View File
@@ -45,19 +45,21 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
block 40. The expert portion is 630 MiB in every block.
The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
from blocks 36 through 40 on CUDA. Those blocks sit next to the output and
include the bundled MTP head. Expert tensors from blocks 0 through 35 move to
CPU through this explicit override:
The provisional service keeps all non-expert tensors on CUDA. It also keeps
expert tensors from blocks 33 through 40 on CUDA. Those blocks sit next to the
output and include the bundled MTP head. Expert tensors from blocks 0 through
32 move to CPU through this explicit override:
```
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
```
This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The
selection leaves the output-adjacent blocks and MTP path on CUDA while the
f16 context cache uses the remaining GPU budget.
The previous five-block configuration placed 3.69 GiB of expert weights on
CUDA and 22.13 GiB on CPU. This provisional eight-block configuration adds
approximately 1.89 GiB of expert weights to CUDA while reducing the context
pool by 98304 cells. The final placement will be selected after measuring this
new headroom and discussing layer priorities.
To list the buffer type names on the host, run a bogus override:
@@ -87,8 +89,8 @@ llama-server \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \
--spec-type draft-mtp \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 589824 -np 3 \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 491520 -np 3 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20
```
@@ -97,10 +99,11 @@ llama-server \
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
box with the selected expert offload.
2. The context pool contains 589824 cells. Three slots receive 196608 cells
each, which is 64K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
override because it preserves the final five expert blocks on CUDA.
2. The provisional context pool contains 491520 cells. Three slots receive
163840 cells each, which is 96K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service currently prefers the
explicit tensor override because it preserves the final eight expert
blocks on CUDA pending the next layer-priority decision.
4. The service uses the Qwen-Sharp template from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
`scripts/chat-template.jinja`.
docs/LLM/deployment.md
+3 -3
View File
@@ -124,10 +124,10 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors from blocks 0 through 35 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
binary with expert tensors from blocks 0 through 32 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 33 through 40
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
speculative decoding, f16 KV, context 491520 (3 slots of 163840 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
docs/LLM/index.md
+1 -1
View File
@@ -20,7 +20,7 @@ graph LR
B[upstream llama.cpp b10121
frank-default-small, port 7770]
C[Ornith-1.5-35B-A3B-CRACK Q6_K
native MTP, selected experts offloaded via override, 589824 ctx]
native MTP, selected experts offloaded via override, 491520 ctx]
A --> B --> C
end