dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce small model context and rebalance GPU layers

Browse Source
This commit is contained in:
Frank
2026-08-25 00:59:41 -07:00
parent 7a17984dbc
commit 339325719e
7 changed files with 38 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/LLM/Dankserver System Specs.md
+3 -3
View File
@@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
### Active Services
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 35 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 32 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
## Hardware Relevance
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 36 through 40. The new context pool contains 589824 f16 cells across three 196608-token slots. The configuration targets about 1.5 to 2 GiB of free VRAM for checkpoint overhead. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the ~22.13 GiB of CPU expert tensors with room to spare.
- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 33 through 40. The provisional context pool contains 491520 f16 cells across three 163840-token slots. The configuration will be rebalanced after measuring the newly available headroom. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds approximately 19.67 GiB of CPU expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
docs/LLM/dealign-ornith-crack.md
+17 -14
View File
@@ -45,19 +45,21 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
block 40. The expert portion is 630 MiB in every block.
The service keeps all non-expert tensors on CUDA. It also keeps expert tensors
from blocks 36 through 40 on CUDA. Those blocks sit next to the output and
include the bundled MTP head. Expert tensors from blocks 0 through 35 move to
CPU through this explicit override:
The provisional service keeps all non-expert tensors on CUDA. It also keeps
expert tensors from blocks 33 through 40 on CUDA. Those blocks sit next to the
output and include the bundled MTP head. Expert tensors from blocks 0 through
32 move to CPU through this explicit override:
```
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
```
This places 3.69 GiB of expert weights on CUDA and 22.13 GiB on CPU. The
selection leaves the output-adjacent blocks and MTP path on CUDA while the
f16 context cache uses the remaining GPU budget.
The previous five-block configuration placed 3.69 GiB of expert weights on
CUDA and 22.13 GiB on CPU. This provisional eight-block configuration adds
approximately 1.89 GiB of expert weights to CUDA while reducing the context
pool by 98304 cells. The final placement will be selected after measuring this
new headroom and discussing layer priorities.
To list the buffer type names on the host, run a bogus override:
@@ -87,8 +89,8 @@ llama-server \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \
--spec-type draft-mtp \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 589824 -np 3 \
--override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 491520 -np 3 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20
```
@@ -97,10 +99,11 @@ llama-server \
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
box with the selected expert offload.
2. The context pool contains 589824 cells. Three slots receive 196608 cells
each, which is 64K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
override because it preserves the final five expert blocks on CUDA.
2. The provisional context pool contains 491520 cells. Three slots receive
163840 cells each, which is 96K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service currently prefers the
explicit tensor override because it preserves the final eight expert
blocks on CUDA pending the next layer-priority decision.
4. The service uses the Qwen-Sharp template from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
`scripts/chat-template.jinja`.
docs/LLM/deployment.md
+3 -3
View File
@@ -124,10 +124,10 @@ curl -sf http://127.0.0.1:7770/health
The active service on Dankserver is upstream llama.cpp (b10121) serving
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small`
on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server`
binary with expert tensors from blocks 0 through 35 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 36 through 40
binary with expert tensors from blocks 0 through 32 pinned to CPU by an
explicit `--override-tensor` regex. Expert tensors from blocks 33 through 40
stay on CUDA. The service uses 7 CPU threads, native `draft-mtp`
speculative decoding, f16 KV, context 589824 (3 slots of 196608 each), batch
speculative decoding, f16 KV, context 491520 (3 slots of 163840 each), batch
4096, ubatch 2048, and `--no-mmap`. The unit is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
docs/LLM/index.md
+1 -1
View File
@@ -20,7 +20,7 @@ graph LR
B[upstream llama.cpp b10121
frank-default-small, port 7770]
C[Ornith-1.5-35B-A3B-CRACK Q6_K
native MTP, selected experts offloaded via override, 589824 ctx]
native MTP, selected experts offloaded via override, 491520 ctx]
A --> B --> C
end
scripts/frank-default-small.env
+7 -7
View File
@@ -28,10 +28,10 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# 3 parallel slots x 196608 ctx each (262144 lowered by 64k per stream for
# cache and checkpoint headroom) = 589824 total context pool. The per-slot
# 196608 matches the per-slot context advertised in models.json (196608).
CTX_SIZE=589824
# 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
# cache and checkpoint headroom) = 491520 total context pool. The per-slot
# 163840 will be advertised in models.json after validation.
CTX_SIZE=491520
PARALLEL=3
# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all
scripts/serve-frank-small.sh
+2 -2
View File
@@ -37,7 +37,7 @@ fi
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=786432}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=draft-mtp}"
: "${SPEC_DRAFT_N_MAX:=4}"
@@ -49,7 +49,7 @@ fi
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
# MoE expert weights and leave every other tensor on the GPU.
: "${OVERRIDE_EXPERT:=\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_BUFT:=CPU}"
: "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=4096}"
tests/frank_small_config.bats
+5 -5
View File
@@ -14,15 +14,15 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q6_K\.gguf' "$SERVE_FILE"
}
@test "frank-default-small uses 196608 context per slot across three slots" {
grep -q '^CTX_SIZE=589824$' "$ENV_FILE"
grep -q '196608 ctx each' "$ENV_FILE"
@test "frank-default-small uses 163840 context per slot across three slots" {
grep -q '^CTX_SIZE=491520$' "$ENV_FILE"
grep -q '163840 ctx each' "$ENV_FILE"
grep -q '^PARALLEL=3$' "$ENV_FILE"
}
@test "frank-default-small keeps final five expert blocks on CUDA" {
@test "frank-default-small keeps final eight expert blocks on CUDA" {
grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE"
grep -Fq '34|35)\.ffn_' "$ENV_FILE"
grep -Fq '31|32)\.ffn_' "$ENV_FILE"
grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE"
grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE"
}