dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce small model context and rebalance GPU layers

Browse Source
This commit is contained in:
Frank
2026-08-25 00:59:41 -07:00
parent 7a17984dbc
commit 339325719e
7 changed files with 38 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+7 -7
View File
@@ -28,10 +28,10 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# 3 parallel slots x 196608 ctx each (262144 lowered by 64k per stream for
# cache and checkpoint headroom) = 589824 total context pool. The per-slot
# 196608 matches the per-slot context advertised in models.json (196608).
CTX_SIZE=589824
# 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
# cache and checkpoint headroom) = 491520 total context pool. The per-slot
# 163840 will be advertised in models.json after validation.
CTX_SIZE=491520
PARALLEL=3
# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all