dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce small model context and rebalance GPU layers

Browse Source
This commit is contained in:
Frank
2026-08-25 00:59:41 -07:00
parent 7a17984dbc
commit 339325719e
7 changed files with 38 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+7 -7
View File
@@ -28,10 +28,10 @@ ALIAS=frank-default-small
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# 3 parallel slots x 196608 ctx each (262144 lowered by 64k per stream for
# cache and checkpoint headroom) = 589824 total context pool. The per-slot
# 196608 matches the per-slot context advertised in models.json (196608).
CTX_SIZE=589824
# 3 parallel slots x 163840 ctx each (262144 lowered by 96k per stream for
# cache and checkpoint headroom) = 491520 total context pool. The per-slot
# 163840 will be advertised in models.json after validation.
CTX_SIZE=491520
PARALLEL=3
# Spec: MoE model with a bundled MTP head -> native draft-mtp, n_max 4
@@ -43,10 +43,10 @@ SPEC_DRAFT_P_MIN=0.0
# Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K
# layer audit found 630 MiB of expert tensors per block. Keep the four
# output-adjacent blocks 36-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-35 to CPU. All non-expert tensors stay on
# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40.
# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on
# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host.
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32|33|34|35)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
N_GPU_LAYERS=all
scripts/serve-frank-small.sh
+2 -2
View File
@@ -37,7 +37,7 @@ fi
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=786432}"
: "${CTX_SIZE:=491520}"
: "${PARALLEL:=3}"
: "${SPEC_TYPE:=draft-mtp}"
: "${SPEC_DRAFT_N_MAX:=4}"
@@ -49,7 +49,7 @@ fi
# Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern
# the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the
# MoE expert weights and leave every other tensor on the GPU.
: "${OVERRIDE_EXPERT:=\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}"
: "${OVERRIDE_BUFT:=CPU}"
: "${N_GPU_LAYERS:=all}"
: "${BATCH_SIZE:=4096}"