dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Offload one more MoE expert to CPU on frank-default-small (32→33 of 40)

Browse Source
This commit is contained in:
Frank
2026-08-11 13:13:17 -07:00
parent 4c5a4a5f3b
commit df41aaafe7
1 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+3 -3
View File
@@ -14,9 +14,9 @@ SPEC_TYPE=ngram-mod
NGRAM_MOD_N_MIN=48 NGRAM_MOD_N_MIN=48
NGRAM_MOD_N_MAX=64 NGRAM_MOD_N_MAX=64
NGRAM_MOD_N_MATCH=24 NGRAM_MOD_N_MATCH=24
# Expert offload: 32 of 40 layers keep MoE weights on CPU; remaining # Expert offload: 33 of 40 layers keep MoE weights on CPU; remaining
# 8 layers have their expert weights resident on GPU for faster routing. # 7 layers have their expert weights resident on GPU for faster routing.
N_CPU_MOE=32 N_CPU_MOE=33
N_GPU_LAYERS=all N_GPU_LAYERS=all
# KV cache on GPU, K quantized to q8_0, V stays f16 (user-locked decision). # KV cache on GPU, K quantized to q8_0, V stays f16 (user-locked decision).
CACHE_TYPE_K=q8_0 CACHE_TYPE_K=q8_0