Offload one more MoE expert to CPU on frank-default-small (34→35 of 40)
- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB VRAM for OOM safety after high-load sessions OOM'd at 32 and 33. - Sync stale doc references (deployment.md, Dankserver System Specs.md) that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
This commit is contained in:
1 parent
c96cb6a529
commit
209223cb4f
4 files changed
+9
-9
No files matched your search
@@ -32,7 +32,7 @@ fi
|
||||
: "${NGRAM_MOD_N_MATCH:=24}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${N_CPU_MOE:=34}" # 34 of 40 layers: MoE expert weights stay on CPU; 6 on GPU
|
||||
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
|
||||
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
|
||||
Reference in new issue
Block a user