Offload one more MoE expert to CPU on frank-default-small (34→35 of 40)

- N_CPU_MOE 34→35: 5 expert layers on GPU instead of 6. Frees ~0.4-0.8 GiB
  VRAM for OOM safety after high-load sessions OOM'd at 32 and 33.
- Sync stale doc references (deployment.md, Dankserver System Specs.md)
  that still claimed 32/8; specs now estimate ~1.8 GiB headroom at 35.
This commit is contained in:
Frank committed 2026-08-13 20:18:20 -07:00
1 parent c96cb6a529
commit 209223cb4f
4 files changed
+9 -9

No files matched your search

+1 -1
View File
@@ -32,7 +32,7 @@ fi
: "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=34}" # 34 of 40 layers: MoE expert weights stay on CPU; 6 on GPU
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"