frank-default-small: ncmoe 35->36 for prefill stability

Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.

ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
This commit is contained in:
Frank committed 2026-08-14 10:56:40 -07:00
1 parent 7daf9896e5
commit 9d7643ec5d
5 files changed
+28 -20

No files matched your search

+1 -1
View File
@@ -32,7 +32,7 @@ fi
: "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
: "${N_CPU_MOE:=36}" # 36 of 40 layers: MoE expert weights stay on CPU; 4 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=3072}"