dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-default-small: ncmoe 35->36 for prefill stability

Browse Source 
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.

ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
This commit is contained in:
Frank
2026-08-14 10:56:40 -07:00
parent 7daf9896e5
commit 9d7643ec5d
5 changed files with 28 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-small.env
+8 -7
View File
@@ -8,17 +8,18 @@ PORT=7770
# (Increased from 3×200k to add a 4th parallel stream.)
CTX_SIZE=800000
PARALLEL=4
# Spec: ngram-mod. n_min=48, n_max=64, n_match=24 (defaults). 5 expert
# layers back on GPU; 35 on CPU. Batch 8192 / ubatch 3072.
# Spec: ngram-mod. n_min=48, n_max=64, n_match=24 (defaults). 4 expert
# layers back on GPU; 36 on CPU. Batch 8192 / ubatch 3072.
SPEC_TYPE=ngram-mod
NGRAM_MOD_N_MIN=48
NGRAM_MOD_N_MAX=64
NGRAM_MOD_N_MATCH=24
# Expert offload: 35 of 40 layers keep MoE weights on CPU; remaining
# 5 layers have their expert weights resident on GPU for faster routing.
# Benchmark-chosen (see frank-default-small-benchmark.md): prefill-max.
# gen ~45.4 t/s, pp ~1458 t/s, ~1.2 GiB headroom.
N_CPU_MOE=35
# Expert offload: 36 of 40 layers keep MoE weights on CPU; remaining
# 4 layers have their expert weights resident on GPU for faster routing.
# Chosen for prefill-max stability at ubatch 3072: frees ~0.75 GiB peak
# VRAM vs ncmoe 35 (grid: peak 8606 vs 9360 MiB) for -2% prefill, -2%
# decode. Fixes the cuBLAS OOM crash class during big prefills.
N_CPU_MOE=36
N_GPU_LAYERS=all
# KV cache on GPU, K quantized to q8_0, V stays f16 (user-locked decision).
CACHE_TYPE_K=q8_0
scripts/serve-llama-frank-small.sh
+1 -1
View File
@@ -32,7 +32,7 @@ fi
: "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=35}" # 35 of 40 layers: MoE expert weights stay on CPU; 5 on GPU
: "${N_CPU_MOE:=36}" # 36 of 40 layers: MoE expert weights stay on CPU; 4 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=3072}"