Add recommended sampling defaults and reduce MoE GPU offload on dankserver

- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
  (per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
  (per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
  layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
  llama-server; clients can still override per-request via API
This commit is contained in:
Frank committed 2026-08-13 16:18:47 -07:00
1 parent 334b6af2ae
commit db3eb01495
4 files changed
+36 -6

No files matched your search

+9 -1
View File
@@ -32,12 +32,16 @@ fi
: "${NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=f16}"
: "${N_CPU_MOE:=40}" # all 40 layers: MoE expert weights stay on CPU
: "${N_CPU_MOE:=34}" # 34 of 40 layers: MoE expert weights stay on CPU; 6 on GPU
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${CTX_CHECKPOINTS:=0}"
: "${NO_MMAP:=0}"
: "${TEMP:=1.0}"
: "${TOP_K:=20}"
: "${TOP_P:=0.95}"
: "${MIN_P:=0.0}"
: "${THREADS:=8}" # 7700X physical cores
: "${THREADS_HTTP:=8}"
@@ -70,6 +74,10 @@ exec "$LLAMA_SERVER" \
--ubatch-size "$UBATCH_SIZE" \
--ctx-checkpoints "$CTX_CHECKPOINTS" \
$([ "$NO_MMAP" = "1" ] && printf -- "--no-mmap") \
--temp "$TEMP" \
--top-k "$TOP_K" \
--top-p "$TOP_P" \
--min-p "$MIN_P" \
--threads "$THREADS" \
--threads-http "$THREADS_HTTP" \
--alias "$ALIAS" \