Add recommended sampling defaults and reduce MoE GPU offload on dankserver

- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
  (per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
  (per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
  layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
  llama-server; clients can still override per-request via API
This commit is contained in:
Frank committed 2026-08-13 16:18:47 -07:00
1 parent 334b6af2ae
commit db3eb01495
4 files changed
+36 -6

No files matched your search

+8
View File
@@ -37,6 +37,10 @@ fi
: "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${TEMP:=1.0}"
: "${TOP_K:=20}"
: "${TOP_P:=0.95}"
: "${MIN_P:=0.0}"
: "${THREADS:=12}"
: "${THREADS_HTTP:=1}"
@@ -68,6 +72,10 @@ exec "$LLAMA_SERVER" \
--parallel "$PARALLEL" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--temp "$TEMP" \
--top-k "$TOP_K" \
--top-p "$TOP_P" \
--min-p "$MIN_P" \
--threads "$THREADS" \
--threads-http "$THREADS_HTTP" \
--alias "$ALIAS" \