Add recommended sampling defaults and reduce MoE GPU offload on dankserver

- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0
  (per HauhauCS Qwen3.6-27B model card, thinking-general mode)
- frank-default-small: add same sampling defaults
  (per HauhauCS Qwen3.6-35B-A3B model card)
- frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert
  layers on GPU instead of 7) to prevent OOM during high-load sessions
- Both serve scripts pass --temp, --top-k, --top-p, --min-p to
  llama-server; clients can still override per-request via API
This commit is contained in:
Frank committed 2026-08-13 16:18:47 -07:00
1 parent 334b6af2ae
commit db3eb01495
4 files changed
+36 -6

No files matched your search

+7
View File
@@ -33,5 +33,12 @@ CACHE_TYPE_V=q8_0
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
BATCH_SIZE=2048
UBATCH_SIZE=512
# Sampling settings — per HauhauCS Qwen3.6-27B model card.
# Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
# Clients may override per-request; these are the server-side defaults.
TEMP=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0
THREADS=12
THREADS_HTTP=1