Add recommended sampling defaults and reduce MoE GPU offload on dankserver
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0 (per HauhauCS Qwen3.6-27B model card, thinking-general mode) - frank-default-small: add same sampling defaults (per HauhauCS Qwen3.6-35B-A3B model card) - frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert layers on GPU instead of 7) to prevent OOM during high-load sessions - Both serve scripts pass --temp, --top-k, --top-p, --min-p to llama-server; clients can still override per-request via API
This commit is contained in:
1 parent
334b6af2ae
commit
db3eb01495
4 files changed
+36
-6
No files matched your search
@@ -33,5 +33,12 @@ CACHE_TYPE_V=q8_0
|
||||
# User-specified ubatch 4096 / batch 12288 is not feasible here (see bench).
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
# Sampling settings — per HauhauCS Qwen3.6-27B model card.
|
||||
# Thinking-general: temp=1.0, top_p=0.95, top_k=20, min_p=0.
|
||||
# Clients may override per-request; these are the server-side defaults.
|
||||
TEMP=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
MIN_P=0.0
|
||||
THREADS=12
|
||||
THREADS_HTTP=1
|
||||
Reference in new issue
Block a user