Add recommended sampling defaults and reduce MoE GPU offload on dankserver
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0 (per HauhauCS Qwen3.6-27B model card, thinking-general mode) - frank-default-small: add same sampling defaults (per HauhauCS Qwen3.6-35B-A3B model card) - frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert layers on GPU instead of 7) to prevent OOM during high-load sessions - Both serve scripts pass --temp, --top-k, --top-p, --min-p to llama-server; clients can still override per-request via API
This commit is contained in:
1 parent
334b6af2ae
commit
db3eb01495
4 files changed
+36
-6
No files matched your search
@@ -37,6 +37,10 @@ fi
|
||||
: "${CACHE_TYPE_V:=q8_0}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${TEMP:=1.0}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.95}"
|
||||
: "${MIN_P:=0.0}"
|
||||
: "${THREADS:=12}"
|
||||
: "${THREADS_HTTP:=1}"
|
||||
|
||||
@@ -68,6 +72,10 @@ exec "$LLAMA_SERVER" \
|
||||
--parallel "$PARALLEL" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--temp "$TEMP" \
|
||||
--top-k "$TOP_K" \
|
||||
--top-p "$TOP_P" \
|
||||
--min-p "$MIN_P" \
|
||||
--threads "$THREADS" \
|
||||
--threads-http "$THREADS_HTTP" \
|
||||
--alias "$ALIAS" \
|
||||
|
||||
Reference in new issue
Block a user