Add recommended sampling defaults and reduce MoE GPU offload on dankserver
- frank-default-large: add TEMP=1.0, TOP_K=20, TOP_P=0.95, MIN_P=0.0 (per HauhauCS Qwen3.6-27B model card, thinking-general mode) - frank-default-small: add same sampling defaults (per HauhauCS Qwen3.6-35B-A3B model card) - frank-default-small: reduce N_CPU_MOE from 33 to 34 (6 expert layers on GPU instead of 7) to prevent OOM during high-load sessions - Both serve scripts pass --temp, --top-k, --top-p, --min-p to llama-server; clients can still override per-request via API
This commit is contained in:
1 parent
334b6af2ae
commit
db3eb01495
4 files changed
+36
-6
No files matched your search
@@ -32,12 +32,16 @@ fi
|
||||
: "${NGRAM_MOD_N_MATCH:=24}"
|
||||
: "${CACHE_TYPE_K:=q8_0}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${N_CPU_MOE:=40}" # all 40 layers: MoE expert weights stay on CPU
|
||||
: "${N_CPU_MOE:=34}" # 34 of 40 layers: MoE expert weights stay on CPU; 6 on GPU
|
||||
: "${N_GPU_LAYERS:=all}" # every non-expert tensor goes to GPU
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${CTX_CHECKPOINTS:=0}"
|
||||
: "${NO_MMAP:=0}"
|
||||
: "${TEMP:=1.0}"
|
||||
: "${TOP_K:=20}"
|
||||
: "${TOP_P:=0.95}"
|
||||
: "${MIN_P:=0.0}"
|
||||
: "${THREADS:=8}" # 7700X physical cores
|
||||
: "${THREADS_HTTP:=8}"
|
||||
|
||||
@@ -70,6 +74,10 @@ exec "$LLAMA_SERVER" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--ctx-checkpoints "$CTX_CHECKPOINTS" \
|
||||
$([ "$NO_MMAP" = "1" ] && printf -- "--no-mmap") \
|
||||
--temp "$TEMP" \
|
||||
--top-k "$TOP_K" \
|
||||
--top-p "$TOP_P" \
|
||||
--min-p "$MIN_P" \
|
||||
--threads "$THREADS" \
|
||||
--threads-http "$THREADS_HTTP" \
|
||||
--alias "$ALIAS" \
|
||||
|
||||
Reference in new issue
Block a user