dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Tune frank-default-large params

Browse Source 
- frank-default-large.env: CTX_SIZE 262144 -> 196608 (192K context),
  BATCH_SIZE/UBATCH_SIZE 4096 -> 8192, CTX_CHECKPOINTS 16 -> 32, add
  preserve_thinking:true to chat_template_kwargs
- kingjones-strix-mtp.md: sync launch excerpt + note on preserve_thinking
  and the peg-native/deepseek chat_format pairing
This commit is contained in:
Frank
2026-08-25 02:30:25 -07:00
parent cd14d7bf68
commit 7a8500f6fa
2 changed files with 17 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-default-large.env
+10 -8
View File
@@ -29,8 +29,10 @@ MMPROJ_FILE=mmproj-Qwen3.8-27B-Q8_0.gguf
# Qwen-Sharp chat template (tracked in the repo, staged on the host).
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
# enable_thinking on by default in the Sharp template; medium effort default.
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true}'
# enable_thinking + preserve_thinking on by default in the Sharp template;
# medium effort default. preserve_thinking keeps <think> blocks in recorded
# assistant messages (set to false to strip them, true to always keep them).
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv).
ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
@@ -41,16 +43,16 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Serving: full native 262144 context, 1 slot. f16 KV, context checkpoints,
# Serving: 196608 (192K) context, 1 slot. f16 KV, context checkpoints,
# prompt cache, draft-mtp with standalone Q4_0 head, n_max 4 (the knee for
# the dense model). Unified memory; rocm-smi reports the 64 GiB budget.
CTX_SIZE=262144
CTX_SIZE=196608
PARALLEL=1
BATCH_SIZE=4096
UBATCH_SIZE=4096
BATCH_SIZE=8192
UBATCH_SIZE=8192
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
CTX_CHECKPOINTS=16
CTX_CHECKPOINTS=32
CACHE_RAM=8192
# Dense-model MTP draft (standalone Q4_0 head, n_max 4). No PromptForge .pfs
@@ -70,4 +72,4 @@ TOP_P=0.95
REASONING=on
REASONING_FORMAT=deepseek
VERBOSITY=4
VERBOSITY=4