Tune frank-default-large params
- frank-default-large.env: CTX_SIZE 262144 -> 196608 (192K context), BATCH_SIZE/UBATCH_SIZE 4096 -> 8192, CTX_CHECKPOINTS 16 -> 32, add preserve_thinking:true to chat_template_kwargs - kingjones-strix-mtp.md: sync launch excerpt + note on preserve_thinking and the peg-native/deepseek chat_format pairing
This commit is contained in:
1 parent
cd14d7bf68
commit
7a8500f6fa
2 files changed
+17
-9
No files matched your search
@@ -68,7 +68,9 @@ llama-server \
|
|||||||
-m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
|
-m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
|
||||||
--mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
|
--mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
|
||||||
--chat-template-file chat-template.jinja \
|
--chat-template-file chat-template.jinja \
|
||||||
-ngl 999 -dev ROCm0 -c 262144 -np 1 \
|
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
|
||||||
|
-ngl 999 -dev ROCm0 -c 196608 -np 1 \
|
||||||
|
-b 8192 -ub 8192 -ctxcp 32 \
|
||||||
--spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
|
--spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
|
||||||
--spec-draft-ngl 999 --spec-draft-n-max 4 \
|
--spec-draft-ngl 999 --spec-draft-n-max 4 \
|
||||||
-fa on --no-mmap --cache-type-k f16 --cache-type-v f16
|
-fa on --no-mmap --cache-type-k f16 --cache-type-v f16
|
||||||
@@ -82,6 +84,10 @@ llama-server \
|
|||||||
`enable_thinking: false` instead.
|
`enable_thinking: false` instead.
|
||||||
4. Small `max_tokens` with thinking on returns empty `content`. The Sharp
|
4. Small `max_tokens` with thinking on returns empty `content`. The Sharp
|
||||||
template drives thinking on by default (effort medium).
|
template drives thinking on by default (effort medium).
|
||||||
|
5. `preserve_thinking:true` (in `chat_template_kwargs`) keeps `<think>`
|
||||||
|
blocks in recorded assistant messages. `preserve_thinking:false` strips
|
||||||
|
them. See [[deployment]] for the chat_format/reasoning_format pairing
|
||||||
|
(`peg-native` + `deepseek`) the Pi agent expects from this endpoint.
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
|
|||||||
@@ -29,8 +29,10 @@ MMPROJ_FILE=mmproj-Qwen3.8-27B-Q8_0.gguf
|
|||||||
|
|
||||||
# Qwen-Sharp chat template (tracked in the repo, staged on the host).
|
# Qwen-Sharp chat template (tracked in the repo, staged on the host).
|
||||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||||
# enable_thinking on by default in the Sharp template; medium effort default.
|
# enable_thinking + preserve_thinking on by default in the Sharp template;
|
||||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true}'
|
# medium effort default. preserve_thinking keeps <think> blocks in recorded
|
||||||
|
# assistant messages (set to false to strip them, true to always keep them).
|
||||||
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
||||||
|
|
||||||
# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv).
|
# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv).
|
||||||
ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
|
ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
|
||||||
@@ -41,16 +43,16 @@ ALIAS=frank-default-large
|
|||||||
BIND_ADDR=0.0.0.0
|
BIND_ADDR=0.0.0.0
|
||||||
LISTEN_PORT=7770
|
LISTEN_PORT=7770
|
||||||
|
|
||||||
# Serving: full native 262144 context, 1 slot. f16 KV, context checkpoints,
|
# Serving: 196608 (192K) context, 1 slot. f16 KV, context checkpoints,
|
||||||
# prompt cache, draft-mtp with standalone Q4_0 head, n_max 4 (the knee for
|
# prompt cache, draft-mtp with standalone Q4_0 head, n_max 4 (the knee for
|
||||||
# the dense model). Unified memory; rocm-smi reports the 64 GiB budget.
|
# the dense model). Unified memory; rocm-smi reports the 64 GiB budget.
|
||||||
CTX_SIZE=262144
|
CTX_SIZE=196608
|
||||||
PARALLEL=1
|
PARALLEL=1
|
||||||
BATCH_SIZE=4096
|
BATCH_SIZE=8192
|
||||||
UBATCH_SIZE=4096
|
UBATCH_SIZE=8192
|
||||||
CACHE_TYPE_K=f16
|
CACHE_TYPE_K=f16
|
||||||
CACHE_TYPE_V=f16
|
CACHE_TYPE_V=f16
|
||||||
CTX_CHECKPOINTS=16
|
CTX_CHECKPOINTS=32
|
||||||
CACHE_RAM=8192
|
CACHE_RAM=8192
|
||||||
|
|
||||||
# Dense-model MTP draft (standalone Q4_0 head, n_max 4). No PromptForge .pfs
|
# Dense-model MTP draft (standalone Q4_0 head, n_max 4). No PromptForge .pfs
|
||||||
@@ -70,4 +72,4 @@ TOP_P=0.95
|
|||||||
REASONING=on
|
REASONING=on
|
||||||
REASONING_FORMAT=deepseek
|
REASONING_FORMAT=deepseek
|
||||||
|
|
||||||
VERBOSITY=4
|
VERBOSITY=4
|
||||||
Reference in new issue
Block a user