frank-large: wire Qwen-Sharp v22.4.0 template via --chat-template-file
The serve script only passed --chat-template-kwargs, so the binary fell back to the built-in peg-native template. A 250-token request produced a 73,600-token prompt (280x the expected ~260). Wiring in --chat-template-file drops the prompt to 62 tokens. Draft acceptance 0.79 (mean 4.14), model answers correctly, reasoning lands in reasoning_content.
This commit is contained in:
@@ -50,7 +50,7 @@ fi
|
||||
: "${PARALLEL_SLOTS:=1}"
|
||||
: "${SPEC_DRAFT_N_MAX:=4}"
|
||||
: "${COMPATIBILITY_MODE:=1}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
TEMPLATE_FILE="${TEMPLATE_FILE:-$SCRIPT_DIR/../dankserver/frank-medium/qwen-sharp-v22.4.0.jinja}"
|
||||
|
||||
SERVER_BIN="$RUNTIME_DIR/bin/llama-server"
|
||||
MODEL_FILE="$MODEL_DIR/Qwen3.8-27B-IU4-Kairic-Edge.gguf"
|
||||
@@ -113,6 +113,7 @@ ARGS=(
|
||||
-m "$MODEL_FILE" --alias frank-large
|
||||
--host "$HOST" --port "$PORT" --jinja
|
||||
--chat-template-kwargs '{"preserve_thinking":true}'
|
||||
--chat-template-file "$TEMPLATE_FILE"
|
||||
-dev ROCm0 -ngl 999
|
||||
-c "$CONTEXT_SIZE" -b 2048 -ub 512 -fa on -ctk f16 -ctv f16
|
||||
-t 16 -tb 32 -np "$PARALLEL_SLOTS" -ctxcp 32 --cache-ram "$CACHE_RAM"
|
||||
|
||||
Reference in New Issue
Block a user