frank-default-small: use the repo chat_template.jinja so tools render in Qwen3 XML

The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
This commit is contained in:
Frank
2026-08-17 17:13:35 -07:00
parent 5108af11aa
commit 20108e5b7a
2 changed files with 9 additions and 1 deletions
+4 -1
View File
@@ -42,9 +42,12 @@ NUM_SPEC_TOKENS=5
# Tool calling. Clients (pi, OpenWebUI) send tool_choice=auto; without
# --enable-auto-tool-choice vLLM rejects it with HTTP 400. The Avesed
# chat template renders tools in the Qwen3 XML format, so the qwen3_xml
# parser is the matching one.
# parser is the matching one. CHAT_TEMPLATE must point at the repo's
# chat_template.jinja (the tokenizer_config.json ships none; vLLM's
# built-in template does not render the XML tool instructions).
ENABLE_AUTO_TOOL_CHOICE=1
TOOL_CALL_PARSER=qwen3_xml
CHAT_TEMPLATE=/home/dank/models/avesed-qwen36-chat_template.jinja
# RDNA flash-attention for the vision encoder (and general FA paths).
# Without this, the mm encoder falls back to torch SDPA and the warmup
+5
View File
@@ -35,6 +35,10 @@ fi
: "${NUM_SPEC_TOKENS:=5}"
: "${ENABLE_AUTO_TOOL_CHOICE:=1}"
: "${TOOL_CALL_PARSER:=qwen3_xml}"
# The Avesed tokenizer_config.json has no chat_template; vLLM would use its
# built-in template which does not render the model's XML tool format. Use
# the repo's chat_template.jinja (ships the <tool_call> instructions).
: "${CHAT_TEMPLATE:=/home/dank/models/avesed-qwen36-chat_template.jinja}"
if [[ ! -x "$VLLM_BIN" ]]; then
echo "missing vllm binary: $VLLM_BIN" >&2
@@ -59,4 +63,5 @@ exec "$VLLM_BIN" serve "$MODEL" \
--speculative-config "{\"method\":\"ngram\",\"prompt_lookup_max\":$NGRAM_LOOKUP_MAX,\"prompt_lookup_min\":$NGRAM_LOOKUP_MIN,\"num_speculative_tokens\":$NUM_SPEC_TOKENS}" \
--reasoning-parser qwen3 \
--trust-remote-code \
--chat-template "$CHAT_TEMPLATE" \
"${TOOL_ARGS[@]}"