frank-default-small: use the repo chat_template.jinja so tools render in Qwen3 XML
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its built-in template, which does not render the <tool_call> instructions; the model then emitted malformed XML (arguments baked into the function name) and the qwen3_xml parser could not extract clean calls.
This commit is contained in:
1 parent
5108af11aa
commit
20108e5b7a
2 files changed
+9
-1
No files matched your search
@@ -42,9 +42,12 @@ NUM_SPEC_TOKENS=5
|
|||||||
# Tool calling. Clients (pi, OpenWebUI) send tool_choice=auto; without
|
# Tool calling. Clients (pi, OpenWebUI) send tool_choice=auto; without
|
||||||
# --enable-auto-tool-choice vLLM rejects it with HTTP 400. The Avesed
|
# --enable-auto-tool-choice vLLM rejects it with HTTP 400. The Avesed
|
||||||
# chat template renders tools in the Qwen3 XML format, so the qwen3_xml
|
# chat template renders tools in the Qwen3 XML format, so the qwen3_xml
|
||||||
# parser is the matching one.
|
# parser is the matching one. CHAT_TEMPLATE must point at the repo's
|
||||||
|
# chat_template.jinja (the tokenizer_config.json ships none; vLLM's
|
||||||
|
# built-in template does not render the XML tool instructions).
|
||||||
ENABLE_AUTO_TOOL_CHOICE=1
|
ENABLE_AUTO_TOOL_CHOICE=1
|
||||||
TOOL_CALL_PARSER=qwen3_xml
|
TOOL_CALL_PARSER=qwen3_xml
|
||||||
|
CHAT_TEMPLATE=/home/dank/models/avesed-qwen36-chat_template.jinja
|
||||||
|
|
||||||
# RDNA flash-attention for the vision encoder (and general FA paths).
|
# RDNA flash-attention for the vision encoder (and general FA paths).
|
||||||
# Without this, the mm encoder falls back to torch SDPA and the warmup
|
# Without this, the mm encoder falls back to torch SDPA and the warmup
|
||||||
|
|||||||
@@ -35,6 +35,10 @@ fi
|
|||||||
: "${NUM_SPEC_TOKENS:=5}"
|
: "${NUM_SPEC_TOKENS:=5}"
|
||||||
: "${ENABLE_AUTO_TOOL_CHOICE:=1}"
|
: "${ENABLE_AUTO_TOOL_CHOICE:=1}"
|
||||||
: "${TOOL_CALL_PARSER:=qwen3_xml}"
|
: "${TOOL_CALL_PARSER:=qwen3_xml}"
|
||||||
|
# The Avesed tokenizer_config.json has no chat_template; vLLM would use its
|
||||||
|
# built-in template which does not render the model's XML tool format. Use
|
||||||
|
# the repo's chat_template.jinja (ships the <tool_call> instructions).
|
||||||
|
: "${CHAT_TEMPLATE:=/home/dank/models/avesed-qwen36-chat_template.jinja}"
|
||||||
|
|
||||||
if [[ ! -x "$VLLM_BIN" ]]; then
|
if [[ ! -x "$VLLM_BIN" ]]; then
|
||||||
echo "missing vllm binary: $VLLM_BIN" >&2
|
echo "missing vllm binary: $VLLM_BIN" >&2
|
||||||
@@ -59,4 +63,5 @@ exec "$VLLM_BIN" serve "$MODEL" \
|
|||||||
--speculative-config "{\"method\":\"ngram\",\"prompt_lookup_max\":$NGRAM_LOOKUP_MAX,\"prompt_lookup_min\":$NGRAM_LOOKUP_MIN,\"num_speculative_tokens\":$NUM_SPEC_TOKENS}" \
|
--speculative-config "{\"method\":\"ngram\",\"prompt_lookup_max\":$NGRAM_LOOKUP_MAX,\"prompt_lookup_min\":$NGRAM_LOOKUP_MIN,\"num_speculative_tokens\":$NUM_SPEC_TOKENS}" \
|
||||||
--reasoning-parser qwen3 \
|
--reasoning-parser qwen3 \
|
||||||
--trust-remote-code \
|
--trust-remote-code \
|
||||||
|
--chat-template "$CHAT_TEMPLATE" \
|
||||||
"${TOOL_ARGS[@]}"
|
"${TOOL_ARGS[@]}"
|
||||||
Reference in new issue
Block a user