frank-default-small: render Qwen-Sharp via --jinja; pull more experts to GPU

--jinja: ik_llama only renders a custom --chat-template-file as real Jinja
when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja
fallback). Without it the Qwen-Sharp template's thinking/image/tool
formatting is silently dropped. Add USE_JINJA=1 -> --jinja.

Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34
back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the
OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
This commit is contained in:
Frank committed 2026-08-26 03:46:54 -07:00
1 parent 4144f0f0e8
commit 3f3b1bdb27
4 files changed
+49 -7

No files matched your search

+6
View File
@@ -45,6 +45,11 @@ fi
: "${MMPROJ_FILE:=mmproj-Ornith-1.5-35B-A3B-f16.gguf}"
: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
# Render the Qwen-Sharp template with the true Jinja engine. ik_llama only
# applies --chat-template-file as real Jinja when --jinja is passed;
# otherwise it falls back to built-in non-Jinja handlers and silently drops
# the custom template's thinking/image/tool formatting.
: "${USE_JINJA:=1}"
: "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
@@ -92,6 +97,7 @@ exec "$LLAMA_SERVER" \
--model "$MODEL" \
--mmproj "$MMPROJ" \
--chat-template-file "$CHAT_TEMPLATE_FILE" \
$([ "$USE_JINJA" = "1" ] && printf -- "--jinja") \
--spec-type "$SPEC_TYPE" \
--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
--cache-type-k "$CACHE_TYPE_K" \