frank-default-small: render Qwen-Sharp via --jinja; pull more experts to GPU
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja fallback). Without it the Qwen-Sharp template's thinking/image/tool formatting is silently dropped. Add USE_JINJA=1 -> --jinja. Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34 back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
This commit is contained in:
@@ -34,6 +34,10 @@ LLAMA_SERVER=/home/dank/ik_llama.cpp/build/bin/llama-server
|
||||
# staged on the host).
|
||||
# Source: https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
# Render the Qwen-Sharp template with the true Jinja engine (--jinja). Without
|
||||
# it ik_llama falls back to built-in non-Jinja handlers and drops the custom
|
||||
# template's thinking/image/tool formatting.
|
||||
USE_JINJA=1
|
||||
|
||||
ALIAS=frank-default-small
|
||||
BIND_ADDR=0.0.0.0
|
||||
@@ -60,7 +64,11 @@ SPEC_TYPE=none
|
||||
# 3,7,11,15,19,23,27,31,35,36,37,38,39,40. Move all other expert tensors to
|
||||
# CPU. All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the
|
||||
# CPU buft name on this host.
|
||||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
#
|
||||
# 2026-08-26: brought back expert blocks 30,32,33,34 to CUDA (~2.5 GiB) to
|
||||
# use the ~5.4 GiB GPU headroom for lower latency; the remaining larger-CPU
|
||||
# range below those stays on CPU.
|
||||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
OVERRIDE_BUFT=CPU
|
||||
# ik_llama --gpu-layers/-ngl does NOT accept the literal "all"; use 999 to
|
||||
# mean "load as many layers as possible". Expert exps are pinned to CPU by
|
||||
|
||||
@@ -45,6 +45,11 @@ fi
|
||||
: "${MMPROJ_FILE:=mmproj-Ornith-1.5-35B-A3B-f16.gguf}"
|
||||
: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
# Render the Qwen-Sharp template with the true Jinja engine. ik_llama only
|
||||
# applies --chat-template-file as real Jinja when --jinja is passed;
|
||||
# otherwise it falls back to built-in non-Jinja handlers and silently drops
|
||||
# the custom template's thinking/image/tool formatting.
|
||||
: "${USE_JINJA:=1}"
|
||||
: "${ALIAS:=frank-default-small}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
@@ -92,6 +97,7 @@ exec "$LLAMA_SERVER" \
|
||||
--model "$MODEL" \
|
||||
--mmproj "$MMPROJ" \
|
||||
--chat-template-file "$CHAT_TEMPLATE_FILE" \
|
||||
$([ "$USE_JINJA" = "1" ] && printf -- "--jinja") \
|
||||
--spec-type "$SPEC_TYPE" \
|
||||
--override-tensor "$OVERRIDE_EXPERT=$OVERRIDE_BUFT" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
|
||||
Reference in New Issue
Block a user