dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-small: render Qwen-Sharp via --jinja; pull more experts to GPU

Browse Source 
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja
when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja
fallback). Without it the Qwen-Sharp template's thinking/image/tool
formatting is silently dropped. Add USE_JINJA=1 -> --jinja.

Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34
back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the
OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
This commit is contained in:
Frank
2026-08-26 03:46:54 -07:00
parent 4144f0f0e8
commit 3f3b1bdb27
4 changed files with 49 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_small_config.bats
+8 -1
View File
@@ -22,7 +22,7 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
@test "frank-default-small keeps full-attention and final blocks on CUDA" {
source "$ENV_FILE"
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
[[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
[[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
[[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
@@ -68,3 +68,10 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
grep -q 'qwen3\.8-froggeric-v22\.4\.0' "$TEMPLATE_FILE"
grep -q 'Qwen-Sharp' "$UNIT_FILE"
}
@test "frank-default-small renders the custom template with --jinja" {
# Without --jinja, ik_llama only supports built-in non-Jinja handlers and
# silently drops the custom Qwen-Sharp template formatting.
grep -q '^USE_JINJA=1$' "$ENV_FILE"
grep -q -- '--jinja' "$SERVE_FILE"
}