docs: document Qwen-Sharp template wiring and 73k-token diagnosis
The serve script now passes --chat-template-file pointing at the Qwen-Sharp v22.4.0 template. Without it the binary falls back to the built-in peg-native template, which produces a ~73,600-token prompt for a ~250-token request. With the sharp template wired in, the same request produces a 62-175-token prompt.
This commit is contained in:
1 parent
293e4e4b09
commit
97f60b35ef
2 files changed
+24
-5
No files matched your search
@@ -298,8 +298,10 @@ via the `PROMPTFORGE_*` env block (the certified runner's exact env). The
|
||||
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF →
|
||||
3 sidecars and exits with a pointer message if any is missing, then `exec`s
|
||||
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
||||
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry
|
||||
thinking blocks across turns; a request-level value still wins).
|
||||
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
|
||||
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
|
||||
falls back to the built-in `peg-native` template, which produces a
|
||||
~73,600-token prompt for a ~250-token request).
|
||||
|
||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||
|
||||
Reference in new issue
Block a user