docs: document Qwen-Sharp template wiring and 73k-token diagnosis

The serve script now passes --chat-template-file pointing at the
Qwen-Sharp v22.4.0 template. Without it the binary falls back to the
built-in peg-native template, which produces a ~73,600-token prompt for
a ~250-token request. With the sharp template wired in, the same request
produces a 62-175-token prompt.
This commit is contained in:
Frank
2026-09-21 20:11:00 -07:00
parent 293e4e4b09
commit 97f60b35ef
2 changed files with 24 additions and 5 deletions
+4 -2
View File
@@ -298,8 +298,10 @@ via the `PROMPTFORGE_*` env block (the certified runner's exact env). The
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF →
3 sidecars and exits with a pointer message if any is missing, then `exec`s
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry
thinking blocks across turns; a request-level value still wins).
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
falls back to the built-in `peg-native` template, which produces a
~73,600-token prompt for a ~250-token request).
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
@@ -127,9 +127,21 @@ The three `.pfs` sidecars are wired via the `PROMPTFORGE_*` env block
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
GGUF → 3 sidecars and exits with a pointer message if any is missing, then
`exec`s `llama-server` with `--jinja --alias frank-large
--chat-template-kwargs '{"preserve_thinking":true}'` (Qwen3.8's template
needs this to carry thinking blocks across turns; a request-level value
still wins).
--chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file
$TEMPLATE_FILE` (Qwen3.8's template needs `preserve_thinking` to carry
thinking blocks across turns; a request-level value still wins).
## Chat template
The serve script passes `--chat-template-file` pointing at
`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
v22.4.0 template, the same one frank-medium uses). Without this flag the
binary falls back to the built-in `peg-native` template, which produces a
~73,600-token prompt for a ~250-token request (280x the expected size).
With the sharp template wired in, the same request produces a 62–175-token
prompt. The `Chat format: peg-native` log line is the format-type label,
not the template indicator — the sharp template is active even when that
line appears.
## Memory budget
@@ -174,6 +186,11 @@ tokens, 22.1 t/s average**. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The
MTP depth 4 + n-gram spec method is working (draft acceptance logged, both
`ngram-mod` and `draft-mtp` active in the journal).
**Template verification** (2026-09-21): a 250-token request produces a
62–175-token prompt (not the 73,600-token prompt the built-in
`peg-native` template produces). Draft acceptance 0.79 (mean 4.14), the
model answers correctly, and reasoning lands in `reasoning_content`.
## Failure modes worth knowing
- **CK `config.h` is untracked**: a `git clean` in the CK checkout removes