docs: document Qwen-Sharp template wiring and 73k-token diagnosis
The serve script now passes --chat-template-file pointing at the Qwen-Sharp v22.4.0 template. Without it the binary falls back to the built-in peg-native template, which produces a ~73,600-token prompt for a ~250-token request. With the sharp template wired in, the same request produces a 62-175-token prompt.
This commit is contained in:
@@ -298,8 +298,10 @@ via the `PROMPTFORGE_*` env block (the certified runner's exact env). The
|
||||
serve script (`scripts/serve-frank-large.sh`) validates binary → main GGUF →
|
||||
3 sidecars and exits with a pointer message if any is missing, then `exec`s
|
||||
`llama-server` with `--jinja --alias frank-large --chat-template-kwargs
|
||||
'{"preserve_thinking":true}'` (Qwen3.8's template needs this to carry
|
||||
thinking blocks across turns; a request-level value still wins).
|
||||
'{"preserve_thinking":true}' --chat-template-file $TEMPLATE_FILE` (the
|
||||
Qwen-Sharp v22.4.0 template; without `--chat-template-file` the binary
|
||||
falls back to the built-in `peg-native` template, which produces a
|
||||
~73,600-token prompt for a ~250-token request).
|
||||
|
||||
The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
|
||||
`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB "dedicated"
|
||||
|
||||
@@ -127,9 +127,21 @@ The three `.pfs` sidecars are wired via the `PROMPTFORGE_*` env block
|
||||
The serve script (`scripts/serve-frank-large.sh`) validates binary → main
|
||||
GGUF → 3 sidecars and exits with a pointer message if any is missing, then
|
||||
`exec`s `llama-server` with `--jinja --alias frank-large
|
||||
--chat-template-kwargs '{"preserve_thinking":true}'` (Qwen3.8's template
|
||||
needs this to carry thinking blocks across turns; a request-level value
|
||||
still wins).
|
||||
--chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file
|
||||
$TEMPLATE_FILE` (Qwen3.8's template needs `preserve_thinking` to carry
|
||||
thinking blocks across turns; a request-level value still wins).
|
||||
|
||||
## Chat template
|
||||
|
||||
The serve script passes `--chat-template-file` pointing at
|
||||
`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
|
||||
v22.4.0 template, the same one frank-medium uses). Without this flag the
|
||||
binary falls back to the built-in `peg-native` template, which produces a
|
||||
~73,600-token prompt for a ~250-token request (280x the expected size).
|
||||
With the sharp template wired in, the same request produces a 62–175-token
|
||||
prompt. The `Chat format: peg-native` log line is the format-type label,
|
||||
not the template indicator — the sharp template is active even when that
|
||||
line appears.
|
||||
|
||||
## Memory budget
|
||||
|
||||
@@ -174,6 +186,11 @@ tokens, 22.1 t/s average**. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The
|
||||
MTP depth 4 + n-gram spec method is working (draft acceptance logged, both
|
||||
`ngram-mod` and `draft-mtp` active in the journal).
|
||||
|
||||
**Template verification** (2026-09-21): a 250-token request produces a
|
||||
62–175-token prompt (not the 73,600-token prompt the built-in
|
||||
`peg-native` template produces). Draft acceptance 0.79 (mean 4.14), the
|
||||
model answers correctly, and reasoning lands in `reasoning_content`.
|
||||
|
||||
## Failure modes worth knowing
|
||||
|
||||
- **CK `config.h` is untracked**: a `git clean` in the CK checkout removes
|
||||
|
||||
Reference in New Issue
Block a user