docs(llm): record a measured NEGATIVE tuning result for frank-large
Asked for iterative decode/prefill improvement at a 32k working point
without touching the model or the quantization. There isn't one: gufo's
served settings are already at this engine's tuned point, so no config
change was made and none is proposed.
Measured on the repo's own prose and shell sources at 32,762 prompt
tokens, salted per run against a warm prefix cache, 3-5 repetitions:
* --prefill-chunk 512 -> 8192 moves prefill 410.7-412.4 t/s. A 16x change
in chunk size, 0.4% out. The lever is dead at this depth.
* --draft-tokens 5 -> 10 gives 13.79-15.86 t/s decode, all inside the
run-to-run noise. The 5-rep confirmation REVERSED the 3-rep leaders:
dt9+chunk4096 looked best and came last (12.86), the deployed config
came first (14.71).
* --draft-policy fixed vs adaptive: no effect. --min-draft-tokens:
rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for
Q8 targets (token_embd.weight is Q8_0).
* --context 262144 -> 40960, tested as a diagnostic: no gain, and it
would break models.yml's declared contextWindow. Not shipped.
What does move the number is speculation itself — 6.65 t/s off vs ~15 t/s
on, 5.3x — and it is already enabled.
The two traps that cost most of the wall time are now in AGENTS.md's
benchmarking policy, because both produced FALSE results:
1. gufo bench cannot rank speculative settings and inverted the ranking.
Its corpus reports acceptance=0.933 +/- 0.000 with an identical trace
sha256 across draft widths — the same highly-predictable text every
run. It reported 29-37 t/s where real traffic gives ~15, and made
dt16 look 6% faster than dt7 when real text made it 14% slower.
2. A live A/B needs >=128 generated tokens. At 8, inter_token is
first-token noise; the first pass produced a nonsense ordering that
the 128-token pass reversed.
Plus the Server-Timing header is `ttft;dur=402.485` — the unit rides in
the NAME half, so splitting on "=" keys the dict "ttft;dur" and every
lookup silently returns nothing. Four configs reported "no Server-Timing"
while the header was present on all of them.
This commit is contained in:
@@ -179,6 +179,40 @@ unchanged by the payload swap.
|
||||
Short-prompt decode is gufo's *winning* regime; see the depth table above
|
||||
before quoting it as the endpoint's number.
|
||||
|
||||
### Tuning: the served flags are already optimal (2026-09-28)
|
||||
|
||||
Frank asked for iterative decode/prefill improvement at a 32k working point
|
||||
without touching the model or the quantization. **Result: no win exists at
|
||||
this engine's served settings, and nothing was changed.** Measured on real
|
||||
repo prose and shell sources at **32,762 prompt tokens**, salted per run so
|
||||
a warm prefix cache could not help, 3–5 repetitions each.
|
||||
|
||||
| lever | range swept | outcome |
|
||||
|---|---|---|
|
||||
| `--prefill-chunk` | 512 → 8192 | **no effect** — 410.7–412.4 t/s prefill across a 16× change |
|
||||
| `--draft-tokens` | 5 → 10 | **flat** — 13.79–15.86 t/s decode, inside run-to-run noise |
|
||||
| `--draft-policy` | fixed vs adaptive | 35.90 vs 35.50 (bench) — no effect |
|
||||
| `--min-draft-tokens` | — | **rejected by the engine**: "DFlash2 requires … 1" |
|
||||
| MTP backend | — | **unsupported for Q8**: `token_embd.weight` is Q8_0 |
|
||||
| `--context` 262144 → 40960 | diagnostic | 13.93 vs 14.71 t/s — no gain, and it would break the declared client context |
|
||||
|
||||
The confirmation round at 5 reps **reversed** the 3-rep leaders: the pair
|
||||
that looked best (`dt9 + chunk4096`) came last at 12.86 t/s, and the
|
||||
**deployed configuration came first at 14.71 t/s**. Every difference sits
|
||||
inside the noise band.
|
||||
|
||||
What *does* move the number is speculative decoding itself: **6.65 t/s off →
|
||||
~15 t/s on, a 5.3× step** — and that is already switched on. Prefill is
|
||||
insensitive to everything tried, which is consistent with gufo's own
|
||||
`EXPERIMENTS.md`, where dozens of kernel-level alternatives are recorded as
|
||||
rejected for "no release throughput improvement".
|
||||
|
||||
**Do not repeat this investigation without new information.** Two traps
|
||||
cost most of the time, and both are in AGENTS.md's benchmarking policy:
|
||||
`gufo bench` reports `acceptance=0.933 ± 0.000` on an identical trace hash
|
||||
and **inverts** the draft-width ranking; and a live A/B needs ≥128 generated
|
||||
tokens or `inter_token` is first-token noise.
|
||||
|
||||
## The template law: Qwen-Sharp is gone
|
||||
|
||||
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
|
||||
|
||||
Reference in New Issue
Block a user