docs(llm): record a measured NEGATIVE tuning result for frank-large

Asked for iterative decode/prefill improvement at a 32k working point
without touching the model or the quantization. There isn't one: gufo's
served settings are already at this engine's tuned point, so no config
change was made and none is proposed.

Measured on the repo's own prose and shell sources at 32,762 prompt
tokens, salted per run against a warm prefix cache, 3-5 repetitions:

* --prefill-chunk 512 -> 8192 moves prefill 410.7-412.4 t/s. A 16x change
  in chunk size, 0.4% out. The lever is dead at this depth.
* --draft-tokens 5 -> 10 gives 13.79-15.86 t/s decode, all inside the
  run-to-run noise. The 5-rep confirmation REVERSED the 3-rep leaders:
  dt9+chunk4096 looked best and came last (12.86), the deployed config
  came first (14.71).
* --draft-policy fixed vs adaptive: no effect. --min-draft-tokens:
  rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for
  Q8 targets (token_embd.weight is Q8_0).
* --context 262144 -> 40960, tested as a diagnostic: no gain, and it
  would break models.yml's declared contextWindow. Not shipped.

What does move the number is speculation itself — 6.65 t/s off vs ~15 t/s
on, 5.3x — and it is already enabled.

The two traps that cost most of the wall time are now in AGENTS.md's
benchmarking policy, because both produced FALSE results:

1. gufo bench cannot rank speculative settings and inverted the ranking.
   Its corpus reports acceptance=0.933 +/- 0.000 with an identical trace
   sha256 across draft widths — the same highly-predictable text every
   run. It reported 29-37 t/s where real traffic gives ~15, and made
   dt16 look 6% faster than dt7 when real text made it 14% slower.
2. A live A/B needs >=128 generated tokens. At 8, inter_token is
   first-token noise; the first pass produced a nonsense ordering that
   the 128-token pass reversed.

Plus the Server-Timing header is `ttft;dur=402.485` — the unit rides in
the NAME half, so splitting on "=" keys the dict "ttft;dur" and every
lookup silently returns nothing. Four configs reported "no Server-Timing"
while the header was present on all of them.
This commit is contained in:
Frank committed 2026-09-29 05:06:06 -07:00
1 parent 3a931ce2c1
commit 67672dc794
3 files changed
+95 -2

No files matched your search

+45
View File
@@ -1,6 +1,51 @@
# Bundle Change Log
## 2026-09-29
- **frank-large tuning investigation: a measured NEGATIVE result, nothing
changed.** Frank asked for iterative decode/prefill improvement at a 32k
working point without touching the model or quantization. **No win exists
at gufo's served settings**, so no config change was shipped. frank-large
was stopped for the whole investigation (~4h) per the benchmarking policy
and restored healthy in 9s afterwards.
- Measured on the repo's own prose and shell sources at **32,762 prompt
tokens**, salted per run so a warm prefix cache could not flatter a
later run, 3–5 repetitions per configuration.
- **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s.
A 16x change in chunk size, 0.4% out.
- **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all
inside the run-to-run noise band. The 5-rep confirmation round
**reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came
**last** (12.86), while the **deployed config came first** (14.71).
- `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`:
**rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported
for Q8 targets** (`Unsupported Qwen MTP tensor type for
token_embd.weight: Q8_0`).
- `--context 262144 → 40960` was tested as a diagnostic (the deployed value
preallocates KV ~8x larger than a 32k conversation needs): **no gain**
(13.93 vs 14.71 t/s), and it would break `models.yml`'s declared
`contextWindow: 262144`. Not shipped.
- **What actually moves the number is speculation itself: 6.65 t/s off →
~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to
everything tried, consistent with gufo's own `EXPERIMENTS.md` where
dozens of kernel variants are recorded rejected for "no release
throughput improvement".
- **Two traps are now in AGENTS.md's benchmarking policy**, because they
cost most of the wall time and both produced *false* results:
(1) **`gufo bench` cannot rank speculative settings and inverted the
ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an
identical trace sha256 across draft widths (the same highly-predictable
text every run, 93% of proposals accepted). It reported 29–37 t/s where
real traffic gives ~15, and made dt16 look 6% faster than dt7 when real
text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**;
at 8, `inter_token` is first-token noise and the first pass of the sweep
produced a nonsense ordering that the 128-token pass reversed.
- Also worth recording: the `Server-Timing` header is
`ttft;dur=402.485` — the unit is in the **name** half, so splitting on
`=` keys the dict `ttft;dur` and every lookup silently returns nothing.
Four configurations reported "no Server-Timing" while the header was
present on all of them.
## 2026-09-28
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
clean up old models. **510 G staged** to