frank-medium: chain DFlash2 with ngram-mod -- measured free where it cannot match

ngram-mod is draftless (added to the spec config vector unconditionally, while
every model-backed drafter needs ctx_dft), so unlike MTP it can share the
process with DFlash2. Chained at the fleet's 48/64 window, over three shapes:

  prose 1k/512      dflash 43.77 | ngram 17.72 (0 drafts) | chain 43.65
  echo 1.5k/400     dflash 76.82 | ngram 102.50          | chain 200.58
  structured/400    dflash 91.08 | ngram 17.80 (0 drafts) | chain 91.22
  identity          dflash 70.37 | ngram 17.77           | chain 70.40

On prose the chain is byte-identical to the drafter alone (same 119 drafts/49
accepts) -- that is what makes it a safe default -- and on prompt-echo traffic
it is 2.6x. VRAM is unchanged: the matcher is a 64 MiB host-RAM hash table.

The window sweep says keep the long one: 48/64 beats 16/32 (161.0) and 8/16
(113.8) on echo, and shortening it costs prose (43.9 -> 36.0) because the
matcher then proposes on ordinary fragments and the rejections still have to be
verified. That is the 'ngram hurts' result this fleet hit twice before.

Two documentation traps recorded with it: ngram-mod prints NO startup banner in
this build (its drafts are only visible behaviourally), and the harness's
'structured' shape is a negative control because its lines differ.
This commit is contained in:
Frank
2026-09-16 16:44:45 -07:00
parent 95936bbc81
commit a4f4429a99
3 changed files with 111 additions and 17 deletions
@@ -242,6 +242,61 @@ same binary, identical prompts at `temperature 0`, unit stopped between runs
the draft model (`creating MTP draft context against the target model ...`).
That is structural, not a tuning question.
### Chaining the draftless n-gram matcher (deployed)
`ngram-mod` **can** ride beside DFlash2, and it is enabled. The reason is in
`common/speculative.cpp`: the n-gram types are added to the config vector
unconditionally while every model-backed drafter is gated on
`ctx_dft != nullptr` — so n-gram needs no draft context and cannot collide with
the drafter over one. (MTP is the opposite: it *is* context-backed, which is why
`draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types
first, then the drafters — also means CLI order is irrelevant.
Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a
proposer can and cannot match:
| Configuration | prose 1k/512 | echo 1.5k/400 (copy 0.94) | structured/400 | identity |
|---|---:|---:|---:|---:|
| `draft-dflash@7` | 43.77 | 76.82 | 91.08 | 70.37 |
| `ngram-mod` alone | 17.72 *(0 drafts)* | 102.50 | 17.80 *(0 drafts)* | 17.77 |
| **`draft-dflash,ngram-mod@7`** | **43.65** | **200.58** | 91.22 | 70.40 |
- **On prompt-echo traffic the chain is 2.6× the drafter alone** (200.6 vs 76.8)
and twice `ngram-mod` by itself, at 100 % acceptance.
- **On prose it is indistinguishable from DFlash2 alone** — the same 119 drafts
and 49 accepts, the same 43.7 tok/s. That is the property that makes it safe
as a default: where the matcher cannot match, it costs nothing.
- `ngram-mod` alone is a *negative control*: zero drafts on prose and identity
(17.7 tok/s = the autoregressive baseline), 100 % acceptance on echo. Nothing
else could have proposed those tokens, which is how its drafts are identified
at all — see the trap below.
- VRAM is unchanged (29.42 GB on card 1 with or without it): the matcher is a
64 MiB host-RAM hash table.
- `structured` (25 records with unique ids) rewards no proposer: its lines
differ, so there is no long span to match. It is a negative control for
*self-repetition*, not evidence about it.
**The window matters, and shorter is not better.** The chain at three windows:
| `SPEC_NGRAM_N_MIN`/`N_MAX` | prose 1k/512 | echo 1.5k/400 |
|---|---:|---:|
| **48/64 — deployed** | **43.9** | **200.6** |
| 16/32 | 36.0 | 161.0 |
| 8/16 | 36.2 | 113.8 |
A short window makes the matcher propose on ordinary prose fragments, where the
proposals are rejected and the verification is still paid — the "ngram hurts"
result this fleet measured twice before on other payloads. At 48/64 the chain
keeps DFlash2's exact numbers on prose and wins on echo.
> **Operational trap: `ngram-mod` prints no startup banner in this build.**
> `journalctl -u frank-medium` shows `adding speculative implementation
> 'draft-dflash'` and nothing for the matcher — the only evidence it is running
> is behavioural (`draft_n` > 0 where text repeats) or the per-task line
> `slot print_timing: ... draft acceptance = 1.00000 (256 accepted / 256
> generated), mean len = 65.00`. Do not conclude from the load log that the
> chain is not engaged.
### The agent-loop shape (fleet harness, 2 families × 2 turns)
`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two