docs: benchmark ngram-mod stacking under MTP on v3 stack (8k/16k/64k); doc CLI order does not matter

2026-08-21 20:43:47 -07:00
parent f1074ade61
commit 3b9379bb5b
1 changed files with 50 additions and 0 deletions
@@ -202,6 +202,56 @@ it and generation proceeds.
 - v3 adds the BF16 vision tower over v2. The service reports `multimodal`
  capability; mmproj loads with the language model (CLIP on ROCm0 backend).

+## ngram-mod + MTP Stacking (benchmarked 2026-08-21)
+
+This page records a research question: does ngram-mod add value when
+stacked under the native MTP draft? The v3 profile runs MTP alone; the
+benchmark below compares that against the stacked pair.
+
+### Feasibility
+
+ngram-mod DOES stack with MTP in this CIRU fork. The v3 build compiles in
+both `ngram-mod` and `draft-mtp` (`common/speculative.cpp`). Pass
+`--spec-type draft-mtp,ngram-mod` (or `ngram-mod,draft-mtp`) and the fork
+initializes both implementations in the same process. The boot log shows:
+
+```
+adding speculative implementation 'ngram-mod'
+initialized ngram_mod with n_match=24, size=4194304 (16.000 MB)
+adding speculative implementation 'draft-mtp'
+```
+
+The model card's "no ngram subsystem" line refers to the CIRU PromptForge
+env lanes (`PROMPTFORGE_*_NGRAM_*`, `PROMPTFORGE_ENABLE_NGRAM_MOD`), not
+the upstream llama.cpp spec chain. Those env lanes must stay unset; the
+`--spec-type ngram-mod` chain is separate and works.
+
+### Ordering — the CLI order does not matter
+
+The `--spec-type` string order is discarded. `common_speculative_init`
+collapses the requested types into a bitmask, then rebuilds the chain in a
+hard-coded priority: ngram types first, then draft-simple, then MTP.
+So `draft-mtp,ngram-mod` and `ngram-mod,draft-mtp` produce the same
+internal order (`ngram-mod` then `draft-mtp`) and the same runtimes.
+Verified: both orders decode ~25.3 t/s at 8k.
+
+### Results (decode generation t/s, 200 tokens, prose prompt)
+
+| ctx | MTP only | MTP + ngram-mod | verdict |
+|-----|----------|-----------------|---------|
+| 8k | 29.42 | 25.27 | ngram hurts -4.15 |
+| 16k | 26.42 | 21.13 | ngram hurts -5.29 |
+| 64k | 19.72 | 18.00 | ngram hurts -1.72 |
+
+At every depth, adding ngram-mod under MTP REDUCED decode throughput. The
+ngram draft costs its own forward pass / occupancy without adding enough
+accepted tokens on top of MTP. Baseline (no ngram) remains the production
+config (`--spec-type draft-mtp` only). Keep ngram-mod off.
+
+The harness lives at [scripts/bench-spec-stack.sh](../scripts/bench-spec-stack.sh);
+it boots the v3 stack on a scratch port and reads authoritative
+`print_timing` lines. It requires the production service to be stopped.
+
 ## See Also

 - [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.