diff --git a/docs/ciru-activefpx-promptforge.md b/docs/ciru-activefpx-promptforge.md index e005ecf..71ff578 100644 --- a/docs/ciru-activefpx-promptforge.md +++ b/docs/ciru-activefpx-promptforge.md @@ -202,6 +202,56 @@ it and generation proceeds. - v3 adds the BF16 vision tower over v2. The service reports `multimodal` capability; mmproj loads with the language model (CLIP on ROCm0 backend). +## ngram-mod + MTP Stacking (benchmarked 2026-08-21) + +This page records a research question: does ngram-mod add value when +stacked under the native MTP draft? The v3 profile runs MTP alone; the +benchmark below compares that against the stacked pair. + +### Feasibility + +ngram-mod DOES stack with MTP in this CIRU fork. The v3 build compiles in +both `ngram-mod` and `draft-mtp` (`common/speculative.cpp`). Pass +`--spec-type draft-mtp,ngram-mod` (or `ngram-mod,draft-mtp`) and the fork +initializes both implementations in the same process. The boot log shows: + +``` +adding speculative implementation 'ngram-mod' +initialized ngram_mod with n_match=24, size=4194304 (16.000 MB) +adding speculative implementation 'draft-mtp' +``` + +The model card's "no ngram subsystem" line refers to the CIRU PromptForge +env lanes (`PROMPTFORGE_*_NGRAM_*`, `PROMPTFORGE_ENABLE_NGRAM_MOD`), not +the upstream llama.cpp spec chain. Those env lanes must stay unset; the +`--spec-type ngram-mod` chain is separate and works. + +### Ordering — the CLI order does not matter + +The `--spec-type` string order is discarded. `common_speculative_init` +collapses the requested types into a bitmask, then rebuilds the chain in a +hard-coded priority: ngram types first, then draft-simple, then MTP. +So `draft-mtp,ngram-mod` and `ngram-mod,draft-mtp` produce the same +internal order (`ngram-mod` then `draft-mtp`) and the same runtimes. +Verified: both orders decode ~25.3 t/s at 8k. + +### Results (decode generation t/s, 200 tokens, prose prompt) + +| ctx | MTP only | MTP + ngram-mod | verdict | +|-----|----------|-----------------|---------| +| 8k | 29.42 | 25.27 | ngram hurts -4.15 | +| 16k | 26.42 | 21.13 | ngram hurts -5.29 | +| 64k | 19.72 | 18.00 | ngram hurts -1.72 | + +At every depth, adding ngram-mod under MTP REDUCED decode throughput. The +ngram draft costs its own forward pass / occupancy without adding enough +accepted tokens on top of MTP. Baseline (no ngram) remains the production +config (`--spec-type draft-mtp` only). Keep ngram-mod off. + +The harness lives at [scripts/bench-spec-stack.sh](../scripts/bench-spec-stack.sh); +it boots the v3 stack on a scratch port and reads authoritative +`print_timing` lines. It requires the production service to be stopped. + ## See Also - [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.