---
name: llama-cpp-spec-decode-ab
description: "Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts."
---

# Spec-decoding A/B on a live llama.cpp endpoint

Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter".
Verdicts here are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.

## Step 0 — settle chaining in the source, before spending GPU time

`common/speculative.cpp` (`common_speculative_init_result`) builds `configs` and gates each type:

- **draftless** (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
- **context-backed** (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) → require `params.draft.ctx_dft != nullptr`

Consequences: one context-backed drafter + any number of ngram types chains fine; **two context-backed types never do**
(`draft-mtp,draft-dflash` → `context type MTP requested but model doesn't contain MTP layers`).
Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.

## Step 1 — shapes (the part that decides the answer)

| Shape | Prompt | Who it can favour |
|---|---|---|
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
| identity | short deterministic question | controls for lossless verification |

Design the echo passage so no long span repeats *inside* it — otherwise you measure the matcher against itself.
For `structured`, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.

## Step 2 — protocol

1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait for the port.
2. Per config: write an env file with overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health` until ok, then measure.
3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
4. Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).

## Step 3 — read the right counter

- Per request: `timings.draft_n` / `timings.draft_n_accepted`, `predicted_per_second`.
- Cumulative: `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`.
- **`draft_n == 0` while tokens flow = the proposer is not engaged**, even when every flag looks right.
- A config with zero drafting is a useful *negative control*: it proves the other config's drafts came from the proposer under test.

## Traps that produce false verdicts

- **Thinking eats the answer.** Echo/structured prompts with thinking on put the copy in `reasoning_content`; `content` is empty, so identity checks compare `''` to `''`. Disable per request: `chat_template_kwargs: {"enable_thinking": false}`.
- **No banner.** ngram-mod prints nothing at startup; the journal shows only the drafter's `adding speculative implementation`. Never conclude "chain not engaged" from the load log.
- **Windows change the verdict.** ngram-mod `--spec-ngram-mod-n-min/--spec-ngram-mod-n-max` decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
- **Unused-tensor warnings are normal**: with `draft-dflash` on an MTP-carrying GGUF, `blk.<n>.nextn.*` is `unused ... ignoring` — correct, not a truncated file.
- **Harness hygiene**: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix), `pkill -x` never `-f`.

## Interpreting

Report `decode tok/s` per shape plus acceptance (`accepted/draft_n`) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.