command-center/agent/managed-skills/spec-decode-worth-it-triage/SKILL.md

name, description

name	description
spec-decode-worth-it-triage	Decide whether speculative decoding is worth deploying for a target/drafter pair: compute break-even acceptance from measured step times, classify the drafter by per-round cost, and check the real attachment surfaces (closed engines vs llama.cpp --spec-draft-device).

Speculative decoding: is it worth deploying?

Use when someone proposes a drafter for an existing endpoint ("use the 3090 as a drafter", "hook a DFlash drafter to X", "wouldn't speculation make Y faster?"). Answer with arithmetic against the INCUMBENT, not against zero.

0. Check whether the target already speculates

Many engines ship a drafter of their own; if it is on, your baseline is A_incumbent / t_s, not 1 / t_s.

• halogen-flash-server (Dankclaw frank-large): MTP head on by default. Read it from its own accounting, e.g. serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round.
• llama.cpp: /metrics carries llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total; per-request lines say draft acceptance = 0.76 (125 accepted / 165 generated).

1. Measure the two step times (never guess)

• target step t_s = 1 / (client-visible tok/s ÷ commit-per-round). Ask the operator for the tok/s they actually SEE: engine-internal decode-only numbers (short generations, prefill and delivery excluded) run ~1.7–2× optimistic.
• drafter step d_s: llama.cpp logs eval time = 2117.32 ms / 178 tokens per task; that is the drafter's per-token price when it is a plain model on the same box.

2. Arithmetic

round_time = K * d_s + 1.1 * t_s        # K draft tokens proposed, K+1 verified in ONE target pass
A          = (1 - p^(K+1)) / (1 - p)    # expected accepted tokens, p = per-position acceptance
rate       = A / round_time             # compare to the incumbent
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)

Worked example that decided a real case: t_s = 67 ms (25 tok/s, 1.67 commit/round), d_s = 12 ms (27B IQ3_S on a 3090) → break-even p ≈ 0.70 at K=3, ≈0.68 at K=5; realistic cross-finetune p = 0.4–0.6 gives 17–20 tok/s, i.e. a LOSS. Drafters that are only ~3× faster per token cannot pay for K sequential forwards.

3. Classify the drafter by per-round cost (this decides more than acceptance)

drafter	cost per round	needs
plain model via -md/draft-simple	K sequential forwards	10–50× faster than the target
n-gram / prompt-lookup (ngram-*)	~0, no model	repeated context (agent edit loops)
trained multi-token head (draft-mtp, MTP/EAGLE/DFlash)	~1 pass	the head must exist FOR THAT CHECKPOINT

Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden states, so a drafter built for one model cannot be re-pointed at another (a dense-27B DFlash2 is useless for a 512-expert MoE trunk).

4. Acceptance reality checks

• Fleet datum: a TRAINED z-lab DFlash2 at depth 7 accepted ~41% on its matched target (draft-dflash 7 → 43.5 tok/s, 41% accepted). Treat 40–60% as the honest band for a matched pair; <85% rarely pays unless the drafter is nearly free.
• Cheapest proxy for p at temp 0, when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights moved.
• llama.cpp can compute the projection without any draft artifact: --spec-synth-rates P0,P1,... / --spec-synth-len L.
• Speculation pays IN PROPORTION TO HOW SLOW THE TARGET IS. The same 41% gave 2.3× on an 18.8 tok/s layer-split target and a loss on a 25 tok/s one. Fast targets are usually sparse MoE (few active experts → little per-token weight traffic) — the worst case for drafting.

5. Check the attachment surface BEFORE the arithmetic

• Closed engines refuse external drafters by design. halogen-flash-server: DRAFTERS = {serial:0, mtp:1, dflash2:2}, DRAFTER_CAP gates dflash2 on the loaded checkpoint, and an unknown/unavailable name is a deliberate 400 (a silent downgrade would make comparisons lie). "Drafter" is a field in its internal loopback token protocol (drafter, spec_verify, spec_commit) — there is no device routing, so a drafter can never sit on another card. Probe it live: POST "drafter":"dflash2" and read the error.
• pure llama.cpp DOES route the draft: --spec-draft-device/-devd <devs> (default: follows --device), plus -ngld/--gpu-layers-draft, -otd, -ctkd/-ctvd, -td/-tbd, -ncmoed, --spec-draft-n-max (default 3), --spec-draft-p-min. In production on the fleet as SPEC_DRAFT_DEVICE=ROCm1 with a layer-split target.
• Devices: one llama.cpp process sees devices only from backends it was built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot coexist); a CUDA-only build has no Vulkan0 to draft with.
• Chaining is not universal: --spec-type takes a comma list, but draft-mtp,draft-dflash was measured to DIE AT LOAD while draft-dflash,ngram-mod is structurally allowed. Test at load, never assume composition.

6. Cross-model pairs need a byte-identical tokenizer

Speculation compares token IDs, so the draft and target tokenizers must map id → the same string. Verify cheaply from the GGUF headers (no full download; a range read of the first ~8 MB over HTTP is enough for a remote file): parse tokenizer.ggml.tokens, print vocab size, merges, pre, and a sha256 over the first 2000 tokens.

vocab=248320 merges=247587 model=gpt2 pre=qwen35
sha256(first 2000 tokens)=8619c68fe4466d3a   # identical => ids are interchangeable

Identity is necessary, NOT sufficient — the drafter still has to be cheap and accepted.

7. Verdict shape

Report, in this order: (a) does the target already draft, and at what rate; (b) break-even p at K=1,3,7 versus the incumbent; (c) the realistic p band and the resulting rate; (d) whether the drafter even fits (VRAM) and whether the attachment surface exists; (e) the one measurement that would settle it, and what you expect it to show.