Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
71 lines
6.0 KiB
Markdown
71 lines
6.0 KiB
Markdown
---
|
||
name: spec-decode-worth-it-triage
|
||
description: "Decide whether speculative decoding is worth deploying for a target/drafter pair: compute break-even acceptance from measured step times, classify the drafter by per-round cost, and check the real attachment surfaces (closed engines vs llama.cpp --spec-draft-device)."
|
||
---
|
||
|
||
# Speculative decoding: is it worth deploying?
|
||
|
||
Use when someone proposes a drafter for an existing endpoint ("use the 3090 as a drafter", "hook a DFlash drafter to X", "wouldn't speculation make Y faster?"). Answer with arithmetic against the INCUMBENT, not against zero.
|
||
|
||
## 0. Check whether the target already speculates
|
||
|
||
Many engines ship a drafter of their own; if it is on, your baseline is `A_incumbent / t_s`, not `1 / t_s`.
|
||
|
||
- halogen-flash-server (Dankclaw frank-large): MTP head **on by default**. Read it from its own accounting, e.g. `serve_api: mtp 63 tok in 1.31s = 47.97 t/s | 33 rounds, commit 1.88/round`.
|
||
- llama.cpp: `/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`; per-request lines say `draft acceptance = 0.76 (125 accepted / 165 generated)`.
|
||
|
||
## 1. Measure the two step times (never guess)
|
||
|
||
- target step `t_s` = 1 / (client-visible tok/s ÷ commit-per-round). Ask the operator for the tok/s they actually SEE: engine-internal decode-only numbers (short generations, prefill and delivery excluded) run ~1.7–2× optimistic.
|
||
- drafter step `d_s`: llama.cpp logs `eval time = 2117.32 ms / 178 tokens` per task; that is the drafter's per-token price when it is a plain model on the same box.
|
||
|
||
## 2. Arithmetic
|
||
|
||
```
|
||
round_time = K * d_s + 1.1 * t_s # K draft tokens proposed, K+1 verified in ONE target pass
|
||
A = (1 - p^(K+1)) / (1 - p) # expected accepted tokens, p = per-position acceptance
|
||
rate = A / round_time # compare to the incumbent
|
||
break-even : A > rate_incumbent * (K*d_s + 1.1*t_s)
|
||
```
|
||
|
||
Worked example that decided a real case: `t_s = 67 ms` (25 tok/s, 1.67 commit/round), `d_s = 12 ms` (27B IQ3_S on a 3090) → break-even `p ≈ 0.70` at K=3, `≈0.68` at K=5; realistic cross-finetune `p = 0.4–0.6` gives 17–20 tok/s, i.e. a LOSS. Drafters that are only ~3× faster per token cannot pay for K sequential forwards.
|
||
|
||
## 3. Classify the drafter by per-round cost (this decides more than acceptance)
|
||
|
||
| drafter | cost per round | needs |
|
||
|---|---|---|
|
||
| plain model via `-md`/`draft-simple` | **K sequential forwards** | 10–50× faster than the target |
|
||
| n-gram / prompt-lookup (`ngram-*`) | ~0, no model | repeated context (agent edit loops) |
|
||
| trained multi-token head (`draft-mtp`, MTP/EAGLE/DFlash) | **~1 pass** | the head must exist FOR THAT CHECKPOINT |
|
||
|
||
Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden states, so a drafter built for one model cannot be re-pointed at another (a dense-27B DFlash2 is useless for a 512-expert MoE trunk).
|
||
|
||
## 4. Acceptance reality checks
|
||
|
||
- Fleet datum: a TRAINED z-lab DFlash2 at depth 7 accepted ~41% on its matched target (`draft-dflash 7 → 43.5 tok/s, 41% accepted`). Treat 40–60% as the honest band for a matched pair; <85% rarely pays unless the drafter is nearly free.
|
||
- Cheapest proxy for `p` at temp 0, when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights moved.
|
||
- llama.cpp can compute the projection without any draft artifact: `--spec-synth-rates P0,P1,...` / `--spec-synth-len L`.
|
||
- Speculation pays IN PROPORTION TO HOW SLOW THE TARGET IS. The same 41% gave 2.3× on an 18.8 tok/s layer-split target and a loss on a 25 tok/s one. Fast targets are usually sparse MoE (few active experts → little per-token weight traffic) — the worst case for drafting.
|
||
|
||
## 5. Check the attachment surface BEFORE the arithmetic
|
||
|
||
- **Closed engines refuse external drafters by design.** halogen-flash-server: `DRAFTERS = {serial:0, mtp:1, dflash2:2}`, `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint, and an unknown/unavailable name is a deliberate 400 (a silent downgrade would make comparisons lie). "Drafter" is a field in its internal loopback token protocol (`drafter`, `spec_verify`, `spec_commit`) — there is no device routing, so a drafter can never sit on another card. Probe it live: POST `"drafter":"dflash2"` and read the error.
|
||
- **pure llama.cpp DOES route the draft**: `--spec-draft-device/-devd <devs>` (default: follows `--device`), plus `-ngld/--gpu-layers-draft`, `-otd`, `-ctkd/-ctvd`, `-td/-tbd`, `-ncmoed`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`. In production on the fleet as `SPEC_DRAFT_DEVICE=ROCm1` with a layer-split target.
|
||
- **Devices**: one llama.cpp process sees devices only from backends it was built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot coexist); a CUDA-only build has no `Vulkan0` to draft with.
|
||
- **Chaining is not universal**: `--spec-type` takes a comma list, but `draft-mtp,draft-dflash` was measured to DIE AT LOAD while `draft-dflash,ngram-mod` is structurally allowed. Test at load, never assume composition.
|
||
|
||
## 6. Cross-model pairs need a byte-identical tokenizer
|
||
|
||
Speculation compares token IDs, so the draft and target tokenizers must map id → the same string. Verify cheaply from the GGUF headers (no full download; a range read of the first ~8 MB over HTTP is enough for a remote file): parse `tokenizer.ggml.tokens`, print vocab size, merges, `pre`, and a sha256 over the first 2000 tokens.
|
||
|
||
```
|
||
vocab=248320 merges=247587 model=gpt2 pre=qwen35
|
||
sha256(first 2000 tokens)=8619c68fe4466d3a # identical => ids are interchangeable
|
||
```
|
||
|
||
Identity is necessary, NOT sufficient — the drafter still has to be cheap and accepted.
|
||
|
||
## 7. Verdict shape
|
||
|
||
Report, in this order: (a) does the target already draft, and at what rate; (b) break-even `p` at K=1,3,7 versus the incumbent; (c) the realistic `p` band and the resulting rate; (d) whether the drafter even fits (VRAM) and whether the attachment surface exists; (e) the one measurement that would settle it, and what you expect it to show.
|