Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
59 lines
4.3 KiB
Markdown
59 lines
4.3 KiB
Markdown
---
|
|
name: llama-cpp-spec-decode-ab
|
|
description: "Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts."
|
|
---
|
|
|
|
# Spec-decoding A/B on a live llama.cpp endpoint
|
|
|
|
Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter".
|
|
Verdicts here are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.
|
|
|
|
## Step 0 — settle chaining in the source, before spending GPU time
|
|
|
|
`common/speculative.cpp` (`common_speculative_init_result`) builds `configs` and gates each type:
|
|
|
|
- **draftless** (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
|
|
- **context-backed** (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) → require `params.draft.ctx_dft != nullptr`
|
|
|
|
Consequences: one context-backed drafter + any number of ngram types chains fine; **two context-backed types never do**
|
|
(`draft-mtp,draft-dflash` → `context type MTP requested but model doesn't contain MTP layers`).
|
|
Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.
|
|
|
|
## Step 1 — shapes (the part that decides the answer)
|
|
|
|
| Shape | Prompt | Who it can favour |
|
|
|---|---|---|
|
|
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
|
|
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
|
|
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
|
|
| identity | short deterministic question | controls for lossless verification |
|
|
|
|
Design the echo passage so no long span repeats *inside* it — otherwise you measure the matcher against itself.
|
|
For `structured`, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.
|
|
|
|
## Step 2 — protocol
|
|
|
|
1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait for the port.
|
|
2. Per config: write an env file with overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health` until ok, then measure.
|
|
3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
|
|
4. Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).
|
|
|
|
## Step 3 — read the right counter
|
|
|
|
- Per request: `timings.draft_n` / `timings.draft_n_accepted`, `predicted_per_second`.
|
|
- Cumulative: `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`.
|
|
- **`draft_n == 0` while tokens flow = the proposer is not engaged**, even when every flag looks right.
|
|
- A config with zero drafting is a useful *negative control*: it proves the other config's drafts came from the proposer under test.
|
|
|
|
## Traps that produce false verdicts
|
|
|
|
- **Thinking eats the answer.** Echo/structured prompts with thinking on put the copy in `reasoning_content`; `content` is empty, so identity checks compare `''` to `''`. Disable per request: `chat_template_kwargs: {"enable_thinking": false}`.
|
|
- **No banner.** ngram-mod prints nothing at startup; the journal shows only the drafter's `adding speculative implementation`. Never conclude "chain not engaged" from the load log.
|
|
- **Windows change the verdict.** ngram-mod `--spec-ngram-mod-n-min/--spec-ngram-mod-n-max` decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
|
|
- **Unused-tensor warnings are normal**: with `draft-dflash` on an MTP-carrying GGUF, `blk.<n>.nextn.*` is `unused ... ignoring` — correct, not a truncated file.
|
|
- **Harness hygiene**: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix), `pkill -x` never `-f`.
|
|
|
|
## Interpreting
|
|
|
|
Report `decode tok/s` per shape plus acceptance (`accepted/draft_n`) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.
|