agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,58 @@
|
||||
---
|
||||
name: llama-cpp-spec-decode-ab
|
||||
description: "Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts."
|
||||
---
|
||||
|
||||
# Spec-decoding A/B on a live llama.cpp endpoint
|
||||
|
||||
Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter".
|
||||
Verdicts here are **traffic-shaped**: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.
|
||||
|
||||
## Step 0 — settle chaining in the source, before spending GPU time
|
||||
|
||||
`common/speculative.cpp` (`common_speculative_init_result`) builds `configs` and gates each type:
|
||||
|
||||
- **draftless** (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
|
||||
- **context-backed** (`draft-mtp`, `draft-dflash`, `draft-dspark`, `draft-eagle3`) → require `params.draft.ctx_dft != nullptr`
|
||||
|
||||
Consequences: one context-backed drafter + any number of ngram types chains fine; **two context-backed types never do**
|
||||
(`draft-mtp,draft-dflash` → `context type MTP requested but model doesn't contain MTP layers`).
|
||||
Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.
|
||||
|
||||
## Step 1 — shapes (the part that decides the answer)
|
||||
|
||||
| Shape | Prompt | Who it can favour |
|
||||
|---|---|---|
|
||||
| prose | 1k filler + "reply with one sentence" | model-backed drafters only |
|
||||
| echo | ~1.5k passage of UNIQUE lines + "repeat verbatim" | ngram (output IS the prompt) |
|
||||
| structured | repetitive output, no prompt overlap | ngram, if spans really repeat |
|
||||
| identity | short deterministic question | controls for lossless verification |
|
||||
|
||||
Design the echo passage so no long span repeats *inside* it — otherwise you measure the matcher against itself.
|
||||
For `structured`, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.
|
||||
|
||||
## Step 2 — protocol
|
||||
|
||||
1. Stop the unit (`sudo systemctl stop <svc>`), `pkill -x llama-server`; wait for the port.
|
||||
2. Per config: write an env file with overrides, start the serve script with `FRANK_*_ENV_FILE`, poll `/health` until ok, then measure.
|
||||
3. **Restore the unit in a `finally`** and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
|
||||
4. Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).
|
||||
|
||||
## Step 3 — read the right counter
|
||||
|
||||
- Per request: `timings.draft_n` / `timings.draft_n_accepted`, `predicted_per_second`.
|
||||
- Cumulative: `llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`.
|
||||
- **`draft_n == 0` while tokens flow = the proposer is not engaged**, even when every flag looks right.
|
||||
- A config with zero drafting is a useful *negative control*: it proves the other config's drafts came from the proposer under test.
|
||||
|
||||
## Traps that produce false verdicts
|
||||
|
||||
- **Thinking eats the answer.** Echo/structured prompts with thinking on put the copy in `reasoning_content`; `content` is empty, so identity checks compare `''` to `''`. Disable per request: `chat_template_kwargs: {"enable_thinking": false}`.
|
||||
- **No banner.** ngram-mod prints nothing at startup; the journal shows only the drafter's `adding speculative implementation`. Never conclude "chain not engaged" from the load log.
|
||||
- **Windows change the verdict.** ngram-mod `--spec-ngram-mod-n-min/--spec-ngram-mod-n-max` decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
|
||||
- **Unused-tensor warnings are normal**: with `draft-dflash` on an MTP-carrying GGUF, `blk.<n>.nextn.*` is `unused ... ignoring` — correct, not a truncated file.
|
||||
- **Harness hygiene**: absolute script paths (a relative path plus `cd` to its own dir double-counts the prefix), `pkill -x` never `-f`.
|
||||
|
||||
## Interpreting
|
||||
|
||||
Report `decode tok/s` per shape plus acceptance (`accepted/draft_n`) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.
|
||||
Reference in new issue
Block a user