0 Fork 0

Files

T

Frank 135bb2194d agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills

- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh
- models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes
- niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0
- memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills
- managed-skills: 40+ new fleet runbook skills
- agent/skills: import built-in skill set

2026-09-17 12:27:08 -07:00

4.3 KiB

Raw Blame History

name, description

name	description
llama-cpp-spec-decode-ab	Measure speculative-decoding configurations (drafter vs MTP vs ngram-mod vs chains) on a live llama.cpp endpoint without fooling yourself: shape taxonomy, negative controls, unit-stopped protocol, window sweeps, and the traps that produce false verdicts.

Spec-decoding A/B on a live llama.cpp endpoint

Use when asked "is X faster than Y?", "can these be chained?", or "should we use MTP/ngram/drafter". Verdicts here are traffic-shaped: a proposer can only draft text it has seen, so a single prompt shape answers the wrong question.

Step 0 — settle chaining in the source, before spending GPU time

common/speculative.cpp (common_speculative_init_result) builds configs and gates each type:

draftless (ngram family: simple, map-k, map-k4v, mod, cache) → added unconditionally
context-backed (draft-mtp, draft-dflash, draft-dspark, draft-eagle3) → require params.draft.ctx_dft != nullptr

Consequences: one context-backed drafter + any number of ngram types chains fine; two context-backed types never do (draft-mtp,draft-dflash → context type MTP requested but model doesn't contain MTP layers). Internal order is fixed (ngram first, then drafters), so CLI order is irrelevant.

Step 1 — shapes (the part that decides the answer)

Shape	Prompt	Who it can favour
prose	1k filler + "reply with one sentence"	model-backed drafters only
echo	~1.5k passage of UNIQUE lines + "repeat verbatim"	ngram (output IS the prompt)
structured	repetitive output, no prompt overlap	ngram, if spans really repeat
identity	short deterministic question	controls for lossless verification

Design the echo passage so no long span repeats inside it — otherwise you measure the matcher against itself. For structured, use lines that genuinely repeat: unique ids per line make it a negative control, not a repetition test.

Step 2 — protocol

Stop the unit (sudo systemctl stop <svc>), pkill -x llama-server; wait for the port.
Per config: write an env file with overrides, start the serve script with FRANK_*_ENV_FILE, poll /health until ok, then measure.
Restore the unit in a finally and shout if it does not come back healthy. A crashed harness that leaves production down is worse than no harness.
Compare greedy text across configs per shape; warn loudly on any mismatch (speculation must be lossless) and on any EMPTY answer (an empty answer makes the comparison vacuous).

Step 3 — read the right counter

Per request: timings.draft_n / timings.draft_n_accepted, predicted_per_second.
Cumulative: llamacpp:spec_decode_num_drafts_total / ..._accepted_tokens_total.
draft_n == 0 while tokens flow = the proposer is not engaged, even when every flag looks right.
A config with zero drafting is a useful negative control: it proves the other config's drafts came from the proposer under test.

Traps that produce false verdicts

Thinking eats the answer. Echo/structured prompts with thinking on put the copy in reasoning_content; content is empty, so identity checks compare '' to ''. Disable per request: chat_template_kwargs: {"enable_thinking": false}.
No banner. ngram-mod prints nothing at startup; the journal shows only the drafter's adding speculative implementation. Never conclude "chain not engaged" from the load log.
Windows change the verdict. ngram-mod --spec-ngram-mod-n-min/--spec-ngram-mod-n-max decide which traffic it can see: too short → proposals on ordinary prose fragments, rejected, verification still paid (measured 43.9 → 36.0 tok/s). Sweep the window in the same run as the configs.
Unused-tensor warnings are normal: with draft-dflash on an MTP-carrying GGUF, blk.<n>.nextn.* is unused ... ignoring — correct, not a truncated file.
Harness hygiene: absolute script paths (a relative path plus cd to its own dir double-counts the prefix), pkill -x never -f.

Interpreting

Report decode tok/s per shape plus acceptance (accepted/draft_n) and mean accepted length; a config that wins on echo and loses on prose is a traffic decision, not a winner. Then wire the winner into the deployed env file with the measured table as its justification comment, and assert it in the test seam.

4.3 KiB Raw Blame History