command-center/agent/managed-skills/spec-decode-ab-single-gpu-host/SKILL.md

name, description

name	description
spec-decode-ab-single-gpu-host	A/B a speculative-decoding variant (or any serve-flag challenger) against an incumbent LLM endpoint on a single-GPU host that serves users: stop the unit, run the challenger from a patched env file, prove losslessness by identical greedy text, and keep the incumbent unless the challenger wins.

A/B a serve-flag challenger against a live LLM endpoint

Use when asked "try it without X / with Y" for an inference endpoint that is serving users from a GPU you cannot share (one model copy per host). The deliverable is a measurement plus a decision, and the endpoint must end the experiment on the configuration you started it with.

Ready-made harness for llama.cpp endpoints: scripts/bench-spec-ab.py (<env-file> <serve-script> [--base-url]) — it stops the unit, runs each config, compares greedy text and restores the unit. Below is what it encodes and the traps it hit.

Protocol

1. Fixed measurement shapes, one prompt per phase. A large prompt with a tiny budget isolates prefill; a small prompt with a long budget isolates decode. temperature 0 throughout. Never compare across sessions — run the incumbent again in the same session, on the same box, or you are comparing two machines' worth of noise.
2. Prove losslessness, do not claim it. Send one fixed prompt at temp 0 to every config and diff the greedy text. Speculation is only allowed to buy speed; if the texts differ, stop and treat it as a correctness bug.
3. Stop the unit for the duration (benchmarking policy on these hosts: a bench beside a live service starves the GPU). sudo systemctl stop <unit> && passive, run the challenger by hand with setsid, then sudo systemctl start <unit> at the end whatever happens — wrap the restore so an exception cannot leave the fleet dark.
4. Override tunables through a patched env file, never through the shell: serve scripts that source their env file with set -a win over environment variables, so CTX=... bash serve.sh silently does nothing. sed the repo env into /tmp/challenger.env and point the service's env-file variable at it.
5. Read the acceptance counters, not the flags. /metrics (llamacpp:spec_decode_num_drafts_total vs ..._accepted_tokens_total) or the per-request timings.draft_n / draft_n_accepted. A draft rate of 0 while tokens are produced means the drafter is not running even though the flags look right.
6. Sweep the depth, not just the mode. Draft depth is model-specific and single-valued: measure at least two (e.g. n_max 3 and 7). Deeper often loses — at n=7 a third of proposals survived, at n=3 more than half.
7. Watch per-card VRAM (rocm-smi --showmeminfo vram). A proposer is not free: a drafter file cost 2.06 GB on its card while loading the target's own in-model MTP head cost ~3.3 GB.
8. Decide and land it. If the challenger loses, say so and change nothing; if it wins, put the value in the repo env file, run the suite, deploy, and verify with a real completion. Record the losing numbers in the env file with "do not switch without re-measuring" so nobody re-litigates it.

Traps that cost time here

• pgrep -f <script> and pkill -f <pattern> match your own command line (and the ssh wrapper's). Use pgrep -x llama-server / pkill -x and confirm with ps -eo pid,args | grep "[s]cript".
• PID 1 is not the app inside a container; for units, systemctl restart is the reload.
• A retired predecessor's port lingers: a published-port proxy leaves the endpoint port in TIME_WAIT. Wait for the port to be free (and for the manual instance to be killed) before starting the challenger, or the bind fails with address-in-use long after the unit stopped.
• Nested f-strings with backslash escapes are a SyntaxError on the fleet's Python (<3.12 semantics). Build log lines with string concatenation.
• If a config dies at load, read the error literally before "fixing" it. context type MTP requested but model doesn't contain MTP layers was not a bad flag: a chain has ONE draft context, so MTP met the drafter's file. Two proposers that each own the draft slot cannot be combined, and pinning that in a seam prevents someone "fixing" the flag later.

Reporting

One table: config, decode tok/s, acceptance, VRAM per card. Then one line per finding, each with its number. Then the decision, plus whether the deployed state changed. "The thing you asked me to try lost to what was already running" is a complete and useful answer.