---
name: spec-decode-ab-single-gpu-host
description: "A/B a speculative-decoding variant (or any serve-flag challenger) against an incumbent LLM endpoint on a single-GPU host that serves users: stop the unit, run the challenger from a patched env file, prove losslessness by identical greedy text, and keep the incumbent unless the challenger wins."
---

# A/B a serve-flag challenger against a live LLM endpoint

Use when asked "try it without X / with Y" for an inference endpoint that is
serving users from a GPU you cannot share (one model copy per host). The
deliverable is a measurement plus a decision, and the endpoint must end the
experiment on the configuration you started it with.

Ready-made harness for llama.cpp endpoints: `scripts/bench-spec-ab.py`
(`<env-file> <serve-script> [--base-url]`) — it stops the unit, runs each
config, compares greedy text and restores the unit. Below is what it encodes
and the traps it hit.

## Protocol

1. **Fixed measurement shapes, one prompt per phase.** A large prompt with a
tiny budget isolates prefill; a small prompt with a long budget isolates
decode. `temperature 0` throughout. Never compare across sessions — run the
*incumbent* again in the same session, on the same box, or you are comparing
two machines' worth of noise.
2. **Prove losslessness, do not claim it.** Send one fixed prompt at temp 0 to
every config and diff the greedy text. Speculation is only allowed to buy
speed; if the texts differ, stop and treat it as a correctness bug.
3. **Stop the unit for the duration** (benchmarking policy on these hosts: a
bench beside a live service starves the GPU). `sudo systemctl stop <unit> &&
passive`, run the challenger by hand with `setsid`, then `sudo systemctl start
<unit>` at the end *whatever happens* — wrap the restore so an exception
cannot leave the fleet dark.
4. **Override tunables through a patched env file**, never through the shell:
serve scripts that `source` their env file with `set -a` win over environment
variables, so `CTX=... bash serve.sh` silently does nothing. `sed` the repo env
into `/tmp/challenger.env` and point the service's env-file variable at it.
5. **Read the acceptance counters, not the flags.** `/metrics`
(`llamacpp:spec_decode_num_drafts_total` vs `..._accepted_tokens_total`) or the
per-request `timings.draft_n` / `draft_n_accepted`. A draft rate of 0 while
tokens are produced means the drafter is not running even though the flags look
right.
6. **Sweep the depth, not just the mode.** Draft depth is model-specific and
single-valued: measure at least two (e.g. n_max 3 and 7). Deeper often loses —
at n=7 a third of proposals survived, at n=3 more than half.
7. **Watch per-card VRAM** (`rocm-smi --showmeminfo vram`). A proposer is not
free: a drafter file cost 2.06 GB on its card while loading the target's own
in-model MTP head cost ~3.3 GB.
8. **Decide and land it.** If the challenger loses, say so and change nothing;
if it wins, put the value in the repo env file, run the suite, deploy, and
verify with a real completion. Record the losing numbers in the env file with
"do not switch without re-measuring" so nobody re-litigates it.

## Traps that cost time here

- **`pgrep -f <script>` and `pkill -f <pattern>` match your own command line**
  (and the ssh wrapper's). Use `pgrep -x llama-server` / `pkill -x` and confirm
  with `ps -eo pid,args | grep "[s]cript"`.
- **PID 1 is not the app** inside a container; for units, `systemctl restart`
is the reload.
- **A retired predecessor's port lingers**: a published-port proxy leaves the
endpoint port in TIME_WAIT. Wait for the port to be free (and for the manual
instance to be killed) before starting the challenger, or the bind fails with
address-in-use long after the unit stopped.
- **Nested f-strings with backslash escapes are a SyntaxError on the fleet's
Python** (<3.12 semantics). Build log lines with string concatenation.
- **If a config dies at load, read the error literally before "fixing" it.**
  `context type MTP requested but model doesn't contain MTP layers` was not a
  bad flag: a chain has ONE draft context, so MTP met the *drafter's* file. Two
  proposers that each own the draft slot cannot be combined, and pinning that
  in a seam prevents someone "fixing" the flag later.

## Reporting

One table: config, decode tok/s, acceptance, VRAM per card. Then one line per
finding, each with its number. Then the decision, plus whether the deployed
state changed. "The thing you asked me to try lost to what was already running"
is a complete and useful answer.