Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
78 lines
4.5 KiB
Markdown
78 lines
4.5 KiB
Markdown
---
|
|
name: spec-decode-ab-single-gpu-host
|
|
description: "A/B a speculative-decoding variant (or any serve-flag challenger) against an incumbent LLM endpoint on a single-GPU host that serves users: stop the unit, run the challenger from a patched env file, prove losslessness by identical greedy text, and keep the incumbent unless the challenger wins."
|
|
---
|
|
|
|
# A/B a serve-flag challenger against a live LLM endpoint
|
|
|
|
Use when asked "try it without X / with Y" for an inference endpoint that is
|
|
serving users from a GPU you cannot share (one model copy per host). The
|
|
deliverable is a measurement plus a decision, and the endpoint must end the
|
|
experiment on the configuration you started it with.
|
|
|
|
Ready-made harness for llama.cpp endpoints: `scripts/bench-spec-ab.py`
|
|
(`<env-file> <serve-script> [--base-url]`) — it stops the unit, runs each
|
|
config, compares greedy text and restores the unit. Below is what it encodes
|
|
and the traps it hit.
|
|
|
|
## Protocol
|
|
|
|
1. **Fixed measurement shapes, one prompt per phase.** A large prompt with a
|
|
tiny budget isolates prefill; a small prompt with a long budget isolates
|
|
decode. `temperature 0` throughout. Never compare across sessions — run the
|
|
*incumbent* again in the same session, on the same box, or you are comparing
|
|
two machines' worth of noise.
|
|
2. **Prove losslessness, do not claim it.** Send one fixed prompt at temp 0 to
|
|
every config and diff the greedy text. Speculation is only allowed to buy
|
|
speed; if the texts differ, stop and treat it as a correctness bug.
|
|
3. **Stop the unit for the duration** (benchmarking policy on these hosts: a
|
|
bench beside a live service starves the GPU). `sudo systemctl stop <unit> &&
|
|
passive`, run the challenger by hand with `setsid`, then `sudo systemctl start
|
|
<unit>` at the end *whatever happens* — wrap the restore so an exception
|
|
cannot leave the fleet dark.
|
|
4. **Override tunables through a patched env file**, never through the shell:
|
|
serve scripts that `source` their env file with `set -a` win over environment
|
|
variables, so `CTX=... bash serve.sh` silently does nothing. `sed` the repo env
|
|
into `/tmp/challenger.env` and point the service's env-file variable at it.
|
|
5. **Read the acceptance counters, not the flags.** `/metrics`
|
|
(`llamacpp:spec_decode_num_drafts_total` vs `..._accepted_tokens_total`) or the
|
|
per-request `timings.draft_n` / `draft_n_accepted`. A draft rate of 0 while
|
|
tokens are produced means the drafter is not running even though the flags look
|
|
right.
|
|
6. **Sweep the depth, not just the mode.** Draft depth is model-specific and
|
|
single-valued: measure at least two (e.g. n_max 3 and 7). Deeper often loses —
|
|
at n=7 a third of proposals survived, at n=3 more than half.
|
|
7. **Watch per-card VRAM** (`rocm-smi --showmeminfo vram`). A proposer is not
|
|
free: a drafter file cost 2.06 GB on its card while loading the target's own
|
|
in-model MTP head cost ~3.3 GB.
|
|
8. **Decide and land it.** If the challenger loses, say so and change nothing;
|
|
if it wins, put the value in the repo env file, run the suite, deploy, and
|
|
verify with a real completion. Record the losing numbers in the env file with
|
|
"do not switch without re-measuring" so nobody re-litigates it.
|
|
|
|
## Traps that cost time here
|
|
|
|
- **`pgrep -f <script>` and `pkill -f <pattern>` match your own command line**
|
|
(and the ssh wrapper's). Use `pgrep -x llama-server` / `pkill -x` and confirm
|
|
with `ps -eo pid,args | grep "[s]cript"`.
|
|
- **PID 1 is not the app** inside a container; for units, `systemctl restart`
|
|
is the reload.
|
|
- **A retired predecessor's port lingers**: a published-port proxy leaves the
|
|
endpoint port in TIME_WAIT. Wait for the port to be free (and for the manual
|
|
instance to be killed) before starting the challenger, or the bind fails with
|
|
address-in-use long after the unit stopped.
|
|
- **Nested f-strings with backslash escapes are a SyntaxError on the fleet's
|
|
Python** (<3.12 semantics). Build log lines with string concatenation.
|
|
- **If a config dies at load, read the error literally before "fixing" it.**
|
|
`context type MTP requested but model doesn't contain MTP layers` was not a
|
|
bad flag: a chain has ONE draft context, so MTP met the *drafter's* file. Two
|
|
proposers that each own the draft slot cannot be combined, and pinning that
|
|
in a seam prevents someone "fixing" the flag later.
|
|
|
|
## Reporting
|
|
|
|
One table: config, decode tok/s, acceptance, VRAM per card. Then one line per
|
|
finding, each with its number. Then the decision, plus whether the deployed
|
|
state changed. "The thing you asked me to try lost to what was already running"
|
|
is a complete and useful answer.
|