agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
Frank committed
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,77 @@
|
||||
---
|
||||
name: spec-decode-ab-single-gpu-host
|
||||
description: "A/B a speculative-decoding variant (or any serve-flag challenger) against an incumbent LLM endpoint on a single-GPU host that serves users: stop the unit, run the challenger from a patched env file, prove losslessness by identical greedy text, and keep the incumbent unless the challenger wins."
|
||||
---
|
||||
|
||||
# A/B a serve-flag challenger against a live LLM endpoint
|
||||
|
||||
Use when asked "try it without X / with Y" for an inference endpoint that is
|
||||
serving users from a GPU you cannot share (one model copy per host). The
|
||||
deliverable is a measurement plus a decision, and the endpoint must end the
|
||||
experiment on the configuration you started it with.
|
||||
|
||||
Ready-made harness for llama.cpp endpoints: `scripts/bench-spec-ab.py`
|
||||
(`<env-file> <serve-script> [--base-url]`) — it stops the unit, runs each
|
||||
config, compares greedy text and restores the unit. Below is what it encodes
|
||||
and the traps it hit.
|
||||
|
||||
## Protocol
|
||||
|
||||
1. **Fixed measurement shapes, one prompt per phase.** A large prompt with a
|
||||
tiny budget isolates prefill; a small prompt with a long budget isolates
|
||||
decode. `temperature 0` throughout. Never compare across sessions — run the
|
||||
*incumbent* again in the same session, on the same box, or you are comparing
|
||||
two machines' worth of noise.
|
||||
2. **Prove losslessness, do not claim it.** Send one fixed prompt at temp 0 to
|
||||
every config and diff the greedy text. Speculation is only allowed to buy
|
||||
speed; if the texts differ, stop and treat it as a correctness bug.
|
||||
3. **Stop the unit for the duration** (benchmarking policy on these hosts: a
|
||||
bench beside a live service starves the GPU). `sudo systemctl stop <unit> &&
|
||||
passive`, run the challenger by hand with `setsid`, then `sudo systemctl start
|
||||
<unit>` at the end *whatever happens* — wrap the restore so an exception
|
||||
cannot leave the fleet dark.
|
||||
4. **Override tunables through a patched env file**, never through the shell:
|
||||
serve scripts that `source` their env file with `set -a` win over environment
|
||||
variables, so `CTX=... bash serve.sh` silently does nothing. `sed` the repo env
|
||||
into `/tmp/challenger.env` and point the service's env-file variable at it.
|
||||
5. **Read the acceptance counters, not the flags.** `/metrics`
|
||||
(`llamacpp:spec_decode_num_drafts_total` vs `..._accepted_tokens_total`) or the
|
||||
per-request `timings.draft_n` / `draft_n_accepted`. A draft rate of 0 while
|
||||
tokens are produced means the drafter is not running even though the flags look
|
||||
right.
|
||||
6. **Sweep the depth, not just the mode.** Draft depth is model-specific and
|
||||
single-valued: measure at least two (e.g. n_max 3 and 7). Deeper often loses —
|
||||
at n=7 a third of proposals survived, at n=3 more than half.
|
||||
7. **Watch per-card VRAM** (`rocm-smi --showmeminfo vram`). A proposer is not
|
||||
free: a drafter file cost 2.06 GB on its card while loading the target's own
|
||||
in-model MTP head cost ~3.3 GB.
|
||||
8. **Decide and land it.** If the challenger loses, say so and change nothing;
|
||||
if it wins, put the value in the repo env file, run the suite, deploy, and
|
||||
verify with a real completion. Record the losing numbers in the env file with
|
||||
"do not switch without re-measuring" so nobody re-litigates it.
|
||||
|
||||
## Traps that cost time here
|
||||
|
||||
- **`pgrep -f <script>` and `pkill -f <pattern>` match your own command line**
|
||||
(and the ssh wrapper's). Use `pgrep -x llama-server` / `pkill -x` and confirm
|
||||
with `ps -eo pid,args | grep "[s]cript"`.
|
||||
- **PID 1 is not the app** inside a container; for units, `systemctl restart`
|
||||
is the reload.
|
||||
- **A retired predecessor's port lingers**: a published-port proxy leaves the
|
||||
endpoint port in TIME_WAIT. Wait for the port to be free (and for the manual
|
||||
instance to be killed) before starting the challenger, or the bind fails with
|
||||
address-in-use long after the unit stopped.
|
||||
- **Nested f-strings with backslash escapes are a SyntaxError on the fleet's
|
||||
Python** (<3.12 semantics). Build log lines with string concatenation.
|
||||
- **If a config dies at load, read the error literally before "fixing" it.**
|
||||
`context type MTP requested but model doesn't contain MTP layers` was not a
|
||||
bad flag: a chain has ONE draft context, so MTP met the *drafter's* file. Two
|
||||
proposers that each own the draft slot cannot be combined, and pinning that
|
||||
in a seam prevents someone "fixing" the flag later.
|
||||
|
||||
## Reporting
|
||||
|
||||
One table: config, decode tok/s, acceptance, VRAM per card. Then one line per
|
||||
finding, each with its number. Then the decision, plus whether the deployed
|
||||
state changed. "The thing you asked me to try lost to what was already running"
|
||||
is a complete and useful answer.
|
||||
Reference in new issue
Block a user