agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,70 @@
|
||||
---
|
||||
name: live-llm-benchmark-hygiene
|
||||
description: "Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification."
|
||||
---
|
||||
|
||||
# Benchmarking a live LLM server honestly
|
||||
|
||||
Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance,
|
||||
any OpenAI-compatible endpoint). The point is to avoid the four confounds that
|
||||
turned a night's numbers into fiction before each one was fixed.
|
||||
|
||||
## 1. Match the corpus to the workload
|
||||
|
||||
Synthetic token soup (`vllm bench serve --dataset-name random`) under-reports
|
||||
speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
|
||||
code, and acceptance — not step time — is what moves throughput. For a coding
|
||||
fleet, build the corpus from the repo's own sources: file body plus an
|
||||
instruction that requires reading it. Then `tok/update` and TPOT mean something.
|
||||
|
||||
## 2. Salt every run so the prefix cache cannot help
|
||||
|
||||
vLLM runs with `--enable-prefix-caching`, and a repeated corpus silently
|
||||
measures a warm cache. Put a unique per-run token in the first bytes (it is both
|
||||
the engine's cache key and, if a proxy routes on it, the routing key). Without
|
||||
this, the second run of any comparison is not comparable to the first.
|
||||
|
||||
## 3. Measure turns, not only batches
|
||||
|
||||
A batch of distinct prompts parallelises prefill and hides the real cost of a
|
||||
conversation: an agent loop re-sends its context every turn. Two workloads need
|
||||
two clients:
|
||||
|
||||
- **batch/throughput** — `vllm bench serve --dataset-name custom --dataset-path
|
||||
<jsonl>` with `--custom-output-len`, several concurrency levels.
|
||||
- **sequential turns** — a small client that sends turn after turn with growing
|
||||
history and a large shared prefix, timing **TTFT of the first streamed chunk**
|
||||
per turn. This is the only measurement that exposes prefix-cache behaviour.
|
||||
|
||||
## 4. Read the engine's own ledger for capacity
|
||||
|
||||
Do not infer memory from the card's size. The server prints the truth at boot:
|
||||
|
||||
```
|
||||
Available KV cache memory: 5.44 GiB
|
||||
GPU KV cache size: 122,636 tokens
|
||||
Maximum concurrency for 114,688 tokens per request: 1.07x
|
||||
```
|
||||
|
||||
That is the number to quote for "how many concurrent conversations fit".
|
||||
Related trap: the arena/KV budget is **not monotonic in chunk size** — on one
|
||||
host, available KV was 5.44 GiB at `CHUNK=8192`, 4.22 GiB at 16384 and 4.16 GiB
|
||||
at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
|
||||
measure three points before treating a knob as a lever.
|
||||
|
||||
## 5. Soak before believing a tuning change
|
||||
|
||||
Run several families concurrently for 15-30 minutes and count failures, not just
|
||||
latency: a nightly soak returned 155/155 answers with zero failovers while only
|
||||
25% of turns hit the cache — which is how the capacity ceiling was found as a
|
||||
*latency* effect rather than an error. Report hit/miss split, medians, and the
|
||||
min/max, because the median of a bimodal distribution describes neither mode.
|
||||
|
||||
## 6. Verification that is not the benchmark
|
||||
|
||||
- Run the client the fleet actually uses (e.g. `omp --model <alias> -p
|
||||
--no-tools`) — a config file that parses is not a config that works, and a
|
||||
non-obvious API mode (raw `/v1/completions`) may still be fine because the
|
||||
client templates its own prompt.
|
||||
- A deployed inference unit is not proven until an **unattended reboot** brings
|
||||
it back: verify `is-active`, `/health`, and one real completion afterwards.
|
||||
Reference in new issue
Block a user