Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
3.4 KiB
name, description
| name | description |
|---|---|
| live-llm-benchmark-hygiene | Measure and tune a live LLM server without fooling yourself: workload-matched corpora, salted runs against warm prefix caches, turn-level clients, engine ledger for capacity, soak tests, and reboot verification. |
Benchmarking a live LLM server honestly
Use when measuring or tuning a served model (vLLM, llama.cpp, halogen, radiance, any OpenAI-compatible endpoint). The point is to avoid the four confounds that turned a night's numbers into fiction before each one was fixed.
1. Match the corpus to the workload
Synthetic token soup (vllm bench serve --dataset-name random) under-reports
speculative decoding badly: measured 2.9-3.6 tok/update against 3.86 on real
code, and acceptance — not step time — is what moves throughput. For a coding
fleet, build the corpus from the repo's own sources: file body plus an
instruction that requires reading it. Then tok/update and TPOT mean something.
2. Salt every run so the prefix cache cannot help
vLLM runs with --enable-prefix-caching, and a repeated corpus silently
measures a warm cache. Put a unique per-run token in the first bytes (it is both
the engine's cache key and, if a proxy routes on it, the routing key). Without
this, the second run of any comparison is not comparable to the first.
3. Measure turns, not only batches
A batch of distinct prompts parallelises prefill and hides the real cost of a conversation: an agent loop re-sends its context every turn. Two workloads need two clients:
- batch/throughput —
vllm bench serve --dataset-name custom --dataset-path <jsonl>with--custom-output-len, several concurrency levels. - sequential turns — a small client that sends turn after turn with growing history and a large shared prefix, timing TTFT of the first streamed chunk per turn. This is the only measurement that exposes prefix-cache behaviour.
4. Read the engine's own ledger for capacity
Do not infer memory from the card's size. The server prints the truth at boot:
Available KV cache memory: 5.44 GiB
GPU KV cache size: 122,636 tokens
Maximum concurrency for 114,688 tokens per request: 1.07x
That is the number to quote for "how many concurrent conversations fit".
Related trap: the arena/KV budget is not monotonic in chunk size — on one
host, available KV was 5.44 GiB at CHUNK=8192, 4.22 GiB at 16384 and 4.16 GiB
at 4096 (where it would not even boot). Never assume "smaller chunk frees KV";
measure three points before treating a knob as a lever.
5. Soak before believing a tuning change
Run several families concurrently for 15-30 minutes and count failures, not just latency: a nightly soak returned 155/155 answers with zero failovers while only 25% of turns hit the cache — which is how the capacity ceiling was found as a latency effect rather than an error. Report hit/miss split, medians, and the min/max, because the median of a bimodal distribution describes neither mode.
6. Verification that is not the benchmark
- Run the client the fleet actually uses (e.g.
omp --model <alias> -p --no-tools) — a config file that parses is not a config that works, and a non-obvious API mode (raw/v1/completions) may still be fine because the client templates its own prompt. - A deployed inference unit is not proven until an unattended reboot brings
it back: verify
is-active,/health, and one real completion afterwards.