Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
68 lines
3.2 KiB
Markdown
68 lines
3.2 KiB
Markdown
---
|
|
name: llm-serving-benchmark-methodology
|
|
description: "Measure and A/B a self-hosted multi-replica LLM endpoint honestly: corpus construction, per-arm salting, repeated samples, engine-counter semantics, and the traps that make results meaningless."
|
|
---
|
|
|
|
# Bench a multi-replica LLM deployment so the numbers mean something
|
|
|
|
Use when measuring or tuning a self-hosted LLM endpoint (vLLM, llama.cpp,
|
|
radiance/MXFP4), especially behind a proxy in front of several replicas.
|
|
|
|
## Set up a client that can actually send requests
|
|
|
|
`vllm bench serve` needs `--tokenizer <local model path>` with the model
|
|
directory mounted. Given only `--model <served-alias>` it tries to resolve that
|
|
alias on HuggingFace and dies before the first request — a run that looks like
|
|
it worked and produced nothing. Confirm with a single prompt before benching.
|
|
|
|
## Build the workload from real material
|
|
|
|
`--dataset-name random` generates token soup whose draft acceptance is nothing
|
|
like production (measured 2.9-3.6 tok/update against 5.8 on code), so it
|
|
mis-ranks speculative-decoding settings. Generate a JSONL corpus from the
|
|
repository's own sources — `{"prompt": "<instruction>\n\n<file body>"}` — and
|
|
fix output length with `--custom-output-len`. Keep two corpus shapes:
|
|
|
|
- **distinct prompts** (batch behaviour, prefill parallelism)
|
|
- **a shared long prefix with a varying tail** (conversation turns, where
|
|
prefix caching decides everything)
|
|
|
|
## Salt every arm
|
|
|
|
Put a unique marker in the first bytes of every prompt, different per A/B arm.
|
|
Without it the second arm silently measures the first arm's warm cache and both
|
|
arms look identical.
|
|
|
|
## Measure the same workload before and after
|
|
|
|
Keep the workload fixed and change one knob. Repeat each arm — run-to-run
|
|
variance of 5-10 % is normal, and a single sample nearly made me ship the wrong
|
|
setting. Report medians per family/class, not one aggregate number.
|
|
|
|
## Read engine counters correctly
|
|
|
|
- vLLM's `Prefix cache hit rate:` line is a **lifetime average** across every
|
|
prompt that replica ever saw. It cannot answer "is this conversation served
|
|
warm?". Send the same prompt twice to an idle engine and compare TTFTs
|
|
instead (measured: 22.3 s cold → 1.4 s warm for a 50k-token prefix).
|
|
- `GPU KV cache usage` tells you the pool is not full; it does not tell you a
|
|
prefix survived. Capacity per card is the boot-time ledger line
|
|
(`GPU KV cache size: N tokens`), not the card's VRAM.
|
|
- Reserve memory is not allocation: a bigger reserve does not free anything for
|
|
anyone else, it only removes cache.
|
|
|
|
## Route-shape traps
|
|
|
|
A cold batch of distinct prompts prefills in parallel across replicas; the same
|
|
conversation's next turn only pays off if it lands where its prefix is cached.
|
|
These pull in opposite directions, so measure both shapes separately and route
|
|
by request shape (spread single-turn, pin continuations).
|
|
|
|
## When the answer is "no"
|
|
|
|
Write down rejected hypotheses with their numbers and the mechanism. Placement
|
|
balance, concurrency admission and chunk-size tweaks all looked obvious and all
|
|
measured flat or worse on this fleet; the wall turned out to be arithmetic
|
|
(cached prefixes plus live sequences share one pool). A runbook that records
|
|
that saves the next operator a night.
|