Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
5.1 KiB
name, description
| name | description |
|---|---|
| llm-replica-routing-prefix-cache | Route requests across multiple LLM replicas by workload shape (pin agent-loop continuations for prefix-cache hits, spread batches for parallel prefill), with the measurement traps and the capacity wall that no routing policy fixes. |
Prefix-cache-aware routing for multi-replica LLM serving
Use when several single-GPU inference replicas sit behind one endpoint and clients send long shared prefixes (agent loops, chat with a big system prompt, RAG with a fixed context).
The two failure modes, both measurable
Measure with a sequential-turn client, not only a batch benchmark: a batch of distinct prompts parallelises prefill across replicas, so round-robin looks great there and hides the problem.
- Round-robin loses every other cache hit. The loop resends the same prefix each turn; alternating replicas means half the turns prefill from scratch. Measured: 90k-token contexts, warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned.
- Pinning everything loses a card. A cold batch pays one card's prefill throughput: 16 requests in 18.6 s spread vs 30.1 s pinned.
The rule that gets both
Route by request shape:
- Continuation (request carries more than one message — the signature of an agent loop re-sending its context) -> pin to the replica that served that prefix before.
- Anything else (one-shot question, batch member, parallel subagent) -> spread across replicas with a counter, not with a hash of the prefix.
Measured on the same host: 19.97 s burst and 1095 ms warm turns.
Also add failover: a replica that restarts is invisible to a health table refreshed every 10 s, so a request routed to it returns 502 even though the other replica is up. On connection failure, demote it immediately and retry the request on a live replica before writing anything to the client (the OpenAI endpoints are stateless, so re-sending a request that never reached an engine is safe; never retry after streaming started). Measured: six requests during a replica stop, six 200s, one logged failover, no client-visible error.
The capacity wall that no policy fixes
Per-card KV is finite and a running sequence and its cached prefix draw on the same pool. With N families on M cards, one card carries two of them, and its budget must hold both cached prefixes plus whichever sequence is generating. Three concurrent 50-80k-token families against 122,636 tokens per card thrash regardless of how requests are placed:
| policy | result |
|---|---|
| counter placement | heavy 25.4 s median (23 % cached), mediums 34.7 / 45.2 s |
| load-balanced placement (new families to the card carrying less context) | no improvement — reverted |
| one large request admitted per card | no improvement — reverted |
| smaller/larger prefill chunk | costs KV in both directions (a sweet spot exists; find it) |
So: measure placement and admission before shipping them, and when the working set exceeds aggregate KV, say so — the levers are capacity (another card), fewer concurrent large contexts (client-side context compaction), or a different platform. Routing cannot rescue arithmetic.
Measurement traps
- A lifetime engine counter is not a per-request verdict. "Prefix cache hit rate: 21 %" is an average over every prompt the engine ever saw, including benchmark traffic that never repeats. Probe the question directly instead: send the same long prompt to an idle replica twice — 22.3 s cold then 1.4 s warm is what a working cache looks like (and it holds with a varying tail).
- Salt your corpora. Two runs sharing a prefix measure a warm cache, not the thing under test; a unique leading line per run makes the prefix new.
- Record the sample size. A 14-minute arm and an 8-minute arm are not a comparison, and medians from five turns say little.
- Check the tool's own requirements:
vllm bench serveneeds--tokenizer <local path>(plus the model directory mounted) or it tries to resolve the served alias on HuggingFace and dies before sending a request — a run that produces no numbers and no obvious error. - Verify metric names against the live endpoint before writing dashboards or alerts: guessing one that does not exist is invisible until someone needs the graph.
Test shape that catches it
Drive the routing function directly with synthetic request bodies (keep the policy in a stdlib-only module so tests need no HTTP framework):
- a batch of single-turn requests -> assert both replicas are used and consecutive picks alternate;
- a growing conversation (same leading bytes, longer history) -> assert every turn agrees with the first turn's replica;
- each mode with a replica marked down -> assert the live one is chosen, and -1 when none is;
- the fixed comparison modes (
roundrobin,affinity) kept behind an env var so the choice can be re-measured rather than argued.
One more: the identity of a conversation is its first N KiB of prompt. If the tested prefix is shorter than that window its varying tail falls inside it, turns hash differently, and pinning silently never happens.