Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
101 lines
5.1 KiB
Markdown
101 lines
5.1 KiB
Markdown
---
|
|
name: llm-replica-routing-prefix-cache
|
|
description: "Route requests across multiple LLM replicas by workload shape (pin agent-loop continuations for prefix-cache hits, spread batches for parallel prefill), with the measurement traps and the capacity wall that no routing policy fixes."
|
|
---
|
|
|
|
# Prefix-cache-aware routing for multi-replica LLM serving
|
|
|
|
Use when several single-GPU inference replicas sit behind one endpoint and
|
|
clients send long shared prefixes (agent loops, chat with a big system prompt,
|
|
RAG with a fixed context).
|
|
|
|
## The two failure modes, both measurable
|
|
|
|
Measure with a **sequential-turn client**, not only a batch benchmark: a batch
|
|
of distinct prompts parallelises prefill across replicas, so round-robin looks
|
|
great there and hides the problem.
|
|
|
|
1. **Round-robin loses every other cache hit.** The loop resends the same
|
|
prefix each turn; alternating replicas means half the turns prefill from
|
|
scratch. Measured: 90k-token contexts, warm-turn TTFT 2676 ms round-robin
|
|
vs 1096 ms pinned.
|
|
2. **Pinning everything loses a card.** A cold batch pays one card's prefill
|
|
throughput: 16 requests in 18.6 s spread vs 30.1 s pinned.
|
|
|
|
## The rule that gets both
|
|
|
|
Route by request shape:
|
|
|
|
- **Continuation** (request carries more than one message — the signature of an
|
|
agent loop re-sending its context) -> pin to the replica that served that
|
|
prefix before.
|
|
- **Anything else** (one-shot question, batch member, parallel subagent) ->
|
|
spread across replicas with a **counter**, not with a hash of the prefix.
|
|
|
|
Measured on the same host: 19.97 s burst *and* 1095 ms warm turns.
|
|
|
|
Also add **failover**: a replica that restarts is invisible to a health table
|
|
refreshed every 10 s, so a request routed to it returns 502 even though the
|
|
other replica is up. On connection failure, demote it immediately and retry the
|
|
request on a live replica before writing anything to the client (the OpenAI
|
|
endpoints are stateless, so re-sending a request that never reached an engine
|
|
is safe; never retry after streaming started). Measured: six requests during a
|
|
replica stop, six 200s, one logged failover, no client-visible error.
|
|
|
|
## The capacity wall that no policy fixes
|
|
|
|
Per-card KV is finite and **a running sequence and its cached prefix draw on
|
|
the same pool**. With N families on M cards, one card carries two of them, and
|
|
its budget must hold both cached prefixes plus whichever sequence is
|
|
generating. Three concurrent 50-80k-token families against 122,636 tokens per
|
|
card thrash regardless of how requests are placed:
|
|
|
|
| policy | result |
|
|
|---|---|
|
|
| counter placement | heavy 25.4 s median (23 % cached), mediums 34.7 / 45.2 s |
|
|
| load-balanced placement (new families to the card carrying less context) | no improvement — reverted |
|
|
| one large request admitted per card | no improvement — reverted |
|
|
| smaller/larger prefill chunk | *costs* KV in both directions (a sweet spot exists; find it) |
|
|
|
|
So: measure placement and admission before shipping them, and when the working
|
|
set exceeds aggregate KV, say so — the levers are capacity (another card),
|
|
fewer concurrent large contexts (client-side context compaction), or a different
|
|
platform. Routing cannot rescue arithmetic.
|
|
|
|
## Measurement traps
|
|
|
|
- **A lifetime engine counter is not a per-request verdict.** "Prefix cache hit
|
|
rate: 21 %" is an average over every prompt the engine ever saw, including
|
|
benchmark traffic that never repeats. Probe the question directly instead:
|
|
send the same long prompt to an idle replica twice — 22.3 s cold then 1.4 s
|
|
warm is what a working cache looks like (and it holds with a varying tail).
|
|
- **Salt your corpora.** Two runs sharing a prefix measure a warm cache, not
|
|
the thing under test; a unique leading line per run makes the prefix new.
|
|
- **Record the sample size.** A 14-minute arm and an 8-minute arm are not a
|
|
comparison, and medians from five turns say little.
|
|
- **Check the tool's own requirements**: `vllm bench serve` needs
|
|
`--tokenizer <local path>` (plus the model directory mounted) or it tries to
|
|
resolve the served alias on HuggingFace and dies before sending a request —
|
|
a run that produces no numbers and no obvious error.
|
|
- **Verify metric names against the live endpoint** before writing dashboards
|
|
or alerts: guessing one that does not exist is invisible until someone needs
|
|
the graph.
|
|
|
|
## Test shape that catches it
|
|
|
|
Drive the routing function directly with synthetic request bodies (keep the
|
|
policy in a stdlib-only module so tests need no HTTP framework):
|
|
|
|
- a batch of single-turn requests -> assert both replicas are used and
|
|
consecutive picks alternate;
|
|
- a growing conversation (same leading bytes, longer history) -> assert every
|
|
turn agrees with the first turn's replica;
|
|
- each mode with a replica marked down -> assert the live one is chosen, and
|
|
-1 when none is;
|
|
- the fixed comparison modes (`roundrobin`, `affinity`) kept behind an env var
|
|
so the choice can be re-measured rather than argued.
|
|
|
|
One more: the identity of a conversation is its **first N KiB** of prompt. If
|
|
the tested prefix is shorter than that window its varying tail falls inside it,
|
|
turns hash differently, and pinning silently never happens.
|