---
name: llm-replica-routing-prefix-cache
description: "Route requests across multiple LLM replicas by workload shape (pin agent-loop continuations for prefix-cache hits, spread batches for parallel prefill), with the measurement traps and the capacity wall that no routing policy fixes."
---

# Prefix-cache-aware routing for multi-replica LLM serving

Use when several single-GPU inference replicas sit behind one endpoint and
clients send long shared prefixes (agent loops, chat with a big system prompt,
RAG with a fixed context).

## The two failure modes, both measurable

Measure with a **sequential-turn client**, not only a batch benchmark: a batch
of distinct prompts parallelises prefill across replicas, so round-robin looks
great there and hides the problem.

1. **Round-robin loses every other cache hit.** The loop resends the same
   prefix each turn; alternating replicas means half the turns prefill from
   scratch. Measured: 90k-token contexts, warm-turn TTFT 2676 ms round-robin
   vs 1096 ms pinned.
2. **Pinning everything loses a card.** A cold batch pays one card's prefill
   throughput: 16 requests in 18.6 s spread vs 30.1 s pinned.

## The rule that gets both

Route by request shape:

- **Continuation** (request carries more than one message — the signature of an
  agent loop re-sending its context) -> pin to the replica that served that
  prefix before.
- **Anything else** (one-shot question, batch member, parallel subagent) ->
  spread across replicas with a **counter**, not with a hash of the prefix.

Measured on the same host: 19.97 s burst *and* 1095 ms warm turns.

Also add **failover**: a replica that restarts is invisible to a health table
refreshed every 10 s, so a request routed to it returns 502 even though the
other replica is up. On connection failure, demote it immediately and retry the
request on a live replica before writing anything to the client (the OpenAI
endpoints are stateless, so re-sending a request that never reached an engine
is safe; never retry after streaming started). Measured: six requests during a
replica stop, six 200s, one logged failover, no client-visible error.

## The capacity wall that no policy fixes

Per-card KV is finite and **a running sequence and its cached prefix draw on
the same pool**. With N families on M cards, one card carries two of them, and
its budget must hold both cached prefixes plus whichever sequence is
generating. Three concurrent 50-80k-token families against 122,636 tokens per
card thrash regardless of how requests are placed:

| policy | result |
|---|---|
| counter placement | heavy 25.4 s median (23 % cached), mediums 34.7 / 45.2 s |
| load-balanced placement (new families to the card carrying less context) | no improvement — reverted |
| one large request admitted per card | no improvement — reverted |
| smaller/larger prefill chunk | *costs* KV in both directions (a sweet spot exists; find it) |

So: measure placement and admission before shipping them, and when the working
set exceeds aggregate KV, say so — the levers are capacity (another card),
fewer concurrent large contexts (client-side context compaction), or a different
platform. Routing cannot rescue arithmetic.

## Measurement traps

- **A lifetime engine counter is not a per-request verdict.** "Prefix cache hit
  rate: 21 %" is an average over every prompt the engine ever saw, including
  benchmark traffic that never repeats. Probe the question directly instead:
  send the same long prompt to an idle replica twice — 22.3 s cold then 1.4 s
  warm is what a working cache looks like (and it holds with a varying tail).
- **Salt your corpora.** Two runs sharing a prefix measure a warm cache, not
  the thing under test; a unique leading line per run makes the prefix new.
- **Record the sample size.** A 14-minute arm and an 8-minute arm are not a
  comparison, and medians from five turns say little.
- **Check the tool's own requirements**: `vllm bench serve` needs
  `--tokenizer <local path>` (plus the model directory mounted) or it tries to
  resolve the served alias on HuggingFace and dies before sending a request —
  a run that produces no numbers and no obvious error.
- **Verify metric names against the live endpoint** before writing dashboards
  or alerts: guessing one that does not exist is invisible until someone needs
  the graph.

## Test shape that catches it

Drive the routing function directly with synthetic request bodies (keep the
policy in a stdlib-only module so tests need no HTTP framework):

- a batch of single-turn requests -> assert both replicas are used and
  consecutive picks alternate;
- a growing conversation (same leading bytes, longer history) -> assert every
  turn agrees with the first turn's replica;
- each mode with a replica marked down -> assert the live one is chosen, and
  -1 when none is;
- the fixed comparison modes (`roundrobin`, `affinity`) kept behind an env var
  so the choice can be re-measured rather than argued.

One more: the identity of a conversation is its **first N KiB** of prompt. If
the tested prefix is shorter than that window its varying tail falls inside it,
turns hash differently, and pinning silently never happens.