Two more hypotheses, both measured on the same three-family workload (one
~80k-token family plus two ~50k-token families, each turn re-sending its
document):
1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt
sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too
(22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%"
is an average across every prompt that replica ever saw, not a verdict on
any one request, and the pinning policy is safe.
2. Serialising large requests does not fix the overload. Admission control
(at most one >16k-token request per card, small requests untouched) gave no
clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums
34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was
reverted like the placement change before it.
The arithmetic now closes: with three families on two cards, one card holds two
families and must carry 2 x 50k of cached prefix plus the 50k of the sequence
generating — 150k against 122,636 tokens. Those co-tenants evict each other
regardless of concurrency limits. The wall is capacity, and the runbook says so
with the numbers.
Frank
4f2f49d2d0