frank-medium: measure admission control and cache health, keep neither change

Two more hypotheses, both measured on the same three-family workload (one
~80k-token family plus two ~50k-token families, each turn re-sending its
document):

1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt
   sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too
   (22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%"
   is an average across every prompt that replica ever saw, not a verdict on
   any one request, and the pinning policy is safe.

2. Serialising large requests does not fix the overload. Admission control
   (at most one >16k-token request per card, small requests untouched) gave no
   clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums
   34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was
   reverted like the placement change before it.

The arithmetic now closes: with three families on two cards, one card holds two
families and must carry 2 x 50k of cached prefix plus the 50k of the sequence
generating — 150k against 122,636 tokens. Those co-tenants evict each other
regardless of concurrency limits. The wall is capacity, and the runbook says so
with the numbers.
This commit is contained in:
Frank
2026-09-16 06:31:01 -07:00
parent 36007666cf
commit 4f2f49d2d0
@@ -381,6 +381,34 @@ whether one is. Three concurrent large sessions do not fit in two cards'
capacity, and the levers are therefore scheduling (fewer large contexts in
flight, which is what the harness's context compaction does) or a third card.
### Caching itself is healthy; the sum is not
Before blaming the cache, it was measured directly: one card, an unloaded
engine, a ~50k-token prompt sent three times — **22.3 s cold, then 1.4 s, then
1.4 s** — and the same result with a varying tail between turns (22.4 s cold,
1.41 s warm). So long-prefix reuse works exactly as the pinning policy assumes,
and the engine's lifetime "Prefix cache hit rate: 29 %" line is an average over
every benchmark prompt a replica has ever seen, not a statement about any one
request.
**Serialising large requests does not fix the overload either.** Admission was
implemented — at most one >16k-token request per card at a time (`LARGE_TOKENS`
/ `LARGE_LIMIT`), small requests untouched — and run against the same
three-family workload:
| | heavy | medium 1 | medium 2 |
|---|---|---|---|
| no admission | 13 turns, median 25.4 s, 23 % cached | 34.7 s, 14 % | 45.2 s, 15 % |
| one large per card | 5 turns, median 48.2 s, 20 % cached | 22.7 s, 14 % | 22.6 s, 33 % |
No clear win on small samples, so it was reverted too. The reason is now
visible in the arithmetic: with three families on two cards, one card must hold
**two** families, and its pool has to carry 2 × 50k of cached prefix *plus* the
50k of the sequence currently generating — 150k against 122,636. Those two
co-tenants evict each other turn by turn whatever the concurrency limit is. A
card can hold two large families' caches, or one cache plus a live generation,
but not both. That is the wall, and it is capacity, not scheduling.
## Citations
- `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer