frank-medium: measure admission control and cache health, keep neither change
Two more hypotheses, both measured on the same three-family workload (one ~80k-token family plus two ~50k-token families, each turn re-sending its document): 1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too (22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%" is an average across every prompt that replica ever saw, not a verdict on any one request, and the pinning policy is safe. 2. Serialising large requests does not fix the overload. Admission control (at most one >16k-token request per card, small requests untouched) gave no clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums 34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was reverted like the placement change before it. The arithmetic now closes: with three families on two cards, one card holds two families and must carry 2 x 50k of cached prefix plus the 50k of the sequence generating — 150k against 122,636 tokens. Those co-tenants evict each other regardless of concurrency limits. The wall is capacity, and the runbook says so with the numbers.
This commit is contained in:
@@ -381,6 +381,34 @@ whether one is. Three concurrent large sessions do not fit in two cards'
|
||||
capacity, and the levers are therefore scheduling (fewer large contexts in
|
||||
flight, which is what the harness's context compaction does) or a third card.
|
||||
|
||||
### Caching itself is healthy; the sum is not
|
||||
|
||||
Before blaming the cache, it was measured directly: one card, an unloaded
|
||||
engine, a ~50k-token prompt sent three times — **22.3 s cold, then 1.4 s, then
|
||||
1.4 s** — and the same result with a varying tail between turns (22.4 s cold,
|
||||
1.41 s warm). So long-prefix reuse works exactly as the pinning policy assumes,
|
||||
and the engine's lifetime "Prefix cache hit rate: 29 %" line is an average over
|
||||
every benchmark prompt a replica has ever seen, not a statement about any one
|
||||
request.
|
||||
|
||||
**Serialising large requests does not fix the overload either.** Admission was
|
||||
implemented — at most one >16k-token request per card at a time (`LARGE_TOKENS`
|
||||
/ `LARGE_LIMIT`), small requests untouched — and run against the same
|
||||
three-family workload:
|
||||
|
||||
| | heavy | medium 1 | medium 2 |
|
||||
|---|---|---|---|
|
||||
| no admission | 13 turns, median 25.4 s, 23 % cached | 34.7 s, 14 % | 45.2 s, 15 % |
|
||||
| one large per card | 5 turns, median 48.2 s, 20 % cached | 22.7 s, 14 % | 22.6 s, 33 % |
|
||||
|
||||
No clear win on small samples, so it was reverted too. The reason is now
|
||||
visible in the arithmetic: with three families on two cards, one card must hold
|
||||
**two** families, and its pool has to carry 2 × 50k of cached prefix *plus* the
|
||||
50k of the sequence currently generating — 150k against 122,636. Those two
|
||||
co-tenants evict each other turn by turn whatever the concurrency limit is. A
|
||||
card can hold two large families' caches, or one cache plus a live generation,
|
||||
but not both. That is the wall, and it is capacity, not scheduling.
|
||||
|
||||
## Citations
|
||||
|
||||
- `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer
|
||||
|
||||
Reference in New Issue
Block a user