diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 3084c16..9d1a50b 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -381,6 +381,34 @@ whether one is. Three concurrent large sessions do not fit in two cards' capacity, and the levers are therefore scheduling (fewer large contexts in flight, which is what the harness's context compaction does) or a third card. +### Caching itself is healthy; the sum is not + +Before blaming the cache, it was measured directly: one card, an unloaded +engine, a ~50k-token prompt sent three times — **22.3 s cold, then 1.4 s, then +1.4 s** — and the same result with a varying tail between turns (22.4 s cold, +1.41 s warm). So long-prefix reuse works exactly as the pinning policy assumes, +and the engine's lifetime "Prefix cache hit rate: 29 %" line is an average over +every benchmark prompt a replica has ever seen, not a statement about any one +request. + +**Serialising large requests does not fix the overload either.** Admission was +implemented — at most one >16k-token request per card at a time (`LARGE_TOKENS` +/ `LARGE_LIMIT`), small requests untouched — and run against the same +three-family workload: + +| | heavy | medium 1 | medium 2 | +|---|---|---|---| +| no admission | 13 turns, median 25.4 s, 23 % cached | 34.7 s, 14 % | 45.2 s, 15 % | +| one large per card | 5 turns, median 48.2 s, 20 % cached | 22.7 s, 14 % | 22.6 s, 33 % | + +No clear win on small samples, so it was reverted too. The reason is now +visible in the arithmetic: with three families on two cards, one card must hold +**two** families, and its pool has to carry 2 × 50k of cached prefix *plus* the +50k of the sequence currently generating — 150k against 122,636. Those two +co-tenants evict each other turn by turn whatever the concurrency limit is. A +card can hold two large families' caches, or one cache plus a live generation, +but not both. That is the wall, and it is capacity, not scheduling. + ## Citations - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer