From 4f2f49d2d0697eb63f5062ef32e1fda2c131654f Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Wed, 16 Sep 2026 06:31:01 -0700
Subject: [PATCH] frank-medium: measure admission control and cache health,
 keep neither change
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Two more hypotheses, both measured on the same three-family workload (one
~80k-token family plus two ~50k-token families, each turn re-sending its
document):

1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt
   sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too
   (22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%"
   is an average across every prompt that replica ever saw, not a verdict on
   any one request, and the pinning policy is safe.

2. Serialising large requests does not fix the overload. Admission control
   (at most one >16k-token request per card, small requests untouched) gave no
   clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums
   34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was
   reverted like the placement change before it.

The arithmetic now closes: with three families on two cards, one card holds two
families and must carry 2 x 50k of cached prefix plus the 50k of the sequence
generating — 150k against 122,636 tokens. Those co-tenants evict each other
regardless of concurrency limits. The wall is capacity, and the runbook says so
with the numbers.
---
 .../LLM/frank-medium-radiance-mxfp4.md        | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
index 3084c16..9d1a50b 100644
--- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
@@ -381,6 +381,34 @@ whether one is. Three concurrent large sessions do not fit in two cards'
 capacity, and the levers are therefore scheduling (fewer large contexts in
 flight, which is what the harness's context compaction does) or a third card.
 
+### Caching itself is healthy; the sum is not
+
+Before blaming the cache, it was measured directly: one card, an unloaded
+engine, a ~50k-token prompt sent three times — **22.3 s cold, then 1.4 s, then
+1.4 s** — and the same result with a varying tail between turns (22.4 s cold,
+1.41 s warm). So long-prefix reuse works exactly as the pinning policy assumes,
+and the engine's lifetime "Prefix cache hit rate: 29 %" line is an average over
+every benchmark prompt a replica has ever seen, not a statement about any one
+request.
+
+**Serialising large requests does not fix the overload either.** Admission was
+implemented — at most one >16k-token request per card at a time (`LARGE_TOKENS`
+/ `LARGE_LIMIT`), small requests untouched — and run against the same
+three-family workload:
+
+| | heavy | medium 1 | medium 2 |
+|---|---|---|---|
+| no admission | 13 turns, median 25.4 s, 23 % cached | 34.7 s, 14 % | 45.2 s, 15 % |
+| one large per card | 5 turns, median 48.2 s, 20 % cached | 22.7 s, 14 % | 22.6 s, 33 % |
+
+No clear win on small samples, so it was reverted too. The reason is now
+visible in the arithmetic: with three families on two cards, one card must hold
+**two** families, and its pool has to carry 2 × 50k of cached prefix *plus* the
+50k of the sequence currently generating — 150k against 122,636. Those two
+co-tenants evict each other turn by turn whatever the concurrency limit is. A
+card can hold two large families' caches, or one cache plus a live generation,
+but not both. That is the wall, and it is capacity, not scheduling.
+
 ## Citations
 
 - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer