diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 9b6201b..3084c16 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -354,6 +354,33 @@ at 16384, and 4.16 GiB at 4096, where the engine refuses to start at all memory (4.16 GiB)`). Both directions from the deployed value cost capacity, so the real lever for more concurrent large contexts is another card. +### Placement does not rescue an overloaded working set + +If families compete for cache space, the obvious idea is to place a new +conversation on the card carrying less context instead of alternating. That was +implemented (decayed prompt-token estimate per replica, new families to the +lighter card, continuations still pinned) and measured against the counter on +the workload it should help: one ~80k-token family plus two ~50k-token +families, each turn re-sending the same document. + +| placement | heavy | medium 1 | medium 2 | +|---|---|---|---| +| counter (13 min) | 13 turns, median 25.4 s, 3 cached (23 %) | 14 turns, median 34.7 s, 2 cached (14 %) | 13 turns, median 45.2 s, 2 cached (15 %) | +| balanced by load (~8 min) | 5 turns, median 46.7 s, 1 cached (20 %) | 7 turns, median 23.4 s, 0 cached | 6 turns, median 34.3 s, 0 cached | + +No improvement — the load arm shows the same near-total loss of cache reuse +(note its smaller sample: the arm was stopped early once the pattern was +clear), so the change was reverted. `routing.py` keeps the counter and the +policy surface stays as small as it was. + +The reason is arithmetic rather than policy: a running request's *live* +sequence and its cached prefix draw on the same pool, so a card that is running +two 50k-token generations has already exceeded its 122,636 tokens before any +cache entry is considered. Placement can decide *which* card is over, not +whether one is. Three concurrent large sessions do not fit in two cards' +capacity, and the levers are therefore scheduling (fewer large contexts in +flight, which is what the harness's context compaction does) or a third card. + ## Citations - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer