dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: measure load-balanced placement, find it does not help, revert

Browse Source 
The obvious next lever for cache thrash was placement: send a new conversation
to the card carrying less context instead of alternating. Implemented it
(decayed prompt-token estimate per replica, continuations still pinned) and
A/B'd it against the counter on the workload it targets — one ~80k-token
family plus two ~50k-token families, each turn re-sending the same document:

  counter          heavy 13 turns median 25.4 s (23% cached)
                   medium 34.7 s / 45.2 s (14% / 15% cached)
  balanced by load heavy  5 turns median 46.7 s (20% cached)
                   medium 23.4 s / 34.3 s (0% cached, small sample)

No improvement, so the change is reverted and routing.py keeps the counter.
The reason is arithmetic: a running request's live sequence and its cached
prefix draw on the same pool, so a card running two 50k-token generations has
already spent its 122,636 tokens before any cache entry is considered.
Placement chooses which card is over capacity, not whether one is.

Third hypothesis measured and rejected tonight (with SPEC depth kept and CHUNK
disproved); the runbook records all three so the next operator starts from
evidence rather than from these ideas looking obvious.
This commit is contained in:
Frank
2026-09-16 06:20:19 -07:00
parent 99ed1aca9a
commit 36007666cf
1 changed files with 27 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+27
View File
@@ -354,6 +354,33 @@ at 16384, and 4.16 GiB at 4096, where the engine refuses to start at all
memory (4.16 GiB)`). Both directions from the deployed value cost capacity, so
the real lever for more concurrent large contexts is another card.
### Placement does not rescue an overloaded working set
If families compete for cache space, the obvious idea is to place a new
conversation on the card carrying less context instead of alternating. That was
implemented (decayed prompt-token estimate per replica, new families to the
lighter card, continuations still pinned) and measured against the counter on
the workload it should help: one ~80k-token family plus two ~50k-token
families, each turn re-sending the same document.
| placement | heavy | medium 1 | medium 2 |
|---|---|---|---|
| counter (13 min) | 13 turns, median 25.4 s, 3 cached (23 %) | 14 turns, median 34.7 s, 2 cached (14 %) | 13 turns, median 45.2 s, 2 cached (15 %) |
| balanced by load (~8 min) | 5 turns, median 46.7 s, 1 cached (20 %) | 7 turns, median 23.4 s, 0 cached | 6 turns, median 34.3 s, 0 cached |
No improvement — the load arm shows the same near-total loss of cache reuse
(note its smaller sample: the arm was stopped early once the pattern was
clear), so the change was reverted. `routing.py` keeps the counter and the
policy surface stays as small as it was.
The reason is arithmetic rather than policy: a running request's *live*
sequence and its cached prefix draw on the same pool, so a card that is running
two 50k-token generations has already exceeded its 122,636 tokens before any
cache entry is considered. Placement can decide *which* card is over, not
whether one is. Three concurrent large sessions do not fit in two cards'
capacity, and the levers are therefore scheduling (fewer large contexts in
flight, which is what the harness's context compaction does) or a third card.
## Citations
- `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer