The obvious next lever for cache thrash was placement: send a new conversation
to the card carrying less context instead of alternating. Implemented it
(decayed prompt-token estimate per replica, continuations still pinned) and
A/B'd it against the counter on the workload it targets — one ~80k-token
family plus two ~50k-token families, each turn re-sending the same document:
counter heavy 13 turns median 25.4 s (23% cached)
medium 34.7 s / 45.2 s (14% / 15% cached)
balanced by load heavy 5 turns median 46.7 s (20% cached)
medium 23.4 s / 34.3 s (0% cached, small sample)
No improvement, so the change is reverted and routing.py keeps the counter.
The reason is arithmetic: a running request's live sequence and its cached
prefix draw on the same pool, so a card running two 50k-token generations has
already spent its 122,636 tokens before any cache entry is considered.
Placement chooses which card is over capacity, not whether one is.
Third hypothesis measured and rejected tonight (with SPEC depth kept and CHUNK
disproved); the runbook records all three so the next operator starts from
evidence rather than from these ideas looking obvious.
Frank
36007666cf