From 24687f561f1980840d35d9d934d75f87f72078f7 Mon Sep 17 00:00:00 2001
From: Frank <frank@dankware>
Date: Wed, 16 Sep 2026 05:49:54 -0700
Subject: [PATCH] docs: frank-medium capacity budget and the CHUNK sweet spot
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

A 25-minute soak with three concurrent large-context agent loops (each turn
re-sending a ~30k-token document, growing history) came back clean — 155
requests, 155 answers, no errors, no failover, both replicas steady — but only
25% of turns hit the prefix cache, the rest paying a 13.9 s median re-prefill.
That is the capacity budget showing: 122,636 KV tokens per card is about two
30k-token conversations with their cached prefixes, and a third family on the
same card evicts.

Chasing the obvious lever disproved it. Available KV per card is 5.44 GiB at
CHUNK=8192, 4.22 GiB at 16384 and 4.16 GiB at 4096 — where the engine refuses
to start. The budget is not monotonic in the chunk size, so 8192 is a sweet
spot rather than a compromise and freeing KV needs another card, not a knob.
---
 .../LLM/frank-medium-radiance-mxfp4.md        | 34 +++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
index 5eee249..0217ad7 100644
--- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
@@ -299,6 +299,40 @@ to `SPEC=5` if batch throughput ever matters more than turn latency — it is on
 environment line in the unit, at the cost of a recompile (the spec depth is
 part of the graph cache key).
 
+### Capacity: how many big conversations fit
+
+Each replica's own ledger at boot states the ceiling:
+
+```
+GPU KV cache size: 122,636 tokens
+Maximum concurrency for 114,688 tokens per request: 1.07x
+[radiance] kv cache groups: size 8, 9 groups, 202 blocks/request (upstream would pick size 5)
+Add 3 padding layers, may waste at most 60.00% KV cache memory
+```
+
+So one card holds ~122k tokens of KV — one full-context conversation plus a
+little, or about two 30k-token agent loops with their cached prefixes. A 3-family
+soak (three concurrent loops, each re-sending a ~30k-token document every turn
+with growing history) shows what happens at that limit: turns alternate between
+**~1.1 s** when the prefix is resident and **12-22 s** when it has been evicted
+and must be prefilled again, because two families per card plus their cached
+prefixes exceed the pool. Nothing failed — ~120 requests, no errors, no failover,
+both replicas steady — and per-family pinning kept the fast turns on the replica
+that holds the prefix.
+
+Read that as a capacity budget, not a defect: two concurrent large-context
+sessions per card are comfortable, three or more of that size trade turns
+against each other. The soak ran 155 requests, 155 answers, no errors, no
+failover and both replicas steady — the cost shows up as latency, not failure.
+
+**`CHUNK` is not a lever for this.** The arena/KV budget is not monotonic in
+the chunk size, and 8192 is a sweet spot rather than a compromise — measured
+available KV per card: **5.44 GiB at `CHUNK=8192`** (122,636 tokens), 4.22 GiB
+at 16384, and 4.16 GiB at 4096, where the engine refuses to start at all
+(`5.0 GiB KV cache is needed, which is larger than the available KV cache
+memory (4.16 GiB)`). Both directions from the deployed value cost capacity, so
+the real lever for more concurrent large contexts is another card.
+
 ## Citations
 
 - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer