diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 5eee249..0217ad7 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -299,6 +299,40 @@ to `SPEC=5` if batch throughput ever matters more than turn latency — it is on environment line in the unit, at the cost of a recompile (the spec depth is part of the graph cache key). +### Capacity: how many big conversations fit + +Each replica's own ledger at boot states the ceiling: + +``` +GPU KV cache size: 122,636 tokens +Maximum concurrency for 114,688 tokens per request: 1.07x +[radiance] kv cache groups: size 8, 9 groups, 202 blocks/request (upstream would pick size 5) +Add 3 padding layers, may waste at most 60.00% KV cache memory +``` + +So one card holds ~122k tokens of KV — one full-context conversation plus a +little, or about two 30k-token agent loops with their cached prefixes. A 3-family +soak (three concurrent loops, each re-sending a ~30k-token document every turn +with growing history) shows what happens at that limit: turns alternate between +**~1.1 s** when the prefix is resident and **12-22 s** when it has been evicted +and must be prefilled again, because two families per card plus their cached +prefixes exceed the pool. Nothing failed — ~120 requests, no errors, no failover, +both replicas steady — and per-family pinning kept the fast turns on the replica +that holds the prefix. + +Read that as a capacity budget, not a defect: two concurrent large-context +sessions per card are comfortable, three or more of that size trade turns +against each other. The soak ran 155 requests, 155 answers, no errors, no +failover and both replicas steady — the cost shows up as latency, not failure. + +**`CHUNK` is not a lever for this.** The arena/KV budget is not monotonic in +the chunk size, and 8192 is a sweet spot rather than a compromise — measured +available KV per card: **5.44 GiB at `CHUNK=8192`** (122,636 tokens), 4.22 GiB +at 16384, and 4.16 GiB at 4096, where the engine refuses to start at all +(`5.0 GiB KV cache is needed, which is larger than the available KV cache +memory (4.16 GiB)`). Both directions from the deployed value cost capacity, so +the real lever for more concurrent large contexts is another card. + ## Citations - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer