frank-medium: route by workload — spread batches, pin agent loops

The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.

Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.

So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.

The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.

ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.

serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md

@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: frank-medium vllm-radiance MXFP4 Runbook
-description: vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201), as two single-GPU replicas behind a round-robin proxy — and the PCIe-atomics reason TP=2 cannot run on this host.
+description: vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201), as two single-GPU replicas behind an adaptive-routing proxy — and the PCIe-atomics reason TP=2 cannot run on this host.
 tags:
   - frank-medium
   - dankserver
@@ -24,7 +24,7 @@ patches, serving `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` on the two
 Radeon AI PRO R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX
 llama.cpp deployment ([[frank-medium-rocmfpx]]).
 
-**Deployed shape: two single-GPU replicas behind a round-robin proxy.** The
+**Deployed shape: two single-GPU replicas behind an adaptive-routing proxy.** The
 intended shape — one tensor-parallel process across both cards (TP=2) — cannot
 launch a collective on this host. The reason is a hardware property of the
 motherboard's PCIe topology, not the image, the RCCL build, or the kernel
@@ -34,7 +34,7 @@ version. The evidence is in [Why TP=2 cannot run here](#why-tp2-cannot-run-here)
 
 ```mermaid
 graph LR
-    C[clients: Hermes, harness] --> P["frank-medium-proxy<br/>:7770 round-robin"]
+    C[clients: Hermes, harness] --> P["frank-medium-proxy<br/>:7770 adaptive routing"]
     P --> A["frank-medium-a<br/>R9700 #1, TP=1, :7771"]
     P --> B["frank-medium-b<br/>R9700 #2, TP=1, :7772"]
 ```
@@ -45,11 +45,12 @@ Three rootless Quadlet units, all `WantedBy=default.target` + `Restart=on-failur
 |---|---|---|---|---|
 | `frank-medium-a` | `frank-medium-a` | `HIP_VISIBLE_DEVICES=0` | 7771 | replica |
 | `frank-medium-b` | `frank-medium-b` | `HIP_VISIBLE_DEVICES=1` | 7772 | replica |
-| `frank-medium-proxy` | `frank-medium-proxy` | — | 7770 | round-robin |
+| `frank-medium-proxy` | `frank-medium-proxy` | — | 7770 | routing |
 
 The fleet contract is unchanged: one endpoint on :7770, served model name
 `frank-medium`, OpenAI `/v1`, streaming included. The vLLM OpenAI API is
-stateless per request, so round-robin needs no session affinity.
+stateless per request, so no session affinity is needed for correctness. Routing
+is described in [Routing](#routing).
 
 Only `HIP_VISIBLE_DEVICES` is set per replica, never a matching
 `ROCR_VISIBLE_DEVICES`: ROCR filters first and HIP then indexes into the
@@ -140,6 +141,39 @@ boot with the write already in place) would change it — and even then the two
   platform with more CPU lanes (Threadripper/EPYC) is the structural answer.
 - Until then, **two replicas** are how both cards serve.
 
+## Routing
+
+The proxy decides per request, from `proxy/routing.py` (policy) and
+`proxy/affinity-proxy.py` (HTTP plumbing):
+
+- **A continuation** — a request carrying more than one message, i.e. an agent
+  loop re-sending its context — is pinned to the replica that served the
+  prefix before, because `--enable-prefix-caching` has that prefix warm there.
+- **Anything else** — a one-shot question, a batch member, a burst of parallel
+  subagents — spreads across the cards by counter, so both prefill in parallel.
+
+Measured on this host, 90k-token contexts and 15k-token prompts, `vllm bench`
+plus a small sequential-turn client:
+
+| strategy | cold 16-request burst | warm-turn TTFT (4 turns × 2 families) |
+|---|--:|--:|
+| `roundrobin` (counter) | 18.6 s | 2676 ms — every other turn pays a cold prefill |
+| `affinity` (hash of prefix) | 30.1 s — one card does all the prefilling | 1096 ms |
+| `adaptive` (deployed) | 19.97 s | 1095 ms |
+
+Two failures found by measuring rather than reasoning, both now covered by
+`tests/frank_medium_proxy.bats`:
+
+- Hashing a fresh prefix to pick a card sends *every* member of a same-prefix
+  burst to one card (29.8 s for the burst above). Only continuations may be
+  pinned; the spread has to come from a counter.
+- The identity of a conversation is its **first 4 KiB** (`AFFINITY_BYTES`).
+  A prefix shorter than that window is indistinguishable from its own tail, so
+  turns of the same conversation hash differently and never pin.
+
+`ROUTING=roundrobin` and `ROUTING=affinity` remain available to re-measure the
+choice; the default is `adaptive`.
+
 ## Deployed configuration
 
 Both replicas, uniform except GPU index and port: