frank-medium: route by workload — spread batches, pin agent loops

The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.

Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.

So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.

The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.

ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.

dankserver/frank-medium/quadlet/frank-medium-proxy.container

@@ -3,7 +3,7 @@
 # Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
 # Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
 #
-# WHAT IT IS. A round-robin reverse proxy in front of the two single-GPU
+# WHAT IT IS. A prefix-affinity reverse proxy in front of the two single-GPU
 # frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
 # the fleet contract: one endpoint on host port 7770 answering as the served
 # model name frank-medium, OpenAI /v1, streaming included.
@@ -13,17 +13,22 @@
 # need PCIe atomics, and the second card's chipset path advertises
 # AtomicOpsCap: Routing-). Two replicas are how both cards get used.
 #
-# ROUTING. vLLM's OpenAI API is stateless per request, so round-robin is
-# sound; there is no session affinity to preserve. /health answers 200 while
-# at least one replica is up and names the ones that are down. The proxy runs
-# on host networking so it can bind 7770 and reach the replicas on loopback.
+# ROUTING. Requests are keyed on a hash of their leading 4 KiB of prompt text,
+# so a conversation family — an agent loop re-sending the same system prompt,
+# tool list and file context every turn — keeps landing on the same replica,
+# where --enable-prefix-caching has that prefix warm. Plain round-robin threw
+# half of those cache hits away. A replica that is down is skipped, and
+# ROUTING=roundrobin restores the old behaviour for A/B measurement. /health
+# answers 200 while at least one replica is up and names the ones that are
+# down. The proxy runs on host networking so it can bind 7770 and reach the
+# replicas on loopback.
 #
 # The script is a few dozen lines of aiohttp, mounted read-only out of
 # /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
 # It reuses the same image as the replicas: aiohttp is already there and no
 # extra image has to be pulled on this host.
 [Unit]
-Description=frank-medium proxy — round-robin in front of the two single-GPU replicas
+Description=frank-medium proxy — prefix-affinity in front of the two single-GPU replicas
 After=network-online.target
 Wants=frank-medium-a.service frank-medium-b.service
 After=frank-medium-a.service frank-medium-b.service
@@ -34,9 +39,13 @@ Image=localhost/radiance-vllm:0.9.3
 PodmanArgs=--network=host
 Environment=PORT=7770
 Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
-Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/roundrobin-proxy.py:/usr/local/bin/roundrobin-proxy.py:ro
+# The proxy is two files: routing.py holds the replica-selection policy, the
+# server holds the HTTP plumbing. Both are mounted because the server imports
+# the policy by name from its own directory (/usr/local/bin is sys.path[0]).
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/routing.py:/usr/local/bin/routing.py:ro
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/affinity-proxy.py:/usr/local/bin/affinity-proxy.py:ro
 Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
-PodmanArgs=--entrypoint /usr/local/bin/roundrobin-proxy.py
+PodmanArgs=--entrypoint /usr/local/bin/affinity-proxy.py
 HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
 HealthInterval=60s
 HealthTimeout=30s