frank-medium: the agent-loop shape on one slot, and a harness header that matches the runtime

Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.

The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.

scripts/bench-turn-latency.py

@@ -3,29 +3,27 @@
 
 Every other harness in this directory measures a *batch* of independent
 requests. That answers throughput, and it is the wrong instrument for the number
-a user feels: an agent loop re-sends the same large prefix turn after turn, and
+a user feels: an agent loop re-sends the same large prefix turn after turn, so
-the deployed routing pins such a conversation to one replica so its prefix cache
+what matters is whether that prefix is served from the cache or re-prefilled.
-stays warm. This script drives that shape and prints TTFT per turn.
+This script drives that shape and prints TTFT per turn.
-
-It is also the measurement that produced the routing decision on 2026-09-16
-(round-robin 2676 ms mean warm turn against pinned 1096 ms on 90k-token
-contexts), and the one that shows a cold-vs-warm prefill directly: pump the same
-prompt twice and the second is a cache lookup.
 
 Usage:
-    bench-turn-latency.py <base-url> <minutes|turns> [families] [salt]
+    bench-turn-latency.py <base-url> <turns> [families] [salt]
     bench-turn-latency.py http://127.0.0.1:7770 5 3 SALT1     # 3 families, 5 turns each
 
-Prompts carry the salt as their FIRST bytes: that changes both the router's
+Prompts carry the salt as their FIRST bytes: that changes the prompt itself and
-identity window and vLLM's prefix-cache key, which is what keeps a run from
+therefore llama.cpp's prefix-cache key, which is what keeps a run from silently
-silently inheriting a warm cache from the previous one.
+inheriting a warm cache from the previous one.
 
-Two cards behind a proxy: send against the proxy (:7770) to measure the fleet,
+One endpoint: send against it directly (:7770). The routing measurement this
-or a replica directly (:7771, :7772) to measure one card.
+harness originally produced (round-robin 2676 ms vs pinned 1096 ms warm turns on
+90k-token contexts, 2026-09-16) belonged to the retired two-replica proxy and is
+kept in the runbook as history. The current deployment is ONE process with ONE
+slot, so concurrent families queue rather than spread — the harness reports that
+shape honestly, and a saturated slot shows up as first-turn TTFT that includes
+the wait.
 """
 
-from __future__ import annotations
-
 import json
 import statistics
 import sys

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -201,6 +201,25 @@ vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
 llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
 no proxy.
 
+### The agent-loop shape (fleet harness, 2 families × 2 turns)
+
+`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
+agent loops re-sending a ~30k-token document concurrently:
+
+| | turn 1 | turn 2 |
+|---|---:|---:|
+| family 1 | 24,799 ms | 30,697 ms |
+| family 0 | 51,424 ms | 8,015 ms |
+
+Read that against the prefill table, not as a latency regression: one slot
+means the second family's first turn waits for the first (25 s of cold 30k
+prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that
+the cache could only partly serve. **The single slot is now the limiting
+resource for concurrent agent loops**, which is the trade this deployment makes
+for giving one conversation the whole 262144-token window instead of splitting
+two. Raising `--parallel` splits the window (and the KV) between slots; it does
+not add throughput.
+
 ## Observability
 
 The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace