frank-medium: the agent-loop shape on one slot, and a harness header that matches the runtime

Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.

The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.
This commit is contained in:
Frank
2026-09-16 15:38:20 -07:00
parent dc76dc5dc4
commit 87492c7b10
2 changed files with 33 additions and 16 deletions
@@ -201,6 +201,25 @@ vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
no proxy.
### The agent-loop shape (fleet harness, 2 families × 2 turns)
`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
agent loops re-sending a ~30k-token document concurrently:
| | turn 1 | turn 2 |
|---|---:|---:|
| family 1 | 24,799 ms | 30,697 ms |
| family 0 | 51,424 ms | 8,015 ms |
Read that against the prefill table, not as a latency regression: one slot
means the second family's first turn waits for the first (25 s of cold 30k
prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that
the cache could only partly serve. **The single slot is now the limiting
resource for concurrent agent loops**, which is the trade this deployment makes
for giving one conversation the whole 262144-token window instead of splitting
two. Raising `--parallel` splits the window (and the KV) between slots; it does
not add throughput.
## Observability
The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace