diff --git a/scripts/bench-turn-latency.py b/scripts/bench-turn-latency.py index 6df45a6..cac9a4f 100755 --- a/scripts/bench-turn-latency.py +++ b/scripts/bench-turn-latency.py @@ -3,29 +3,27 @@ Every other harness in this directory measures a *batch* of independent requests. That answers throughput, and it is the wrong instrument for the number -a user feels: an agent loop re-sends the same large prefix turn after turn, and -the deployed routing pins such a conversation to one replica so its prefix cache -stays warm. This script drives that shape and prints TTFT per turn. - -It is also the measurement that produced the routing decision on 2026-09-16 -(round-robin 2676 ms mean warm turn against pinned 1096 ms on 90k-token -contexts), and the one that shows a cold-vs-warm prefill directly: pump the same -prompt twice and the second is a cache lookup. +a user feels: an agent loop re-sends the same large prefix turn after turn, so +what matters is whether that prefix is served from the cache or re-prefilled. +This script drives that shape and prints TTFT per turn. Usage: - bench-turn-latency.py [families] [salt] + bench-turn-latency.py [families] [salt] bench-turn-latency.py http://127.0.0.1:7770 5 3 SALT1 # 3 families, 5 turns each -Prompts carry the salt as their FIRST bytes: that changes both the router's -identity window and vLLM's prefix-cache key, which is what keeps a run from -silently inheriting a warm cache from the previous one. +Prompts carry the salt as their FIRST bytes: that changes the prompt itself and +therefore llama.cpp's prefix-cache key, which is what keeps a run from silently +inheriting a warm cache from the previous one. -Two cards behind a proxy: send against the proxy (:7770) to measure the fleet, -or a replica directly (:7771, :7772) to measure one card. +One endpoint: send against it directly (:7770). The routing measurement this +harness originally produced (round-robin 2676 ms vs pinned 1096 ms warm turns on +90k-token contexts, 2026-09-16) belonged to the retired two-replica proxy and is +kept in the runbook as history. The current deployment is ONE process with ONE +slot, so concurrent families queue rather than spread — the harness reports that +shape honestly, and a saturated slot shows up as first-turn TTFT that includes +the wait. """ -from __future__ import annotations - import json import statistics import sys diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 2313edc..314881c 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -201,6 +201,25 @@ vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards, no proxy. +### The agent-loop shape (fleet harness, 2 families × 2 turns) + +`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two +agent loops re-sending a ~30k-token document concurrently: + +| | turn 1 | turn 2 | +|---|---:|---:| +| family 1 | 24,799 ms | 30,697 ms | +| family 0 | 51,424 ms | 8,015 ms | + +Read that against the prefill table, not as a latency regression: one slot +means the second family's first turn waits for the first (25 s of cold 30k +prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that +the cache could only partly serve. **The single slot is now the limiting +resource for concurrent agent loops**, which is the trade this deployment makes +for giving one conversation the whole 262144-token window instead of splitting +two. Raising `--parallel` splits the window (and the KV) between slots; it does +not add throughput. + ## Observability The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace