dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-medium: the agent-loop shape on one slot, and a harness header that matches the runtime

Browse Source 
Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.

The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.
This commit is contained in:
Frank
2026-09-16 15:38:20 -07:00
parent dc76dc5dc4
commit 87492c7b10
2 changed files with 33 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/bench-turn-latency.py
+14 -16
View File
@@ -3,29 +3,27 @@
Every other harness in this directory measures a *batch* of independent
requests. That answers throughput, and it is the wrong instrument for the number
a user feels: an agent loop re-sends the same large prefix turn after turn, and
the deployed routing pins such a conversation to one replica so its prefix cache
stays warm. This script drives that shape and prints TTFT per turn.
It is also the measurement that produced the routing decision on 2026-09-16
(round-robin 2676 ms mean warm turn against pinned 1096 ms on 90k-token
contexts), and the one that shows a cold-vs-warm prefill directly: pump the same
prompt twice and the second is a cache lookup.
a user feels: an agent loop re-sends the same large prefix turn after turn, so
what matters is whether that prefix is served from the cache or re-prefilled.
This script drives that shape and prints TTFT per turn.
Usage:
bench-turn-latency.py <base-url> <minutes|turns> [families] [salt]
bench-turn-latency.py <base-url> <turns> [families] [salt]
bench-turn-latency.py http://127.0.0.1:7770 5 3 SALT1 # 3 families, 5 turns each
Prompts carry the salt as their FIRST bytes: that changes both the router's
identity window and vLLM's prefix-cache key, which is what keeps a run from
silently inheriting a warm cache from the previous one.
Prompts carry the salt as their FIRST bytes: that changes the prompt itself and
therefore llama.cpp's prefix-cache key, which is what keeps a run from silently
inheriting a warm cache from the previous one.
Two cards behind a proxy: send against the proxy (:7770) to measure the fleet,
or a replica directly (:7771, :7772) to measure one card.
One endpoint: send against it directly (:7770). The routing measurement this
harness originally produced (round-robin 2676 ms vs pinned 1096 ms warm turns on
90k-token contexts, 2026-09-16) belonged to the retired two-replica proxy and is
kept in the runbook as history. The current deployment is ONE process with ONE
slot, so concurrent families queue rather than spread — the harness reports that
shape honestly, and a saturated slot shows up as first-turn TTFT that includes
the wait.
"""
from __future__ import annotations
import json
import statistics
import sys