frank-medium: the agent-loop shape on one slot, and a harness header that matches the runtime
Two families of ~30k-token turns ran against the deployed endpoint: 24.8 s and
51.4 s for the first turn (the second waits for the slot), 30.7 s and 8.0 s for
the second. Against the prefill table that is ~25 s of cold prefill plus the
wait, so the honest reading is that ONE slot is now the limiting resource for
concurrent agent loops — the trade this deployment makes for giving a single
conversation the whole 262144-token window. --parallel splits the window; it
does not add throughput.
The harness header still described the retired two-replica proxy ('the deployed
routing pins such a conversation to one replica', vLLM's prefix-cache key). It
now says what is true: one endpoint, one slot, concurrent families queue, and
the old routing numbers are history in the runbook.
This commit is contained in:
@@ -201,6 +201,25 @@ vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
|
||||
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
|
||||
no proxy.
|
||||
|
||||
### The agent-loop shape (fleet harness, 2 families × 2 turns)
|
||||
|
||||
`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
|
||||
agent loops re-sending a ~30k-token document concurrently:
|
||||
|
||||
| | turn 1 | turn 2 |
|
||||
|---|---:|---:|
|
||||
| family 1 | 24,799 ms | 30,697 ms |
|
||||
| family 0 | 51,424 ms | 8,015 ms |
|
||||
|
||||
Read that against the prefill table, not as a latency regression: one slot
|
||||
means the second family's first turn waits for the first (25 s of cold 30k
|
||||
prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that
|
||||
the cache could only partly serve. **The single slot is now the limiting
|
||||
resource for concurrent agent loops**, which is the trade this deployment makes
|
||||
for giving one conversation the whole 262144-token window instead of splitting
|
||||
two. Raising `--parallel` splits the window (and the KV) between slots; it does
|
||||
not add throughput.
|
||||
|
||||
## Observability
|
||||
|
||||
The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
|
||||
|
||||
Reference in New Issue
Block a user