docs: record the proxy's failover behaviour and how to verify it

This commit is contained in:
Frank committed 2026-09-16 05:29:45 -07:00
1 parent 3946ce9a3a
commit d8189b65f4
1 file changed
+19
@@ -226,6 +226,25 @@ The same call works against either replica directly (`:7771`, `:7772`) — when
one of them is down the other still serves, which is why the proxy reports the one of them is down the other still serves, which is why the proxy reports the
individual replica states instead of a single boolean. individual replica states instead of a single boolean.
A replica that dies mid-flight does not cost a client a request: a failed
connection demotes that replica immediately and the proxy retries on the live
one (vLLM's endpoints are stateless, so re-sending a request that never reached
an engine is safe) and logs the failover. Verify with card 1 stopped:
```
systemctl --user stop frank-medium-a.service
for i in 1 2 3 4 5 6; do
curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \
-d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \
http://127.0.0.1:7770/v1/chat/completions
done; echo
podman logs --since 60s frank-medium-proxy | grep -i failover
systemctl --user start frank-medium-a.service
```
Six 200s and one `proxy: failover — … unreachable` line is the pass condition.
Restarting a stopped replica takes about 150 s (warm compile cache).
## Benchmarks ## Benchmarks
`vllm bench serve` against the proxy (both cards serving), random dataset, `vllm bench serve` against the proxy (both cards serving), random dataset,