docs: record the proxy's failover behaviour and how to verify it

This commit is contained in:
Frank
2026-09-16 05:29:45 -07:00
parent 3946ce9a3a
commit d8189b65f4
@@ -226,6 +226,25 @@ The same call works against either replica directly (`:7771`, `:7772`) — when
one of them is down the other still serves, which is why the proxy reports the
individual replica states instead of a single boolean.
A replica that dies mid-flight does not cost a client a request: a failed
connection demotes that replica immediately and the proxy retries on the live
one (vLLM's endpoints are stateless, so re-sending a request that never reached
an engine is safe) and logs the failover. Verify with card 1 stopped:
```
systemctl --user stop frank-medium-a.service
for i in 1 2 3 4 5 6; do
curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \
-d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \
http://127.0.0.1:7770/v1/chat/completions
done; echo
podman logs --since 60s frank-medium-proxy | grep -i failover
systemctl --user start frank-medium-a.service
```
Six 200s and one `proxy: failover — … unreachable` line is the pass condition.
Restarting a stopped replica takes about 150 s (warm compile cache).
## Benchmarks
`vllm bench serve` against the proxy (both cards serving), random dataset,