docs: record the proxy's failover behaviour and how to verify it
This commit is contained in:
@@ -226,6 +226,25 @@ The same call works against either replica directly (`:7771`, `:7772`) — when
|
||||
one of them is down the other still serves, which is why the proxy reports the
|
||||
individual replica states instead of a single boolean.
|
||||
|
||||
A replica that dies mid-flight does not cost a client a request: a failed
|
||||
connection demotes that replica immediately and the proxy retries on the live
|
||||
one (vLLM's endpoints are stateless, so re-sending a request that never reached
|
||||
an engine is safe) and logs the failover. Verify with card 1 stopped:
|
||||
|
||||
```
|
||||
systemctl --user stop frank-medium-a.service
|
||||
for i in 1 2 3 4 5 6; do
|
||||
curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \
|
||||
-d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \
|
||||
http://127.0.0.1:7770/v1/chat/completions
|
||||
done; echo
|
||||
podman logs --since 60s frank-medium-proxy | grep -i failover
|
||||
systemctl --user start frank-medium-a.service
|
||||
```
|
||||
|
||||
Six 200s and one `proxy: failover — … unreachable` line is the pass condition.
|
||||
Restarting a stopped replica takes about 150 s (warm compile cache).
|
||||
|
||||
## Benchmarks
|
||||
|
||||
`vllm bench serve` against the proxy (both cards serving), random dataset,
|
||||
|
||||
Reference in New Issue
Block a user