docs: record the proxy's failover behaviour and how to verify it
This commit is contained in:
1 parent
3946ce9a3a
commit
d8189b65f4
1 file changed
+19
@@ -226,6 +226,25 @@ The same call works against either replica directly (`:7771`, `:7772`) — when
|
|||||||
one of them is down the other still serves, which is why the proxy reports the
|
one of them is down the other still serves, which is why the proxy reports the
|
||||||
individual replica states instead of a single boolean.
|
individual replica states instead of a single boolean.
|
||||||
|
|
||||||
|
A replica that dies mid-flight does not cost a client a request: a failed
|
||||||
|
connection demotes that replica immediately and the proxy retries on the live
|
||||||
|
one (vLLM's endpoints are stateless, so re-sending a request that never reached
|
||||||
|
an engine is safe) and logs the failover. Verify with card 1 stopped:
|
||||||
|
|
||||||
|
```
|
||||||
|
systemctl --user stop frank-medium-a.service
|
||||||
|
for i in 1 2 3 4 5 6; do
|
||||||
|
curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \
|
||||||
|
-d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \
|
||||||
|
http://127.0.0.1:7770/v1/chat/completions
|
||||||
|
done; echo
|
||||||
|
podman logs --since 60s frank-medium-proxy | grep -i failover
|
||||||
|
systemctl --user start frank-medium-a.service
|
||||||
|
```
|
||||||
|
|
||||||
|
Six 200s and one `proxy: failover — … unreachable` line is the pass condition.
|
||||||
|
Restarting a stopped replica takes about 150 s (warm compile cache).
|
||||||
|
|
||||||
## Benchmarks
|
## Benchmarks
|
||||||
|
|
||||||
`vllm bench serve` against the proxy (both cards serving), random dataset,
|
`vllm bench serve` against the proxy (both cards serving), random dataset,
|
||||||
|
|||||||
Reference in new issue
Block a user