diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 73e4c28..5eee249 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -226,6 +226,25 @@ The same call works against either replica directly (`:7771`, `:7772`) — when one of them is down the other still serves, which is why the proxy reports the individual replica states instead of a single boolean. +A replica that dies mid-flight does not cost a client a request: a failed +connection demotes that replica immediately and the proxy retries on the live +one (vLLM's endpoints are stateless, so re-sending a request that never reached +an engine is safe) and logs the failover. Verify with card 1 stopped: + +``` +systemctl --user stop frank-medium-a.service +for i in 1 2 3 4 5 6; do + curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \ + -d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \ + http://127.0.0.1:7770/v1/chat/completions +done; echo +podman logs --since 60s frank-medium-proxy | grep -i failover +systemctl --user start frank-medium-a.service +``` + +Six 200s and one `proxy: failover — … unreachable` line is the pass condition. +Restarting a stopped replica takes about 150 s (warm compile cache). + ## Benchmarks `vllm bench serve` against the proxy (both cards serving), random dataset,