Frank
83625e84a7
The two replicas behind :7770 were on plain round-robin, which throws away prefix-cache hits: an agent loop re-sends its system prompt and file context every turn, and alternating replicas means every other turn pays a cold prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms. Pinning everything to one replica is not the answer either — a cold batch (the shape a benchmark or a burst of parallel subagents produces) then loses a card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread. So the policy routes by request shape. A continuation (more than one message — the agent-loop signature) is pinned to the replica that served its prefix; anything else spreads across the cards by counter. Measured: 19.97 s for the burst and 1095 ms warm-turn TTFT — the best of both, without a regression on either workload. The policy lives in proxy/routing.py, separate from the HTTP plumbing in proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the first version hashed fresh prefixes to pick a card, which sends every member of a same-prefix burst to one card (29.8 s, measured), and a stale index in the pin path returned 500 on every continuation. Both are now assertions. ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the choice on this host; the default is adaptive.
62 lines
3.0 KiB
INI
62 lines
3.0 KiB
INI
# Quadlet unit: frank-medium proxy (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
|
|
# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
|
|
#
|
|
# WHAT IT IS. A prefix-affinity reverse proxy in front of the two single-GPU
|
|
# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
|
|
# the fleet contract: one endpoint on host port 7770 answering as the served
|
|
# model name frank-medium, OpenAI /v1, streaming included.
|
|
#
|
|
# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible
|
|
# on this host (see frank-medium-a.container and the runbook: RCCL's kernels
|
|
# need PCIe atomics, and the second card's chipset path advertises
|
|
# AtomicOpsCap: Routing-). Two replicas are how both cards get used.
|
|
#
|
|
# ROUTING. Requests are keyed on a hash of their leading 4 KiB of prompt text,
|
|
# so a conversation family — an agent loop re-sending the same system prompt,
|
|
# tool list and file context every turn — keeps landing on the same replica,
|
|
# where --enable-prefix-caching has that prefix warm. Plain round-robin threw
|
|
# half of those cache hits away. A replica that is down is skipped, and
|
|
# ROUTING=roundrobin restores the old behaviour for A/B measurement. /health
|
|
# answers 200 while at least one replica is up and names the ones that are
|
|
# down. The proxy runs on host networking so it can bind 7770 and reach the
|
|
# replicas on loopback.
|
|
#
|
|
# The script is a few dozen lines of aiohttp, mounted read-only out of
|
|
# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
|
|
# It reuses the same image as the replicas: aiohttp is already there and no
|
|
# extra image has to be pulled on this host.
|
|
[Unit]
|
|
Description=frank-medium proxy — prefix-affinity in front of the two single-GPU replicas
|
|
After=network-online.target
|
|
Wants=frank-medium-a.service frank-medium-b.service
|
|
After=frank-medium-a.service frank-medium-b.service
|
|
|
|
[Container]
|
|
ContainerName=frank-medium-proxy
|
|
Image=localhost/radiance-vllm:0.9.3
|
|
PodmanArgs=--network=host
|
|
Environment=PORT=7770
|
|
Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
|
|
# The proxy is two files: routing.py holds the replica-selection policy, the
|
|
# server holds the HTTP plumbing. Both are mounted because the server imports
|
|
# the policy by name from its own directory (/usr/local/bin is sys.path[0]).
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/routing.py:/usr/local/bin/routing.py:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/affinity-proxy.py:/usr/local/bin/affinity-proxy.py:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
|
PodmanArgs=--entrypoint /usr/local/bin/affinity-proxy.py
|
|
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
|
HealthInterval=60s
|
|
HealthTimeout=30s
|
|
HealthRetries=3
|
|
HealthStartPeriod=60s
|
|
Pull=never
|
|
|
|
[Service]
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|