Frank
05c0d3a00a
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
62 lines
3.0 KiB
INI
62 lines
3.0 KiB
INI
# Quadlet unit: frank-medium proxy (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
|
|
# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
|
|
#
|
|
# WHAT IT IS. A prefix-affinity reverse proxy in front of the two single-GPU
|
|
# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
|
|
# the fleet contract: one endpoint on host port 7770 answering as the served
|
|
# model name frank-medium, OpenAI /v1, streaming included.
|
|
#
|
|
# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible
|
|
# on this host (see frank-medium-a.container and the runbook: RCCL's kernels
|
|
# need PCIe atomics, and the second card's chipset path advertises
|
|
# AtomicOpsCap: Routing-). Two replicas are how both cards get used.
|
|
#
|
|
# ROUTING. Requests are keyed on a hash of their leading 4 KiB of prompt text,
|
|
# so a conversation family — an agent loop re-sending the same system prompt,
|
|
# tool list and file context every turn — keeps landing on the same replica,
|
|
# where --enable-prefix-caching has that prefix warm. Plain round-robin threw
|
|
# half of those cache hits away. A replica that is down is skipped, and
|
|
# ROUTING=roundrobin restores the old behaviour for A/B measurement. /health
|
|
# answers 200 while at least one replica is up and names the ones that are
|
|
# down. The proxy runs on host networking so it can bind 7770 and reach the
|
|
# replicas on loopback.
|
|
#
|
|
# The script is a few dozen lines of aiohttp, mounted read-only out of
|
|
# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
|
|
# It reuses the same image as the replicas: aiohttp is already there and no
|
|
# extra image has to be pulled on this host.
|
|
[Unit]
|
|
Description=frank-medium proxy — prefix-affinity in front of the two single-GPU replicas
|
|
After=network-online.target
|
|
Wants=frank-medium-a.service frank-medium-b.service
|
|
After=frank-medium-a.service frank-medium-b.service
|
|
|
|
[Container]
|
|
ContainerName=frank-medium-proxy
|
|
Image=localhost/radiance-vllm:0.9.3
|
|
PodmanArgs=--network=host
|
|
Environment=PORT=7770
|
|
Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
|
|
# The proxy is two files: routing.py holds the replica-selection policy, the
|
|
# server holds the HTTP plumbing. Both are mounted because the server imports
|
|
# the policy by name from its own directory (/usr/local/bin is sys.path[0]).
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/routing.py:/usr/local/bin/routing.py:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/affinity-proxy.py:/usr/local/bin/affinity-proxy.py:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
|
PodmanArgs=--entrypoint /usr/local/bin/affinity-proxy.py
|
|
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
|
HealthInterval=60s
|
|
HealthTimeout=30s
|
|
HealthRetries=3
|
|
HealthStartPeriod=60s
|
|
Pull=never
|
|
|
|
[Service]
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|