dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: deploy two single-GPU replicas — TP=2 is blocked by PCIe atomics

Browse Source 
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.

Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).

Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.

Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.

Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
This commit is contained in:
Frank
2026-09-16 03:47:11 -07:00
parent ad7a9c42f9
commit 4fca739bf8
15 changed files with 817 additions and 105 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dankserver/frank-medium/frank-medium-healthcheck.py
+9 -5
View File
@@ -1,17 +1,21 @@
#!/usr/bin/env python3
"""frank-medium container healthcheck: vLLM /health must answer 200.
"""frank-medium container healthcheck: the port's /health must answer 200.
Runs INSIDE the container. The vLLM /health endpoint only answers once the
engine is up (weights loaded, cudagraphs captured), so a healthy container is
a serving container, not merely a live process. The HealthCmd in the quadlet
splits on spaces into an argv array (podman 5.x v1 generator), which is why
this lives in a file rather than an inline python one-liner.
a serving container, not merely a live process. PORT selects the endpoint so
the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can
share this probe. The HealthCmd in the quadlet splits on spaces into an argv
array (podman 5.x v1 generator), which is why this lives in a file rather
than an inline python one-liner.
"""
import os
import sys
import urllib.request
port = os.environ.get("PORT", "7770")
try:
with urllib.request.urlopen("http://127.0.0.1:7770/health", timeout=5) as r:
with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=5) as r:
sys.exit(0 if r.status == 200 else 1)
except Exception:
sys.exit(1)
dankserver/frank-medium/proxy/roundrobin-proxy.py
Executable
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env python3
"""Round-robin proxy in front of the frank-medium TP=1 replicas.
Why this exists: TP=2 (one vLLM process spanning both R9700s) needs RCCL
collectives, and RCCL's kernels require PCIe atomics on every participating
GPU. On this host the second R9700 sits behind chipset bridges that advertise
`AtomicOpsCap: Routing-` (see the runbook), so no collective can launch on it
and TP=2 is impossible. Two single-GPU replicas use both cards instead; this
proxy keeps the fleet contract of one endpoint on :7770.
The vLLM OpenAI API is stateless per request, so round-robin is sound. /health
returns 200 while at least one replica is up and names the ones that are not.
"""
from __future__ import annotations
import asyncio
import itertools
import os
from aiohttp import ClientSession, ClientTimeout, web
UPSTREAMS = [u.strip() for u in os.environ.get(
"UPSTREAMS", "http://127.0.0.1:7771,http://127.0.0.1:7772").split(",") if u.strip()]
PORT = int(os.environ.get("PORT", "7770"))
# Hop-by-hop headers must not be forwarded (RFC 9110 §7.6.1) — forwarding
# transfer-encoding or content-length of a rewritten body breaks clients.
HOP_BY_HOP = {
"host", "content-length", "transfer-encoding", "connection",
"keep-alive", "proxy-authenticate", "proxy-authorization", "te",
"trailers", "upgrade",
}
_cycle = itertools.cycle(range(len(UPSTREAMS)))
async def health(session: ClientSession) -> web.Response:
results = await asyncio.gather(
*(session.get(u + "/health", timeout=ClientTimeout(total=5)) for u in UPSTREAMS),
return_exceptions=True,
)
lines, up = [], 0
for url, res in zip(UPSTREAMS, results):
if isinstance(res, Exception) or res.status != 200:
lines.append(f"{url} DOWN")
else:
up += 1
lines.append(f"{url} UP")
body = "\n".join(lines) + "\n"
return web.Response(status=200 if up else 503, text=body)
async def proxy(request: web.Request) -> web.StreamResponse:
session: ClientSession = request.app["session"]
if request.path == "/health":
return await health(session)
target = UPSTREAMS[next(_cycle)] + request.path_qs
headers = {k: v for k, v in request.headers.items() if k.lower() not in HOP_BY_HOP}
body = await request.read()
try:
upstream = await session.request(
request.method, target, headers=headers, data=body or None,
timeout=ClientTimeout(total=None, sock_connect=10, sock_read=3600),
)
except Exception as exc: # upstream refused/unreachable
return web.Response(status=502, text=f"proxy: {target} unreachable: {exc}\n")
out = web.StreamResponse(status=upstream.status, reason=upstream.reason)
for k, v in upstream.headers.items():
if k.lower() not in HOP_BY_HOP:
out.headers[k] = v
await out.prepare(request)
try:
async for chunk in upstream.content.iter_any():
await out.write(chunk)
finally:
upstream.release()
await out.write_eof()
return out
async def main() -> None:
app = web.Application(client_max_size=1024 ** 3)
app["session"] = ClientSession()
app.router.add_route("*", "/{tail:.*}", proxy)
runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner, "0.0.0.0", PORT)
await site.start()
print(f"proxy: :{PORT} -> {', '.join(UPSTREAMS)}", flush=True)
await asyncio.Event().wait()
if __name__ == "__main__":
asyncio.run(main())
dankserver/frank-medium/quadlet/frank-medium-a.container
+98
View File
@@ -0,0 +1,98 @@
# Quadlet unit: frank-medium replica A (rootless, boot-persistent).
#
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-a.container
# Unit name: frank-medium-a.service (container name: frank-medium-a)
#
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
# tensor-parallel process across both R9700s (see frank-medium.container, kept
# as the target). That shape cannot run on this host: RCCL's collective kernels
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
# atomics. Result: every collective that touches GPU 2 dies at launch.
# Two independent single-GPU replicas use both cards instead; the proxy unit
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
#
# SHAPE. vLLM binds 0.0.0.0:7771 on the host network directly; no PublishPort.
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
# this shape instead of using the 2x7551-32624 pin, which is only valid at
# TP=2/262144.
#
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=0 (the first R9700). The
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
# first and HIP indexes into the filtered list, so a single-valued pair
# (ROCR=1,HIP=1) leaves an empty device set.
#
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
[Unit]
Description=frank-medium replica A — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #1
After=network-online.target
[Container]
ContainerName=frank-medium-a
Image=localhost/radiance-vllm:0.9.3
AddDevice=/dev/kfd
AddDevice=/dev/dri
GroupAdd=keep-groups
# The upstream launcher's exact podman posture (rootless).
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
Environment=HIP_VISIBLE_DEVICES=0
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
Environment=SERVED_NAMES=frank-medium
Environment=PORT=7771
Environment=TP=1
Environment=MAXSEQS=4
Environment=CHUNK=8192
Environment=MAXLEN=114688
Environment=GPU_UTIL=0.98
Environment=KV_MEM=0
Environment=SPEC_METHOD=dflash
Environment=SPEC=7
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
Volume=/home/dank/models:/models
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
# Compile cache. Shared with the other replica and with the TP=2 target: the
# entries are content-addressed, so shapes simply add subdirectories.
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
# The pinned radiance clone: patches applied at every container start.
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
# Entrypoint + healthcheck from this checkout, mounted read-only.
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
# The entrypoint must be the container command; the image's own entrypoint
# (which the patched script execs at the end) must not run unpatched.
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
# 200 only once the engine serves; during first-start compilation the
# container is legitimately unhealthy. HealthCmd is split on spaces into an
# argv array by the podman 5.x v1 generator, hence the file-based probe.
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
HealthInterval=60s
HealthTimeout=30s
HealthRetries=3
HealthStartPeriod=1800s
Pull=never
[Service]
# First start compiles Triton/inductor kernels (several minutes) before the
# engine loads ~20 GiB of weights.
TimeoutStartSec=1800
Restart=on-failure
RestartSec=10
[Install]
WantedBy=default.target
dankserver/frank-medium/quadlet/frank-medium-b.container
+98
View File
@@ -0,0 +1,98 @@
# Quadlet unit: frank-medium replica B (rootless, boot-persistent).
#
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-b.container
# Unit name: frank-medium-b.service (container name: frank-medium-a)
#
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
# tensor-parallel process across both R9700s (see frank-medium.container, kept
# as the target). That shape cannot run on this host: RCCL's collective kernels
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
# atomics. Result: every collective that touches GPU 2 dies at launch.
# Two independent single-GPU replicas use both cards instead; the proxy unit
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
#
# SHAPE. vLLM binds 0.0.0.0:7772 on the host network directly; no PublishPort.
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
# this shape instead of using the 2x7551-32624 pin, which is only valid at
# TP=2/262144.
#
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=1 (the second R9700). The
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
# first and HIP indexes into the filtered list, so a single-valued pair
# (ROCR=1,HIP=1) leaves an empty device set.
#
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
[Unit]
Description=frank-medium replica B — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #2
After=network-online.target
[Container]
ContainerName=frank-medium-b
Image=localhost/radiance-vllm:0.9.3
AddDevice=/dev/kfd
AddDevice=/dev/dri
GroupAdd=keep-groups
# The upstream launcher's exact podman posture (rootless).
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
Environment=HIP_VISIBLE_DEVICES=1
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
Environment=SERVED_NAMES=frank-medium
Environment=PORT=7772
Environment=TP=1
Environment=MAXSEQS=4
Environment=CHUNK=8192
Environment=MAXLEN=114688
Environment=GPU_UTIL=0.98
Environment=KV_MEM=0
Environment=SPEC_METHOD=dflash
Environment=SPEC=7
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
Volume=/home/dank/models:/models
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
# Compile cache. Shared with the other replica and with the TP=2 target: the
# entries are content-addressed, so shapes simply add subdirectories.
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
# The pinned radiance clone: patches applied at every container start.
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
# Entrypoint + healthcheck from this checkout, mounted read-only.
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
# The entrypoint must be the container command; the image's own entrypoint
# (which the patched script execs at the end) must not run unpatched.
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
# 200 only once the engine serves; during first-start compilation the
# container is legitimately unhealthy. HealthCmd is split on spaces into an
# argv array by the podman 5.x v1 generator, hence the file-based probe.
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
HealthInterval=60s
HealthTimeout=30s
HealthRetries=3
HealthStartPeriod=1800s
Pull=never
[Service]
# First start compiles Triton/inductor kernels (several minutes) before the
# engine loads ~20 GiB of weights.
TimeoutStartSec=1800
Restart=on-failure
RestartSec=10
[Install]
WantedBy=default.target
dankserver/frank-medium/quadlet/frank-medium-proxy.container
+52
View File
@@ -0,0 +1,52 @@
# Quadlet unit: frank-medium proxy (rootless, boot-persistent).
#
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
#
# WHAT IT IS. A round-robin reverse proxy in front of the two single-GPU
# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
# the fleet contract: one endpoint on host port 7770 answering as the served
# model name frank-medium, OpenAI /v1, streaming included.
#
# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible
# on this host (see frank-medium-a.container and the runbook: RCCL's kernels
# need PCIe atomics, and the second card's chipset path advertises
# AtomicOpsCap: Routing-). Two replicas are how both cards get used.
#
# ROUTING. vLLM's OpenAI API is stateless per request, so round-robin is
# sound; there is no session affinity to preserve. /health answers 200 while
# at least one replica is up and names the ones that are down. The proxy runs
# on host networking so it can bind 7770 and reach the replicas on loopback.
#
# The script is a few dozen lines of aiohttp, mounted read-only out of
# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
# It reuses the same image as the replicas: aiohttp is already there and no
# extra image has to be pulled on this host.
[Unit]
Description=frank-medium proxy — round-robin in front of the two single-GPU replicas
After=network-online.target
Wants=frank-medium-a.service frank-medium-b.service
After=frank-medium-a.service frank-medium-b.service
[Container]
ContainerName=frank-medium-proxy
Image=localhost/radiance-vllm:0.9.3
PodmanArgs=--network=host
Environment=PORT=7770
Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/roundrobin-proxy.py:/usr/local/bin/roundrobin-proxy.py:ro
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
PodmanArgs=--entrypoint /usr/local/bin/roundrobin-proxy.py
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
HealthInterval=60s
HealthTimeout=30s
HealthRetries=3
HealthStartPeriod=60s
Pull=never
[Service]
Restart=on-failure
RestartSec=10
[Install]
WantedBy=default.target
dankserver/frank-medium/quadlet/frank-medium.container
+9
View File
@@ -19,6 +19,15 @@
# what this unit does too — no PublishPort. The fleet contract is unchanged:
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
#
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
# process across both R9700s. It cannot launch a collective here — RCCL's
# kernels need PCIe atomics and the second card sits behind chipset bridges
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
# two-replica trio in this directory (frank-medium-a/b/proxy) and
# deploy-remote.sh deliberately does not install this unit. Keep it for a
# host whose second card is on an atomic-capable root port; the full evidence
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
#
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
# and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192
dankserver/frank-medium/radiance-entrypoint.sh
+2 -1
View File
@@ -27,7 +27,8 @@ set -euo pipefail
: "${PORT:=7770}"
: "${SERVED_NAMES:=frank-medium}"
: "${MAXLEN:=262144}"
: "${MAXSEQS:=8}"
: "${KV_MEM:=18563072000}"
if [ "$KV_MEM" = "0" ]; then KV_MEM=""; fi # 0 = let vLLM profile (the launcher's rule)
: "${CHUNK:=8192}"
: "${GPU_UTIL:=0.98}"
: "${TP:=2}"