frank-medium: deploy two single-GPU replicas — TP=2 is blocked by PCIe atomics
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
This commit is contained in:
Frank
@@ -23,7 +23,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 |
|
||||
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind a round-robin proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 |
|
||||
| Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 |
|
||||
|
||||
Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in
|
||||
@@ -248,23 +248,61 @@ Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`.
|
||||
Rollback: [[halogen-flash-server]] (quadlet archived in
|
||||
`~/.AGENTTRASH/20260915-ik-swap/` on dankclaw).
|
||||
|
||||
### frank-medium — ROCmFPX llama.cpp (Dankserver)
|
||||
### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE)
|
||||
|
||||
charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for gfx1201 with the fork's
|
||||
self-contained ROCm 7.14 toolchain (RPATH-bundled — host ROCm packages never
|
||||
participate). Serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF`
|
||||
(ftype 115; stock llama.cpp cannot load it) with `--split-mode layer` across
|
||||
both R9700s, 131072 ctx / 1 slot, f16 KV, flash-attn on, MTP draft head
|
||||
pinned to ROCm1, reasoning on. Env `scripts/frank-medium.env`, serve
|
||||
`scripts/serve-frank-medium.sh`.
|
||||
`stilldeadcode/vllm-radiance:0.9.3` plus the `ggz14/radiance-vllm-mxfp4`
|
||||
patches (pinned clone at `/home/dank/GitPackages/radiance-vllm-mxfp4`; its
|
||||
patches are applied at every container start), serving
|
||||
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` with the
|
||||
`Qwen3.8-27B-DFlash2-FP8` block-diffusion drafter (`SPEC=7`).
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md`.
|
||||
The deployment is **two single-GPU replicas behind a round-robin proxy** —
|
||||
`frank-medium-a` (:7771, R9700 #1), `frank-medium-b` (:7772, R9700 #2) and
|
||||
`frank-medium-proxy` (:7770, the fleet endpoint), three rootless Quadlets
|
||||
under `dankserver/frank-medium/quadlet/`. Each replica is TP=1 at
|
||||
114688 context / 4 sequences with vLLM-profiled KV (`KV_MEM=0`).
|
||||
|
||||
**TP=2 across both cards cannot run on this host.** RCCL's collective kernels
|
||||
carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on
|
||||
a device without PCIe atomics; the second R9700 sits behind the chipset
|
||||
bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a
|
||||
read-only capability, so no software can enable atomics on that path. Not the
|
||||
kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL
|
||||
build (a from-source rebuild fails identically). A slot on a CPU root port —
|
||||
there is none free on this AM5 platform for a second x16 card — or a platform
|
||||
with more CPU lanes restores TP=2. `frank-medium.container` stays in the repo
|
||||
as the TP=2 target; the deploy handler does not install it.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
|
||||
Prior runtime: [[frank-medium-rocmfpx]] (llama.cpp era, same day).
|
||||
|
||||
## Key Lessons Learned
|
||||
|
||||
Durable operational lessons from the fleet's history. Full detail in the
|
||||
linked runbooks.
|
||||
|
||||
**PCIe / multi-GPU**
|
||||
|
||||
- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
|
||||
kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
|
||||
them when the device has no atomics, and the failure surfaces as
|
||||
`hipErrorIllegalState` ("the operation cannot be performed in the present
|
||||
state") from `hipLaunchKernel`, not as anything that names atomics. The
|
||||
value is consumed at driver probe time, so writing the AtomicOp Requester
|
||||
bit with `setpci` at runtime changes nothing. Check the *chain*:
|
||||
`lspci -vv -s <gpu>` and every upstream bridge must show
|
||||
`AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
|
||||
endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
|
||||
6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
|
||||
and the rest behind the chipset (usually not) — that asymmetry is what
|
||||
blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
|
||||
short-circuits at `world_size=1`.
|
||||
- **When a GPU collective dies, read the runtime, not the framework.** The
|
||||
message that identified the cause came from `AMD_LOG_LEVEL=3`
|
||||
(`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
|
||||
failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
|
||||
prints the failing HIP call with its return code.
|
||||
|
||||
|
||||
**ROCm / GPU**
|
||||
|
||||
- **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor
|
||||
@@ -359,7 +397,7 @@ Active:
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-medium (ROCmFPX, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` |
|
||||
| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
|
||||
@@ -18,13 +18,13 @@ providers:
|
||||
discover_models: false
|
||||
models:
|
||||
frank-large:
|
||||
context_length: 262144
|
||||
context_length: 114688
|
||||
dankserver-small:
|
||||
api: http://192.168.0.235:7770/v1
|
||||
discover_models: false
|
||||
models:
|
||||
frank-medium:
|
||||
context_length: 163840
|
||||
context_length: 114688
|
||||
|
||||
# Auxiliary routing from the live configuration. Named providers route via
|
||||
# their `api:` URL; `custom` entries route by their explicit `base_url`.
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
#!/usr/bin/env python3
|
||||
"""frank-medium container healthcheck: vLLM /health must answer 200.
|
||||
"""frank-medium container healthcheck: the port's /health must answer 200.
|
||||
|
||||
Runs INSIDE the container. The vLLM /health endpoint only answers once the
|
||||
engine is up (weights loaded, cudagraphs captured), so a healthy container is
|
||||
a serving container, not merely a live process. The HealthCmd in the quadlet
|
||||
splits on spaces into an argv array (podman 5.x v1 generator), which is why
|
||||
this lives in a file rather than an inline python one-liner.
|
||||
a serving container, not merely a live process. PORT selects the endpoint so
|
||||
the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can
|
||||
share this probe. The HealthCmd in the quadlet splits on spaces into an argv
|
||||
array (podman 5.x v1 generator), which is why this lives in a file rather
|
||||
than an inline python one-liner.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
port = os.environ.get("PORT", "7770")
|
||||
try:
|
||||
with urllib.request.urlopen("http://127.0.0.1:7770/health", timeout=5) as r:
|
||||
with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=5) as r:
|
||||
sys.exit(0 if r.status == 200 else 1)
|
||||
except Exception:
|
||||
sys.exit(1)
|
||||
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Round-robin proxy in front of the frank-medium TP=1 replicas.
|
||||
|
||||
Why this exists: TP=2 (one vLLM process spanning both R9700s) needs RCCL
|
||||
collectives, and RCCL's kernels require PCIe atomics on every participating
|
||||
GPU. On this host the second R9700 sits behind chipset bridges that advertise
|
||||
`AtomicOpsCap: Routing-` (see the runbook), so no collective can launch on it
|
||||
and TP=2 is impossible. Two single-GPU replicas use both cards instead; this
|
||||
proxy keeps the fleet contract of one endpoint on :7770.
|
||||
|
||||
The vLLM OpenAI API is stateless per request, so round-robin is sound. /health
|
||||
returns 200 while at least one replica is up and names the ones that are not.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import itertools
|
||||
import os
|
||||
from aiohttp import ClientSession, ClientTimeout, web
|
||||
|
||||
UPSTREAMS = [u.strip() for u in os.environ.get(
|
||||
"UPSTREAMS", "http://127.0.0.1:7771,http://127.0.0.1:7772").split(",") if u.strip()]
|
||||
PORT = int(os.environ.get("PORT", "7770"))
|
||||
# Hop-by-hop headers must not be forwarded (RFC 9110 §7.6.1) — forwarding
|
||||
# transfer-encoding or content-length of a rewritten body breaks clients.
|
||||
HOP_BY_HOP = {
|
||||
"host", "content-length", "transfer-encoding", "connection",
|
||||
"keep-alive", "proxy-authenticate", "proxy-authorization", "te",
|
||||
"trailers", "upgrade",
|
||||
}
|
||||
|
||||
_cycle = itertools.cycle(range(len(UPSTREAMS)))
|
||||
|
||||
|
||||
async def health(session: ClientSession) -> web.Response:
|
||||
results = await asyncio.gather(
|
||||
*(session.get(u + "/health", timeout=ClientTimeout(total=5)) for u in UPSTREAMS),
|
||||
return_exceptions=True,
|
||||
)
|
||||
lines, up = [], 0
|
||||
for url, res in zip(UPSTREAMS, results):
|
||||
if isinstance(res, Exception) or res.status != 200:
|
||||
lines.append(f"{url} DOWN")
|
||||
else:
|
||||
up += 1
|
||||
lines.append(f"{url} UP")
|
||||
body = "\n".join(lines) + "\n"
|
||||
return web.Response(status=200 if up else 503, text=body)
|
||||
|
||||
|
||||
async def proxy(request: web.Request) -> web.StreamResponse:
|
||||
session: ClientSession = request.app["session"]
|
||||
if request.path == "/health":
|
||||
return await health(session)
|
||||
|
||||
target = UPSTREAMS[next(_cycle)] + request.path_qs
|
||||
headers = {k: v for k, v in request.headers.items() if k.lower() not in HOP_BY_HOP}
|
||||
body = await request.read()
|
||||
|
||||
try:
|
||||
upstream = await session.request(
|
||||
request.method, target, headers=headers, data=body or None,
|
||||
timeout=ClientTimeout(total=None, sock_connect=10, sock_read=3600),
|
||||
)
|
||||
except Exception as exc: # upstream refused/unreachable
|
||||
return web.Response(status=502, text=f"proxy: {target} unreachable: {exc}\n")
|
||||
|
||||
out = web.StreamResponse(status=upstream.status, reason=upstream.reason)
|
||||
for k, v in upstream.headers.items():
|
||||
if k.lower() not in HOP_BY_HOP:
|
||||
out.headers[k] = v
|
||||
await out.prepare(request)
|
||||
try:
|
||||
async for chunk in upstream.content.iter_any():
|
||||
await out.write(chunk)
|
||||
finally:
|
||||
upstream.release()
|
||||
await out.write_eof()
|
||||
return out
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
app = web.Application(client_max_size=1024 ** 3)
|
||||
app["session"] = ClientSession()
|
||||
app.router.add_route("*", "/{tail:.*}", proxy)
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, "0.0.0.0", PORT)
|
||||
await site.start()
|
||||
print(f"proxy: :{PORT} -> {', '.join(UPSTREAMS)}", flush=True)
|
||||
await asyncio.Event().wait()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,98 @@
|
||||
# Quadlet unit: frank-medium replica A (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-a.container
|
||||
# Unit name: frank-medium-a.service (container name: frank-medium-a)
|
||||
#
|
||||
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
|
||||
# tensor-parallel process across both R9700s (see frank-medium.container, kept
|
||||
# as the target). That shape cannot run on this host: RCCL's collective kernels
|
||||
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
|
||||
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
|
||||
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
|
||||
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
|
||||
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
|
||||
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
|
||||
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
|
||||
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
|
||||
# atomics. Result: every collective that touches GPU 2 dies at launch.
|
||||
# Two independent single-GPU replicas use both cards instead; the proxy unit
|
||||
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
|
||||
#
|
||||
# SHAPE. vLLM binds 0.0.0.0:7771 on the host network directly; no PublishPort.
|
||||
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
|
||||
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
|
||||
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
|
||||
# this shape instead of using the 2x7551-32624 pin, which is only valid at
|
||||
# TP=2/262144.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=0 (the first R9700). The
|
||||
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
|
||||
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
|
||||
# first and HIP indexes into the filtered list, so a single-valued pair
|
||||
# (ROCR=1,HIP=1) leaves an empty device set.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium replica A — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #1
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium-a
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
# The upstream launcher's exact podman posture (rootless).
|
||||
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
||||
Environment=HIP_VISIBLE_DEVICES=0
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
Environment=PORT=7771
|
||||
Environment=TP=1
|
||||
Environment=MAXSEQS=4
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=114688
|
||||
Environment=GPU_UTIL=0.98
|
||||
Environment=KV_MEM=0
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
|
||||
Volume=/home/dank/models:/models
|
||||
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Shared with the other replica and with the TP=2 target: the
|
||||
# entries are content-addressed, so shapes simply add subdirectories.
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
|
||||
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
||||
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
# 200 only once the engine serves; during first-start compilation the
|
||||
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
||||
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=1800s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start compiles Triton/inductor kernels (several minutes) before the
|
||||
# engine loads ~20 GiB of weights.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,98 @@
|
||||
# Quadlet unit: frank-medium replica B (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-b.container
|
||||
# Unit name: frank-medium-b.service (container name: frank-medium-a)
|
||||
#
|
||||
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
|
||||
# tensor-parallel process across both R9700s (see frank-medium.container, kept
|
||||
# as the target). That shape cannot run on this host: RCCL's collective kernels
|
||||
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
|
||||
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
|
||||
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
|
||||
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
|
||||
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
|
||||
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
|
||||
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
|
||||
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
|
||||
# atomics. Result: every collective that touches GPU 2 dies at launch.
|
||||
# Two independent single-GPU replicas use both cards instead; the proxy unit
|
||||
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
|
||||
#
|
||||
# SHAPE. vLLM binds 0.0.0.0:7772 on the host network directly; no PublishPort.
|
||||
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
|
||||
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
|
||||
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
|
||||
# this shape instead of using the 2x7551-32624 pin, which is only valid at
|
||||
# TP=2/262144.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=1 (the second R9700). The
|
||||
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
|
||||
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
|
||||
# first and HIP indexes into the filtered list, so a single-valued pair
|
||||
# (ROCR=1,HIP=1) leaves an empty device set.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium replica B — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #2
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium-b
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
GroupAdd=keep-groups
|
||||
# The upstream launcher's exact podman posture (rootless).
|
||||
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
||||
Environment=HIP_VISIBLE_DEVICES=1
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
Environment=PORT=7772
|
||||
Environment=TP=1
|
||||
Environment=MAXSEQS=4
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=114688
|
||||
Environment=GPU_UTIL=0.98
|
||||
Environment=KV_MEM=0
|
||||
Environment=SPEC_METHOD=dflash
|
||||
Environment=SPEC=7
|
||||
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
|
||||
Volume=/home/dank/models:/models
|
||||
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Shared with the other replica and with the TP=2 target: the
|
||||
# entries are content-addressed, so shapes simply add subdirectories.
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
|
||||
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
||||
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
# 200 only once the engine serves; during first-start compilation the
|
||||
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
||||
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=1800s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
# First start compiles Triton/inductor kernels (several minutes) before the
|
||||
# engine loads ~20 GiB of weights.
|
||||
TimeoutStartSec=1800
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -0,0 +1,52 @@
|
||||
# Quadlet unit: frank-medium proxy (rootless, boot-persistent).
|
||||
#
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
|
||||
# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
|
||||
#
|
||||
# WHAT IT IS. A round-robin reverse proxy in front of the two single-GPU
|
||||
# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
|
||||
# the fleet contract: one endpoint on host port 7770 answering as the served
|
||||
# model name frank-medium, OpenAI /v1, streaming included.
|
||||
#
|
||||
# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible
|
||||
# on this host (see frank-medium-a.container and the runbook: RCCL's kernels
|
||||
# need PCIe atomics, and the second card's chipset path advertises
|
||||
# AtomicOpsCap: Routing-). Two replicas are how both cards get used.
|
||||
#
|
||||
# ROUTING. vLLM's OpenAI API is stateless per request, so round-robin is
|
||||
# sound; there is no session affinity to preserve. /health answers 200 while
|
||||
# at least one replica is up and names the ones that are down. The proxy runs
|
||||
# on host networking so it can bind 7770 and reach the replicas on loopback.
|
||||
#
|
||||
# The script is a few dozen lines of aiohttp, mounted read-only out of
|
||||
# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
|
||||
# It reuses the same image as the replicas: aiohttp is already there and no
|
||||
# extra image has to be pulled on this host.
|
||||
[Unit]
|
||||
Description=frank-medium proxy — round-robin in front of the two single-GPU replicas
|
||||
After=network-online.target
|
||||
Wants=frank-medium-a.service frank-medium-b.service
|
||||
After=frank-medium-a.service frank-medium-b.service
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium-proxy
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
PodmanArgs=--network=host
|
||||
Environment=PORT=7770
|
||||
Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/roundrobin-proxy.py:/usr/local/bin/roundrobin-proxy.py:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
PodmanArgs=--entrypoint /usr/local/bin/roundrobin-proxy.py
|
||||
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
||||
HealthInterval=60s
|
||||
HealthTimeout=30s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=60s
|
||||
Pull=never
|
||||
|
||||
[Service]
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -19,6 +19,15 @@
|
||||
# what this unit does too — no PublishPort. The fleet contract is unchanged:
|
||||
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
|
||||
#
|
||||
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
|
||||
# process across both R9700s. It cannot launch a collective here — RCCL's
|
||||
# kernels need PCIe atomics and the second card sits behind chipset bridges
|
||||
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
|
||||
# two-replica trio in this directory (frank-medium-a/b/proxy) and
|
||||
# deploy-remote.sh deliberately does not install this unit. Keep it for a
|
||||
# host whose second card is on an atomic-capable root port; the full evidence
|
||||
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
|
||||
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
|
||||
# and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192
|
||||
|
||||
@@ -27,7 +27,8 @@ set -euo pipefail
|
||||
: "${PORT:=7770}"
|
||||
: "${SERVED_NAMES:=frank-medium}"
|
||||
: "${MAXLEN:=262144}"
|
||||
: "${MAXSEQS:=8}"
|
||||
: "${KV_MEM:=18563072000}"
|
||||
if [ "$KV_MEM" = "0" ]; then KV_MEM=""; fi # 0 = let vLLM profile (the launcher's rule)
|
||||
: "${CHUNK:=8192}"
|
||||
: "${GPU_UTIL:=0.98}"
|
||||
: "${TP:=2}"
|
||||
|
||||
@@ -991,9 +991,30 @@ deploy_llm_medium() {
|
||||
# The compile cache must exist before the quadlet mounts resolve; the
|
||||
# one-time setup (setup-mxfp4.sh / PREPARE_ONLY) normally creates it.
|
||||
mkdir -p /home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq/{vllm,inductor,triton,aiter}
|
||||
install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium
|
||||
systemctl --user restart frank-medium.service
|
||||
wait_for_health frank-medium http://127.0.0.1:7770/health 1200
|
||||
# Three units, not one. TP=2 across both R9700s cannot launch a collective
|
||||
# on this host — RCCL's kernels need PCIe atomics and the second card's
|
||||
# chipset path advertises AtomicOpsCap: Routing- — so the endpoint is two
|
||||
# single-GPU replicas behind a round-robin proxy. frank-medium.container
|
||||
# stays in the repo as the TP=2 target for a host whose second card sits
|
||||
# on a CPU root port; it is deliberately NOT installed here.
|
||||
install_quadlet frank-medium-a dankserver/frank-medium/quadlet frank-medium-a
|
||||
install_quadlet frank-medium-b dankserver/frank-medium/quadlet frank-medium-b
|
||||
install_quadlet frank-medium-proxy dankserver/frank-medium/quadlet frank-medium-proxy
|
||||
# Drop a stale TP=2 unit from an earlier deploy of this service, or the
|
||||
# generator keeps trying to start a container that can never become
|
||||
# healthy. Never `rm`: it moves into the home trash.
|
||||
if [[ -e "$quadlet_unit_dir/frank-medium.container" ]]; then
|
||||
local trash="$HOME/.AGENTTRASH/20260916-frank-medium-tp2"
|
||||
mkdir -p "$trash"
|
||||
systemctl --user stop frank-medium.service 2>/dev/null || true
|
||||
mv -- "$quadlet_unit_dir/frank-medium.container" "$trash/frank-medium.container"
|
||||
systemctl --user daemon-reload
|
||||
echo "retired the TP=2 quadlet -> $trash (see the runbook for why)"
|
||||
fi
|
||||
# The proxy waits on the replicas, but a restart of all three is what a
|
||||
# deploy does: the replicas reload the model, the proxy reconnects.
|
||||
systemctl --user restart frank-medium-a.service frank-medium-b.service frank-medium-proxy.service
|
||||
wait_for_health frank-medium http://127.0.0.1:7770/health 1800
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
|
||||
@@ -0,0 +1,236 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-medium vllm-radiance MXFP4 Runbook
|
||||
description: vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201), as two single-GPU replicas behind a round-robin proxy — and the PCIe-atomics reason TP=2 cannot run on this host.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
- vllm
|
||||
- vllm-radiance
|
||||
- mxfp4
|
||||
- gfx1201
|
||||
- r9700
|
||||
- rccl
|
||||
- pcie-atomics
|
||||
- quadlet
|
||||
timestamp: 2026-09-16
|
||||
---
|
||||
|
||||
# frank-medium vllm-radiance MXFP4 Runbook
|
||||
|
||||
The ACTIVE frank-medium deployment (since 2026-09-15/16): the
|
||||
`stilldeadcode/vllm-radiance:0.9.3` image plus the `ggz14/radiance-vllm-mxfp4`
|
||||
patches, serving `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` on the two
|
||||
Radeon AI PRO R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX
|
||||
llama.cpp deployment ([[frank-medium-rocmfpx]]).
|
||||
|
||||
**Deployed shape: two single-GPU replicas behind a round-robin proxy.** The
|
||||
intended shape — one tensor-parallel process across both cards (TP=2) — cannot
|
||||
launch a collective on this host. The reason is a hardware property of the
|
||||
motherboard's PCIe topology, not the image, the RCCL build, or the kernel
|
||||
version. The evidence is in [Why TP=2 cannot run here](#why-tp2-cannot-run-here).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
C[clients: Hermes, harness] --> P["frank-medium-proxy<br/>:7770 round-robin"]
|
||||
P --> A["frank-medium-a<br/>R9700 #1, TP=1, :7771"]
|
||||
P --> B["frank-medium-b<br/>R9700 #2, TP=1, :7772"]
|
||||
```
|
||||
|
||||
Three rootless Quadlet units, all `WantedBy=default.target` + `Restart=on-failure`:
|
||||
|
||||
| Unit | Container | GPU | Port | Role |
|
||||
|---|---|---|---|---|
|
||||
| `frank-medium-a` | `frank-medium-a` | `HIP_VISIBLE_DEVICES=0` | 7771 | replica |
|
||||
| `frank-medium-b` | `frank-medium-b` | `HIP_VISIBLE_DEVICES=1` | 7772 | replica |
|
||||
| `frank-medium-proxy` | `frank-medium-proxy` | — | 7770 | round-robin |
|
||||
|
||||
The fleet contract is unchanged: one endpoint on :7770, served model name
|
||||
`frank-medium`, OpenAI `/v1`, streaming included. The vLLM OpenAI API is
|
||||
stateless per request, so round-robin needs no session affinity.
|
||||
|
||||
Only `HIP_VISIBLE_DEVICES` is set per replica, never a matching
|
||||
`ROCR_VISIBLE_DEVICES`: ROCR filters first and HIP then indexes into the
|
||||
filtered list, so the single-valued pair (`ROCR=1,HIP=1`) leaves an empty
|
||||
device set and the engine dies with `hipErrorNoDevice`.
|
||||
|
||||
`frank-medium.container` (TP=2, `KV_MEM=18563072000`, `MAXLEN=262144`) stays in
|
||||
the repo as the target shape for a host whose second card sits on an
|
||||
atomic-capable root port. `deploy-remote.sh` deliberately does not install it
|
||||
and moves any stale copy into `~/.AGENTTRASH/`, because a TP=2 unit on this
|
||||
host would restart forever without ever becoming healthy.
|
||||
|
||||
## Why TP=2 cannot run here
|
||||
|
||||
### Symptom
|
||||
|
||||
Any two-rank collective dies at launch:
|
||||
|
||||
```
|
||||
[FATAL ERROR]: HIP failure: 'the operation cannot be performed in the present state'
|
||||
group.cc:337 -> group.cc:656 -> group.cc:801 -> enqueue.cc:3111
|
||||
```
|
||||
|
||||
With the runtime's own API log (`AMD_LOG_LEVEL=3`) the failing call is explicit:
|
||||
|
||||
```
|
||||
ShaderName : ncclDevKernel_Generic_4(ncclDevKernelArgsStorage<4096ul>)
|
||||
:1:rocvirtual.cpp :4208: Pcie atomics not enabled, hostcall not supported
|
||||
:1:rocvirtual.cpp :4636: AQL dispatch failed!
|
||||
:3:hip_module.cpp :893 : hipLaunchKernel: Returned hipErrorIllegalState
|
||||
```
|
||||
|
||||
ROCclr refuses to hand the kernel a hostcall buffer when the device has no
|
||||
PCIe atomics (`rocclr/device/rocm/rocvirtual.cpp:3963`), and the launch aborts.
|
||||
`hipErrorIllegalState` is that refusal, not a driver fault.
|
||||
|
||||
### Chain of causation
|
||||
|
||||
1. RCCL's collective kernels carry a hidden **hostcall buffer** parameter.
|
||||
2. ROCr passes that buffer only when `pcie_atomics_` is true for the device;
|
||||
otherwise it returns "not supported" and the dispatch fails.
|
||||
3. `pcie_atomics_` is set once, at device init, from the HSA link info between
|
||||
the CPU agent and the GPU (`rocclr/device/rocm/rocdevice.cpp:186`,
|
||||
`Device::checkAtomicSupport`). It is not re-read per process.
|
||||
4. The KFD driver populates that link info from
|
||||
`pci_enable_atomic_ops_to_root()`, which — per the PCIe spec and the
|
||||
kernel's implementation — requires **every upstream bridge** between the
|
||||
device and its root port to advertise `AtomicOp Routing` capability
|
||||
(`drivers/pci/pci.c`).
|
||||
|
||||
### The hardware difference between the two cards
|
||||
|
||||
| Card | PCIe chain (endpoint → root port) | `AtomicOpsCap` on every hop | AtomicOpsCtl |
|
||||
|---|---|---|---|
|
||||
| R9700 #1 (`03:00.0`) | `02:00.0` → `01:00.0` → `00:01.1` | Routing+ / Routing+ / Routing+ | `ReqEn+` (atomics ON) |
|
||||
| R9700 #2 (`0b:00.0`) | `0a:00.0` → `09:00.0` → `06:09.0` → `05:00.0` → `00:02.1` | Routing+ / Routing+ / **Routing−** / **Routing−** / Routing+ | `ReqEn−` (atomics OFF) |
|
||||
|
||||
Card #1 is on a CPU root port; card #2 is behind the chipset bridges
|
||||
`05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-`. That capability
|
||||
bit is **read-only** — no software can enable atomic routing through those
|
||||
bridges — so the driver's atomic enablement call fails for card #2 and the KFD
|
||||
marks the CPU↔GPU2 link `NO_ATOMICS`.
|
||||
|
||||
Writing the AtomicOp Requester bit into the endpoint's Device Control 2 by hand
|
||||
(`setpci -s 0b:00.0 CAP_EXP+28.w`) does not help: the driver reads the chain at
|
||||
probe time, and ROCr consumes the driver's cached answer. Only a re-probe (a
|
||||
boot with the write already in place) would change it — and even then the two
|
||||
`Routing-` bridges would still make `pci_enable_atomic_ops_to_root()` fail.
|
||||
|
||||
### What was ruled out
|
||||
|
||||
| Suspect | Result |
|
||||
|---|---|
|
||||
| Kernel version | 7.0.0-28, -30 and -31 fail identically (full `apt full-upgrade` to the newest available kernel changed nothing) |
|
||||
| IOMMU | identical before and after `iommu=pt` |
|
||||
| P2P hardware | fine: 45.3 GB/s bidirectional copy between the two cards; both cards pass `rocm-bandwidth-test` |
|
||||
| The shipped librccl | a from-source rebuild of `ROCm/rccl` develop (with local fixes for that branch's dangling references) reproduces the same failure at the same call path — the library is not at fault |
|
||||
| vLLM/pynccl | a bare two-rank `torch.distributed` allreduce with no vLLM involved fails the same way; single-rank collectives succeed on both cards (NCCL short-circuits at `world_size=1`, which is why a one-card test proves nothing) |
|
||||
| Transport knobs | `NCCL_PROTO=Simple`, `NCCL_P2P_DISABLE`, `NCCL_SHM_DISABLE`, `NCCL_CUMEM_ENABLE=0`, MSCCL off, channel counts, launch modes, `HSA_ENABLE_SDMA=0` — all identical |
|
||||
| Image/support matrix | the same image on a different host with the same kernel (Proxmox VM, Codeberg issue #22) fails byte-identically |
|
||||
|
||||
### What would fix it
|
||||
|
||||
- **Move card #2 to a slot whose path is atomic-capable** (a CPU root port, as
|
||||
card #1 has). This is the only software-free fix and the one that restores
|
||||
true TP=2. On this AM5 platform the CPU provides the x16 slot plus x4 lanes,
|
||||
so a second x16-class card necessarily lands on the chipset path — a
|
||||
platform with more CPU lanes (Threadripper/EPYC) is the structural answer.
|
||||
- Until then, **two replicas** are how both cards serve.
|
||||
|
||||
## Deployed configuration
|
||||
|
||||
Both replicas, uniform except GPU index and port:
|
||||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| `TP` | 1 | one card per replica |
|
||||
| `MAXLEN` | 114688 | the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.14 available, and 131072 needs 5.58 against 5.22 |
|
||||
| `KV_MEM` | 0 | vLLM profiles this shape; the upstream pin (2x7551-32624) is a TP=2/262144 measurement and is invalid here |
|
||||
| `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency |
|
||||
| `CHUNK` | 8192 | prefill chunk (upstream default) |
|
||||
| `GPU_UTIL` | 0.98 | upstream default; the card is dedicated |
|
||||
| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter, upstream's measured production depth |
|
||||
| `SNAP` / `DRAFTER` | `/models/Qwen3.8-27B-Uncensored-MXFP4-awq` / `/models/Qwen3.8-27B-DFlash2-FP8` | the served checkpoint and its drafter |
|
||||
|
||||
Mounts are the TP=2 unit's minus nothing: `/models`, the HF cache, the shared
|
||||
compile cache, `/patches` (the pinned radiance clone; its patches are applied
|
||||
at every container start), `/r4d` (the patched libr4d — the image's shipped
|
||||
`r4d.so` predates the gated-delta-net overflow fix and NaNs this model), and
|
||||
the entrypoint/healthcheck/proxy scripts mounted read-only out of
|
||||
`/home/dank/serverAppConfigs`.
|
||||
|
||||
The healthcheck reads `PORT`, so one probe serves the replicas (:7771/:7772),
|
||||
the proxy (:7770) and the TP=2 target.
|
||||
|
||||
## Deploy and verify
|
||||
|
||||
```
|
||||
./tests/run.sh # unit/config seams
|
||||
./scripts/deploy.sh dankserver --service frank-medium
|
||||
```
|
||||
|
||||
The handler re-tags `docker.io/stilldeadcode/vllm-radiance:0.9.3` as
|
||||
`localhost/radiance-vllm:0.9.3` (this host's quadlet path cannot resolve
|
||||
registry-prefixed refs from local storage at create time), installs the three
|
||||
units, retires any stale TP=2 unit into the trash, restarts the trio and waits
|
||||
for `http://127.0.0.1:7770/health`.
|
||||
|
||||
Verification that means something:
|
||||
|
||||
```
|
||||
curl -s http://192.168.0.235:7770/health # proxy: names any replica that is down
|
||||
curl -s http://192.168.0.235:7770/v1/models # served name must be frank-medium
|
||||
# and a real completion through the proxy:
|
||||
curl -s http://192.168.0.235:7770/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"frank-medium","messages":[{"role":"user","content":"Say OK"}],"max_tokens":400}'
|
||||
```
|
||||
|
||||
The same call works against either replica directly (`:7771`, `:7772`) — when
|
||||
one of them is down the other still serves, which is why the proxy reports the
|
||||
individual replica states instead of a single boolean.
|
||||
|
||||
## Benchmarks
|
||||
|
||||
`vllm bench serve` against the proxy (both cards serving), random dataset,
|
||||
`--ignore-eos`, `2026-09-16`:
|
||||
|
||||
| Case | Output t/s | TTFT p50 | TPOT p50 | Acceptance |
|
||||
|---|--:|--:|--:|--:|
|
||||
| conc 1, in 1k / out 512 | 42.8 | 382 ms | 23.6 ms | 43.1 % |
|
||||
| conc 4, in 1k / out 512 | 147.6 | 471 ms | 25.3 ms | 34.2 % |
|
||||
| conc 8, in 1k / out 512 | 230.8 | 913 ms | 28.8 ms | 39.0 % |
|
||||
| conc 1, in 32k / out 32 | 2.5 | 11837 ms | 29.6 ms | 25.0 % |
|
||||
|
||||
The 32k row is a prefill measurement: 32768 prompt tokens in 11.8 s is
|
||||
**≈2.8k tok/s of prefill on a single card**. The replicas split the fleet's
|
||||
traffic, they do not pool capacity for one request, against the upstream
|
||||
reference's 4585 t/s at 32k on TP=2 — one card doing the whole prompt lands
|
||||
where you would expect.
|
||||
|
||||
Read these against the upstream repo's own reference table (2x R9700, TP=2,
|
||||
BetterBench corpus): step time 22.8-23.2 ms per update, combined decode
|
||||
224 t/s, prefill 4585 t/s at 32k. Step time (TPOT) is the comparable number —
|
||||
tokens/s swings with draft acceptance, and synthetic random prompts accept far
|
||||
worse than the code/prose mix upstream measures (43 % here against 5.8
|
||||
tokens/update there), so a lower token rate at the same step time is the
|
||||
dataset, not the deployment.
|
||||
|
||||
Bench caveat: the client must be given `--tokenizer
|
||||
/models/Qwen3.8-27B-Uncensored-MXFP4-awq` with `/models` mounted. Passing only
|
||||
`--model frank-medium` makes the client try to resolve that alias on
|
||||
HuggingFace and die before sending a request — a green-looking run that
|
||||
produced no numbers at all.
|
||||
|
||||
## Citations
|
||||
|
||||
- `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer
|
||||
refusal; `rocclr/device/rocm/rocdevice.cpp:186` — `checkAtomicSupport`.
|
||||
- Linux `drivers/pci/pci.c` — `pci_enable_atomic_ops_to_root()` and its
|
||||
`PCI_EXP_DEVCAP2_ATOMIC_ROUTE` requirements on upstream bridges.
|
||||
- Codeberg `StillDeadcode/vllm-radiance` issue #22 — an independent host with
|
||||
the same image and kernel 7.0.0-30 failing identically.
|
||||
- `ggz14/radiance-vllm-mxfp4` — the patch set and the reference benchmark table
|
||||
this page compares against.
|
||||
@@ -13,6 +13,12 @@ tags:
|
||||
timestamp: 2026-09-15
|
||||
---
|
||||
|
||||
> **PRIOR deployment (retired 2026-09-16).** This page documents the
|
||||
> llama.cpp era of `frank-medium`, which served the endpoint for part of
|
||||
> 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE
|
||||
> deployment is [[frank-medium-radiance-mxfp4]]; this page is kept as the
|
||||
> record of the runtime it replaced.
|
||||
|
||||
# frank-medium ROCmFPX Runbook
|
||||
|
||||
`frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15,
|
||||
|
||||
@@ -48,10 +48,17 @@ i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook:
|
||||
charlie12345/ROCmFPX llama.cpp serving kingjones777/
|
||||
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (MTP draft on) on Dankserver's
|
||||
2x Radeon AI PRO R9700 (since 2026-09-15; replaced frank-small)
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the
|
||||
vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
|
||||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon
|
||||
AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, with
|
||||
the PCIe-atomics evidence that rules out TP=2 on this host (since
|
||||
2026-09-15/16)
|
||||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||||
2026-09-15): charlie12345/ROCmFPX serving
|
||||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||||
runtime it replaced
|
||||
- [[freetoken-frank-small]] — RETIRED 2026-09-15 with the RTX 3090: the
|
||||
FreeToken v0.1.2 frank-small runbook (batsclamp Huihui-Qwen3.6-35B-A3B-
|
||||
abliterated-FP8), kept as the retirement record
|
||||
|
||||
@@ -114,10 +114,16 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook:
|
||||
charlie12345/ROCmFPX llama.cpp serving kingjones777/
|
||||
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver's 2x Radeon AI PRO
|
||||
R9700 (since 2026-09-15; replaced the FreeToken frank-small)
|
||||
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the
|
||||
vllm-radiance MXFP4 stack serving
|
||||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon
|
||||
AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, plus
|
||||
the PCIe-atomics evidence that rules out TP=2 on this host (since
|
||||
2026-09-15/16)
|
||||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
|
||||
charlie12345/ROCmFPX serving kingjones777/
|
||||
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
|
||||
replaced
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
|
||||
- [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
|
||||
standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
|
||||
|
||||
+114
-74
@@ -1,127 +1,167 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-medium quadlet on dankserver.
|
||||
# frank-medium became a rootless Quadlet container on 2026-09-15 when the
|
||||
# Seam: the frank-medium service on dankserver.
|
||||
# frank-medium became a rootless Quadlet deployment on 2026-09-15 when the
|
||||
# vllm-radiance MXFP4 stack replaced the charlie12345/ROCmFPX llama.cpp
|
||||
# runtime on the 2x Radeon AI PRO R9700. This seam pins the properties that
|
||||
# make it work and that no reviewer can see by reading the flags alone: GPU
|
||||
# device access on the fixed device indices, the upstream launcher's podman
|
||||
# posture, host networking with the fleet's client port and alias, the
|
||||
# measured KV pin for this exact hardware signature, boot persistence, and a
|
||||
# deploy handler that retires the old root unit without deleting anything.
|
||||
# runtime on the 2x Radeon AI PRO R9700. The intended shape is one
|
||||
# tensor-parallel process across both cards (frank-medium.container, kept as
|
||||
# the target); it cannot run on this host because RCCL's kernels require PCIe
|
||||
# atomics and the second card's chipset path advertises AtomicOpsCap: Routing-.
|
||||
# The deployed shape is therefore two single-GPU replicas behind a round-robin
|
||||
# proxy, which keeps the fleet contract of one :7770 endpoint. This seam pins
|
||||
# the properties that make that work: GPU access on the fixed device indices,
|
||||
# the upstream launcher's podman posture, host networking with the fleet's
|
||||
# client port and alias, per-replica resource shape, boot persistence, and a
|
||||
# deploy handler that moves the unusable TP=2 unit into the trash.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
UNIT_A="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-a.container"
|
||||
UNIT_B="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-b.container"
|
||||
UNIT_PROXY="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-proxy.container"
|
||||
UNIT_TP2="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
ENTRYPOINT="$REPO_ROOT/dankserver/frank-medium/radiance-entrypoint.sh"
|
||||
HEALTHCHECK="$REPO_ROOT/dankserver/frank-medium/frank-medium-healthcheck.py"
|
||||
PROXY="$REPO_ROOT/dankserver/frank-medium/proxy/roundrobin-proxy.py"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
|
||||
@test "frank-medium quadlet exists under dankserver/frank-medium/quadlet/" {
|
||||
[ -f "$UNIT" ]
|
||||
@test "the frank-medium units live under dankserver/frank-medium/quadlet/" {
|
||||
[ -f "$UNIT_A" ]
|
||||
[ -f "$UNIT_B" ]
|
||||
[ -f "$UNIT_PROXY" ]
|
||||
}
|
||||
|
||||
@test "frank-medium unit is boot-persistent (WantedBy=default.target)" {
|
||||
@test "the TP=2 unit is kept as the documented target, not deleted" {
|
||||
# It becomes runnable again only on a host whose second card sits on an
|
||||
# atomic-capable root port, so the artifact stays in the repo.
|
||||
[ -f "$UNIT_TP2" ]
|
||||
grep -q '^Environment=TP=2$' "$UNIT_TP2"
|
||||
}
|
||||
|
||||
@test "every frank-medium unit is boot-persistent (WantedBy=default.target)" {
|
||||
# Generated units cannot be enabled on the podman 5.x v1 quadlet
|
||||
# generator; [Install] plus linger is the entire boot-persistence story.
|
||||
grep -q '^WantedBy=default.target$' "$UNIT"
|
||||
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
|
||||
grep -q '^WantedBy=default.target$' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-medium unit is self-healing" {
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
@test "every frank-medium unit is self-healing" {
|
||||
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
|
||||
grep -q '^Restart=on-failure$' "$unit"
|
||||
grep -q '^RestartSec=' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-medium image is pinned to an exact version tag" {
|
||||
grep -Eq '^Image=localhost/radiance-vllm:[0-9]+\.[0-9]+\.[0-9]+$' "$UNIT"
|
||||
grep -q '^Pull=never$' "$UNIT"
|
||||
@test "frank-medium units pin the image tag and never pull at boot" {
|
||||
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
|
||||
grep -Eq '^Image=localhost/radiance-vllm:[0-9]+\.[0-9]+\.[0-9]+$' "$unit"
|
||||
grep -q '^Pull=never$' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-medium can reach the GPU (kfd + DRM render node) on the fixed indices" {
|
||||
grep -q '^AddDevice=/dev/kfd$' "$UNIT"
|
||||
grep -q '^AddDevice=/dev/dri$' "$UNIT"
|
||||
grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT"
|
||||
grep -q '^Environment=ROCR_VISIBLE_DEVICES=0,1$' "$UNIT"
|
||||
@test "the replicas each get exactly one GPU on the fixed device index" {
|
||||
grep -q '^AddDevice=/dev/kfd$' "$UNIT_A"
|
||||
grep -q '^AddDevice=/dev/dri$' "$UNIT_A"
|
||||
# One index per replica: ROCR filters first and HIP indexes into the
|
||||
# filtered list, so setting both to the same non-zero value leaves an
|
||||
# empty device set.
|
||||
grep -q '^Environment=HIP_VISIBLE_DEVICES=0$' "$UNIT_A"
|
||||
grep -q '^Environment=HIP_VISIBLE_DEVICES=1$' "$UNIT_B"
|
||||
! grep -q '^Environment=ROCR_VISIBLE_DEVICES=' "$UNIT_A"
|
||||
! grep -q '^Environment=ROCR_VISIBLE_DEVICES=' "$UNIT_B"
|
||||
}
|
||||
|
||||
@test "frank-medium runs with the upstream launcher's podman posture" {
|
||||
# serve-mxfp4.sh runs the image privileged, with host IPC, an unconfined
|
||||
# seccomp profile and SYS_PTRACE; host networking carries the port.
|
||||
grep -q -- '--privileged' "$UNIT"
|
||||
grep -q -- '--ipc=host' "$UNIT"
|
||||
grep -q -- '--security-opt seccomp=unconfined' "$UNIT"
|
||||
grep -q -- '--network=host' "$UNIT"
|
||||
@test "the replicas run with the upstream launcher's podman posture" {
|
||||
for unit in "$UNIT_A" "$UNIT_B"; do
|
||||
grep -q -- '--privileged' "$unit"
|
||||
grep -q -- '--ipc=host' "$unit"
|
||||
grep -q -- '--security-opt seccomp=unconfined' "$unit"
|
||||
grep -q -- '--network=host' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-medium keeps the fleet endpoint: host-network port 7770, alias frank-medium" {
|
||||
# vLLM binds 0.0.0.0:7770 on the host network; no PublishPort exists.
|
||||
[ "$(grep -c '^PublishPort=' "$UNIT")" -eq 0 ]
|
||||
grep -q '^Environment=PORT=7770$' "$UNIT"
|
||||
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT"
|
||||
@test "the replicas serve single-GPU shapes that fit one card" {
|
||||
for unit in "$UNIT_A" "$UNIT_B"; do
|
||||
grep -q '^Environment=TP=1$' "$unit"
|
||||
# 262144 does not fit one card (needs 9.61 GiB of KV, 5.14 available)
|
||||
# and 131072 does not either (5.58 needed vs 5.22 available); 114688
|
||||
# is the largest measured shape that boots.
|
||||
grep -q '^Environment=MAXLEN=114688$' "$unit"
|
||||
# KV_MEM=0 makes vLLM profile this shape: the 2x7551-32624 pin is a
|
||||
# TP=2/262144 measurement and invalid here.
|
||||
grep -q '^Environment=KV_MEM=0$' "$unit"
|
||||
grep -q '^Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq$' "$unit"
|
||||
grep -q '^Environment=SPEC_METHOD=dflash$' "$unit"
|
||||
grep -q '^Environment=SPEC=7$' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-medium pins the measured KV cache for its hardware signature" {
|
||||
# The upstream repo's shipped kv-profiles.tsv row for
|
||||
# 2x7551-32624 at maxseqs=8 chunk=8192 maxlen=262144 spec=dflash.
|
||||
# A pin is only valid at that batch shape; if the shape changes, this
|
||||
# value must be recalibrated or set to 0 (vLLM profiling).
|
||||
grep -q '^Environment=KV_MEM=18563072000$' "$UNIT"
|
||||
grep -q '^Environment=MAXSEQS=8$' "$UNIT"
|
||||
grep -q '^Environment=CHUNK=8192$' "$UNIT"
|
||||
grep -q '^Environment=MAXLEN=262144$' "$UNIT"
|
||||
grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT"
|
||||
grep -q '^Environment=SPEC=7$' "$UNIT"
|
||||
@test "the proxy keeps the fleet endpoint: host-network port 7770, alias frank-medium" {
|
||||
grep -q '^Environment=PORT=7770$' "$UNIT_PROXY"
|
||||
grep -q '^Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772$' "$UNIT_PROXY"
|
||||
grep -q -- '--network=host' "$UNIT_PROXY"
|
||||
[ "$(grep -c '^PublishPort=' "$UNIT_PROXY")" -eq 0 ]
|
||||
# The replicas answer as the same served model name, so a client cannot
|
||||
# tell which one served a request.
|
||||
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT_A"
|
||||
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT_B"
|
||||
grep -q '^Environment=PORT=7771$' "$UNIT_A"
|
||||
grep -q '^Environment=PORT=7772$' "$UNIT_B"
|
||||
}
|
||||
|
||||
@test "frank-medium mounts the pinned radiance clone and the patched libr4d" {
|
||||
# The patches are applied at every container start from /patches; the
|
||||
# image's shipped r4d.so predates the GDN overflow fix and NaNs this
|
||||
# model, so /r4d (b9e42ab-rx6) must be mounted and copied over it.
|
||||
grep -q 'Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z$' "$UNIT"
|
||||
grep -q 'Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z$' "$UNIT"
|
||||
@test "the proxy really is a round-robin over the replicas" {
|
||||
[ -f "$PROXY" ]
|
||||
python3 -m py_compile "$PROXY"
|
||||
grep -q 'itertools.cycle' "$PROXY"
|
||||
grep -q 'UPSTREAMS' "$PROXY"
|
||||
}
|
||||
|
||||
@test "frank-medium entrypoint is a syntax-clean fork of the upstream launcher" {
|
||||
[ -f "$ENTRYPOINT" ]
|
||||
bash -n "$ENTRYPOINT"
|
||||
# Pinned clone: the entrypoint must name the commit it was forked from so
|
||||
# a clone update is forced through a re-diff, not drifted into silently.
|
||||
grep -q '9735329348ae1c9319f01cbe5f7004ec5f4dba63' "$ENTRYPOINT"
|
||||
}
|
||||
|
||||
@test "frank-medium healthcheck probes the serving endpoint, not the process" {
|
||||
@test "frank-medium healthcheck probes the serving endpoint on its own port" {
|
||||
[ -f "$HEALTHCHECK" ]
|
||||
python3 -m py_compile "$HEALTHCHECK"
|
||||
grep -q 'frank-medium-healthcheck.py' "$UNIT"
|
||||
grep -q '^HealthStartPeriod=' "$UNIT"
|
||||
# One probe serves the replicas, the proxy and the TP=2 target because it
|
||||
# reads PORT instead of hardcoding 7770.
|
||||
grep -q 'os.environ.get("PORT"' "$HEALTHCHECK"
|
||||
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
|
||||
grep -q 'frank-medium-healthcheck.py' "$unit"
|
||||
grep -q '^HealthStartPeriod=' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "frank-medium unit carries no secret material" {
|
||||
! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$UNIT"
|
||||
@test "frank-medium units carry no secret material" {
|
||||
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
|
||||
! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$unit"
|
||||
done
|
||||
}
|
||||
|
||||
@test "the frank-medium checkpoint mount is the designated target" {
|
||||
grep -q '^Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq$' "$UNIT"
|
||||
grep -q '^Volume=/home/dank/models:/models$' "$UNIT"
|
||||
@test "the deploy handler installs the trio and retires the TP=2 unit" {
|
||||
grep -q 'install_quadlet frank-medium-a dankserver/frank-medium/quadlet frank-medium-a' "$REMOTE_DEPLOY"
|
||||
grep -q 'install_quadlet frank-medium-b dankserver/frank-medium/quadlet frank-medium-b' "$REMOTE_DEPLOY"
|
||||
grep -q 'install_quadlet frank-medium-proxy dankserver/frank-medium/quadlet frank-medium-proxy' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user restart frank-medium-a.service frank-medium-b.service frank-medium-proxy.service' "$REMOTE_DEPLOY"
|
||||
# The TP=2 unit cannot become healthy on this host; a leftover copy would
|
||||
# leave the generator restarting it forever. It moves to the trash.
|
||||
grep -q 'mv -- "$quadlet_unit_dir/frank-medium.container"' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?"?\$quadlet_unit_dir/frank-medium' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler never tries to enable a generated unit" {
|
||||
! grep -q 'systemctl --user enable frank-medium' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the old root unit without deleting it" {
|
||||
grep -q 'retire_root_llm_medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl stop frank-medium.service' "$REMOTE_DEPLOY"
|
||||
# Never `rm`: the retired ROCmFPX unit moves into the home trash so the
|
||||
# llama.cpp era stays recoverable.
|
||||
grep -q 'mv -- /etc/systemd/system/frank-medium.service' "$REMOTE_DEPLOY"
|
||||
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?/etc/systemd/system/frank-medium' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler never tries to enable a generated unit" {
|
||||
# `systemctl --user enable frank-medium.service` fails on the v1
|
||||
# generator ("transient or generated"); the deploy must install and
|
||||
# restart instead.
|
||||
! grep -q 'systemctl --user enable frank-medium.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user restart frank-medium.service' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler re-tags the registry image to the localhost alias" {
|
||||
grep -q 'podman tag docker.io/stilldeadcode/vllm-radiance:0.9.3 localhost/radiance-vllm:0.9.3' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user