diff --git a/AGENTS.md b/AGENTS.md index a8ef820..c686a8d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 | +| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind a round-robin proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 | | Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 | Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in @@ -248,23 +248,61 @@ Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`. Rollback: [[halogen-flash-server]] (quadlet archived in `~/.AGENTTRASH/20260915-ik-swap/` on dankclaw). -### frank-medium — ROCmFPX llama.cpp (Dankserver) +### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE) -charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for gfx1201 with the fork's -self-contained ROCm 7.14 toolchain (RPATH-bundled — host ROCm packages never -participate). Serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` -(ftype 115; stock llama.cpp cannot load it) with `--split-mode layer` across -both R9700s, 131072 ctx / 1 slot, f16 KV, flash-attn on, MTP draft head -pinned to ROCm1, reasoning on. Env `scripts/frank-medium.env`, serve -`scripts/serve-frank-medium.sh`. +`stilldeadcode/vllm-radiance:0.9.3` plus the `ggz14/radiance-vllm-mxfp4` +patches (pinned clone at `/home/dank/GitPackages/radiance-vllm-mxfp4`; its +patches are applied at every container start), serving +`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` with the +`Qwen3.8-27B-DFlash2-FP8` block-diffusion drafter (`SPEC=7`). -Runbook: `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md`. +The deployment is **two single-GPU replicas behind a round-robin proxy** — +`frank-medium-a` (:7771, R9700 #1), `frank-medium-b` (:7772, R9700 #2) and +`frank-medium-proxy` (:7770, the fleet endpoint), three rootless Quadlets +under `dankserver/frank-medium/quadlet/`. Each replica is TP=1 at +114688 context / 4 sequences with vLLM-profiled KV (`KV_MEM=0`). + +**TP=2 across both cards cannot run on this host.** RCCL's collective kernels +carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on +a device without PCIe atomics; the second R9700 sits behind the chipset +bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a +read-only capability, so no software can enable atomics on that path. Not the +kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL +build (a from-source rebuild fails identically). A slot on a CPU root port — +there is none free on this AM5 platform for a second x16 card — or a platform +with more CPU lanes restores TP=2. `frank-medium.container` stays in the repo +as the TP=2 target; the deploy handler does not install it. + +Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`. +Prior runtime: [[frank-medium-rocmfpx]] (llama.cpp era, same day). ## Key Lessons Learned - Durable operational lessons from the fleet's history. Full detail in the linked runbooks. +**PCIe / multi-GPU** + +- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's + kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch + them when the device has no atomics, and the failure surfaces as + `hipErrorIllegalState` ("the operation cannot be performed in the present + state") from `hipLaunchKernel`, not as anything that names atomics. The + value is consumed at driver probe time, so writing the AtomicOp Requester + bit with `setpci` at runtime changes nothing. Check the *chain*: + `lspci -vv -s ` and every upstream bridge must show + `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the + endpoint's Device Control 2 (`setpci -s CAP_EXP+28.w`) must have bit + 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) + and the rest behind the chipset (usually not) — that asymmetry is what + blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL + short-circuits at `world_size=1`. +- **When a GPU collective dies, read the runtime, not the framework.** The + message that identified the cause came from `AMD_LOG_LEVEL=3` + (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch + failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and + prints the failing HIP call with its return code. + + **ROCm / GPU** - **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor @@ -359,7 +397,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | -| frank-medium (ROCmFPX, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` | +| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | diff --git a/dankclaw/hermes/config.yaml.example b/dankclaw/hermes/config.yaml.example index e7c2119..afaa784 100644 --- a/dankclaw/hermes/config.yaml.example +++ b/dankclaw/hermes/config.yaml.example @@ -18,13 +18,13 @@ providers: discover_models: false models: frank-large: - context_length: 262144 + context_length: 114688 dankserver-small: api: http://192.168.0.235:7770/v1 discover_models: false models: frank-medium: - context_length: 163840 + context_length: 114688 # Auxiliary routing from the live configuration. Named providers route via # their `api:` URL; `custom` entries route by their explicit `base_url`. diff --git a/dankserver/frank-medium/frank-medium-healthcheck.py b/dankserver/frank-medium/frank-medium-healthcheck.py index fb0455a..a48fa55 100755 --- a/dankserver/frank-medium/frank-medium-healthcheck.py +++ b/dankserver/frank-medium/frank-medium-healthcheck.py @@ -1,17 +1,21 @@ #!/usr/bin/env python3 -"""frank-medium container healthcheck: vLLM /health must answer 200. +"""frank-medium container healthcheck: the port's /health must answer 200. Runs INSIDE the container. The vLLM /health endpoint only answers once the engine is up (weights loaded, cudagraphs captured), so a healthy container is -a serving container, not merely a live process. The HealthCmd in the quadlet -splits on spaces into an argv array (podman 5.x v1 generator), which is why -this lives in a file rather than an inline python one-liner. +a serving container, not merely a live process. PORT selects the endpoint so +the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can +share this probe. The HealthCmd in the quadlet splits on spaces into an argv +array (podman 5.x v1 generator), which is why this lives in a file rather +than an inline python one-liner. """ +import os import sys import urllib.request +port = os.environ.get("PORT", "7770") try: - with urllib.request.urlopen("http://127.0.0.1:7770/health", timeout=5) as r: + with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=5) as r: sys.exit(0 if r.status == 200 else 1) except Exception: sys.exit(1) diff --git a/dankserver/frank-medium/proxy/roundrobin-proxy.py b/dankserver/frank-medium/proxy/roundrobin-proxy.py new file mode 100755 index 0000000..a93706c --- /dev/null +++ b/dankserver/frank-medium/proxy/roundrobin-proxy.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Round-robin proxy in front of the frank-medium TP=1 replicas. + +Why this exists: TP=2 (one vLLM process spanning both R9700s) needs RCCL +collectives, and RCCL's kernels require PCIe atomics on every participating +GPU. On this host the second R9700 sits behind chipset bridges that advertise +`AtomicOpsCap: Routing-` (see the runbook), so no collective can launch on it +and TP=2 is impossible. Two single-GPU replicas use both cards instead; this +proxy keeps the fleet contract of one endpoint on :7770. + +The vLLM OpenAI API is stateless per request, so round-robin is sound. /health +returns 200 while at least one replica is up and names the ones that are not. +""" + +from __future__ import annotations + +import asyncio +import itertools +import os +from aiohttp import ClientSession, ClientTimeout, web + +UPSTREAMS = [u.strip() for u in os.environ.get( + "UPSTREAMS", "http://127.0.0.1:7771,http://127.0.0.1:7772").split(",") if u.strip()] +PORT = int(os.environ.get("PORT", "7770")) +# Hop-by-hop headers must not be forwarded (RFC 9110 §7.6.1) — forwarding +# transfer-encoding or content-length of a rewritten body breaks clients. +HOP_BY_HOP = { + "host", "content-length", "transfer-encoding", "connection", + "keep-alive", "proxy-authenticate", "proxy-authorization", "te", + "trailers", "upgrade", +} + +_cycle = itertools.cycle(range(len(UPSTREAMS))) + + +async def health(session: ClientSession) -> web.Response: + results = await asyncio.gather( + *(session.get(u + "/health", timeout=ClientTimeout(total=5)) for u in UPSTREAMS), + return_exceptions=True, + ) + lines, up = [], 0 + for url, res in zip(UPSTREAMS, results): + if isinstance(res, Exception) or res.status != 200: + lines.append(f"{url} DOWN") + else: + up += 1 + lines.append(f"{url} UP") + body = "\n".join(lines) + "\n" + return web.Response(status=200 if up else 503, text=body) + + +async def proxy(request: web.Request) -> web.StreamResponse: + session: ClientSession = request.app["session"] + if request.path == "/health": + return await health(session) + + target = UPSTREAMS[next(_cycle)] + request.path_qs + headers = {k: v for k, v in request.headers.items() if k.lower() not in HOP_BY_HOP} + body = await request.read() + + try: + upstream = await session.request( + request.method, target, headers=headers, data=body or None, + timeout=ClientTimeout(total=None, sock_connect=10, sock_read=3600), + ) + except Exception as exc: # upstream refused/unreachable + return web.Response(status=502, text=f"proxy: {target} unreachable: {exc}\n") + + out = web.StreamResponse(status=upstream.status, reason=upstream.reason) + for k, v in upstream.headers.items(): + if k.lower() not in HOP_BY_HOP: + out.headers[k] = v + await out.prepare(request) + try: + async for chunk in upstream.content.iter_any(): + await out.write(chunk) + finally: + upstream.release() + await out.write_eof() + return out + + +async def main() -> None: + app = web.Application(client_max_size=1024 ** 3) + app["session"] = ClientSession() + app.router.add_route("*", "/{tail:.*}", proxy) + runner = web.AppRunner(app) + await runner.setup() + site = web.TCPSite(runner, "0.0.0.0", PORT) + await site.start() + print(f"proxy: :{PORT} -> {', '.join(UPSTREAMS)}", flush=True) + await asyncio.Event().wait() + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/dankserver/frank-medium/quadlet/frank-medium-a.container b/dankserver/frank-medium/quadlet/frank-medium-a.container new file mode 100644 index 0000000..4f121cc --- /dev/null +++ b/dankserver/frank-medium/quadlet/frank-medium-a.container @@ -0,0 +1,98 @@ +# Quadlet unit: frank-medium replica A (rootless, boot-persistent). +# +# Install path on dankserver: ~/.config/containers/systemd/frank-medium-a.container +# Unit name: frank-medium-a.service (container name: frank-medium-a) +# +# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one +# tensor-parallel process across both R9700s (see frank-medium.container, kept +# as the target). That shape cannot run on this host: RCCL's collective kernels +# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to +# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled, +# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The +# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise +# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable +# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root() +# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS. +# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have +# atomics. Result: every collective that touches GPU 2 dies at launch. +# Two independent single-GPU replicas use both cards instead; the proxy unit +# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook. +# +# SHAPE. vLLM binds 0.0.0.0:7771 on the host network directly; no PublishPort. +# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the +# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and +# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile +# this shape instead of using the 2x7551-32624 pin, which is only valid at +# TP=2/262144. +# +# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=0 (the first R9700). The +# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin. +# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters +# first and HIP indexes into the filtered list, so a single-valued pair +# (ROCR=1,HIP=1) leaves an empty device set. +# +# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the +# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident). +[Unit] +Description=frank-medium replica A — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #1 +After=network-online.target + +[Container] +ContainerName=frank-medium-a +Image=localhost/radiance-vllm:0.9.3 +AddDevice=/dev/kfd +AddDevice=/dev/dri +GroupAdd=keep-groups +# The upstream launcher's exact podman posture (rootless). +PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host +Environment=HIP_VISIBLE_DEVICES=0 +Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq +Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8 +Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja +Environment=SERVED_NAMES=frank-medium +Environment=PORT=7771 +Environment=TP=1 +Environment=MAXSEQS=4 +Environment=CHUNK=8192 +Environment=MAXLEN=114688 +Environment=GPU_UTIL=0.98 +Environment=KV_MEM=0 +Environment=SPEC_METHOD=dflash +Environment=SPEC=7 +# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards). +Volume=/home/dank/models:/models +# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint. +Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface +# Compile cache. Shared with the other replica and with the TP=2 target: the +# entries are content-addressed, so shapes simply add subdirectories. +Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache +# The pinned radiance clone: patches applied at every container start. +Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z +# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's +# shipped r4d.so predates the GDN overflow fix and NaNs this model. +Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z +# Entrypoint + healthcheck from this checkout, mounted read-only. +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro +# The entrypoint must be the container command; the image's own entrypoint +# (which the patched script execs at the end) must not run unpatched. +PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh +# 200 only once the engine serves; during first-start compilation the +# container is legitimately unhealthy. HealthCmd is split on spaces into an +# argv array by the podman 5.x v1 generator, hence the file-based probe. +HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py +HealthInterval=60s +HealthTimeout=30s +HealthRetries=3 +HealthStartPeriod=1800s +Pull=never + +[Service] +# First start compiles Triton/inductor kernels (several minutes) before the +# engine loads ~20 GiB of weights. +TimeoutStartSec=1800 +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=default.target diff --git a/dankserver/frank-medium/quadlet/frank-medium-b.container b/dankserver/frank-medium/quadlet/frank-medium-b.container new file mode 100644 index 0000000..4839383 --- /dev/null +++ b/dankserver/frank-medium/quadlet/frank-medium-b.container @@ -0,0 +1,98 @@ +# Quadlet unit: frank-medium replica B (rootless, boot-persistent). +# +# Install path on dankserver: ~/.config/containers/systemd/frank-medium-b.container +# Unit name: frank-medium-b.service (container name: frank-medium-a) +# +# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one +# tensor-parallel process across both R9700s (see frank-medium.container, kept +# as the target). That shape cannot run on this host: RCCL's collective kernels +# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to +# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled, +# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The +# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise +# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable +# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root() +# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS. +# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have +# atomics. Result: every collective that touches GPU 2 dies at launch. +# Two independent single-GPU replicas use both cards instead; the proxy unit +# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook. +# +# SHAPE. vLLM binds 0.0.0.0:7772 on the host network directly; no PublishPort. +# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the +# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and +# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile +# this shape instead of using the 2x7551-32624 pin, which is only valid at +# TP=2/262144. +# +# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=1 (the second R9700). The +# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin. +# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters +# first and HIP indexes into the filtered list, so a single-valued pair +# (ROCR=1,HIP=1) leaves an empty device set. +# +# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the +# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident). +[Unit] +Description=frank-medium replica B — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #2 +After=network-online.target + +[Container] +ContainerName=frank-medium-b +Image=localhost/radiance-vllm:0.9.3 +AddDevice=/dev/kfd +AddDevice=/dev/dri +GroupAdd=keep-groups +# The upstream launcher's exact podman posture (rootless). +PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host +Environment=HIP_VISIBLE_DEVICES=1 +Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq +Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8 +Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja +Environment=SERVED_NAMES=frank-medium +Environment=PORT=7772 +Environment=TP=1 +Environment=MAXSEQS=4 +Environment=CHUNK=8192 +Environment=MAXLEN=114688 +Environment=GPU_UTIL=0.98 +Environment=KV_MEM=0 +Environment=SPEC_METHOD=dflash +Environment=SPEC=7 +# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards). +Volume=/home/dank/models:/models +# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint. +Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface +# Compile cache. Shared with the other replica and with the TP=2 target: the +# entries are content-addressed, so shapes simply add subdirectories. +Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache +# The pinned radiance clone: patches applied at every container start. +Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z +# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's +# shipped r4d.so predates the GDN overflow fix and NaNs this model. +Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z +# Entrypoint + healthcheck from this checkout, mounted read-only. +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro +# The entrypoint must be the container command; the image's own entrypoint +# (which the patched script execs at the end) must not run unpatched. +PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh +# 200 only once the engine serves; during first-start compilation the +# container is legitimately unhealthy. HealthCmd is split on spaces into an +# argv array by the podman 5.x v1 generator, hence the file-based probe. +HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py +HealthInterval=60s +HealthTimeout=30s +HealthRetries=3 +HealthStartPeriod=1800s +Pull=never + +[Service] +# First start compiles Triton/inductor kernels (several minutes) before the +# engine loads ~20 GiB of weights. +TimeoutStartSec=1800 +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=default.target diff --git a/dankserver/frank-medium/quadlet/frank-medium-proxy.container b/dankserver/frank-medium/quadlet/frank-medium-proxy.container new file mode 100644 index 0000000..412310e --- /dev/null +++ b/dankserver/frank-medium/quadlet/frank-medium-proxy.container @@ -0,0 +1,52 @@ +# Quadlet unit: frank-medium proxy (rootless, boot-persistent). +# +# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container +# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy) +# +# WHAT IT IS. A round-robin reverse proxy in front of the two single-GPU +# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves +# the fleet contract: one endpoint on host port 7770 answering as the served +# model name frank-medium, OpenAI /v1, streaming included. +# +# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible +# on this host (see frank-medium-a.container and the runbook: RCCL's kernels +# need PCIe atomics, and the second card's chipset path advertises +# AtomicOpsCap: Routing-). Two replicas are how both cards get used. +# +# ROUTING. vLLM's OpenAI API is stateless per request, so round-robin is +# sound; there is no session affinity to preserve. /health answers 200 while +# at least one replica is up and names the ones that are down. The proxy runs +# on host networking so it can bind 7770 and reach the replicas on loopback. +# +# The script is a few dozen lines of aiohttp, mounted read-only out of +# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch). +# It reuses the same image as the replicas: aiohttp is already there and no +# extra image has to be pulled on this host. +[Unit] +Description=frank-medium proxy — round-robin in front of the two single-GPU replicas +After=network-online.target +Wants=frank-medium-a.service frank-medium-b.service +After=frank-medium-a.service frank-medium-b.service + +[Container] +ContainerName=frank-medium-proxy +Image=localhost/radiance-vllm:0.9.3 +PodmanArgs=--network=host +Environment=PORT=7770 +Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772 +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/roundrobin-proxy.py:/usr/local/bin/roundrobin-proxy.py:ro +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro +PodmanArgs=--entrypoint /usr/local/bin/roundrobin-proxy.py +HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py +HealthInterval=60s +HealthTimeout=30s +HealthRetries=3 +HealthStartPeriod=60s +Pull=never + +[Service] +Restart=on-failure +RestartSec=10 + +[Install] +WantedBy=default.target diff --git a/dankserver/frank-medium/quadlet/frank-medium.container b/dankserver/frank-medium/quadlet/frank-medium.container index 0e9e6a4..c129bd7 100644 --- a/dankserver/frank-medium/quadlet/frank-medium.container +++ b/dankserver/frank-medium/quadlet/frank-medium.container @@ -19,6 +19,15 @@ # what this unit does too — no PublishPort. The fleet contract is unchanged: # :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1. # +# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel +# process across both R9700s. It cannot launch a collective here — RCCL's +# kernels need PCIe atomics and the second card sits behind chipset bridges +# advertising `AtomicOpsCap: Routing-` — so the active deployment is the +# two-replica trio in this directory (frank-medium-a/b/proxy) and +# deploy-remote.sh deliberately does not install this unit. Keep it for a +# host whose second card is on an atomic-capable root port; the full evidence +# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md. +# # DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device # indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2 # and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192 diff --git a/dankserver/frank-medium/radiance-entrypoint.sh b/dankserver/frank-medium/radiance-entrypoint.sh index dd34cd5..cb12dab 100755 --- a/dankserver/frank-medium/radiance-entrypoint.sh +++ b/dankserver/frank-medium/radiance-entrypoint.sh @@ -27,7 +27,8 @@ set -euo pipefail : "${PORT:=7770}" : "${SERVED_NAMES:=frank-medium}" : "${MAXLEN:=262144}" -: "${MAXSEQS:=8}" +: "${KV_MEM:=18563072000}" +if [ "$KV_MEM" = "0" ]; then KV_MEM=""; fi # 0 = let vLLM profile (the launcher's rule) : "${CHUNK:=8192}" : "${GPU_UTIL:=0.98}" : "${TP:=2}" diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 556e9f0..b9f93c6 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -991,9 +991,30 @@ deploy_llm_medium() { # The compile cache must exist before the quadlet mounts resolve; the # one-time setup (setup-mxfp4.sh / PREPARE_ONLY) normally creates it. mkdir -p /home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq/{vllm,inductor,triton,aiter} - install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium - systemctl --user restart frank-medium.service - wait_for_health frank-medium http://127.0.0.1:7770/health 1200 + # Three units, not one. TP=2 across both R9700s cannot launch a collective + # on this host — RCCL's kernels need PCIe atomics and the second card's + # chipset path advertises AtomicOpsCap: Routing- — so the endpoint is two + # single-GPU replicas behind a round-robin proxy. frank-medium.container + # stays in the repo as the TP=2 target for a host whose second card sits + # on a CPU root port; it is deliberately NOT installed here. + install_quadlet frank-medium-a dankserver/frank-medium/quadlet frank-medium-a + install_quadlet frank-medium-b dankserver/frank-medium/quadlet frank-medium-b + install_quadlet frank-medium-proxy dankserver/frank-medium/quadlet frank-medium-proxy + # Drop a stale TP=2 unit from an earlier deploy of this service, or the + # generator keeps trying to start a container that can never become + # healthy. Never `rm`: it moves into the home trash. + if [[ -e "$quadlet_unit_dir/frank-medium.container" ]]; then + local trash="$HOME/.AGENTTRASH/20260916-frank-medium-tp2" + mkdir -p "$trash" + systemctl --user stop frank-medium.service 2>/dev/null || true + mv -- "$quadlet_unit_dir/frank-medium.container" "$trash/frank-medium.container" + systemctl --user daemon-reload + echo "retired the TP=2 quadlet -> $trash (see the runbook for why)" + fi + # The proxy waits on the replicas, but a restart of all three is what a + # deploy does: the replicas reload the model, the proxy reconnects. + systemctl --user restart frank-medium-a.service frank-medium-b.service frank-medium-proxy.service + wait_for_health frank-medium http://127.0.0.1:7770/health 1800 } wait_for_health() { diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md new file mode 100644 index 0000000..802481a --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -0,0 +1,236 @@ +--- +type: configuration +title: frank-medium vllm-radiance MXFP4 Runbook +description: vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201), as two single-GPU replicas behind a round-robin proxy — and the PCIe-atomics reason TP=2 cannot run on this host. +tags: + - frank-medium + - dankserver + - vllm + - vllm-radiance + - mxfp4 + - gfx1201 + - r9700 + - rccl + - pcie-atomics + - quadlet +timestamp: 2026-09-16 +--- + +# frank-medium vllm-radiance MXFP4 Runbook + +The ACTIVE frank-medium deployment (since 2026-09-15/16): the +`stilldeadcode/vllm-radiance:0.9.3` image plus the `ggz14/radiance-vllm-mxfp4` +patches, serving `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` on the two +Radeon AI PRO R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX +llama.cpp deployment ([[frank-medium-rocmfpx]]). + +**Deployed shape: two single-GPU replicas behind a round-robin proxy.** The +intended shape — one tensor-parallel process across both cards (TP=2) — cannot +launch a collective on this host. The reason is a hardware property of the +motherboard's PCIe topology, not the image, the RCCL build, or the kernel +version. The evidence is in [Why TP=2 cannot run here](#why-tp2-cannot-run-here). + +## Architecture + +```mermaid +graph LR + C[clients: Hermes, harness] --> P["frank-medium-proxy
:7770 round-robin"] + P --> A["frank-medium-a
R9700 #1, TP=1, :7771"] + P --> B["frank-medium-b
R9700 #2, TP=1, :7772"] +``` + +Three rootless Quadlet units, all `WantedBy=default.target` + `Restart=on-failure`: + +| Unit | Container | GPU | Port | Role | +|---|---|---|---|---| +| `frank-medium-a` | `frank-medium-a` | `HIP_VISIBLE_DEVICES=0` | 7771 | replica | +| `frank-medium-b` | `frank-medium-b` | `HIP_VISIBLE_DEVICES=1` | 7772 | replica | +| `frank-medium-proxy` | `frank-medium-proxy` | — | 7770 | round-robin | + +The fleet contract is unchanged: one endpoint on :7770, served model name +`frank-medium`, OpenAI `/v1`, streaming included. The vLLM OpenAI API is +stateless per request, so round-robin needs no session affinity. + +Only `HIP_VISIBLE_DEVICES` is set per replica, never a matching +`ROCR_VISIBLE_DEVICES`: ROCR filters first and HIP then indexes into the +filtered list, so the single-valued pair (`ROCR=1,HIP=1`) leaves an empty +device set and the engine dies with `hipErrorNoDevice`. + +`frank-medium.container` (TP=2, `KV_MEM=18563072000`, `MAXLEN=262144`) stays in +the repo as the target shape for a host whose second card sits on an +atomic-capable root port. `deploy-remote.sh` deliberately does not install it +and moves any stale copy into `~/.AGENTTRASH/`, because a TP=2 unit on this +host would restart forever without ever becoming healthy. + +## Why TP=2 cannot run here + +### Symptom + +Any two-rank collective dies at launch: + +``` +[FATAL ERROR]: HIP failure: 'the operation cannot be performed in the present state' + group.cc:337 -> group.cc:656 -> group.cc:801 -> enqueue.cc:3111 +``` + +With the runtime's own API log (`AMD_LOG_LEVEL=3`) the failing call is explicit: + +``` +ShaderName : ncclDevKernel_Generic_4(ncclDevKernelArgsStorage<4096ul>) +:1:rocvirtual.cpp :4208: Pcie atomics not enabled, hostcall not supported +:1:rocvirtual.cpp :4636: AQL dispatch failed! +:3:hip_module.cpp :893 : hipLaunchKernel: Returned hipErrorIllegalState +``` + +ROCclr refuses to hand the kernel a hostcall buffer when the device has no +PCIe atomics (`rocclr/device/rocm/rocvirtual.cpp:3963`), and the launch aborts. +`hipErrorIllegalState` is that refusal, not a driver fault. + +### Chain of causation + +1. RCCL's collective kernels carry a hidden **hostcall buffer** parameter. +2. ROCr passes that buffer only when `pcie_atomics_` is true for the device; + otherwise it returns "not supported" and the dispatch fails. +3. `pcie_atomics_` is set once, at device init, from the HSA link info between + the CPU agent and the GPU (`rocclr/device/rocm/rocdevice.cpp:186`, + `Device::checkAtomicSupport`). It is not re-read per process. +4. The KFD driver populates that link info from + `pci_enable_atomic_ops_to_root()`, which — per the PCIe spec and the + kernel's implementation — requires **every upstream bridge** between the + device and its root port to advertise `AtomicOp Routing` capability + (`drivers/pci/pci.c`). + +### The hardware difference between the two cards + +| Card | PCIe chain (endpoint → root port) | `AtomicOpsCap` on every hop | AtomicOpsCtl | +|---|---|---|---| +| R9700 #1 (`03:00.0`) | `02:00.0` → `01:00.0` → `00:01.1` | Routing+ / Routing+ / Routing+ | `ReqEn+` (atomics ON) | +| R9700 #2 (`0b:00.0`) | `0a:00.0` → `09:00.0` → `06:09.0` → `05:00.0` → `00:02.1` | Routing+ / Routing+ / **Routing−** / **Routing−** / Routing+ | `ReqEn−` (atomics OFF) | + +Card #1 is on a CPU root port; card #2 is behind the chipset bridges +`05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-`. That capability +bit is **read-only** — no software can enable atomic routing through those +bridges — so the driver's atomic enablement call fails for card #2 and the KFD +marks the CPU↔GPU2 link `NO_ATOMICS`. + +Writing the AtomicOp Requester bit into the endpoint's Device Control 2 by hand +(`setpci -s 0b:00.0 CAP_EXP+28.w`) does not help: the driver reads the chain at +probe time, and ROCr consumes the driver's cached answer. Only a re-probe (a +boot with the write already in place) would change it — and even then the two +`Routing-` bridges would still make `pci_enable_atomic_ops_to_root()` fail. + +### What was ruled out + +| Suspect | Result | +|---|---| +| Kernel version | 7.0.0-28, -30 and -31 fail identically (full `apt full-upgrade` to the newest available kernel changed nothing) | +| IOMMU | identical before and after `iommu=pt` | +| P2P hardware | fine: 45.3 GB/s bidirectional copy between the two cards; both cards pass `rocm-bandwidth-test` | +| The shipped librccl | a from-source rebuild of `ROCm/rccl` develop (with local fixes for that branch's dangling references) reproduces the same failure at the same call path — the library is not at fault | +| vLLM/pynccl | a bare two-rank `torch.distributed` allreduce with no vLLM involved fails the same way; single-rank collectives succeed on both cards (NCCL short-circuits at `world_size=1`, which is why a one-card test proves nothing) | +| Transport knobs | `NCCL_PROTO=Simple`, `NCCL_P2P_DISABLE`, `NCCL_SHM_DISABLE`, `NCCL_CUMEM_ENABLE=0`, MSCCL off, channel counts, launch modes, `HSA_ENABLE_SDMA=0` — all identical | +| Image/support matrix | the same image on a different host with the same kernel (Proxmox VM, Codeberg issue #22) fails byte-identically | + +### What would fix it + +- **Move card #2 to a slot whose path is atomic-capable** (a CPU root port, as + card #1 has). This is the only software-free fix and the one that restores + true TP=2. On this AM5 platform the CPU provides the x16 slot plus x4 lanes, + so a second x16-class card necessarily lands on the chipset path — a + platform with more CPU lanes (Threadripper/EPYC) is the structural answer. +- Until then, **two replicas** are how both cards serve. + +## Deployed configuration + +Both replicas, uniform except GPU index and port: + +| Knob | Value | Why | +|---|---|---| +| `TP` | 1 | one card per replica | +| `MAXLEN` | 114688 | the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.14 available, and 131072 needs 5.58 against 5.22 | +| `KV_MEM` | 0 | vLLM profiles this shape; the upstream pin (2x7551-32624) is a TP=2/262144 measurement and is invalid here | +| `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency | +| `CHUNK` | 8192 | prefill chunk (upstream default) | +| `GPU_UTIL` | 0.98 | upstream default; the card is dedicated | +| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter, upstream's measured production depth | +| `SNAP` / `DRAFTER` | `/models/Qwen3.8-27B-Uncensored-MXFP4-awq` / `/models/Qwen3.8-27B-DFlash2-FP8` | the served checkpoint and its drafter | + +Mounts are the TP=2 unit's minus nothing: `/models`, the HF cache, the shared +compile cache, `/patches` (the pinned radiance clone; its patches are applied +at every container start), `/r4d` (the patched libr4d — the image's shipped +`r4d.so` predates the gated-delta-net overflow fix and NaNs this model), and +the entrypoint/healthcheck/proxy scripts mounted read-only out of +`/home/dank/serverAppConfigs`. + +The healthcheck reads `PORT`, so one probe serves the replicas (:7771/:7772), +the proxy (:7770) and the TP=2 target. + +## Deploy and verify + +``` +./tests/run.sh # unit/config seams +./scripts/deploy.sh dankserver --service frank-medium +``` + +The handler re-tags `docker.io/stilldeadcode/vllm-radiance:0.9.3` as +`localhost/radiance-vllm:0.9.3` (this host's quadlet path cannot resolve +registry-prefixed refs from local storage at create time), installs the three +units, retires any stale TP=2 unit into the trash, restarts the trio and waits +for `http://127.0.0.1:7770/health`. + +Verification that means something: + +``` +curl -s http://192.168.0.235:7770/health # proxy: names any replica that is down +curl -s http://192.168.0.235:7770/v1/models # served name must be frank-medium +# and a real completion through the proxy: +curl -s http://192.168.0.235:7770/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"frank-medium","messages":[{"role":"user","content":"Say OK"}],"max_tokens":400}' +``` + +The same call works against either replica directly (`:7771`, `:7772`) — when +one of them is down the other still serves, which is why the proxy reports the +individual replica states instead of a single boolean. + +## Benchmarks + +`vllm bench serve` against the proxy (both cards serving), random dataset, +`--ignore-eos`, `2026-09-16`: + +| Case | Output t/s | TTFT p50 | TPOT p50 | Acceptance | +|---|--:|--:|--:|--:| +| conc 1, in 1k / out 512 | 42.8 | 382 ms | 23.6 ms | 43.1 % | +| conc 4, in 1k / out 512 | 147.6 | 471 ms | 25.3 ms | 34.2 % | +| conc 8, in 1k / out 512 | 230.8 | 913 ms | 28.8 ms | 39.0 % | +| conc 1, in 32k / out 32 | 2.5 | 11837 ms | 29.6 ms | 25.0 % | + +The 32k row is a prefill measurement: 32768 prompt tokens in 11.8 s is +**≈2.8k tok/s of prefill on a single card**. The replicas split the fleet's +traffic, they do not pool capacity for one request, against the upstream +reference's 4585 t/s at 32k on TP=2 — one card doing the whole prompt lands +where you would expect. + +Read these against the upstream repo's own reference table (2x R9700, TP=2, +BetterBench corpus): step time 22.8-23.2 ms per update, combined decode +224 t/s, prefill 4585 t/s at 32k. Step time (TPOT) is the comparable number — +tokens/s swings with draft acceptance, and synthetic random prompts accept far +worse than the code/prose mix upstream measures (43 % here against 5.8 +tokens/update there), so a lower token rate at the same step time is the +dataset, not the deployment. + +Bench caveat: the client must be given `--tokenizer +/models/Qwen3.8-27B-Uncensored-MXFP4-awq` with `/models` mounted. Passing only +`--model frank-medium` makes the client try to resolve that alias on +HuggingFace and die before sending a request — a green-looking run that +produced no numbers at all. + +## Citations + +- `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer + refusal; `rocclr/device/rocm/rocdevice.cpp:186` — `checkAtomicSupport`. +- Linux `drivers/pci/pci.c` — `pci_enable_atomic_ops_to_root()` and its + `PCI_EXP_DEVCAP2_ATOMIC_ROUTE` requirements on upstream bridges. +- Codeberg `StillDeadcode/vllm-radiance` issue #22 — an independent host with + the same image and kernel 7.0.0-30 failing identically. +- `ggz14/radiance-vllm-mxfp4` — the patch set and the reference benchmark table + this page compares against. diff --git a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md index 8f1e0b6..36b1bda 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md +++ b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md @@ -13,6 +13,12 @@ tags: timestamp: 2026-09-15 --- +> **PRIOR deployment (retired 2026-09-16).** This page documents the +> llama.cpp era of `frank-medium`, which served the endpoint for part of +> 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE +> deployment is [[frank-medium-radiance-mxfp4]]; this page is kept as the +> record of the runtime it replaced. + # frank-medium ROCmFPX Runbook `frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15, diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 8de0379..9491af3 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -48,10 +48,17 @@ i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision] ## Active Deployment -- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook: - charlie12345/ROCmFPX llama.cpp serving kingjones777/ - Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (MTP draft on) on Dankserver's - 2x Radeon AI PRO R9700 (since 2026-09-15; replaced frank-small) +- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the + vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + + ggz14/radiance-vllm-mxfp4 patches) serving + just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon + AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, with + the PCIe-atomics evidence that rules out TP=2 on this host (since + 2026-09-15/16) +- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era, + 2026-09-15): charlie12345/ROCmFPX serving + kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the + runtime it replaced - [[freetoken-frank-small]] — RETIRED 2026-09-15 with the RTX 3090: the FreeToken v0.1.2 frank-small runbook (batsclamp Huihui-Qwen3.6-35B-A3B- abliterated-FP8), kept as the retirement record diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 3a1625b..91958a7 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -114,10 +114,16 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook: - charlie12345/ROCmFPX llama.cpp serving kingjones777/ - Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver's 2x Radeon AI PRO - R9700 (since 2026-09-15; replaced the FreeToken frank-small) +- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the + vllm-radiance MXFP4 stack serving + just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon + AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, plus + the PCIe-atomics evidence that rules out TP=2 on this host (since + 2026-09-15/16) +- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era): + charlie12345/ROCmFPX serving kingjones777/ + Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it + replaced - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090 - [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical) diff --git a/tests/frank_medium_quadlet.bats b/tests/frank_medium_quadlet.bats index d773a27..067e8ac 100755 --- a/tests/frank_medium_quadlet.bats +++ b/tests/frank_medium_quadlet.bats @@ -1,127 +1,167 @@ #!/usr/bin/env bats -# Seam: the frank-medium quadlet on dankserver. -# frank-medium became a rootless Quadlet container on 2026-09-15 when the +# Seam: the frank-medium service on dankserver. +# frank-medium became a rootless Quadlet deployment on 2026-09-15 when the # vllm-radiance MXFP4 stack replaced the charlie12345/ROCmFPX llama.cpp -# runtime on the 2x Radeon AI PRO R9700. This seam pins the properties that -# make it work and that no reviewer can see by reading the flags alone: GPU -# device access on the fixed device indices, the upstream launcher's podman -# posture, host networking with the fleet's client port and alias, the -# measured KV pin for this exact hardware signature, boot persistence, and a -# deploy handler that retires the old root unit without deleting anything. +# runtime on the 2x Radeon AI PRO R9700. The intended shape is one +# tensor-parallel process across both cards (frank-medium.container, kept as +# the target); it cannot run on this host because RCCL's kernels require PCIe +# atomics and the second card's chipset path advertises AtomicOpsCap: Routing-. +# The deployed shape is therefore two single-GPU replicas behind a round-robin +# proxy, which keeps the fleet contract of one :7770 endpoint. This seam pins +# the properties that make that work: GPU access on the fixed device indices, +# the upstream launcher's podman posture, host networking with the fleet's +# client port and alias, per-replica resource shape, boot persistence, and a +# deploy handler that moves the unusable TP=2 unit into the trash. load helpers -UNIT="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container" +UNIT_A="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-a.container" +UNIT_B="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-b.container" +UNIT_PROXY="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-proxy.container" +UNIT_TP2="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container" ENTRYPOINT="$REPO_ROOT/dankserver/frank-medium/radiance-entrypoint.sh" HEALTHCHECK="$REPO_ROOT/dankserver/frank-medium/frank-medium-healthcheck.py" +PROXY="$REPO_ROOT/dankserver/frank-medium/proxy/roundrobin-proxy.py" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" -@test "frank-medium quadlet exists under dankserver/frank-medium/quadlet/" { - [ -f "$UNIT" ] +@test "the frank-medium units live under dankserver/frank-medium/quadlet/" { + [ -f "$UNIT_A" ] + [ -f "$UNIT_B" ] + [ -f "$UNIT_PROXY" ] } -@test "frank-medium unit is boot-persistent (WantedBy=default.target)" { +@test "the TP=2 unit is kept as the documented target, not deleted" { + # It becomes runnable again only on a host whose second card sits on an + # atomic-capable root port, so the artifact stays in the repo. + [ -f "$UNIT_TP2" ] + grep -q '^Environment=TP=2$' "$UNIT_TP2" +} + +@test "every frank-medium unit is boot-persistent (WantedBy=default.target)" { # Generated units cannot be enabled on the podman 5.x v1 quadlet # generator; [Install] plus linger is the entire boot-persistence story. - grep -q '^WantedBy=default.target$' "$UNIT" + for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do + grep -q '^WantedBy=default.target$' "$unit" + done } -@test "frank-medium unit is self-healing" { - grep -q '^Restart=on-failure$' "$UNIT" - grep -q '^RestartSec=' "$UNIT" +@test "every frank-medium unit is self-healing" { + for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do + grep -q '^Restart=on-failure$' "$unit" + grep -q '^RestartSec=' "$unit" + done } -@test "frank-medium image is pinned to an exact version tag" { - grep -Eq '^Image=localhost/radiance-vllm:[0-9]+\.[0-9]+\.[0-9]+$' "$UNIT" - grep -q '^Pull=never$' "$UNIT" +@test "frank-medium units pin the image tag and never pull at boot" { + for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do + grep -Eq '^Image=localhost/radiance-vllm:[0-9]+\.[0-9]+\.[0-9]+$' "$unit" + grep -q '^Pull=never$' "$unit" + done } -@test "frank-medium can reach the GPU (kfd + DRM render node) on the fixed indices" { - grep -q '^AddDevice=/dev/kfd$' "$UNIT" - grep -q '^AddDevice=/dev/dri$' "$UNIT" - grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT" - grep -q '^Environment=ROCR_VISIBLE_DEVICES=0,1$' "$UNIT" +@test "the replicas each get exactly one GPU on the fixed device index" { + grep -q '^AddDevice=/dev/kfd$' "$UNIT_A" + grep -q '^AddDevice=/dev/dri$' "$UNIT_A" + # One index per replica: ROCR filters first and HIP indexes into the + # filtered list, so setting both to the same non-zero value leaves an + # empty device set. + grep -q '^Environment=HIP_VISIBLE_DEVICES=0$' "$UNIT_A" + grep -q '^Environment=HIP_VISIBLE_DEVICES=1$' "$UNIT_B" + ! grep -q '^Environment=ROCR_VISIBLE_DEVICES=' "$UNIT_A" + ! grep -q '^Environment=ROCR_VISIBLE_DEVICES=' "$UNIT_B" } -@test "frank-medium runs with the upstream launcher's podman posture" { - # serve-mxfp4.sh runs the image privileged, with host IPC, an unconfined - # seccomp profile and SYS_PTRACE; host networking carries the port. - grep -q -- '--privileged' "$UNIT" - grep -q -- '--ipc=host' "$UNIT" - grep -q -- '--security-opt seccomp=unconfined' "$UNIT" - grep -q -- '--network=host' "$UNIT" +@test "the replicas run with the upstream launcher's podman posture" { + for unit in "$UNIT_A" "$UNIT_B"; do + grep -q -- '--privileged' "$unit" + grep -q -- '--ipc=host' "$unit" + grep -q -- '--security-opt seccomp=unconfined' "$unit" + grep -q -- '--network=host' "$unit" + done } -@test "frank-medium keeps the fleet endpoint: host-network port 7770, alias frank-medium" { - # vLLM binds 0.0.0.0:7770 on the host network; no PublishPort exists. - [ "$(grep -c '^PublishPort=' "$UNIT")" -eq 0 ] - grep -q '^Environment=PORT=7770$' "$UNIT" - grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT" +@test "the replicas serve single-GPU shapes that fit one card" { + for unit in "$UNIT_A" "$UNIT_B"; do + grep -q '^Environment=TP=1$' "$unit" + # 262144 does not fit one card (needs 9.61 GiB of KV, 5.14 available) + # and 131072 does not either (5.58 needed vs 5.22 available); 114688 + # is the largest measured shape that boots. + grep -q '^Environment=MAXLEN=114688$' "$unit" + # KV_MEM=0 makes vLLM profile this shape: the 2x7551-32624 pin is a + # TP=2/262144 measurement and invalid here. + grep -q '^Environment=KV_MEM=0$' "$unit" + grep -q '^Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq$' "$unit" + grep -q '^Environment=SPEC_METHOD=dflash$' "$unit" + grep -q '^Environment=SPEC=7$' "$unit" + done } -@test "frank-medium pins the measured KV cache for its hardware signature" { - # The upstream repo's shipped kv-profiles.tsv row for - # 2x7551-32624 at maxseqs=8 chunk=8192 maxlen=262144 spec=dflash. - # A pin is only valid at that batch shape; if the shape changes, this - # value must be recalibrated or set to 0 (vLLM profiling). - grep -q '^Environment=KV_MEM=18563072000$' "$UNIT" - grep -q '^Environment=MAXSEQS=8$' "$UNIT" - grep -q '^Environment=CHUNK=8192$' "$UNIT" - grep -q '^Environment=MAXLEN=262144$' "$UNIT" - grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT" - grep -q '^Environment=SPEC=7$' "$UNIT" +@test "the proxy keeps the fleet endpoint: host-network port 7770, alias frank-medium" { + grep -q '^Environment=PORT=7770$' "$UNIT_PROXY" + grep -q '^Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772$' "$UNIT_PROXY" + grep -q -- '--network=host' "$UNIT_PROXY" + [ "$(grep -c '^PublishPort=' "$UNIT_PROXY")" -eq 0 ] + # The replicas answer as the same served model name, so a client cannot + # tell which one served a request. + grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT_A" + grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT_B" + grep -q '^Environment=PORT=7771$' "$UNIT_A" + grep -q '^Environment=PORT=7772$' "$UNIT_B" } -@test "frank-medium mounts the pinned radiance clone and the patched libr4d" { - # The patches are applied at every container start from /patches; the - # image's shipped r4d.so predates the GDN overflow fix and NaNs this - # model, so /r4d (b9e42ab-rx6) must be mounted and copied over it. - grep -q 'Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z$' "$UNIT" - grep -q 'Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z$' "$UNIT" +@test "the proxy really is a round-robin over the replicas" { + [ -f "$PROXY" ] + python3 -m py_compile "$PROXY" + grep -q 'itertools.cycle' "$PROXY" + grep -q 'UPSTREAMS' "$PROXY" } @test "frank-medium entrypoint is a syntax-clean fork of the upstream launcher" { [ -f "$ENTRYPOINT" ] bash -n "$ENTRYPOINT" - # Pinned clone: the entrypoint must name the commit it was forked from so - # a clone update is forced through a re-diff, not drifted into silently. grep -q '9735329348ae1c9319f01cbe5f7004ec5f4dba63' "$ENTRYPOINT" } -@test "frank-medium healthcheck probes the serving endpoint, not the process" { +@test "frank-medium healthcheck probes the serving endpoint on its own port" { [ -f "$HEALTHCHECK" ] python3 -m py_compile "$HEALTHCHECK" - grep -q 'frank-medium-healthcheck.py' "$UNIT" - grep -q '^HealthStartPeriod=' "$UNIT" + # One probe serves the replicas, the proxy and the TP=2 target because it + # reads PORT instead of hardcoding 7770. + grep -q 'os.environ.get("PORT"' "$HEALTHCHECK" + for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do + grep -q 'frank-medium-healthcheck.py' "$unit" + grep -q '^HealthStartPeriod=' "$unit" + done } -@test "frank-medium unit carries no secret material" { - ! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$UNIT" +@test "frank-medium units carry no secret material" { + for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do + ! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$unit" + done } -@test "the frank-medium checkpoint mount is the designated target" { - grep -q '^Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq$' "$UNIT" - grep -q '^Volume=/home/dank/models:/models$' "$UNIT" +@test "the deploy handler installs the trio and retires the TP=2 unit" { + grep -q 'install_quadlet frank-medium-a dankserver/frank-medium/quadlet frank-medium-a' "$REMOTE_DEPLOY" + grep -q 'install_quadlet frank-medium-b dankserver/frank-medium/quadlet frank-medium-b' "$REMOTE_DEPLOY" + grep -q 'install_quadlet frank-medium-proxy dankserver/frank-medium/quadlet frank-medium-proxy' "$REMOTE_DEPLOY" + grep -q 'systemctl --user restart frank-medium-a.service frank-medium-b.service frank-medium-proxy.service' "$REMOTE_DEPLOY" + # The TP=2 unit cannot become healthy on this host; a leftover copy would + # leave the generator restarting it forever. It moves to the trash. + grep -q 'mv -- "$quadlet_unit_dir/frank-medium.container"' "$REMOTE_DEPLOY" + ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?"?\$quadlet_unit_dir/frank-medium' "$REMOTE_DEPLOY" +} + +@test "the deploy handler never tries to enable a generated unit" { + ! grep -q 'systemctl --user enable frank-medium' "$REMOTE_DEPLOY" } @test "the deploy handler retires the old root unit without deleting it" { grep -q 'retire_root_llm_medium' "$REMOTE_DEPLOY" grep -q 'systemctl stop frank-medium.service' "$REMOTE_DEPLOY" - # Never `rm`: the retired ROCmFPX unit moves into the home trash so the - # llama.cpp era stays recoverable. grep -q 'mv -- /etc/systemd/system/frank-medium.service' "$REMOTE_DEPLOY" ! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?/etc/systemd/system/frank-medium' "$REMOTE_DEPLOY" } -@test "the deploy handler never tries to enable a generated unit" { - # `systemctl --user enable frank-medium.service` fails on the v1 - # generator ("transient or generated"); the deploy must install and - # restart instead. - ! grep -q 'systemctl --user enable frank-medium.service' "$REMOTE_DEPLOY" - grep -q 'install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium' "$REMOTE_DEPLOY" - grep -q 'systemctl --user restart frank-medium.service' "$REMOTE_DEPLOY" -} - @test "the deploy handler re-tags the registry image to the localhost alias" { grep -q 'podman tag docker.io/stilldeadcode/vllm-radiance:0.9.3 localhost/radiance-vllm:0.9.3' "$REMOTE_DEPLOY" }