dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: deploy two single-GPU replicas — TP=2 is blocked by PCIe atomics

Browse Source 
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.

Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).

Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.

Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.

Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
This commit is contained in:
Frank committed 2026-09-16 03:47:11 -07:00
1 parent ad7a9c42f9
commit 4fca739bf8
15 files changed
+817 -105
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/frank_medium_quadlet.bats
+114 -74
View File
@@ -1,127 +1,167 @@
#!/usr/bin/env bats
# Seam: the frank-medium quadlet on dankserver.
# frank-medium became a rootless Quadlet container on 2026-09-15 when the
# Seam: the frank-medium service on dankserver.
# frank-medium became a rootless Quadlet deployment on 2026-09-15 when the
# vllm-radiance MXFP4 stack replaced the charlie12345/ROCmFPX llama.cpp
# runtime on the 2x Radeon AI PRO R9700. This seam pins the properties that
# make it work and that no reviewer can see by reading the flags alone: GPU
# device access on the fixed device indices, the upstream launcher's podman
# posture, host networking with the fleet's client port and alias, the
# measured KV pin for this exact hardware signature, boot persistence, and a
# deploy handler that retires the old root unit without deleting anything.
# runtime on the 2x Radeon AI PRO R9700. The intended shape is one
# tensor-parallel process across both cards (frank-medium.container, kept as
# the target); it cannot run on this host because RCCL's kernels require PCIe
# atomics and the second card's chipset path advertises AtomicOpsCap: Routing-.
# The deployed shape is therefore two single-GPU replicas behind a round-robin
# proxy, which keeps the fleet contract of one :7770 endpoint. This seam pins
# the properties that make that work: GPU access on the fixed device indices,
# the upstream launcher's podman posture, host networking with the fleet's
# client port and alias, per-replica resource shape, boot persistence, and a
# deploy handler that moves the unusable TP=2 unit into the trash.
load helpers
UNIT="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container"
UNIT_A="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-a.container"
UNIT_B="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-b.container"
UNIT_PROXY="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium-proxy.container"
UNIT_TP2="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container"
ENTRYPOINT="$REPO_ROOT/dankserver/frank-medium/radiance-entrypoint.sh"
HEALTHCHECK="$REPO_ROOT/dankserver/frank-medium/frank-medium-healthcheck.py"
PROXY="$REPO_ROOT/dankserver/frank-medium/proxy/roundrobin-proxy.py"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
@test "frank-medium quadlet exists under dankserver/frank-medium/quadlet/" {
[ -f "$UNIT" ]
@test "the frank-medium units live under dankserver/frank-medium/quadlet/" {
[ -f "$UNIT_A" ]
[ -f "$UNIT_B" ]
[ -f "$UNIT_PROXY" ]
}
@test "frank-medium unit is boot-persistent (WantedBy=default.target)" {
@test "the TP=2 unit is kept as the documented target, not deleted" {
# It becomes runnable again only on a host whose second card sits on an
# atomic-capable root port, so the artifact stays in the repo.
[ -f "$UNIT_TP2" ]
grep -q '^Environment=TP=2$' "$UNIT_TP2"
}
@test "every frank-medium unit is boot-persistent (WantedBy=default.target)" {
# Generated units cannot be enabled on the podman 5.x v1 quadlet
# generator; [Install] plus linger is the entire boot-persistence story.
grep -q '^WantedBy=default.target$' "$UNIT"
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
grep -q '^WantedBy=default.target$' "$unit"
done
}
@test "frank-medium unit is self-healing" {
grep -q '^Restart=on-failure$' "$UNIT"
grep -q '^RestartSec=' "$UNIT"
@test "every frank-medium unit is self-healing" {
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
grep -q '^Restart=on-failure$' "$unit"
grep -q '^RestartSec=' "$unit"
done
}
@test "frank-medium image is pinned to an exact version tag" {
grep -Eq '^Image=localhost/radiance-vllm:[0-9]+\.[0-9]+\.[0-9]+$' "$UNIT"
grep -q '^Pull=never$' "$UNIT"
@test "frank-medium units pin the image tag and never pull at boot" {
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
grep -Eq '^Image=localhost/radiance-vllm:[0-9]+\.[0-9]+\.[0-9]+$' "$unit"
grep -q '^Pull=never$' "$unit"
done
}
@test "frank-medium can reach the GPU (kfd + DRM render node) on the fixed indices" {
grep -q '^AddDevice=/dev/kfd$' "$UNIT"
grep -q '^AddDevice=/dev/dri$' "$UNIT"
grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT"
grep -q '^Environment=ROCR_VISIBLE_DEVICES=0,1$' "$UNIT"
@test "the replicas each get exactly one GPU on the fixed device index" {
grep -q '^AddDevice=/dev/kfd$' "$UNIT_A"
grep -q '^AddDevice=/dev/dri$' "$UNIT_A"
# One index per replica: ROCR filters first and HIP indexes into the
# filtered list, so setting both to the same non-zero value leaves an
# empty device set.
grep -q '^Environment=HIP_VISIBLE_DEVICES=0$' "$UNIT_A"
grep -q '^Environment=HIP_VISIBLE_DEVICES=1$' "$UNIT_B"
! grep -q '^Environment=ROCR_VISIBLE_DEVICES=' "$UNIT_A"
! grep -q '^Environment=ROCR_VISIBLE_DEVICES=' "$UNIT_B"
}
@test "frank-medium runs with the upstream launcher's podman posture" {
# serve-mxfp4.sh runs the image privileged, with host IPC, an unconfined
# seccomp profile and SYS_PTRACE; host networking carries the port.
grep -q -- '--privileged' "$UNIT"
grep -q -- '--ipc=host' "$UNIT"
grep -q -- '--security-opt seccomp=unconfined' "$UNIT"
grep -q -- '--network=host' "$UNIT"
@test "the replicas run with the upstream launcher's podman posture" {
for unit in "$UNIT_A" "$UNIT_B"; do
grep -q -- '--privileged' "$unit"
grep -q -- '--ipc=host' "$unit"
grep -q -- '--security-opt seccomp=unconfined' "$unit"
grep -q -- '--network=host' "$unit"
done
}
@test "frank-medium keeps the fleet endpoint: host-network port 7770, alias frank-medium" {
# vLLM binds 0.0.0.0:7770 on the host network; no PublishPort exists.
[ "$(grep -c '^PublishPort=' "$UNIT")" -eq 0 ]
grep -q '^Environment=PORT=7770$' "$UNIT"
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT"
@test "the replicas serve single-GPU shapes that fit one card" {
for unit in "$UNIT_A" "$UNIT_B"; do
grep -q '^Environment=TP=1$' "$unit"
# 262144 does not fit one card (needs 9.61 GiB of KV, 5.14 available)
# and 131072 does not either (5.58 needed vs 5.22 available); 114688
# is the largest measured shape that boots.
grep -q '^Environment=MAXLEN=114688$' "$unit"
# KV_MEM=0 makes vLLM profile this shape: the 2x7551-32624 pin is a
# TP=2/262144 measurement and invalid here.
grep -q '^Environment=KV_MEM=0$' "$unit"
grep -q '^Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq$' "$unit"
grep -q '^Environment=SPEC_METHOD=dflash$' "$unit"
grep -q '^Environment=SPEC=7$' "$unit"
done
}
@test "frank-medium pins the measured KV cache for its hardware signature" {
# The upstream repo's shipped kv-profiles.tsv row for
# 2x7551-32624 at maxseqs=8 chunk=8192 maxlen=262144 spec=dflash.
# A pin is only valid at that batch shape; if the shape changes, this
# value must be recalibrated or set to 0 (vLLM profiling).
grep -q '^Environment=KV_MEM=18563072000$' "$UNIT"
grep -q '^Environment=MAXSEQS=8$' "$UNIT"
grep -q '^Environment=CHUNK=8192$' "$UNIT"
grep -q '^Environment=MAXLEN=262144$' "$UNIT"
grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT"
grep -q '^Environment=SPEC=7$' "$UNIT"
@test "the proxy keeps the fleet endpoint: host-network port 7770, alias frank-medium" {
grep -q '^Environment=PORT=7770$' "$UNIT_PROXY"
grep -q '^Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772$' "$UNIT_PROXY"
grep -q -- '--network=host' "$UNIT_PROXY"
[ "$(grep -c '^PublishPort=' "$UNIT_PROXY")" -eq 0 ]
# The replicas answer as the same served model name, so a client cannot
# tell which one served a request.
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT_A"
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT_B"
grep -q '^Environment=PORT=7771$' "$UNIT_A"
grep -q '^Environment=PORT=7772$' "$UNIT_B"
}
@test "frank-medium mounts the pinned radiance clone and the patched libr4d" {
# The patches are applied at every container start from /patches; the
# image's shipped r4d.so predates the GDN overflow fix and NaNs this
# model, so /r4d (b9e42ab-rx6) must be mounted and copied over it.
grep -q 'Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z$' "$UNIT"
grep -q 'Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z$' "$UNIT"
@test "the proxy really is a round-robin over the replicas" {
[ -f "$PROXY" ]
python3 -m py_compile "$PROXY"
grep -q 'itertools.cycle' "$PROXY"
grep -q 'UPSTREAMS' "$PROXY"
}
@test "frank-medium entrypoint is a syntax-clean fork of the upstream launcher" {
[ -f "$ENTRYPOINT" ]
bash -n "$ENTRYPOINT"
# Pinned clone: the entrypoint must name the commit it was forked from so
# a clone update is forced through a re-diff, not drifted into silently.
grep -q '9735329348ae1c9319f01cbe5f7004ec5f4dba63' "$ENTRYPOINT"
}
@test "frank-medium healthcheck probes the serving endpoint, not the process" {
@test "frank-medium healthcheck probes the serving endpoint on its own port" {
[ -f "$HEALTHCHECK" ]
python3 -m py_compile "$HEALTHCHECK"
grep -q 'frank-medium-healthcheck.py' "$UNIT"
grep -q '^HealthStartPeriod=' "$UNIT"
# One probe serves the replicas, the proxy and the TP=2 target because it
# reads PORT instead of hardcoding 7770.
grep -q 'os.environ.get("PORT"' "$HEALTHCHECK"
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
grep -q 'frank-medium-healthcheck.py' "$unit"
grep -q '^HealthStartPeriod=' "$unit"
done
}
@test "frank-medium unit carries no secret material" {
! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$UNIT"
@test "frank-medium units carry no secret material" {
for unit in "$UNIT_A" "$UNIT_B" "$UNIT_PROXY"; do
! grep -Eq '^(Environment|EnvironmentFile)=.*(TOKEN|PASSWORD|SECRET|KEY)' "$unit"
done
}
@test "the frank-medium checkpoint mount is the designated target" {
grep -q '^Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq$' "$UNIT"
grep -q '^Volume=/home/dank/models:/models$' "$UNIT"
@test "the deploy handler installs the trio and retires the TP=2 unit" {
grep -q 'install_quadlet frank-medium-a dankserver/frank-medium/quadlet frank-medium-a' "$REMOTE_DEPLOY"
grep -q 'install_quadlet frank-medium-b dankserver/frank-medium/quadlet frank-medium-b' "$REMOTE_DEPLOY"
grep -q 'install_quadlet frank-medium-proxy dankserver/frank-medium/quadlet frank-medium-proxy' "$REMOTE_DEPLOY"
grep -q 'systemctl --user restart frank-medium-a.service frank-medium-b.service frank-medium-proxy.service' "$REMOTE_DEPLOY"
# The TP=2 unit cannot become healthy on this host; a leftover copy would
# leave the generator restarting it forever. It moves to the trash.
grep -q 'mv -- "$quadlet_unit_dir/frank-medium.container"' "$REMOTE_DEPLOY"
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?"?\$quadlet_unit_dir/frank-medium' "$REMOTE_DEPLOY"
}
@test "the deploy handler never tries to enable a generated unit" {
! grep -q 'systemctl --user enable frank-medium' "$REMOTE_DEPLOY"
}
@test "the deploy handler retires the old root unit without deleting it" {
grep -q 'retire_root_llm_medium' "$REMOTE_DEPLOY"
grep -q 'systemctl stop frank-medium.service' "$REMOTE_DEPLOY"
# Never `rm`: the retired ROCmFPX unit moves into the home trash so the
# llama.cpp era stays recoverable.
grep -q 'mv -- /etc/systemd/system/frank-medium.service' "$REMOTE_DEPLOY"
! grep -qE '(^|[^[:alnum:]_])rm (-[a-zA-Z]+ )?/etc/systemd/system/frank-medium' "$REMOTE_DEPLOY"
}
@test "the deploy handler never tries to enable a generated unit" {
# `systemctl --user enable frank-medium.service` fails on the v1
# generator ("transient or generated"); the deploy must install and
# restart instead.
! grep -q 'systemctl --user enable frank-medium.service' "$REMOTE_DEPLOY"
grep -q 'install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium' "$REMOTE_DEPLOY"
grep -q 'systemctl --user restart frank-medium.service' "$REMOTE_DEPLOY"
}
@test "the deploy handler re-tags the registry image to the localhost alias" {
grep -q 'podman tag docker.io/stilldeadcode/vllm-radiance:0.9.3 localhost/radiance-vllm:0.9.3' "$REMOTE_DEPLOY"
}