Frank
4fca739bf8
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
148 lines
7.7 KiB
INI
148 lines
7.7 KiB
INI
# Quadlet unit: frank-medium (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
|
|
# Unit name: frank-medium.service (container name: frank-medium)
|
|
#
|
|
# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15
|
|
# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3
|
|
# image + the ggz14/radiance-vllm-mxfp4 patches) serving
|
|
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
|
# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp
|
|
# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF;
|
|
# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the
|
|
# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production
|
|
# configuration of the upstream repo, on our exact hardware signature
|
|
# (2x7551-32624, see the KV pin below).
|
|
#
|
|
# SHAPE. The launcher the upstream repo ships binds the server with
|
|
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
|
# what this unit does too — no PublishPort. The fleet contract is unchanged:
|
|
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
|
|
#
|
|
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
|
|
# process across both R9700s. It cannot launch a collective here — RCCL's
|
|
# kernels need PCIe atomics and the second card sits behind chipset bridges
|
|
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
|
|
# two-replica trio in this directory (frank-medium-a/b/proxy) and
|
|
# deploy-remote.sh deliberately does not install this unit. Keep it for a
|
|
# host whose second card is on an atomic-capable root port; the full evidence
|
|
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
|
#
|
|
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
|
|
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
|
|
# and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192
|
|
# logic this unit replaces. --privileged --ipc=host --security-opt
|
|
# seccomp=unconfined --cap-add SYS_PTRACE is the upstream launcher's exact
|
|
# podman posture.
|
|
#
|
|
# MOUNTS. /models (rw checkpoints), the HF cache (tokenizer files),
|
|
# /cache (Triton/inductor/aiter/vLLM compile cache — first start compiles for
|
|
# several minutes, later starts reuse), /patches (the pinned radiance clone —
|
|
# its patches are applied at EVERY container start by the entrypoint), /r4d
|
|
# (the patched libr4d build; the image's shipped r4d.so NaNs the
|
|
# gated-delta-net and must never be used), and the entrypoint + healthcheck
|
|
# scripts from this checkout (mounted read-only straight out of
|
|
# /home/dank/serverAppConfigs — deploy.sh git-pulls the checkout before
|
|
# dispatching).
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time (same as frank-large on dankclaw). deploy-remote.sh pulls the
|
|
# docker.io ref and re-tags it before the restart. Pull=never keeps boots off
|
|
# the registry.
|
|
#
|
|
# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated,
|
|
# re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new
|
|
# serve-mxfp4.sh container command and bump the pin deliberately.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
|
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
[Unit]
|
|
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-medium
|
|
Image=localhost/radiance-vllm:0.9.3
|
|
# vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort.
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
GroupAdd=keep-groups
|
|
# The upstream launcher's exact podman posture (rootless; podman-only flags
|
|
# like --replace are not needed under systemd management).
|
|
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
|
# The two R9700s. The Raphael iGPU (index 2, 2 GiB) stays out: shared system
|
|
# memory is not a compute target.
|
|
Environment=ROCR_VISIBLE_DEVICES=0,1
|
|
Environment=HIP_VISIBLE_DEVICES=0,1
|
|
# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
|
|
# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
|
|
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
|
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
|
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
|
# Fleet identity: the id at /v1/models and in every response body. Every
|
|
# client (Hermes, the harness, services.json) points at frank-medium.
|
|
Environment=SERVED_NAMES=frank-medium
|
|
Environment=PORT=7770
|
|
# Fixed 2-card tensor parallel: the tested configuration of the upstream
|
|
# repo; the R4D one-shot all-reduce kernel serves it natively.
|
|
Environment=TP=2
|
|
# Measured production batch shape (upstream README): MAXSEQS 8 peaks
|
|
# aggregate throughput at 573 t/s; MAXLEN is the model's native context.
|
|
Environment=MAXSEQS=8
|
|
Environment=CHUNK=8192
|
|
Environment=MAXLEN=262144
|
|
Environment=GPU_UTIL=0.98
|
|
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
|
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
|
# prefill) was measured with.
|
|
Environment=SPEC_METHOD=dflash
|
|
Environment=SPEC=7
|
|
# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
|
|
# sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
|
|
# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
|
|
# explicitly because this unit does not run the launcher's lookup; valid
|
|
# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
|
|
# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
|
|
# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
|
|
Environment=KV_MEM=18563072000
|
|
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
|
|
# keep parity with the launcher's rw mount).
|
|
Volume=/home/dank/models:/models
|
|
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the
|
|
# entrypoint.
|
|
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
|
# Compile cache. Keyed on model + stack version + every graph-changing knob;
|
|
# never share it across configurations (upstream rule).
|
|
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
|
# The pinned radiance clone: patches applied at every container start.
|
|
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
|
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The
|
|
# image's shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
|
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
|
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
|
# The entrypoint must be the container command; the image's own entrypoint
|
|
# (which the patched script execs at the end) must not run unpatched.
|
|
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
|
# 200 only once the engine serves; during first-start compilation the
|
|
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
|
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
|
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
|
HealthInterval=60s
|
|
HealthTimeout=30s
|
|
HealthRetries=3
|
|
HealthStartPeriod=1800s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# First start compiles Triton/inductor kernels (several minutes) before the
|
|
# engine loads ~9.5 GiB of weights per card.
|
|
TimeoutStartSec=1800
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|