Frank
05c0d3a00a
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
99 lines
4.9 KiB
INI
99 lines
4.9 KiB
INI
# Quadlet unit: frank-medium replica B (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankserver: ~/.config/containers/systemd/frank-medium-b.container
|
|
# Unit name: frank-medium-b.service (container name: frank-medium-a)
|
|
#
|
|
# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
|
|
# tensor-parallel process across both R9700s (see frank-medium.container, kept
|
|
# as the target). That shape cannot run on this host: RCCL's collective kernels
|
|
# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
|
|
# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
|
|
# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
|
|
# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
|
|
# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
|
|
# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
|
|
# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
|
|
# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
|
|
# atomics. Result: every collective that touches GPU 2 dies at launch.
|
|
# Two independent single-GPU replicas use both cards instead; the proxy unit
|
|
# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
|
|
#
|
|
# SHAPE. vLLM binds 0.0.0.0:7772 on the host network directly; no PublishPort.
|
|
# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
|
|
# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
|
|
# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
|
|
# this shape instead of using the 2x7551-32624 pin, which is only valid at
|
|
# TP=2/262144.
|
|
#
|
|
# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=1 (the second R9700). The
|
|
# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
|
|
# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
|
|
# first and HIP indexes into the filtered list, so a single-valued pair
|
|
# (ROCR=1,HIP=1) leaves an empty device set.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
|
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
[Unit]
|
|
Description=frank-medium replica B — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #2
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-medium-b
|
|
Image=localhost/radiance-vllm:0.9.3
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
GroupAdd=keep-groups
|
|
# The upstream launcher's exact podman posture (rootless).
|
|
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
|
Environment=HIP_VISIBLE_DEVICES=1
|
|
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
|
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
|
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
|
Environment=SERVED_NAMES=frank-medium
|
|
Environment=PORT=7772
|
|
Environment=TP=1
|
|
Environment=MAXSEQS=4
|
|
Environment=CHUNK=8192
|
|
Environment=MAXLEN=114688
|
|
Environment=GPU_UTIL=0.98
|
|
Environment=KV_MEM=0
|
|
Environment=SPEC_METHOD=dflash
|
|
Environment=SPEC=7
|
|
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
|
|
Volume=/home/dank/models:/models
|
|
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
|
|
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
|
# Compile cache. Shared with the other replica and with the TP=2 target: the
|
|
# entries are content-addressed, so shapes simply add subdirectories.
|
|
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
|
# The pinned radiance clone: patches applied at every container start.
|
|
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
|
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
|
|
# shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
|
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
|
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
|
# The entrypoint must be the container command; the image's own entrypoint
|
|
# (which the patched script execs at the end) must not run unpatched.
|
|
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
|
# 200 only once the engine serves; during first-start compilation the
|
|
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
|
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
|
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
|
HealthInterval=60s
|
|
HealthTimeout=30s
|
|
HealthRetries=3
|
|
HealthStartPeriod=1800s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# First start compiles Triton/inductor kernels (several minutes) before the
|
|
# engine loads ~20 GiB of weights.
|
|
TimeoutStartSec=1800
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|