Frank
c2116658f8
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
159 lines
8.3 KiB
INI
159 lines
8.3 KiB
INI
# Quadlet unit: frank-medium (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
|
|
# Unit name: frank-medium.service (container name: frank-medium)
|
|
#
|
|
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
|
|
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
|
|
# the ggz14/radiance-vllm-mxfp4 patches) serving
|
|
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
|
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
|
|
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
|
|
# It replaced the mainline llama.cpp layer-split deployment, which served the
|
|
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
|
|
#
|
|
# SHAPE. The launcher the upstream repo ships binds the server with
|
|
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
|
# what this unit does too — no PublishPort. The fleet contract is unchanged:
|
|
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
|
|
#
|
|
# TP=2 IS INSTALLABLE HERE NOW, and it won on measurement. Until 2026-09-20
|
|
# this unit was archived un-installed because the second card sat behind B650
|
|
# chipset bridges advertising `AtomicOpsCap: Routing-`, which stopped RCCL
|
|
# kernel dispatch. The board swap (ProArt B850-Creator WiFi NEO) put both
|
|
# cards on CPU GPP root ports — Gen5 x8 each, every bridge Routing+, both
|
|
# endpoints ReqEn+ — and the collective-free R4D path runs at TP=2 without
|
|
# complaint. The conversation battery (scripts/bench-conversation.py, salted,
|
|
# same 10 prompts, 2026-09-20) measured this shape against the incumbent:
|
|
# decode geomean 95.8 t/s vs 45.7, mean TTFT 0.21 s vs 0.72 s, and 259 t/s
|
|
# aggregate against 4 concurrent streams vs 71.7. Evidence and the llama-side
|
|
# caveats: serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md and
|
|
# Dual-R9700-Cooperation.md.
|
|
#
|
|
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
|
|
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
|
|
# and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192
|
|
# logic this unit replaces. --privileged --ipc=host --security-opt
|
|
# seccomp=unconfined --cap-add SYS_PTRACE is the upstream launcher's exact
|
|
# podman posture.
|
|
#
|
|
# MOUNTS. /models (rw checkpoints), the HF cache (tokenizer files),
|
|
# /cache (Triton/inductor/aiter/vLLM compile cache — first start compiles for
|
|
# several minutes, later starts reuse), /patches (the pinned radiance clone —
|
|
# its patches are applied at EVERY container start by the entrypoint), /r4d
|
|
# (the patched libr4d build; the image's shipped r4d.so NaNs the
|
|
# gated-delta-net and must never be used), and the entrypoint + healthcheck
|
|
# scripts from this checkout (mounted read-only straight out of
|
|
# /home/dank/serverAppConfigs — deploy.sh git-pulls the checkout before
|
|
# dispatching).
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time (same as frank-large on dankclaw). deploy-remote.sh pulls the
|
|
# docker.io ref and re-tags it before the restart. Pull=never keeps boots off
|
|
# the registry.
|
|
#
|
|
# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated,
|
|
# re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new
|
|
# serve-mxfp4.sh container command and bump the pin deliberately.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
|
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
[Unit]
|
|
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-medium
|
|
Image=localhost/radiance-vllm:0.9.3
|
|
# vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort.
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
GroupAdd=keep-groups
|
|
# The upstream launcher's exact podman posture (rootless; podman-only flags
|
|
# like --replace are not needed under systemd management).
|
|
PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
|
|
# The two R9700s. The Raphael iGPU (index 2, 2 GiB) stays out: shared system
|
|
# memory is not a compute target.
|
|
Environment=ROCR_VISIBLE_DEVICES=0,1
|
|
Environment=HIP_VISIBLE_DEVICES=0,1
|
|
# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
|
|
# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
|
|
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
|
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
|
Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
|
|
# Fleet identity: the id at /v1/models and in every response body. Every
|
|
# client (Hermes, the harness, services.json) points at frank-medium.
|
|
Environment=SERVED_NAMES=frank-medium
|
|
Environment=PORT=7770
|
|
# Fixed 2-card tensor parallel: the tested configuration of the upstream
|
|
# repo; the R4D one-shot all-reduce kernel serves it natively.
|
|
Environment=TP=2
|
|
# Measured production batch shape (upstream README): MAXSEQS 8 peaks
|
|
# aggregate throughput at 573 t/s; MAXLEN is the model's native context.
|
|
Environment=MAXSEQS=8
|
|
Environment=CHUNK=8192
|
|
Environment=MAXLEN=262144
|
|
# MAXLEN is the whole pool shared across the MAXSEQS slots: 8 concurrent
|
|
# sessions split 262,144 tokens between them, so a full-window request
|
|
# excludes the others until it finishes (vLLM preempts rather than refuses).
|
|
# The llama deployment it replaced advertised 327,680 across 2 slots.
|
|
#
|
|
# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the
|
|
# launcher loads no vision projector, so an input_audio/image request is a
|
|
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
|
|
# modelRoles vision -> frank-small).
|
|
Environment=GPU_UTIL=0.98
|
|
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
|
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
|
# prefill) was measured with.
|
|
Environment=SPEC_METHOD=dflash
|
|
Environment=SPEC=7
|
|
# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
|
|
# sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
|
|
# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
|
|
# explicitly because this unit does not run the launcher's lookup; valid
|
|
# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
|
|
# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
|
|
# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
|
|
Environment=KV_MEM=18563072000
|
|
# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
|
|
# keep parity with the launcher's rw mount).
|
|
Volume=/home/dank/models:/models
|
|
# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the
|
|
# entrypoint.
|
|
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
|
# Compile cache. Keyed on model + stack version + every graph-changing knob;
|
|
# never share it across configurations (upstream rule).
|
|
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
|
# The pinned radiance clone: patches applied at every container start.
|
|
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
|
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The
|
|
# image's shipped r4d.so predates the GDN overflow fix and NaNs this model.
|
|
Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
|
# Entrypoint + healthcheck from this checkout, mounted read-only.
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
|
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
|
# The entrypoint must be the container command; the image's own entrypoint
|
|
# (which the patched script execs at the end) must not run unpatched.
|
|
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
|
# 200 only once the engine serves; during first-start compilation the
|
|
# container is legitimately unhealthy. HealthCmd is split on spaces into an
|
|
# argv array by the podman 5.x v1 generator, hence the file-based probe.
|
|
HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
|
|
HealthInterval=60s
|
|
HealthTimeout=30s
|
|
HealthRetries=3
|
|
HealthStartPeriod=1800s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# First start compiles Triton/inductor kernels (several minutes) before the
|
|
# engine loads ~9.5 GiB of weights per card.
|
|
TimeoutStartSec=1800
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|