feat(frank-medium): cut the endpoint over to radiance TP=2
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
This commit is contained in:
+4
-4
@@ -4,10 +4,10 @@
|
||||
Runs INSIDE the container. The vLLM /health endpoint only answers once the
|
||||
engine is up (weights loaded, cudagraphs captured), so a healthy container is
|
||||
a serving container, not merely a live process. PORT selects the endpoint so
|
||||
the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can
|
||||
share this probe. The HealthCmd in the quadlet splits on spaces into an argv
|
||||
array (podman 5.x v1 generator), which is why this lives in a file rather
|
||||
than an inline python one-liner.
|
||||
the probe follows the unit's Environment=PORT rather than hard-coding the
|
||||
fleet port. The HealthCmd in the quadlet splits on spaces into an argv array
|
||||
(podman 5.x v1 generator), which is why this lives in a file rather than an
|
||||
inline python one-liner.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
+28
-17
@@ -3,30 +3,32 @@
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15
|
||||
# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3
|
||||
# image + the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
|
||||
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
|
||||
# the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
||||
# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp
|
||||
# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF;
|
||||
# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the
|
||||
# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production
|
||||
# configuration of the upstream repo, on our exact hardware signature
|
||||
# (2x7551-32624, see the KV pin below).
|
||||
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
|
||||
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
|
||||
# It replaced the mainline llama.cpp layer-split deployment, which served the
|
||||
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
# what this unit does too — no PublishPort. The fleet contract is unchanged:
|
||||
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
|
||||
#
|
||||
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
|
||||
# process across both R9700s. It cannot launch a collective here — RCCL's
|
||||
# kernels need PCIe atomics and the second card sits behind chipset bridges
|
||||
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
|
||||
# two-replica trio in this directory (frank-medium-a/b/proxy) and
|
||||
# deploy-remote.sh deliberately does not install this unit. Keep it for a
|
||||
# host whose second card is on an atomic-capable root port; the full evidence
|
||||
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
||||
# TP=2 IS INSTALLABLE HERE NOW, and it won on measurement. Until 2026-09-20
|
||||
# this unit was archived un-installed because the second card sat behind B650
|
||||
# chipset bridges advertising `AtomicOpsCap: Routing-`, which stopped RCCL
|
||||
# kernel dispatch. The board swap (ProArt B850-Creator WiFi NEO) put both
|
||||
# cards on CPU GPP root ports — Gen5 x8 each, every bridge Routing+, both
|
||||
# endpoints ReqEn+ — and the collective-free R4D path runs at TP=2 without
|
||||
# complaint. The conversation battery (scripts/bench-conversation.py, salted,
|
||||
# same 10 prompts, 2026-09-20) measured this shape against the incumbent:
|
||||
# decode geomean 95.8 t/s vs 45.7, mean TTFT 0.21 s vs 0.72 s, and 259 t/s
|
||||
# aggregate against 4 concurrent streams vs 71.7. Evidence and the llama-side
|
||||
# caveats: serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md and
|
||||
# Dual-R9700-Cooperation.md.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
|
||||
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
|
||||
@@ -92,6 +94,15 @@ Environment=TP=2
|
||||
Environment=MAXSEQS=8
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=262144
|
||||
# MAXLEN is the whole pool shared across the MAXSEQS slots: 8 concurrent
|
||||
# sessions split 262,144 tokens between them, so a full-window request
|
||||
# excludes the others until it finishes (vLLM preempts rather than refuses).
|
||||
# The llama deployment it replaced advertised 327,680 across 2 slots.
|
||||
#
|
||||
# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the
|
||||
# launcher loads no vision projector, so an input_audio/image request is a
|
||||
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
|
||||
# modelRoles vision -> frank-small).
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
||||
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
||||
Reference in New Issue
Block a user