feat(frank-medium): cut the endpoint over to radiance TP=2

The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:

  arm I  llama.cpp layer split 32:33 + DFlash2, Q8_0  geo decode  45.7 t/s, TTFT 0.72 s, burst-4  71.7 t/s
  arm L  llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4  54.3 t/s
  arm R  radiance vLLM TP=2 + DFlash2-FP8, MXFP4      geo decode  95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s

Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.

  - un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
    it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
    The replica pair and its affinity proxy stay archived; one process owns
    both cards now.
  - retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
    bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
    rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
    stay: they are the tooling that rollback needs.
  - deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
    re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
    the drafter, the pinned patch clone and the patched libr4d; retires the
    llama root unit into the home trash instead of relinking it; then
    install_quadlet + wait_container_healthy 1800s (first start compiles).
    retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
    a/b/proxy trio only, since frank-medium is installed again.
  - planner + test-changed map dankserver/frank-medium/** to frank-medium
    and its seam; the scripts-trio mapping is gone with the files.
  - tests/frank_medium_unit.bats rewritten for the container contract: boot
    persistence of a GENERATED unit (no enable on this host's v1 generator),
    TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
    the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
    was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
    contract, and the archive invariants.
  - homepage card + docs updated, and they were stale twice over: they still
    described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.

Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
This commit is contained in:
Frank
2026-09-20 18:15:27 -07:00
parent bcdd898750
commit c2116658f8
15 changed files with 345 additions and 433 deletions
@@ -4,10 +4,10 @@
Runs INSIDE the container. The vLLM /health endpoint only answers once the
engine is up (weights loaded, cudagraphs captured), so a healthy container is
a serving container, not merely a live process. PORT selects the endpoint so
the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can
share this probe. The HealthCmd in the quadlet splits on spaces into an argv
array (podman 5.x v1 generator), which is why this lives in a file rather
than an inline python one-liner.
the probe follows the unit's Environment=PORT rather than hard-coding the
fleet port. The HealthCmd in the quadlet splits on spaces into an argv array
(podman 5.x v1 generator), which is why this lives in a file rather than an
inline python one-liner.
"""
import os
import sys
@@ -3,30 +3,32 @@
# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
# Unit name: frank-medium.service (container name: frank-medium)
#
# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15
# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3
# image + the ggz14/radiance-vllm-mxfp4 patches) serving
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
# the ggz14/radiance-vllm-mxfp4 patches) serving
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp
# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF;
# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the
# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production
# configuration of the upstream repo, on our exact hardware signature
# (2x7551-32624, see the KV pin below).
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
# It replaced the mainline llama.cpp layer-split deployment, which served the
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
#
# SHAPE. The launcher the upstream repo ships binds the server with
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
# what this unit does too — no PublishPort. The fleet contract is unchanged:
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
#
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
# process across both R9700s. It cannot launch a collective here — RCCL's
# kernels need PCIe atomics and the second card sits behind chipset bridges
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
# two-replica trio in this directory (frank-medium-a/b/proxy) and
# deploy-remote.sh deliberately does not install this unit. Keep it for a
# host whose second card is on an atomic-capable root port; the full evidence
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
# TP=2 IS INSTALLABLE HERE NOW, and it won on measurement. Until 2026-09-20
# this unit was archived un-installed because the second card sat behind B650
# chipset bridges advertising `AtomicOpsCap: Routing-`, which stopped RCCL
# kernel dispatch. The board swap (ProArt B850-Creator WiFi NEO) put both
# cards on CPU GPP root ports — Gen5 x8 each, every bridge Routing+, both
# endpoints ReqEn+ — and the collective-free R4D path runs at TP=2 without
# complaint. The conversation battery (scripts/bench-conversation.py, salted,
# same 10 prompts, 2026-09-20) measured this shape against the incumbent:
# decode geomean 95.8 t/s vs 45.7, mean TTFT 0.21 s vs 0.72 s, and 259 t/s
# aggregate against 4 concurrent streams vs 71.7. Evidence and the llama-side
# caveats: serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md and
# Dual-R9700-Cooperation.md.
#
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
@@ -92,6 +94,15 @@ Environment=TP=2
Environment=MAXSEQS=8
Environment=CHUNK=8192
Environment=MAXLEN=262144
# MAXLEN is the whole pool shared across the MAXSEQS slots: 8 concurrent
# sessions split 262,144 tokens between them, so a full-window request
# excludes the others until it finishes (vLLM preempts rather than refuses).
# The llama deployment it replaced advertised 327,680 across 2 slots.
#
# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the
# launcher loads no vision projector, so an input_audio/image request is a
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
# modelRoles vision -> frank-small).
Environment=GPU_UTIL=0.98
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
# every published number (224 t/s single-stream decode, 4749-4810 t/s