feat(frank-medium): cut the endpoint over to radiance TP=2
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
This commit is contained in:
Frank
+8
-11
@@ -369,11 +369,14 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
_add(services, "node-exporter")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-medium":
|
||||
# frank-medium is a ROOT llama.cpp unit since 2026-09-16: its
|
||||
# runtime is a native gfx1201 binary, not a container, so there is
|
||||
# nothing under dankserver/frank-medium/ to plan any more. The
|
||||
# retired vllm-radiance quadlets live in .AGENTTRASH/, which is
|
||||
# archive material and plans nothing.
|
||||
# frank-medium is a CONTAINER again since 2026-09-20: the TP=2
|
||||
# radiance Quadlet, its entrypoint and its healthcheck all live
|
||||
# under this directory, and the container mounts the latter two
|
||||
# straight out of this checkout — so any file here changes deployed
|
||||
# state. (It was a root llama.cpp unit from 2026-09-16, whose trio
|
||||
# lived in scripts/; that shape is archived in
|
||||
# scripts/.AGENTTRASH/20260920-llama-layersplit/ and plans nothing.)
|
||||
_add(services, "frank-medium")
|
||||
return services
|
||||
return services
|
||||
|
||||
@@ -405,12 +408,6 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
|
||||
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
|
||||
# files here the same way. The files that BUILD and FETCH its
|
||||
# runtime and payload are host-side one-offs: they change no
|
||||
# deployed state, so they select nothing.
|
||||
_add(services, "frank-medium")
|
||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
||||
# CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
|
||||
# serve files here the same way: a root unit on a host-built
|
||||
|
||||
+102
-54
@@ -1137,31 +1137,43 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a
|
||||
# ROOT unit since the 2026-09-16 cutover: the runtime is a native gfx1201 HIP
|
||||
# binary, so there is no image to install and no generated unit to dodge. It
|
||||
# replaces the vllm-radiance replica pair + routing proxy, which held this
|
||||
# endpoint for a day and could never use both cards for one model (TP=2 needs
|
||||
# an RCCL collective, and the second card cannot dispatch one).
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a
|
||||
# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2)
|
||||
# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter.
|
||||
# It replaced mainline llama.cpp's layer split, which held the endpoint since
|
||||
# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a
|
||||
# third of the time-to-first-token with 4x the concurrent throughput
|
||||
# (scripts/bench-conversation.py, 2026-09-20).
|
||||
#
|
||||
# Layer split in ONE process is what the successor does instead: layers are
|
||||
# distributed across the cards and only the activation at the boundary crosses
|
||||
# PCIe, once per forward pass. That is the "pipeline parallelism" the endpoint
|
||||
# wanted, with no collective anywhere in it.
|
||||
LLM_MEDIUM_BIN="${LLM_MEDIUM_BIN:-/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}"
|
||||
LLM_MEDIUM_MODEL="${LLM_MEDIUM_MODEL:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/Qwen3.8-27B-Uncensored-Q8_0.gguf}"
|
||||
LLM_MEDIUM_DRAFT="${LLM_MEDIUM_DRAFT:-/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
LLM_MEDIUM_MMPROJ="${LLM_MEDIUM_MMPROJ:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/mmproj-Qwen3.8-27B-Uncensored-f16.gguf}"
|
||||
|
||||
# The three vLLM user units are retired HERE, not left to rot: the proxy
|
||||
# published 7770 on the host network, so the llama-server below cannot bind
|
||||
# until they are stopped and the port is released. Never `rm` — the unit files
|
||||
# move into the home trash; the images stay in local storage.
|
||||
retire_vllm_medium_quadlets() {
|
||||
# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this
|
||||
# unit because TP=2 could not dispatch a collective on the second card (B650
|
||||
# chipset bridges, AtomicOpsCap: Routing-), and the two-replica + routing-proxy
|
||||
# trio was the workaround. The board swap cleared the atomics wall, so one
|
||||
# TP=2 process is both possible and the fastest shape; the replicas and their
|
||||
# affinity proxy stay retired.
|
||||
#
|
||||
# HOST ASSETS. Nothing here installs the image's payloads: the checkpoints, the
|
||||
# pinned radiance patch clone, the patched libr4d build and the Triton compile
|
||||
# cache live outside the checkout (see the unit file's mount comments). The
|
||||
# gates below refuse a deploy that would start a container guaranteed to die.
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}"
|
||||
LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}"
|
||||
LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}"
|
||||
LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}"
|
||||
|
||||
# The three replica/proxy user units from the 2026-09-16 era are retired HERE,
|
||||
# never left to rot: they must not exist alongside the TP=2 unit (the proxy
|
||||
# published 7770 on the host network). Never `rm` — unit files move into the
|
||||
# home trash; the images stay in local storage. `frank-medium` itself is NOT in
|
||||
# the list: it is the unit this handler installs.
|
||||
retire_vllm_replica_quadlets() {
|
||||
local trash="$HOME/.AGENTTRASH/20260916-vllm-radiance-retire" unit
|
||||
systemctl --user stop frank-medium-a.service frank-medium-b.service \
|
||||
frank-medium-proxy.service 2>/dev/null || true
|
||||
for unit in frank-medium-a frank-medium-b frank-medium-proxy frank-medium; do
|
||||
for unit in frank-medium-a frank-medium-b frank-medium-proxy; do
|
||||
if [[ -e "$quadlet_unit_dir/$unit.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/$unit.container" "$trash/$unit.container"
|
||||
@@ -1171,54 +1183,90 @@ retire_vllm_medium_quadlets() {
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
|
||||
# The llama.cpp root unit is retired HERE, into the home trash, never `rm`:
|
||||
# /etc/systemd/system/frank-medium.service is a relink_unit symlink into this
|
||||
# checkout, and the checkout pull that precedes this handler already moved the
|
||||
# repo file away, leaving the symlink dangling. Stop it while systemd still has
|
||||
# it loaded (an active unit stays loaded without its file), then take both the
|
||||
# symlink and its [Install] alias. Nothing can start it at boot afterwards.
|
||||
retire_llama_medium_unit() {
|
||||
local trash="$HOME/.AGENTTRASH/20260920-llama-layersplit-frank-medium"
|
||||
run_privileged systemctl stop frank-medium.service 2>/dev/null || true
|
||||
if [[ -L /etc/systemd/system/frank-medium.service ||
|
||||
-L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
[[ -L /etc/systemd/system/frank-medium.service ]] &&
|
||||
run_privileged mv -- /etc/systemd/system/frank-medium.service \
|
||||
"$trash/etc-frank-medium.service.symlink"
|
||||
[[ -L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]] &&
|
||||
run_privileged mv -- /etc/systemd/system/multi-user.target.wants/frank-medium.service \
|
||||
"$trash/etc-frank-medium.wants.symlink"
|
||||
echo "retired llama root unit frank-medium.service -> $trash"
|
||||
fi
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_medium() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
# Refuse rather than restart-loop. A container whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the endpoint
|
||||
# dark with nothing in the deploy log to say why.
|
||||
local missing=0
|
||||
if [[ ! -x "$LLM_MEDIUM_BIN" ]]; then
|
||||
echo "ERROR: llama-server absent at $LLM_MEDIUM_BIN" >&2
|
||||
echo "frank-medium: build it ON the host (gfx1201): bash scripts/build-llama-rdna4.sh" >&2
|
||||
if ! podman image exists "$LLM_MEDIUM_IMAGE_LOCAL"; then
|
||||
if podman image exists "$LLM_MEDIUM_IMAGE"; then
|
||||
podman tag "$LLM_MEDIUM_IMAGE" "$LLM_MEDIUM_IMAGE_LOCAL"
|
||||
else
|
||||
echo "ERROR: radiance image absent ($LLM_MEDIUM_IMAGE / $LLM_MEDIUM_IMAGE_LOCAL)" >&2
|
||||
echo "frank-medium: podman pull $LLM_MEDIUM_IMAGE && podman tag $LLM_MEDIUM_IMAGE $LLM_MEDIUM_IMAGE_LOCAL" >&2
|
||||
missing=1
|
||||
fi
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_SNAP/config.json" ]]; then
|
||||
echo "ERROR: MXFP4 checkpoint absent at $LLM_MEDIUM_SNAP" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_MODEL" ]]; then
|
||||
echo "ERROR: target model absent at $LLM_MEDIUM_MODEL" >&2
|
||||
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
|
||||
if [[ ! -f "$LLM_MEDIUM_DRAFTER/config.json" ]]; then
|
||||
echo "ERROR: DFlash2-FP8 drafter absent at $LLM_MEDIUM_DRAFTER" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_DRAFT" ]]; then
|
||||
echo "ERROR: DFlash2 drafter absent at $LLM_MEDIUM_DRAFT" >&2
|
||||
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
|
||||
# The patches clone is mounted and its scripts run at EVERY container start;
|
||||
# the chat template is the fleet's reasoning-format contract.
|
||||
if [[ ! -f "$LLM_MEDIUM_PATCHES/qwen-fixed-v22.3.jinja" ]]; then
|
||||
echo "ERROR: pinned patch clone / chat template absent at $LLM_MEDIUM_PATCHES" >&2
|
||||
echo "frank-medium: git clone https://github.com/ggz14/radiance-vllm-mxfp4 $LLM_MEDIUM_PATCHES" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_MMPROJ" ]]; then
|
||||
echo "ERROR: vision projector absent at $LLM_MEDIUM_MMPROJ" >&2
|
||||
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
|
||||
# The image's shipped r4d.so predates the gated-delta-net overflow fix and
|
||||
# NaNs this model, so the patched build is not optional.
|
||||
if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then
|
||||
echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2
|
||||
missing=1
|
||||
fi
|
||||
[[ "$missing" -eq 0 ]] || return 1
|
||||
mkdir -p "$LLM_MEDIUM_CACHE"
|
||||
|
||||
retire_vllm_medium_quadlets
|
||||
# The thing holding 7770 is now THIS service: on a re-deploy the previous
|
||||
# llama-server is still bound to it, so waiting for the port without
|
||||
# stopping the unit deadlocks the deploy (measured 2026-09-16: "FAIL: port
|
||||
# 7770 is still held after 90s", with the old server happily serving a
|
||||
# scrape). Stop the unit first, then sweep a stray manual run of the same
|
||||
# binary -- that is how a benchmark harness leaves the port busy. `pkill -x
|
||||
# llama-server` matches the process name exactly and can never match the
|
||||
# deploying shell; never `pkill -f` here.
|
||||
run_privileged systemctl stop frank-medium.service 2>/dev/null || true
|
||||
retire_vllm_replica_quadlets
|
||||
retire_llama_medium_unit
|
||||
# 7770 is a host-network bind: nothing new can be created until the old
|
||||
# holder released it (the same gate as the 2026-09-16 cutover, where the
|
||||
# retired proxy held the port and the new engine failed at create). Sweep a
|
||||
# stray manual llama-server too — that is how a bench harness leaves the
|
||||
# port busy. `pkill -x` matches the process name exactly and can never
|
||||
# match the deploying shell; never `pkill -f` here.
|
||||
pkill -x llama-server 2>/dev/null || true
|
||||
pkill -x VLLM::EngineCor 2>/dev/null || true
|
||||
sleep 2
|
||||
# pasta released the retired proxy's socket at container teardown, and a
|
||||
# stopped engine lingers in TIME_WAIT just past that. Same gate as
|
||||
# frank-large's cutover (2026-09-16 addressed-in-use failure).
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-medium
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-medium.service
|
||||
run_privileged systemctl restart frank-medium.service
|
||||
wait_for_health frank-medium http://127.0.0.1:7770/health 900
|
||||
|
||||
install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence is the unit's [Install] WantedBy=default.target.
|
||||
systemctl --user restart frank-medium.service
|
||||
# Triton/inductor compile on a cold cache runs minutes before the engine
|
||||
# loads ~9.5 GiB per card; HealthStartPeriod in the unit is 1800s to match.
|
||||
wait_container_healthy frank-medium 1800 || {
|
||||
podman logs --tail 40 frank-medium 2>&1 || true
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
|
||||
@@ -180,17 +180,16 @@ for path in "${CHANGED[@]}"; do
|
||||
scripts/mount-entropy.sh)
|
||||
add_test tests/mount_entropy.bats
|
||||
;;
|
||||
# frank-medium is llama.cpp on the two R9700s as a ROOT unit since
|
||||
# 2026-09-16: a native gfx1201 HIP binary, so (like frank-small) its
|
||||
# env, unit and serve script all live in scripts/, and this case has to
|
||||
# precede the generic scripts/*.service fallback below. The retired
|
||||
# vllm-radiance service dir is under dankserver/.AGENTTRASH/, which is
|
||||
# archive material and maps to nothing.
|
||||
scripts/frank-medium.*|scripts/serve-frank-medium.sh)
|
||||
# frank-medium is a rootless quadlet again since the 2026-09-20
|
||||
# tensor-parallel cutover: the TP=2 radiance unit, its entrypoint and
|
||||
# its healthcheck all live under dankserver/frank-medium/, and the
|
||||
# container mounts the latter two straight out of the checkout. The
|
||||
# llama.cpp trio that held this endpoint before that date is archive
|
||||
# material in scripts/.AGENTTRASH/ and maps to nothing.
|
||||
dankserver/frank-medium/*)
|
||||
add_test tests/frank_medium_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
|
||||
Reference in New Issue
Block a user