feat(frank-medium): cut the endpoint over to radiance TP=2
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
This commit is contained in:
Frank
+4
-4
@@ -4,10 +4,10 @@
|
||||
Runs INSIDE the container. The vLLM /health endpoint only answers once the
|
||||
engine is up (weights loaded, cudagraphs captured), so a healthy container is
|
||||
a serving container, not merely a live process. PORT selects the endpoint so
|
||||
the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can
|
||||
share this probe. The HealthCmd in the quadlet splits on spaces into an argv
|
||||
array (podman 5.x v1 generator), which is why this lives in a file rather
|
||||
than an inline python one-liner.
|
||||
the probe follows the unit's Environment=PORT rather than hard-coding the
|
||||
fleet port. The HealthCmd in the quadlet splits on spaces into an argv array
|
||||
(podman 5.x v1 generator), which is why this lives in a file rather than an
|
||||
inline python one-liner.
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
+28
-17
@@ -3,30 +3,32 @@
|
||||
# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15
|
||||
# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3
|
||||
# image + the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20:
|
||||
# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image +
|
||||
# the ggz14/radiance-vllm-mxfp4 patches) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
|
||||
# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp
|
||||
# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF;
|
||||
# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the
|
||||
# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production
|
||||
# configuration of the upstream repo, on our exact hardware signature
|
||||
# (2x7551-32624, see the KV pin below).
|
||||
# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot
|
||||
# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7).
|
||||
# It replaced the mainline llama.cpp layer-split deployment, which served the
|
||||
# same alias from the orcarouter Q8_0 GGUF across the same two cards.
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
# what this unit does too — no PublishPort. The fleet contract is unchanged:
|
||||
# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
|
||||
#
|
||||
# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
|
||||
# process across both R9700s. It cannot launch a collective here — RCCL's
|
||||
# kernels need PCIe atomics and the second card sits behind chipset bridges
|
||||
# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
|
||||
# two-replica trio in this directory (frank-medium-a/b/proxy) and
|
||||
# deploy-remote.sh deliberately does not install this unit. Keep it for a
|
||||
# host whose second card is on an atomic-capable root port; the full evidence
|
||||
# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
|
||||
# TP=2 IS INSTALLABLE HERE NOW, and it won on measurement. Until 2026-09-20
|
||||
# this unit was archived un-installed because the second card sat behind B650
|
||||
# chipset bridges advertising `AtomicOpsCap: Routing-`, which stopped RCCL
|
||||
# kernel dispatch. The board swap (ProArt B850-Creator WiFi NEO) put both
|
||||
# cards on CPU GPP root ports — Gen5 x8 each, every bridge Routing+, both
|
||||
# endpoints ReqEn+ — and the collective-free R4D path runs at TP=2 without
|
||||
# complaint. The conversation battery (scripts/bench-conversation.py, salted,
|
||||
# same 10 prompts, 2026-09-20) measured this shape against the incumbent:
|
||||
# decode geomean 95.8 t/s vs 45.7, mean TTFT 0.21 s vs 0.72 s, and 259 t/s
|
||||
# aggregate against 4 concurrent streams vs 71.7. Evidence and the llama-side
|
||||
# caveats: serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md and
|
||||
# Dual-R9700-Cooperation.md.
|
||||
#
|
||||
# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
|
||||
# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
|
||||
@@ -92,6 +94,15 @@ Environment=TP=2
|
||||
Environment=MAXSEQS=8
|
||||
Environment=CHUNK=8192
|
||||
Environment=MAXLEN=262144
|
||||
# MAXLEN is the whole pool shared across the MAXSEQS slots: 8 concurrent
|
||||
# sessions split 262,144 tokens between them, so a full-window request
|
||||
# excludes the others until it finishes (vLLM preempts rather than refuses).
|
||||
# The llama deployment it replaced advertised 327,680 across 2 slots.
|
||||
#
|
||||
# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the
|
||||
# launcher loads no vision projector, so an input_audio/image request is a
|
||||
# 400 here. Vision traffic routes to frank-small on dankclaw (harness
|
||||
# modelRoles vision -> frank-small).
|
||||
Environment=GPU_UTIL=0.98
|
||||
# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
|
||||
# every published number (224 t/s single-stream decode, 4749-4810 t/s
|
||||
+8
-11
@@ -369,11 +369,14 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
_add(services, "node-exporter")
|
||||
return services
|
||||
if len(parts) >= 2 and parts[1] == "frank-medium":
|
||||
# frank-medium is a ROOT llama.cpp unit since 2026-09-16: its
|
||||
# runtime is a native gfx1201 binary, not a container, so there is
|
||||
# nothing under dankserver/frank-medium/ to plan any more. The
|
||||
# retired vllm-radiance quadlets live in .AGENTTRASH/, which is
|
||||
# archive material and plans nothing.
|
||||
# frank-medium is a CONTAINER again since 2026-09-20: the TP=2
|
||||
# radiance Quadlet, its entrypoint and its healthcheck all live
|
||||
# under this directory, and the container mounts the latter two
|
||||
# straight out of this checkout — so any file here changes deployed
|
||||
# state. (It was a root llama.cpp unit from 2026-09-16, whose trio
|
||||
# lived in scripts/; that shape is archived in
|
||||
# scripts/.AGENTTRASH/20260920-llama-layersplit/ and plans nothing.)
|
||||
_add(services, "frank-medium")
|
||||
return services
|
||||
return services
|
||||
|
||||
@@ -405,12 +408,6 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-small")
|
||||
if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}:
|
||||
# llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve
|
||||
# files here the same way. The files that BUILD and FETCH its
|
||||
# runtime and payload are host-side one-offs: they change no
|
||||
# deployed state, so they select nothing.
|
||||
_add(services, "frank-medium")
|
||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
||||
# CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
|
||||
# serve files here the same way: a root unit on a host-built
|
||||
|
||||
+102
-54
@@ -1137,31 +1137,43 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a
|
||||
# ROOT unit since the 2026-09-16 cutover: the runtime is a native gfx1201 HIP
|
||||
# binary, so there is no image to install and no generated unit to dodge. It
|
||||
# replaces the vllm-radiance replica pair + routing proxy, which held this
|
||||
# endpoint for a day and could never use both cards for one model (TP=2 needs
|
||||
# an RCCL collective, and the second card cannot dispatch one).
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a
|
||||
# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2)
|
||||
# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter.
|
||||
# It replaced mainline llama.cpp's layer split, which held the endpoint since
|
||||
# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a
|
||||
# third of the time-to-first-token with 4x the concurrent throughput
|
||||
# (scripts/bench-conversation.py, 2026-09-20).
|
||||
#
|
||||
# Layer split in ONE process is what the successor does instead: layers are
|
||||
# distributed across the cards and only the activation at the boundary crosses
|
||||
# PCIe, once per forward pass. That is the "pipeline parallelism" the endpoint
|
||||
# wanted, with no collective anywhere in it.
|
||||
LLM_MEDIUM_BIN="${LLM_MEDIUM_BIN:-/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}"
|
||||
LLM_MEDIUM_MODEL="${LLM_MEDIUM_MODEL:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/Qwen3.8-27B-Uncensored-Q8_0.gguf}"
|
||||
LLM_MEDIUM_DRAFT="${LLM_MEDIUM_DRAFT:-/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
LLM_MEDIUM_MMPROJ="${LLM_MEDIUM_MMPROJ:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/mmproj-Qwen3.8-27B-Uncensored-f16.gguf}"
|
||||
|
||||
# The three vLLM user units are retired HERE, not left to rot: the proxy
|
||||
# published 7770 on the host network, so the llama-server below cannot bind
|
||||
# until they are stopped and the port is released. Never `rm` — the unit files
|
||||
# move into the home trash; the images stay in local storage.
|
||||
retire_vllm_medium_quadlets() {
|
||||
# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this
|
||||
# unit because TP=2 could not dispatch a collective on the second card (B650
|
||||
# chipset bridges, AtomicOpsCap: Routing-), and the two-replica + routing-proxy
|
||||
# trio was the workaround. The board swap cleared the atomics wall, so one
|
||||
# TP=2 process is both possible and the fastest shape; the replicas and their
|
||||
# affinity proxy stay retired.
|
||||
#
|
||||
# HOST ASSETS. Nothing here installs the image's payloads: the checkpoints, the
|
||||
# pinned radiance patch clone, the patched libr4d build and the Triton compile
|
||||
# cache live outside the checkout (see the unit file's mount comments). The
|
||||
# gates below refuse a deploy that would start a container guaranteed to die.
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}"
|
||||
LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}"
|
||||
LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}"
|
||||
LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}"
|
||||
|
||||
# The three replica/proxy user units from the 2026-09-16 era are retired HERE,
|
||||
# never left to rot: they must not exist alongside the TP=2 unit (the proxy
|
||||
# published 7770 on the host network). Never `rm` — unit files move into the
|
||||
# home trash; the images stay in local storage. `frank-medium` itself is NOT in
|
||||
# the list: it is the unit this handler installs.
|
||||
retire_vllm_replica_quadlets() {
|
||||
local trash="$HOME/.AGENTTRASH/20260916-vllm-radiance-retire" unit
|
||||
systemctl --user stop frank-medium-a.service frank-medium-b.service \
|
||||
frank-medium-proxy.service 2>/dev/null || true
|
||||
for unit in frank-medium-a frank-medium-b frank-medium-proxy frank-medium; do
|
||||
for unit in frank-medium-a frank-medium-b frank-medium-proxy; do
|
||||
if [[ -e "$quadlet_unit_dir/$unit.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/$unit.container" "$trash/$unit.container"
|
||||
@@ -1171,54 +1183,90 @@ retire_vllm_medium_quadlets() {
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
|
||||
# The llama.cpp root unit is retired HERE, into the home trash, never `rm`:
|
||||
# /etc/systemd/system/frank-medium.service is a relink_unit symlink into this
|
||||
# checkout, and the checkout pull that precedes this handler already moved the
|
||||
# repo file away, leaving the symlink dangling. Stop it while systemd still has
|
||||
# it loaded (an active unit stays loaded without its file), then take both the
|
||||
# symlink and its [Install] alias. Nothing can start it at boot afterwards.
|
||||
retire_llama_medium_unit() {
|
||||
local trash="$HOME/.AGENTTRASH/20260920-llama-layersplit-frank-medium"
|
||||
run_privileged systemctl stop frank-medium.service 2>/dev/null || true
|
||||
if [[ -L /etc/systemd/system/frank-medium.service ||
|
||||
-L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]]; then
|
||||
mkdir -p "$trash"
|
||||
[[ -L /etc/systemd/system/frank-medium.service ]] &&
|
||||
run_privileged mv -- /etc/systemd/system/frank-medium.service \
|
||||
"$trash/etc-frank-medium.service.symlink"
|
||||
[[ -L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]] &&
|
||||
run_privileged mv -- /etc/systemd/system/multi-user.target.wants/frank-medium.service \
|
||||
"$trash/etc-frank-medium.wants.symlink"
|
||||
echo "retired llama root unit frank-medium.service -> $trash"
|
||||
fi
|
||||
run_privileged systemctl daemon-reload
|
||||
}
|
||||
|
||||
deploy_llm_medium() {
|
||||
# Refuse rather than restart-loop. A unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the deploy log to say why.
|
||||
# Refuse rather than restart-loop. A container whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the endpoint
|
||||
# dark with nothing in the deploy log to say why.
|
||||
local missing=0
|
||||
if [[ ! -x "$LLM_MEDIUM_BIN" ]]; then
|
||||
echo "ERROR: llama-server absent at $LLM_MEDIUM_BIN" >&2
|
||||
echo "frank-medium: build it ON the host (gfx1201): bash scripts/build-llama-rdna4.sh" >&2
|
||||
if ! podman image exists "$LLM_MEDIUM_IMAGE_LOCAL"; then
|
||||
if podman image exists "$LLM_MEDIUM_IMAGE"; then
|
||||
podman tag "$LLM_MEDIUM_IMAGE" "$LLM_MEDIUM_IMAGE_LOCAL"
|
||||
else
|
||||
echo "ERROR: radiance image absent ($LLM_MEDIUM_IMAGE / $LLM_MEDIUM_IMAGE_LOCAL)" >&2
|
||||
echo "frank-medium: podman pull $LLM_MEDIUM_IMAGE && podman tag $LLM_MEDIUM_IMAGE $LLM_MEDIUM_IMAGE_LOCAL" >&2
|
||||
missing=1
|
||||
fi
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_SNAP/config.json" ]]; then
|
||||
echo "ERROR: MXFP4 checkpoint absent at $LLM_MEDIUM_SNAP" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_MODEL" ]]; then
|
||||
echo "ERROR: target model absent at $LLM_MEDIUM_MODEL" >&2
|
||||
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
|
||||
if [[ ! -f "$LLM_MEDIUM_DRAFTER/config.json" ]]; then
|
||||
echo "ERROR: DFlash2-FP8 drafter absent at $LLM_MEDIUM_DRAFTER" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_DRAFT" ]]; then
|
||||
echo "ERROR: DFlash2 drafter absent at $LLM_MEDIUM_DRAFT" >&2
|
||||
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
|
||||
# The patches clone is mounted and its scripts run at EVERY container start;
|
||||
# the chat template is the fleet's reasoning-format contract.
|
||||
if [[ ! -f "$LLM_MEDIUM_PATCHES/qwen-fixed-v22.3.jinja" ]]; then
|
||||
echo "ERROR: pinned patch clone / chat template absent at $LLM_MEDIUM_PATCHES" >&2
|
||||
echo "frank-medium: git clone https://github.com/ggz14/radiance-vllm-mxfp4 $LLM_MEDIUM_PATCHES" >&2
|
||||
missing=1
|
||||
fi
|
||||
if [[ ! -f "$LLM_MEDIUM_MMPROJ" ]]; then
|
||||
echo "ERROR: vision projector absent at $LLM_MEDIUM_MMPROJ" >&2
|
||||
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
|
||||
# The image's shipped r4d.so predates the gated-delta-net overflow fix and
|
||||
# NaNs this model, so the patched build is not optional.
|
||||
if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then
|
||||
echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2
|
||||
missing=1
|
||||
fi
|
||||
[[ "$missing" -eq 0 ]] || return 1
|
||||
mkdir -p "$LLM_MEDIUM_CACHE"
|
||||
|
||||
retire_vllm_medium_quadlets
|
||||
# The thing holding 7770 is now THIS service: on a re-deploy the previous
|
||||
# llama-server is still bound to it, so waiting for the port without
|
||||
# stopping the unit deadlocks the deploy (measured 2026-09-16: "FAIL: port
|
||||
# 7770 is still held after 90s", with the old server happily serving a
|
||||
# scrape). Stop the unit first, then sweep a stray manual run of the same
|
||||
# binary -- that is how a benchmark harness leaves the port busy. `pkill -x
|
||||
# llama-server` matches the process name exactly and can never match the
|
||||
# deploying shell; never `pkill -f` here.
|
||||
run_privileged systemctl stop frank-medium.service 2>/dev/null || true
|
||||
retire_vllm_replica_quadlets
|
||||
retire_llama_medium_unit
|
||||
# 7770 is a host-network bind: nothing new can be created until the old
|
||||
# holder released it (the same gate as the 2026-09-16 cutover, where the
|
||||
# retired proxy held the port and the new engine failed at create). Sweep a
|
||||
# stray manual llama-server too — that is how a bench harness leaves the
|
||||
# port busy. `pkill -x` matches the process name exactly and can never
|
||||
# match the deploying shell; never `pkill -f` here.
|
||||
pkill -x llama-server 2>/dev/null || true
|
||||
pkill -x VLLM::EngineCor 2>/dev/null || true
|
||||
sleep 2
|
||||
# pasta released the retired proxy's socket at container teardown, and a
|
||||
# stopped engine lingers in TIME_WAIT just past that. Same gate as
|
||||
# frank-large's cutover (2026-09-16 addressed-in-use failure).
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-medium
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-medium.service
|
||||
run_privileged systemctl restart frank-medium.service
|
||||
wait_for_health frank-medium http://127.0.0.1:7770/health 900
|
||||
|
||||
install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium
|
||||
# No enable: generated units cannot be enabled on this host (v1 quadlet
|
||||
# generator); boot persistence is the unit's [Install] WantedBy=default.target.
|
||||
systemctl --user restart frank-medium.service
|
||||
# Triton/inductor compile on a cold cache runs minutes before the engine
|
||||
# loads ~9.5 GiB per card; HealthStartPeriod in the unit is 1800s to match.
|
||||
wait_container_healthy frank-medium 1800 || {
|
||||
podman logs --tail 40 frank-medium 2>&1 || true
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
|
||||
@@ -180,17 +180,16 @@ for path in "${CHANGED[@]}"; do
|
||||
scripts/mount-entropy.sh)
|
||||
add_test tests/mount_entropy.bats
|
||||
;;
|
||||
# frank-medium is llama.cpp on the two R9700s as a ROOT unit since
|
||||
# 2026-09-16: a native gfx1201 HIP binary, so (like frank-small) its
|
||||
# env, unit and serve script all live in scripts/, and this case has to
|
||||
# precede the generic scripts/*.service fallback below. The retired
|
||||
# vllm-radiance service dir is under dankserver/.AGENTTRASH/, which is
|
||||
# archive material and maps to nothing.
|
||||
scripts/frank-medium.*|scripts/serve-frank-medium.sh)
|
||||
# frank-medium is a rootless quadlet again since the 2026-09-20
|
||||
# tensor-parallel cutover: the TP=2 radiance unit, its entrypoint and
|
||||
# its healthcheck all live under dankserver/frank-medium/, and the
|
||||
# container mounts the latter two straight out of the checkout. The
|
||||
# llama.cpp trio that held this endpoint before that date is archive
|
||||
# material in scripts/.AGENTTRASH/ and maps to nothing.
|
||||
dankserver/frank-medium/*)
|
||||
add_test tests/frank_medium_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
# frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
|
||||
@@ -64,10 +64,10 @@
|
||||
|
||||
<h2>frank-medium — the 27B generalist</h2>
|
||||
<ul>
|
||||
<li><strong>Model:</strong> Swift-Qwen3.8-27B-Uncensored at UD-Q8_K_XL, with its own MTP head, on mainline llama.cpp.</li>
|
||||
<li><strong>Hardware:</strong> Dankserver's two Radeon R9700s, one process split 35:30 across the cards.</li>
|
||||
<li><strong>Best for:</strong> general chat and reasoning when the big brain is busy; the biggest pure-language model we run.</li>
|
||||
<li><strong>Capacity:</strong> 393k-token context over two conversations. Also sees images (BF16 projector).</li>
|
||||
<li><strong>Model:</strong> Qwen3.8-27B-Uncensored at MXFP4, on vllm-radiance with a DFlash2 draft head.</li>
|
||||
<li><strong>Hardware:</strong> Dankserver's two Radeon R9700s as ONE tensor-parallel engine (TP=2).</li>
|
||||
<li><strong>Best for:</strong> general chat and reasoning when the big brain is busy; the biggest pure-language model we run, and the fastest we have measured.</li>
|
||||
<li><strong>Capacity:</strong> 262k-token context, shared by up to eight conversations at once. Text only — pictures go to <code>frank-small</code>.</li>
|
||||
</ul>
|
||||
|
||||
<p class="foot-note">Which one to pick? Screenshots and photos: <code>frank-small</code>.
|
||||
|
||||
@@ -244,7 +244,7 @@
|
||||
{
|
||||
"name": "frank-medium",
|
||||
"category": "user",
|
||||
"description": "The 27B generalist. ajgazin Swift-Qwen3.8-27B-Uncensored at UD-Q8_K_XL with its own MTP head, mainline llama.cpp in one 35:30 layer-split process across Dankserver's two R9700s, 393k-token context.",
|
||||
"description": "The 27B generalist. Qwen3.8-27B-Uncensored at MXFP4 on vllm-radiance, one tensor-parallel process across Dankserver's two R9700s with a DFlash2 drafter; 262k-token context shared by up to eight conversations.",
|
||||
"url": "",
|
||||
"docs": "/docs#models",
|
||||
"probe": "192.168.0.235:7770",
|
||||
|
||||
@@ -21,10 +21,18 @@ load helpers
|
||||
[[ "$output" != *"tests/homepage_pages.bats"* ]]
|
||||
}
|
||||
|
||||
@test "changed-test wrapper maps the frank-medium trio to its focused tests" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/frank-medium.env
|
||||
@test "changed-test wrapper maps the frank-medium dir to its focused tests" {
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths \
|
||||
dankserver/frank-medium/quadlet/frank-medium.container
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" == *"tests/frank_medium_unit.bats"* ]]
|
||||
[[ "$output" == *"tests/deployment_scoping.bats"* ]]
|
||||
[[ "$output" == *"tests/docs_completeness.bats"* ]]
|
||||
[[ "$output" != *"tests/homepage_pages.bats"* ]]
|
||||
# The llama trio's old path must no longer select the container seam: the
|
||||
# mapping was removed with the runtime, and a stale case here would keep
|
||||
# firing tests for an archived service.
|
||||
run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/frank-medium.env
|
||||
[ "$status" -eq 0 ]
|
||||
[[ "$output" != *"tests/frank_medium_unit.bats"* ]]
|
||||
}
|
||||
|
||||
@@ -46,18 +46,25 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "deployment planner maps the frank-medium scripts trio to frank-medium only" {
|
||||
@test "deployment planner maps the frank-medium container dir to frank-medium only" {
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/frank-medium.env scripts/frank-medium.service scripts/serve-frank-medium.sh
|
||||
dankserver/frank-medium/quadlet/frank-medium.container \
|
||||
dankserver/frank-medium/radiance-entrypoint.sh \
|
||||
dankserver/frank-medium/frank-medium-healthcheck.py
|
||||
[ "$status" -eq 0 ]
|
||||
[ "$output" = "frank-medium" ]
|
||||
|
||||
# The path is not the assertion: with the vllm-radiance service dir
|
||||
# retired into the trash, a planner rule still naming dankserver/frank-medium
|
||||
# would plan nothing and look identical to a correct one. Assert the
|
||||
# invariant (the dir is gone) as well as the mapping above.
|
||||
[ ! -d "$REPO_ROOT/dankserver/frank-medium" ]
|
||||
run python3 "$PLANNER" --target dankserver --paths dankserver/frank-medium/quadlet/frank-medium-a.container
|
||||
# The path is not the assertion: both retired shapes are still in the tree
|
||||
# as archive, and a rule matching either would restart a service nobody
|
||||
# runs. Assert the negatives explicitly, plus the llama trio's old paths,
|
||||
# whose mapping was removed with the runtime.
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-a.container
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
run python3 "$PLANNER" --target dankserver --paths \
|
||||
scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.env.llamacpp-layersplit \
|
||||
scripts/frank-medium.env
|
||||
[ "$status" -eq 0 ]
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
+165
-323
@@ -1,351 +1,193 @@
|
||||
#!/usr/bin/env bats
|
||||
# Seam: the frank-medium mainline-llama.cpp root unit on dankserver.
|
||||
# frank-medium became a native llama.cpp service on 2026-09-16, replacing the
|
||||
# vllm-radiance replica pair + routing proxy that served this endpoint for a
|
||||
# day. What this seam pins is what the flags alone do not say: boot persistence
|
||||
# for a NATIVE runtime (no generated quadlet is involved), that BOTH cards work
|
||||
# on ONE model through a layer split with no collective in the path, that the
|
||||
# DFlash2 drafter is wired with a depth its block size actually allows, that the
|
||||
# runtime and payload are pinned rather than floated, and that one slot's KV
|
||||
# still fits a 32 GiB card beside the weights it holds.
|
||||
# Seam: the frank-medium vllm-radiance container on dankserver.
|
||||
# frank-medium became a CONTAINER again on 2026-09-20, when the board swap put
|
||||
# both R9700s on CPU root ports and a measured three-arm comparison retired the
|
||||
# mainline-llama.cpp layer split (scripts/.AGENTTRASH/20260920-llama-layersplit/).
|
||||
# What this seam pins is what the flags alone do not say: boot persistence for a
|
||||
# GENERATED unit (this host's v1 quadlet generator cannot enable units, so the
|
||||
# [Install] section is the only thing that survives a reboot), that BOTH cards
|
||||
# work on ONE model through tensor parallelism — the thing the pre-2026-09-20
|
||||
# hardware could not do — the KV pin's coupling to the batch shape it was
|
||||
# calibrated at, and that the pool still holds one full-window request.
|
||||
|
||||
load helpers
|
||||
|
||||
UNIT="$REPO_ROOT/scripts/frank-medium.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-medium.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-medium.env"
|
||||
BUILD="$REPO_ROOT/scripts/build-llama-rdna4.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-medium-gguf.sh"
|
||||
UNIT="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
ENTRY="$REPO_ROOT/dankserver/frank-medium/radiance-entrypoint.sh"
|
||||
HEALTH="$REPO_ROOT/dankserver/frank-medium/frank-medium-healthcheck.py"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
LLAMA_TRASH="$REPO_ROOT/scripts/.AGENTTRASH/20260920-llama-layersplit"
|
||||
REPLICA_TRASH="$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance"
|
||||
|
||||
# Measured constants (rocm-smi / HF blob sizes), used by the fit arithmetic.
|
||||
CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it
|
||||
MODEL_BYTES=29047084416 # orcarouter Q8_0 target
|
||||
DRAFT_BYTES=1143006816 # DFlash2 Q4_K_M drafter
|
||||
|
||||
@test "frank-medium's unit, serve script, env and its build/fetch scripts exist" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -f "$SERVE" ]
|
||||
[ -f "$ENV_FILE" ]
|
||||
[ -f "$BUILD" ]
|
||||
[ -f "$FETCH" ]
|
||||
# Measured constants, used by the fit arithmetic.
|
||||
CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it (32624 MiB)
|
||||
KV_BYTES_PER_TOKEN=70656 # --kv-cache-memory 18563072000 => "262,720 tokens"
|
||||
env_value() {
|
||||
sed -nE "s/^Environment=$1=(-?[0-9]+)$/\1/p" "$UNIT" | head -n1
|
||||
}
|
||||
# GPU_UTIL is a fraction, not a count; the integer reader would drop it.
|
||||
env_ratio() {
|
||||
sed -nE "s/^Environment=$1=([0-9.]+)$/\1/p" "$UNIT" | head -n1
|
||||
}
|
||||
|
||||
@test "frank-medium is boot-persistent, self-healing and unprivileged" {
|
||||
grep -q '^WantedBy=multi-user.target$' "$UNIT"
|
||||
@test "frank-medium's unit, entrypoint and healthcheck exist" {
|
||||
[ -f "$UNIT" ]
|
||||
[ -x "$ENTRY" ]
|
||||
[ -f "$HEALTH" ]
|
||||
}
|
||||
|
||||
@test "frank-medium is boot-persistent and self-healing" {
|
||||
# Generated units cannot be enabled on this host, so WantedBy in the file is
|
||||
# the whole boot story (AGENTS.md: Boot Persistence & Self-Healing).
|
||||
grep -q '^WantedBy=default.target$' "$UNIT"
|
||||
grep -q '^Restart=on-failure$' "$UNIT"
|
||||
grep -q '^RestartSec=' "$UNIT"
|
||||
grep -q '^User=dank$' "$UNIT"
|
||||
grep -q '^LimitMEMLOCK=infinity$' "$UNIT"
|
||||
# First start compiles Triton/inductor kernels before the engine loads ~9.5
|
||||
# GiB per card; a shorter timeout kills a healthy boot.
|
||||
grep -q '^TimeoutStartSec=1800$' "$UNIT"
|
||||
grep -q '^HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py$' "$UNIT"
|
||||
grep -q '^HealthStartPeriod=1800s$' "$UNIT"
|
||||
# Boots must not depend on the registry.
|
||||
grep -q '^Pull=never$' "$UNIT"
|
||||
}
|
||||
|
||||
@test "frank-medium runs the repo's own serve script with the repo's env" {
|
||||
grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-medium.sh$' "$UNIT"
|
||||
grep -q 'FRANK_MEDIUM_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-medium.env' "$UNIT"
|
||||
@test "both R9700s serve ONE model as one tensor-parallel process" {
|
||||
grep -q '^Environment=TP=2$' "$UNIT"
|
||||
grep -q '^Environment=ROCR_VISIBLE_DEVICES=0,1$' "$UNIT"
|
||||
grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT"
|
||||
grep -q -- '--tensor-parallel-size "$TP"' "$ENTRY"
|
||||
# The iGPU (index 2, 2 GiB) must stay out of the visible set: a third rank
|
||||
# would take layers onto system RAM.
|
||||
! grep -q '^Environment=.*VISIBLE_DEVICES=0,1,2' "$UNIT"
|
||||
# TP needs the all-reduce to actually run. R4D is the collective-free path
|
||||
# this host measured; the entrypoint must still select it, and the RCCL
|
||||
# fallback must not be re-disabled by the pre-board-swap workarounds.
|
||||
grep -q 'RADIANCE_USE_R4D_AR' "$ENTRY"
|
||||
grep -q -- '--attention-backend "$ATTN"' "$ENTRY"
|
||||
! grep -q 'NCCL_P2P_DISABLE' "$UNIT"
|
||||
! grep -q 'RCCL_MSCCL_ENABLE' "$UNIT"
|
||||
}
|
||||
|
||||
@test "both R9700s serve ONE model, layer-split inside one process" {
|
||||
grep -q '^SPLIT_MODE=layer$' "$ENV_FILE"
|
||||
grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE"
|
||||
# The split is a layer PAIR: the shares add up to the model's 65 blocks (one
|
||||
# -ts unit per layer). The operator asked for the tensors BALANCED across
|
||||
# the two cards (2026-09-19), which for 65 layers is 32:33 -- the most even
|
||||
# split that exists, with ROCm1 (the last-layers card, which also carries
|
||||
# the drafter and the output layers) holding the extra one. WHICH pair
|
||||
# balances the cards is the fit test's job, which recomputes it from this
|
||||
# file; pinning a literal here would only mean that re-measuring the
|
||||
# balance breaks a test about parallelism.
|
||||
local split_hi split_lo
|
||||
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ]
|
||||
[ $(( split_hi + split_lo )) -eq 65 ]
|
||||
[ $(( split_lo - split_hi )) -le 1 ]
|
||||
grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE"
|
||||
grep -q -- '--device "$DEVICES"' "$SERVE"
|
||||
grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE"
|
||||
# Tensor split is the one shape this host cannot run: it needs a cross-card
|
||||
# collective, and the second card cannot dispatch one (see the env file).
|
||||
! grep -q '^SPLIT_MODE=tensor$' "$ENV_FILE"
|
||||
@test "HIP never reads one of these tunables as its own setting" {
|
||||
# HIP reads bare TEMP, HOST and GPU_MAX_HEAP_SIZE (2026-08-13 incident:
|
||||
# TEMP=1.0 sent the JIT cache to path "1.0" and segfaulted every load).
|
||||
# Serve scripts export env files with `set -a`; a container inherits the
|
||||
# same hazard from Environment=.
|
||||
! grep -qE '^Environment=(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$UNIT"
|
||||
}
|
||||
|
||||
@test "the iGPU is hidden so no layer can land on system RAM" {
|
||||
grep -q '^HIP_VISIBLE_DEVICES=0,1$' "$ENV_FILE"
|
||||
grep -q '^export HIP_VISIBLE_DEVICES$' "$SERVE"
|
||||
@test "the drafter is DFlash2 at the measured depth and the kernel band covers the batch" {
|
||||
grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT"
|
||||
local maxseqs spec chunk
|
||||
spec="$(env_value SPEC)"
|
||||
maxseqs="$(env_value MAXSEQS)"
|
||||
chunk="$(env_value CHUNK)"
|
||||
[ -n "$spec" ] && [ -n "$maxseqs" ] && [ -n "$chunk" ]
|
||||
# DFlash2 drafts at most 7 tokens per step; deeper is a different drafter's
|
||||
# contract and costs acceptance, not just VRAM.
|
||||
[ "$spec" -eq 7 ]
|
||||
# The MXFP4 decode kernel band covers MAXSEQS x (SPEC+1) rows, and the
|
||||
# shipped default covers 64. Exceeding it without the entrypoint's >8 bump
|
||||
# is a launch failure at the worst moment (first concurrent request).
|
||||
[ $(( maxseqs * (spec + 1) )) -le 64 ]
|
||||
grep -q 'RADIANCE_MXFP4_DECODE_MAX_M' "$ENTRY"
|
||||
# The chunk is the AR size gate's input (CHUNK x hidden x 2 bytes); a chunk
|
||||
# larger than this row's silently changes which all-reduce path runs.
|
||||
[ "$chunk" -le 8192 ]
|
||||
}
|
||||
|
||||
@test "the drafter is the DFlash2 sidecar at the measured depth" {
|
||||
# The DFlash2 sidecar is the deployed drafter (operator call 2026-09-19).
|
||||
# n_max 7 is the measured DFlash2 winner -- 43.5 tok/s at 41 % acceptance
|
||||
# against 32.4 for the in-file MTP head at n=3 (the env's table; identical
|
||||
# greedy text at every depth, so this is pure speed). The target file
|
||||
# also carries its own MTP head (nextn_predict_layers=1), so draft-mtp is
|
||||
# one SPEC_TYPE change away; the drafter file stays on disk for that.
|
||||
grep -qE '^SPEC_TYPE=draft-dflash$' "$ENV_FILE"
|
||||
grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE"
|
||||
local n_max
|
||||
n_max="$(sed -nE 's/^SPEC_DRAFT_N_MAX=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$n_max" ] && [ "$n_max" -eq 7 ]
|
||||
# The Q4_K_M rung is the operator's requested drafter quant -- the
|
||||
# drafter's job is throughput, not output quality, and the target
|
||||
# verifies every drafted token. ngram-mod's window is retained in the env
|
||||
# for the chain, so it may not be dropped.
|
||||
grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DFlash2-Q4_K_M\.gguf$' "$ENV_FILE"
|
||||
grep -q '^SPEC_NGRAM_N_MATCH=24$' "$ENV_FILE"
|
||||
# ...and neither is loaded: ngram-mod's flags only go out when the type
|
||||
# contains ngram, so a pure dflash configuration is not handed them.
|
||||
python3 -c "
|
||||
lines = open('$SERVE').read().splitlines()
|
||||
guard = next(i for i, l in enumerate(lines) if '*ngram*' in l)
|
||||
indent = len(lines[guard]) - len(lines[guard].lstrip())
|
||||
end = next(i for i in range(guard + 1, len(lines))
|
||||
if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent)
|
||||
assert '--spec-ngram-mod-n-match' in chr(10).join(lines[guard:end]), 'ngram flags must sit in the ngram branch'
|
||||
"
|
||||
@test "the KV pin is valid only at the shape it was calibrated for" {
|
||||
# The pin is the upstream kv-profiles.tsv row for sig 2x7551-32624 at
|
||||
# maxseqs=8 chunk=8192 maxlen=262144 spec=dflash — exactly this host. It is
|
||||
# not a general-purpose budget: changing MAXSEQS, CHUNK or SPEC_METHOD
|
||||
# invalidates it, so the shape has to move with it or the pin goes to 0
|
||||
# (let the image profile the shape).
|
||||
local kv_mem maxseqs chunk maxlen spec
|
||||
kv_mem="$(env_value KV_MEM)"
|
||||
maxseqs="$(env_value MAXSEQS)"
|
||||
chunk="$(env_value CHUNK)"
|
||||
maxlen="$(env_value MAXLEN)"
|
||||
spec="$(env_value SPEC)"
|
||||
[ -n "$kv_mem" ] && [ -n "$maxseqs" ] && [ -n "$chunk" ] && [ -n "$maxlen" ]
|
||||
grep -q 'kv-profiles.tsv' "$UNIT"
|
||||
[ "$maxseqs" -eq 8 ] && [ "$chunk" -eq 8192 ] && [ "$spec" -eq 7 ]
|
||||
grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT"
|
||||
# Pool arithmetic, from the engine's own boot line: 18563072000 bytes
|
||||
# reserved => "GPU KV cache size: 262,720 tokens". The pool must hold at
|
||||
# least ONE full-window request or the endpoint refuses its own
|
||||
# --max-model-len, and it must fit on one card beside the weights.
|
||||
[ $(( kv_mem / KV_BYTES_PER_TOKEN )) -ge "$maxlen" ]
|
||||
# 9.5 GiB of MXFP4 weights per card + the KV pool, under the 0.98 util cap.
|
||||
local weights=10200547328 # ~9.5 GiB per card of the 19 GB MXFP4 checkpoint
|
||||
local util util_pct floor
|
||||
util="$(env_ratio GPU_UTIL)"
|
||||
[ -n "$util" ]
|
||||
util_pct="${util#0.}"
|
||||
floor=$(( 1024 * 1024 * 1024 )) # 1 GiB per card
|
||||
[ $(( weights + kv_mem / 2 )) -le $(( CARD_BYTES * util_pct / 100 - floor )) ]
|
||||
}
|
||||
|
||||
@test "a draft MODEL is passed only for DFlash types, never for MTP" {
|
||||
# MTP builds its context against the TARGET model (its head lives in that
|
||||
# GGUF). Passing --spec-draft-model alongside it points the draft context at
|
||||
# the drafter's file, which has no MTP layers, and the server dies at load:
|
||||
# context type MTP requested but model doesn't contain MTP layers
|
||||
# failed to create MTP context
|
||||
# So the flag must appear exactly once, inside the dflash branch.
|
||||
python3 -c "
|
||||
lines = open('$SERVE').read().splitlines()
|
||||
guard = next(i for i, l in enumerate(lines) if '*dflash*' in l)
|
||||
indent = len(lines[guard]) - len(lines[guard].lstrip())
|
||||
end = next(i for i in range(guard + 1, len(lines))
|
||||
if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent)
|
||||
block = chr(10).join(lines[guard:end])
|
||||
assert '--spec-draft-model' in block, 'the dflash branch must pass the draft model'
|
||||
assert sum(l.count('--spec-draft-model') for l in lines) == 1, 'only that branch may pass it'
|
||||
print('draft-model wiring ok')
|
||||
"
|
||||
@test "the fleet endpoint contract survives the runtime swap" {
|
||||
grep -q '^Environment=PORT=7770$' "$UNIT"
|
||||
grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT"
|
||||
grep -q -- '--port "$PORT"' "$ENTRY"
|
||||
grep -q -- '--served-model-name $SERVED_NAMES' "$ENTRY"
|
||||
# Host network: vLLM binds the port itself, so no PublishPort (and no
|
||||
# pasta bind, which is what made the retired proxy's port replacement
|
||||
# fail at container create).
|
||||
grep -q -- '--network=host' "$UNIT"
|
||||
! grep -q '^PublishPort=' "$UNIT"
|
||||
# Tools and reasoning are part of the harness contract; both parsers must
|
||||
# be selected, and thinking must land in its own field.
|
||||
grep -q -- '--enable-auto-tool-choice' "$ENTRY"
|
||||
grep -q -- '--tool-call-parser qwen3_coder' "$ENTRY"
|
||||
grep -q -- '--reasoning-parser qwen3' "$ENTRY"
|
||||
grep -q -- '--enable-prefix-caching' "$ENTRY"
|
||||
# Requests that send no sampler fields get the predecessor's defaults.
|
||||
grep -q '"temperature":0.7' "$ENTRY"
|
||||
grep -q '"top_p":0.95' "$ENTRY"
|
||||
}
|
||||
|
||||
@test "the runtime is pinned to a commit and to gfx1201 alone" {
|
||||
# A floating ref would silently move the service between engine versions,
|
||||
# and an unpinned arch would build for a sibling GPU that loads and then
|
||||
# segfaults on these cards.
|
||||
grep -Eq '^LLAMA_REF="\$\{LLAMA_REF:-[0-9a-f]{40}\}"$' "$BUILD"
|
||||
grep -Eq '^HIP_ARCH="\$\{HIP_ARCH:-gfx1201\}"$' "$BUILD"
|
||||
grep -q 'GGML_HIP=ON' "$BUILD"
|
||||
grep -q 'frank-medium.env' "$SERVE"
|
||||
@test "the healthcheck proves the engine serves, not merely that a process lives" {
|
||||
grep -q '"/health"' "$HEALTH" || grep -q '/health' "$HEALTH"
|
||||
grep -q 'status == 200' "$HEALTH"
|
||||
# PORT-aware, and it must not hard-code a replica port.
|
||||
grep -q 'os.environ.get("PORT"' "$HEALTH"
|
||||
! grep -qE '7771|7772' "$HEALTH"
|
||||
}
|
||||
|
||||
@test "the payload is fetched resumably and verified by size and sha256" {
|
||||
grep -q 'Qwen3\.8-27B-Uncensored-Q8_0\.gguf' "$FETCH"
|
||||
grep -q 'Qwen3\.8-27B-DFlash2-Q4_K_M\.gguf' "$FETCH"
|
||||
grep -q -- '-fL -C -' "$FETCH"
|
||||
grep -q 'verifying sizes and sha256' "$FETCH"
|
||||
# The target repo is gated: the script must fail loud without a token
|
||||
# rather than write a 401 body to disk and report success.
|
||||
grep -q 'no HF token is available' "$FETCH"
|
||||
}
|
||||
|
||||
@test "both cards stay balanced and keep the operator's 1.5 GiB floor" {
|
||||
local ctx parallel cache_k cache_v split_hi split_lo batch ubatch
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
|
||||
cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
|
||||
split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")"
|
||||
batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
spec_type="$(sed -nE 's/^SPEC_TYPE=(.+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$spec_type" ]
|
||||
[ -n "$split_hi" ] && [ -n "$split_lo" ] && [ -n "$batch" ] && [ -n "$ubatch" ]
|
||||
[ "$parallel" -ge 1 ]
|
||||
[ "$batch" -ge "$ubatch" ]
|
||||
[ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ]
|
||||
# Per card, calibrated against rocm-smi at the 35:30 / ctx 262144 / batch
|
||||
# 6144 x ubatch 512 / two slots / DFlash2 point, where the two cards
|
||||
# measured 6.44 / 6.90 GiB free. The fit reproduces card 1 to 0.00 GiB
|
||||
# and card 0 to 0.86 GiB at that point; it is a GATE, not a meter -- the
|
||||
# deployed point is re-measured on the host after every split/ctx/batch
|
||||
# move (the current one is 32:33 / 327680 / DFlash2 Q4_K_M, the balanced
|
||||
# split the operator asked for on 2026-09-19 -- see the TENSOR_SPLIT
|
||||
# header in frank-medium.env).
|
||||
#
|
||||
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
|
||||
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
|
||||
# x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit
|
||||
# of compute buffers
|
||||
# card 1 = the same for its own share, plus the drafter. The drafter's
|
||||
# charge depends on WHAT it is: DFlash2 is a sidecar file, so
|
||||
# its weights (2.06e9 B) plus window-bounded KV (20480 B/pos x
|
||||
# 2048) plus ~0.89 GiB of constant buffers; MTP's head lives in
|
||||
# the target file (already charged above), so its cost is one
|
||||
# nextn layer's KV (4096 B/position) plus small buffers. Its
|
||||
# compute buffers are the heavier ones either way, because it
|
||||
# holds the output layers and the drafter's graphs.
|
||||
#
|
||||
# BOTH ubatch coefficients are measured at the deployed shape, from the
|
||||
# 1536 -> 512 move: 1.30 GiB on card 0 and 2.41 GiB on card 1. That
|
||||
# asymmetry is why shrinking the batch moved the balance toward card 1 and
|
||||
# cost card 0 a layer -- the split and the batch cannot be chosen apart.
|
||||
#
|
||||
# The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus
|
||||
# 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary
|
||||
# around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention
|
||||
# one -- which makes the drift check below CONSERVATIVE: it overstates the gap
|
||||
# a one-layer move creates, and so trips on moves a measurement might bless.
|
||||
# That is the intent: moving the split means re-measuring, not re-arguing.
|
||||
#
|
||||
# The slack is the DEPLOYED gap, rounded up: at each shape the balance
|
||||
# cannot be tuned finer than one layer, so the deployed split's gap IS
|
||||
# what the check must accept, and a tighter slack would only reject it.
|
||||
#
|
||||
# What this model cannot see is CONTIGUITY -- the mechanism by which the
|
||||
# DSpark-era 8192/2048 failed, a 2020 MiB pp buffer that found no room while
|
||||
# every total looked fine. Total bytes are not the whole story for one big
|
||||
# allocation.
|
||||
local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack
|
||||
local card_hi card_lo diff
|
||||
local mmproj=931145984 # mmproj-Qwen3.8-27B-Uncensored-f16.gguf, on the primary device
|
||||
# 1.5 GiB was the floor until the 2x196k shape: the 34:31 / 393216 /
|
||||
# MTP point measured 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
|
||||
# floor is the measured 1.08 GiB rounded down. The 32:33 / 327680 /
|
||||
# DFlash2 point is roomier than that (fit: 6.97 / 4.89 GiB free), so the
|
||||
# floor stays at the measured 1.0 GiB -- if a large allocation fails at
|
||||
# load, the next lever is a ctx cut, not a floor change.
|
||||
floor=$(( 1024 * 1024 * 1024 )) # 1.0 GiB per card
|
||||
# The deployed point (32:33 at ctx 327680 with DFlash2) has a fit gap of
|
||||
# 2082 MiB -- the balanced split the operator asked for, not the
|
||||
# tightest reachable one. The slack sits at 2150 MiB: it accepts the
|
||||
# deployed split and rejects a two-layer re-skew (34:31 / 31:34 both
|
||||
# move the fit gap well past it). The x16-lean 35:30 that served the
|
||||
# predecessor payload (gap 0.58 GiB) no longer applies: the operator
|
||||
# chose symmetry over the x1-card bottleneck.
|
||||
slack=$(( 2150 * 1024 * 1024 ))
|
||||
card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
|
||||
+ kv_pos * ctx * split_hi / (split_hi + split_lo) \
|
||||
+ 381898377 \
|
||||
+ mmproj \
|
||||
+ 1363 * 1024 * ubatch ))
|
||||
if [[ "$spec_type" == *dflash* || "$spec_type" == *dspark* ]]; then
|
||||
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ DRAFT_BYTES \
|
||||
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
|
||||
+ draft_kv_pos * (ctx < 2048 ? ctx : 2048) \
|
||||
+ 955046413 \
|
||||
+ 2468 * 1024 * ubatch ))
|
||||
else
|
||||
# MTP: the head is in the target file (charged above); one nextn
|
||||
# layer's KV plus buffers/graphs. The constant is CALIBRATED to the
|
||||
# deployed point (2026-09-17, 34:31 / 393216): card 1 measured 30.78
|
||||
# GiB, of which 4.20 GiB is MTP's total charge -- 1.61 GiB of nextn KV
|
||||
# plus 2.70 GiB of buffers and graphs, far more than the 0.2 GiB
|
||||
# estimate the DFlash2-era shape suggested.
|
||||
card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \
|
||||
+ kv_pos * ctx * split_lo / (split_hi + split_lo) \
|
||||
+ mtp_kv_pos * ctx \
|
||||
+ 2901231913 \
|
||||
+ 2468 * 1024 * ubatch ))
|
||||
fi
|
||||
[ "$card_hi" -le $(( CARD_BYTES - floor )) ]
|
||||
[ "$card_lo" -le $(( CARD_BYTES - floor )) ]
|
||||
diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff ))
|
||||
[ "$diff" -le "$slack" ]
|
||||
# ...and the window may not wander far past what the checkpoint was
|
||||
# trained for (262144; the GGUF carries no rope scaling, so positions
|
||||
# past that extrapolate on the base rope). The deployed 327680 pool is
|
||||
# two 163840 slots -- entirely trained territory.
|
||||
[ "$ctx" -le 393216 ]
|
||||
}
|
||||
|
||||
@test "the host-RAM prompt cache can hold a full slot's state" {
|
||||
# --cache-ram is the one cache tunable that is NOT VRAM: it caps the total
|
||||
# bytes of saved prompt states in host RAM (server_prompt_cache, in
|
||||
# tools/server/server-task.cpp). alloc() REFUSES any single state larger
|
||||
# than the whole cap -- silently, with one WRN line -- so a budget below
|
||||
# one slot's full context makes every deep conversation uncacheable and
|
||||
# every turn past it re-prefill from scratch. That was the deployed state
|
||||
# on 2026-09-18: 137 refusals of 11.0-11.4 GiB states against the 8192 MiB
|
||||
# cap, in two days of journal.
|
||||
#
|
||||
# The state law at this shape, per token of sequence: 16 full-attention
|
||||
# layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB, plus 4096 B for
|
||||
# the MTP draft context's own KV. The DeltaNet state is constant-size and
|
||||
# is left out of the sum -- which only makes this gate more conservative.
|
||||
local ctx parallel cache_mib per_token slot_state host_mib
|
||||
ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
cache_mib="$(sed -nE 's/^CACHE_RAM=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
[ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_mib" ]
|
||||
per_token=$(( (16 * 4 * 256 * 2 * 2) + (4 * 256 * 2 * 2) ))
|
||||
slot_state=$(( per_token * (ctx / parallel) ))
|
||||
# one full slot must fit, or the cache is decorative
|
||||
[ "$cache_mib" -ge $(( slot_state / 1024 / 1024 )) ]
|
||||
# ...and this budget is host RAM: never more than half of dankserver's
|
||||
# 62220 MiB, which the media-stack co-tenants and the page cache share.
|
||||
host_mib=62220
|
||||
[ "$cache_mib" -le $(( host_mib / 2 )) ]
|
||||
# the arithmetic above is about nothing unless the flag goes out
|
||||
grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE"
|
||||
grep -q -- '--cache-prompt' "$SERVE"
|
||||
}
|
||||
|
||||
@test "vision is enabled: the f16 projector loads with the model" {
|
||||
grep -q '^NO_MMPROJ=0$' "$ENV_FILE"
|
||||
grep -q '^MM_PROJ_FILE=mmproj-Qwen3\.8-27B-Uncensored-f16\.gguf$' "$ENV_FILE"
|
||||
# Explicit flag, not autoload: this build starts text-only without it,
|
||||
# and refuses to start without the projector file, so a missing file is
|
||||
# a deploy error, not a silent text-only server.
|
||||
grep -q -- '--mmproj "$MM_PROJ"' "$SERVE"
|
||||
grep -q 'vision is enabled but the projector is missing' "$SERVE"
|
||||
# And the download script fetches and byte-verifies it.
|
||||
grep -q '^MM_PROJ_SIZE=931145984$' "$FETCH"
|
||||
grep -q -- 'check "$MODEL_DIR/$MM_PROJ_FILE" "$MM_PROJ_SIZE"' "$FETCH"
|
||||
}
|
||||
|
||||
@test "the fleet endpoint and the predecessor's sampler defaults survive" {
|
||||
grep -q '^ALIAS=frank-medium$' "$ENV_FILE"
|
||||
grep -q '^LISTEN_PORT=7770$' "$ENV_FILE"
|
||||
grep -q -- '--alias "$ALIAS"' "$SERVE"
|
||||
grep -q -- '--port "$LISTEN_PORT"' "$SERVE"
|
||||
grep -q -- '--host "$BIND_ADDR"' "$SERVE"
|
||||
# Prometheus scrapes this endpoint for LLM fleet metrics.
|
||||
grep -q -- '--metrics' "$SERVE"
|
||||
# Requests that send no sampler fields get these instead of an error.
|
||||
grep -q '^TEMPERATURE=0.7$' "$ENV_FILE"
|
||||
grep -q '^TOP_P=0.95$' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "the deploy handler installs the root unit and refuses a missing payload" {
|
||||
@test "the deploy handler gates on host assets and installs the quadlet" {
|
||||
grep -q 'deploy_llm_medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'relink_unit frank-medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl enable frank-medium.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl restart frank-medium.service' "$REMOTE_DEPLOY"
|
||||
grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY"
|
||||
grep -q 'build-llama-rdna4.sh' "$REMOTE_DEPLOY"
|
||||
grep -q 'download-frank-medium-gguf.sh' "$REMOTE_DEPLOY"
|
||||
grep -q '\[\[ "$missing" -eq 0 \]\] || return 1' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the deploy handler retires the vLLM quadlets and frees the port they held" {
|
||||
grep -q 'retire_vllm_medium_quadlets' "$REMOTE_DEPLOY"
|
||||
grep -q 'systemctl --user stop frank-medium-a.service frank-medium-b.service' "$REMOTE_DEPLOY"
|
||||
grep -q 'install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_container_healthy frank-medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY"
|
||||
# No image left to tag: the runtime is a native binary now.
|
||||
# ...and no image work or quadlet install of the predecessor survives.
|
||||
grep -q '\[\[ "$missing" -eq 0 \]\] || return 1' "$REMOTE_DEPLOY"
|
||||
# The image is only pulled when absent, and re-tagged to the name the unit
|
||||
# actually runs (the quadlet path cannot resolve registry refs at create).
|
||||
grep -q 'podman image exists "$LLM_MEDIUM_IMAGE_LOCAL"' "$REMOTE_DEPLOY"
|
||||
grep -q 'podman tag "$LLM_MEDIUM_IMAGE" "$LLM_MEDIUM_IMAGE_LOCAL"' "$REMOTE_DEPLOY"
|
||||
# The patched libr4d is not optional (the image's own build NaNs this model).
|
||||
grep -q 'patched libr4d absent' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "a generated unit is never asked to enable, and the llama unit is gone" {
|
||||
# podman 5.x v1 generator: generated units cannot be enabled; asking makes
|
||||
# the deploy fail on a unit that is already boot-persistent via [Install].
|
||||
! grep -q 'systemctl --user enable frank-medium' "$REMOTE_DEPLOY"
|
||||
# ...and the root llama unit must be retired, not relinked: relinking would
|
||||
# resurrect a service whose repo file no longer exists.
|
||||
! grep -q 'relink_unit frank-medium' "$REMOTE_DEPLOY"
|
||||
grep -q 'retire_llama_medium_unit' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the llama layer split and the replica trio are archived, not deleted" {
|
||||
[ -f "$LLAMA_TRASH/frank-medium.service.llamacpp-layersplit" ]
|
||||
[ -f "$LLAMA_TRASH/frank-medium.env.llamacpp-layersplit" ]
|
||||
[ -f "$LLAMA_TRASH/serve-frank-medium.sh.llamacpp-layersplit" ]
|
||||
[ ! -f "$REPO_ROOT/scripts/frank-medium.env" ]
|
||||
[ -f "$REPLICA_TRASH/frank-medium-a.container" ]
|
||||
[ -f "$REPLICA_TRASH/frank-medium-proxy.container" ]
|
||||
# The replicas stay retired: the handler stops them but never installs them.
|
||||
grep -q 'systemctl --user stop frank-medium-a.service frank-medium-b.service' "$REMOTE_DEPLOY"
|
||||
! grep -q 'install_quadlet frank-medium-a' "$REMOTE_DEPLOY"
|
||||
! grep -q 'podman tag docker.io/stilldeadcode' "$REMOTE_DEPLOY"
|
||||
}
|
||||
|
||||
@test "the retired vLLM deployment is gone from the repo, not merely unmapped" {
|
||||
[ ! -d "$REPO_ROOT/dankserver/frank-medium" ]
|
||||
[ ! -f "$REPO_ROOT/tests/frank_medium_quadlet.bats" ]
|
||||
# ...and the archive is present, so the two assertions above cannot pass
|
||||
# because a path went missing rather than because it stayed missing.
|
||||
[ -f "$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-a.container" ]
|
||||
[ -f "$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance/radiance-entrypoint.sh" ]
|
||||
}
|
||||
|
||||
@test "the serve script refuses to start on an incomplete host" {
|
||||
grep -q 'frank-medium: missing\|missing executable runtime\|missing or unreadable' "$SERVE"
|
||||
grep -q 'missing or unreadable target model' "$SERVE"
|
||||
grep -q 'missing or unreadable draft model' "$SERVE"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user