From c2116658f89fcb1201dd42dcfdc1d1f68e65d310 Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 20 Sep 2026 18:13:33 -0700 Subject: [PATCH] feat(frank-medium): cut the endpoint over to radiance TP=2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The conversation battery (scripts/bench-conversation.py, salted, same 10 prompts, 2026-09-20) measured three arms on the two R9700s: arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s Radiance wins every axis, so it takes the alias. The board swap is what made arm R possible: TP=2 was archived on 2026-09-16 because the second card sat behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+). The R4D one-shot all-reduce needs no RCCL collective anyway. - un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten — it is the deployed shape now), radiance-entrypoint.sh, the healthcheck. The replica pair and its affinity proxy stay archived; one process owns both cards now. - retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh stay: they are the tooling that rollback needs. - deploy-remote.sh: deploy_llm_medium now gates on the image (pull once, re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint, the drafter, the pinned patch clone and the patched libr4d; retires the llama root unit into the home trash instead of relinking it; then install_quadlet + wait_container_healthy 1800s (first start compiles). retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the a/b/proxy trio only, since frank-medium is installed again. - planner + test-changed map dankserver/frank-medium/** to frank-medium and its seam; the scripts-trio mapping is gone with the files. - tests/frank_medium_unit.bats rewritten for the container contract: boot persistence of a GENERATED unit (no enable on this host's v1 generator), TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning contract, and the archive invariants. - homepage card + docs updated, and they were stale twice over: they still described the ajgazin Swift 35:30 / 393k payload, not the deployed shape. Known regression, documented in the unit: vision. The MXFP4 checkpoint is language_model_only and the launcher loads no projector, so images are a 400 here; the harness's vision role already routes to frank-small. --- .../frank-medium-healthcheck.py | 8 +- .../quadlet}/frank-medium.container | 45 +- .../radiance-entrypoint.sh | 0 .../bench-spec-ab.py.llamacpp-layersplit} | 0 .../frank-medium.env.llamacpp-layersplit} | 0 .../frank-medium.service.llamacpp-layersplit} | 0 ...serve-frank-medium.sh.llamacpp-layersplit} | 0 scripts/deploy-plan.py | 19 +- scripts/deploy-remote.sh | 156 ++++-- scripts/test-changed.sh | 17 +- shared/homepage/docs.html | 8 +- shared/homepage/services.json | 2 +- tests/changed_tests.bats | 12 +- tests/deployment_scoping.bats | 23 +- tests/frank_medium_unit.bats | 488 ++++++------------ 15 files changed, 345 insertions(+), 433 deletions(-) rename dankserver/{.AGENTTRASH/20260916-vllm-radiance => frank-medium}/frank-medium-healthcheck.py (69%) rename dankserver/{.AGENTTRASH/20260916-vllm-radiance => frank-medium/quadlet}/frank-medium.container (77%) rename dankserver/{.AGENTTRASH/20260916-vllm-radiance => frank-medium}/radiance-entrypoint.sh (100%) rename scripts/{bench-spec-ab.py => .AGENTTRASH/20260920-llama-layersplit/bench-spec-ab.py.llamacpp-layersplit} (100%) rename scripts/{frank-medium.env => .AGENTTRASH/20260920-llama-layersplit/frank-medium.env.llamacpp-layersplit} (100%) rename scripts/{frank-medium.service => .AGENTTRASH/20260920-llama-layersplit/frank-medium.service.llamacpp-layersplit} (100%) rename scripts/{serve-frank-medium.sh => .AGENTTRASH/20260920-llama-layersplit/serve-frank-medium.sh.llamacpp-layersplit} (100%) diff --git a/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-healthcheck.py b/dankserver/frank-medium/frank-medium-healthcheck.py similarity index 69% rename from dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-healthcheck.py rename to dankserver/frank-medium/frank-medium-healthcheck.py index a48fa55..e792150 100755 --- a/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-healthcheck.py +++ b/dankserver/frank-medium/frank-medium-healthcheck.py @@ -4,10 +4,10 @@ Runs INSIDE the container. The vLLM /health endpoint only answers once the engine is up (weights loaded, cudagraphs captured), so a healthy container is a serving container, not merely a live process. PORT selects the endpoint so -the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can -share this probe. The HealthCmd in the quadlet splits on spaces into an argv -array (podman 5.x v1 generator), which is why this lives in a file rather -than an inline python one-liner. +the probe follows the unit's Environment=PORT rather than hard-coding the +fleet port. The HealthCmd in the quadlet splits on spaces into an argv array +(podman 5.x v1 generator), which is why this lives in a file rather than an +inline python one-liner. """ import os import sys diff --git a/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium.container b/dankserver/frank-medium/quadlet/frank-medium.container similarity index 77% rename from dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium.container rename to dankserver/frank-medium/quadlet/frank-medium.container index c129bd7..48afd90 100644 --- a/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium.container +++ b/dankserver/frank-medium/quadlet/frank-medium.container @@ -3,30 +3,32 @@ # Install path on dankserver: ~/.config/containers/systemd/frank-medium.container # Unit name: frank-medium.service (container name: frank-medium) # -# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15 -# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 -# image + the ggz14/radiance-vllm-mxfp4 patches) serving +# The dankserver LLM endpoint (:7770, alias frank-medium) since 2026-09-20: +# the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 image + +# the ggz14/radiance-vllm-mxfp4 patches) serving # just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO -# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp -# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF; -# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the -# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production -# configuration of the upstream repo, on our exact hardware signature -# (2x7551-32624, see the KV pin below). +# R9700 (gfx1201) as ONE tensor-parallel process (TP=2, R4D one-shot +# all-reduce) with the DFlash2-FP8 block-diffusion drafter (SPEC=7). +# It replaced the mainline llama.cpp layer-split deployment, which served the +# same alias from the orcarouter Q8_0 GGUF across the same two cards. # # SHAPE. The launcher the upstream repo ships binds the server with # --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is # what this unit does too — no PublishPort. The fleet contract is unchanged: # :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1. # -# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel -# process across both R9700s. It cannot launch a collective here — RCCL's -# kernels need PCIe atomics and the second card sits behind chipset bridges -# advertising `AtomicOpsCap: Routing-` — so the active deployment is the -# two-replica trio in this directory (frank-medium-a/b/proxy) and -# deploy-remote.sh deliberately does not install this unit. Keep it for a -# host whose second card is on an atomic-capable root port; the full evidence -# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md. +# TP=2 IS INSTALLABLE HERE NOW, and it won on measurement. Until 2026-09-20 +# this unit was archived un-installed because the second card sat behind B650 +# chipset bridges advertising `AtomicOpsCap: Routing-`, which stopped RCCL +# kernel dispatch. The board swap (ProArt B850-Creator WiFi NEO) put both +# cards on CPU GPP root ports — Gen5 x8 each, every bridge Routing+, both +# endpoints ReqEn+ — and the collective-free R4D path runs at TP=2 without +# complaint. The conversation battery (scripts/bench-conversation.py, salted, +# same 10 prompts, 2026-09-20) measured this shape against the incumbent: +# decode geomean 95.8 t/s vs 45.7, mean TTFT 0.21 s vs 0.72 s, and 259 t/s +# aggregate against 4 concurrent streams vs 71.7. Evidence and the llama-side +# caveats: serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md and +# Dual-R9700-Cooperation.md. # # DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device # indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2 @@ -92,6 +94,15 @@ Environment=TP=2 Environment=MAXSEQS=8 Environment=CHUNK=8192 Environment=MAXLEN=262144 +# MAXLEN is the whole pool shared across the MAXSEQS slots: 8 concurrent +# sessions split 262,144 tokens between them, so a full-window request +# excludes the others until it finishes (vLLM preempts rather than refuses). +# The llama deployment it replaced advertised 327,680 across 2 slots. +# +# TEXT ONLY. The MXFP4 checkpoint carries language_model_only=true and the +# launcher loads no vision projector, so an input_audio/image request is a +# 400 here. Vision traffic routes to frank-small on dankclaw (harness +# modelRoles vision -> frank-small). Environment=GPU_UTIL=0.98 # Speculative decoding: the DFlash2 drafter at depth 7 is the configuration # every published number (224 t/s single-stream decode, 4749-4810 t/s diff --git a/dankserver/.AGENTTRASH/20260916-vllm-radiance/radiance-entrypoint.sh b/dankserver/frank-medium/radiance-entrypoint.sh similarity index 100% rename from dankserver/.AGENTTRASH/20260916-vllm-radiance/radiance-entrypoint.sh rename to dankserver/frank-medium/radiance-entrypoint.sh diff --git a/scripts/bench-spec-ab.py b/scripts/.AGENTTRASH/20260920-llama-layersplit/bench-spec-ab.py.llamacpp-layersplit similarity index 100% rename from scripts/bench-spec-ab.py rename to scripts/.AGENTTRASH/20260920-llama-layersplit/bench-spec-ab.py.llamacpp-layersplit diff --git a/scripts/frank-medium.env b/scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.env.llamacpp-layersplit similarity index 100% rename from scripts/frank-medium.env rename to scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.env.llamacpp-layersplit diff --git a/scripts/frank-medium.service b/scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.service.llamacpp-layersplit similarity index 100% rename from scripts/frank-medium.service rename to scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.service.llamacpp-layersplit diff --git a/scripts/serve-frank-medium.sh b/scripts/.AGENTTRASH/20260920-llama-layersplit/serve-frank-medium.sh.llamacpp-layersplit similarity index 100% rename from scripts/serve-frank-medium.sh rename to scripts/.AGENTTRASH/20260920-llama-layersplit/serve-frank-medium.sh.llamacpp-layersplit diff --git a/scripts/deploy-plan.py b/scripts/deploy-plan.py index ad2ae9d..1d2cda2 100755 --- a/scripts/deploy-plan.py +++ b/scripts/deploy-plan.py @@ -369,11 +369,14 @@ def services_for_path(path: str, target: str) -> set[str]: _add(services, "node-exporter") return services if len(parts) >= 2 and parts[1] == "frank-medium": - # frank-medium is a ROOT llama.cpp unit since 2026-09-16: its - # runtime is a native gfx1201 binary, not a container, so there is - # nothing under dankserver/frank-medium/ to plan any more. The - # retired vllm-radiance quadlets live in .AGENTTRASH/, which is - # archive material and plans nothing. + # frank-medium is a CONTAINER again since 2026-09-20: the TP=2 + # radiance Quadlet, its entrypoint and its healthcheck all live + # under this directory, and the container mounts the latter two + # straight out of this checkout — so any file here changes deployed + # state. (It was a root llama.cpp unit from 2026-09-16, whose trio + # lived in scripts/; that shape is archived in + # scripts/.AGENTTRASH/20260920-llama-layersplit/ and plans nothing.) + _add(services, "frank-medium") return services return services @@ -405,12 +408,6 @@ def services_for_path(path: str, target: str) -> set[str]: # trio that lived here for two days is in scripts/.AGENTTRASH/, # which is archive material and plans nothing. _add(services, "frank-small") - if name in {"frank-medium.env", "frank-medium.service", "serve-frank-medium.sh"}: - # llama.cpp frank-medium (2026-09-16) keeps its unit, env and serve - # files here the same way. The files that BUILD and FETCH its - # runtime and payload are host-side one-offs: they change no - # deployed state, so they select nothing. - _add(services, "frank-medium") if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}: # CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and # serve files here the same way: a root unit on a host-built diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 6444a85..5ba06d1 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1137,31 +1137,43 @@ deploy_llm_small() { wait_for_health frank-small http://127.0.0.1:7771/health 900 } -# deploy_llm_medium — frank-medium is mainline llama.cpp on the two R9700s as a -# ROOT unit since the 2026-09-16 cutover: the runtime is a native gfx1201 HIP -# binary, so there is no image to install and no generated unit to dodge. It -# replaces the vllm-radiance replica pair + routing proxy, which held this -# endpoint for a day and could never use both cards for one model (TP=2 needs -# an RCCL collective, and the second card cannot dispatch one). +# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a +# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2) +# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter. +# It replaced mainline llama.cpp's layer split, which held the endpoint since +# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a +# third of the time-to-first-token with 4x the concurrent throughput +# (scripts/bench-conversation.py, 2026-09-20). # -# Layer split in ONE process is what the successor does instead: layers are -# distributed across the cards and only the activation at the boundary crosses -# PCIe, once per forward pass. That is the "pipeline parallelism" the endpoint -# wanted, with no collective anywhere in it. -LLM_MEDIUM_BIN="${LLM_MEDIUM_BIN:-/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}" -LLM_MEDIUM_MODEL="${LLM_MEDIUM_MODEL:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/Qwen3.8-27B-Uncensored-Q8_0.gguf}" -LLM_MEDIUM_DRAFT="${LLM_MEDIUM_DRAFT:-/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}" -LLM_MEDIUM_MMPROJ="${LLM_MEDIUM_MMPROJ:-/home/dank/models/Qwen3.8-27B-Uncensored-GGUF/mmproj-Qwen3.8-27B-Uncensored-f16.gguf}" - -# The three vLLM user units are retired HERE, not left to rot: the proxy -# published 7770 on the host network, so the llama-server below cannot bind -# until they are stopped and the port is released. Never `rm` — the unit files -# move into the home trash; the images stay in local storage. -retire_vllm_medium_quadlets() { +# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this +# unit because TP=2 could not dispatch a collective on the second card (B650 +# chipset bridges, AtomicOpsCap: Routing-), and the two-replica + routing-proxy +# trio was the workaround. The board swap cleared the atomics wall, so one +# TP=2 process is both possible and the fastest shape; the replicas and their +# affinity proxy stay retired. +# +# HOST ASSETS. Nothing here installs the image's payloads: the checkpoints, the +# pinned radiance patch clone, the patched libr4d build and the Triton compile +# cache live outside the checkout (see the unit file's mount comments). The +# gates below refuse a deploy that would start a container guaranteed to die. +LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}" +LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}" +LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}" +LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}" +LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}" +LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}" +LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}" + +# The three replica/proxy user units from the 2026-09-16 era are retired HERE, +# never left to rot: they must not exist alongside the TP=2 unit (the proxy +# published 7770 on the host network). Never `rm` — unit files move into the +# home trash; the images stay in local storage. `frank-medium` itself is NOT in +# the list: it is the unit this handler installs. +retire_vllm_replica_quadlets() { local trash="$HOME/.AGENTTRASH/20260916-vllm-radiance-retire" unit systemctl --user stop frank-medium-a.service frank-medium-b.service \ frank-medium-proxy.service 2>/dev/null || true - for unit in frank-medium-a frank-medium-b frank-medium-proxy frank-medium; do + for unit in frank-medium-a frank-medium-b frank-medium-proxy; do if [[ -e "$quadlet_unit_dir/$unit.container" ]]; then mkdir -p "$trash" mv -- "$quadlet_unit_dir/$unit.container" "$trash/$unit.container" @@ -1171,54 +1183,90 @@ retire_vllm_medium_quadlets() { systemctl --user daemon-reload } +# The llama.cpp root unit is retired HERE, into the home trash, never `rm`: +# /etc/systemd/system/frank-medium.service is a relink_unit symlink into this +# checkout, and the checkout pull that precedes this handler already moved the +# repo file away, leaving the symlink dangling. Stop it while systemd still has +# it loaded (an active unit stays loaded without its file), then take both the +# symlink and its [Install] alias. Nothing can start it at boot afterwards. +retire_llama_medium_unit() { + local trash="$HOME/.AGENTTRASH/20260920-llama-layersplit-frank-medium" + run_privileged systemctl stop frank-medium.service 2>/dev/null || true + if [[ -L /etc/systemd/system/frank-medium.service || + -L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]]; then + mkdir -p "$trash" + [[ -L /etc/systemd/system/frank-medium.service ]] && + run_privileged mv -- /etc/systemd/system/frank-medium.service \ + "$trash/etc-frank-medium.service.symlink" + [[ -L /etc/systemd/system/multi-user.target.wants/frank-medium.service ]] && + run_privileged mv -- /etc/systemd/system/multi-user.target.wants/frank-medium.service \ + "$trash/etc-frank-medium.wants.symlink" + echo "retired llama root unit frank-medium.service -> $trash" + fi + run_privileged systemctl daemon-reload +} + deploy_llm_medium() { - # Refuse rather than restart-loop. A unit whose runtime or payload is - # missing never becomes healthy, and a green deploy would leave the - # endpoint dark with nothing in the deploy log to say why. + # Refuse rather than restart-loop. A container whose image or payload is + # missing never becomes healthy, and a green deploy would leave the endpoint + # dark with nothing in the deploy log to say why. local missing=0 - if [[ ! -x "$LLM_MEDIUM_BIN" ]]; then - echo "ERROR: llama-server absent at $LLM_MEDIUM_BIN" >&2 - echo "frank-medium: build it ON the host (gfx1201): bash scripts/build-llama-rdna4.sh" >&2 + if ! podman image exists "$LLM_MEDIUM_IMAGE_LOCAL"; then + if podman image exists "$LLM_MEDIUM_IMAGE"; then + podman tag "$LLM_MEDIUM_IMAGE" "$LLM_MEDIUM_IMAGE_LOCAL" + else + echo "ERROR: radiance image absent ($LLM_MEDIUM_IMAGE / $LLM_MEDIUM_IMAGE_LOCAL)" >&2 + echo "frank-medium: podman pull $LLM_MEDIUM_IMAGE && podman tag $LLM_MEDIUM_IMAGE $LLM_MEDIUM_IMAGE_LOCAL" >&2 + missing=1 + fi + fi + if [[ ! -f "$LLM_MEDIUM_SNAP/config.json" ]]; then + echo "ERROR: MXFP4 checkpoint absent at $LLM_MEDIUM_SNAP" >&2 missing=1 fi - if [[ ! -f "$LLM_MEDIUM_MODEL" ]]; then - echo "ERROR: target model absent at $LLM_MEDIUM_MODEL" >&2 - echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2 + if [[ ! -f "$LLM_MEDIUM_DRAFTER/config.json" ]]; then + echo "ERROR: DFlash2-FP8 drafter absent at $LLM_MEDIUM_DRAFTER" >&2 missing=1 fi - if [[ ! -f "$LLM_MEDIUM_DRAFT" ]]; then - echo "ERROR: DFlash2 drafter absent at $LLM_MEDIUM_DRAFT" >&2 - echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2 + # The patches clone is mounted and its scripts run at EVERY container start; + # the chat template is the fleet's reasoning-format contract. + if [[ ! -f "$LLM_MEDIUM_PATCHES/qwen-fixed-v22.3.jinja" ]]; then + echo "ERROR: pinned patch clone / chat template absent at $LLM_MEDIUM_PATCHES" >&2 + echo "frank-medium: git clone https://github.com/ggz14/radiance-vllm-mxfp4 $LLM_MEDIUM_PATCHES" >&2 missing=1 fi - if [[ ! -f "$LLM_MEDIUM_MMPROJ" ]]; then - echo "ERROR: vision projector absent at $LLM_MEDIUM_MMPROJ" >&2 - echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2 + # The image's shipped r4d.so predates the gated-delta-net overflow fix and + # NaNs this model, so the patched build is not optional. + if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then + echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2 missing=1 fi [[ "$missing" -eq 0 ]] || return 1 + mkdir -p "$LLM_MEDIUM_CACHE" - retire_vllm_medium_quadlets - # The thing holding 7770 is now THIS service: on a re-deploy the previous - # llama-server is still bound to it, so waiting for the port without - # stopping the unit deadlocks the deploy (measured 2026-09-16: "FAIL: port - # 7770 is still held after 90s", with the old server happily serving a - # scrape). Stop the unit first, then sweep a stray manual run of the same - # binary -- that is how a benchmark harness leaves the port busy. `pkill -x - # llama-server` matches the process name exactly and can never match the - # deploying shell; never `pkill -f` here. - run_privileged systemctl stop frank-medium.service 2>/dev/null || true + retire_vllm_replica_quadlets + retire_llama_medium_unit + # 7770 is a host-network bind: nothing new can be created until the old + # holder released it (the same gate as the 2026-09-16 cutover, where the + # retired proxy held the port and the new engine failed at create). Sweep a + # stray manual llama-server too — that is how a bench harness leaves the + # port busy. `pkill -x` matches the process name exactly and can never + # match the deploying shell; never `pkill -f` here. pkill -x llama-server 2>/dev/null || true + pkill -x VLLM::EngineCor 2>/dev/null || true sleep 2 - # pasta released the retired proxy's socket at container teardown, and a - # stopped engine lingers in TIME_WAIT just past that. Same gate as - # frank-large's cutover (2026-09-16 addressed-in-use failure). wait_port_free 7770 || return 1 - relink_unit frank-medium - run_privileged systemctl daemon-reload - run_privileged systemctl enable frank-medium.service - run_privileged systemctl restart frank-medium.service - wait_for_health frank-medium http://127.0.0.1:7770/health 900 + + install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium + # No enable: generated units cannot be enabled on this host (v1 quadlet + # generator); boot persistence is the unit's [Install] WantedBy=default.target. + systemctl --user restart frank-medium.service + # Triton/inductor compile on a cold cache runs minutes before the engine + # loads ~9.5 GiB per card; HealthStartPeriod in the unit is 1800s to match. + wait_container_healthy frank-medium 1800 || { + podman logs --tail 40 frank-medium 2>&1 || true + return 1 + } } wait_for_health() { diff --git a/scripts/test-changed.sh b/scripts/test-changed.sh index 042e22b..62b21f9 100755 --- a/scripts/test-changed.sh +++ b/scripts/test-changed.sh @@ -180,17 +180,16 @@ for path in "${CHANGED[@]}"; do scripts/mount-entropy.sh) add_test tests/mount_entropy.bats ;; - # frank-medium is llama.cpp on the two R9700s as a ROOT unit since - # 2026-09-16: a native gfx1201 HIP binary, so (like frank-small) its - # env, unit and serve script all live in scripts/, and this case has to - # precede the generic scripts/*.service fallback below. The retired - # vllm-radiance service dir is under dankserver/.AGENTTRASH/, which is - # archive material and maps to nothing. - scripts/frank-medium.*|scripts/serve-frank-medium.sh) + # frank-medium is a rootless quadlet again since the 2026-09-20 + # tensor-parallel cutover: the TP=2 radiance unit, its entrypoint and + # its healthcheck all live under dankserver/frank-medium/, and the + # container mounts the latter two straight out of the checkout. The + # llama.cpp trio that held this endpoint before that date is archive + # material in scripts/.AGENTTRASH/ and maps to nothing. + dankserver/frank-medium/*) add_test tests/frank_medium_unit.bats - add_test tests/llm_units.bats add_test tests/deployment_scoping.bats - add_test tests/scripts_syntax.bats + add_test tests/docs_completeness.bats ;; # frank-small is llama.cpp on the 3090 as a ROOT unit: a native CUDA # binary, so no generated quadlet is involved. Its env, unit and serve diff --git a/shared/homepage/docs.html b/shared/homepage/docs.html index 934dbb9..34481ba 100644 --- a/shared/homepage/docs.html +++ b/shared/homepage/docs.html @@ -64,10 +64,10 @@

frank-medium — the 27B generalist

Which one to pick? Screenshots and photos: frank-small. diff --git a/shared/homepage/services.json b/shared/homepage/services.json index dd91063..8810bd7 100644 --- a/shared/homepage/services.json +++ b/shared/homepage/services.json @@ -244,7 +244,7 @@ { "name": "frank-medium", "category": "user", - "description": "The 27B generalist. ajgazin Swift-Qwen3.8-27B-Uncensored at UD-Q8_K_XL with its own MTP head, mainline llama.cpp in one 35:30 layer-split process across Dankserver's two R9700s, 393k-token context.", + "description": "The 27B generalist. Qwen3.8-27B-Uncensored at MXFP4 on vllm-radiance, one tensor-parallel process across Dankserver's two R9700s with a DFlash2 drafter; 262k-token context shared by up to eight conversations.", "url": "", "docs": "/docs#models", "probe": "192.168.0.235:7770", diff --git a/tests/changed_tests.bats b/tests/changed_tests.bats index 49c8c08..19c730c 100644 --- a/tests/changed_tests.bats +++ b/tests/changed_tests.bats @@ -21,10 +21,18 @@ load helpers [[ "$output" != *"tests/homepage_pages.bats"* ]] } -@test "changed-test wrapper maps the frank-medium trio to its focused tests" { - run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/frank-medium.env +@test "changed-test wrapper maps the frank-medium dir to its focused tests" { + run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths \ + dankserver/frank-medium/quadlet/frank-medium.container [ "$status" -eq 0 ] [[ "$output" == *"tests/frank_medium_unit.bats"* ]] [[ "$output" == *"tests/deployment_scoping.bats"* ]] + [[ "$output" == *"tests/docs_completeness.bats"* ]] [[ "$output" != *"tests/homepage_pages.bats"* ]] + # The llama trio's old path must no longer select the container seam: the + # mapping was removed with the runtime, and a stale case here would keep + # firing tests for an archived service. + run bash "$REPO_ROOT/scripts/test-changed.sh" --list --paths scripts/frank-medium.env + [ "$status" -eq 0 ] + [[ "$output" != *"tests/frank_medium_unit.bats"* ]] } diff --git a/tests/deployment_scoping.bats b/tests/deployment_scoping.bats index 35146af..30d0ee3 100644 --- a/tests/deployment_scoping.bats +++ b/tests/deployment_scoping.bats @@ -46,18 +46,25 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" [ -z "$output" ] } -@test "deployment planner maps the frank-medium scripts trio to frank-medium only" { +@test "deployment planner maps the frank-medium container dir to frank-medium only" { run python3 "$PLANNER" --target dankserver --paths \ - scripts/frank-medium.env scripts/frank-medium.service scripts/serve-frank-medium.sh + dankserver/frank-medium/quadlet/frank-medium.container \ + dankserver/frank-medium/radiance-entrypoint.sh \ + dankserver/frank-medium/frank-medium-healthcheck.py [ "$status" -eq 0 ] [ "$output" = "frank-medium" ] - # The path is not the assertion: with the vllm-radiance service dir - # retired into the trash, a planner rule still naming dankserver/frank-medium - # would plan nothing and look identical to a correct one. Assert the - # invariant (the dir is gone) as well as the mapping above. - [ ! -d "$REPO_ROOT/dankserver/frank-medium" ] - run python3 "$PLANNER" --target dankserver --paths dankserver/frank-medium/quadlet/frank-medium-a.container + # The path is not the assertion: both retired shapes are still in the tree + # as archive, and a rule matching either would restart a service nobody + # runs. Assert the negatives explicitly, plus the llama trio's old paths, + # whose mapping was removed with the runtime. + run python3 "$PLANNER" --target dankserver --paths \ + dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-a.container + [ "$status" -eq 0 ] + [ -z "$output" ] + run python3 "$PLANNER" --target dankserver --paths \ + scripts/.AGENTTRASH/20260920-llama-layersplit/frank-medium.env.llamacpp-layersplit \ + scripts/frank-medium.env [ "$status" -eq 0 ] [ -z "$output" ] } diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index a41939a..8a099b8 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -1,351 +1,193 @@ #!/usr/bin/env bats -# Seam: the frank-medium mainline-llama.cpp root unit on dankserver. -# frank-medium became a native llama.cpp service on 2026-09-16, replacing the -# vllm-radiance replica pair + routing proxy that served this endpoint for a -# day. What this seam pins is what the flags alone do not say: boot persistence -# for a NATIVE runtime (no generated quadlet is involved), that BOTH cards work -# on ONE model through a layer split with no collective in the path, that the -# DFlash2 drafter is wired with a depth its block size actually allows, that the -# runtime and payload are pinned rather than floated, and that one slot's KV -# still fits a 32 GiB card beside the weights it holds. +# Seam: the frank-medium vllm-radiance container on dankserver. +# frank-medium became a CONTAINER again on 2026-09-20, when the board swap put +# both R9700s on CPU root ports and a measured three-arm comparison retired the +# mainline-llama.cpp layer split (scripts/.AGENTTRASH/20260920-llama-layersplit/). +# What this seam pins is what the flags alone do not say: boot persistence for a +# GENERATED unit (this host's v1 quadlet generator cannot enable units, so the +# [Install] section is the only thing that survives a reboot), that BOTH cards +# work on ONE model through tensor parallelism — the thing the pre-2026-09-20 +# hardware could not do — the KV pin's coupling to the batch shape it was +# calibrated at, and that the pool still holds one full-window request. load helpers -UNIT="$REPO_ROOT/scripts/frank-medium.service" -SERVE="$REPO_ROOT/scripts/serve-frank-medium.sh" -ENV_FILE="$REPO_ROOT/scripts/frank-medium.env" -BUILD="$REPO_ROOT/scripts/build-llama-rdna4.sh" -FETCH="$REPO_ROOT/scripts/download-frank-medium-gguf.sh" +UNIT="$REPO_ROOT/dankserver/frank-medium/quadlet/frank-medium.container" +ENTRY="$REPO_ROOT/dankserver/frank-medium/radiance-entrypoint.sh" +HEALTH="$REPO_ROOT/dankserver/frank-medium/frank-medium-healthcheck.py" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" +LLAMA_TRASH="$REPO_ROOT/scripts/.AGENTTRASH/20260920-llama-layersplit" +REPLICA_TRASH="$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance" -# Measured constants (rocm-smi / HF blob sizes), used by the fit arithmetic. -CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it -MODEL_BYTES=29047084416 # orcarouter Q8_0 target -DRAFT_BYTES=1143006816 # DFlash2 Q4_K_M drafter - -@test "frank-medium's unit, serve script, env and its build/fetch scripts exist" { - [ -f "$UNIT" ] - [ -f "$SERVE" ] - [ -f "$ENV_FILE" ] - [ -f "$BUILD" ] - [ -f "$FETCH" ] +# Measured constants, used by the fit arithmetic. +CARD_BYTES=34208743424 # one R9700 as rocm-smi reports it (32624 MiB) +KV_BYTES_PER_TOKEN=70656 # --kv-cache-memory 18563072000 => "262,720 tokens" +env_value() { + sed -nE "s/^Environment=$1=(-?[0-9]+)$/\1/p" "$UNIT" | head -n1 +} +# GPU_UTIL is a fraction, not a count; the integer reader would drop it. +env_ratio() { + sed -nE "s/^Environment=$1=([0-9.]+)$/\1/p" "$UNIT" | head -n1 } -@test "frank-medium is boot-persistent, self-healing and unprivileged" { - grep -q '^WantedBy=multi-user.target$' "$UNIT" +@test "frank-medium's unit, entrypoint and healthcheck exist" { + [ -f "$UNIT" ] + [ -x "$ENTRY" ] + [ -f "$HEALTH" ] +} + +@test "frank-medium is boot-persistent and self-healing" { + # Generated units cannot be enabled on this host, so WantedBy in the file is + # the whole boot story (AGENTS.md: Boot Persistence & Self-Healing). + grep -q '^WantedBy=default.target$' "$UNIT" grep -q '^Restart=on-failure$' "$UNIT" grep -q '^RestartSec=' "$UNIT" - grep -q '^User=dank$' "$UNIT" - grep -q '^LimitMEMLOCK=infinity$' "$UNIT" + # First start compiles Triton/inductor kernels before the engine loads ~9.5 + # GiB per card; a shorter timeout kills a healthy boot. + grep -q '^TimeoutStartSec=1800$' "$UNIT" + grep -q '^HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py$' "$UNIT" + grep -q '^HealthStartPeriod=1800s$' "$UNIT" + # Boots must not depend on the registry. + grep -q '^Pull=never$' "$UNIT" } -@test "frank-medium runs the repo's own serve script with the repo's env" { - grep -q '^ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-medium.sh$' "$UNIT" - grep -q 'FRANK_MEDIUM_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-medium.env' "$UNIT" +@test "both R9700s serve ONE model as one tensor-parallel process" { + grep -q '^Environment=TP=2$' "$UNIT" + grep -q '^Environment=ROCR_VISIBLE_DEVICES=0,1$' "$UNIT" + grep -q '^Environment=HIP_VISIBLE_DEVICES=0,1$' "$UNIT" + grep -q -- '--tensor-parallel-size "$TP"' "$ENTRY" + # The iGPU (index 2, 2 GiB) must stay out of the visible set: a third rank + # would take layers onto system RAM. + ! grep -q '^Environment=.*VISIBLE_DEVICES=0,1,2' "$UNIT" + # TP needs the all-reduce to actually run. R4D is the collective-free path + # this host measured; the entrypoint must still select it, and the RCCL + # fallback must not be re-disabled by the pre-board-swap workarounds. + grep -q 'RADIANCE_USE_R4D_AR' "$ENTRY" + grep -q -- '--attention-backend "$ATTN"' "$ENTRY" + ! grep -q 'NCCL_P2P_DISABLE' "$UNIT" + ! grep -q 'RCCL_MSCCL_ENABLE' "$UNIT" } -@test "both R9700s serve ONE model, layer-split inside one process" { - grep -q '^SPLIT_MODE=layer$' "$ENV_FILE" - grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE" - # The split is a layer PAIR: the shares add up to the model's 65 blocks (one - # -ts unit per layer). The operator asked for the tensors BALANCED across - # the two cards (2026-09-19), which for 65 layers is 32:33 -- the most even - # split that exists, with ROCm1 (the last-layers card, which also carries - # the drafter and the output layers) holding the extra one. WHICH pair - # balances the cards is the fit test's job, which recomputes it from this - # file; pinning a literal here would only mean that re-measuring the - # balance breaks a test about parallelism. - local split_hi split_lo - split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")" - split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")" - [ -n "$split_hi" ] && [ -n "$split_lo" ] - [ $(( split_hi + split_lo )) -eq 65 ] - [ $(( split_lo - split_hi )) -le 1 ] - grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE" - grep -q -- '--device "$DEVICES"' "$SERVE" - grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE" - # Tensor split is the one shape this host cannot run: it needs a cross-card - # collective, and the second card cannot dispatch one (see the env file). - ! grep -q '^SPLIT_MODE=tensor$' "$ENV_FILE" +@test "HIP never reads one of these tunables as its own setting" { + # HIP reads bare TEMP, HOST and GPU_MAX_HEAP_SIZE (2026-08-13 incident: + # TEMP=1.0 sent the JIT cache to path "1.0" and segfaulted every load). + # Serve scripts export env files with `set -a`; a container inherits the + # same hazard from Environment=. + ! grep -qE '^Environment=(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$UNIT" } -@test "the iGPU is hidden so no layer can land on system RAM" { - grep -q '^HIP_VISIBLE_DEVICES=0,1$' "$ENV_FILE" - grep -q '^export HIP_VISIBLE_DEVICES$' "$SERVE" +@test "the drafter is DFlash2 at the measured depth and the kernel band covers the batch" { + grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT" + local maxseqs spec chunk + spec="$(env_value SPEC)" + maxseqs="$(env_value MAXSEQS)" + chunk="$(env_value CHUNK)" + [ -n "$spec" ] && [ -n "$maxseqs" ] && [ -n "$chunk" ] + # DFlash2 drafts at most 7 tokens per step; deeper is a different drafter's + # contract and costs acceptance, not just VRAM. + [ "$spec" -eq 7 ] + # The MXFP4 decode kernel band covers MAXSEQS x (SPEC+1) rows, and the + # shipped default covers 64. Exceeding it without the entrypoint's >8 bump + # is a launch failure at the worst moment (first concurrent request). + [ $(( maxseqs * (spec + 1) )) -le 64 ] + grep -q 'RADIANCE_MXFP4_DECODE_MAX_M' "$ENTRY" + # The chunk is the AR size gate's input (CHUNK x hidden x 2 bytes); a chunk + # larger than this row's silently changes which all-reduce path runs. + [ "$chunk" -le 8192 ] } -@test "the drafter is the DFlash2 sidecar at the measured depth" { - # The DFlash2 sidecar is the deployed drafter (operator call 2026-09-19). - # n_max 7 is the measured DFlash2 winner -- 43.5 tok/s at 41 % acceptance - # against 32.4 for the in-file MTP head at n=3 (the env's table; identical - # greedy text at every depth, so this is pure speed). The target file - # also carries its own MTP head (nextn_predict_layers=1), so draft-mtp is - # one SPEC_TYPE change away; the drafter file stays on disk for that. - grep -qE '^SPEC_TYPE=draft-dflash$' "$ENV_FILE" - grep -q -- '--spec-type "$SPEC_TYPE"' "$SERVE" - local n_max - n_max="$(sed -nE 's/^SPEC_DRAFT_N_MAX=([0-9]+)$/\1/p' "$ENV_FILE")" - [ -n "$n_max" ] && [ "$n_max" -eq 7 ] - # The Q4_K_M rung is the operator's requested drafter quant -- the - # drafter's job is throughput, not output quality, and the target - # verifies every drafted token. ngram-mod's window is retained in the env - # for the chain, so it may not be dropped. - grep -q '^DRAFT_MODEL_FILE=Qwen3\.8-27B-DFlash2-Q4_K_M\.gguf$' "$ENV_FILE" - grep -q '^SPEC_NGRAM_N_MATCH=24$' "$ENV_FILE" - # ...and neither is loaded: ngram-mod's flags only go out when the type - # contains ngram, so a pure dflash configuration is not handed them. - python3 -c " -lines = open('$SERVE').read().splitlines() -guard = next(i for i, l in enumerate(lines) if '*ngram*' in l) -indent = len(lines[guard]) - len(lines[guard].lstrip()) -end = next(i for i in range(guard + 1, len(lines)) - if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent) -assert '--spec-ngram-mod-n-match' in chr(10).join(lines[guard:end]), 'ngram flags must sit in the ngram branch' -" +@test "the KV pin is valid only at the shape it was calibrated for" { + # The pin is the upstream kv-profiles.tsv row for sig 2x7551-32624 at + # maxseqs=8 chunk=8192 maxlen=262144 spec=dflash — exactly this host. It is + # not a general-purpose budget: changing MAXSEQS, CHUNK or SPEC_METHOD + # invalidates it, so the shape has to move with it or the pin goes to 0 + # (let the image profile the shape). + local kv_mem maxseqs chunk maxlen spec + kv_mem="$(env_value KV_MEM)" + maxseqs="$(env_value MAXSEQS)" + chunk="$(env_value CHUNK)" + maxlen="$(env_value MAXLEN)" + spec="$(env_value SPEC)" + [ -n "$kv_mem" ] && [ -n "$maxseqs" ] && [ -n "$chunk" ] && [ -n "$maxlen" ] + grep -q 'kv-profiles.tsv' "$UNIT" + [ "$maxseqs" -eq 8 ] && [ "$chunk" -eq 8192 ] && [ "$spec" -eq 7 ] + grep -q '^Environment=SPEC_METHOD=dflash$' "$UNIT" + # Pool arithmetic, from the engine's own boot line: 18563072000 bytes + # reserved => "GPU KV cache size: 262,720 tokens". The pool must hold at + # least ONE full-window request or the endpoint refuses its own + # --max-model-len, and it must fit on one card beside the weights. + [ $(( kv_mem / KV_BYTES_PER_TOKEN )) -ge "$maxlen" ] + # 9.5 GiB of MXFP4 weights per card + the KV pool, under the 0.98 util cap. + local weights=10200547328 # ~9.5 GiB per card of the 19 GB MXFP4 checkpoint + local util util_pct floor + util="$(env_ratio GPU_UTIL)" + [ -n "$util" ] + util_pct="${util#0.}" + floor=$(( 1024 * 1024 * 1024 )) # 1 GiB per card + [ $(( weights + kv_mem / 2 )) -le $(( CARD_BYTES * util_pct / 100 - floor )) ] } -@test "a draft MODEL is passed only for DFlash types, never for MTP" { - # MTP builds its context against the TARGET model (its head lives in that - # GGUF). Passing --spec-draft-model alongside it points the draft context at - # the drafter's file, which has no MTP layers, and the server dies at load: - # context type MTP requested but model doesn't contain MTP layers - # failed to create MTP context - # So the flag must appear exactly once, inside the dflash branch. - python3 -c " -lines = open('$SERVE').read().splitlines() -guard = next(i for i, l in enumerate(lines) if '*dflash*' in l) -indent = len(lines[guard]) - len(lines[guard].lstrip()) -end = next(i for i in range(guard + 1, len(lines)) - if lines[i].strip() == 'fi' and len(lines[i]) - len(lines[i].lstrip()) == indent) -block = chr(10).join(lines[guard:end]) -assert '--spec-draft-model' in block, 'the dflash branch must pass the draft model' -assert sum(l.count('--spec-draft-model') for l in lines) == 1, 'only that branch may pass it' -print('draft-model wiring ok') -" +@test "the fleet endpoint contract survives the runtime swap" { + grep -q '^Environment=PORT=7770$' "$UNIT" + grep -q '^Environment=SERVED_NAMES=frank-medium$' "$UNIT" + grep -q -- '--port "$PORT"' "$ENTRY" + grep -q -- '--served-model-name $SERVED_NAMES' "$ENTRY" + # Host network: vLLM binds the port itself, so no PublishPort (and no + # pasta bind, which is what made the retired proxy's port replacement + # fail at container create). + grep -q -- '--network=host' "$UNIT" + ! grep -q '^PublishPort=' "$UNIT" + # Tools and reasoning are part of the harness contract; both parsers must + # be selected, and thinking must land in its own field. + grep -q -- '--enable-auto-tool-choice' "$ENTRY" + grep -q -- '--tool-call-parser qwen3_coder' "$ENTRY" + grep -q -- '--reasoning-parser qwen3' "$ENTRY" + grep -q -- '--enable-prefix-caching' "$ENTRY" + # Requests that send no sampler fields get the predecessor's defaults. + grep -q '"temperature":0.7' "$ENTRY" + grep -q '"top_p":0.95' "$ENTRY" } -@test "the runtime is pinned to a commit and to gfx1201 alone" { - # A floating ref would silently move the service between engine versions, - # and an unpinned arch would build for a sibling GPU that loads and then - # segfaults on these cards. - grep -Eq '^LLAMA_REF="\$\{LLAMA_REF:-[0-9a-f]{40}\}"$' "$BUILD" - grep -Eq '^HIP_ARCH="\$\{HIP_ARCH:-gfx1201\}"$' "$BUILD" - grep -q 'GGML_HIP=ON' "$BUILD" - grep -q 'frank-medium.env' "$SERVE" +@test "the healthcheck proves the engine serves, not merely that a process lives" { + grep -q '"/health"' "$HEALTH" || grep -q '/health' "$HEALTH" + grep -q 'status == 200' "$HEALTH" + # PORT-aware, and it must not hard-code a replica port. + grep -q 'os.environ.get("PORT"' "$HEALTH" + ! grep -qE '7771|7772' "$HEALTH" } -@test "the payload is fetched resumably and verified by size and sha256" { - grep -q 'Qwen3\.8-27B-Uncensored-Q8_0\.gguf' "$FETCH" - grep -q 'Qwen3\.8-27B-DFlash2-Q4_K_M\.gguf' "$FETCH" - grep -q -- '-fL -C -' "$FETCH" - grep -q 'verifying sizes and sha256' "$FETCH" - # The target repo is gated: the script must fail loud without a token - # rather than write a 401 body to disk and report success. - grep -q 'no HF token is available' "$FETCH" -} - -@test "both cards stay balanced and keep the operator's 1.5 GiB floor" { - local ctx parallel cache_k cache_v split_hi split_lo batch ubatch - ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")" - cache_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")" - cache_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")" - split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")" - split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")" - batch="$(sed -nE 's/^BATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - ubatch="$(sed -nE 's/^UBATCH_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - spec_type="$(sed -nE 's/^SPEC_TYPE=(.+)$/\1/p' "$ENV_FILE")" - [ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_k" ] && [ -n "$cache_v" ] && [ -n "$spec_type" ] - [ -n "$split_hi" ] && [ -n "$split_lo" ] && [ -n "$batch" ] && [ -n "$ubatch" ] - [ "$parallel" -ge 1 ] - [ "$batch" -ge "$ubatch" ] - [ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ] - # Per card, calibrated against rocm-smi at the 35:30 / ctx 262144 / batch - # 6144 x ubatch 512 / two slots / DFlash2 point, where the two cards - # measured 6.44 / 6.90 GiB free. The fit reproduces card 1 to 0.00 GiB - # and card 0 to 0.86 GiB at that point; it is a GATE, not a meter -- the - # deployed point is re-measured on the host after every split/ctx/batch - # move (the current one is 32:33 / 327680 / DFlash2 Q4_K_M, the balanced - # split the operator asked for on 2026-09-19 -- see the TENSOR_SPLIT - # header in frank-medium.env). - # - # card 0 = its share of the weights and of the KV (qwen35 spends KV on - # its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim - # x 2 B = 64 KiB per token at f16) plus ~1.4e6 B per ubatch unit - # of compute buffers - # card 1 = the same for its own share, plus the drafter. The drafter's - # charge depends on WHAT it is: DFlash2 is a sidecar file, so - # its weights (2.06e9 B) plus window-bounded KV (20480 B/pos x - # 2048) plus ~0.89 GiB of constant buffers; MTP's head lives in - # the target file (already charged above), so its cost is one - # nextn layer's KV (4096 B/position) plus small buffers. Its - # compute buffers are the heavier ones either way, because it - # holds the output layers and the drafter's graphs. - # - # BOTH ubatch coefficients are measured at the deployed shape, from the - # 1536 -> 512 move: 1.30 GiB on card 0 and 2.41 GiB on card 1. That - # asymmetry is why shrinking the batch moved the balance toward card 1 and - # cost card 0 a layer -- the split and the batch cannot be chosen apart. - # - # The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus - # 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary - # around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention - # one -- which makes the drift check below CONSERVATIVE: it overstates the gap - # a one-layer move creates, and so trips on moves a measurement might bless. - # That is the intent: moving the split means re-measuring, not re-arguing. - # - # The slack is the DEPLOYED gap, rounded up: at each shape the balance - # cannot be tuned finer than one layer, so the deployed split's gap IS - # what the check must accept, and a tighter slack would only reject it. - # - # What this model cannot see is CONTIGUITY -- the mechanism by which the - # DSpark-era 8192/2048 failed, a 2020 MiB pp buffer that found no room while - # every total looked fine. Total bytes are not the whole story for one big - # allocation. - local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack - local card_hi card_lo diff - local mmproj=931145984 # mmproj-Qwen3.8-27B-Uncensored-f16.gguf, on the primary device - # 1.5 GiB was the floor until the 2x196k shape: the 34:31 / 393216 / - # MTP point measured 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the - # floor is the measured 1.08 GiB rounded down. The 32:33 / 327680 / - # DFlash2 point is roomier than that (fit: 6.97 / 4.89 GiB free), so the - # floor stays at the measured 1.0 GiB -- if a large allocation fails at - # load, the next lever is a ctx cut, not a floor change. - floor=$(( 1024 * 1024 * 1024 )) # 1.0 GiB per card - # The deployed point (32:33 at ctx 327680 with DFlash2) has a fit gap of - # 2082 MiB -- the balanced split the operator asked for, not the - # tightest reachable one. The slack sits at 2150 MiB: it accepts the - # deployed split and rejects a two-layer re-skew (34:31 / 31:34 both - # move the fit gap well past it). The x16-lean 35:30 that served the - # predecessor payload (gap 0.58 GiB) no longer applies: the operator - # chose symmetry over the x1-card bottleneck. - slack=$(( 2150 * 1024 * 1024 )) - card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \ - + kv_pos * ctx * split_hi / (split_hi + split_lo) \ - + 381898377 \ - + mmproj \ - + 1363 * 1024 * ubatch )) - if [[ "$spec_type" == *dflash* || "$spec_type" == *dspark* ]]; then - card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ - + DRAFT_BYTES \ - + kv_pos * ctx * split_lo / (split_hi + split_lo) \ - + draft_kv_pos * (ctx < 2048 ? ctx : 2048) \ - + 955046413 \ - + 2468 * 1024 * ubatch )) - else - # MTP: the head is in the target file (charged above); one nextn - # layer's KV plus buffers/graphs. The constant is CALIBRATED to the - # deployed point (2026-09-17, 34:31 / 393216): card 1 measured 30.78 - # GiB, of which 4.20 GiB is MTP's total charge -- 1.61 GiB of nextn KV - # plus 2.70 GiB of buffers and graphs, far more than the 0.2 GiB - # estimate the DFlash2-era shape suggested. - card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ - + kv_pos * ctx * split_lo / (split_hi + split_lo) \ - + mtp_kv_pos * ctx \ - + 2901231913 \ - + 2468 * 1024 * ubatch )) - fi - [ "$card_hi" -le $(( CARD_BYTES - floor )) ] - [ "$card_lo" -le $(( CARD_BYTES - floor )) ] - diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff )) - [ "$diff" -le "$slack" ] - # ...and the window may not wander far past what the checkpoint was - # trained for (262144; the GGUF carries no rope scaling, so positions - # past that extrapolate on the base rope). The deployed 327680 pool is - # two 163840 slots -- entirely trained territory. - [ "$ctx" -le 393216 ] -} - -@test "the host-RAM prompt cache can hold a full slot's state" { - # --cache-ram is the one cache tunable that is NOT VRAM: it caps the total - # bytes of saved prompt states in host RAM (server_prompt_cache, in - # tools/server/server-task.cpp). alloc() REFUSES any single state larger - # than the whole cap -- silently, with one WRN line -- so a budget below - # one slot's full context makes every deep conversation uncacheable and - # every turn past it re-prefill from scratch. That was the deployed state - # on 2026-09-18: 137 refusals of 11.0-11.4 GiB states against the 8192 MiB - # cap, in two days of journal. - # - # The state law at this shape, per token of sequence: 16 full-attention - # layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB, plus 4096 B for - # the MTP draft context's own KV. The DeltaNet state is constant-size and - # is left out of the sum -- which only makes this gate more conservative. - local ctx parallel cache_mib per_token slot_state host_mib - ctx="$(sed -nE 's/^CTX_SIZE=([0-9]+)$/\1/p' "$ENV_FILE")" - parallel="$(sed -nE 's/^PARALLEL=([0-9]+)$/\1/p' "$ENV_FILE")" - cache_mib="$(sed -nE 's/^CACHE_RAM=([0-9]+)$/\1/p' "$ENV_FILE")" - [ -n "$ctx" ] && [ -n "$parallel" ] && [ -n "$cache_mib" ] - per_token=$(( (16 * 4 * 256 * 2 * 2) + (4 * 256 * 2 * 2) )) - slot_state=$(( per_token * (ctx / parallel) )) - # one full slot must fit, or the cache is decorative - [ "$cache_mib" -ge $(( slot_state / 1024 / 1024 )) ] - # ...and this budget is host RAM: never more than half of dankserver's - # 62220 MiB, which the media-stack co-tenants and the page cache share. - host_mib=62220 - [ "$cache_mib" -le $(( host_mib / 2 )) ] - # the arithmetic above is about nothing unless the flag goes out - grep -q -- '--cache-ram "$CACHE_RAM"' "$SERVE" - grep -q -- '--cache-prompt' "$SERVE" -} - -@test "vision is enabled: the f16 projector loads with the model" { - grep -q '^NO_MMPROJ=0$' "$ENV_FILE" - grep -q '^MM_PROJ_FILE=mmproj-Qwen3\.8-27B-Uncensored-f16\.gguf$' "$ENV_FILE" - # Explicit flag, not autoload: this build starts text-only without it, - # and refuses to start without the projector file, so a missing file is - # a deploy error, not a silent text-only server. - grep -q -- '--mmproj "$MM_PROJ"' "$SERVE" - grep -q 'vision is enabled but the projector is missing' "$SERVE" - # And the download script fetches and byte-verifies it. - grep -q '^MM_PROJ_SIZE=931145984$' "$FETCH" - grep -q -- 'check "$MODEL_DIR/$MM_PROJ_FILE" "$MM_PROJ_SIZE"' "$FETCH" -} - -@test "the fleet endpoint and the predecessor's sampler defaults survive" { - grep -q '^ALIAS=frank-medium$' "$ENV_FILE" - grep -q '^LISTEN_PORT=7770$' "$ENV_FILE" - grep -q -- '--alias "$ALIAS"' "$SERVE" - grep -q -- '--port "$LISTEN_PORT"' "$SERVE" - grep -q -- '--host "$BIND_ADDR"' "$SERVE" - # Prometheus scrapes this endpoint for LLM fleet metrics. - grep -q -- '--metrics' "$SERVE" - # Requests that send no sampler fields get these instead of an error. - grep -q '^TEMPERATURE=0.7$' "$ENV_FILE" - grep -q '^TOP_P=0.95$' "$ENV_FILE" -} - -@test "the deploy handler installs the root unit and refuses a missing payload" { +@test "the deploy handler gates on host assets and installs the quadlet" { grep -q 'deploy_llm_medium' "$REMOTE_DEPLOY" - grep -q 'relink_unit frank-medium' "$REMOTE_DEPLOY" - grep -q 'systemctl enable frank-medium.service' "$REMOTE_DEPLOY" - grep -q 'systemctl restart frank-medium.service' "$REMOTE_DEPLOY" - grep -q '127.0.0.1:7770/health' "$REMOTE_DEPLOY" - grep -q 'build-llama-rdna4.sh' "$REMOTE_DEPLOY" - grep -q 'download-frank-medium-gguf.sh' "$REMOTE_DEPLOY" - grep -q '\[\[ "$missing" -eq 0 \]\] || return 1' "$REMOTE_DEPLOY" -} - -@test "the deploy handler retires the vLLM quadlets and frees the port they held" { - grep -q 'retire_vllm_medium_quadlets' "$REMOTE_DEPLOY" - grep -q 'systemctl --user stop frank-medium-a.service frank-medium-b.service' "$REMOTE_DEPLOY" + grep -q 'install_quadlet frank-medium dankserver/frank-medium/quadlet frank-medium' "$REMOTE_DEPLOY" + grep -q 'wait_container_healthy frank-medium' "$REMOTE_DEPLOY" grep -q 'wait_port_free 7770' "$REMOTE_DEPLOY" - # No image left to tag: the runtime is a native binary now. - # ...and no image work or quadlet install of the predecessor survives. + grep -q '\[\[ "$missing" -eq 0 \]\] || return 1' "$REMOTE_DEPLOY" + # The image is only pulled when absent, and re-tagged to the name the unit + # actually runs (the quadlet path cannot resolve registry refs at create). + grep -q 'podman image exists "$LLM_MEDIUM_IMAGE_LOCAL"' "$REMOTE_DEPLOY" + grep -q 'podman tag "$LLM_MEDIUM_IMAGE" "$LLM_MEDIUM_IMAGE_LOCAL"' "$REMOTE_DEPLOY" + # The patched libr4d is not optional (the image's own build NaNs this model). + grep -q 'patched libr4d absent' "$REMOTE_DEPLOY" +} + +@test "a generated unit is never asked to enable, and the llama unit is gone" { + # podman 5.x v1 generator: generated units cannot be enabled; asking makes + # the deploy fail on a unit that is already boot-persistent via [Install]. + ! grep -q 'systemctl --user enable frank-medium' "$REMOTE_DEPLOY" + # ...and the root llama unit must be retired, not relinked: relinking would + # resurrect a service whose repo file no longer exists. + ! grep -q 'relink_unit frank-medium' "$REMOTE_DEPLOY" + grep -q 'retire_llama_medium_unit' "$REMOTE_DEPLOY" +} + +@test "the llama layer split and the replica trio are archived, not deleted" { + [ -f "$LLAMA_TRASH/frank-medium.service.llamacpp-layersplit" ] + [ -f "$LLAMA_TRASH/frank-medium.env.llamacpp-layersplit" ] + [ -f "$LLAMA_TRASH/serve-frank-medium.sh.llamacpp-layersplit" ] + [ ! -f "$REPO_ROOT/scripts/frank-medium.env" ] + [ -f "$REPLICA_TRASH/frank-medium-a.container" ] + [ -f "$REPLICA_TRASH/frank-medium-proxy.container" ] + # The replicas stay retired: the handler stops them but never installs them. + grep -q 'systemctl --user stop frank-medium-a.service frank-medium-b.service' "$REMOTE_DEPLOY" ! grep -q 'install_quadlet frank-medium-a' "$REMOTE_DEPLOY" - ! grep -q 'podman tag docker.io/stilldeadcode' "$REMOTE_DEPLOY" -} - -@test "the retired vLLM deployment is gone from the repo, not merely unmapped" { - [ ! -d "$REPO_ROOT/dankserver/frank-medium" ] - [ ! -f "$REPO_ROOT/tests/frank_medium_quadlet.bats" ] - # ...and the archive is present, so the two assertions above cannot pass - # because a path went missing rather than because it stayed missing. - [ -f "$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-a.container" ] - [ -f "$REPO_ROOT/dankserver/.AGENTTRASH/20260916-vllm-radiance/radiance-entrypoint.sh" ] -} - -@test "the serve script refuses to start on an incomplete host" { - grep -q 'frank-medium: missing\|missing executable runtime\|missing or unreadable' "$SERVE" - grep -q 'missing or unreadable target model' "$SERVE" - grep -q 'missing or unreadable draft model' "$SERVE" }