frank-medium: serve the new R9700s, retire frank-small
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all, so the `frank-small` service is retired and `frank-medium` takes the endpoint. New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3 never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF -- a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load. Placeholder config: --split-mode tensor across both cards, 131072 context, 1 slot, f16 KV, MTP draft head on ROCm0. deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped, disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units before frank-medium starts, because `systemctl disable` has been observed to delete the unit file itself on this host (2026-08-26 incident). The retired frank-small repo files (env, unit, serve script) move to scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the deployment test seams now name frank-medium. Follow-up (same session, measured): llama-bench on the host with the correct multi-device syntax (-dev ROCm0/ROCm1) overturned the tensor choice: single R9700 pp4096 1160.9 / tg128 19.42 (d32768: 512 / 18.4) layer 2x pp4096 1991.2 / tg128 18.76 (d32768: 859 / 17.6) tensor 2x pp4096 271.8 / tg128 26.26 (d32768: 229 / 25.1) This build has no RCCL, so tensor mode falls back to a "meta-backend butterfly" allreduce and prefill collapses 7x. Served path with MTP: layer 35.7-46.2 tok/s vs tensor 28.7. SPLIT_MODE=layer, draft pinned to ROCm1 (the card holding the last target layers). Served prefix cache with the draft head verified live: turn 2 reprocessed 22 tokens instead of 949.
This commit is contained in:
1 parent
6085a99dd8
commit
d53962fec5
2 files changed
+36
-28
No files matched your search
@@ -27,11 +27,13 @@
|
||||
# what makes the MTP draft land ~0.95 of its guesses. A stock llama.cpp
|
||||
# cannot load this GGUF at all - Q8_0_ROCMFPX exists only in this fork.
|
||||
#
|
||||
# Parallelism: --split-mode tensor across both cards. Tensor parallel
|
||||
# splits the weights AND the KV cache across the participating devices via
|
||||
# the fused "meta device" (unlike -sm layer, which pipelines layers and
|
||||
# keeps each layer's KV on its own card). It requires flash attention and
|
||||
# a non-quantized KV cache, and it is disabled from auto-fit (--fit off).
|
||||
# Parallelism: --split-mode layer across both cards (operator call
|
||||
# 2026-09-15). Layers are distributed in order and activations hop PCIe at
|
||||
# the boundary; measured llama-bench: layer 1991/18.8 tok/s (pp4096/tg128)
|
||||
# vs tensor 272/26.3 and single-card 1161/19.4. Tensor wins decode +40 %
|
||||
# but its prefill collapses 7x on this build (no RCCL: "AllReduce init
|
||||
# failed; falling back to meta-backend butterfly"), so layer split wins the
|
||||
# prefill-heavy agentic workload.
|
||||
#
|
||||
# Tunables come from frank-medium.env unless FRANK_MEDIUM_ENV_FILE
|
||||
# overrides that path.
|
||||
|
||||
Reference in new issue
Block a user