From d53962fec500eb149f1437a129e567b4df607b10 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 15 Sep 2026 19:03:30 -0700 Subject: [PATCH] frank-medium: serve the new R9700s, retire frank-small The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all, so the `frank-small` service is retired and `frank-medium` takes the endpoint. New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3 never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF -- a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load. Placeholder config: --split-mode tensor across both cards, 131072 context, 1 slot, f16 KV, MTP draft head on ROCm0. deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped, disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units before frank-medium starts, because `systemctl disable` has been observed to delete the unit file itself on this host (2026-08-26 incident). The retired frank-small repo files (env, unit, serve script) move to scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the deployment test seams now name frank-medium. Follow-up (same session, measured): llama-bench on the host with the correct multi-device syntax (-dev ROCm0/ROCm1) overturned the tensor choice: single R9700 pp4096 1160.9 / tg128 19.42 (d32768: 512 / 18.4) layer 2x pp4096 1991.2 / tg128 18.76 (d32768: 859 / 17.6) tensor 2x pp4096 271.8 / tg128 26.26 (d32768: 229 / 25.1) This build has no RCCL, so tensor mode falls back to a "meta-backend butterfly" allreduce and prefill collapses 7x. Served path with MTP: layer 35.7-46.2 tok/s vs tensor 28.7. SPLIT_MODE=layer, draft pinned to ROCm1 (the card holding the last target layers). Served prefix cache with the draft head verified live: turn 2 reprocessed 22 tokens instead of 949. --- scripts/frank-medium.env | 52 +++++++++++++++++++---------------- scripts/serve-frank-medium.sh | 12 ++++---- 2 files changed, 36 insertions(+), 28 deletions(-) diff --git a/scripts/frank-medium.env b/scripts/frank-medium.env index 6522631..a9fdd09 100644 --- a/scripts/frank-medium.env +++ b/scripts/frank-medium.env @@ -25,12 +25,20 @@ # MTP draft acceptance (0.953 vs 0.911) and that is the whole point of this # checkpoint. Model card documents 131,072 context. # -# Parallelism (--split-mode tensor): tensor parallel splits the weights AND -# the KV cache across both cards through the fused "meta device". It is the -# only split mode here that gives each card half the KV cache; layer split -# would give each card the KV of the layers it owns, with one PCIe handoff -# per token per boundary. TENSOR mode's own constraints, all met below: -# flash attention ON, non-quantized KV cache, and no auto-fit. +# Parallelism (--split-mode layer, operator call 2026-09-15 "pipeline +# parallelism"): layers are distributed across the two cards in order and +# activations hop PCIe at the boundary. Measured with llama-bench +# (pp4096 / tg128 / same at depth 32768, f16 KV, fa on, 3 reps): +# +# single R9700 1160.9 / 19.42 (fits: 26.27 GiB weights, ~5 GiB left) +# layer 2x 1991.2 / 18.76 859.1 / 17.64 +# tensor 2x 271.8 / 26.26 228.6 / 25.09 +# +# Tensor parallel wins decode (+40 %) but collapses prefill 7x: this build +# has no RCCL (llama.cpp logs "AllReduce init failed; falling back to +# meta-backend butterfly"), so every layer pays a slow cross-card reduction. +# Agentic traffic is prefill-heavy, so layer split wins the workload. Revisit +# if the fork ships a working RCCL tensor path. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP @@ -59,20 +67,20 @@ LISTEN_PORT=7770 # can drift onto shared system RAM; DEVICES is what llama.cpp splits across. HIP_VISIBLE_DEVICES=0,1 DEVICES=ROCm0,ROCm1 -SPLIT_MODE=tensor +SPLIT_MODE=layer -# TENSOR_SPLIT empty = an even split between the two cards. +# TENSOR_SPLIT empty = layers distributed by free VRAM (equal here). TENSOR_SPLIT= NGL=999 -# Tensor mode ignores auto-fit, and auto-fit has no business resizing a -# context we set explicitly. +# Auto-fit has no business resizing a context we set explicitly. FIT=off FLASH_ATTN=on -# 131,072 tokens, one slot (the model card's documented window). Tensor mode -# puts half the KV cache on each card, so the window is not limited by a -# single card's free memory. +# 131,072 tokens, one slot (the model card's documented window; the GGUF +# reports n_ctx_train 262144, untested by the model author). Layer split +# keeps each layer's KV on the card that owns the layer, so the window is +# bounded by per-card free memory: ~5.6 GiB of KV per card at 131072 f16. CTX_SIZE=131072 PARALLEL=1 @@ -83,8 +91,7 @@ UBATCH_SIZE=512 NUM_THREADS=8 THREAD_BATCH=8 -# f16 K/V: tensor mode refuses a quantized KV cache, and an agent checkpoint -# is the wrong place to spend KV precision anyway. +# f16 K/V: an agent checkpoint is the wrong place to spend KV precision. CACHE_TYPE_K=f16 CACHE_TYPE_V=f16 CTX_CHECKPOINTS=8 @@ -122,11 +129,10 @@ SPEC_DRAFT_TYPE_K=f16 SPEC_DRAFT_TYPE_V=f16 SPEC_DRAFT_POLL=1 SPEC_DRAFT_POLL_BATCH=1 -# The draft head must run on ONE device. With --split-mode tensor the draft -# otherwise joins the meta-device allocation and the KV-cache build aborts: -# ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed -# <- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor -# (measured 2026-09-15 on the built runtime: identical flags are healthy with -# --spec-draft-device ROCm0 and abort with it empty). ROCm0 is the card that -# also holds the target model's main GPU. -SPEC_DRAFT_DEVICE=ROCm0 +# The draft head rides the card that holds the target model's LAST layers: +# the MTP head consumes the final hidden state, so pinning it to ROCm1 keeps +# the per-token hop off the wire. (Under the abandoned tensor split the +# draft must also be pinned to exactly one device or the meta-device KV +# allocation aborts: GGML_ASSERT(buffer) in ggml-backend.cpp, measured +# 2026-09-15.) +SPEC_DRAFT_DEVICE=ROCm1 diff --git a/scripts/serve-frank-medium.sh b/scripts/serve-frank-medium.sh index 453c110..6e26fba 100755 --- a/scripts/serve-frank-medium.sh +++ b/scripts/serve-frank-medium.sh @@ -27,11 +27,13 @@ # what makes the MTP draft land ~0.95 of its guesses. A stock llama.cpp # cannot load this GGUF at all - Q8_0_ROCMFPX exists only in this fork. # -# Parallelism: --split-mode tensor across both cards. Tensor parallel -# splits the weights AND the KV cache across the participating devices via -# the fused "meta device" (unlike -sm layer, which pipelines layers and -# keeps each layer's KV on its own card). It requires flash attention and -# a non-quantized KV cache, and it is disabled from auto-fit (--fit off). +# Parallelism: --split-mode layer across both cards (operator call +# 2026-09-15). Layers are distributed in order and activations hop PCIe at +# the boundary; measured llama-bench: layer 1991/18.8 tok/s (pp4096/tg128) +# vs tensor 272/26.3 and single-card 1161/19.4. Tensor wins decode +40 % +# but its prefill collapses 7x on this build (no RCCL: "AllReduce init +# failed; falling back to meta-backend butterfly"), so layer split wins the +# prefill-heavy agentic workload. # # Tunables come from frank-medium.env unless FRANK_MEDIUM_ENV_FILE # overrides that path.