serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	frank-medium ROCmFPX Runbook	charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201).	frank-medium dankserver rocmfpx llama-cpp mtp gfx1201 r9700	2026-09-15

PRIOR deployment (retired 2026-09-16). This page documents the llama.cpp era of frank-medium, which served the endpoint for part of 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE deployment is frank-medium-llamacpp-dflash2; this page is kept as the record of the runtime it replaced.

frank-medium ROCmFPX Runbook

frank-medium is the Dankserver LLM endpoint (port 7770) since 2026-09-15, when the RTX 3090 left the box and two AMD Radeon AI PRO R9700 cards (Navi 48, RDNA4, gfx1201, 32,624 MiB each, PCIe 5.0 x16) took its place. It replaced the retired FreeToken/CUDA frank-small (freetoken-frank-small), which cannot run on this hardware at all.

Stack: charlie12345/ROCmFPX llama.cpp (main, c49ebdb, 2026-08-22) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF with the companion MTP draft head. Stock llama.cpp cannot load this GGUF — Q8_0_ROCMFPX_AGENT is ftype 115 / GGML_TYPE_Q8_0_ROCMFPX (type 103) and exists only in the fork.

Hardware

Property	Value
GPUs	2x AMD Radeon AI PRO R9700 (DID 0x7551), rev c0
Architecture	gfx1201 (Navi 48, RDNA4)
VRAM	32,624 MiB each (rocm-smi)
Link	PCIe, no XGMI; cards sit in different IOMMU groups
iGPU	Raphael (gfx1036) enumerates as ROCm device 2 — hidden from HIP
CPU / RAM	Ryzen 7 7700X (8c/8t), 60 GiB RAM, 63 GiB swap
Kernel driver	in-tree amdgpu; no new driver needed

HIP_VISIBLE_DEVICES=0,1 hides the iGPU so nothing can drift onto shared system RAM. Devices are addressed as ROCm0,ROCm1 (llama-server) or ROCm0/ROCm1 (llama-bench multi-device syntax — see the bench gotcha below).

Runtime build (self-contained ROCm 7.14)

The host ROCm install is quarantined rather than repaired: the build uses the fork's self-contained toolchain, so the binary carries its own libamdhip64/libhsa-runtime64/rocblas/hipblaslt with RPATH=$ORIGIN/../lib and never touches system ROCm.

cd /home/dank/GitPackages/charlie-ROCmFPX
env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
# -> build-rdna4-rocm714/bin/llama-server

The script downloads the gfx120X-all TheRock ROCm 7.14 nightly tarball into ~/.local/share/rocm-7.14.0a20260624 (~17 GiB extracted; tarball kept next to the build tree) and builds gfx1200;gfx1201.

gfx1201 is not interchangeable with gfx1200 — the fork's docs/BUILD-AMD-ARCHITECTURES.md warns a wrong-sibling build links, loads the model, then segfaults. scripts/rocmfpx-hip-arch.sh auto-detects and refuses.

Model

File	Size	Role
Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf	28,214,425,952 B (26.28 GiB, 8.39 bpw)	target
mtp-Qwen3.8-27B-Q4_0.gguf	1,680,271,648 B	MTP draft head
mmproj-Qwen3.8-27B-BF16.gguf	931,145,888 B	vision tower — unused (--no-mmproj)

The AGENT routing keeps 13 of the 17 *.output.weight tensors at full Q8_0 (12 more than the plain Q8_0_ROCMFPX build), which raises MTP draft acceptance (0.953 vs 0.911 on the author's gfx1151 box). Take this variant only when serving with MTP. Fetch with scripts/download-q8agent.sh (resumable, size-verified; run --verify-only to re-check).

The model card documents 131,072 context; llama.cpp reports n_ctx_train (262144) from the GGUF, but the author never tested beyond 131k, so the deployment uses 131,072.

Parallelism: layer split, not tensor (measured)

llama-bench on the host (pp4096 / tg128, same at depth 32768, f16 KV, flash-attn on, 3 reps):

Split	pp4096	tg128	pp4096 @d32768	tg128 @d32768
1x R9700 (-sm none)	1160.9	19.42	~512	~18.4
2x layer (-sm layer)	1991.2	18.76	859.1	17.64
2x tensor (-sm tensor)	271.8	26.26	228.6	25.09

Tensor parallel wins decode by ~40 % but collapses prefill 7x: this build ships without RCCL, so llama.cpp logs internal AllReduce init failed (n_devices != 2?); falling back to meta-backend butterfly and every layer pays a slow cross-card reduction. The workload is prefill-heavy (agentic, long prompts, reasoning traces), so the operator switched to layer split on 2026-09-15. Revisit if the fork ships a working RCCL tensor path (-DGGML_HIP_RCCL=ON).

Served-path comparison with MTP on (real completions): layer 35.7–46.2 tok/s vs tensor 28.7 tok/s — layer wins decode too once the draft head is in play.

llama-bench multi-device gotcha

llama-bench and llama-server disagree on device syntax. In llama-bench, -dev ROCm0,ROCm1 means two separate single-card benches (the value is split on , into separate device sets, one model load each); a single split bench needs -dev ROCm0/ROCm1. The invalid form silently produces single-card numbers in a table that looks like a split test.

Tensor-mode draft abort (if tensor is ever revisited)

Under --split-mode tensor the MTP draft head must be pinned to exactly one device (--spec-draft-device ROCm0); with it empty the draft joins the meta-device KV allocation and the server aborts while building the KV cache:

ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
  <- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor

Deployment

File	Role
scripts/frank-medium.env	all tunables (sampler, split, draft, context)
scripts/serve-frank-medium.sh	env-driven llama-server launcher
scripts/frank-medium.service	root systemd unit (relinked into /etc/systemd/system)
scripts/download-q8agent.sh	model fetch + size verification

./scripts/deploy.sh dankserver --service frank-medium

First deploy (2026-09-15) also ran retire_root_llm_small() in scripts/deploy-remote.sh: the CUDA-era frank-small unit was stopped, disabled and moved to ~/.AGENTTRASH/20260915-frank-zealot-retire/host-units (the host's systemctl disable has been observed to delete unit files, 2026-08-26 incident, so both outcomes are tolerated).

Deployed flags (see scripts/frank-medium.env for the full set): --split-mode layer --device ROCm0,ROCm1 --gpu-layers 999 --fit off, --ctx-size 131072 --parallel 1, f16 K/V, flash-attn on, --spec-type draft-mtp (n_max 6, p_min 0.6) with the draft pinned to ROCm1 — the card holding the target model's last layers, where the final hidden state is produced. Reasoning on, deepseek format, unlimited budget; sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8, top-k 20, min-p 0, presence 1.5). Text only — vision stays on frank-dragoon (Dankclaw).

Verified serving behavior (2026-09-15)

• Health {"status":"ok"} ~18 s after start; model load ~17 s (warm page cache).
• MTP engaged: draft acceptance 60–69 % across prompts (timings.draft_n / draft_n_accepted in every chat response).
• Decode 35.7–46.2 tok/s on a 200-token budget; cold prompt 949 tokens prefilled in 2036 ms (~466 tok/s at ubatch 512).
• Prefix caching works with the draft head: the model card's mtp-prompt-cache-fix.patch defect (draft head disables prefix caching) is already fixed in c49ebdb — the speculative boundary rides inside the context checkpoint (common_prompt_checkpoint::data_spec). Measured: turn 2 of a conversation reprocessed 22 tokens (cache_n 945) instead of 949.
• Quirk: the request-level "speculative": {"n_max": 0} override does not disable drafting (still drafts and accepts). To serve without MTP, restart with SPEC_TYPE=none.

Client contract

OpenAI-compatible /v1 on port 7770, alias frank-medium. Reasoning traces are projected into reasoning_content (deepseek format); requests that send no sampler fields get the server's CLI defaults, not greedy. Text only. Harness routing after the cutover: blankpad smol/tiny/commit/default → frank-medium, Hermes title_generation (and its other dankserver-small routes) → frank-medium, vision stays frank-dragoon.

Host ROCm posture (post-cleanup)

Historically the host had a broken mix: AMD-repo ROCm 6.3.3 in /opt/rocm-6.3.3 hijacked libhsa-runtime64 via /etc/ld.so.conf.d/10-rocm-opencl.conf, breaking Ubuntu's ROCm 7.1 userspace (undefined reference to 'hsa_amd_memory_get_preferred_copy_engine'). On 2026-09-15 the stale 6.3.3 packages (~24.04-suffixed, 167 pkgs) were purged, the hijack conf and the repo.radeon.com apt pin moved to ~/.AGENTTRASH/20260915-rocm633-cleanup/, and ldconfig now resolves only Ubuntu's ROCm 7.1 libs. rocm-smi is Ubuntu's own 7.1.1 package. The self-contained toolchain in ~/.local/share/rocm-7.14.0a20260624 does not depend on any of it.

Troubleshooting

• Wrong arch build: a gfx1200 binary on this card links, loads, then segfaults. Rebuild with the pinned script and check the "Build complete for gfx1200;gfx1201" line plus the emitted code objects.
• Crash at load: read the journal (sudo journalctl -u frank-medium -n 100), then run the serve script directly; the bundled libs make the script and the raw binary behave identically. The known abort shape is the tensor-mode draft KV abort above.
• MTP "not engaged": check timings.draft_n in a response; zero means the draft head is not loaded (look for common_speculative_init: adding speculative implementation 'draft-mtp' in the journal).
• Model files missing/truncated: bash scripts/download-q8agent.sh --verify-only.

See Also

• freetoken-frank-small — the retired predecessor (FreeToken/CUDA on the 3090)
• kingjones-q8agent-rocmfpx — the gfx1151 investigation of this checkpoint family
• Llama Cpp Runtimes — where the ROCmFPX fork sits among the runtimes
• Dankserver System Specs — current hardware
• deployment — the deploy flow