---
type: configuration
title: frank-medium ROCmFPX Runbook
description: charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201).
tags:
  - frank-medium
  - dankserver
  - rocmfpx
  - llama-cpp
  - mtp
  - gfx1201
  - r9700
timestamp: 2026-09-15
---

> **PRIOR deployment (retired 2026-09-16).** This page documents the
> llama.cpp era of `frank-medium`, which served the endpoint for part of
> 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE
> deployment is [[frank-medium-llamacpp-dflash2]]; this page is kept as the
> record of the runtime it replaced.

# frank-medium ROCmFPX Runbook

`frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15,
when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards
(Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place.
It replaced the retired FreeToken/CUDA `frank-small`
([[freetoken-frank-small]]), which cannot run on this hardware at all.

Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22)
serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the
companion MTP draft head. Stock llama.cpp cannot load this GGUF —
`Q8_0_ROCMFPX_AGENT` is ftype 115 / `GGML_TYPE_Q8_0_ROCMFPX` (type 103) and
exists only in the fork.

## Hardware

| Property | Value |
|----------|-------|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), rev c0 |
| Architecture | gfx1201 (Navi 48, RDNA4) |
| VRAM | 32,624 MiB each (rocm-smi) |
| Link | PCIe, no XGMI; cards sit in different IOMMU groups |
| iGPU | Raphael (gfx1036) enumerates as ROCm device 2 — hidden from HIP |
| CPU / RAM | Ryzen 7 7700X (8c/8t), 60 GiB RAM, 63 GiB swap |
| Kernel driver | in-tree `amdgpu`; no new driver needed |

`HIP_VISIBLE_DEVICES=0,1` hides the iGPU so nothing can drift onto shared
system RAM. Devices are addressed as `ROCm0,ROCm1` (llama-server) or
`ROCm0/ROCm1` (llama-bench multi-device syntax — see the bench gotcha below).

## Runtime build (self-contained ROCm 7.14)

The host ROCm install is quarantined rather than repaired: the build uses the
fork's self-contained toolchain, so the binary carries its own
`libamdhip64`/`libhsa-runtime64`/`rocblas`/`hipblaslt` with
`RPATH=$ORIGIN/../lib` and never touches system ROCm.

```bash
cd /home/dank/GitPackages/charlie-ROCmFPX
env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
# -> build-rdna4-rocm714/bin/llama-server
```

The script downloads the gfx120X-all TheRock ROCm 7.14 nightly tarball into
`~/.local/share/rocm-7.14.0a20260624` (~17 GiB extracted; tarball kept next to
the build tree) and builds `gfx1200;gfx1201`.

**gfx1201 is not interchangeable with gfx1200** — the fork's
`docs/BUILD-AMD-ARCHITECTURES.md` warns a wrong-sibling build links, loads the
model, then segfaults. `scripts/rocmfpx-hip-arch.sh` auto-detects and refuses.

## Model

| File | Size | Role |
|------|------|------|
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28,214,425,952 B (26.28 GiB, 8.39 bpw) | target |
| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1,680,271,648 B | MTP draft head |
| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,145,888 B | vision tower — **unused** (`--no-mmproj`) |

The AGENT routing keeps 13 of the 17 `*.output.weight` tensors at full Q8_0
(12 more than the plain `Q8_0_ROCMFPX` build), which raises MTP draft
acceptance (0.953 vs 0.911 on the author's gfx1151 box). Take this variant
only when serving with MTP. Fetch with `scripts/download-q8agent.sh`
(resumable, size-verified; run `--verify-only` to re-check).

The model card documents 131,072 context; llama.cpp reports
`n_ctx_train (262144)` from the GGUF, but the author never tested beyond 131k,
so the deployment uses 131,072.

## Parallelism: layer split, not tensor (measured)

llama-bench on the host (pp4096 / tg128, same at depth 32768, f16 KV,
flash-attn on, 3 reps):

| Split | pp4096 | tg128 | pp4096 @d32768 | tg128 @d32768 |
|-------|-------:|------:|---------------:|--------------:|
| 1x R9700 (`-sm none`) | 1160.9 | 19.42 | ~512 | ~18.4 |
| **2x layer (`-sm layer`)** | **1991.2** | 18.76 | **859.1** | 17.64 |
| 2x tensor (`-sm tensor`) | 271.8 | 26.26 | 228.6 | 25.09 |

Tensor parallel wins decode by ~40 % but collapses prefill **7x**: this build
ships without RCCL, so llama.cpp logs
`internal AllReduce init failed (n_devices != 2?); falling back to meta-backend
butterfly` and every layer pays a slow cross-card reduction. The workload is
prefill-heavy (agentic, long prompts, reasoning traces), so the operator
switched to layer split on 2026-09-15. Revisit if the fork ships a working
RCCL tensor path (`-DGGML_HIP_RCCL=ON`).

Served-path comparison with MTP on (real completions): layer 35.7–46.2 tok/s
vs tensor 28.7 tok/s — layer wins decode too once the draft head is in play.

### llama-bench multi-device gotcha

`llama-bench` and `llama-server` disagree on device syntax. In llama-bench,
`-dev ROCm0,ROCm1` means *two separate single-card benches* (the value is
split on `,` into separate device sets, one model load each); a single split
bench needs `-dev ROCm0/ROCm1`. The invalid form silently produces
single-card numbers in a table that looks like a split test.

### Tensor-mode draft abort (if tensor is ever revisited)

Under `--split-mode tensor` the MTP draft head must be pinned to exactly one
device (`--spec-draft-device ROCm0`); with it empty the draft joins the
meta-device KV allocation and the server aborts while building the KV cache:

```
ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
  <- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor
```

## Deployment

| File | Role |
|------|------|
| `scripts/frank-medium.env` | all tunables (sampler, split, draft, context) |
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher |
| `scripts/frank-medium.service` | root systemd unit (relinked into /etc/systemd/system) |
| `scripts/download-q8agent.sh` | model fetch + size verification |

```bash
./scripts/deploy.sh dankserver --service frank-medium
```

First deploy (2026-09-15) also ran `retire_root_llm_small()` in
`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
disabled and moved to `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units`
(the host's `systemctl disable` has been observed to delete unit files,
2026-08-26 incident, so both outcomes are tolerated).

Deployed flags (see `scripts/frank-medium.env` for the full set):
`--split-mode layer --device ROCm0,ROCm1 --gpu-layers 999 --fit off`,
`--ctx-size 131072 --parallel 1`, f16 K/V, flash-attn on,
`--spec-type draft-mtp` (n_max 6, p_min 0.6) with the draft pinned to
`ROCm1` — the card holding the target model's last layers, where the final
hidden state is produced. Reasoning on, deepseek format, unlimited budget;
sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8,
top-k 20, min-p 0, presence 1.5). Text only — vision stays on
`frank-large` (Dankclaw).

## Verified serving behavior (2026-09-15)

- Health `{"status":"ok"}` ~18 s after start; model load ~17 s (warm page cache).
- MTP engaged: draft acceptance 60–69 % across prompts
  (`timings.draft_n` / `draft_n_accepted` in every chat response).
- Decode 35.7–46.2 tok/s on a 200-token budget; cold prompt 949 tokens
  prefilled in 2036 ms (~466 tok/s at ubatch 512).
- **Prefix caching works with the draft head**: the model card's
  `mtp-prompt-cache-fix.patch` defect (draft head disables prefix caching) is
  already fixed in `c49ebdb` — the speculative boundary rides inside the
  context checkpoint (`common_prompt_checkpoint::data_spec`). Measured: turn 2
  of a conversation reprocessed 22 tokens (`cache_n` 945) instead of 949.
- Quirk: the request-level `"speculative": {"n_max": 0}` override does **not**
  disable drafting (still drafts and accepts). To serve without MTP, restart
  with `SPEC_TYPE=none`.

## Client contract

OpenAI-compatible `/v1` on port 7770, alias `frank-medium`. Reasoning traces
are projected into `reasoning_content` (deepseek format); requests that send
no sampler fields get the server's CLI defaults, not greedy. Text only.
Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default`
→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small
routes) → `frank-medium`, vision stays `frank-large`.

## Host ROCm posture (post-cleanup)

Historically the host had a broken mix: AMD-repo ROCm 6.3.3 in
`/opt/rocm-6.3.3` hijacked `libhsa-runtime64` via
`/etc/ld.so.conf.d/10-rocm-opencl.conf`, breaking Ubuntu's ROCm 7.1 userspace
(`undefined reference to 'hsa_amd_memory_get_preferred_copy_engine'`).
On 2026-09-15 the stale 6.3.3 packages (~24.04-suffixed, 167 pkgs) were
purged, the hijack conf and the `repo.radeon.com` apt pin moved to
`~/.AGENTTRASH/20260915-rocm633-cleanup/`, and `ldconfig` now resolves only
Ubuntu's ROCm 7.1 libs. `rocm-smi` is Ubuntu's own 7.1.1 package. The
self-contained toolchain in `~/.local/share/rocm-7.14.0a20260624` does not
depend on any of it.

## Troubleshooting

- **Wrong arch build**: a gfx1200 binary on this card links, loads, then
  segfaults. Rebuild with the pinned script and check the "Build complete for
  gfx1200;gfx1201" line plus the emitted code objects.
- **Crash at load**: read the journal (`sudo journalctl -u frank-medium -n 100`),
  then run the serve script directly; the bundled libs make the script and the
  raw binary behave identically. The known abort shape is the tensor-mode
  draft KV abort above.
- **MTP "not engaged"**: check `timings.draft_n` in a response; zero means the
  draft head is not loaded (look for `common_speculative_init: adding
  speculative implementation 'draft-mtp'` in the journal).
- **Model files missing/truncated**: `bash scripts/download-q8agent.sh
  --verify-only`.

## See Also

- [[freetoken-frank-small]] — the retired predecessor (FreeToken/CUDA on the 3090)
- [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family
- [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes
- [[Dankserver System Specs]] — current hardware
- [[deployment]] — the deploy flow