Frank
ab51d61198
The runbook for the new deployment, plus the sweep of everything that still described the vllm-radiance replica pair: the vault index and its ACTIVE marker, the superseded banner on the radiance page, the pointer from the ROCmFPX page, the homepage card, the Grafana dashboard description and the observability page's scrape table. The new page carries the two facts a future agent needs and the flags do not say: llama.cpp's layer split is what lets BOTH cards serve ONE model without any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a healthy-looking build fail on the first request.
221 lines
10 KiB
Markdown
221 lines
10 KiB
Markdown
---
|
||
type: configuration
|
||
title: frank-medium ROCmFPX Runbook
|
||
description: charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201).
|
||
tags:
|
||
- frank-medium
|
||
- dankserver
|
||
- rocmfpx
|
||
- llama-cpp
|
||
- mtp
|
||
- gfx1201
|
||
- r9700
|
||
timestamp: 2026-09-15
|
||
---
|
||
|
||
> **PRIOR deployment (retired 2026-09-16).** This page documents the
|
||
> llama.cpp era of `frank-medium`, which served the endpoint for part of
|
||
> 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE
|
||
> deployment is [[frank-medium-llamacpp-dflash2]]; this page is kept as the
|
||
> record of the runtime it replaced.
|
||
|
||
# frank-medium ROCmFPX Runbook
|
||
|
||
`frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15,
|
||
when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards
|
||
(Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place.
|
||
It replaced the retired FreeToken/CUDA `frank-small`
|
||
([[freetoken-frank-small]]), which cannot run on this hardware at all.
|
||
|
||
Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22)
|
||
serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the
|
||
companion MTP draft head. Stock llama.cpp cannot load this GGUF —
|
||
`Q8_0_ROCMFPX_AGENT` is ftype 115 / `GGML_TYPE_Q8_0_ROCMFPX` (type 103) and
|
||
exists only in the fork.
|
||
|
||
## Hardware
|
||
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), rev c0 |
|
||
| Architecture | gfx1201 (Navi 48, RDNA4) |
|
||
| VRAM | 32,624 MiB each (rocm-smi) |
|
||
| Link | PCIe, no XGMI; cards sit in different IOMMU groups |
|
||
| iGPU | Raphael (gfx1036) enumerates as ROCm device 2 — hidden from HIP |
|
||
| CPU / RAM | Ryzen 7 7700X (8c/8t), 60 GiB RAM, 63 GiB swap |
|
||
| Kernel driver | in-tree `amdgpu`; no new driver needed |
|
||
|
||
`HIP_VISIBLE_DEVICES=0,1` hides the iGPU so nothing can drift onto shared
|
||
system RAM. Devices are addressed as `ROCm0,ROCm1` (llama-server) or
|
||
`ROCm0/ROCm1` (llama-bench multi-device syntax — see the bench gotcha below).
|
||
|
||
## Runtime build (self-contained ROCm 7.14)
|
||
|
||
The host ROCm install is quarantined rather than repaired: the build uses the
|
||
fork's self-contained toolchain, so the binary carries its own
|
||
`libamdhip64`/`libhsa-runtime64`/`rocblas`/`hipblaslt` with
|
||
`RPATH=$ORIGIN/../lib` and never touches system ROCm.
|
||
|
||
```bash
|
||
cd /home/dank/GitPackages/charlie-ROCmFPX
|
||
env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
|
||
# -> build-rdna4-rocm714/bin/llama-server
|
||
```
|
||
|
||
The script downloads the gfx120X-all TheRock ROCm 7.14 nightly tarball into
|
||
`~/.local/share/rocm-7.14.0a20260624` (~17 GiB extracted; tarball kept next to
|
||
the build tree) and builds `gfx1200;gfx1201`.
|
||
|
||
**gfx1201 is not interchangeable with gfx1200** — the fork's
|
||
`docs/BUILD-AMD-ARCHITECTURES.md` warns a wrong-sibling build links, loads the
|
||
model, then segfaults. `scripts/rocmfpx-hip-arch.sh` auto-detects and refuses.
|
||
|
||
## Model
|
||
|
||
| File | Size | Role |
|
||
|------|------|------|
|
||
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28,214,425,952 B (26.28 GiB, 8.39 bpw) | target |
|
||
| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1,680,271,648 B | MTP draft head |
|
||
| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,145,888 B | vision tower — **unused** (`--no-mmproj`) |
|
||
|
||
The AGENT routing keeps 13 of the 17 `*.output.weight` tensors at full Q8_0
|
||
(12 more than the plain `Q8_0_ROCMFPX` build), which raises MTP draft
|
||
acceptance (0.953 vs 0.911 on the author's gfx1151 box). Take this variant
|
||
only when serving with MTP. Fetch with `scripts/download-q8agent.sh`
|
||
(resumable, size-verified; run `--verify-only` to re-check).
|
||
|
||
The model card documents 131,072 context; llama.cpp reports
|
||
`n_ctx_train (262144)` from the GGUF, but the author never tested beyond 131k,
|
||
so the deployment uses 131,072.
|
||
|
||
## Parallelism: layer split, not tensor (measured)
|
||
|
||
llama-bench on the host (pp4096 / tg128, same at depth 32768, f16 KV,
|
||
flash-attn on, 3 reps):
|
||
|
||
| Split | pp4096 | tg128 | pp4096 @d32768 | tg128 @d32768 |
|
||
|-------|-------:|------:|---------------:|--------------:|
|
||
| 1x R9700 (`-sm none`) | 1160.9 | 19.42 | ~512 | ~18.4 |
|
||
| **2x layer (`-sm layer`)** | **1991.2** | 18.76 | **859.1** | 17.64 |
|
||
| 2x tensor (`-sm tensor`) | 271.8 | 26.26 | 228.6 | 25.09 |
|
||
|
||
Tensor parallel wins decode by ~40 % but collapses prefill **7x**: this build
|
||
ships without RCCL, so llama.cpp logs
|
||
`internal AllReduce init failed (n_devices != 2?); falling back to meta-backend
|
||
butterfly` and every layer pays a slow cross-card reduction. The workload is
|
||
prefill-heavy (agentic, long prompts, reasoning traces), so the operator
|
||
switched to layer split on 2026-09-15. Revisit if the fork ships a working
|
||
RCCL tensor path (`-DGGML_HIP_RCCL=ON`).
|
||
|
||
Served-path comparison with MTP on (real completions): layer 35.7–46.2 tok/s
|
||
vs tensor 28.7 tok/s — layer wins decode too once the draft head is in play.
|
||
|
||
### llama-bench multi-device gotcha
|
||
|
||
`llama-bench` and `llama-server` disagree on device syntax. In llama-bench,
|
||
`-dev ROCm0,ROCm1` means *two separate single-card benches* (the value is
|
||
split on `,` into separate device sets, one model load each); a single split
|
||
bench needs `-dev ROCm0/ROCm1`. The invalid form silently produces
|
||
single-card numbers in a table that looks like a split test.
|
||
|
||
### Tensor-mode draft abort (if tensor is ever revisited)
|
||
|
||
Under `--split-mode tensor` the MTP draft head must be pinned to exactly one
|
||
device (`--spec-draft-device ROCm0`); with it empty the draft joins the
|
||
meta-device KV allocation and the server aborts while building the KV cache:
|
||
|
||
```
|
||
ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
|
||
<- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor
|
||
```
|
||
|
||
## Deployment
|
||
|
||
| File | Role |
|
||
|------|------|
|
||
| `scripts/frank-medium.env` | all tunables (sampler, split, draft, context) |
|
||
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher |
|
||
| `scripts/frank-medium.service` | root systemd unit (relinked into /etc/systemd/system) |
|
||
| `scripts/download-q8agent.sh` | model fetch + size verification |
|
||
|
||
```bash
|
||
./scripts/deploy.sh dankserver --service frank-medium
|
||
```
|
||
|
||
First deploy (2026-09-15) also ran `retire_root_llm_small()` in
|
||
`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
|
||
disabled and moved to `~/.AGENTTRASH/20260915-frank-small-retire/host-units`
|
||
(the host's `systemctl disable` has been observed to delete unit files,
|
||
2026-08-26 incident, so both outcomes are tolerated).
|
||
|
||
Deployed flags (see `scripts/frank-medium.env` for the full set):
|
||
`--split-mode layer --device ROCm0,ROCm1 --gpu-layers 999 --fit off`,
|
||
`--ctx-size 131072 --parallel 1`, f16 K/V, flash-attn on,
|
||
`--spec-type draft-mtp` (n_max 6, p_min 0.6) with the draft pinned to
|
||
`ROCm1` — the card holding the target model's last layers, where the final
|
||
hidden state is produced. Reasoning on, deepseek format, unlimited budget;
|
||
sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8,
|
||
top-k 20, min-p 0, presence 1.5). Text only — vision stays on
|
||
`frank-large` (Dankclaw).
|
||
|
||
## Verified serving behavior (2026-09-15)
|
||
|
||
- Health `{"status":"ok"}` ~18 s after start; model load ~17 s (warm page cache).
|
||
- MTP engaged: draft acceptance 60–69 % across prompts
|
||
(`timings.draft_n` / `draft_n_accepted` in every chat response).
|
||
- Decode 35.7–46.2 tok/s on a 200-token budget; cold prompt 949 tokens
|
||
prefilled in 2036 ms (~466 tok/s at ubatch 512).
|
||
- **Prefix caching works with the draft head**: the model card's
|
||
`mtp-prompt-cache-fix.patch` defect (draft head disables prefix caching) is
|
||
already fixed in `c49ebdb` — the speculative boundary rides inside the
|
||
context checkpoint (`common_prompt_checkpoint::data_spec`). Measured: turn 2
|
||
of a conversation reprocessed 22 tokens (`cache_n` 945) instead of 949.
|
||
- Quirk: the request-level `"speculative": {"n_max": 0}` override does **not**
|
||
disable drafting (still drafts and accepts). To serve without MTP, restart
|
||
with `SPEC_TYPE=none`.
|
||
|
||
## Client contract
|
||
|
||
OpenAI-compatible `/v1` on port 7770, alias `frank-medium`. Reasoning traces
|
||
are projected into `reasoning_content` (deepseek format); requests that send
|
||
no sampler fields get the server's CLI defaults, not greedy. Text only.
|
||
Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default`
|
||
→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small
|
||
routes) → `frank-medium`, vision stays `frank-large`.
|
||
|
||
## Host ROCm posture (post-cleanup)
|
||
|
||
Historically the host had a broken mix: AMD-repo ROCm 6.3.3 in
|
||
`/opt/rocm-6.3.3` hijacked `libhsa-runtime64` via
|
||
`/etc/ld.so.conf.d/10-rocm-opencl.conf`, breaking Ubuntu's ROCm 7.1 userspace
|
||
(`undefined reference to 'hsa_amd_memory_get_preferred_copy_engine'`).
|
||
On 2026-09-15 the stale 6.3.3 packages (~24.04-suffixed, 167 pkgs) were
|
||
purged, the hijack conf and the `repo.radeon.com` apt pin moved to
|
||
`~/.AGENTTRASH/20260915-rocm633-cleanup/`, and `ldconfig` now resolves only
|
||
Ubuntu's ROCm 7.1 libs. `rocm-smi` is Ubuntu's own 7.1.1 package. The
|
||
self-contained toolchain in `~/.local/share/rocm-7.14.0a20260624` does not
|
||
depend on any of it.
|
||
|
||
## Troubleshooting
|
||
|
||
- **Wrong arch build**: a gfx1200 binary on this card links, loads, then
|
||
segfaults. Rebuild with the pinned script and check the "Build complete for
|
||
gfx1200;gfx1201" line plus the emitted code objects.
|
||
- **Crash at load**: read the journal (`sudo journalctl -u frank-medium -n 100`),
|
||
then run the serve script directly; the bundled libs make the script and the
|
||
raw binary behave identically. The known abort shape is the tensor-mode
|
||
draft KV abort above.
|
||
- **MTP "not engaged"**: check `timings.draft_n` in a response; zero means the
|
||
draft head is not loaded (look for `common_speculative_init: adding
|
||
speculative implementation 'draft-mtp'` in the journal).
|
||
- **Model files missing/truncated**: `bash scripts/download-q8agent.sh
|
||
--verify-only`.
|
||
|
||
## See Also
|
||
|
||
- [[freetoken-frank-small]] — the retired predecessor (FreeToken/CUDA on the 3090)
|
||
- [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family
|
||
- [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes
|
||
- [[Dankserver System Specs]] — current hardware
|
||
- [[deployment]] — the deploy flow
|