serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

---
type: configuration
title: frank-medium llama.cpp + DFlash2 Runbook
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.
tags:
  - frank-medium
  - dankserver
  - llama-cpp
  - dflash2
  - qwen35
  - gfx1201
  - r9700
  - layer-split
timestamp: 2026-09-16
---

# frank-medium — llama.cpp + DFlash2

The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
with the **z-lab DFlash2** block-diffusion drafter, as **one process across
both Radeon AI PRO R9700 cards**, layer-split.

It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the
replicas, the proxy and their unit files are retired.

## Why one process and layer split

The endpoint's whole history is one constraint: **the two cards cannot take
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
and ROCr refuses to supply it to a device without PCIe atomics; the second
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
every collective that touches it dies with `hipErrorIllegalState`. That rules
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
whose stage handoff is a `send`/`recv` of the hidden state on the device
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
The full matrix is in [[Dual-R9700-Cooperation]].

llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
distributed across the cards in order and the boundary activation is a plain
per-backend copy, once per forward pass. Measured on this host with the
predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):

| Split | pp4096 | tg128 |
|---|---:|---:|
| one R9700 (`-sm none`) | 1160.9 | 19.42 |
| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |

Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
shape — prefill nearly doubles and one model serves from both cards.

## Hardware

| Property | Value |
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |

## Runtime

Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
toolchain (TheRock nightly) that is **bundled next to the binary** —
`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
packages.

Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
earlier the same day (its tip, `c49ebdb`, predates both):

- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
  convolution plus the candidate selector, detected from the draft GGUF
  (`selector_top_k > 0`) and served through the ordinary
  `--spec-type draft-dflash` path.
- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
  single trained head.

Build with:

```bash
bash scripts/build-llama-rdna4.sh     # ~25 min on this box, JOBS=$(nproc)
```

> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
> failure is invisible at startup: the engine loads 29 GiB of weights, answers
> `/health`, and dies on the **first request** with
> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
> such file or directory for GPU arch : gfx1201`. The build script now declares
> the destination and copies contents.

## Model and drafter

| File | Bytes | Role |
|---|---:|---|
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |

Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
a truncated 31 GB GGUF is otherwise a load-time crash).

The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
the same size, on Unsloth's importance matrix) of an abliterated Swift
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.

**The file also carries the model's MTP head** (`blk.64.nextn.*`). With
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled by default because the DFlash2 path is the measured
one.

## Deployment

| File | Role |
|---|---|
| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |

```bash
./scripts/deploy.sh dankserver --service frank-medium
```

The handler verifies the runtime and payload exist (and refuses otherwise),
retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
waits for **port 7770 to be released** (the retired proxy held it on the host
network), relinks and restarts the unit, and gates on
`http://127.0.0.1:7770/health`.

Deployed flags:

| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `--no-mmproj` | — | text only |

## Verified serving behaviour (2026-09-16)

- Load: `model loaded` ~29 s after start with a warm page cache, then
  `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports
  `modalities: {vision: false}`, one slot, and the sampler defaults above.
- DFlash2 engaged — the init line is the proof, not the flag:

  ```
  common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
  common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
  common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
  ```

  `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
  what distinguishes it from DFlash 1.
- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on
  GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention
  layers' KV and the compute buffers all sit on the second card, which keeps
  ~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB.
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one
  slot: the raised window is real, not just declared, and it is 22 % beyond the
  114688 the vLLM predecessor could offer.

- **Nothing is replicated.** The loader does not print a per-device ledger at
  this verbosity, so this is measured instead of read: raising CTX_SIZE from
  131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards
  carry the KV for their own attention layers, about half each — and neither
  card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB
  file, so no card carries a full copy of the weights. Compare the shape this
  replaced: two vLLM replicas, i.e. two complete copies of the model, one per
  card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there
  at any usable context.

### Measured throughput (2026-09-16, deployed service, `temperature 0`)

Same filler corpus for every shape, so these are shapes rather than a
benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
with a long budget isolates decode. `draft`/`acc` are per request; acceptance
moves with the text (filler prose accepts far worse than code or a real agent
loop).

| Shape | Prefill | Decode | Acceptance |
|---|---|---|---|
| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |

The last row is the prefix cache doing its job: the second request of the same
conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
no proxy.

### What the drafter is worth — and why MTP is not the alternative

Measured 2026-09-16 with `scripts/bench-spec-ab.py`, four configurations of the
same binary, identical prompts at `temperature 0`, unit stopped between runs
(one host, one copy of a 29 GiB model). 1k-token prompt, 512-token budget:

| Configuration | Decode | Drafting | VRAM (card 1) |
|---|---:|---|---:|
| `none` — autoregressive | 17.7 tok/s | — | 26.07 GB |
| `draft-mtp` n_max 3 (the model's own head) | 32.4 tok/s | 58 % of 72 | 29.35 GB |
| `draft-mtp` n_max 7 | 26.7 tok/s | 33 % of 147 | 29.65 GB |
| **`draft-dflash` n_max 7 — deployed** | **43.5 tok/s** | 41 % of 119 | 29.42 GB |

- **DFlash2 is worth 2.46× over autoregressive and 34 % over the best MTP.**
  Deeper MTP drafting costs more than it returns: at depth 7 only a third of
  the proposals survive, at depth 3 more than half do.
- **All four produced identical greedy text** (`2, 3, 5, 7, 11`): speculation
  buys speed, never a different answer. The harness warns loudly if a future
  configuration ever breaks that.
- **MTP is not the cheaper option.** Selecting it loads `blk.64.nextn.*` from
  the target and adds ~3.3 GB to the card that holds it, against 2.06 GB for
  the drafter's file — which is also why the load log says
  `unused tensor blk.64.nextn... ignoring` under `draft-dflash`: correct
  behaviour, not a truncated download.
- **They cannot be chained.** `--spec-type draft-mtp,draft-dflash` dies at load
  with `context type MTP requested but model doesn't contain MTP layers`. A
  chain has ONE draft context, and MTP builds against whatever model that
  context holds — here the DFlash2 file. MTP's head is loaded only when MTP *is*
  the draft model (`creating MTP draft context against the target model ...`).
  That is structural, not a tuning question.

### Chaining the draftless n-gram matcher (deployed)

`ngram-mod` **can** ride beside DFlash2, and it is enabled. The reason is in
`common/speculative.cpp`: the n-gram types are added to the config vector
unconditionally while every model-backed drafter is gated on
`ctx_dft != nullptr` — so n-gram needs no draft context and cannot collide with
the drafter over one. (MTP is the opposite: it *is* context-backed, which is why
`draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types
first, then the drafters — also means CLI order is irrelevant.

Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a
proposer can and cannot match:

| Configuration | prose 1k/512 | echo 1.5k/400 (copy 0.94) | structured/400 | identity |
|---|---:|---:|---:|---:|
| `draft-dflash@7` | 43.77 | 76.82 | 91.08 | 70.37 |
| `ngram-mod` alone | 17.72 *(0 drafts)* | 102.50 | 17.80 *(0 drafts)* | 17.77 |
| **`draft-dflash,ngram-mod@7`** | **43.65** | **200.58** | 91.22 | 70.40 |

- **On prompt-echo traffic the chain is 2.6× the drafter alone** (200.6 vs 76.8)
  and twice `ngram-mod` by itself, at 100 % acceptance.
- **On prose it is indistinguishable from DFlash2 alone** — the same 119 drafts
  and 49 accepts, the same 43.7 tok/s. That is the property that makes it safe
  as a default: where the matcher cannot match, it costs nothing.
- `ngram-mod` alone is a *negative control*: zero drafts on prose and identity
  (17.7 tok/s = the autoregressive baseline), 100 % acceptance on echo. Nothing
  else could have proposed those tokens, which is how its drafts are identified
  at all — see the trap below.
- VRAM is unchanged (29.42 GB on card 1 with or without it): the matcher is a
  64 MiB host-RAM hash table.
- `structured` (25 records with unique ids) rewards no proposer: its lines
  differ, so there is no long span to match. It is a negative control for
  *self-repetition*, not evidence about it.

**The window matters, and shorter is not better.** The chain at three windows:

| `SPEC_NGRAM_N_MIN`/`N_MAX` | prose 1k/512 | echo 1.5k/400 |
|---|---:|---:|
| **48/64 — deployed** | **43.9** | **200.6** |
| 16/32 | 36.0 | 161.0 |
| 8/16 | 36.2 | 113.8 |

A short window makes the matcher propose on ordinary prose fragments, where the
proposals are rejected and the verification is still paid — the "ngram hurts"
result this fleet measured twice before on other payloads. At 48/64 the chain
keeps DFlash2's exact numbers on prose and wins on echo.

> **Operational trap: `ngram-mod` prints no startup banner in this build.**
> `journalctl -u frank-medium` shows `adding speculative implementation
> 'draft-dflash'` and nothing for the matcher — the only evidence it is running
> is behavioural (`draft_n` > 0 where text repeats) or the per-task line
> `slot print_timing: ... draft acceptance = 1.00000 (256 accepted / 256
> generated), mean len = 65.00`. Do not conclude from the load log that the
> chain is not engaged.

### The agent-loop shape (fleet harness, 2 families × 2 turns)

`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
agent loops re-sending a ~30k-token document concurrently:

| | turn 1 | turn 2 |
|---|---:|---:|
| family 1 | 24,799 ms | 30,697 ms |
| family 0 | 51,424 ms | 8,015 ms |

Read that against the prefill table, not as a latency regression: one slot
means the second family's first turn waits for the first (25 s of cold 30k
prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that
the cache could only partly serve. **The single slot is now the limiting
resource for concurrent agent loops**, which is the trade this deployment makes
for giving one conversation the whole 262144-token window instead of splitting
two. Raising `--parallel` splits the window (and the KV) between slots; it does
not add throughput.

## Observability

The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
(`--metrics`), which is what the fleet's scrape and dashboard read — the
`vllm:` series the predecessor exported no longer exist:

| Need | Series |
|---|---|
| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |

`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
drafter actually running": the flags can all be right while the draft model
failed to load.

## Client contract

OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
are projected into `reasoning_content`. A request that sends no sampler fields
gets the server defaults above; a client that sends its own wins. Text only.

## Troubleshooting

- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
  bundled device-library depth (see the bundling trap above); check
  `build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
  `0,1`; the iGPU enumerates as device 2 and must stay hidden.
- **Drafting not engaged** — read the journal for
  `common_speculative_impl_draft_dflash`; a missing drafter file makes the
  serve script exit before llama-server starts, and a non-DFlash2 drafter (or
  an older llama.cpp) fails the load instead.
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
  correct for `draft-dflash`. It is not a truncated download:
  `bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
  gfx1200 sibling binary links, loads the model and then segfaults on these
  cards.

## See Also

- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
  (replicas + proxy) and the PCIe-atomics evidence
- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
  2026-09-15, whose `--split-mode layer` measurements this service reuses
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
  collective, and every software escape that is closed
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
  3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory