serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

---
type: configuration
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split, 262144 ctx over two slots.
tags:
  - frank-medium
  - dankserver
  - llama-cpp
  - dflash2
  - dspark
  - qwen35
  - gfx1201
  - r9700
  - layer-split
timestamp: 2026-09-16
---

# frank-medium — llama.cpp (DFlash2 drafter)

The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
with the **z-lab DFlash2** block-diffusion drafter, as **one process across both
Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's
place for a few hours the same day and reverted — see *The DSpark excursion*.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the
replicas, the proxy and their unit files are retired.

## Why one process and layer split

The endpoint's whole history is one constraint: **the two cards cannot take
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
and ROCr refuses to supply it to a device without PCIe atomics; the second
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
every collective that touches it dies with `hipErrorIllegalState`. That rules
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
whose stage handoff is a `send`/`recv` of the hidden state on the device
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
The full matrix is in [[Dual-R9700-Cooperation]].

llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
distributed across the cards in order and the boundary activation is a plain
per-backend copy, once per forward pass. Measured on this host with the
predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):

| Split | pp4096 | tg128 |
|---|---:|---:|
| one R9700 (`-sm none`) | 1160.9 | 19.42 |
| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |

Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
shape — prefill nearly doubles and one model serves from both cards.

## Hardware

| Property | Value |
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |

## Runtime

Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
toolchain (TheRock nightly) that is **bundled next to the binary** —
`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
packages.

Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
earlier the same day (its tip, `c49ebdb`, predates both):

- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
  convolution plus the candidate selector, detected from the draft GGUF
  (`selector_top_k > 0`) and served through the ordinary
  `--spec-type draft-dflash` path.
- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
  single trained head.

Build with:

```bash
bash scripts/build-llama-rdna4.sh     # ~25 min on this box, JOBS=$(nproc)
```

> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
> failure is invisible at startup: the engine loads 29 GiB of weights, answers
> `/health`, and dies on the **first request** with
> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
> such file or directory for GPU arch : gfx1201`. The build script now declares
> the destination and copies contents.

## Model and drafter

| File | Bytes | Role |
|---|---:|---|
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |

Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
a truncated 31 GB GGUF is otherwise a load-time crash).

The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
the same size, on Unsloth's importance matrix) of an abliterated Swift
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.

**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled because the DFlash2 path is the measured one (below)
and because a chain cannot hold two context-backed drafters at all.

## Deployment

| File | Role |
|---|---|
| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |

```bash
./scripts/deploy.sh dankserver --service frank-medium
```

The handler verifies the runtime and payload exist (and refuses otherwise),
retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
waits for **port 7770 to be released** (the retired proxy held it on the host
network), relinks and restarts the unit, and gates on
`http://127.0.0.1:7770/health`.

Deployed flags:

| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `--no-mmproj` | — | text only |

## The DSpark excursion (2026-09-16, tried and reverted)

The endpoint went live on **z-lab DFlash2**, was switched to
**magnitudedev/Qwen3.8-27B-DSpark** for a few hours the same day, and was put
back. Both are 5-layer block-diffusion sidecars, and their differences are what
decided it:

| | DFlash2 (deployed) | DSpark (tried, reverted) |
|---|---|---|
| file | 2,056,414,816 B — 1.92 GiB | 1,455,376,576 B — 1.36 GiB |
| draft depth | `block_size 8` -> `n_max 7` | `block_size 7` -> `n_max 6` (llama.cpp asserts `n_max <= block_size - 1`) |
| attention | `sliding_window 2048`, `causal false`, candidate selector (rank 256, top-k 16) | **no sliding window**; a Markov head in place of the selector |
| draft KV at 262144 | ~40 MiB — the window bounds it | **5.0 GiB** — the whole context |

> **A fetch-script bug this threw up.** The DSpark file's size was recorded as
> 1,359,284,737 when it was fetched — 96 MB short of the Hub's 1,455,376,576 —
> and that wrong number became the fetch script's declared size, so
> `--verify-only` failed on a payload that was in fact byte-correct. Both
> drafters' sha256 have since been checked against the Hub's LFS oids and match.

**The draft context is sized from the target's.** `cparams.n_ctx =
llama_n_ctx(ctx_tgt)` in `common/speculative.cpp`, with no flag to cap it
independently, so a drafter that declares no sliding window caches the entire
target window on whichever device `--spec-draft-device` names. DSpark's KV is
5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position.

That cost has to be paid somewhere, and the failure was instructive: at a 1:1
split the model **loaded fine, answered `/health`, and then died on the first
draft decode** with `ROCm error: out of memory` inside
`common_speculative_impl_draft_dflash::process` — ROCm1 sat at 33.9 GB of
34.2 GB. A unit that crash-loops with an allocation failure *after* a
successful load is a sizing failure, not a broken drafter.

Read `sliding_window` out of a drafter's GGUF header before sizing a
spec-decode deployment — 40 lines of `struct` parsing is enough, and the
flag-level configuration looks identical either way.

### The split moved twice: once for the batch, once for the drafter

**Under DSpark.** Tripling the batch (2048/512 -> 6144/1536) made the two cards
diverge: at 262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against
ROCm0's ~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs —
which left the draft-carrying card at 0.08 GiB free against ROCm0's 3.78 under
the 38:27 split that had been balanced at the smaller batch:

| `-ts` | GPU 0 free | GPU 1 free |
|---|---:|---:|
| 38,27 | 3.78 GiB | 0.08 GiB |
| 41,24 | 1.47 GiB | 2.38 GiB |
| 40,25 | 1.88 GiB | 1.88 GiB |

**Then DSpark came off**, which frees ~4.3 GiB on ROCm1 (5.0 GiB of draft KV
out, 0.65 GiB of heavier DFlash2 weights in) and moves the balance the other
way. Re-measured at the same batch, the same two slots and the same window:

| `-ts` | GPU 0 free | GPU 1 free | gap |
|---|---:|---:|---:|
| 39,26 | 2.42 GiB | 7.20 GiB | 4.78 |
| 38,27 | 3.85 GiB | 5.77 GiB | 1.92 |
| 37,28 | 4.28 GiB | 5.34 GiB | 1.06 |
| **36,29 — deployed** | **4.71 GiB** | **4.91 GiB** | **0.20** |
| 35,30 | 5.14 GiB | 4.48 GiB | 0.66 |
| 34,31 | 6.57 GiB | 3.05 GiB | 3.52 |

**One `-ts` step is one layer, and this stack's layers are not the same size.**
qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth
**~0.43 GiB** on a DeltaNet boundary and **~1.43 GiB** on a full-attention one
(0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144). Both fell out
of that table — 37,28 -> 38,27 moves a DeltaNet layer, 38,27 -> 39,26 an
attention layer. Balance by measurement, never by interpolating.

**Then the batch moved, and the split followed it** (2026-09-16, the same day
— see [[frank-medium-batch-ubatch]]). 36:29 was the balance at ubatch **1536**;
the sweep showed 512 is both faster and 3.71 GiB cheaper, and — because the
compute buffers cost ROCm1 nearly twice what they cost ROCm0 — shrinking them
handed ROCm1 1.31 GiB more than it handed ROCm0:

| `-ts` | ubatch | GPU 0 free | GPU 1 free | gap |
|---|---:|---:|---:|---:|
| 36,29 | 1536 | 4.71 GiB | 4.91 GiB | 0.20 |
| 36,29 | 512 | 6.01 GiB | 7.32 GiB | 1.31 |
| **35,30 — deployed** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46** |

35:30's 0.46 GiB gap is one DeltaNet layer's worth, which is the floor here: no
split can be tuned finer than a layer, so the 0.20 GiB of the old configuration
is not reachable at this batch. It clears the operator's 1.5 GiB-per-card floor
by ~5 GiB, so no headroom is bought out of `CTX_SIZE` and the model's full
262144-token window stays.

A batch of 8192/2048 failed here under DSpark — its 2020 MiB pp buffer could not
be allocated on ROCm1, which had 0.08 GiB free. That was a *contiguity* failure,
not a total-bytes one, and the sweep has since removed the reason to care: with
the batch geometry measured, there is nothing to gain by making either knob
larger, so the question is closed rather than pending.

> **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its
> eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the
> engine surfaced `ROCm error: an illegal memory access was encountered` ->
> `ggml_abort` — a fault in the cross-card boundary copy, which is this layer
> split's hot path. It has not reproduced (36:29 served the whole verification
> battery and the e2e gate, and 35:30 at ubatch 512 is the deployed shape now;
> 39:26 itself was only *loaded* at the deployed batch, not exercised), and it
> is **not** an OOM: no amount of headroom would prevent it.

## Verified serving behaviour (2026-09-16)

- Load: `model loaded` ~25 s after start with a warm page cache, then
  `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports
  `modalities: {vision: false}` and the sampler defaults above.
- DFlash2 engaged — the init lines are the proof, not the flags:

  ```
  common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf'
  common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
  common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
  common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
  ```

  `block_size=8` and `n_extract=5` are DFlash2's own metadata; the candidate
  selector is what distinguishes it from DFlash 1. DSpark logs under the same
  `draft_dflash` symbol with `block_size=7` and a Markov head, so read the
  numbers, not the implementation name.
- Both cards hold the model and they are **balanced**: at 262144 with the
  deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
  6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
  measurement of the same shape to 0.01 GiB. That is the reading straight after
  load; once the service has served traffic it settles ~0.2 GiB lower
  (**6.23 / 6.64**, the same with two concurrent requests in flight and after
  them). Both figures are ~5 GiB above the operator's 1.5 GiB floor.
- Drafting is live on **both slots**, which is what a 2-slot deployment has to
  prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
  70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
  `0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair.
  Acceptance moves with the text; the point is that neither slot is idle.
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) was measured
  through a *single* slot, at the previous configuration: the raised window is
  real rather than declared, and it is 22 % beyond the 114688 the vLLM
  predecessor could offer. The window is now split over two slots.

- **Nothing is replicated.** The loader does not print a per-device ledger at
  this verbosity, so this is measured instead of read: raising CTX_SIZE from
  131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards
  carry the KV for their own attention layers, about half each — and neither
  card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB
  file, so no card carries a full copy of the weights. Compare the shape this
  replaced: two vLLM replicas, i.e. two complete copies of the model, one per
  card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there
  at any usable context.

### Measured throughput (2026-09-16, deployed service, `temperature 0`)

Same filler corpus for every shape, so these are shapes rather than a
benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
with a long budget isolates decode. `draft`/`acc` are per request; acceptance
moves with the text (filler prose accepts far worse than code or a real agent
loop).

| Shape | Prefill | Decode | Acceptance |
|---|---|---|---|
| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |

The last row is the prefix cache doing its job: the second request of the same
conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
no proxy.

### What the drafter is worth — and why MTP is not the alternative

Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter —
the one deployed now: four configurations of the same binary, identical prompts
at `temperature 0`, unit stopped between runs (one host, one copy of a 29 GiB
model). 1k-token prompt, 512-token budget. These are the numbers that selected a
trained multi-token drafter over the model's own MTP head.

| Configuration | Decode | Drafting | VRAM (card 1) |
|---|---:|---|---:|
| `none` — autoregressive | 17.7 tok/s | — | 26.07 GB |
| `draft-mtp` n_max 3 (the model's own head) | 32.4 tok/s | 58 % of 72 | 29.35 GB |
| `draft-mtp` n_max 7 | 26.7 tok/s | 33 % of 147 | 29.65 GB |
| **`draft-dflash` n_max 7 — deployed** | **43.5 tok/s** | 41 % of 119 | 29.42 GB |

- **DFlash2 is worth 2.46× over autoregressive and 34 % over the best MTP.**
  Deeper MTP drafting costs more than it returns: at depth 7 only a third of
  the proposals survive, at depth 3 more than half do.
- **All four produced identical greedy text** (`2, 3, 5, 7, 11`): speculation
  buys speed, never a different answer. The harness warns loudly if a future
  configuration ever breaks that.
- **MTP is not the cheaper option.** Selecting it loads `blk.64.nextn.*` from
  the target and adds ~3.3 GB to the card that holds it, against 2.06 GB for
  the drafter's file — which is also why the load log says
  `unused tensor blk.64.nextn... ignoring` under `draft-dflash`: correct
  behaviour, not a truncated download.
- **They cannot be chained.** `--spec-type draft-mtp,draft-dflash` dies at load
  with `context type MTP requested but model doesn't contain MTP layers`. A
  chain has ONE draft context, and MTP builds against whatever model that
  context holds — here the DFlash2 file. MTP's head is loaded only when MTP *is*
  the draft model (`creating MTP draft context against the target model ...`).
  That is structural, not a tuning question.

### Chaining the draftless n-gram matcher (deployed)

`ngram-mod` **can** ride beside a context-backed drafter, and it is enabled.
The reason is in `common/speculative.cpp`: the n-gram types are added to the config vector
unconditionally while every model-backed drafter is gated on
`ctx_dft != nullptr` — so n-gram needs no draft context and cannot collide with
the drafter over one. (MTP is the opposite: it *is* context-backed, which is why
`draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types
first, then the drafters — also means CLI order is irrelevant.

The measurements below are DFlash2's — the deployed configuration. (During the
DSpark excursion the chain was `draft-dspark,ngram-mod`; the matcher's behaviour
is a property of the matcher, not of the drafter beside it.)

Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a
proposer can and cannot match:

| Configuration | prose 1k/512 | echo 1.5k/400 (copy 0.94) | structured/400 | identity |
|---|---:|---:|---:|---:|
| `draft-dflash@7` | 43.77 | 76.82 | 91.08 | 70.37 |
| `ngram-mod` alone | 17.72 *(0 drafts)* | 102.50 | 17.80 *(0 drafts)* | 17.77 |
| **`draft-dflash,ngram-mod@7`** | **43.65** | **200.58** | 91.22 | 70.40 |

- **On prompt-echo traffic the chain is 2.6× the drafter alone** (200.6 vs 76.8)
  and twice `ngram-mod` by itself, at 100 % acceptance.
- **On prose it is indistinguishable from DFlash2 alone** — the same 119 drafts
  and 49 accepts, the same 43.7 tok/s. That is the property that makes it safe
  as a default: where the matcher cannot match, it costs nothing.
- `ngram-mod` alone is a *negative control*: zero drafts on prose and identity
  (17.7 tok/s = the autoregressive baseline), 100 % acceptance on echo. Nothing
  else could have proposed those tokens, which is how its drafts are identified
  at all — see the trap below.
- VRAM is unchanged (29.42 GB on card 1 with or without it): the matcher is a
  64 MiB host-RAM hash table.
- `structured` (25 records with unique ids) rewards no proposer: its lines
  differ, so there is no long span to match. It is a negative control for
  *self-repetition*, not evidence about it.

**The window matters, and shorter is not better.** The chain at three windows:

| `SPEC_NGRAM_N_MIN`/`N_MAX` | prose 1k/512 | echo 1.5k/400 |
|---|---:|---:|
| **48/64 — deployed** | **43.9** | **200.6** |
| 16/32 | 36.0 | 161.0 |
| 8/16 | 36.2 | 113.8 |

A short window makes the matcher propose on ordinary prose fragments, where the
proposals are rejected and the verification is still paid — the "ngram hurts"
result this fleet measured twice before on other payloads. At 48/64 the chain
keeps DFlash2's exact numbers on prose and wins on echo.

> **Operational trap: `ngram-mod` prints no startup banner in this build.**
> `journalctl -u frank-medium` shows `adding speculative implementation
> 'draft-dflash'` and nothing for the matcher — the only evidence it is running
> is behavioural (`draft_n` > 0 where text repeats) or the per-task line
> `slot print_timing: ... draft acceptance = 1.00000 (256 accepted / 256
> generated), mean len = 65.00`. Do not conclude from the load log that the
> chain is not engaged.

### The agent-loop shape (fleet harness, 2 families × 2 turns)

`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
agent loops re-sending a ~30k-token document concurrently:

| | turn 1 | turn 2 |
|---|---:|---:|
| family 1 | 24,799 ms | 30,697 ms |
| family 0 | 51,424 ms | 8,015 ms |

Read that against the prefill table, not as a latency regression: with **one**
slot the second family's first turn waited for the first (25 s of cold 30k
prefill + the wait ≈ 51 s), and the last number — 8.0 s — was a re-prefill the
cache could only partly serve. That was the 1-slot configuration. The
deployment now runs **two slots of 131072**, so a second agent loop has a slot
of its own instead of queueing, at the cost of halving the window each one
gets. `--parallel` splits the context and the KV between slots; it does not add
throughput, so the trade is concurrency against conversation length.

### Benchmarking this endpoint

`scripts/bench-batch-ubatch.sh` sweeps `--batch-size` against `--ubatch-size`
with `llama-bench` and records, per cell, the prefill rate, the decode rate and
each card's **peak** VRAM. Run it from a transient unit rather than a shell:

```bash
sudo systemd-run --unit=fm-bench --collect --no-block --property=Type=oneshot \
     --property=LimitMEMLOCK=infinity --property=User=dank \
     scripts/bench-batch-ubatch.sh
```

Properties of `llama-bench` shape that harness, and three of them have already
produced wrong numbers on this fleet:

- **It separates devices differently from `llama-server`.** `/` splits devices
  *within one configuration*, `,` splits *sweep values* — so this service's
  `--tensor-split 35,30` is `-ts 35/30` to llama-bench, and the comma form
  silently benches each card on its own instead. Likewise `--device
  ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
- **There is no `-c`.** The context is a consequence of the test shape:
  `n_ctx = n_prompt + n_gen + n_depth`. No bench can therefore prove that a
  configuration's activation buffers fit beside the *deployed* KV — that needs
  the live unit at 262144 across two slots.
- **A test that cannot allocate calls `exit(1)`**, taking the whole process with
  it, so the harness runs one invocation per cell and a failure costs one TSV
  line instead of the rest of the sweep.
- **A shell's memlock limit is not the unit's.** The unit sets
  `LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell here gets
  8 MiB, which fails the load for a reason that has nothing to do with the
  model being benched.
- **No `--spec-*` and no `--parallel`.** The drafter cannot take part, so a
  `llama-bench` decode figure is the *drafterless* baseline — 15.9 tok/s at 64k
  depth against the deployed 43.8 — and slots are never exercised. `-d <n>` is
  what puts KV in front of the measurement at all: a `tg` instance runs with
  `n_prompt = 0`, i.e. at ~zero depth unless `-d` says otherwise.

## Observability

The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
(`--metrics`), which is what the fleet's scrape and dashboard read — the
`vllm:` series the predecessor exported no longer exist:

| Need | Series |
|---|---|
| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |

`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
drafter actually running": the flags can all be right while the draft model
failed to load.

## Client contract

OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
are projected into `reasoning_content`. A request that sends no sampler fields
gets the server defaults above; a client that sends its own wins. Text only.

## Troubleshooting

- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
  bundled device-library depth (see the bundling trap above); check
  `build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
  `0,1`; the iGPU enumerates as device 2 and must stay hidden.
- **Drafting not engaged** — read the journal for
  `common_speculative_impl_draft_dflash` (DFlash2 and DSpark both log under that
  symbol); a missing drafter file makes the serve script exit before
  llama-server starts, and a drafter whose GGUF the build does not recognise
  fails the load instead.
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
  correct for `draft-dflash`. It is not a truncated download:
  `bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
  gfx1200 sibling binary links, loads the model and then segfaults on these
  cards.

## See Also

- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
  (replicas + proxy) and the PCIe-atomics evidence
- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
  2026-09-15, whose `--split-mode layer` measurements this service reuses
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
  collective, and every software escape that is closed
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
  3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
  geometry and moved this split from 36:29 to 35:30: the ubatch optimum at 512,
  why the logical batch is free, and the harness traps