Frank
dc76dc5dc4
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and 29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a 140,069-token prompt prefilled through the single slot in 134 s to prove the window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this runtime's. Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to 140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance, and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt. The signals move with the runtime, so the fleet's rules and dashboard are re-cut rather than renamed: vllm: series do not exist any more, and two of the old rules watched things llama.cpp does not export at all (KV-pool fill, p95 latency). Those are retired with the reason written down, and LlmSpecDecodeOff takes one of the slots -- a lost drafter is the failure no client can see. The observability seam now fails if any rule names a series the endpoint does not export, which is what would have caught the stale vllm: expressions. omp models.yml (outside the repo) is updated in the same pass: contextWindow 262144, and the full reasoning-effort map, because Qwen3.8's template answers 500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer 200. Subagents run at high.
257 lines
13 KiB
Markdown
257 lines
13 KiB
Markdown
---
|
||
type: configuration
|
||
title: frank-medium llama.cpp + DFlash2 Runbook
|
||
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.
|
||
tags:
|
||
- frank-medium
|
||
- dankserver
|
||
- llama-cpp
|
||
- dflash2
|
||
- qwen35
|
||
- gfx1201
|
||
- r9700
|
||
- layer-split
|
||
timestamp: 2026-09-16
|
||
---
|
||
|
||
# frank-medium — llama.cpp + DFlash2
|
||
|
||
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
|
||
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
|
||
with the **z-lab DFlash2** block-diffusion drafter, as **one process across
|
||
both Radeon AI PRO R9700 cards**, layer-split.
|
||
|
||
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
|
||
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
|
||
the same day the pair went live. It is also the box's **only** LLM runtime: the
|
||
replicas, the proxy and their unit files are retired.
|
||
|
||
## Why one process and layer split
|
||
|
||
The endpoint's whole history is one constraint: **the two cards cannot take
|
||
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
|
||
and ROCr refuses to supply it to a device without PCIe atomics; the second
|
||
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
|
||
every collective that touches it dies with `hipErrorIllegalState`. That rules
|
||
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
|
||
whose stage handoff is a `send`/`recv` of the hidden state on the device
|
||
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
|
||
The full matrix is in [[Dual-R9700-Cooperation]].
|
||
|
||
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
|
||
distributed across the cards in order and the boundary activation is a plain
|
||
per-backend copy, once per forward pass. Measured on this host with the
|
||
predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):
|
||
|
||
| Split | pp4096 | tg128 |
|
||
|---|---:|---:|
|
||
| one R9700 (`-sm none`) | 1160.9 | 19.42 |
|
||
| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
|
||
| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |
|
||
|
||
Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
|
||
prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
|
||
shape — prefill nearly doubles and one model serves from both cards.
|
||
|
||
## Hardware
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
|
||
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
|
||
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
|
||
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
|
||
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
|
||
|
||
## Runtime
|
||
|
||
Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
|
||
`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
|
||
toolchain (TheRock nightly) that is **bundled next to the binary** —
|
||
`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
|
||
packages.
|
||
|
||
Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
|
||
earlier the same day (its tip, `c49ebdb`, predates both):
|
||
|
||
- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
|
||
convolution plus the candidate selector, detected from the draft GGUF
|
||
(`selector_top_k > 0`) and served through the ordinary
|
||
`--spec-type draft-dflash` path.
|
||
- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
|
||
single trained head.
|
||
|
||
Build with:
|
||
|
||
```bash
|
||
bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc)
|
||
```
|
||
|
||
> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
|
||
> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
|
||
> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
|
||
> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
|
||
> failure is invisible at startup: the engine loads 29 GiB of weights, answers
|
||
> `/health`, and dies on the **first request** with
|
||
> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
|
||
> such file or directory for GPU arch : gfx1201`. The build script now declares
|
||
> the destination and copies contents.
|
||
|
||
## Model and drafter
|
||
|
||
| File | Bytes | Role |
|
||
|---|---:|---|
|
||
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
|
||
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
|
||
|
||
Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
|
||
a truncated 31 GB GGUF is otherwise a load-time crash).
|
||
|
||
The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
|
||
the same size, on Unsloth's importance matrix) of an abliterated Swift
|
||
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
|
||
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
|
||
endpoint is frank-small.
|
||
|
||
**The file also carries the model's MTP head** (`blk.64.nextn.*`). With
|
||
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
|
||
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
|
||
because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining)
|
||
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
|
||
head in; it is not enabled by default because the DFlash2 path is the measured
|
||
one.
|
||
|
||
## Deployment
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
|
||
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
|
||
| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |
|
||
|
||
```bash
|
||
./scripts/deploy.sh dankserver --service frank-medium
|
||
```
|
||
|
||
The handler verifies the runtime and payload exist (and refuses otherwise),
|
||
retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
|
||
waits for **port 7770 to be released** (the retired proxy held it on the host
|
||
network), relinks and restarts the unit, and gates on
|
||
`http://127.0.0.1:7770/health`.
|
||
|
||
Deployed flags:
|
||
|
||
| Flag | Value | Why |
|
||
|---|---|---|
|
||
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
|
||
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
|
||
| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context |
|
||
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
|
||
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
|
||
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
|
||
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
|
||
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
|
||
| `--no-mmproj` | — | text only |
|
||
|
||
## Verified serving behaviour (2026-09-16)
|
||
|
||
- Load: `model loaded` ~29 s after start with a warm page cache, then
|
||
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
|
||
- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports
|
||
`modalities: {vision: false}`, one slot, and the sampler defaults above.
|
||
- DFlash2 engaged — the init line is the proof, not the flag:
|
||
|
||
```
|
||
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
|
||
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
|
||
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
|
||
```
|
||
|
||
`block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
|
||
what distinguishes it from DFlash 1.
|
||
- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on
|
||
GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention
|
||
layers' KV and the compute buffers all sit on the second card, which keeps
|
||
~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB.
|
||
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one
|
||
slot: the raised window is real, not just declared, and it is 22 % beyond the
|
||
114688 the vLLM predecessor could offer.
|
||
|
||
### Measured throughput (2026-09-16, deployed service, `temperature 0`)
|
||
|
||
Same filler corpus for every shape, so these are shapes rather than a
|
||
benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
|
||
with a long budget isolates decode. `draft`/`acc` are per request; acceptance
|
||
moves with the text (filler prose accepts far worse than code or a real agent
|
||
loop).
|
||
|
||
| Shape | Prefill | Decode | Acceptance |
|
||
|---|---|---|---|
|
||
| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
|
||
| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
|
||
| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
|
||
| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
|
||
| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
|
||
| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |
|
||
|
||
The last row is the prefix cache doing its job: the second request of the same
|
||
conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
|
||
4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
|
||
vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
|
||
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
|
||
no proxy.
|
||
|
||
## Observability
|
||
|
||
The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
|
||
(`--metrics`), which is what the fleet's scrape and dashboard read — the
|
||
`vllm:` series the predecessor exported no longer exist:
|
||
|
||
| Need | Series |
|
||
|---|---|
|
||
| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
|
||
| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
|
||
| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
|
||
| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |
|
||
|
||
`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
|
||
drafter actually running": the flags can all be right while the draft model
|
||
failed to load.
|
||
|
||
## Client contract
|
||
|
||
OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
|
||
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
|
||
are projected into `reasoning_content`. A request that sends no sampler fields
|
||
gets the server defaults above; a client that sends its own wins. Text only.
|
||
|
||
## Troubleshooting
|
||
|
||
- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
|
||
bundled device-library depth (see the bundling trap above); check
|
||
`build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
|
||
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
|
||
`0,1`; the iGPU enumerates as device 2 and must stay hidden.
|
||
- **Drafting not engaged** — read the journal for
|
||
`common_speculative_impl_draft_dflash`; a missing drafter file makes the
|
||
serve script exit before llama-server starts, and a non-DFlash2 drafter (or
|
||
an older llama.cpp) fails the load instead.
|
||
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
|
||
correct for `draft-dflash`. It is not a truncated download:
|
||
`bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
|
||
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
|
||
gfx1200 sibling binary links, loads the model and then segfaults on these
|
||
cards.
|
||
|
||
## See Also
|
||
|
||
- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
|
||
(replicas + proxy) and the PCIe-atomics evidence
|
||
- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
|
||
2026-09-15, whose `--split-mode layer` measurements this service reuses
|
||
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
|
||
collective, and every software escape that is closed
|
||
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
|
||
3090, MTP, the image path)
|
||
- [[Dankserver System Specs]] — hardware inventory
|