Frank
ff52543d8a
The loader prints no per-device ledger at this verbosity, so the claim is measured rather than read: +131072 tokens of context added 4.83 GB to one card and 4.84 GB to the other (both carry their own attention layers' KV), and neither card's total usage could hold the 31.46 GB file. Worth writing down because the retired shape WAS two full copies — that is what a replica is, and it is why the 29.8 GB FP8 checkpoint could never run there.
285 lines
14 KiB
Markdown
285 lines
14 KiB
Markdown
---
|
||
type: configuration
|
||
title: frank-medium llama.cpp + DFlash2 Runbook
|
||
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.
|
||
tags:
|
||
- frank-medium
|
||
- dankserver
|
||
- llama-cpp
|
||
- dflash2
|
||
- qwen35
|
||
- gfx1201
|
||
- r9700
|
||
- layer-split
|
||
timestamp: 2026-09-16
|
||
---
|
||
|
||
# frank-medium — llama.cpp + DFlash2
|
||
|
||
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
|
||
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
|
||
with the **z-lab DFlash2** block-diffusion drafter, as **one process across
|
||
both Radeon AI PRO R9700 cards**, layer-split.
|
||
|
||
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
|
||
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
|
||
the same day the pair went live. It is also the box's **only** LLM runtime: the
|
||
replicas, the proxy and their unit files are retired.
|
||
|
||
## Why one process and layer split
|
||
|
||
The endpoint's whole history is one constraint: **the two cards cannot take
|
||
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
|
||
and ROCr refuses to supply it to a device without PCIe atomics; the second
|
||
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
|
||
every collective that touches it dies with `hipErrorIllegalState`. That rules
|
||
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
|
||
whose stage handoff is a `send`/`recv` of the hidden state on the device
|
||
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
|
||
The full matrix is in [[Dual-R9700-Cooperation]].
|
||
|
||
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
|
||
distributed across the cards in order and the boundary activation is a plain
|
||
per-backend copy, once per forward pass. Measured on this host with the
|
||
predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):
|
||
|
||
| Split | pp4096 | tg128 |
|
||
|---|---:|---:|
|
||
| one R9700 (`-sm none`) | 1160.9 | 19.42 |
|
||
| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
|
||
| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |
|
||
|
||
Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
|
||
prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
|
||
shape — prefill nearly doubles and one model serves from both cards.
|
||
|
||
## Hardware
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
|
||
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
|
||
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
|
||
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
|
||
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
|
||
|
||
## Runtime
|
||
|
||
Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
|
||
`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
|
||
toolchain (TheRock nightly) that is **bundled next to the binary** —
|
||
`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
|
||
packages.
|
||
|
||
Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
|
||
earlier the same day (its tip, `c49ebdb`, predates both):
|
||
|
||
- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
|
||
convolution plus the candidate selector, detected from the draft GGUF
|
||
(`selector_top_k > 0`) and served through the ordinary
|
||
`--spec-type draft-dflash` path.
|
||
- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
|
||
single trained head.
|
||
|
||
Build with:
|
||
|
||
```bash
|
||
bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc)
|
||
```
|
||
|
||
> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
|
||
> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
|
||
> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
|
||
> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
|
||
> failure is invisible at startup: the engine loads 29 GiB of weights, answers
|
||
> `/health`, and dies on the **first request** with
|
||
> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
|
||
> such file or directory for GPU arch : gfx1201`. The build script now declares
|
||
> the destination and copies contents.
|
||
|
||
## Model and drafter
|
||
|
||
| File | Bytes | Role |
|
||
|---|---:|---|
|
||
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
|
||
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
|
||
|
||
Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
|
||
a truncated 31 GB GGUF is otherwise a load-time crash).
|
||
|
||
The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
|
||
the same size, on Unsloth's importance matrix) of an abliterated Swift
|
||
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
|
||
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
|
||
endpoint is frank-small.
|
||
|
||
**The file also carries the model's MTP head** (`blk.64.nextn.*`). With
|
||
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
|
||
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
|
||
because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining)
|
||
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
|
||
head in; it is not enabled by default because the DFlash2 path is the measured
|
||
one.
|
||
|
||
## Deployment
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
|
||
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
|
||
| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |
|
||
|
||
```bash
|
||
./scripts/deploy.sh dankserver --service frank-medium
|
||
```
|
||
|
||
The handler verifies the runtime and payload exist (and refuses otherwise),
|
||
retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
|
||
waits for **port 7770 to be released** (the retired proxy held it on the host
|
||
network), relinks and restarts the unit, and gates on
|
||
`http://127.0.0.1:7770/health`.
|
||
|
||
Deployed flags:
|
||
|
||
| Flag | Value | Why |
|
||
|---|---|---|
|
||
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
|
||
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
|
||
| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context |
|
||
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
|
||
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
|
||
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
|
||
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
|
||
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
|
||
| `--no-mmproj` | — | text only |
|
||
|
||
## Verified serving behaviour (2026-09-16)
|
||
|
||
- Load: `model loaded` ~29 s after start with a warm page cache, then
|
||
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
|
||
- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports
|
||
`modalities: {vision: false}`, one slot, and the sampler defaults above.
|
||
- DFlash2 engaged — the init line is the proof, not the flag:
|
||
|
||
```
|
||
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
|
||
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
|
||
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
|
||
```
|
||
|
||
`block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
|
||
what distinguishes it from DFlash 1.
|
||
- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on
|
||
GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention
|
||
layers' KV and the compute buffers all sit on the second card, which keeps
|
||
~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB.
|
||
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one
|
||
slot: the raised window is real, not just declared, and it is 22 % beyond the
|
||
114688 the vLLM predecessor could offer.
|
||
|
||
- **Nothing is replicated.** The loader does not print a per-device ledger at
|
||
this verbosity, so this is measured instead of read: raising CTX_SIZE from
|
||
131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards
|
||
carry the KV for their own attention layers, about half each — and neither
|
||
card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB
|
||
file, so no card carries a full copy of the weights. Compare the shape this
|
||
replaced: two vLLM replicas, i.e. two complete copies of the model, one per
|
||
card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there
|
||
at any usable context.
|
||
### Measured throughput (2026-09-16, deployed service, `temperature 0`)
|
||
|
||
Same filler corpus for every shape, so these are shapes rather than a
|
||
benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
|
||
with a long budget isolates decode. `draft`/`acc` are per request; acceptance
|
||
moves with the text (filler prose accepts far worse than code or a real agent
|
||
loop).
|
||
|
||
| Shape | Prefill | Decode | Acceptance |
|
||
|---|---|---|---|
|
||
| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
|
||
| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
|
||
| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
|
||
| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
|
||
| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
|
||
| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |
|
||
|
||
The last row is the prefix cache doing its job: the second request of the same
|
||
conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
|
||
4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
|
||
vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
|
||
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
|
||
no proxy.
|
||
|
||
### The agent-loop shape (fleet harness, 2 families × 2 turns)
|
||
|
||
`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
|
||
agent loops re-sending a ~30k-token document concurrently:
|
||
|
||
| | turn 1 | turn 2 |
|
||
|---|---:|---:|
|
||
| family 1 | 24,799 ms | 30,697 ms |
|
||
| family 0 | 51,424 ms | 8,015 ms |
|
||
|
||
Read that against the prefill table, not as a latency regression: one slot
|
||
means the second family's first turn waits for the first (25 s of cold 30k
|
||
prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that
|
||
the cache could only partly serve. **The single slot is now the limiting
|
||
resource for concurrent agent loops**, which is the trade this deployment makes
|
||
for giving one conversation the whole 262144-token window instead of splitting
|
||
two. Raising `--parallel` splits the window (and the KV) between slots; it does
|
||
not add throughput.
|
||
|
||
## Observability
|
||
|
||
The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
|
||
(`--metrics`), which is what the fleet's scrape and dashboard read — the
|
||
`vllm:` series the predecessor exported no longer exist:
|
||
|
||
| Need | Series |
|
||
|---|---|
|
||
| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
|
||
| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
|
||
| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
|
||
| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |
|
||
|
||
`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
|
||
drafter actually running": the flags can all be right while the draft model
|
||
failed to load.
|
||
|
||
## Client contract
|
||
|
||
OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
|
||
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
|
||
are projected into `reasoning_content`. A request that sends no sampler fields
|
||
gets the server defaults above; a client that sends its own wins. Text only.
|
||
|
||
## Troubleshooting
|
||
|
||
- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
|
||
bundled device-library depth (see the bundling trap above); check
|
||
`build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
|
||
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
|
||
`0,1`; the iGPU enumerates as device 2 and must stay hidden.
|
||
- **Drafting not engaged** — read the journal for
|
||
`common_speculative_impl_draft_dflash`; a missing drafter file makes the
|
||
serve script exit before llama-server starts, and a non-DFlash2 drafter (or
|
||
an older llama.cpp) fails the load instead.
|
||
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
|
||
correct for `draft-dflash`. It is not a truncated download:
|
||
`bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
|
||
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
|
||
gfx1200 sibling binary links, loads the model and then segfaults on these
|
||
cards.
|
||
|
||
## See Also
|
||
|
||
- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
|
||
(replicas + proxy) and the PCIe-atomics evidence
|
||
- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
|
||
2026-09-15, whose `--split-mode layer` measurements this service reuses
|
||
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
|
||
collective, and every software escape that is closed
|
||
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
|
||
3090, MTP, the image path)
|
||
- [[Dankserver System Specs]] — hardware inventory
|