dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-medium is llama.cpp with DFlash2 on both cards

Browse Source 
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.

The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
This commit is contained in:
Frank committed 2026-09-16 15:15:02 -07:00
1 parent c9c42e7489
commit ab51d61198
7 files changed
+266 -21
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+231
View File
@@ -0,0 +1,231 @@
---
type: configuration
title: frank-medium llama.cpp + DFlash2 Runbook
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.
tags:
- frank-medium
- dankserver
- llama-cpp
- dflash2
- qwen35
- gfx1201
- r9700
- layer-split
timestamp: 2026-09-16
---
# frank-medium — llama.cpp + DFlash2
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
with the **z-lab DFlash2** block-diffusion drafter, as **one process across
both Radeon AI PRO R9700 cards**, layer-split.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the
replicas, the proxy and their unit files are retired.
## Why one process and layer split
The endpoint's whole history is one constraint: **the two cards cannot take
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
and ROCr refuses to supply it to a device without PCIe atomics; the second
R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
every collective that touches it dies with `hipErrorIllegalState`. That rules
out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
whose stage handoff is a `send`/`recv` of the hidden state on the device
backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
The full matrix is in [[Dual-R9700-Cooperation]].
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
distributed across the cards in order and the boundary activation is a plain
per-backend copy, once per forward pass. Measured on this host with the
predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):
| Split | pp4096 | tg128 |
|---|---:|---:|
| one R9700 (`-sm none`) | 1160.9 | 19.42 |
| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |
Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
shape — prefill nearly doubles and one model serves from both cards.
## Hardware
| Property | Value |
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
## Runtime
Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
toolchain (TheRock nightly) that is **bundled next to the binary** —
`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
packages.
Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
earlier the same day (its tip, `c49ebdb`, predates both):
- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
convolution plus the candidate selector, detected from the draft GGUF
(`selector_top_k > 0`) and served through the ordinary
`--spec-type draft-dflash` path.
- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
single trained head.
Build with:
```bash
bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc)
```
> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
> failure is invisible at startup: the engine loads 29 GiB of weights, answers
> `/health`, and dies on the **first request** with
> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
> such file or directory for GPU arch : gfx1201`. The build script now declares
> the destination and copies contents.
## Model and drafter
| File | Bytes | Role |
|---|---:|---|
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
a truncated 31 GB GGUF is otherwise a load-time crash).
The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
the same size, on Unsloth's importance matrix) of an abliterated Swift
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.
**The file also carries the model's MTP head** (`blk.64.nextn.*`). With
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled by default because the DFlash2 path is the measured
one.
## Deployment
| File | Role |
|---|---|
| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |
```bash
./scripts/deploy.sh dankserver --service frank-medium
```
The handler verifies the runtime and payload exist (and refuses otherwise),
retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
waits for **port 7770 to be released** (the retired proxy held it on the host
network), relinks and restarts the unit, and gates on
`http://127.0.0.1:7770/health`.
Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `131072` / `1` | one slot; `--parallel` **divides** the context |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `--no-mmproj` | — | text only |
## Verified serving behaviour (2026-09-16)
- Load: `model loaded` ~29 s after start with a warm page cache, then
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports
`modalities: {vision: false}`, one slot, and the sampler defaults above.
- DFlash2 engaged — the init line is the proof, not the flag:
```
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
```
`block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
what distinguishes it from DFlash 1.
- Both cards hold the model: at 131072 context, `rocm-smi` reports
**17.9 GiB on GPU 0 and 22.0 GiB on GPU 1** (the drafter and the attention
layers' KV sit on the second).
- A real completion (60-token prompt, 43 output tokens incl. reasoning):
**28.3 tok/s**, `draft_n = 49`, `draft_n_accepted = 36` → **73 % acceptance**.
The reasoning trace arrives in `reasoning_content`.
## Observability
The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
(`--metrics`), which is what the fleet's scrape and dashboard read — the
`vllm:` series the predecessor exported no longer exist:
| Need | Series |
|---|---|
| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |
`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
drafter actually running": the flags can all be right while the draft model
failed to load.
## Client contract
OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
are projected into `reasoning_content`. A request that sends no sampler fields
gets the server defaults above; a client that sends its own wins. Text only.
## Troubleshooting
- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
bundled device-library depth (see the bundling trap above); check
`build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
`0,1`; the iGPU enumerates as device 2 and must stay hidden.
- **Drafting not engaged** — read the journal for
`common_speculative_impl_draft_dflash`; a missing drafter file makes the
serve script exit before llama-server starts, and a non-DFlash2 drafter (or
an older llama.cpp) fails the load instead.
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
correct for `draft-dflash`. It is not a truncated download:
`bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
gfx1200 sibling binary links, loads the model and then segfaults on these
cards.
## See Also
- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
(replicas + proxy) and the PCIe-atomics evidence
- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
2026-09-15, whose `--split-mode layer` measurements this service reuses
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
collective, and every software escape that is closed
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory