serverAppConfigs/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
configuration	frank-medium llama.cpp + DFlash2 Runbook	The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.	frank-medium dankserver llama-cpp dflash2 qwen35 gfx1201 r9700 layer-split	2026-09-16

frank-medium — llama.cpp + DFlash2

The ACTIVE frank-medium deployment since 2026-09-16: mainline llama.cpp serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the z-lab DFlash2 block-diffusion drafter, as one process across both Radeon AI PRO R9700 cards, layer-split.

It replaced the vllm-radiance MXFP4 replica pair + routing proxy (frank-medium-radiance-mxfp4, the record and the PCIe-atomics evidence) in the same day the pair went live. It is also the box's only LLM runtime: the replicas, the proxy and their unit files are retired.

Why one process and layer split

The endpoint's whole history is one constraint: the two cards cannot take part in an RCCL collective. RCCL's kernels declare a hidden hostcall buffer, and ROCr refuses to supply it to a device without PCIe atomics; the second R9700 hangs off the B650 chipset (AtomicOpsCap: Routing-, read-only), so every collective that touches it dies with hipErrorIllegalState. That rules out tensor parallelism — and it rules out vLLM's pipeline parallelism too, whose stage handoff is a send/recv of the hidden state on the device backend (vllm/distributed/parallel_state.py), the very primitive that fails. The full matrix is in Dual-R9700-Cooperation.

llama.cpp's --split-mode layer needs no collective at all: layers are distributed across the cards in order and the boundary activation is a plain per-backend copy, once per forward pass. Measured on this host with the predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):

Split	pp4096	tg128
one R9700 (-sm none)	1160.9	19.42
two cards, layer (-sm layer)	1991.2	18.76
two cards, tensor (-sm tensor, no RCCL: butterfly fallback)	271.8	26.26

Tensor split buys ~40 % decode and loses 7× prefill. The workload here is prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the shape — prefill nearly doubles and one model serves from both cards.

Hardware

Property	Value
GPUs	2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4)
VRAM	34,208,743,424 B (31.86 GiB) each, as rocm-smi reports it
Link	PCIe, no XGMI; card #2 behind the chipset, no atomic routing
iGPU	Raphael (gfx1036) enumerates as HIP device 2 — hidden via HIP_VISIBLE_DEVICES=0,1
CPU / RAM	Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap

Runtime

Mainline llama.cpp at fb27a525 (2026-09-16), built for gfx1201 only by scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain (TheRock nightly) that is bundled next to the binary — RPATH=$ORIGIN/../lib, no LD_LIBRARY_PATH, nothing from the host's ROCm 7.1 packages.

Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used earlier the same day (its tip, c49ebdb, predates both):

• DFlash2 (ggml-org/llama.cpp#27342): grouped dynamic depthwise convolution plus the candidate selector, detected from the draft GGUF (selector_top_k > 0) and served through the ordinary --spec-type draft-dflash path.
• qwen35 MTP: the draft-mtp implementation covers this architecture's single trained head.

Build with:

bash scripts/build-llama-rdna4.sh     # ~25 min on this box, JOBS=$(nproc)

Bundling trap, measured. rocBLAS and hipBLASLt look for their device code at a fixed depth under the binary's lib dir (../lib/rocblas/library, ../lib/hipblaslt/library/<arch>). cp -r SRC DST/ with a missing DST creates DST as a copy of SRC and flattens it one level, and the failure is invisible at startup: the engine loads 29 GiB of weights, answers /health, and dies on the first request with rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1201. The build script now declares the destination and copies contents.

Model and drafter

File	Bytes	Role
Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf	31,457,990,112	target, 29.29 GiB
Qwen3.8-27B-DFlash2-Q8_0.gguf	2,056,414,816	DFlash2 drafter, 1.92 GiB

Fetched by scripts/download-frank-medium-gguf.sh (resumable, byte-verified — a truncated 31 GB GGUF is otherwise a load-time crash).

The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of the same size, on Unsloth's importance matrix) of an abliterated Swift fine-tune: orcarouter's single refusal direction projected out of 131 tensors, vision tower untouched. Text only here (--no-mmproj); the fleet's image endpoint is frank-small.

The file also carries the model's MTP head (blk.64.nextn.*). With --spec-type draft-dflash those tensors are deliberately not loaded — the log says model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring — because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining) --spec-type draft-mtp,draft-dflash is a one-line env change that pulls the head in; it is not enabled by default because the DFlash2 path is the measured one.

Deployment

File	Role
scripts/frank-medium.env	every tunable, with the sizing arithmetic in comments
scripts/serve-frank-medium.sh	env-driven llama-server launcher; no arguments
scripts/frank-medium.service	ROOT systemd unit, User=dank, WantedBy=multi-user.target

./scripts/deploy.sh dankserver --service frank-medium

The handler verifies the runtime and payload exist (and refuses otherwise), retires the three vLLM quadlets into ~/.AGENTTRASH/20260916-vllm-radiance-retire/, waits for port 7770 to be released (the retired proxy held it on the host network), relinks and restarts the unit, and gates on http://127.0.0.1:7770/health.

Deployed flags:

Flag	Value	Why
--split-mode / --device / --tensor-split	layer / ROCm0,ROCm1 / 1,1	both cards, one model, no collective
--gpu-layers / --fit	999 / off	everything on the cards; auto-fit must not resize a context we set
--ctx-size / --parallel	131072 / 1	one slot; --parallel divides the context
--flash-attn / cache types	on / f16 K and V	an agentic checkpoint is the wrong place to spend KV precision
--spec-type / --spec-draft-n-max	draft-dflash / 7	DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows
--spec-draft-device	ROCm1	the drafter rides the card that ends up holding the target's last layers
--temp / --top-p / --top-k	0.7 / 0.95 / 20	the values the vLLM predecessor served on this endpoint
--reasoning-format / --reasoning-budget	deepseek / -1	trace into reasoning_content, unlimited
--no-mmproj	—	text only

Verified serving behaviour (2026-09-16)

• Load: model loaded ~29 s after start with a warm page cache, then listening on http://0.0.0.0:7770. The drafter adds ~2.5 s.

• Slot shape: n_slots = 1, n_ctx_slot = 131072; /props reports modalities: {vision: false}, one slot, and the sampler defaults above.

• DFlash2 engaged — the init line is the proof, not the flag:

  common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
  common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
  common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
  

  block_size=8 and n_extract=5 are DFlash2's own metadata; the selector is what distinguishes it from DFlash 1.

• Both cards hold the model: at 131072 context, rocm-smi reports 17.9 GiB on GPU 0 and 22.0 GiB on GPU 1 (the drafter and the attention layers' KV sit on the second).

• A real completion (60-token prompt, 43 output tokens incl. reasoning): 28.3 tok/s, draft_n = 49, draft_n_accepted = 36 → 73 % acceptance. The reasoning trace arrives in reasoning_content.

Observability

The endpoint exports Prometheus metrics under the llamacpp: namespace (--metrics), which is what the fleet's scrape and dashboard read — the vllm: series the predecessor exported no longer exist:

Need	Series
Throughput	llamacpp:prompt_tokens_seconds, llamacpp:predicted_tokens_seconds
Queue	llamacpp:requests_processing, llamacpp:requests_deferred
Cache reuse	llamacpp:prompt_tokens_cached_total vs llamacpp:prompt_tokens_total
Speculative decoding is live	llamacpp:spec_decode_num_drafts_total and llamacpp:spec_decode_num_accepted_tokens_total (their ratio is the acceptance rate)

spec_decode_num_drafts_total staying at 0 is the one-line answer to "is the drafter actually running": the flags can all be right while the draft model failed to load.

Client contract

OpenAI-compatible /v1 on 7770, alias frank-medium, streaming and /health, /props, /metrics as llama.cpp provides them. Reasoning traces are projected into reasoning_content. A request that sends no sampler fields gets the server defaults above; a client that sends its own wins. Text only.

Troubleshooting

• Loads, then dies on the first request with a rocBLAS Tensile error — the bundled device-library depth (see the bundling trap above); check build-rdna4-rocm714/lib/rocblas/library/ exists and is populated.
• hipErrorNoDevice / a card missing — HIP_VISIBLE_DEVICES must be 0,1; the iGPU enumerates as device 2 and must stay hidden.
• Drafting not engaged — read the journal for common_speculative_impl_draft_dflash; a missing drafter file makes the serve script exit before llama-server starts, and a non-DFlash2 drafter (or an older llama.cpp) fails the load instead.
• Why is blk.64.* "unused"? That is the model's own MTP head and it is correct for draft-dflash. It is not a truncated download: bash scripts/download-frank-medium-gguf.sh --verify-only proves the bytes.
• A rebuild segfaults at load — the build is gfx1201-only on purpose; a gfx1200 sibling binary links, loads the model and then segfaults on these cards.

See Also

• frank-medium-radiance-mxfp4 — the retired vllm-radiance MXFP4 deployment (replicas + proxy) and the PCIe-atomics evidence
• frank-medium-rocmfpx — the llama.cpp/ROCmFPX deployment earlier on 2026-09-15, whose --split-mode layer measurements this service reuses
• Dual-R9700-Cooperation — why the two cards cannot take part in a collective, and every software escape that is closed
• frank-small-qwen38-gsq-rco — the fleet's other llama.cpp endpoint (RTX 3090, MTP, the image path)
• Dankserver System Specs — hardware inventory