Frank
05c0d3a00a
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
147 lines
6.6 KiB
Bash
147 lines
6.6 KiB
Bash
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
|
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
|
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
|
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
|
# routing proxy are retired.
|
|
#
|
|
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
|
|
# IOMMU groups -- every cross-card byte goes over PCIe. The Raphael iGPU
|
|
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
|
|
# nothing can drift onto shared system RAM.
|
|
#
|
|
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
|
|
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
|
|
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
|
|
# on a device without PCIe atomics -- the second card sits behind the B650
|
|
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
|
|
# touches it dies with hipErrorIllegalState. Measured on this box 2026-09-15
|
|
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
|
|
#
|
|
# single R9700 1160.9 / 19.42
|
|
# layer 2x 1991.2 / 18.76 (859.1 / 17.64 at depth 32768)
|
|
# tensor 2x 271.8 / 26.26 (no RCCL: butterfly fallback, -7x prefill)
|
|
#
|
|
# Layer split distributes layers in order; only the activation at the boundary
|
|
# crosses the wire, once per forward pass. That is the "pipeline parallelism"
|
|
# this service wanted, and it needs no collective at all.
|
|
#
|
|
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
|
|
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
|
|
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
|
|
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
|
|
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
|
|
#
|
|
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
|
|
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
|
|
# own quant of the same size, on Unsloth's importance matrix). Abliteration is
|
|
# orcarouter's single refusal direction projected out of 131 tensors; the vision
|
|
# tower is untouched but unused here (text only, see NO_MMPROJ). The file also
|
|
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
|
# --spec-type draft-mtp is available without a second copy in VRAM.
|
|
#
|
|
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
|
# 5120, block-diffusion proposal with the candidate selector. Decoding stays
|
|
# lossless -- the target verifies every drafted token.
|
|
#
|
|
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
|
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
|
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
|
|
# 256 head_dim. A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
|
|
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
|
|
# state at any depth.
|
|
#
|
|
# SIZING (per card, 31.86 GiB): weights split ~14.7 GiB; the drafter rides one
|
|
# card; KV at CTX_SIZE 131,072 is 8.00 GiB total (~4 GiB per card, since the 16
|
|
# attention layers are split with everything else); the loader's compute buffers
|
|
# complete the ledger. Read the loader's own "KV self size" / "compute buffer
|
|
# size" lines rather than trusting this arithmetic.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
|
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
|
# as flags only.
|
|
|
|
# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
|
|
|
|
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
|
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
|
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
|
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
|
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
|
|
|
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
|
# at /v1/models and what every client asks for.
|
|
ALIAS=frank-medium
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
|
|
# Placement. HIP_VISIBLE_DEVICES hides the iGPU; DEVICES is llama.cpp's own
|
|
# list, which is the pair the split runs across.
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
DEVICES=ROCm0,ROCm1
|
|
SPLIT_MODE=layer
|
|
# Empty = distribute layers by free VRAM (equal here, so 1,1 by hand).
|
|
TENSOR_SPLIT=1,1
|
|
|
|
NGL=999
|
|
# Auto-fit has no business resizing a context we set explicitly.
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
|
|
# One slot at the model's documented-configured window. --parallel divides
|
|
# --ctx-size between slots; it does not multiply. The target model ships
|
|
# n_ctx_train 262144 -- raising CTX_SIZE is a measurement, not a guess (watch
|
|
# the per-card ledger), and this service was deployed at 131072 first.
|
|
CTX_SIZE=131072
|
|
PARALLEL=1
|
|
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
|
|
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
CTX_CHECKPOINTS=8
|
|
CACHE_RAM=8192
|
|
|
|
# Sampling defaults for requests that send none -- the values the vLLM
|
|
# predecessor served on this endpoint (`--override-generation-config`), so a
|
|
# client sees no change across the runtime swap. The model card's own
|
|
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
|
|
# models.yml, so this is the floor, not the contract.
|
|
TEMPERATURE=0.7
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
# Reasoning on, projected into reasoning_content (deepseek format) with no
|
|
# budget: the model decides how long to think.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|
|
|
|
# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
|
|
CHAT_TEMPLATE_FILE=
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
|
|
|
# Speculative decoding: the DFlash2 block-diffusion drafter. Its GGUF carries
|
|
# block_size 8, so at most 7 draft tokens per verification step (llama.cpp
|
|
# asserts n_max <= block_size - 1); 7 is also what the drafter was published
|
|
# with. The drafter rides ROCm1, the card that ends up holding the target's
|
|
# last layers -- the hidden states the proposal consumes.
|
|
SPEC_TYPE=draft-dflash
|
|
SPEC_DRAFT_N_MAX=7
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm1
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16
|
|
|
|
# Text only: the server must not autoload a vision tower. Vision stays on
|
|
# frank-small (Dankclaw, RTX 3090), which is the fleet's image endpoint.
|
|
NO_MMPROJ=1
|