Frank
dc76dc5dc4
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and 29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a 140,069-token prompt prefilled through the single slot in 134 s to prove the window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this runtime's. Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to 140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance, and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt. The signals move with the runtime, so the fleet's rules and dashboard are re-cut rather than renamed: vllm: series do not exist any more, and two of the old rules watched things llama.cpp does not export at all (KV-pool fill, p95 latency). Those are retired with the reason written down, and LlmSpecDecodeOff takes one of the slots -- a lost drafter is the failure no client can see. The observability seam now fails if any rule names a series the endpoint does not export, which is what would have caught the stale vllm: expressions. omp models.yml (outside the repo) is updated in the same pass: contextWindow 262144, and the full reasoning-effort map, because Qwen3.8's template answers 500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer 200. Subagents run at high.
155 lines
7.1 KiB
Bash
155 lines
7.1 KiB
Bash
# Tunables for frank-medium (ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP
|
|
# UD-Q8_K_XL + z-lab/Qwen3.8-27B-DFlash2-GGUF Q8_0 drafter), mainline llama.cpp
|
|
# on Dankserver, 2x AMD Radeon AI PRO R9700 (gfx1201). Since 2026-09-16 this
|
|
# is the SOLE LLM runtime on the box: the vllm-radiance replica pair and its
|
|
# routing proxy are retired.
|
|
#
|
|
# Hardware: two R9700 (Navi 48, RDNA4, 32,624 MiB each), no XGMI, different
|
|
# IOMMU groups -- every cross-card byte goes over PCIe. The Raphael iGPU
|
|
# (gfx1036/Radeon 610M) enumerates as HIP device 2 and is hidden below so
|
|
# nothing can drift onto shared system RAM.
|
|
#
|
|
# WHY LAYER SPLIT AND NOT TENSOR (operator call, 2026-09-16): tensor parallel
|
|
# needs a cross-card allreduce per layer, and this host cannot dispatch one.
|
|
# RCCL's kernels carry a hidden hostcall-buffer parameter, and ROCr refuses it
|
|
# on a device without PCIe atomics -- the second card sits behind the B650
|
|
# chipset bridges (AtomicOpsCap: Routing-, read-only), so every collective that
|
|
# touches it dies with hipErrorIllegalState. Measured on this box 2026-09-15
|
|
# (llama-bench, pp4096 / tg128, f16 KV, fa on):
|
|
#
|
|
# single R9700 1160.9 / 19.42
|
|
# layer 2x 1991.2 / 18.76 (859.1 / 17.64 at depth 32768)
|
|
# tensor 2x 271.8 / 26.26 (no RCCL: butterfly fallback, -7x prefill)
|
|
#
|
|
# Layer split distributes layers in order; only the activation at the boundary
|
|
# crosses the wire, once per forward pass. That is the "pipeline parallelism"
|
|
# this service wanted, and it needs no collective at all.
|
|
#
|
|
# Runtime: mainline llama.cpp master fb27a525 (2026-09-16), gfx1201, built by
|
|
# scripts/build-llama-rdna4.sh against a private ROCm 7.14.0a20260624 toolchain
|
|
# that is bundled next to the binary (RPATH $ORIGIN/../lib, no LD_LIBRARY_PATH).
|
|
# The ref is pinned by TWO features: DFlash2 (ggml-org/llama.cpp#27342 -- the
|
|
# drafter is detected from its GGUF) and qwen35 MTP. Do not float it.
|
|
#
|
|
# Model: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL,
|
|
# 29.29 GiB, arch qwen35, Unsloth Dynamic 3.0 (per-tensor types from Unsloth's
|
|
# own quant of the same size, on Unsloth's importance matrix). Abliteration is
|
|
# orcarouter's single refusal direction projected out of 131 tensors; the vision
|
|
# tower is untouched but unused here (text only, see NO_MMPROJ). The file also
|
|
# carries the model's MTP head (blk.64.nextn.*), so a file-level switch to
|
|
# --spec-type draft-mtp is available without a second copy in VRAM.
|
|
#
|
|
# Drafter: z-lab/Qwen3.8-27B-DFlash2-GGUF @ Q8_0, 1.92 GiB: 5 layers, hidden
|
|
# 5120, block-diffusion proposal with the candidate selector. Decoding stays
|
|
# lossless -- the target verifies every drafted token.
|
|
#
|
|
# Architecture (GGUF arch qwen35): 64 layers as 16 x (3 x (Gated DeltaNet ->
|
|
# FFN) + 1 x (Gated Attention -> FFN)) -- 48 linear-attention layers whose
|
|
# per-sequence state is CONSTANT-SIZE, 16 full-attention layers at 4 KV heads x
|
|
# 256 head_dim. A token therefore costs 2 x 4 x 256 x 2 B = 64 KiB of KV at f16
|
|
# in the attention layers only, plus ~0.16 GiB per sequence of f32 DeltaNet
|
|
# state at any depth.
|
|
#
|
|
# SIZING, measured (rocm-smi, 2026-09-16). Weights split across the two cards;
|
|
# the drafter rides the second. The loader's compute buffers are part of what
|
|
# is seen below -- they are allocated at load, not per request:
|
|
#
|
|
# CTX_SIZE GPU 0 GPU 1 headroom on the busier card
|
|
# 131072 19.89 GB 24.30 GB 9.9 GB / 31 %
|
|
# 262144 24.72 GB 29.14 GB 5.07 GB / 15 % <- deployed
|
|
#
|
|
# 262144 is the window the model ships (n_ctx_train) and it fits with room, so
|
|
# the fleet gets the whole thing; the arithmetic that used to cap this endpoint
|
|
# at 114688 was the vLLM predecessor's, not this runtime's. If a future change
|
|
# adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers
|
|
# rather than assuming, and prefer CTX_SIZE over --gpu-layers.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads
|
|
# those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp
|
|
# as flags only.
|
|
|
|
# llama-server: mainline fb27a525 built for gfx1201 by scripts/build-llama-rdna4.sh.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server
|
|
|
|
# Payload, fetched with scripts/download-frank-medium-gguf.sh.
|
|
MODEL_DIR=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
|
MODEL_FILE=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
|
DRAFT_MODEL_DIR=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF
|
|
DRAFT_MODEL_FILE=Qwen3.8-27B-DFlash2-Q8_0.gguf
|
|
|
|
# Fleet identity. 7770 is the fleet's dankserver endpoint; the alias is the id
|
|
# at /v1/models and what every client asks for.
|
|
ALIAS=frank-medium
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
# Device placement. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2)
|
|
# so nothing can drift onto shared system RAM; DEVICES is what llama.cpp splits
|
|
# across, and neither of the two cards has an XGMI link to the other.
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
DEVICES=ROCm0,ROCm1
|
|
SPLIT_MODE=layer
|
|
# Empty = distribute layers by free VRAM (equal here, so 1,1 by hand).
|
|
TENSOR_SPLIT=1,1
|
|
|
|
NGL=999
|
|
# Auto-fit has no business resizing a context we set explicitly.
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
|
|
# One slot at the model's full configured window (n_ctx_train 262144), which was
|
|
# measured to fit before it was deployed: see SIZING above. --parallel DIVIDES
|
|
# --ctx-size between slots, so one slot carries the whole window and a second
|
|
# concurrent request queues behind it.
|
|
CTX_SIZE=262144
|
|
PARALLEL=1
|
|
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
|
|
# Threads cover tokenization, sampling and HTTP only: every layer is on a GPU.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# f16 K/V: an agentic checkpoint is the wrong place to spend KV precision.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
CTX_CHECKPOINTS=8
|
|
CACHE_RAM=8192
|
|
|
|
# Sampling defaults for requests that send none -- the values the vLLM
|
|
# predecessor served on this endpoint (`--override-generation-config`), so a
|
|
# client sees no change across the runtime swap. The model card's own
|
|
# recommendation is temp 1.0 / top_p 0.95; the harness sets its own per model in
|
|
# models.yml, so this is the floor, not the contract.
|
|
TEMPERATURE=0.7
|
|
TOP_P=0.95
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=0.0
|
|
|
|
# Reasoning on, projected into reasoning_content (deepseek format) with no
|
|
# budget: the model decides how long to think.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|
|
|
|
# Chat template: empty = the one embedded in the GGUF (Qwen3.8's own).
|
|
CHAT_TEMPLATE_FILE=
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
|
|
|
# Speculative decoding: the DFlash2 block-diffusion drafter. Its GGUF carries
|
|
# block_size 8, so at most 7 draft tokens per verification step (llama.cpp
|
|
# asserts n_max <= block_size - 1); 7 is also what the drafter was published
|
|
# with. The drafter rides ROCm1, the card that ends up holding the target's
|
|
# last layers -- the hidden states the proposal consumes.
|
|
SPEC_TYPE=draft-dflash
|
|
SPEC_DRAFT_N_MAX=7
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_DEVICE=ROCm1
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16
|
|
|
|
# Text only: the server must not autoload a vision tower. Vision stays on
|
|
# frank-small (Dankclaw, RTX 3090), which is the fleet's image endpoint.
|
|
NO_MMPROJ=1
|