frank-medium: swap ROCmFPX llama.cpp runtime for vllm-radiance MXFP4
Rootless Quadlet on dankserver serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's designated alternate target; ships its fp8 MTP head, no rewrite step) with the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network, :7770, alias frank-medium unchanged). - dankserver/frank-medium/: quadlet unit, container entrypoint (fork of serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck - dispatcher retires the root llama.cpp unit, re-tags the registry image to the localhost alias, installs the quadlet, waits on /health - planner maps dankserver/frank-medium/* to frank-medium; scripts-era mappings removed - ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
This commit is contained in:
Frank committed
1 parent
db29c4d227
commit
6b647c1558
13 files changed
+566
-47
No files matched your search
@@ -0,0 +1,138 @@
|
||||
# Tunables for frank-medium (kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF,
|
||||
# charlie12345/ROCmFPX llama.cpp, Dankserver, 2x AMD Radeon AI PRO R9700).
|
||||
#
|
||||
# Hardware (2026-09-15): the RTX 3090 left Dankserver and two Radeon AI PRO
|
||||
# R9700 cards (Navi 48, gfx1201, RDNA4, 32,624 MiB each, PCIe 5.0 x16) took
|
||||
# its place. The Raphael iGPU (gfx1036) is still enumerated by HIP as the
|
||||
# third device. There is no XGMI link between the two cards and they sit in
|
||||
# different IOMMU groups, so any cross-GPU traffic goes over PCIe.
|
||||
#
|
||||
# Runtime: charlie12345/ROCmFPX (llama.cpp fork, ROCmFPX quant family; the
|
||||
# Q8_0_ROCMFPX_AGENT tensor type exists nowhere else - stock llama.cpp cannot
|
||||
# load this GGUF). Built for gfx1201 with the fork's self-contained ROCm
|
||||
# 7.14 toolchain so the host's stale /opt/rocm-6.3.3 install is never used:
|
||||
#
|
||||
# cd /home/dank/GitPackages/charlie-ROCmFPX
|
||||
# env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
|
||||
#
|
||||
# gfx1201 must be the build target: a gfx1200 binary links, loads the model
|
||||
# and then segfaults on this card (fork's docs/BUILD-AMD-ARCHITECTURES.md).
|
||||
#
|
||||
# Model: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT - Q8_0_ROCMFPX_AGENT
|
||||
# (ftype 115), 26.28 GiB, 8.39 bpw, arch qwen35 (MoE), quantized from a
|
||||
# 51.3 GiB BF16 source. The AGENT routing keeps 12 more attention-output
|
||||
# projections at full Q8_0 than the plain 8-bit build, which buys a higher
|
||||
# MTP draft acceptance (0.953 vs 0.911) and that is the whole point of this
|
||||
# checkpoint. Model card documents 131,072 context.
|
||||
#
|
||||
# Parallelism (--split-mode layer, operator call 2026-09-15 "pipeline
|
||||
# parallelism"): layers are distributed across the two cards in order and
|
||||
# activations hop PCIe at the boundary. Measured with llama-bench
|
||||
# (pp4096 / tg128 / same at depth 32768, f16 KV, fa on, 3 reps):
|
||||
#
|
||||
# single R9700 1160.9 / 19.42 (fits: 26.27 GiB weights, ~5 GiB left)
|
||||
# layer 2x 1991.2 / 18.76 859.1 / 17.64
|
||||
# tensor 2x 271.8 / 26.26 228.6 / 25.09
|
||||
#
|
||||
# Tensor parallel wins decode (+40 %) but collapses prefill 7x: this build
|
||||
# has no RCCL (llama.cpp logs "AllReduce init failed; falling back to
|
||||
# meta-backend butterfly"), so every layer pays a slow cross-card reduction.
|
||||
# Agentic traffic is prefill-heavy, so layer split wins the workload. Revisit
|
||||
# if the fork ships a working RCCL tensor path.
|
||||
#
|
||||
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
||||
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
||||
# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
|
||||
# env vars.
|
||||
|
||||
# charlie12345/ROCmFPX llama-server (self-contained ROCm 7.14 build, RPATH
|
||||
# points at the bundled runtime libs; no LD_LIBRARY_PATH needed).
|
||||
LLAMA_BIN=/home/dank/GitPackages/charlie-ROCmFPX/build-rdna4-rocm714/bin/llama-server
|
||||
|
||||
# Model package, fetched with scripts/download-q8agent.sh:
|
||||
# Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 26.28 GiB main weights
|
||||
# mtp-Qwen3.8-27B-Q4_0.gguf 1.68 GiB MTP draft head
|
||||
# mmproj-Qwen3.8-27B-BF16.gguf 0.93 GiB vision tower (unused)
|
||||
MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
|
||||
MODEL_FILE=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
||||
DRAFT_MODEL_FILE=mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
|
||||
# Alias + network. The port is the fleet's dankserver endpoint; the alias is
|
||||
# what clients ask for, and it replaces the retired `frank-small`.
|
||||
ALIAS=frank-medium
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
# Device placement. HIP_VISIBLE_DEVICES hides the iGPU (device 2) so nothing
|
||||
# can drift onto shared system RAM; DEVICES is what llama.cpp splits across.
|
||||
HIP_VISIBLE_DEVICES=0,1
|
||||
DEVICES=ROCm0,ROCm1
|
||||
SPLIT_MODE=layer
|
||||
|
||||
# TENSOR_SPLIT empty = layers distributed by free VRAM (equal here).
|
||||
TENSOR_SPLIT=
|
||||
|
||||
NGL=999
|
||||
# Auto-fit has no business resizing a context we set explicitly.
|
||||
FIT=off
|
||||
FLASH_ATTN=on
|
||||
|
||||
# 131,072 tokens, one slot (the model card's documented window; the GGUF
|
||||
# reports n_ctx_train 262144, untested by the model author). Layer split
|
||||
# keeps each layer's KV on the card that owns the layer, so the window is
|
||||
# bounded by per-card free memory: ~5.6 GiB of KV per card at 131072 f16.
|
||||
CTX_SIZE=131072
|
||||
PARALLEL=1
|
||||
|
||||
# 26.28 GiB of weights + MTP draft + KV cache leaves ~15 GiB per card at this
|
||||
# context, so the batch shape is chosen for prefill throughput, not memory.
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=512
|
||||
NUM_THREADS=8
|
||||
THREAD_BATCH=8
|
||||
|
||||
# f16 K/V: an agent checkpoint is the wrong place to spend KV precision.
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
CTX_CHECKPOINTS=8
|
||||
CACHE_RAM=8192
|
||||
|
||||
# Sampling: the checkpoint's instruct table.
|
||||
TEMPERATURE=0.7
|
||||
TOP_P=0.8
|
||||
TOP_K=20
|
||||
MIN_P=0.0
|
||||
PRESENCE_PENALTY=1.5
|
||||
|
||||
# Reasoning on; DeepSeek format projects the trace into reasoning_content
|
||||
# instead of literal thinking tags in content.
|
||||
REASONING=on
|
||||
REASONING_FORMAT=deepseek
|
||||
REASONING_BUDGET=-1
|
||||
|
||||
# Chat template: empty = the template embedded in the GGUF. Point this at
|
||||
# /home/dank/serverAppConfigs/scripts/chat-template.jinja to force the repo's
|
||||
# Sharp agent template instead.
|
||||
CHAT_TEMPLATE_FILE=
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
||||
|
||||
# MTP speculative decoding. n_max 6 / p_min 0.6 is the fork's measured
|
||||
# sweet spot for a MTP head with high acceptance; p_min 0.0 drafts
|
||||
# low-probability tokens that get rejected and makes MTP a net loss.
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=6
|
||||
SPEC_DRAFT_N_MIN=0
|
||||
SPEC_DRAFT_P_MIN=0.6
|
||||
SPEC_DRAFT_P_SPLIT=0.10
|
||||
SPEC_DRAFT_NGL=999
|
||||
SPEC_DRAFT_TYPE_K=f16
|
||||
SPEC_DRAFT_TYPE_V=f16
|
||||
SPEC_DRAFT_POLL=1
|
||||
SPEC_DRAFT_POLL_BATCH=1
|
||||
# The draft head rides the card that holds the target model's LAST layers:
|
||||
# the MTP head consumes the final hidden state, so pinning it to ROCm1 keeps
|
||||
# the per-token hop off the wire. (Under the abandoned tensor split the
|
||||
# draft must also be pinned to exactly one device or the meta-device KV
|
||||
# allocation aborts: GGML_ASSERT(buffer) in ggml-backend.cpp, measured
|
||||
# 2026-09-15.)
|
||||
SPEC_DRAFT_DEVICE=ROCm1
|
||||
Reference in new issue
Block a user