Frank
6085a99dd8
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all, so the `frank-small` service is retired and `frank-medium` takes the endpoint. New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3 never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF -- a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load. Placeholder config: --split-mode tensor across both cards, 131072 context, 1 slot, f16 KV, MTP draft head on ROCm0. deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped, disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units before frank-medium starts, because `systemctl disable` has been observed to delete the unit file itself on this host (2026-08-26 incident). The retired frank-small repo files (env, unit, serve script) move to scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the deployment test seams now name frank-medium.
133 lines
5.5 KiB
Bash
133 lines
5.5 KiB
Bash
# Tunables for frank-medium (kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF,
|
|
# charlie12345/ROCmFPX llama.cpp, Dankserver, 2x AMD Radeon AI PRO R9700).
|
|
#
|
|
# Hardware (2026-09-15): the RTX 3090 left Dankserver and two Radeon AI PRO
|
|
# R9700 cards (Navi 48, gfx1201, RDNA4, 32,624 MiB each, PCIe 5.0 x16) took
|
|
# its place. The Raphael iGPU (gfx1036) is still enumerated by HIP as the
|
|
# third device. There is no XGMI link between the two cards and they sit in
|
|
# different IOMMU groups, so any cross-GPU traffic goes over PCIe.
|
|
#
|
|
# Runtime: charlie12345/ROCmFPX (llama.cpp fork, ROCmFPX quant family; the
|
|
# Q8_0_ROCMFPX_AGENT tensor type exists nowhere else - stock llama.cpp cannot
|
|
# load this GGUF). Built for gfx1201 with the fork's self-contained ROCm
|
|
# 7.14 toolchain so the host's stale /opt/rocm-6.3.3 install is never used:
|
|
#
|
|
# cd /home/dank/GitPackages/charlie-ROCmFPX
|
|
# env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
|
|
#
|
|
# gfx1201 must be the build target: a gfx1200 binary links, loads the model
|
|
# and then segfaults on this card (fork's docs/BUILD-AMD-ARCHITECTURES.md).
|
|
#
|
|
# Model: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT - Q8_0_ROCMFPX_AGENT
|
|
# (ftype 115), 26.28 GiB, 8.39 bpw, arch qwen35 (MoE), quantized from a
|
|
# 51.3 GiB BF16 source. The AGENT routing keeps 12 more attention-output
|
|
# projections at full Q8_0 than the plain 8-bit build, which buys a higher
|
|
# MTP draft acceptance (0.953 vs 0.911) and that is the whole point of this
|
|
# checkpoint. Model card documents 131,072 context.
|
|
#
|
|
# Parallelism (--split-mode tensor): tensor parallel splits the weights AND
|
|
# the KV cache across both cards through the fused "meta device". It is the
|
|
# only split mode here that gives each card half the KV cache; layer split
|
|
# would give each card the KV of the layers it owns, with one PCIe handoff
|
|
# per token per boundary. TENSOR mode's own constraints, all met below:
|
|
# flash attention ON, non-quantized KV cache, and no auto-fit.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
|
|
# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
|
|
# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
|
|
# env vars.
|
|
|
|
# charlie12345/ROCmFPX llama-server (self-contained ROCm 7.14 build, RPATH
|
|
# points at the bundled runtime libs; no LD_LIBRARY_PATH needed).
|
|
LLAMA_BIN=/home/dank/GitPackages/charlie-ROCmFPX/build-rdna4-rocm714/bin/llama-server
|
|
|
|
# Model package, fetched with scripts/download-q8agent.sh:
|
|
# Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf 26.28 GiB main weights
|
|
# mtp-Qwen3.8-27B-Q4_0.gguf 1.68 GiB MTP draft head
|
|
# mmproj-Qwen3.8-27B-BF16.gguf 0.93 GiB vision tower (unused)
|
|
MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
|
|
MODEL_FILE=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
|
DRAFT_MODEL_FILE=mtp-Qwen3.8-27B-Q4_0.gguf
|
|
|
|
# Alias + network. The port is the fleet's dankserver endpoint; the alias is
|
|
# what clients ask for, and it replaces the retired `frank-small`.
|
|
ALIAS=frank-medium
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7770
|
|
|
|
# Device placement. HIP_VISIBLE_DEVICES hides the iGPU (device 2) so nothing
|
|
# can drift onto shared system RAM; DEVICES is what llama.cpp splits across.
|
|
HIP_VISIBLE_DEVICES=0,1
|
|
DEVICES=ROCm0,ROCm1
|
|
SPLIT_MODE=tensor
|
|
|
|
# TENSOR_SPLIT empty = an even split between the two cards.
|
|
TENSOR_SPLIT=
|
|
|
|
NGL=999
|
|
# Tensor mode ignores auto-fit, and auto-fit has no business resizing a
|
|
# context we set explicitly.
|
|
FIT=off
|
|
FLASH_ATTN=on
|
|
|
|
# 131,072 tokens, one slot (the model card's documented window). Tensor mode
|
|
# puts half the KV cache on each card, so the window is not limited by a
|
|
# single card's free memory.
|
|
CTX_SIZE=131072
|
|
PARALLEL=1
|
|
|
|
# 26.28 GiB of weights + MTP draft + KV cache leaves ~15 GiB per card at this
|
|
# context, so the batch shape is chosen for prefill throughput, not memory.
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# f16 K/V: tensor mode refuses a quantized KV cache, and an agent checkpoint
|
|
# is the wrong place to spend KV precision anyway.
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=f16
|
|
CTX_CHECKPOINTS=8
|
|
CACHE_RAM=8192
|
|
|
|
# Sampling: the checkpoint's instruct table.
|
|
TEMPERATURE=0.7
|
|
TOP_P=0.8
|
|
TOP_K=20
|
|
MIN_P=0.0
|
|
PRESENCE_PENALTY=1.5
|
|
|
|
# Reasoning on; DeepSeek format projects the trace into reasoning_content
|
|
# instead of literal thinking tags in content.
|
|
REASONING=on
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|
|
|
|
# Chat template: empty = the template embedded in the GGUF. Point this at
|
|
# /home/dank/serverAppConfigs/scripts/chat-template.jinja to force the repo's
|
|
# Sharp agent template instead.
|
|
CHAT_TEMPLATE_FILE=
|
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}'
|
|
|
|
# MTP speculative decoding. n_max 6 / p_min 0.6 is the fork's measured
|
|
# sweet spot for a MTP head with high acceptance; p_min 0.0 drafts
|
|
# low-probability tokens that get rejected and makes MTP a net loss.
|
|
SPEC_TYPE=draft-mtp
|
|
SPEC_DRAFT_N_MAX=6
|
|
SPEC_DRAFT_N_MIN=0
|
|
SPEC_DRAFT_P_MIN=0.6
|
|
SPEC_DRAFT_P_SPLIT=0.10
|
|
SPEC_DRAFT_NGL=999
|
|
SPEC_DRAFT_TYPE_K=f16
|
|
SPEC_DRAFT_TYPE_V=f16
|
|
SPEC_DRAFT_POLL=1
|
|
SPEC_DRAFT_POLL_BATCH=1
|
|
# The draft head must run on ONE device. With --split-mode tensor the draft
|
|
# otherwise joins the meta-device allocation and the KV-cache build aborts:
|
|
# ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
|
|
# <- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor
|
|
# (measured 2026-09-15 on the built runtime: identical flags are healthy with
|
|
# --spec-draft-device ROCm0 and abort with it empty). ROCm0 is the card that
|
|
# also holds the target model's main GPU.
|
|
SPEC_DRAFT_DEVICE=ROCm0
|