dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c9c42e7489b57b1138447356e52f1e04edf4e580
serverAppConfigs/scripts/serve-frank-medium.sh
T
Frank 05c0d3a00a frank-medium: llama.cpp layer split across both R9700s, DFlash2 drafting 
The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.

llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.

Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.

Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
2026-09-16 14:59:23 -07:00

182 lines
6.3 KiB
Bash
Executable File
Raw Blame History

#!/usr/bin/env bash
# Serve ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF @ UD-Q8_K_XL as
# frank-medium on Dankserver's two Radeon AI PRO R9700 (gfx1201), with the
# z-lab DFlash2 block-diffusion drafter, using mainline llama.cpp.
#
# Runtime: llama.cpp master fb27a525 (2026-09-16), a gfx1201 HIP build that
# bundles its own ROCm 7.14 runtime (RPATH $ORIGIN/../lib), produced by
# scripts/build-llama-rdna4.sh. Two features pin that ref: DFlash2
# (ggml-org/llama.cpp#27342) and qwen35 MTP. The charlie12345/ROCmFPX fork
# this deployment replaced has neither (its tip is c49ebdb, 2026-08-22).
#
# Parallelism: --split-mode layer across both cards. Tensor split would need a
# cross-card allreduce and this host cannot dispatch one -- RCCL's kernels
# require a hostcall buffer that ROCr refuses on a card without PCIe atomics,
# and the second R9700 sits behind the chipset (AtomicOpsCap: Routing-). Layer
# split crosses the wire once per forward pass instead of once per layer.
#
# The vLLM replica pair and its routing proxy were retired with this cutover:
# one process, one model, both cards.
#
# Tunables come from frank-medium.env unless FRANK_MEDIUM_ENV_FILE overrides it.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_MEDIUM_ENV_FILE:-$SCRIPT_DIR/frank-medium.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) ------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}"
: "${MODEL_DIR:=/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF}"
: "${MODEL_FILE:=Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
: "${DRAFT_MODEL_DIR:=/home/dank/models/Qwen3.8-27B-DFlash2-GGUF}"
: "${DRAFT_MODEL_FILE:=Qwen3.8-27B-DFlash2-Q8_0.gguf}"
: "${ALIAS:=frank-medium}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
# Devices. HIP_VISIBLE_DEVICES hides the Raphael iGPU (HIP device 2) so nothing
# can drift onto shared system RAM; DEVICES is what llama.cpp splits across.
: "${HIP_VISIBLE_DEVICES:=0,1}"
: "${DEVICES:=ROCm0,ROCm1}"
: "${SPLIT_MODE:=layer}"
: "${TENSOR_SPLIT:=1,1}"
: "${NGL:=999}"
: "${FIT:=off}"
: "${FLASH_ATTN:=on}"
: "${CTX_SIZE:=131072}"
: "${PARALLEL:=1}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
: "${NUM_THREADS:=8}"
: "${THREAD_BATCH:=8}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CTX_CHECKPOINTS:=8}"
: "${CACHE_RAM:=8192}"
# Sampling defaults for silent clients: the values the vLLM predecessor served.
: "${TEMPERATURE:=0.7}"
: "${TOP_P:=0.95}"
: "${TOP_K:=20}"
: "${MIN_P:=0.0}"
: "${PRESENCE_PENALTY:=0.0}"
: "${REASONING:=on}"
: "${REASONING_FORMAT:=deepseek}"
: "${REASONING_BUDGET:=-1}"
: "${CHAT_TEMPLATE_FILE:=}"
: "${CHAT_TEMPLATE_KWARGS:={\"enable_thinking\":true,\"preserve_thinking\":true}}"
# Speculative decoding: draft-dflash (the DFlash2 drafter is a file, not a
# flag) or draft-mtp (the target GGUF's own MTP head -- no draft file).
: "${SPEC_TYPE:=draft-dflash}"
: "${SPEC_DRAFT_N_MAX:=7}"
: "${SPEC_DRAFT_N_MIN:=0}"
: "${SPEC_DRAFT_P_MIN:=0.0}"
: "${SPEC_DRAFT_P_SPLIT:=0.10}"
: "${SPEC_DRAFT_NGL:=999}"
: "${SPEC_DRAFT_DEVICE:=}"
: "${SPEC_DRAFT_TYPE_K:=f16}"
: "${SPEC_DRAFT_TYPE_V:=f16}"
: "${SPEC_DRAFT_POLL:=1}"
: "${SPEC_DRAFT_POLL_BATCH:=1}"
NO_MMPROJ="${NO_MMPROJ:-1}"
export HIP_VISIBLE_DEVICES
MODEL="$MODEL_DIR/$MODEL_FILE"
DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE"
if [[ ! -x "$LLAMA_BIN" ]]; then
echo "missing executable runtime: $LLAMA_BIN" >&2
echo "build it with: bash $SCRIPT_DIR/build-llama-rdna4.sh" >&2
exit 1
fi
if [[ ! -r "$MODEL" ]]; then
echo "missing or unreadable target model: $MODEL" >&2
echo "fetch it with: bash $SCRIPT_DIR/download-frank-medium-gguf.sh" >&2
exit 1
fi
if [[ -n "$CHAT_TEMPLATE_FILE" && ! -r "$CHAT_TEMPLATE_FILE" ]]; then
echo "missing or unreadable chat template: $CHAT_TEMPLATE_FILE" >&2
exit 1
fi
# --- speculative decoding -------------------------------------------------
SPEC_ARGS=()
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
SPEC_ARGS=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
if [[ "$SPEC_TYPE" == *dflash* ]]; then
if [[ ! -r "$DRAFT_MODEL" ]]; then
echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
exit 1
fi
SPEC_ARGS+=(
--spec-draft-model "$DRAFT_MODEL"
--spec-draft-ngl "$SPEC_DRAFT_NGL"
--spec-draft-type-k "$SPEC_DRAFT_TYPE_K"
--spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
)
fi
if [[ "$SPEC_TYPE" == *dspark* || ( "$SPEC_DRAFT_P_MIN" != "0.0" && "$SPEC_DRAFT_P_MIN" != "0" ) ]]; then
SPEC_ARGS+=(--spec-draft-n-min "$SPEC_DRAFT_N_MIN" --spec-draft-p-min "$SPEC_DRAFT_P_MIN")
fi
if [[ -n "$SPEC_DRAFT_DEVICE" ]]; then
SPEC_ARGS+=(--spec-draft-device "$SPEC_DRAFT_DEVICE")
fi
fi
# --- template + placement -------------------------------------------------
TEMPLATE_ARGS=()
if [[ -n "$CHAT_TEMPLATE_FILE" ]]; then
TEMPLATE_ARGS+=(--chat-template-file "$CHAT_TEMPLATE_FILE")
fi
TEMPLATE_ARGS+=(--chat-template-kwargs "$CHAT_TEMPLATE_KWARGS")
PLACEMENT_ARGS=(--device "$DEVICES" --split-mode "$SPLIT_MODE" --tensor-split "$TENSOR_SPLIT")
MMPROJ_ARGS=()
[[ "$NO_MMPROJ" == "1" ]] && MMPROJ_ARGS=(--no-mmproj)
exec "$LLAMA_BIN" \
--model "$MODEL" \
--alias "$ALIAS" \
--host "$BIND_ADDR" \
--port "$LISTEN_PORT" \
--jinja \
--gpu-layers "$NGL" \
--fit "$FIT" \
--parallel "$PARALLEL" \
--ctx-size "$CTX_SIZE" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--flash-attn "$FLASH_ATTN" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--threads "$NUM_THREADS" \
--threads-batch "$THREAD_BATCH" \
--ctx-checkpoints "$CTX_CHECKPOINTS" \
--cache-ram "$CACHE_RAM" \
--cache-prompt \
--metrics \
--temp "$TEMPERATURE" \
--top-p "$TOP_P" \
--top-k "$TOP_K" \
--min-p "$MIN_P" \
--presence-penalty "$PRESENCE_PENALTY" \
--reasoning "$REASONING" \
--reasoning-format "$REASONING_FORMAT" \
--reasoning-budget "$REASONING_BUDGET" \
"${TEMPLATE_ARGS[@]}" \
"${PLACEMENT_ARGS[@]}" \
"${MMPROJ_ARGS[@]}" \
"${SPEC_ARGS[@]}"
Reference in New Issue View Git Blame Copy Permalink