frank-medium: 36:29 split, 327680 context, vision enabled

The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV,
and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point,
computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82
GiB free, both above the 1.5 GiB floor.

36:29 is one layer over the 35:30 balance on purpose: at this context the
balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0
to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so
the seam's slack is that gap rounded up (3225 MiB) with the trade written down,
and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries
no scaling; positions past the trained 262144 extrapolate on the base rope).

Vision: NO_MMPROJ=0, the serve script now refuses to start without the
projector when vision is on (a missing file is a deploy error, not a silent
text-only server), the download script fetches and byte-verifies it, and the
deploy handler's preflight checks for it. The projector is already on the host,
byte- and sha256-verified against the Hub.
This commit is contained in:
Frank committed 2026-09-17 01:37:55 -07:00
1 parent 24cdbb0e45
commit d95f55b822
5 files changed
+83 -27

No files matched your search

+8 -2
View File
@@ -1,6 +1,5 @@
#!/usr/bin/env bash
# Download the frank-medium payload for Dankserver (2x Radeon AI PRO R9700):
# the target GGUF and the DFlash2 draft model, into /home/dank/models.
# the target GGUF, the BF16 vision projector and the DFlash2 draft model.
#
# bash scripts/download-frank-medium-gguf.sh
# bash scripts/download-frank-medium-gguf.sh --verify-only
@@ -27,6 +26,10 @@ MODEL_REPO="ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
MODEL_DIR="/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
MODEL_FILE="Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
MODEL_SIZE=31457990112
# The BF16 vision projector from the same repo; llama-server autoloads any
# mmproj*.gguf beside the target, so it must land in MODEL_DIR.
MM_PROJ_FILE="mmproj-BF16.gguf"
MM_PROJ_SIZE=931145888
DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF"
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF"
@@ -55,6 +58,8 @@ dl() { # dl <repo> <dir> <file>
if [[ "$mode" == "download" ]]; then
echo "target: $MODEL_REPO"
dl "$MODEL_REPO" "$MODEL_DIR" "$MODEL_FILE"
echo "projector: $MODEL_REPO"
dl "$MODEL_REPO" "$MODEL_DIR" "$MM_PROJ_FILE"
echo "draft: $DRAFT_REPO"
dl "$DRAFT_REPO" "$DRAFT_DIR" "$DRAFT_FILE"
fi
@@ -72,6 +77,7 @@ check() { # check <path> <expected>
fi
}
check "$MODEL_DIR/$MODEL_FILE" "$MODEL_SIZE"
check "$MODEL_DIR/$MM_PROJ_FILE" "$MM_PROJ_SIZE"
check "$DRAFT_DIR/$DRAFT_FILE" "$DRAFT_SIZE"
[[ -z "$token" ]] && echo "note: no token used; the byte counts above are the integrity check"
[[ "$ok" -eq 1 ]] || exit 1