frank-medium: 36:29 split, 327680 context, vision enabled

The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV,
and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point,
computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82
GiB free, both above the 1.5 GiB floor.

36:29 is one layer over the 35:30 balance on purpose: at this context the
balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0
to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so
the seam's slack is that gap rounded up (3225 MiB) with the trade written down,
and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries
no scaling; positions past the trained 262144 extrapolate on the base rope).

Vision: NO_MMPROJ=0, the serve script now refuses to start without the
projector when vision is on (a missing file is a deploy error, not a silent
text-only server), the download script fetches and byte-verifies it, and the
deploy handler's preflight checks for it. The projector is already on the host,
byte- and sha256-verified against the Hub.
This commit is contained in:
Frank committed 2026-09-17 01:37:55 -07:00
1 parent 24cdbb0e45
commit d95f55b822
5 files changed
+83 -27

No files matched your search

+10 -1
View File
@@ -89,7 +89,8 @@ fi
: "${SPEC_NGRAM_N_MIN:=48}"
: "${SPEC_NGRAM_N_MAX:=64}"
NO_MMPROJ="${NO_MMPROJ:-1}"
NO_MMPROJ="${NO_MMPROJ:-0}"
: "${MM_PROJ_FILE:=mmproj-BF16.gguf}"
export HIP_VISIBLE_DEVICES
@@ -106,6 +107,14 @@ if [[ ! -r "$MODEL" ]]; then
echo "fetch it with: bash $SCRIPT_DIR/download-frank-medium-gguf.sh" >&2
exit 1
fi
if [[ "$NO_MMPROJ" != "1" ]]; then
MM_PROJ="$MODEL_DIR/$MM_PROJ_FILE"
if [[ ! -r "$MM_PROJ" ]]; then
echo "vision is enabled but the projector is missing: $MM_PROJ" >&2
echo "fetch it with: bash $SCRIPT_DIR/download-frank-medium-gguf.sh" >&2
exit 1
fi
fi
if [[ -n "$CHAT_TEMPLATE_FILE" && ! -r "$CHAT_TEMPLATE_FILE" ]]; then
echo "missing or unreadable chat template: $CHAT_TEMPLATE_FILE" >&2
exit 1