dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: 36:29 split, 327680 context, vision enabled

Browse Source 
The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV,
and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point,
computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82
GiB free, both above the 1.5 GiB floor.

36:29 is one layer over the 35:30 balance on purpose: at this context the
balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0
to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so
the seam's slack is that gap rounded up (3225 MiB) with the trade written down,
and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries
no scaling; positions past the trained 262144 extrapolate on the base rope).

Vision: NO_MMPROJ=0, the serve script now refuses to start without the
projector when vision is on (a missing file is a deploy error, not a silent
text-only server), the download script fetches and byte-verifies it, and the
deploy handler's preflight checks for it. The projector is already on the host,
byte- and sha256-verified against the Hub.
This commit is contained in:
Frank
2026-09-17 01:37:55 -07:00
parent 24cdbb0e45
commit d95f55b822
5 changed files with 83 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/deploy-remote.sh
+6
View File
@@ -1114,6 +1114,7 @@ deploy_llm_small() {
LLM_MEDIUM_BIN="${LLM_MEDIUM_BIN:-/home/dank/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714/bin/llama-server}"
LLM_MEDIUM_MODEL="${LLM_MEDIUM_MODEL:-/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
LLM_MEDIUM_DRAFT="${LLM_MEDIUM_DRAFT:-/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf}"
LLM_MEDIUM_MMPROJ="${LLM_MEDIUM_MMPROJ:-/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF/mmproj-BF16.gguf}"
# The three vLLM user units are retired HERE, not left to rot: the proxy
# published 7770 on the host network, so the llama-server below cannot bind
@@ -1153,6 +1154,11 @@ deploy_llm_medium() {
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
missing=1
fi
if [[ ! -f "$LLM_MEDIUM_MMPROJ" ]]; then
echo "ERROR: vision projector absent at $LLM_MEDIUM_MMPROJ" >&2
echo "frank-medium: fetch it: bash scripts/download-frank-medium-gguf.sh" >&2
missing=1
fi
[[ "$missing" -eq 0 ]] || return 1
retire_vllm_medium_quadlets