Frank
d95f55b822
The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV, and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point, computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82 GiB free, both above the 1.5 GiB floor. 36:29 is one layer over the 35:30 balance on purpose: at this context the balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0 to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so the seam's slack is that gap rounded up (3225 MiB) with the trade written down, and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries no scaling; positions past the trained 262144 extrapolate on the base rope). Vision: NO_MMPROJ=0, the serve script now refuses to start without the projector when vision is on (a missing file is a deploy error, not a silent text-only server), the download script fetches and byte-verifies it, and the deploy handler's preflight checks for it. The projector is already on the host, byte- and sha256-verified against the Hub.
85 lines
3.2 KiB
Bash
Executable File
85 lines
3.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# the target GGUF, the BF16 vision projector and the DFlash2 draft model.
|
|
#
|
|
# bash scripts/download-frank-medium-gguf.sh
|
|
# bash scripts/download-frank-medium-gguf.sh --verify-only
|
|
#
|
|
# Both repositories are public (anonymous download; no token needed). If a
|
|
# token exists at ~/.cache/huggingface/token it is used, which keeps a 31 GB
|
|
# transfer clear of the anonymous rate limit. The script is resumable
|
|
# (curl -C -) and checks every artifact's exact byte count afterwards, so a
|
|
# truncated file fails here instead of load-crashing the server.
|
|
#
|
|
# WHY THESE TWO FILES:
|
|
# * Target: ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at
|
|
# UD-Q8_K_XL -- Unsloth Dynamic 3.0 quant (per-tensor types from Unsloth's
|
|
# own quant of the same size, on Unsloth's importance matrix). The file
|
|
# carries the model's MTP layer (`blk.64.nextn.*`), so llama.cpp can also
|
|
# draft from the model itself with no second copy in VRAM.
|
|
# * Draft: z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0 -- the block-diffusion
|
|
# drafter (5 layers, hidden 5120, block_size 8, a 2048-token sliding
|
|
# window). Q8_0 is 1.92 GiB against BF16's 3.6 GiB and the drafter's job is
|
|
# throughput, not output quality (the target verifies every drafted token).
|
|
set -euo pipefail
|
|
|
|
MODEL_REPO="ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
|
|
MODEL_DIR="/home/dank/models/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF"
|
|
MODEL_FILE="Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
|
|
MODEL_SIZE=31457990112
|
|
# The BF16 vision projector from the same repo; llama-server autoloads any
|
|
# mmproj*.gguf beside the target, so it must land in MODEL_DIR.
|
|
MM_PROJ_FILE="mmproj-BF16.gguf"
|
|
MM_PROJ_SIZE=931145888
|
|
|
|
DRAFT_REPO="z-lab/Qwen3.8-27B-DFlash2-GGUF"
|
|
DRAFT_DIR="/home/dank/models/Qwen3.8-27B-DFlash2-GGUF"
|
|
DRAFT_FILE="Qwen3.8-27B-DFlash2-Q8_0.gguf"
|
|
DRAFT_SIZE=2056414816
|
|
|
|
mode="download"
|
|
[[ "${1:-}" == "--verify-only" ]] && mode="verify"
|
|
|
|
token=""
|
|
auth=()
|
|
if [[ -r "$HOME/.cache/huggingface/token" ]]; then
|
|
token="$(cat "$HOME/.cache/huggingface/token")"
|
|
auth=(-H "Authorization: Bearer $token")
|
|
else
|
|
echo "no HF token at ~/.cache/huggingface/token; downloading anonymously"
|
|
fi
|
|
|
|
dl() { # dl <repo> <dir> <file>
|
|
mkdir -p "$2"
|
|
echo " fetching $3"
|
|
curl -fL -C - -o "$2/$3" --retry 10 --retry-delay 5 --retry-max-time 21600 \
|
|
"${auth[@]}" "https://huggingface.co/$1/resolve/main/$3"
|
|
}
|
|
|
|
if [[ "$mode" == "download" ]]; then
|
|
echo "target: $MODEL_REPO"
|
|
dl "$MODEL_REPO" "$MODEL_DIR" "$MODEL_FILE"
|
|
echo "projector: $MODEL_REPO"
|
|
dl "$MODEL_REPO" "$MODEL_DIR" "$MM_PROJ_FILE"
|
|
echo "draft: $DRAFT_REPO"
|
|
dl "$DRAFT_REPO" "$DRAFT_DIR" "$DRAFT_FILE"
|
|
fi
|
|
|
|
echo "verifying sizes"
|
|
ok=1
|
|
check() { # check <path> <expected>
|
|
local act
|
|
act="$(stat -c%s "$1" 2>/dev/null || echo 0)"
|
|
if [[ "$act" -eq "$2" ]]; then
|
|
echo " OK $1 ($act bytes)"
|
|
else
|
|
echo " BAD $1 ($act bytes, want $2)" >&2
|
|
ok=0
|
|
fi
|
|
}
|
|
check "$MODEL_DIR/$MODEL_FILE" "$MODEL_SIZE"
|
|
check "$MODEL_DIR/$MM_PROJ_FILE" "$MM_PROJ_SIZE"
|
|
check "$DRAFT_DIR/$DRAFT_FILE" "$DRAFT_SIZE"
|
|
[[ -z "$token" ]] && echo "note: no token used; the byte counts above are the integrity check"
|
|
[[ "$ok" -eq 1 ]] || exit 1
|
|
echo "all artifacts present and sized correctly"
|