frank-large: Q4_K_XL payload + 4 x 163840 sessions (f16 KV)
Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB, SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB, sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer uncensored — that was one of the reasons for the Swift payload on 2026-09-28. New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public and ungated; size + sha256 verified after transfer). The speculative drafter and its fetcher are unchanged. Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so f16 is inherent. KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT. Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
This commit is contained in:
1 parent
35e24db7e6
commit
8029119b19
7 files changed
+149
-26
No files matched your search
@@ -19,13 +19,13 @@ load helpers
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
|
||||
UNIT="$REPO_ROOT/scripts/frank-large.service"
|
||||
SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
|
||||
FETCH="$REPO_ROOT/scripts/download-frank-large-q4.sh"
|
||||
DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
|
||||
BUILD="$REPO_ROOT/scripts/build-gufo.sh"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
# The deployed target's filename, kept here so a rename in the env file and a
|
||||
# rename in this stub cannot drift apart unnoticed.
|
||||
TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
|
||||
TARGET_STUB="Qwen3.8-27B-UD-Q4_K_XL.gguf"
|
||||
setup() {
|
||||
STAGE="$(mktemp -d)"
|
||||
RUNTIME_DIR="$STAGE/gufo"
|
||||
@@ -144,8 +144,10 @@ argv_value() {
|
||||
[ "$(argv_value --served-model-name)" = "frank-large" ]
|
||||
[ "$(argv_value --host)" = "0.0.0.0" ]
|
||||
[ "$(argv_value --port)" = "7770" ]
|
||||
[ "$(argv_value --context)" -eq 262144 ]
|
||||
[ "$(argv_value --sessions)" -eq 1 ]
|
||||
# 2026-10-01: four sessions x 163840 context (Frank's "4x 160k"), f16 KV.
|
||||
# 163840 tokens x 64 KiB/token = 10 GiB of KV per session exactly.
|
||||
[ "$(argv_value --context)" -eq 163840 ]
|
||||
[ "$(argv_value --sessions)" -eq 4 ]
|
||||
[ "$(argv_value --speculative)" = "dflash2" ]
|
||||
[ "$(argv_value --draft-tokens)" -gt 0 ]
|
||||
[ "$(argv_value --draft-policy)" = "adaptive" ]
|
||||
@@ -222,7 +224,7 @@ argv_value() {
|
||||
# so a plain grep stayed green after the URL was repointed at another
|
||||
# repository -- the script would then download something the serve script
|
||||
# never asked for.
|
||||
grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
|
||||
grep -qE '^URL="https://huggingface\.co/unsloth/Qwen3\.8-27B-GGUF/resolve/main/' "$FETCH"
|
||||
grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
|
||||
for f in "$FETCH" "$DRAFTER_FETCH"; do
|
||||
# -f so an HTTP error is a failure rather than a small "artifact";
|
||||
@@ -252,9 +254,10 @@ argv_value() {
|
||||
}
|
||||
|
||||
@test "the drafter is still fetched by the script that carries its Hub repo" {
|
||||
# Guards a future "simplification" that folds the drafter into the Swift
|
||||
# script and silently orphans the artifact the serve script still points
|
||||
# at via DRAFTER_DIR.
|
||||
# Guards a future "simplification" that folds the drafter into the
|
||||
# payload-fetch script and silently orphans the artifact the serve script
|
||||
# still points at via DRAFTER_DIR. The literal drafter name must not
|
||||
# appear in that script AT ALL, comments included.
|
||||
grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
|
||||
! grep -q 'DFlash2' "$FETCH"
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user