frank-large: Q4_K_XL payload + 4 x 163840 sessions (f16 KV)
Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB, SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB, sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer uncensored — that was one of the reasons for the Swift payload on 2026-09-28. New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public and ungated; size + sha256 verified after transfer). The speculative drafter and its fetcher are unchanged. Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so f16 is inherent. KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT. Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
This commit is contained in:
1 parent
35e24db7e6
commit
8029119b19
7 files changed
+149
-26
No files matched your search
@@ -18,7 +18,7 @@
|
||||
# is not an OSI licence.
|
||||
#
|
||||
# Shape: one session, full 262144 context (the model's native window). The
|
||||
# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
|
||||
# 27B dense weights (16.35 GiB at UD-Q4_K_XL) + the DFlash2 drafter (1.07) + KV +
|
||||
# compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
|
||||
# context measured 66.5 GiB of the 116.7 GiB device.
|
||||
#
|
||||
@@ -78,7 +78,7 @@ GUFO_BIN="$RUNTIME_DIR/bin/gufo"
|
||||
# and keeps its own long filename, while the DFlash2 drafter is unchanged and
|
||||
# still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
|
||||
# the deployed payload, so the env file only has to state them when it moves.
|
||||
: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
|
||||
: "${MODEL_FILE:=$MODEL_DIR/Qwen3.8-27B-UD-Q4_K_XL.gguf}"
|
||||
: "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
|
||||
: "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
|
||||
|
||||
|
||||
Reference in new issue
Block a user