frank-large: Q4_K_XL payload + 4 x 163840 sessions (f16 KV)

Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB,
SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB,
sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer
uncensored — that was one of the reasons for the Swift payload on 2026-09-28.

New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public
and ungated; size + sha256 verified after transfer). The speculative drafter
and its fetcher are unchanged.

Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV
quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so
f16 is inherent.

KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x
256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a
per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each
session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT.

Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
This commit is contained in:
Frank committed 2026-10-01 14:08:50 -07:00
1 parent 35e24db7e6
commit 8029119b19
7 files changed
+149 -26

No files matched your search

+29 -11
View File
@@ -8,14 +8,20 @@
# runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
# exception because the serve script itself consumes it for --host.
# The target payload (scripts/download-frank-large-swift.sh): the serve
# script refuses to start without the main GGUF and the DFlash2 drafter.
# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
# are structurally interchangeable for gufo (same qwen35 architecture, the
# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
# layout, the same MTP head), so only this path changed.
MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
# The target payload (scripts/download-frank-large-q4.sh): the serve script
# refuses to start without the main GGUF and the DFlash2 drafter.
#
# 2026-10-01 swap (Frank): unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL, 16.35 GiB,
# replacing ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB). Two
# consequences worth knowing at a glance:
# * this is the VANILLA (non-abliterated) Qwen3.8-27B — the endpoint is no
# longer uncensored, which was one of the reasons the Swift payload was
# chosen on 2026-09-28;
# * ~13 GiB of weights became available for KV, which is what makes four
# full-sized f16 sessions fit.
# gufo needs no template for either: its Qwen renderer is compiled in.
MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL
MODEL_FILE=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf
# The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
# the previous target, so it carries its own directory. It has a sliding
@@ -32,9 +38,21 @@ ALIAS=frank-large
HOST=0.0.0.0
PORT=7770
# One session, full 262144 context (the model's native window).
CONTEXT_SIZE=262144
SESSIONS=1
# Four sessions, 163840 context each (Frank, 2026-10-01: "4x 160k"). The
# context is per session and the KV is UNQUANTIZED (f16) — gufo has no KV
# quantization lever at all (execution_policy.hpp pins kv_cache_storage to
# kFp16), so "unquantized" is inherent rather than configured.
#
# The arithmetic that makes this fit: f16 KV costs 64 KiB/token on this model
# (16 full-attention layers x 4 KV heads x 256 head_dim x 2 (K,V) x 2 bytes;
# the other 48 layers are linear-attention with a per-SEQUENCE fp32 state of
# ~144 MiB, not a per-token cache). 4 x 163840 = 655,360 tokens = 40.0 GiB of
# KV, and each session's 163,840 tokens is exactly 10 GiB. Budget: weights
# 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = ~62 GiB
# against the 114 GiB GTT — roughly 50 GiB of headroom, so the session count
# or the per-session context can both grow from here.
CONTEXT_SIZE=163840
SESSIONS=4
# DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
# block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is