dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: Q4_K_XL payload + 4 x 163840 sessions (f16 KV)

Browse Source 
Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB,
SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB,
sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer
uncensored — that was one of the reasons for the Swift payload on 2026-09-28.

New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public
and ungated; size + sha256 verified after transfer). The speculative drafter
and its fetcher are unchanged.

Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV
quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so
f16 is inherent.

KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x
256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a
per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each
session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT.

Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
This commit is contained in:
Frank
2026-10-01 14:08:50 -07:00
parent 35e24db7e6
commit 8029119b19
7 changed files with 149 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/index.md
+2 -2
View File
@@ -12,8 +12,8 @@ measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
runs two endpoints —
gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M
drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter
on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small,
HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
port 7771).