frank-large: Q4_K_XL payload + 4 x 163840 sessions (f16 KV)

Frank's call. Payload: ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB, SWIFT OPEN LICENSE) -> unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (16.35 GiB, sha256 3f227079...b01e, vanilla/Apache-2.0). NOTE: the endpoint is no longer uncensored — that was one of the reasons for the Swift payload on 2026-09-28. New fetcher scripts/download-frank-large-q4.sh (no auth header: repo is public and ungated; size + sha256 verified after transfer). The speculative drafter and its fetcher are unchanged. Shape: --sessions 4 --context 163840 with unquantized KV. gufo exposes no KV quantization lever (execution_policy.hpp pins kv_cache_storage to kFp16), so f16 is inherent. KV arithmetic: f16 KV = 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim x 2 x 2B; the other 48 layers are linear-attention with a per-sequence fp32 state). 4 x 163840 = 655,360 tokens = 40.0 GiB KV, each session exactly 10 GiB. Budget ~62 GiB of the 114 GiB GTT. Seam retargeted (FETCH -> the q4 script, TARGET_STUB, context/sessions pins).
2026-10-01 14:08:50 -07:00
parent 35e24db7e6
commit 8029119b19
7 changed files with 149 additions and 26 deletions
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 |
-| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 |
+| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | `unsloth/Qwen3.8-27B-GGUF` **UD-Q4_K_XL** (27B dense, **vanilla — no longer abliterated**, 16.35 GiB, Apache-2.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), **4 sessions x 163840 ctx, unquantized (f16) KV** — gufo has no KV-quant lever — greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only. Swapped 2026-10-01 from ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB): ~13 GiB less weights is what funds four f16 sessions, at the cost of the uncensored behaviour | 7770 |
 | Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 |

 Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -437,8 +437,13 @@ Details in the runbook.
 Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`;
 chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses
 llama.cpp's field name); a red-square image answers "Red"
-(`image_tokens: 64`); 4 concurrent 54k-token requests all answer; a
-170,019-token prompt prefills in 50.5 s (~3,370 t/s eager). Decode
+(`image_tokens: 64`). At the 4 x 151,552 shape: 4 concurrent 54k-token
+requests answered; a 170,019-token prompt prefilled in 50.5 s (~3,370 t/s
+eager). At the 2 x 262,144 shape (reslice, same day): boot flags read
+`context_len=262144 max_running_requests=2 max_total_num_tokens=606208`;
+a 200,000-token prompt — well past frank-primary's 183,501 compaction
+trigger — prefilled and completed in ONE request (97.3 s wall, ~2,056
+tok/s eager); two concurrent 50k-token requests both answered. Decode
 **83.9 tok/s**.
 Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`.
 Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk).
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Download the Q4_K_XL frank-large payload on Dankclaw:
+#   unsloth/Qwen3.8-27B-GGUF  ->  Qwen3.8-27B-UD-Q4_K_XL.gguf
+#
+#   bash scripts/download-frank-large-q4.sh
+#   bash scripts/download-frank-large-q4.sh --verify-only
+#
+# Integrity: the file is checked against its declared byte count and sha256
+# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
+# the repo is public and ungated (gated: false), and a token on the curl
+# command line is world-readable via /proc/<pid>/cmdline. If this ever needs
+# a token, put it in a mode-600 file and pass it with `curl -K`.
+#
+# Layout after a successful run (MODEL_DIR):
+#   Qwen3.8-27B-UD-Q4_K_XL.gguf   17559178144  (16.35 GiB)
+#
+# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated)
+# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every
+# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the
+# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim
+# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
+#
+# The speculative drafter is UNCHANGED and still fetched by its own script
+# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
+# appear here at all — the frank-large seam greps THIS file for it to prove
+# the drafter was not folded into the payload fetch.
+#
+# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
+# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
+# spec decode here is the block-diffusion drafter, not MTP.
+set -euo pipefail
+
+MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL"
+REL="Qwen3.8-27B-UD-Q4_K_XL.gguf"
+SIZE=17559178144
+SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e
+URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}"
+
+mode="download"
+[[ "${1:-}" == "--verify-only" ]] && mode="verify"
+
+if [[ "$mode" == "download" ]]; then
+    mkdir -p "$MODEL_DIR"
+    if [[ -s "$MODEL_DIR/$REL" ]]; then
+        echo "already present: $MODEL_DIR/$REL"
+    else
+        echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB"
+        # -f so an HTTP error is a failure, not a 153-byte "artifact".
+        # -C - so an interrupted run resumes instead of restarting.
+        curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL"
+    fi
+fi
+
+echo "verifying $MODEL_DIR against declared size + sha256"
+python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY'
+import hashlib, os, sys
+model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4]
+dest = os.path.join(model_dir, rel)
+if not os.path.isfile(dest):
+    print(f"  MISSING {rel}")
+    sys.exit(1)
+actual = os.path.getsize(dest)
+if actual != size:
+    print(f"  SIZE MISMATCH {rel}: {actual} != {size}")
+    sys.exit(1)
+h = hashlib.sha256()
+with open(dest, "rb") as f:
+    for chunk in iter(lambda: f.read(1 << 24), b""):
+        h.update(chunk)
+if h.hexdigest() != sha:
+    print(f"  SHA256 MISMATCH {rel}: {h.hexdigest()}")
+    sys.exit(1)
+print(f"  OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)")
+PY
+
+echo "Q4_K_XL payload complete: $MODEL_DIR"
@@ -8,14 +8,20 @@
 # runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one
 # exception because the serve script itself consumes it for --host.

-# The target payload (scripts/download-frank-large-swift.sh): the serve
-# script refuses to start without the main GGUF and the DFlash2 drafter.
-# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two
-# are structurally interchangeable for gufo (same qwen35 architecture, the
-# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16
-# layout, the same MTP head), so only this path changed.
-MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL
-MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
+# The target payload (scripts/download-frank-large-q4.sh): the serve script
+# refuses to start without the main GGUF and the DFlash2 drafter.
+#
+# 2026-10-01 swap (Frank): unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL, 16.35 GiB,
+# replacing ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB). Two
+# consequences worth knowing at a glance:
+#   * this is the VANILLA (non-abliterated) Qwen3.8-27B — the endpoint is no
+#     longer uncensored, which was one of the reasons the Swift payload was
+#     chosen on 2026-09-28;
+#   * ~13 GiB of weights became available for KV, which is what makes four
+#     full-sized f16 sessions fit.
+# gufo needs no template for either: its Qwen renderer is compiled in.
+MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL
+MODEL_FILE=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf

 # The DFlash2 drafter is UNCHANGED by the model swap and still lives beside
 # the previous target, so it carries its own directory. It has a sliding
@@ -32,9 +38,21 @@ ALIAS=frank-large
 HOST=0.0.0.0
 PORT=7770

-# One session, full 262144 context (the model's native window).
-CONTEXT_SIZE=262144
-SESSIONS=1
+# Four sessions, 163840 context each (Frank, 2026-10-01: "4x 160k"). The
+# context is per session and the KV is UNQUANTIZED (f16) — gufo has no KV
+# quantization lever at all (execution_policy.hpp pins kv_cache_storage to
+# kFp16), so "unquantized" is inherent rather than configured.
+#
+# The arithmetic that makes this fit: f16 KV costs 64 KiB/token on this model
+# (16 full-attention layers x 4 KV heads x 256 head_dim x 2 (K,V) x 2 bytes;
+# the other 48 layers are linear-attention with a per-SEQUENCE fp32 state of
+# ~144 MiB, not a per-token cache). 4 x 163840 = 655,360 tokens = 40.0 GiB of
+# KV, and each session's 163,840 tokens is exactly 10 GiB. Budget: weights
+# 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = ~62 GiB
+# against the 114 GiB GTT — roughly 50 GiB of headroom, so the session count
+# or the per-session context can both grow from here.
+CONTEXT_SIZE=163840
+SESSIONS=4

 # DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft
 # block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is
@@ -18,7 +18,7 @@
 # is not an OSI licence.
 #
 # Shape: one session, full 262144 context (the model's native window). The
-# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV +
+# 27B dense weights (16.35 GiB at UD-Q4_K_XL) + the DFlash2 drafter (1.07) + KV +
 # compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this
 # context measured 66.5 GiB of the 116.7 GiB device.
 #
@@ -78,7 +78,7 @@ GUFO_BIN="$RUNTIME_DIR/bin/gufo"
 # and keeps its own long filename, while the DFlash2 drafter is unchanged and
 # still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to
 # the deployed payload, so the env file only has to state them when it moves.
-: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}"
+: "${MODEL_FILE:=$MODEL_DIR/Qwen3.8-27B-UD-Q4_K_XL.gguf}"
 : "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}"
 : "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"

@@ -12,8 +12,8 @@ measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
 the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
 runs two endpoints —
 gufo v0.1.1 (the Strix Halo engine, built from source; frank-large,
-Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M
-drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
+unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter
+on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small,
 HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090,
 port 7771).

@@ -688,6 +688,27 @@ planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
 with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
 planned arena; it comes out of free VRAM instead.

+## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions
+
+Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
+UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
+sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
+uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2
+drafter and its fetcher are unchanged.
+
+Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
+KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
+to `kFp16`), so "unquantized" is inherent.
+
+KV arithmetic (why this fits where one 262144 session used to sit):
+f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
+`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
+2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
+per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
+655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
+weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
+~62 GiB against the 114 GiB GTT.
+
 ## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner

 `specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
@@ -19,13 +19,13 @@ load helpers
 ENV_FILE="$REPO_ROOT/scripts/frank-large.env"
 UNIT="$REPO_ROOT/scripts/frank-large.service"
 SERVE="$REPO_ROOT/scripts/serve-frank-large.sh"
-FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh"
+FETCH="$REPO_ROOT/scripts/download-frank-large-q4.sh"
 DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh"
 BUILD="$REPO_ROOT/scripts/build-gufo.sh"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
 # The deployed target's filename, kept here so a rename in the env file and a
 # rename in this stub cannot drift apart unnoticed.
-TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf"
+TARGET_STUB="Qwen3.8-27B-UD-Q4_K_XL.gguf"
 setup() {
    STAGE="$(mktemp -d)"
    RUNTIME_DIR="$STAGE/gufo"
@@ -144,8 +144,10 @@ argv_value() {
    [ "$(argv_value --served-model-name)" = "frank-large" ]
    [ "$(argv_value --host)" = "0.0.0.0" ]
    [ "$(argv_value --port)" = "7770" ]
-    [ "$(argv_value --context)" -eq 262144 ]
-    [ "$(argv_value --sessions)" -eq 1 ]
+    # 2026-10-01: four sessions x 163840 context (Frank's "4x 160k"), f16 KV.
+    # 163840 tokens x 64 KiB/token = 10 GiB of KV per session exactly.
+    [ "$(argv_value --context)" -eq 163840 ]
+    [ "$(argv_value --sessions)" -eq 4 ]
    [ "$(argv_value --speculative)" = "dflash2" ]
    [ "$(argv_value --draft-tokens)" -gt 0 ]
    [ "$(argv_value --draft-policy)" = "adaptive" ]
@@ -222,7 +224,7 @@ argv_value() {
    # so a plain grep stayed green after the URL was repointed at another
    # repository -- the script would then download something the serve script
    # never asked for.
-    grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH"
+    grep -qE '^URL="https://huggingface\.co/unsloth/Qwen3\.8-27B-GGUF/resolve/main/' "$FETCH"
    grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH"
    for f in "$FETCH" "$DRAFTER_FETCH"; do
        # -f so an HTTP error is a failure rather than a small "artifact";
@@ -252,9 +254,10 @@ argv_value() {
 }

@test "the drafter is still fetched by the script that carries its Hub repo" {
-    # Guards a future "simplification" that folds the drafter into the Swift
-    # script and silently orphans the artifact the serve script still points
-    # at via DRAFTER_DIR.
+    # Guards a future "simplification" that folds the drafter into the
+    # payload-fetch script and silently orphans the artifact the serve script
+    # still points at via DRAFTER_DIR. The literal drafter name must not
+    # appear in that script AT ALL, comments included.
    grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH"
    ! grep -q 'DFlash2' "$FETCH"
 }