diff --git a/AGENTS.md b/AGENTS.md index 93859e8..3cb6a79 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **bf16 KV** (`--kv-cache-dtype auto`), **no speculative decode**, MNBT 2048, MoE gather 32×64, **2 seqs × 262144 ctx (the model's full native window)** at KV_BYTES 7,300,000,000/rank (bf16 measures 13,839 B/token), with **`--expert-precise-memory`** — the fitted 5.25 GiB overhead instead of the 7.60 GiB default, which hands the compile/graph-capture headroom to the expert arena (~7,278 slots vs 6,248) — util 0.98, and the client contract wired: `--reasoning-parser qwen3` (thinking in `reasoning`, never inline), `--enable-auto-tool-choice --tool-call-parser qwen3_xml`, default chat-template kwargs `enable_thinking` + `preserve_thinking`. Capacity history: five full-length contexts boot but leave ~0 bytes free at runtime (a real request with `max_tokens=261149` killed the worker: `Tried to allocate 50.00 MiB ... 0 bytes free`); six never fit (6×241664 asked 10.70 GiB vs 9.18 GiB free). DCP cannot help here — it requires TP > the model's 2 KV heads, i.e. four GPUs. fp8+MTP4 shape restorable from git tag `frank-primary-fp8-mtp-20261001` | 7770 | -| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL (27B dense, ABLITERATED/uncensored, 29.3 GiB, SWIFT OPEN LICENSE 1.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), 262144 ctx single session, greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only | 7770 | +| Dankclaw | `frank-large` | gufo `gufo-org/gufo` v0.1.1, the Strix Halo engine, BUILT FROM SOURCE (root systemd unit; links the host ROCm 7.2.4) | `unsloth/Qwen3.8-27B-GGUF` **UD-Q4_K_XL** (27B dense, **vanilla — no longer abliterated**, 16.35 GiB, Apache-2.0) with the z-lab DFlash2 Q4_K_M drafter (adaptive block policy, cap 7) on the gfx1151 iGPU (Strix Halo LPDDR, 114 GiB GTT), **4 sessions x 163840 ctx, unquantized (f16) KV** — gufo has no KV-quant lever — greedy, gufo's compiled-in Qwen renderer (NO Qwen-Sharp), text only. Swapped 2026-10-01 from ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB): ~13 GiB less weights is what funds four f16 sessions, at the cost of the uncensored behaviour | 7770 | | Dankclaw | `frank-small` | SGLang in a self-contained python3.12 venv (root systemd unit; the venv carries its own CUDA 13 stack — see the runbook) | letechlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, 12.0 GiB, INT8 W8A16 AutoRound, in-checkpoint vision tower, NO MTP weights shipped so no spec decode) on the RTX 3090, fp8 KV, pinned 606,208-token pool / 2 slots of 262,144 (native window; 2026-10-01 reslice from 4 x 151,552 for compaction headroom), bf16 mamba states, eager prefill, 83.9 tok/s decode | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -437,8 +437,13 @@ Details in the runbook. Verified at deploy (2026-10-01): `/v1/models` reports `frank-small`; chat returns content plus `reasoning_content` (SGLang's qwen3 parser uses llama.cpp's field name); a red-square image answers "Red" -(`image_tokens: 64`); 4 concurrent 54k-token requests all answer; a -170,019-token prompt prefills in 50.5 s (~3,370 t/s eager). Decode +(`image_tokens: 64`). At the 4 x 151,552 shape: 4 concurrent 54k-token +requests answered; a 170,019-token prompt prefilled in 50.5 s (~3,370 t/s +eager). At the 2 x 262,144 shape (reslice, same day): boot flags read +`context_len=262144 max_running_requests=2 max_total_num_tokens=606208`; +a 200,000-token prompt — well past frank-primary's 183,501 compaction +trigger — prefilled and completed in ONE request (97.3 s wall, ~2,056 +tok/s eager); two concurrent 50k-token requests both answered. Decode **83.9 tok/s**. Runbook: `serverAppConfigs-docs/LLM/frank-small-sglang-mimo-v26.md`. Rollback: [[frank-small-gemma4-12b]] (payload and binary intact on disk). diff --git a/scripts/download-frank-large-q4.sh b/scripts/download-frank-large-q4.sh new file mode 100755 index 0000000..00649e7 --- /dev/null +++ b/scripts/download-frank-large-q4.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# Download the Q4_K_XL frank-large payload on Dankclaw: +# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf +# +# bash scripts/download-frank-large-q4.sh +# bash scripts/download-frank-large-q4.sh --verify-only +# +# Integrity: the file is checked against its declared byte count and sha256 +# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED: +# the repo is public and ungated (gated: false), and a token on the curl +# command line is world-readable via /proc//cmdline. If this ever needs +# a token, put it in a mode-600 file and pass it with `curl -K`. +# +# Layout after a successful run (MODEL_DIR): +# Qwen3.8-27B-UD-Q4_K_XL.gguf 17559178144 (16.35 GiB) +# +# WHY Q4 AND NOT THE Q8 SWIFT PAYLOAD: this is the vanilla (non-abliterated) +# Qwen3.8-27B at UD-Q4_K_XL, ~13 GiB smaller than the Swift UD-Q8_K_XL. Every +# GiB of weights saved is a GiB of f16 KV the endpoint can hold instead: the +# KV costs 64 KiB/token (16 full-attention layers x 4 KV heads x 256 head_dim +# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB. +# +# The speculative drafter is UNCHANGED and still fetched by its own script +# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not +# appear here at all — the frank-large seam greps THIS file for it to prove +# the drafter was not folded into the payload fetch. +# +# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one +# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it — +# spec decode here is the block-diffusion drafter, not MTP. +set -euo pipefail + +MODEL_DIR="/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL" +REL="Qwen3.8-27B-UD-Q4_K_XL.gguf" +SIZE=17559178144 +SHA=3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e +URL="https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/${REL}" + +mode="download" +[[ "${1:-}" == "--verify-only" ]] && mode="verify" + +if [[ "$mode" == "download" ]]; then + mkdir -p "$MODEL_DIR" + if [[ -s "$MODEL_DIR/$REL" ]]; then + echo "already present: $MODEL_DIR/$REL" + else + echo "fetching $REL (17559178144 bytes) — this is ~16.4 GiB" + # -f so an HTTP error is a failure, not a 153-byte "artifact". + # -C - so an interrupted run resumes instead of restarting. + curl -fL -C - --retry 5 --retry-delay 3 -o "$MODEL_DIR/$REL" "$URL" + fi +fi + +echo "verifying $MODEL_DIR against declared size + sha256" +python3 - "$MODEL_DIR" "$REL" "$SIZE" "$SHA" <<'PY' +import hashlib, os, sys +model_dir, rel, size, sha = sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4] +dest = os.path.join(model_dir, rel) +if not os.path.isfile(dest): + print(f" MISSING {rel}") + sys.exit(1) +actual = os.path.getsize(dest) +if actual != size: + print(f" SIZE MISMATCH {rel}: {actual} != {size}") + sys.exit(1) +h = hashlib.sha256() +with open(dest, "rb") as f: + for chunk in iter(lambda: f.read(1 << 24), b""): + h.update(chunk) +if h.hexdigest() != sha: + print(f" SHA256 MISMATCH {rel}: {h.hexdigest()}") + sys.exit(1) +print(f" OK {rel} ({actual} bytes, sha256 matches the Hub LFS oid)") +PY + +echo "Q4_K_XL payload complete: $MODEL_DIR" diff --git a/scripts/frank-large.env b/scripts/frank-large.env index 764b142..d823d47 100644 --- a/scripts/frank-large.env +++ b/scripts/frank-large.env @@ -8,14 +8,20 @@ # runtime reads those bare (the 2026-08-13 TEMP incident). HOST is the one # exception because the serve script itself consumes it for --host. -# The target payload (scripts/download-frank-large-swift.sh): the serve -# script refuses to start without the main GGUF and the DFlash2 drafter. -# Swapped in from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL on 2026-09-28. The two -# are structurally interchangeable for gufo (same qwen35 architecture, the -# same 866 tensors in the same order, the same per-tensor Q8_0/F32/BF16 -# layout, the same MTP head), so only this path changed. -MODEL_DIR=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL -MODEL_FILE=/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf +# The target payload (scripts/download-frank-large-q4.sh): the serve script +# refuses to start without the main GGUF and the DFlash2 drafter. +# +# 2026-10-01 swap (Frank): unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL, 16.35 GiB, +# replacing ajgazin's abliterated Swift-1.5 UD-Q8_K_XL (29.3 GiB). Two +# consequences worth knowing at a glance: +# * this is the VANILLA (non-abliterated) Qwen3.8-27B — the endpoint is no +# longer uncensored, which was one of the reasons the Swift payload was +# chosen on 2026-09-28; +# * ~13 GiB of weights became available for KV, which is what makes four +# full-sized f16 sessions fit. +# gufo needs no template for either: its Qwen renderer is compiled in. +MODEL_DIR=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL +MODEL_FILE=/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf # The DFlash2 drafter is UNCHANGED by the model swap and still lives beside # the previous target, so it carries its own directory. It has a sliding @@ -32,9 +38,21 @@ ALIAS=frank-large HOST=0.0.0.0 PORT=7770 -# One session, full 262144 context (the model's native window). -CONTEXT_SIZE=262144 -SESSIONS=1 +# Four sessions, 163840 context each (Frank, 2026-10-01: "4x 160k"). The +# context is per session and the KV is UNQUANTIZED (f16) — gufo has no KV +# quantization lever at all (execution_policy.hpp pins kv_cache_storage to +# kFp16), so "unquantized" is inherent rather than configured. +# +# The arithmetic that makes this fit: f16 KV costs 64 KiB/token on this model +# (16 full-attention layers x 4 KV heads x 256 head_dim x 2 (K,V) x 2 bytes; +# the other 48 layers are linear-attention with a per-SEQUENCE fp32 state of +# ~144 MiB, not a per-token cache). 4 x 163840 = 655,360 tokens = 40.0 GiB of +# KV, and each session's 163,840 tokens is exactly 10 GiB. Budget: weights +# 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = ~62 GiB +# against the 114 GiB GTT — roughly 50 GiB of headroom, so the session count +# or the per-session context can both grow from here. +CONTEXT_SIZE=163840 +SESSIONS=4 # DFlash2 spec decode (a block-diffusion drafter). gufo sizes the draft # block itself; DRAFT_TOKENS caps the proposals per step (gufo's default is diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 578a20d..caf524c 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -18,7 +18,7 @@ # is not an OSI licence. # # Shape: one session, full 262144 context (the model's native window). The -# 27B dense weights (29.3 GiB) + the DFlash2 drafter (1.07 GiB) + KV + +# 27B dense weights (16.35 GiB at UD-Q4_K_XL) + the DFlash2 drafter (1.07) + KV + # compute buffers fit the iGPU's GTT aperture (~114 GiB); a load at this # context measured 66.5 GiB of the 116.7 GiB device. # @@ -78,7 +78,7 @@ GUFO_BIN="$RUNTIME_DIR/bin/gufo" # and keeps its own long filename, while the DFlash2 drafter is unchanged and # still lives beside the previous target. MODEL_FILE/DRAFTER_FILE default to # the deployed payload, so the env file only has to state them when it moves. -: "${MODEL_FILE:=$MODEL_DIR/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf}" +: "${MODEL_FILE:=$MODEL_DIR/Qwen3.8-27B-UD-Q4_K_XL.gguf}" : "${DRAFTER_DIR:=/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL}" : "${DRAFTER_FILE:=$DRAFTER_DIR/Qwen3.8-27B-DFlash2-Q4_K_M.gguf}" diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 86f0976..9240710 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -12,8 +12,8 @@ measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw runs two endpoints — gufo v0.1.1 (the Strix Halo engine, built from source; frank-large, -Swift-1.5-Qwen3.8-27B-Uncensored UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M -drafter on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small, +unsloth Qwen3.8-27B UD-Q4_K_XL (vanilla) + the z-lab DFlash2 Q4_K_M drafter +on the gfx1151 iGPU, 4 x 163840 ctx with f16 KV, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771). diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 10184c9..5e9c828 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -688,6 +688,27 @@ planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the planned arena; it comes out of free VRAM instead. +## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions + +Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5 +UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB, +sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer +uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2 +drafter and its fetcher are unchanged. + +Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no +KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage` +to `kFp16`), so "unquantized" is inherent. + +KV arithmetic (why this fits where one 262144 session used to sit): +f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is +`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x +2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a +per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 = +655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget: +weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = +~62 GiB against the 114 GiB GTT. + ## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner `specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the diff --git a/tests/frank_large_unit.bats b/tests/frank_large_unit.bats index de27f7f..db607d9 100755 --- a/tests/frank_large_unit.bats +++ b/tests/frank_large_unit.bats @@ -19,13 +19,13 @@ load helpers ENV_FILE="$REPO_ROOT/scripts/frank-large.env" UNIT="$REPO_ROOT/scripts/frank-large.service" SERVE="$REPO_ROOT/scripts/serve-frank-large.sh" -FETCH="$REPO_ROOT/scripts/download-frank-large-swift.sh" +FETCH="$REPO_ROOT/scripts/download-frank-large-q4.sh" DRAFTER_FETCH="$REPO_ROOT/scripts/download-frank-large.sh" BUILD="$REPO_ROOT/scripts/build-gufo.sh" REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh" # The deployed target's filename, kept here so a rename in the env file and a # rename in this stub cannot drift apart unnoticed. -TARGET_STUB="Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf" +TARGET_STUB="Qwen3.8-27B-UD-Q4_K_XL.gguf" setup() { STAGE="$(mktemp -d)" RUNTIME_DIR="$STAGE/gufo" @@ -144,8 +144,10 @@ argv_value() { [ "$(argv_value --served-model-name)" = "frank-large" ] [ "$(argv_value --host)" = "0.0.0.0" ] [ "$(argv_value --port)" = "7770" ] - [ "$(argv_value --context)" -eq 262144 ] - [ "$(argv_value --sessions)" -eq 1 ] + # 2026-10-01: four sessions x 163840 context (Frank's "4x 160k"), f16 KV. + # 163840 tokens x 64 KiB/token = 10 GiB of KV per session exactly. + [ "$(argv_value --context)" -eq 163840 ] + [ "$(argv_value --sessions)" -eq 4 ] [ "$(argv_value --speculative)" = "dflash2" ] [ "$(argv_value --draft-tokens)" -gt 0 ] [ "$(argv_value --draft-policy)" = "adaptive" ] @@ -222,7 +224,7 @@ argv_value() { # so a plain grep stayed green after the URL was repointed at another # repository -- the script would then download something the serve script # never asked for. - grep -qE '^URL="https://huggingface\.co/ajgazin/Swift-1\.5-Qwen3\.8-27B-Uncensored-Dynamic-MTP-GGUF/resolve/main/' "$FETCH" + grep -qE '^URL="https://huggingface\.co/unsloth/Qwen3\.8-27B-GGUF/resolve/main/' "$FETCH" grep -qE 'https://huggingface\.co/z-lab/Qwen3\.8-27B-DFlash2-GGUF/resolve/main/' "$DRAFTER_FETCH" for f in "$FETCH" "$DRAFTER_FETCH"; do # -f so an HTTP error is a failure rather than a small "artifact"; @@ -252,9 +254,10 @@ argv_value() { } @test "the drafter is still fetched by the script that carries its Hub repo" { - # Guards a future "simplification" that folds the drafter into the Swift - # script and silently orphans the artifact the serve script still points - # at via DRAFTER_DIR. + # Guards a future "simplification" that folds the drafter into the + # payload-fetch script and silently orphans the artifact the serve script + # still points at via DRAFTER_DIR. The literal drafter name must not + # appear in that script AT ALL, comments included. grep -q 'Qwen3.8-27B-UD-Q8_K_XL' "$DRAFTER_FETCH" ! grep -q 'DFlash2' "$FETCH" }