From 2b127163c83feca5afa57bc3a472cab41d9aa05b Mon Sep 17 00:00:00 2001 From: Frank Date: Sun, 18 Oct 2026 03:05:09 -0700 Subject: [PATCH] feat(frank-dragoon): double the KV pool to 2 x ctx for slot parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit frank-dragoon's total in-flight context was HALF frank-zealot's: zealot runs one 401,408-position pool split into two 200,704 conversations, while dragoon ran ONE 200,704-position pool behind its two slots, so a single maximum-length request filled it and two long conversations had to stay under ~100k each. HALOGEN_KV_POOL_POSITIONS 200704 -> 401408 = slots x ctx, the top of the seam's legal band (ctx <= pool <= slots * ctx). The PER-REQUEST ceiling does not move: HALOGEN_CTX stays 200704, because the checkpoint's native window is 262,144 and the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN. What this buys is concurrency, not a longer conversation, which is why the harness contextWindow (155648 = ceiling - 45056) is untouched. Cost: ~10.8 GiB of KV where 200,704 cost 5.5 (262,144 cost 7.2; the 2026-10-06 shape's 524,288 measured 14.6), taken from the page cache the 51.2 GB n-gram table reads through — the warm prompt-cache ratio is the number to re-measure after the boot. Engine-side total goes 74.1 -> ~79.4 GiB of 121.2. Also carries scripts/download-frank-zealot-bonsai2-ptq1.sh: the PTQ1_0 candidate payload fetcher (5,946,648,928 B, LFS sha256-verified, mmproj hardlinked), with the publisher's own throughput table in its header — the packing is smaller but the card measures it SLOWER on Ampere, so the swap is decided on our card, not on the card's A100 row. --- AGENTS.md | 49 +++---- scripts/deploy-remote.sh | 5 +- scripts/download-frank-zealot-bonsai2-ptq1.sh | 124 ++++++++++++++++++ scripts/frank-dragoon.env | 45 ++++--- scripts/frank-dragoon.service | 2 +- .../LLM/dankclaw-iommu-translation-off.md | 4 +- .../frank-dragoon-flash-server-v2-native.md | 3 +- serverAppConfigs-docs/LLM/index.md | 17 ++- serverAppConfigs-docs/index.md | 8 +- tests/frank_dragoon_unit.bats | 14 +- 10 files changed, 209 insertions(+), 62 deletions(-) create mode 100755 scripts/download-frank-zealot-bonsai2-ptq1.sh diff --git a/AGENTS.md b/AGENTS.md index 27dca71..cf323b2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 | -| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=200704` pool at `HALOGEN_CTX=200704`** (lowered from the card's 262,144 the same evening — 5.5 GiB reserved where 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; pool == ctx so ONE maximum-length request fills it, and the freed RAM is page cache for the n-gram table), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| +| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once — the parity with frank-zealot's 401,408 pool on total in-flight context; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`; every pool position is page cache the n-gram table loses), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| | Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -396,15 +396,19 @@ request ≈ 156.2k ≤ 200,704); compaction fires at 85 % of the harness window `HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter" law does NOT carry over; batching and speculation coexist (the card's quickstart ships 2 slots with the head), so 2 is concurrent capacity. -`HALOGEN_KV_POOL_POSITIONS=200704` — **lowered from the card's 262,144 the same -evening** (Frank's call: every position costs KV RAM and nothing in the fleet -asks for the biggest window) — is **ONE pool behind both slots**, ledger form -`2 slots over ONE -position KV pool, each request up to HALOGEN_CTX`, and -at pool == ctx a single MAXIMUM-length request fills it: two maximum-length -conversations never coexist (two of ≤100k do). ~28 KiB/position ⇒ the ledger -reserves "about 5.5 GiB", where 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6; the -engine's preflight FORMULA bills slightly more than the reservation (block -scratch + ~115 MiB/slot of O(1) state) — quote the reservation. +`HALOGEN_KV_POOL_POSITIONS=401408` — **2 × ctx, RAISED from 200,704 on 2026-10-18** +(Frank: dragoon's total in-flight context was HALF frank-zealot's 401,408 +pool — doubling the POOL is what fixes that, not doubling the window) — is +**ONE pool behind both slots**, ledger form `2 slots over ONE -position KV +pool, each request up to HALOGEN_CTX`, and at pool == slots × ctx two +MAXIMUM-length conversations DO coexist. The PER-REQUEST ceiling stays +`HALOGEN_CTX=200704`: the checkpoint's native window is 262,144 and the engine's +own 400 text refuses to exceed it without `HALOGEN_ROPE_YARN`, so "more context" +bought here is CONCURRENCY, not a longer single conversation. ~28 KiB/position ⇒ +the ledger reserves "about 10.8 GiB", where 200,704 cost 5.5, 262,144 cost 7.2 +and the 2026-10-06 shape's 524,288 measured 14.6; the engine's preflight FORMULA +bills slightly more than the reservation (block scratch + ~115 MiB/slot of O(1) +state) — quote the reservation. `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table). @@ -421,17 +425,18 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content + `image_url` content parts only; ≥256 px probe images or the model answers "video". -**Memory ledger** (121.2 GiB unified; the referee is the boot ledger — the -200,704 boot printed exactly): weights 62.9 (trunk 62.11 + vision 0.84) + KV -5.5 + working 5.7 (`3.8 + 8.4 × MAX_TOK/16384`) = **74.1 GiB in all**, the -~1.0 GiB prompt cache booked separately by the ledger, + -`HALOGEN_HOST_RESERVE_GIB=20` → ~30 GiB -margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd page -cache beside the checkpoint (ledger: `mapped 51.2 GB, NOT pinned -(Pin::None)`) — which is why every pool cut is a double saving: **RAM the pool -takes is RAM that table loses**. `HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the -v2 loader, so disk-backing the trunk is not a lever on this format. READ the -boot ledger and re-size if the engine prints a clamp. +**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): weights +62.9 (trunk 62.11 + vision 0.84) + KV ~10.8 + working 5.7 +(`3.8 + 8.4 × MAX_TOK/16384`) ≈ **79.4 GiB in all**, the ~1.0 GiB prompt cache +booked separately by the ledger, + `HALOGEN_HOST_RESERVE_GIB=20` → ~22 GiB +margin over baseline co-tenancy (the pool-equals-ctx boot printed 74.1 GiB with +~30 GiB of margin). The 51.2 GB n-gram table rides as mmap'd page cache beside +the checkpoint (ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why +every pool position is a double cost: **RAM the pool takes is RAM that table +loses**, so the warm prompt-cache ratio is re-measured after a raise. +`HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the v2 loader, so disk-backing the +trunk is not a lever on this format. READ the boot ledger and re-size if the +engine prints a clamp. Fetch: `scripts/download-frank-dragoon-v2abl.sh` into `/home/dank/models/halogen-v2-abliterated/` (`-f` + exact byte counts + LFS @@ -1179,7 +1184,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 200,704 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` | +| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 401,408 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` | | frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` | | frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 294a429..ff01b09 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1097,8 +1097,9 @@ deploy_anubis() { # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next — # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident) # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode -# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 200,704-position KV pool at the -# matching ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a +# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx, +# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the +# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a # ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern): # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity diff --git a/scripts/download-frank-zealot-bonsai2-ptq1.sh b/scripts/download-frank-zealot-bonsai2-ptq1.sh new file mode 100755 index 0000000..f3a2506 --- /dev/null +++ b/scripts/download-frank-zealot-bonsai2-ptq1.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# Candidate payload for frank-zealot: prism-ml/Ternary-Bonsai-2-27B-gguf at +# **PTQ1_0** — the DENSE ternary packing (1.75 bits/weight, 5.95 GB) of the same +# weights the live PQ2_0 endpoint serves (2.13 bits/weight, 7.21 GB). Public +# repo: no auth header, and never a token on the command line (a Bearer token is +# world-readable in /proc//cmdline). +# +# WHY THIS ARM EXISTS. The whole argument is footprint: 5,946,648,928 B against +# the live 7,206,168,928 B = **1,201 MiB freed on the card**. On this endpoint +# KV bills 34,816 B/token (16 full-attention layers x 4 KV heads x 256 dim at +# q8_0 K + q8_0 V), so that one gap is ~37,000 tokens of pool per 1,201 MiB — +# context is exactly what the freed weight bytes buy. +# +# WHY IT MIGHT LOSE. The publisher's own throughput table (README, llama-bench, +# batch 1, depth 0): PTQ1_0 wins decode only on Ada/L4 (RTX 4090 91.1 vs 81.2, +# RTX 6000 Ada 90.4 vs 82.8, L4 32.1 vs 29.8) and LOSES on H100 (86.9 vs 113.9) +# and A100 (54.7 vs 73.9); prompt processing "favors PQ2_0 everywhere" (A100 +# 706 vs 1328). The card text says explicitly that dense-trit unpacking costs +# arithmetic and the pack is "slower on Ampere, Hopper, and Blackwell". There is +# NO GA102/RTX 3090 row: the Ampere statement generalizes from an A100, which is +# instruction-bound at batch 1 with 2 TB/s of HBM, while the 3090 is the +# bandwidth-starved member of the family (936 GB/s) — the one regime where +# moving 17 % less weight per step can pay. That is an argument, not a +# measurement: this script exists so the decision is made on OUR card, against +# the PQ2_0 record (76.6 t/s single-stream decode, 1,258 t/s prefill at 35k). +# +# THE RUNTIME LOCK CARRIES OVER. Both packings are custom GGML tensor types and +# stock llama.cpp rejects them ("It rejects PQ2_0 and PTQ1_0 as unknown types" +# — the card), so the PrismML fork build stays mandatory. Confirm the loadability +# against the installed binary BEFORE pointing any unit at this file. +# +# The vision projector is the SAME file the PQ2_0 endpoint already uses (Hub LFS +# oid 6807ede61d57… matches byte-for-byte), so it is HARDLINKED from the existing +# payload dir instead of re-downloaded — a hardlink is a regular file, so the +# serve gates' no-symlink rule stays satisfied. +# +# Usage: +# scripts/download-frank-zealot-bonsai2-ptq1.sh [DEST_DIR] [--verify-only] +set -euo pipefail + +DEST="" +VERIFY_ONLY=0 +for arg in "$@"; do + case "$arg" in + --verify-only) VERIFY_ONLY=1 ;; + *) DEST="$arg" ;; + esac +done +DEST="${DEST:-/home/dank/models/Bonsai2-PTQ1_0}" +LINK_SRC="${LINK_SRC:-/home/dank/models/Bonsai2-PQ2_0}" + +REPO="prism-ml/Ternary-Bonsai-2-27B-gguf" +MMPROJ="Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf" +MMPROJ_SIZE=629246976 +MMPROJ_SHA="6807ede61d570bb86ba34b756a0fa109edc33668604de867c6ea6d8f1d631903" + +# hub-name | dest-name | bytes | sha256 (Hub LFS oid) +FILES=( + "Ternary-Bonsai-2-27B-PTQ1_0.gguf|Ternary-Bonsai-2-27B-PTQ1_0.gguf|5946648928|53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3" +) + +mkdir -p "$DEST" +status=0 + +check() { # file size sha -> 0 when byte- and hash-exact + [[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \ + && sha256sum -c --quiet <(printf '%s %s\n' "$3" "$1") +} + +for entry in "${FILES[@]}"; do + IFS='|' read -r hubname destname size sha <<<"$entry" + f="$DEST/$destname" + if check "$f" "$size" "$sha"; then + echo "OK $destname" + continue + fi + if (( VERIFY_ONLY )); then + echo "MISSING $destname" + status=1 + continue + fi + echo "FETCHING $destname (from $REPO)" + tmp="$f.part" + # No resume: `curl -C -` once produced an oversized file on a 31 GiB payload. + # -f/--fail-with-body so an HTTP 401/403 body is never written as success. + curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \ + -o "$tmp" "https://huggingface.co/${REPO}/resolve/main/${hubname}" + mv "$tmp" "$f" + if ! check "$f" "$size" "$sha"; then + echo "FAIL $destname (size $(stat -c%s "$f" 2>/dev/null || echo none) != $size, or sha mismatch)" + status=1 + continue + fi + echo "VERIFIED $destname" +done + +# Vision projector: hardlink the identical file when a verified copy exists. +mm="$DEST/$MMPROJ" +if check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then + echo "OK $MMPROJ" +elif [[ -f "$LINK_SRC/$MMPROJ" ]] && check "$LINK_SRC/$MMPROJ" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then + ln "$LINK_SRC/$MMPROJ" "$mm" && echo "HARDLINKED $MMPROJ from $LINK_SRC" +elif (( VERIFY_ONLY )); then + echo "MISSING $MMPROJ" + status=1 +else + echo "FETCHING $MMPROJ (no verified source copy to link)" + curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \ + -o "$mm.part" "https://huggingface.co/${REPO}/resolve/main/${MMPROJ}" + mv "$mm.part" "$mm" + if ! check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then + echo "FAIL $MMPROJ" + status=1 + else + echo "VERIFIED $MMPROJ" + fi +fi + +if (( status == 0 )); then + echo "DOWNLOAD_DONE $DEST" +else + echo "DOWNLOAD_FAILED $DEST" >&2 +fi +exit "$status" diff --git a/scripts/frank-dragoon.env b/scripts/frank-dragoon.env index dbf2c50..e8444e5 100644 --- a/scripts/frank-dragoon.env +++ b/scripts/frank-dragoon.env @@ -20,9 +20,11 @@ # thinking tokens 1,476 vs 784 mean — the finetune compressed reasoning, the # base thinks ~1.9x longer per answer. # -# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered both -# to 200,704 (RAM + the harness's compaction budget); the rates above are the -# ceiling-262,144 record and the production ledger is the 200,704 one. +# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered ctx to +# 200,704 (RAM + the harness's compaction budget) and, on 2026-10-18, raised the +# pool to 2 x ctx = 401,408 so BOTH slots can carry a full-length request at +# once; the rates above are the ceiling-262,144 record and the production ledger +# is the ctx-200,704 one. # # WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own # card it "will not load in transformers, vLLM, or llama.cpp" — and the v2 @@ -57,15 +59,21 @@ # est-error ≈ 156.2k <= 200,704 — ~9.5× the 3 % error of margin (robust to a # ~25 % under-count). Compaction rides on frank-zealot: input ≈ threshold − # keepRecent(20k) ≈ 112k, inside its 200,704 slot. -# HALOGEN_KV_POOL_POSITIONS=200704 — ONE pool behind HALOGEN_KV_SLOTS=2 -# (ledger form: "2 slots over ONE -position KV pool, each request up to -# HALOGEN_CTX"). With pool == ctx a single MAXIMUM-length request fills the -# whole pool. ~28 KiB/position (FLAGS.md) ⇒ ~5.4 GiB, where 262144 cost 7.2 -# and the 2026-10-06 shape's 524,288 measured 14.6. The engine's own preflight -# formula bills a little more than the reservation because it adds block -# scratch and ~115 MiB/slot of O(1) state — quote the RESERVATION for the RAM -# ledger. FLAGS.md law: "RAM the pool takes is RAM that table loses", so every -# position cut is page cache for the 47.68 GiB n-gram table long prompts read. +# HALOGEN_KV_POOL_POSITIONS=401408 = SLOTS x CTX (2 x 200,704), the top of the +# legal band (the seam enforces ctx <= pool <= slots * ctx). Still ONE pool +# behind HALOGEN_KV_SLOTS=2 (ledger form: "2 slots over ONE -position KV +# pool, each request up to HALOGEN_CTX"), but sized so BOTH slots can hold a +# MAXIMUM-length request at the same time — the parity with frank-zealot's +# 401,408-position pool that Frank ordered on 2026-10-18. The PER-REQUEST +# ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the +# checkpoint's native window is 262,144 and the engine refuses to go past it +# without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position +# (FLAGS.md) ⇒ ~10.8 GiB, where pool == ctx (200,704) cost 5.5, 262,144 cost +# 7.2 and the 2026-10-06 shape's 524,288 measured 14.6. The engine's preflight +# formula bills a little more than the reservation (block scratch + ~115 +# MiB/slot of O(1) state) — quote the RESERVATION. FLAGS.md law: "RAM the pool +# takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this +# much page cache, so the warm prompt-cache ratio is what to re-measure. # HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP # drafter is NOT slot-locked on this engine (batching and speculation # coexist — DFlash-era slots>1 law does NOT carry over; the card's own @@ -74,13 +82,14 @@ # Memory ledger against the box's 121.2 GiB unified (the n-gram table is # mmap'd page cache beside the checkpoint, not a pinned charge — the boot # ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"): -# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~5.4 + -# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 74 GiB engine-side, with the +# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~10.8 + +# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 79.4 GiB engine-side, with the # prompt cache (~1.0 at this ctx; 1.3 was measured at 262144) booked # separately by the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host -# honest; baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ) -# ~10-15. ~30 GiB margin. The boot ledger is the referee — read it after -# deploy and re-size if the engine prints a clamp. +# honest. Against the 121.2 GiB box that leaves ~22 GiB over the reserve for +# baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot +# ledger is the referee — read it after deploy and re-size if the engine +# prints a clamp. # HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy # turns the MTP head off for whole requests after low acceptance and # HALVES decode as a result. Frank accepted the head always-on trade @@ -174,7 +183,7 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf # model_ids for the :7770 backend MUST name this value. HALOGEN_MODEL_ID=frank-dragoon HALOGEN_CTX=200704 -HALOGEN_KV_POOL_POSITIONS=200704 +HALOGEN_KV_POOL_POSITIONS=401408 HALOGEN_KV_SLOTS=2 HALOGEN_HOST_RESERVE_GIB=20 # Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD); diff --git a/scripts/frank-dragoon.service b/scripts/frank-dragoon.service index 134d652..b26284a 100644 --- a/scripts/frank-dragoon.service +++ b/scripts/frank-dragoon.service @@ -1,5 +1,5 @@ [Unit] -Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 200704-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU) +Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU) After=network-online.target Wants=network-online.target diff --git a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md index f635813..5e9bb8d 100644 --- a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md +++ b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md @@ -201,7 +201,9 @@ and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip. boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5 GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all -answered 200. +answered 200. (Later the same day the pool was doubled to 401,408 positions for +slot parity — that reservation is ~10.8 GiB and the engine-side total ~79.4 GiB; +see [[frank-dragoon-flash-server-v2-native]].) **The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/ Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized diff --git a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md index 4ddd6cf..d3d43f5 100644 --- a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md +++ b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md @@ -68,7 +68,8 @@ ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b8680670 `HALOGEN_CTX=200704` (per-request ceiling — the card's 262,144 lowered the same evening, see below), `HALOGEN_KV_SLOTS=2`, -**`HALOGEN_KV_POOL_POSITIONS=200704`** (one pool behind both slots; the card +**`HALOGEN_KV_POOL_POSITIONS=401408`** (one pool behind both slots, raised to +2 × ctx on 2026-10-18 — see *Pool doubled for slot parity* below; the card recommends it equal to ctx, and that equality is where the memory is), `HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`, `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair), diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index f55dbc4..8cbe984 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -19,7 +19,7 @@ and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2 digest serving the NATIVE v2 conversion of the abliterated Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine, MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE -200,704-position KV pool at ctx 200704, vision ON)** and +401,408-position KV pool at ctx 200704, vision ON)** and **llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48 Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled, @@ -54,7 +54,7 @@ gfx1151 iGPU 114 GiB GTT] E[flash-server container gfx1151, digest-pinned image frank-dragoon, port 7770] F[flash-server v2 native abliterated .hgn + ngram table + vision tower -ONE 200704-position KV pool at ctx 200704, 2 slots, MTP always-on, image input] +ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input] G[llama.cpp PrismML fork CUDA build frank-zealot, port 7771] H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj @@ -87,11 +87,14 @@ frank-zealot, port 7771] **hardlinked** beside it — the card publishes only the main file, and the engine's own tools certified the stock companions (`niah` 18/18, `ppl` 3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus). - `HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=200704`: **one pool - behind two slots** (the card ships 262,144/262,144 = 7.2 GiB; Frank took the - RAM the same evening — 5.5 GiB reserved, one maximum-length request filling - the pool), and the freed RAM becomes page cache for the n-gram - table. MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout + `HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**): + **one pool behind two slots**, sized so TWO maximum-length requests coexist — + raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total + in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 = + 7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a + longer window: the per-request ceiling stays 200,704, because 262,144 is the + native wall and the engine refuses more without `HALOGEN_ROPE_YARN`. + MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout 65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone). diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index f9017ee..a7199ce 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -313,10 +313,10 @@ The LLM inference fleet on the two hosts. Full index: - [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2 abliterated** conversion of orcarouter's pinned BF16 - (`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx/pool **200,704** - (`HALOGEN_KV_POOL_POSITIONS=200704`, ~5.4 GiB — one pool behind both slots, - lowered the same evening from the card's 262,144) and the stock peonist - n-gram/vision companions **hardlinked** + (`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a + **401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, ~10.8 GiB — one + pool behind both slots, raised to 2 × ctx on 2026-10-18 so TWO maximum-length + requests coexist) and the stock peonist n-gram/vision companions **hardlinked** in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP always-on, vision ON, no server-side sampler change - [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17 diff --git a/tests/frank_dragoon_unit.bats b/tests/frank_dragoon_unit.bats index 3ce63f9..8fab8e5 100755 --- a/tests/frank_dragoon_unit.bats +++ b/tests/frank_dragoon_unit.bats @@ -82,7 +82,7 @@ good_env() { "FRANK_DRAGOON_TOKENIZER_FILES=tok.json" "HALOGEN_MODEL_ID=frank-dragoon" "HALOGEN_CTX=200704" - "HALOGEN_KV_POOL_POSITIONS=200704" + "HALOGEN_KV_POOL_POSITIONS=401408" "HALOGEN_KV_SLOTS=2" "HALOGEN_SPEC_ADAPT=0" "HALOGEN_MAX_TOK=8192" @@ -155,11 +155,13 @@ good_env() { reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")" # The pool is ONE budget behind the slots (boot ledger form: "2 slots over # ONE -position KV pool, each request up to HALOGEN_CTX"), NOT - # slots x ctx — one window behind both slots IS the memory saving. What - # still binds: the pool must back ONE full-length request (or the + # a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx + # (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be + # resident at once — frank-zealot parity on total in-flight context. What + # still binds: the pool must back ONE full-length request (below that the # advertised per-request ceiling would be unreachable, hence pool >= ctx), - # it must not pretend to be slots x ctx (pool <= slots * ctx), and the - # ledger sum below must stay inside unified memory. + # it must not pretend to be more than slots x ctx (pool <= slots * ctx), and + # the ledger sum below must stay inside unified memory. [ "$pool" -ge "$ctx" ] [ "$pool" -le $(( slots * ctx )) ] kv_gib=$(( pool * 28 * 1024 / 1073741824 )) @@ -274,7 +276,7 @@ $(cat "$TEST_TMP/serve.log")" [[ "$output" == *"--publish 7770:8731"* ]] [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]] [[ "$output" == *"-e HALOGEN_CTX=200704"* ]] - [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=200704"* ]] + [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]] [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]] [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]] [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]