feat(frank-dragoon): double the KV pool to 2 x ctx for slot parity
frank-dragoon's total in-flight context was HALF frank-zealot's: zealot runs one 401,408-position pool split into two 200,704 conversations, while dragoon ran ONE 200,704-position pool behind its two slots, so a single maximum-length request filled it and two long conversations had to stay under ~100k each. HALOGEN_KV_POOL_POSITIONS 200704 -> 401408 = slots x ctx, the top of the seam's legal band (ctx <= pool <= slots * ctx). The PER-REQUEST ceiling does not move: HALOGEN_CTX stays 200704, because the checkpoint's native window is 262,144 and the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN. What this buys is concurrency, not a longer conversation, which is why the harness contextWindow (155648 = ceiling - 45056) is untouched. Cost: ~10.8 GiB of KV where 200,704 cost 5.5 (262,144 cost 7.2; the 2026-10-06 shape's 524,288 measured 14.6), taken from the page cache the 51.2 GB n-gram table reads through — the warm prompt-cache ratio is the number to re-measure after the boot. Engine-side total goes 74.1 -> ~79.4 GiB of 121.2. Also carries scripts/download-frank-zealot-bonsai2-ptq1.sh: the PTQ1_0 candidate payload fetcher (5,946,648,928 B, LFS sha256-verified, mmproj hardlinked), with the publisher's own throughput table in its header — the packing is smaller but the card measures it SLOWER on Ampere, so the swap is decided on our card, not on the card's A100 row.
This commit is contained in:
Frank
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=200704` pool at `HALOGEN_CTX=200704`** (lowered from the card's 262,144 the same evening — 5.5 GiB reserved where 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; pool == ctx so ONE maximum-length request fills it, and the freed RAM is page cache for the n-gram table), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once — the parity with frank-zealot's 401,408 pool on total in-flight context; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`; every pool position is page cache the n-gram table loses), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -396,15 +396,19 @@ request ≈ 156.2k ≤ 200,704); compaction fires at 85 % of the harness window
|
||||
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
|
||||
law does NOT carry over; batching and speculation coexist (the card's
|
||||
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
|
||||
`HALOGEN_KV_POOL_POSITIONS=200704` — **lowered from the card's 262,144 the same
|
||||
evening** (Frank's call: every position costs KV RAM and nothing in the fleet
|
||||
asks for the biggest window) — is **ONE pool behind both slots**, ledger form
|
||||
`2 slots over ONE <pool>-position KV pool, each request up to HALOGEN_CTX`, and
|
||||
at pool == ctx a single MAXIMUM-length request fills it: two maximum-length
|
||||
conversations never coexist (two of ≤100k do). ~28 KiB/position ⇒ the ledger
|
||||
reserves "about 5.5 GiB", where 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6; the
|
||||
engine's preflight FORMULA bills slightly more than the reservation (block
|
||||
scratch + ~115 MiB/slot of O(1) state) — quote the reservation.
|
||||
`HALOGEN_KV_POOL_POSITIONS=401408` — **2 × ctx, RAISED from 200,704 on 2026-10-18**
|
||||
(Frank: dragoon's total in-flight context was HALF frank-zealot's 401,408
|
||||
pool — doubling the POOL is what fixes that, not doubling the window) — is
|
||||
**ONE pool behind both slots**, ledger form `2 slots over ONE <pool>-position KV
|
||||
pool, each request up to HALOGEN_CTX`, and at pool == slots × ctx two
|
||||
MAXIMUM-length conversations DO coexist. The PER-REQUEST ceiling stays
|
||||
`HALOGEN_CTX=200704`: the checkpoint's native window is 262,144 and the engine's
|
||||
own 400 text refuses to exceed it without `HALOGEN_ROPE_YARN`, so "more context"
|
||||
bought here is CONCURRENCY, not a longer single conversation. ~28 KiB/position ⇒
|
||||
the ledger reserves "about 10.8 GiB", where 200,704 cost 5.5, 262,144 cost 7.2
|
||||
and the 2026-10-06 shape's 524,288 measured 14.6; the engine's preflight FORMULA
|
||||
bills slightly more than the reservation (block scratch + ~115 MiB/slot of O(1)
|
||||
state) — quote the reservation.
|
||||
`HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
||||
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
|
||||
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
|
||||
@@ -421,17 +425,18 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content +
|
||||
`image_url` content parts only; ≥256 px probe images or the model answers
|
||||
"video".
|
||||
|
||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger — the
|
||||
200,704 boot printed exactly): weights 62.9 (trunk 62.11 + vision 0.84) + KV
|
||||
5.5 + working 5.7 (`3.8 + 8.4 × MAX_TOK/16384`) = **74.1 GiB in all**, the
|
||||
~1.0 GiB prompt cache booked separately by the ledger, +
|
||||
`HALOGEN_HOST_RESERVE_GIB=20` → ~30 GiB
|
||||
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd page
|
||||
cache beside the checkpoint (ledger: `mapped 51.2 GB, NOT pinned
|
||||
(Pin::None)`) — which is why every pool cut is a double saving: **RAM the pool
|
||||
takes is RAM that table loses**. `HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the
|
||||
v2 loader, so disk-backing the trunk is not a lever on this format. READ the
|
||||
boot ledger and re-size if the engine prints a clamp.
|
||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): weights
|
||||
62.9 (trunk 62.11 + vision 0.84) + KV ~10.8 + working 5.7
|
||||
(`3.8 + 8.4 × MAX_TOK/16384`) ≈ **79.4 GiB in all**, the ~1.0 GiB prompt cache
|
||||
booked separately by the ledger, + `HALOGEN_HOST_RESERVE_GIB=20` → ~22 GiB
|
||||
margin over baseline co-tenancy (the pool-equals-ctx boot printed 74.1 GiB with
|
||||
~30 GiB of margin). The 51.2 GB n-gram table rides as mmap'd page cache beside
|
||||
the checkpoint (ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why
|
||||
every pool position is a double cost: **RAM the pool takes is RAM that table
|
||||
loses**, so the warm prompt-cache ratio is re-measured after a raise.
|
||||
`HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the v2 loader, so disk-backing the
|
||||
trunk is not a lever on this format. READ the boot ledger and re-size if the
|
||||
engine prints a clamp.
|
||||
|
||||
Fetch: `scripts/download-frank-dragoon-v2abl.sh` into
|
||||
`/home/dank/models/halogen-v2-abliterated/` (`-f` + exact byte counts + LFS
|
||||
@@ -1179,7 +1184,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 200,704 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` |
|
||||
| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 401,408 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` |
|
||||
| frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` |
|
||||
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
|
||||
@@ -1097,8 +1097,9 @@ deploy_anubis() {
|
||||
# conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next —
|
||||
# qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident)
|
||||
# + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode
|
||||
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 200,704-position KV pool at the
|
||||
# matching ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
|
||||
# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 401,408-position KV pool (2 × ctx,
|
||||
# raised 2026-10-18 so BOTH slots can hold a maximum-length request) at the
|
||||
# ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a
|
||||
# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern):
|
||||
# relink_unit points /etc/systemd/system/frank-dragoon.service at the repo
|
||||
# file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity
|
||||
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env bash
|
||||
# Candidate payload for frank-zealot: prism-ml/Ternary-Bonsai-2-27B-gguf at
|
||||
# **PTQ1_0** — the DENSE ternary packing (1.75 bits/weight, 5.95 GB) of the same
|
||||
# weights the live PQ2_0 endpoint serves (2.13 bits/weight, 7.21 GB). Public
|
||||
# repo: no auth header, and never a token on the command line (a Bearer token is
|
||||
# world-readable in /proc/<pid>/cmdline).
|
||||
#
|
||||
# WHY THIS ARM EXISTS. The whole argument is footprint: 5,946,648,928 B against
|
||||
# the live 7,206,168,928 B = **1,201 MiB freed on the card**. On this endpoint
|
||||
# KV bills 34,816 B/token (16 full-attention layers x 4 KV heads x 256 dim at
|
||||
# q8_0 K + q8_0 V), so that one gap is ~37,000 tokens of pool per 1,201 MiB —
|
||||
# context is exactly what the freed weight bytes buy.
|
||||
#
|
||||
# WHY IT MIGHT LOSE. The publisher's own throughput table (README, llama-bench,
|
||||
# batch 1, depth 0): PTQ1_0 wins decode only on Ada/L4 (RTX 4090 91.1 vs 81.2,
|
||||
# RTX 6000 Ada 90.4 vs 82.8, L4 32.1 vs 29.8) and LOSES on H100 (86.9 vs 113.9)
|
||||
# and A100 (54.7 vs 73.9); prompt processing "favors PQ2_0 everywhere" (A100
|
||||
# 706 vs 1328). The card text says explicitly that dense-trit unpacking costs
|
||||
# arithmetic and the pack is "slower on Ampere, Hopper, and Blackwell". There is
|
||||
# NO GA102/RTX 3090 row: the Ampere statement generalizes from an A100, which is
|
||||
# instruction-bound at batch 1 with 2 TB/s of HBM, while the 3090 is the
|
||||
# bandwidth-starved member of the family (936 GB/s) — the one regime where
|
||||
# moving 17 % less weight per step can pay. That is an argument, not a
|
||||
# measurement: this script exists so the decision is made on OUR card, against
|
||||
# the PQ2_0 record (76.6 t/s single-stream decode, 1,258 t/s prefill at 35k).
|
||||
#
|
||||
# THE RUNTIME LOCK CARRIES OVER. Both packings are custom GGML tensor types and
|
||||
# stock llama.cpp rejects them ("It rejects PQ2_0 and PTQ1_0 as unknown types"
|
||||
# — the card), so the PrismML fork build stays mandatory. Confirm the loadability
|
||||
# against the installed binary BEFORE pointing any unit at this file.
|
||||
#
|
||||
# The vision projector is the SAME file the PQ2_0 endpoint already uses (Hub LFS
|
||||
# oid 6807ede61d57… matches byte-for-byte), so it is HARDLINKED from the existing
|
||||
# payload dir instead of re-downloaded — a hardlink is a regular file, so the
|
||||
# serve gates' no-symlink rule stays satisfied.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-zealot-bonsai2-ptq1.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
DEST=""
|
||||
VERIFY_ONLY=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--verify-only) VERIFY_ONLY=1 ;;
|
||||
*) DEST="$arg" ;;
|
||||
esac
|
||||
done
|
||||
DEST="${DEST:-/home/dank/models/Bonsai2-PTQ1_0}"
|
||||
LINK_SRC="${LINK_SRC:-/home/dank/models/Bonsai2-PQ2_0}"
|
||||
|
||||
REPO="prism-ml/Ternary-Bonsai-2-27B-gguf"
|
||||
MMPROJ="Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf"
|
||||
MMPROJ_SIZE=629246976
|
||||
MMPROJ_SHA="6807ede61d570bb86ba34b756a0fa109edc33668604de867c6ea6d8f1d631903"
|
||||
|
||||
# hub-name | dest-name | bytes | sha256 (Hub LFS oid)
|
||||
FILES=(
|
||||
"Ternary-Bonsai-2-27B-PTQ1_0.gguf|Ternary-Bonsai-2-27B-PTQ1_0.gguf|5946648928|53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3"
|
||||
)
|
||||
|
||||
mkdir -p "$DEST"
|
||||
status=0
|
||||
|
||||
check() { # file size sha -> 0 when byte- and hash-exact
|
||||
[[ -f "$1" ]] && [[ "$(stat -c%s "$1")" == "$2" ]] \
|
||||
&& sha256sum -c --quiet <(printf '%s %s\n' "$3" "$1")
|
||||
}
|
||||
|
||||
for entry in "${FILES[@]}"; do
|
||||
IFS='|' read -r hubname destname size sha <<<"$entry"
|
||||
f="$DEST/$destname"
|
||||
if check "$f" "$size" "$sha"; then
|
||||
echo "OK $destname"
|
||||
continue
|
||||
fi
|
||||
if (( VERIFY_ONLY )); then
|
||||
echo "MISSING $destname"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "FETCHING $destname (from $REPO)"
|
||||
tmp="$f.part"
|
||||
# No resume: `curl -C -` once produced an oversized file on a 31 GiB payload.
|
||||
# -f/--fail-with-body so an HTTP 401/403 body is never written as success.
|
||||
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \
|
||||
-o "$tmp" "https://huggingface.co/${REPO}/resolve/main/${hubname}"
|
||||
mv "$tmp" "$f"
|
||||
if ! check "$f" "$size" "$sha"; then
|
||||
echo "FAIL $destname (size $(stat -c%s "$f" 2>/dev/null || echo none) != $size, or sha mismatch)"
|
||||
status=1
|
||||
continue
|
||||
fi
|
||||
echo "VERIFIED $destname"
|
||||
done
|
||||
|
||||
# Vision projector: hardlink the identical file when a verified copy exists.
|
||||
mm="$DEST/$MMPROJ"
|
||||
if check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
||||
echo "OK $MMPROJ"
|
||||
elif [[ -f "$LINK_SRC/$MMPROJ" ]] && check "$LINK_SRC/$MMPROJ" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
||||
ln "$LINK_SRC/$MMPROJ" "$mm" && echo "HARDLINKED $MMPROJ from $LINK_SRC"
|
||||
elif (( VERIFY_ONLY )); then
|
||||
echo "MISSING $MMPROJ"
|
||||
status=1
|
||||
else
|
||||
echo "FETCHING $MMPROJ (no verified source copy to link)"
|
||||
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 \
|
||||
-o "$mm.part" "https://huggingface.co/${REPO}/resolve/main/${MMPROJ}"
|
||||
mv "$mm.part" "$mm"
|
||||
if ! check "$mm" "$MMPROJ_SIZE" "$MMPROJ_SHA"; then
|
||||
echo "FAIL $MMPROJ"
|
||||
status=1
|
||||
else
|
||||
echo "VERIFIED $MMPROJ"
|
||||
fi
|
||||
fi
|
||||
|
||||
if (( status == 0 )); then
|
||||
echo "DOWNLOAD_DONE $DEST"
|
||||
else
|
||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
+27
-18
@@ -20,9 +20,11 @@
|
||||
# thinking tokens 1,476 vs 784 mean — the finetune compressed reasoning, the
|
||||
# base thinks ~1.9x longer per answer.
|
||||
#
|
||||
# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered both
|
||||
# to 200,704 (RAM + the harness's compaction budget); the rates above are the
|
||||
# ceiling-262,144 record and the production ledger is the 200,704 one.
|
||||
# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered ctx to
|
||||
# 200,704 (RAM + the harness's compaction budget) and, on 2026-10-18, raised the
|
||||
# pool to 2 x ctx = 401,408 so BOTH slots can carry a full-length request at
|
||||
# once; the rates above are the ceiling-262,144 record and the production ledger
|
||||
# is the ctx-200,704 one.
|
||||
#
|
||||
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
|
||||
# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
|
||||
@@ -57,15 +59,21 @@
|
||||
# est-error ≈ 156.2k <= 200,704 — ~9.5× the 3 % error of margin (robust to a
|
||||
# ~25 % under-count). Compaction rides on frank-zealot: input ≈ threshold −
|
||||
# keepRecent(20k) ≈ 112k, inside its 200,704 slot.
|
||||
# HALOGEN_KV_POOL_POSITIONS=200704 — ONE pool behind HALOGEN_KV_SLOTS=2
|
||||
# (ledger form: "2 slots over ONE <pool>-position KV pool, each request up to
|
||||
# HALOGEN_CTX"). With pool == ctx a single MAXIMUM-length request fills the
|
||||
# whole pool. ~28 KiB/position (FLAGS.md) ⇒ ~5.4 GiB, where 262144 cost 7.2
|
||||
# and the 2026-10-06 shape's 524,288 measured 14.6. The engine's own preflight
|
||||
# formula bills a little more than the reservation because it adds block
|
||||
# scratch and ~115 MiB/slot of O(1) state — quote the RESERVATION for the RAM
|
||||
# ledger. FLAGS.md law: "RAM the pool takes is RAM that table loses", so every
|
||||
# position cut is page cache for the 47.68 GiB n-gram table long prompts read.
|
||||
# HALOGEN_KV_POOL_POSITIONS=401408 = SLOTS x CTX (2 x 200,704), the top of the
|
||||
# legal band (the seam enforces ctx <= pool <= slots * ctx). Still ONE pool
|
||||
# behind HALOGEN_KV_SLOTS=2 (ledger form: "2 slots over ONE <pool>-position KV
|
||||
# pool, each request up to HALOGEN_CTX"), but sized so BOTH slots can hold a
|
||||
# MAXIMUM-length request at the same time — the parity with frank-zealot's
|
||||
# 401,408-position pool that Frank ordered on 2026-10-18. The PER-REQUEST
|
||||
# ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the
|
||||
# checkpoint's native window is 262,144 and the engine refuses to go past it
|
||||
# without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position
|
||||
# (FLAGS.md) ⇒ ~10.8 GiB, where pool == ctx (200,704) cost 5.5, 262,144 cost
|
||||
# 7.2 and the 2026-10-06 shape's 524,288 measured 14.6. The engine's preflight
|
||||
# formula bills a little more than the reservation (block scratch + ~115
|
||||
# MiB/slot of O(1) state) — quote the RESERVATION. FLAGS.md law: "RAM the pool
|
||||
# takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this
|
||||
# much page cache, so the warm prompt-cache ratio is what to re-measure.
|
||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||
# drafter is NOT slot-locked on this engine (batching and speculation
|
||||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||||
@@ -74,13 +82,14 @@
|
||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the boot
|
||||
# ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"):
|
||||
# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~5.4 +
|
||||
# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 74 GiB engine-side, with the
|
||||
# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~10.8 +
|
||||
# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 79.4 GiB engine-side, with the
|
||||
# prompt cache (~1.0 at this ctx; 1.3 was measured at 262144) booked
|
||||
# separately by the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
||||
# honest; baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ)
|
||||
# ~10-15. ~30 GiB margin. The boot ledger is the referee — read it after
|
||||
# deploy and re-size if the engine prints a clamp.
|
||||
# honest. Against the 121.2 GiB box that leaves ~22 GiB over the reserve for
|
||||
# baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot
|
||||
# ledger is the referee — read it after deploy and re-size if the engine
|
||||
# prints a clamp.
|
||||
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
||||
# turns the MTP head off for whole requests after low acceptance and
|
||||
# HALVES decode as a result. Frank accepted the head always-on trade
|
||||
@@ -174,7 +183,7 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf
|
||||
# model_ids for the :7770 backend MUST name this value.
|
||||
HALOGEN_MODEL_ID=frank-dragoon
|
||||
HALOGEN_CTX=200704
|
||||
HALOGEN_KV_POOL_POSITIONS=200704
|
||||
HALOGEN_KV_POOL_POSITIONS=401408
|
||||
HALOGEN_KV_SLOTS=2
|
||||
HALOGEN_HOST_RESERVE_GIB=20
|
||||
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 200704-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||
Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 401408-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
|
||||
@@ -201,7 +201,9 @@ and PerfOpt is now a bonus nobody is waiting for rather than the reason to flip.
|
||||
boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) + 5.5
|
||||
GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP
|
||||
depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all
|
||||
answered 200.
|
||||
answered 200. (Later the same day the pool was doubled to 401,408 positions for
|
||||
slot parity — that reservation is ~10.8 GiB and the engine-side total ~79.4 GiB;
|
||||
see [[frank-dragoon-flash-server-v2-native]].)
|
||||
|
||||
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
|
||||
Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized
|
||||
|
||||
@@ -68,7 +68,8 @@ ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b8680670
|
||||
|
||||
`HALOGEN_CTX=200704` (per-request ceiling — the card's 262,144 lowered the
|
||||
same evening, see below), `HALOGEN_KV_SLOTS=2`,
|
||||
**`HALOGEN_KV_POOL_POSITIONS=200704`** (one pool behind both slots; the card
|
||||
**`HALOGEN_KV_POOL_POSITIONS=401408`** (one pool behind both slots, raised to
|
||||
2 × ctx on 2026-10-18 — see *Pool doubled for slot parity* below; the card
|
||||
recommends it equal to ctx, and that equality is where the memory is),
|
||||
`HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`,
|
||||
`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair),
|
||||
|
||||
@@ -19,7 +19,7 @@ and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
digest serving the NATIVE v2 conversion of the abliterated
|
||||
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
200,704-position KV pool at ctx 200704, vision ON)** and
|
||||
401,408-position KV pool at ctx 200704, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
|
||||
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
|
||||
@@ -54,7 +54,7 @@ gfx1151 iGPU 114 GiB GTT]
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-dragoon, port 7770]
|
||||
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
||||
ONE 200704-position KV pool at ctx 200704, 2 slots, MTP always-on, image input]
|
||||
ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input]
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj
|
||||
@@ -87,11 +87,14 @@ frank-zealot, port 7771]
|
||||
**hardlinked** beside it — the card publishes only the main file, and the
|
||||
engine's own tools certified the stock companions (`niah` 18/18, `ppl`
|
||||
3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus).
|
||||
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=200704`: **one pool
|
||||
behind two slots** (the card ships 262,144/262,144 = 7.2 GiB; Frank took the
|
||||
RAM the same evening — 5.5 GiB reserved, one maximum-length request filling
|
||||
the pool), and the freed RAM becomes page cache for the n-gram
|
||||
table. MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
|
||||
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
||||
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
||||
in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 =
|
||||
7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a
|
||||
longer window: the per-request ceiling stays 200,704, because 262,144 is the
|
||||
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
||||
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
|
||||
boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18,
|
||||
PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone).
|
||||
|
||||
@@ -313,10 +313,10 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook
|
||||
since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2
|
||||
abliterated** conversion of orcarouter's pinned BF16
|
||||
(`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx/pool **200,704**
|
||||
(`HALOGEN_KV_POOL_POSITIONS=200704`, ~5.4 GiB — one pool behind both slots,
|
||||
lowered the same evening from the card's 262,144) and the stock peonist
|
||||
n-gram/vision companions **hardlinked**
|
||||
(`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a
|
||||
**401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, ~10.8 GiB — one
|
||||
pool behind both slots, raised to 2 × ctx on 2026-10-18 so TWO maximum-length
|
||||
requests coexist) and the stock peonist n-gram/vision companions **hardlinked**
|
||||
in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP
|
||||
always-on, vision ON, no server-side sampler change
|
||||
- [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17
|
||||
|
||||
@@ -82,7 +82,7 @@ good_env() {
|
||||
"FRANK_DRAGOON_TOKENIZER_FILES=tok.json"
|
||||
"HALOGEN_MODEL_ID=frank-dragoon"
|
||||
"HALOGEN_CTX=200704"
|
||||
"HALOGEN_KV_POOL_POSITIONS=200704"
|
||||
"HALOGEN_KV_POOL_POSITIONS=401408"
|
||||
"HALOGEN_KV_SLOTS=2"
|
||||
"HALOGEN_SPEC_ADAPT=0"
|
||||
"HALOGEN_MAX_TOK=8192"
|
||||
@@ -155,11 +155,13 @@ good_env() {
|
||||
reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")"
|
||||
# The pool is ONE budget behind the slots (boot ledger form: "2 slots over
|
||||
# ONE <pool>-position KV pool, each request up to HALOGEN_CTX"), NOT
|
||||
# slots x ctx — one window behind both slots IS the memory saving. What
|
||||
# still binds: the pool must back ONE full-length request (or the
|
||||
# a per-slot private pool. Since 2026-10-18 it is sized pool = slots * ctx
|
||||
# (401,408 = 2 x 200,704), so two MAXIMUM-length conversations can be
|
||||
# resident at once — frank-zealot parity on total in-flight context. What
|
||||
# still binds: the pool must back ONE full-length request (below that the
|
||||
# advertised per-request ceiling would be unreachable, hence pool >= ctx),
|
||||
# it must not pretend to be slots x ctx (pool <= slots * ctx), and the
|
||||
# ledger sum below must stay inside unified memory.
|
||||
# it must not pretend to be more than slots x ctx (pool <= slots * ctx), and
|
||||
# the ledger sum below must stay inside unified memory.
|
||||
[ "$pool" -ge "$ctx" ]
|
||||
[ "$pool" -le $(( slots * ctx )) ]
|
||||
kv_gib=$(( pool * 28 * 1024 / 1073741824 ))
|
||||
@@ -274,7 +276,7 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
[[ "$output" == *"--publish 7770:8731"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_CTX=200704"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=200704"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=401408"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]]
|
||||
[[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]]
|
||||
|
||||
Reference in New Issue
Block a user