diff --git a/AGENTS.md b/AGENTS.md index 792288a..efb74a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 | -| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=262144` pool** (the card's recommendation: the ledger's 7.2 GiB where the old 524,288 pool measured 14.6 — per-request ceiling unchanged at 262,144, two maximum-length conversations no longer coexist, and the freed RAM is page cache for the n-gram table), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| +| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=200704` pool at `HALOGEN_CTX=200704`** (lowered from the card's 262,144 the same evening — ~5.4 GiB where 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; pool == ctx so ONE maximum-length request fills it, and the freed RAM is page cache for the n-gram table), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| | Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -380,25 +380,29 @@ endpoint). with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive policy disables the MTP head for whole requests after low acceptance and HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance -for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400 -naming the limit, never truncation) — RAISED from 200192 on 2026-10-07 -under the 400-zone law: the harness clamps max_tokens to +for it (accepted). `HALOGEN_CTX=200704` is the PER-REQUEST hard ceiling (400 +naming the limit, never truncation) — the checkpoint's native window is +262,144 and was served until 2026-10-17, when Frank lowered it for RAM. The +**400-zone law is invariant to the value**: the harness clamps max_tokens to `min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured: est 152,938 → server 157,388, request 43,158 vs room 42,804), so a harness window == engine ceiling -made every request over ~146k est a guaranteed 400. `models.yml` -`contextWindow: 217088` leaves the 45k gap that absorbs the estimate -error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of -the harness window = 184,524 est ≈ 190k measured (Frank's 180k target). +makes every long request a guaranteed 400. `models.yml` +`contextWindow: 155648` (= ceiling − 45056) keeps that gap (worst-case +request ≈ 156.2k ≤ 200,704); compaction fires at 85 % of the harness window += 132,301 est ≈ 136k measured (the 180k target died with the 262,144 window). `HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter" law does NOT carry over; batching and speculation coexist (the card's quickstart ships 2 slots with the head), so 2 is concurrent capacity. -`HALOGEN_KV_POOL_POSITIONS=262144` (the card's recommendation and the memory -saving of the 2026-10-17 swap; ~28 KiB/position ⇒ 7.2 GiB) is **ONE pool -behind both slots** — ledger `2 slots over ONE 262144-position KV pool, each -request up to 262144` — so the per-request ceiling is untouched while two -MAXIMUM-length conversations no longer coexist (the old 524,288 = 2 × ctx -spent the measured 14.6 GiB and allowed it). +`HALOGEN_KV_POOL_POSITIONS=200704` — **lowered from the card's 262,144 the same +evening** (Frank's call: every position costs KV RAM and nothing in the fleet +asks for the biggest window) — is **ONE pool behind both slots**, ledger form +`2 slots over ONE -position KV pool, each request up to HALOGEN_CTX`, and +at pool == ctx a single MAXIMUM-length request fills it: two maximum-length +conversations never coexist (two of ≤100k do). ~28 KiB/position ⇒ ~5.4 GiB, +where 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6; the +engine's preflight FORMULA bills slightly more than the reservation (block +scratch + ~115 MiB/slot of O(1) state) — quote the reservation. `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table). @@ -416,15 +420,15 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content + "video". **Memory ledger** (121.2 GiB unified; the referee is the boot ledger): -weights ~62.9 (trunk 62.11 + vision 0.84) + KV 7.2 + arena ~8 -(`3.8 + 8.4 × MAX_TOK/16384`) ≈ 78 GiB engine-side + -`HALOGEN_HOST_RESERVE_GIB=20` → ~25 GiB margin over baseline co-tenancy. -The 51.2 GB n-gram table rides as mmap'd page cache beside the checkpoint -(ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why the pool -cut is a double saving: **RAM the pool takes is RAM that table loses**. -`HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the v2 loader, so disk-backing -the trunk is not a lever on this format. READ the boot ledger and re-size -if the engine prints a clamp. +weights ~62.9 (trunk 62.11 + vision 0.84) + KV ~5.4 + working ~5.7 +(`3.8 + 8.4 × MAX_TOK/16384`) ≈ 74 GiB engine-side, the ~1.0 GiB prompt cache +booked separately by the ledger, + `HALOGEN_HOST_RESERVE_GIB=20` → ~30 GiB +margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd page +cache beside the checkpoint (ledger: `mapped 51.2 GB, NOT pinned +(Pin::None)`) — which is why every pool cut is a double saving: **RAM the pool +takes is RAM that table loses**. `HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the +v2 loader, so disk-backing the trunk is not a lever on this format. READ the +boot ledger and re-size if the engine prints a clamp. Fetch: `scripts/download-frank-dragoon-v2abl.sh` into `/home/dank/models/halogen-v2-abliterated/` (`-f` + exact byte counts + LFS @@ -1156,7 +1160,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 262,144 pool, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` | +| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 200,704 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` | | frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` | | frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 8592335..294a429 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1097,9 +1097,9 @@ deploy_anubis() { # conversion (2026-10-11) of the abliterated Qwen3.8-Flash-Next — # qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B, ~62 GiB resident) # + the REQUIRED n-gram lookup table + the vision tower — MTP spec decode -# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 262,144-position KV pool, image -# input — on the Strix Halo iGPU. It stays a ROOT unit because the serve -# script drives `podman run` directly (the frank-archon pattern): +# (HALOGEN_SPEC_ADAPT=0), 2 slots over ONE 200,704-position KV pool at the +# matching ctx 200704 ceiling, image input — on the Strix Halo iGPU. It stays a +# ROOT unit because the serve script drives `podman run` directly (the frank-archon pattern): # relink_unit points /etc/systemd/system/frank-dragoon.service at the repo # file, boot persistence is that unit's [Install], and LimitMEMLOCK=infinity # is fleet parity. 2026-10-17 cutover from the Swift1.5-finetune shape — diff --git a/scripts/frank-dragoon.env b/scripts/frank-dragoon.env index 97410c5..dbf2c50 100644 --- a/scripts/frank-dragoon.env +++ b/scripts/frank-dragoon.env @@ -20,6 +20,10 @@ # thinking tokens 1,476 vs 784 mean — the finetune compressed reasoning, the # base thinks ~1.9x longer per answer. # +# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered both +# to 200,704 (RAM + the harness's compaction budget); the rates above are the +# ceiling-262,144 record and the production ledger is the 200,704 one. +# # WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own # card it "will not load in transformers, vLLM, or llama.cpp" — and the v2 # .hgn loads ONLY in the flash-server line (the old halogen engine is for @@ -29,50 +33,54 @@ # the censored 27B (905 mean), the ABLITERATED base (refusals should NOT # return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint — # red-PNG probe verified on the card; the 27B engine ignored its encoder), -# tool calls verified, and the native 262,144 context. +# tool calls verified, and the native 262,144 context — served at 200,704 by +# choice from 2026-10-17 (see SHAPE: the ceiling is a RAM decision, not a +# capability limit). # # SHAPE — the knobs that are NOT free, with the arithmetic: -# HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard -# ceiling — a prompt over it is a 400 naming the limit, never a silent -# truncation. RAISED from 200192 on 2026-10-07: the harness clamps -# max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts -# the rendered prompt by ~3 % on code/agent traffic, and compaction only -# fires at 85 % — so with window == engine ceiling, every request over -# ~146k est (65-85 % of the window) was a guaranteed 400. Measured -# 2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804 -# — over by 354, margin 4,096 < the 4,450 under-count. -# Harness contract (models.yml): contextWindow 217088, maxTokens 45056. -# Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k -# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 % -# est-error ≈ 219k <= 262144 — the 45k gap between harness window and -# engine ceiling absorbs the estimate error with ~7x margin (robust to a -# 10 % under-count). Compaction rides on frank-zealot: input ≈ -# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot. -# HALOGEN_KV_POOL_POSITIONS=262144 is the card's recommended pool and the -# memory saving of this swap. One budget behind HALOGEN_KV_SLOTS=2 (boot -# ledger: "2 slots over ONE 262144-position KV pool, each request up to -# 262144"), so the per-request ceiling stays HALOGEN_CTX while two -# MAXIMUM-length conversations can no longer coexist — two moderate ones -# can. ~28 KiB/position (FLAGS.md) => 7.2 GiB, and the ledger said exactly -# that: "reserving the KV pool, 262144 positions (about 7.2 GiB)". The -# 2026-10-06 shape's 524,288 cost 14.6 GiB; FLAGS.md law — "RAM the pool -# takes is RAM that table loses" — makes the 7.4 GiB handed back page -# cache for the 47.7 GiB n-gram table, which is what long prompts read. +# HALOGEN_CTX=200704 is the PER-REQUEST ceiling and the per-conversation +# hard limit — a prompt over it is a 400 naming the limit, never a silent +# truncation. LOWERED from 262,144 on 2026-10-17 at Frank's direction (the +# checkpoint's native window is 262,144 and the engine serves anything +# below it): every position costs KV pool RAM and shrinks the prompt-cache +# entry (111 MiB at 262,144, ~85 MiB here). The 400-zone law is INVARIANT +# to the value — it says the harness window must sit ~45k UNDER the engine +# ceiling, because the harness clamps max_tokens to contextWindow − est − +# 4096, its chars/4 est UNDER-counts the rendered prompt by ~3 % on +# code/agent traffic, and compaction only fires at 85 % (measured 2026-10-07 +# at the old 200,192 ceiling: est 152,938, server 157,388, over by 354). +# Harness contract (models.yml): contextWindow 155648 (= ceiling − 45056, +# the gap that absorbs the estimate error), maxTokens 45056. +# Compaction fires at 85 % = 132,301 est ≈ 136k measured — the 180k target +# of the 262,144 era is unreachable at this ceiling: the window itself is +# 155,648. Worst-case request: prompt + budget <= 155648 − 4096 + 3 % +# est-error ≈ 156.2k <= 200,704 — ~9.5× the 3 % error of margin (robust to a +# ~25 % under-count). Compaction rides on frank-zealot: input ≈ threshold − +# keepRecent(20k) ≈ 112k, inside its 200,704 slot. +# HALOGEN_KV_POOL_POSITIONS=200704 — ONE pool behind HALOGEN_KV_SLOTS=2 +# (ledger form: "2 slots over ONE -position KV pool, each request up to +# HALOGEN_CTX"). With pool == ctx a single MAXIMUM-length request fills the +# whole pool. ~28 KiB/position (FLAGS.md) ⇒ ~5.4 GiB, where 262144 cost 7.2 +# and the 2026-10-06 shape's 524,288 measured 14.6. The engine's own preflight +# formula bills a little more than the reservation because it adds block +# scratch and ~115 MiB/slot of O(1) state — quote the RESERVATION for the RAM +# ledger. FLAGS.md law: "RAM the pool takes is RAM that table loses", so every +# position cut is page cache for the 47.68 GiB n-gram table long prompts read. # HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP # drafter is NOT slot-locked on this engine (batching and speculation # coexist — DFlash-era slots>1 law does NOT carry over; the card's own # quickstart ships 2 slots WITH the head), so the second slot is # concurrent capacity, not a speculation tax. # Memory ledger against the box's 121.2 GiB unified (the n-gram table is -# mmap'd page cache beside the checkpoint, not a pinned charge — the -# card's own quickstart arithmetic does not bill it, and the boot ledger -# agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"): -# weights ~62 GiB resident + KV 7.2 + arena (HALOGEN_MAX_TOK=8192 => -# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2 -# = ~79 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host +# mmap'd page cache beside the checkpoint, not a pinned charge — the boot +# ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"): +# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~5.4 + +# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 74 GiB engine-side, with the +# prompt cache (~1.0 at this ctx; 1.3 was measured at 262144) booked +# separately by the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host # honest; baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ) -# ~10-15. ~25 GiB margin. The boot ledger is the referee — read it -# after deploy and re-size if the engine prints a clamp. +# ~10-15. ~30 GiB margin. The boot ledger is the referee — read it after +# deploy and re-size if the engine prints a clamp. # HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy # turns the MTP head off for whole requests after low acceptance and # HALVES decode as a result. Frank accepted the head always-on trade @@ -99,11 +107,11 @@ # so the models.yml fold (high->xhigh) still applies. # * Vision: OpenAI image_url content parts only (the fleet's standard # contract). >=256 px probe images or the model answers "video". -# * TWO maximum-length conversations no longer share the endpoint: the KV -# pool is one 262,144-position budget behind 2 slots. The PER-REQUEST -# ceiling is unchanged (HALOGEN_CTX), so the harness contract below -# (contextWindow 217088) does not move; only concurrency at full depth -# does, and this endpoint carries the slow + security-reviewer roles. +# * Concurrency at depth is tight now: the KV pool is ONE 200,704-position +# budget behind 2 slots — exactly one maximum-length request fills it. Two +# conversations coexist only while they stay under ~100k each. The +# PER-REQUEST ceiling moved WITH the pool (262144 -> 200704), so the harness +# contract moves too: contextWindow 155648 = ceiling − 45k, never higher. # * response_format json_object/json_schema unsupported in this line # (no constrained decoding) — unchanged from the 27B engine. # * The engine's token protocol has no auth and binds loopback INSIDE @@ -165,8 +173,8 @@ FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_conf # scripts and this file keep the frank-dragoon name. OpenWebUI api_configs # model_ids for the :7770 backend MUST name this value. HALOGEN_MODEL_ID=frank-dragoon -HALOGEN_CTX=262144 -HALOGEN_KV_POOL_POSITIONS=262144 +HALOGEN_CTX=200704 +HALOGEN_KV_POOL_POSITIONS=200704 HALOGEN_KV_SLOTS=2 HALOGEN_HOST_RESERVE_GIB=20 # Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD); diff --git a/scripts/frank-dragoon.service b/scripts/frank-dragoon.service index 4e24395..134d652 100644 --- a/scripts/frank-dragoon.service +++ b/scripts/frank-dragoon.service @@ -1,5 +1,5 @@ [Unit] -Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 262144-position pool, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU) +Description=halogen-flash-server container (Quat3rnion NATIVE v2 abliterated Qwen3.8-Flash-Next, MTP spec decode, 2 slots over ONE 200704-position pool at ctx 200704, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU) After=network-online.target Wants=network-online.target diff --git a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md index e265b17..11c3081 100644 --- a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md +++ b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md @@ -66,9 +66,11 @@ ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b8680670 ## Shape — the card's settings, and the three deliberate deviations -`HALOGEN_CTX=262144` (per-request ceiling), `HALOGEN_KV_SLOTS=2`, -**`HALOGEN_KV_POOL_POSITIONS=262144`** (the card's recommendation, and the -memory saving), `HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`, +`HALOGEN_CTX=200704` (per-request ceiling — the card's 262,144 lowered the +same evening, see below), `HALOGEN_KV_SLOTS=2`, +**`HALOGEN_KV_POOL_POSITIONS=200704`** (one pool behind both slots; the card +recommends it equal to ctx, and that equality is where the memory is), +`HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`, `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair), sampler `1.0 / 0.95 / 20`, `HALOGEN_WEIGHTS_LOCK=1`, `HALOGEN_HOST_RESERVE_GIB=20`. @@ -81,24 +83,28 @@ Deviations from the card, each on purpose: (the adaptive default disables the drafter per request and halves decode). - the sampler values are peonist's engine recipe; Quat3rnion publishes none. -**The pool is one budget, not a partition.** Boot ledger: `listening on -127.0.0.1:8730, 2 slots over ONE 262144-position KV pool, each request up to -262144`. The per-request ceiling is unchanged, so the harness contract -(`contextWindow: 217088`) does not move — what changed is that **two -maximum-length conversations can no longer share the endpoint**. That is -acceptable here because frank-dragoon carries the `slow` (medium-thinking) -role and the `security-reviewer` subagent, not the interactive workhorse -(that is frank-archon). +**The pool is one budget, not a partition.** Boot ledger form: `listening on +127.0.0.1:8730, 2 slots over ONE -position KV pool, each request up to +`. Both numbers were lowered to **200,704** the same evening +(the card ships 262,144; Frank took the RAM — nothing else in the fleet asks +for the bigger window), so `models.yml` `contextWindow` moved to **155,648** += ceiling − 45k with it. With pool == ctx, one maximum-length request fills +the pool: **two long conversations can no longer share the endpoint** (two of +≤100k can). That is acceptable here because frank-dragoon carries the `slow` +(medium-thinking) role and the `security-reviewer` subagent, not the +interactive workhorse (that is frank-archon). -**Memory ledger** (121.2 GiB unified), from the boot ledger of the live -candidate: `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk, +**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot +printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk, `0.84 GiB` vision tower, `reserving the KV pool, 262144 positions (about 7.2 GiB)`, and the engine's own sum: `memory: 62.9 GiB of weights locked in -RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …`. The n-gram -table rides as page cache — `mapped 51.2 GB, NOT pinned (Pin::None)` — which -is exactly why cutting the pool is a *double* saving. FLAGS.md law: **RAM the -pool takes is RAM that table loses**; the 7.4 GiB the smaller pool hands back -is page cache for the 47.68 GiB table that long prompts read. +RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …, 75.8 GiB in +all`. After the same-evening ctx/pool cut to 200,704 the reservation is ~5.4 +GiB and the total ~74. The n-gram table rides as page cache — `mapped 51.2 GB, +NOT pinned (Pin::None)` — which is exactly why every pool cut is a *double* +saving. FLAGS.md law: **RAM the pool takes is RAM that table loses**; the ~9 +GiB the two cuts together handed back is page cache for the 47.68 GiB table +that long prompts read. `HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk stays pinned. Disk-backing the weights is not a lever on this format. diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 2c07100..288bea6 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -19,7 +19,7 @@ and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2 digest serving the NATIVE v2 conversion of the abliterated Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine, MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE -262,144-position KV pool, vision ON)** and +200,704-position KV pool at ctx 200704, vision ON)** and **llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48 Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled, @@ -54,7 +54,7 @@ gfx1151 iGPU 114 GiB GTT] E[flash-server container gfx1151, digest-pinned image frank-dragoon, port 7770] F[flash-server v2 native abliterated .hgn + ngram table + vision tower -ONE 262144-position KV pool, 2 slots, MTP always-on, image input] +ONE 200704-position KV pool at ctx 200704, 2 slots, MTP always-on, image input] G[llama.cpp PrismML fork CUDA build frank-zealot, port 7771] H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj @@ -87,9 +87,10 @@ frank-zealot, port 7771] **hardlinked** beside it — the card publishes only the main file, and the engine's own tools certified the stock companions (`niah` 18/18, `ppl` 3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus). - Card-recommended `HALOGEN_KV_POOL_POSITIONS=262144`: **one pool behind two - slots** — 7.2 GiB where the old shape spent 14.6, the per-request ceiling - unchanged at 262,144, and the freed RAM becomes page cache for the n-gram + `HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=200704`: **one pool + behind two slots** (the card ships 262,144/262,144 = 7.2 GiB; Frank took the + RAM the same evening — ~5.4 GiB, one maximum-length request filling the + pool), and the freed RAM becomes page cache for the n-gram table. MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout 65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index b3bae9e..fee777e 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -313,9 +313,10 @@ The LLM inference fleet on the two hosts. Full index: - [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2 abliterated** conversion of orcarouter's pinned BF16 - (`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) with the card's - memory-saving `HALOGEN_KV_POOL_POSITIONS=262144` (7.2 GiB, one pool behind - both slots) and the stock peonist n-gram/vision companions **hardlinked** + (`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx/pool **200,704** + (`HALOGEN_KV_POOL_POSITIONS=200704`, ~5.4 GiB — one pool behind both slots, + lowered the same evening from the card's 262,144) and the stock peonist + n-gram/vision companions **hardlinked** in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP always-on, vision ON, no server-side sampler change - [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17 diff --git a/tests/frank_dragoon_unit.bats b/tests/frank_dragoon_unit.bats index 53d3a14..3ce63f9 100755 --- a/tests/frank_dragoon_unit.bats +++ b/tests/frank_dragoon_unit.bats @@ -3,7 +3,7 @@ # of the abliterated Qwen3.8-Flash-Next (a .hgn flash-server payload) on the # Strix Halo iGPU through the halogen-flash-server CONTAINER (0.16.2, # digest-pinned), driven by the root unit's serve script — 2 slots over ONE -# 262,144-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0) +# 200,704-position KV pool with the MTP head always on (HALOGEN_SPEC_ADAPT=0) # and the vision tower mounted. 2026-10-17 cutover from the Swift1.5-finetune # shape, which itself cut over the halogen 27B-dense shape on 2026-10-06. # @@ -81,8 +81,8 @@ good_env() { "FRANK_DRAGOON_VISION_BYTES=0" "FRANK_DRAGOON_TOKENIZER_FILES=tok.json" "HALOGEN_MODEL_ID=frank-dragoon" - "HALOGEN_CTX=262144" - "HALOGEN_KV_POOL_POSITIONS=262144" + "HALOGEN_CTX=200704" + "HALOGEN_KV_POOL_POSITIONS=200704" "HALOGEN_KV_SLOTS=2" "HALOGEN_SPEC_ADAPT=0" "HALOGEN_MAX_TOK=8192" @@ -153,13 +153,13 @@ good_env() { pool="$(sed -nE 's/^HALOGEN_KV_POOL_POSITIONS=([0-9]+)$/\1/p' "$ENV_FILE")" mt="$(sed -nE 's/^HALOGEN_MAX_TOK=([0-9]+)$/\1/p' "$ENV_FILE")" reserve="$(sed -nE 's/^HALOGEN_HOST_RESERVE_GIB=([0-9]+)$/\1/p' "$ENV_FILE")" - # The pool is ONE budget behind the slots (boot ledger: "2 slots over ONE - # 262144-position KV pool, each request up to 262144"), NOT slots x ctx: - # the card ships it at one window and that IS the memory saving of this - # payload. What still binds is that the pool back ONE full-length - # request, or the advertised per-request ceiling would be unreachable — - # and that it stay inside unified memory, which the ledger sum below - # checks. + # The pool is ONE budget behind the slots (boot ledger form: "2 slots over + # ONE -position KV pool, each request up to HALOGEN_CTX"), NOT + # slots x ctx — one window behind both slots IS the memory saving. What + # still binds: the pool must back ONE full-length request (or the + # advertised per-request ceiling would be unreachable, hence pool >= ctx), + # it must not pretend to be slots x ctx (pool <= slots * ctx), and the + # ledger sum below must stay inside unified memory. [ "$pool" -ge "$ctx" ] [ "$pool" -le $(( slots * ctx )) ] kv_gib=$(( pool * 28 * 1024 / 1073741824 )) @@ -273,8 +273,8 @@ $(cat "$TEST_TMP/serve.log")" [[ "$output" == *"--name frank-dragoon-halogen"* ]] [[ "$output" == *"--publish 7770:8731"* ]] [[ "$output" == *"-e HALOGEN_MODEL_ID=frank-dragoon"* ]] - [[ "$output" == *"-e HALOGEN_CTX=262144"* ]] - [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=262144"* ]] + [[ "$output" == *"-e HALOGEN_CTX=200704"* ]] + [[ "$output" == *"-e HALOGEN_KV_POOL_POSITIONS=200704"* ]] [[ "$output" == *"-e HALOGEN_KV_SLOTS=2"* ]] [[ "$output" == *"-e HALOGEN_SPEC_ADAPT=0"* ]] [[ "$output" == *"-e HALOGEN_CHECKPOINT=/models/qwen.hgn"* ]] @@ -291,23 +291,24 @@ $(cat "$TEST_TMP/serve.log")" } @test "the shipped env keeps the harness fit law inside the ceiling" { - # 262144 (the checkpoint's native window) is the per-slot hard ceiling - # (a longer prompt is a 400 naming the limit, never a silent - # truncation). The harness clamps max_tokens = min(45056, 217088 − est - # − 4096), compacts at 85 % of its declared 217088 window, and its - # chars/4 est UNDER-counts the rendered prompt by ~3 % on code/agent - # traffic (measured 4,450 tokens at est 152,938 on 2026-10-07 — with - # window == ceiling, every request over ~146k est was a guaranteed - # 400 before compaction could fire at 85 %). The 45k gap between the - # harness window and the engine ceiling must absorb that error at - # both monotone maxima of est + budget (est ≤ 217088 − 4096): - # full budget at the compaction threshold: 1.03*184524 + 45056 - # clamped budget at the clamp's zero point: 1.03*212992 + 0 + # HALOGEN_CTX is the hard per-request ceiling (a longer prompt is a 400 + # naming the limit, never a silent truncation). The law is INVARIANT to + # the value: the harness window must sit ~45k UNDER it, because the + # harness clamps max_tokens = min(45056, window − est − 4096), compacts + # at 85 % of its declared window, and its chars/4 est UNDER-counts the + # rendered prompt by ~3 % on code/agent traffic (measured 4,450 tokens at + # est 152,938 on 2026-10-07 — with window == ceiling, every request over + # ~65 % of the window was a guaranteed 400 before compaction could fire). + # The shipped pair is ctx 200704 / window 155648 (lowered from + # 262144/217088 on 2026-10-17); the gap must absorb the error at BOTH + # monotone maxima of est + budget (est ≤ window − 4096): + # full budget at the compaction threshold: 1.03*132300 + 45056 + # clamped budget at the clamp's zero point: 1.03*151552 + 0 # and the compaction request that rides on frank-zealot (its # 200,704-token slot) must fit: input ≈ threshold − keepRecent(20k). - grep -qE '^HALOGEN_CTX=262144$' "$ENV_FILE" - ctx=262144 - hw=217088; mt=45056 + grep -qE '^HALOGEN_CTX=200704$' "$ENV_FILE" + ctx=200704 + hw=155648; mt=45056 thr=$(( hw * 85 / 100 )) [ $(( thr * 103 / 100 + mt )) -le "$ctx" ] zero=$(( hw - 4096 ))