diff --git a/AGENTS.md b/AGENTS.md index af008fd..d740025 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| | Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 | -| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770| +| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=262144` pool** (the card's recommendation: the ledger's 7.2 GiB where the old 524,288 pool measured 14.6 — per-request ceiling unchanged at 262,144, two maximum-length conversations no longer coexist, and the freed RAM is page cache for the n-gram table), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770| | Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). | Retired endpoints (`frank-npu`, and every prior runtime) are in @@ -334,7 +334,7 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (neither the MiMo nor the Qwen3.5 tower carries audio). -### frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE) +### frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, ACTIVE) The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container @@ -346,19 +346,23 @@ run` FOREGROUND — the container's exit code is the service's exit so 7770 from the Strix Halo iGPU. Engine stdout goes to `/home/dank/logs/frank_dragoon.log` — **the boot ledger (weights resident, KV pool, MTP head, prompt cache, any clamp) lives there, not in the journal**. -Cutover 2026-10-06 from the halogen 27B-dense era -([[frank-dragoon-halogen27b-dflash2]], retired the same window). +Cutover 2026-10-17 from the Swift1.5-finetune shape +([[frank-dragoon-flash-server-v2-swift15]], retired the same window; it +itself cut over the halogen 27B-dense era on 2026-10-06). -Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`: -`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1 -GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B -append-only original; the abliteration workflow appends and the repo's -tooling drops dead padding, live tensors byte-identical) + the **REQUIRED** -n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B, -peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn` -(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files, -fetched fresh from peonist). Cutover 2026-10-06 from the halogen -27B-dense shape, chosen by Frank for the ABLITERATED base. +Serving `Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated`: +`qwen38-flash-next-v2-abliterated.hgn` (66,687,678,432 B ≈ 62.1 GiB, sha256 +`b5bd925f…` — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned +BF16 checkpoint, abliterated; NOT the writer transplant that carried this +filename, which measured 66,687,678,464 B — 32 bytes apart, same filename +family, different model) + the **REQUIRED** n-gram lookup table +`qwen38-flash-next-ngram.hgn` (51,200,246,144 B, peonist @5cc17cea) + the +vision tower `qwen38-flash-next-vision.hgn` (897,916,416 B, @e053f488) + the +flat `tokenizer/` directory (6 files) — the three companions **hardlinked** +out of the swift15 payload dir, because the release publishes only the main +file and no source-matched pair exists. Cutover 2026-10-17 from the Swift1.5 +finetune, chosen by Frank for the base checkpoint itself plus the card's +memory-saving pool. **The format is the lock.** The v2 `.hgn` loads ONLY in the flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the @@ -389,8 +393,13 @@ the harness window = 184,524 est ≈ 190k measured (Frank's 180k target). `HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter" law does NOT carry over; batching and speculation coexist (the card's quickstart ships 2 slots with the head), so 2 is concurrent capacity. -`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒ -14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the +`HALOGEN_KV_POOL_POSITIONS=262144` (the card's recommendation and the memory +saving of the 2026-10-17 swap; ~28 KiB/position ⇒ 7.2 GiB) is **ONE pool +behind both slots** — ledger `2 slots over ONE 262144-position KV pool, each +request up to 262144` — so the per-request ceiling is untouched while two +MAXIMUM-length conversations no longer coexist (the old 524,288 = 2 × ctx +spent the measured 14.6 GiB and allowed it). +`HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table). **A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` = @@ -407,27 +416,34 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content + "video". **Memory ledger** (121.2 GiB unified; the referee is the boot ledger): -weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision -~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB -margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd -page cache beside the checkpoint — the card's quickstart arithmetic does -not bill it as resident, but READ the boot ledger and re-size if the -engine prints a clamp. +weights ~62.9 (trunk 62.11 + vision 0.84) + KV 7.2 + arena ~8 +(`3.8 + 8.4 × MAX_TOK/16384`) ≈ 78 GiB engine-side + +`HALOGEN_HOST_RESERVE_GIB=20` → ~25 GiB margin over baseline co-tenancy. +The 51.2 GB n-gram table rides as mmap'd page cache beside the checkpoint +(ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why the pool +cut is a double saving: **RAM the pool takes is RAM that table loses**. +`HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the v2 loader, so disk-backing +the trunk is not a lever on this format. READ the boot ledger and re-size +if the engine prints a clamp. -Fetch: `scripts/download-frank-dragoon-flashnext.sh` into -`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS -sha256 + `--verify-only`, no token on the command line; resume goes -against the RESOLVED CDN URL because curl drops `Range` across HF's 302). -The serve gates mirror the deployer EXACTLY (byte-exact checkpoint + -n-gram + vision tower, flat tokenizer with all six files as regular -files) — the 2026-10-01 crash-loop lesson. +Fetch: `scripts/download-frank-dragoon-v2abl.sh` into +`/home/dank/models/halogen-v2-abliterated/` (`-f` + exact byte counts + LFS +sha256 + `--verify-only`, no token on the command line; resume goes against +the RESOLVED CDN URL because curl drops `Range` across HF's 302) — the main +file is fetched, the three companions are HARDLINKED from the verified +swift15 dir (a hardlink is a regular file, so the no-symlink serve gate +passes without a 47.7 GiB copy). The serve gates mirror the deployer EXACTLY +(byte-exact checkpoint + n-gram + vision tower, flat tokenizer with all six +files as regular files) — the 2026-10-01 crash-loop lesson. Runbook: -`serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`. -Rollback: revert the swap commit and redeploy — the 27B payload -(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the -halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind -that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched. +`serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md`. +Rollback: revert the swap commit and redeploy — the Swift1.5 payload +(`/home/dank/models/halogen-flashnext-v2/`, re-verified by +`scripts/download-frank-dragoon-flashnext.sh`), the 27B payload +(`/home/dank/models/halogen-qwen3.8-27b/`) and the halogen 0.1.4 image +digest all remain on disk; the ROCmFPX door behind that +(`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched. ### frank-zealot — llama.cpp (PrismML fork) on the RTX 3090 (Dankclaw, ACTIVE) @@ -1140,7 +1156,7 @@ Active: | Service | Runbook | |---------|---------| -| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` | +| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 262,144 pool, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` | | frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` | | frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | @@ -1153,6 +1169,7 @@ Retired (rollback targets and history): | radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` purged 2026-10-17, byte-exact re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` | | llama.cpp CUDA (master) Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-17 (rollback target — payload + master CUDA build on disk; owns the 32 KiB/token 8-full-layer KV law and the 2→3-slot reslice law) | `frank-zealot-llama-qwen35-9b-q6k-2x200704.md` | | llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` | +| halogen-flash-server 0.16.2 abliterated Swift1.5 v2 `.hgn` + MTP always-on, 2 slots × 200,192 over a 524,288 pool (`frank-dragoon`) | Dankclaw | 2026-10-06 → 10-17 (rollback target — payload + fetch script on disk; owns the 400-zone law and the sampler/prompt-cache contract) | `frank-dragoon-flash-server-v2-swift15.md` | | ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` | | tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` | | FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-archon`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-archon-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-archon-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-archon-freetoken-pr587.md` | diff --git a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md new file mode 100644 index 0000000..23d167c --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md @@ -0,0 +1,187 @@ +# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, Strix Halo iGPU) + +**ACTIVE since 2026-10-17.** Rollback = revert the swap commit and redeploy +(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`, +`scripts/frank-dragoon.service`); the Swift1.5 payload stays on disk at +`/home/dank/models/halogen-flashnext-v2/` (its fetch script +`scripts/download-frank-dragoon-flashnext.sh` stays the door back), and the +27B-era payload beside that at `/home/dank/models/halogen-qwen3.8-27b/`. +Era write-up: [[frank-dragoon-flash-server-v2-swift15]]. + +## Why this payload + +Frank's ask: serve +`https://huggingface.co/Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated` +with the card's recommended settings, and **save memory**. + +This is the **native 2026-10-11 v2 conversion** of orcarouter's pinned BF16 +Qwen3.8-Flash-Next, abliterated — the base itself, converted straight into +the v2 `.hgn` format. It is **not** the writer transplant that carried this +filename before: the artifact measures **66,687,678,432 B** where the +Swift1.5 r2 measured 66,687,678,464 B — 32 bytes apart, same filename +family, different model. Byte counts and the Hub LFS sha256 are the only way +to tell them apart, which is why the fetch script and both gates check them. + +What the swap buys, measured (table below): retrieval at depth, decode, +prefill, and **7.4 GiB of RAM handed back**. What it costs: the Swift1.5 +finetune's compressed reasoning and its small perplexity edge. + +## The companion law (the reason this swap needed a referee) + +The release publishes **only the main file**. The n-gram lookup table +(REQUIRED — the engine reads it beside the checkpoint), the vision tower, +and the tokenizer are peonist's, and the card does **not** establish them as +interchangeable with this native conversion. No source-matched pair is +published anywhere, so the only available companions are the stock ones. + +They are **hardlinked** out of the verified Swift1.5 payload directory +(`scripts/download-frank-dragoon-v2abl.sh`): a hardlink is a regular file, so +it satisfies the serve gate's no-symlink law, and it costs no disk and no +copy next to a 62 GB payload. The engine's own checkpoint tools are the +referee, and they answered: **NIAH 18/18** (vs 17/18 for the payload those +files were published with), needle retrieval exact at both depths, the red-PNG +vision probe exact, perplexity in the normal band. A wrong PLE table does not +whisper — see Verification. + +## The payload — one new file, three inherited ones + +`/home/dank/models/halogen-v2-abliterated/`, fetched by +`scripts/download-frank-dragoon-v2abl.sh` (`-f` + exact byte counts + LFS +sha256 + `--verify-only`; no token on the command line): + +|File|Bytes|sha256|Source| +|---|---|---|---| +|`qwen38-flash-next-v2-abliterated.hgn`|66,687,678,432|`b5bd925f…`|Quat3rnion `@main`, fetched| +|`qwen38-flash-next-ngram.hgn`|51,200,246,144|`3450acada…`|peonist `@5cc17cea`, **hardlinked**, REQUIRED| +|`qwen38-flash-next-vision.hgn`|897,916,416|`d62e0ae5…`|peonist `@e053f488`, hardlinked, vision ON| +|`tokenizer/` (6 files)| — |`tokenizer.json` LFS|peonist `@e053f488`, hardlinked| + +The engine stays **halogen-flash-server 0.16.2**, digest-pinned (the card +tested 0.15.0; 0.16.2 is newer in the same line and is what the v2 format was +qualified on in this fleet): + +``` +ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a +``` + +## Shape — the card's settings, and the three deliberate deviations + +`HALOGEN_CTX=262144` (per-request ceiling), `HALOGEN_KV_SLOTS=2`, +**`HALOGEN_KV_POOL_POSITIONS=262144`** (the card's recommendation, and the +memory saving), `HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`, +`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair), +sampler `1.0 / 0.95 / 20`, `HALOGEN_WEIGHTS_LOCK=1`, +`HALOGEN_HOST_RESERVE_GIB=20`. + +Deviations from the card, each on purpose: + +- `HALOGEN_MODEL_ID=frank-dragoon`, not the card's `halogen-v2-abliterated` — + the fleet naming law (service = unit = served id). +- `HALOGEN_SPEC_ADAPT=0` — the card is silent; this is a measured 0.16.2 law + (the adaptive default disables the drafter per request and halves decode). +- the sampler values are peonist's engine recipe; Quat3rnion publishes none. + +**The pool is one budget, not a partition.** Boot ledger: `listening on +127.0.0.1:8730, 2 slots over ONE 262144-position KV pool, each request up to +262144`. The per-request ceiling is unchanged, so the harness contract +(`contextWindow: 217088`) does not move — what changed is that **two +maximum-length conversations can no longer share the endpoint**. That is +acceptable here because frank-dragoon carries the `slow` (medium-thinking) +role and the `security-reviewer` subagent, not the interactive workhorse +(that is frank-archon). + +**Memory ledger** (121.2 GiB unified), from the boot ledger of the live +candidate: `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk, +`0.84 GiB` vision tower, `reserving the KV pool, 262144 positions (about +7.2 GiB)`, and the engine's own sum: `memory: 62.9 GiB of weights locked in +RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …`. The n-gram +table rides as page cache — `mapped 51.2 GB, NOT pinned (Pin::None)` — which +is exactly why cutting the pool is a *double* saving. FLAGS.md law: **RAM the +pool takes is RAM that table loses**; the 7.4 GiB the smaller pool hands back +is page cache for the 47.68 GiB table that long prompts read. + +`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk +stays pinned. Disk-backing the weights is not a lever on this format. + +## Measured A/B — same battery, both boots (2026-10-17) + +Behavior probes ran against the live service (tagged `swift15` / `v2abl`); +the engine tools ran with the service **stopped**, per the fleet benchmarking +law, under one identical knob set (`scripts/census-frank-dragoon.sh`). + +|Probe|Swift1.5 r2 (out)|native v2 (in)| +|---|---|---| +|`verify` on the file|—|OK| +|NIAH 18 cases (4k/32k × 3 depths × 3 positions)|17/18 (94.4 %)|**18/18 (100 %)**| +|wikitext-2 PPL (297,053 ids, 2 × 262,144 seqs)|**3.8772**|3.9679 (+2.3 %)| +|KL of the new model against the old model's logits|—|mean 0.155 / median 0.057, top-1 85.1 %| +|refusal census (5 hostile prompts)|0/5 refusing|0/5 refusing| +|thinking tokens, mean of 6 questions (temp 1.0)|**784**|1,476 (×1.9)| +|tool call, 1-turn / 2-turn|`{"city": "Seattle"}` / answer|`{"city": "Seattle"}` / answer| +|vision (512 px solid red, `image_url` part)|Red|Red| +|needle @21,568 tokens, planted 0.85 / 0.10|hit / hit|hit / hit| +|decode, greedy, 512 tokens|46.8 t/s (spread 25.6–46.8)|50.0 t/s (steady 50.0/50.2)| +|prefill, cold, 59,024 tokens|1,424.8 t/s|**1,604.3 t/s (+12.6 %)**| +|prompt-cache ratio (same prompt, second pass)|136.7×|131.6×| +|KV pool|14.6 GiB (524,288 positions)|**7.2 GiB (262,144)**| + +Read the KL honestly: it is a **model-vs-model** divergence, not the +quant-vs-source comparison the tool was built for. Two different models on +the same text *should* differ; median 0.057 with 85 % top-1 agreement is the +signature of two sound models that disagree on content, and the worst rows +are date/name completions. A n-gram table mismatched to the checkpoint +corrupts the forward pass and shows up as elevated KL **everywhere** plus +degraded retrieval — which NIAH (18/18) and the needle probes rule out. + +The decode line needs care: the baseline's own spread was 25.6–46.8 t/s +because its samples were taken while the 62 GB fetch was still streaming into +page cache, and the candidate's **first** sample after the battery's long +prompts read 32.8 before settling at 50.0/50.2. Honest claim: **no decode +regression, plausibly faster**; never quote either single number. + +## Verification — how to reproduce + +``` +# 1. payload: bytes + LFS sha256 + the hardlinked trio +bash scripts/download-frank-dragoon-v2abl.sh --verify-only +# 2. engine tools on a STOPPED service (fleet law: no GPU experiment beside +# a live endpoint; stop the unit first — mask fails on a relinked unit) +bash scripts/census-frank-dragoon.sh verify # no GPU, safe while serving +sudo systemctl stop frank-dragoon +bash scripts/census-frank-dragoon.sh ref # reference dump, old payload +bash scripts/census-frank-dragoon.sh cand # PPL + KL/top-1 vs that dump +bash scripts/census-frank-dragoon.sh niah-cur # baseline retrieval +bash scripts/census-frank-dragoon.sh niah-new # candidate retrieval +# 3. boot the candidate through a transient drop-in, then the battery +bash scripts/trial-frank-dragoon.sh candidate +python3 scripts/battery-frank-dragoon.py v2abl all +bash scripts/trial-frank-dragoon.sh restore # back onto the repo env +``` + +The battery needs two staged assets in `/tmp/halogen-eval/` (a 512 px solid +red PNG as base64, `red512.b64`, and the wikitext-2 raw test split, +`wikitext-2-raw-test.txt`). Laws it encodes: + +- **the engine bills wikitext at ~4.33 characters/token** — the harness + chars/4 estimate under-bills by ~8 %, and a prompt over `HALOGEN_CTX` is + refused with a 400 that names the exact overage (never truncated). Size + haystacks in characters against a **token** budget. +- the wikitext-2 raw test split as staged joins rows with a **single** + newline (2,891 lines, zero blank lines), so a blank-line split returns one + 1.29 MB "paragraph" and every budget silently becomes "everything". +- the first request after a boot is a warmup artifact; the battery discards + it (`warmup` row) and never measures it. +- `ppl`/`niah` **load the model** → stopped service. `verify` reads the file + back with no GPU and can run while serving. + +## Contract (unchanged from the Swift1.5 era) + +`reasoning_content` carries the thinking and counts against `max_tokens` +(≥512-token budget on any vision probe or the endpoint looks blind while it +is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px +probes; `response_format` unsupported; the chat template rejects +`reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays +mandatory); unknown sampling params are a 400; the prompt cache is on by +default, so a warm 100k follow-up costs seconds. See +[[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that +carries verbatim, and [[deployment]] for the deploy/health gate. diff --git a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md index 5f4348a..a05e2e0 100644 --- a/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md +++ b/serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md @@ -1,10 +1,13 @@ # frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU) -**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio -(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`, -`scripts/frank-dragoon.service`); the retired 27B payload stays on disk at -`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script -(`scripts/download-frank-dragoon-halogen.sh`) stays in the repo. +**RETIRED 2026-10-06 → 2026-10-17**, superseded by the native v2 conversion: +[[frank-dragoon-flash-server-v2-native]]. Rollback target — the payload +stays on disk at `/home/dank/models/halogen-flashnext-v2/`, +`scripts/download-frank-dragoon-flashnext.sh` re-verifies it, and restoring +the era is a revert of the 2026-10-17 swap commit. The 27B payload this era +replaced stays at `/home/dank/models/halogen-qwen3.8-27b/` with +`scripts/download-frank-dragoon-halogen.sh`. Everything below is the era as +it shipped; the engine-flag detail still governs the current page. ## Why this shape exists diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 215e2a8..992e9f3 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -16,10 +16,10 @@ disk); tags frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003), and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2 (frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned -digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of -Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine, -MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192 -ctx, vision ON)** and +digest serving the NATIVE v2 conversion of the abliterated +Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine, +MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE +262,144-position KV pool, vision ON)** and **llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48 Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled, @@ -53,12 +53,12 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2, gfx1151 iGPU 114 GiB GTT] E[flash-server container gfx1151, digest-pinned image frank-dragoon, port 7770] - F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower -2 x 200192 ctx, MTP always-on, image input] - G[llama.cpp master CUDA sm_86 + F[flash-server v2 native abliterated .hgn + ngram table + vision tower +ONE 262144-position KV pool, 2 slots, MTP always-on, image input] + G[llama.cpp PrismML fork CUDA build frank-zealot, port 7771] - H[Gemma4-26B-A4B QAT Uncensored Q4_K_M GGUF -1 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision] + H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj +2 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision] E2 --> E --> F E2 --> G --> H end @@ -77,23 +77,31 @@ frank-zealot, port 7771] ## Active Deployment -- [[frank-dragoon-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06** +- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE since 2026-10-17** (root systemd unit + podman, port 7770): the digest-pinned `ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the - Quat3rnion ABLITERATED Swift1.5 finetune of peonist's - Qwen3.8-Flash-Next v2 checkpoint (`qwen38-flash-next-v2-swift15-abliterated.hgn`, - 66.7 GB compacted r2 of the card's 89.8 GB append-only original, - sha256-verified) with the REQUIRED n-gram lookup table (51.2 GB, - @peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488) beside - it. TWO slots of 200,192 ctx, MTP head ALWAYS ON (`HALOGEN_SPEC_ADAPT=0` - — the adaptive default turns the head off per request and halves - decode), server-side sampler at the card's 1.0/0.95/20, coupled - cap/timeout 65536/7200, prompt cache ON by default (the 27B engine's - manual `HALOGEN_CACHE_MB` pin era is over — warm 100k turn ≈ 2 s vs - ≈ 88 s cold). Vision ON (image_url parts only). Card: 53-57 t/s - greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens. - Rollback: revert the swap commit; the 27B payload + fetch script stay - on disk + **native v2 conversion** of the abliterated Qwen3.8-Flash-Next + (`qwen38-flash-next-v2-abliterated.hgn`, 66,687,678,432 B, Hub LFS + sha256-verified at fetch) with the REQUIRED n-gram lookup table (51.2 GB, + @peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488) + **hardlinked** beside it — the card publishes only the main file, and the + engine's own tools certified the stock companions (`niah` 18/18, `ppl` + 3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus). + Card-recommended `HALOGEN_KV_POOL_POSITIONS=262144`: **one pool behind two + slots** — 7.2 GiB where the old shape spent 14.6, the per-request ceiling + unchanged at 262,144, and the freed RAM becomes page cache for the n-gram + table. MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout + 65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5 + boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, + PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone). + Rollback: revert the swap commit; the swift15 payload and its fetch script + stay on disk +- [[frank-dragoon-flash-server-v2-swift15]] — **RETIRED 2026-10-06 → 10-17 + (rollback target: payload + fetch script on disk)**: the Swift1.5-finetuned + v2 `.hgn` (compacted r2, 66,687,678,464 B) at TWO slots × 200,192 ctx over + a 524,288-position pool; owns the 400-zone law (the harness chars/4 + under-count against the engine ceiling), the MTP-always-on law, and the + sampler / prompt-cache / vision contract the native era carries verbatim - [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06 (rollback target; payload + fetch script + image on disk)**: the digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 803f3a0..b3bae9e 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -302,17 +302,27 @@ The LLM inference fleet on the two hosts. Full index: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record) -- [[frank-dragoon-halogen27b-dflash2]] — the ACTIVE frank-dragoon runbook since - 2026-10-05 (root systemd unit + podman, port 7770): the digest-pinned +- [[frank-dragoon-halogen27b-dflash2]] — RETIRED 2026-10-05 → 10-06 frank-dragoon + runbook (root systemd unit + podman, port 7770): the digest-pinned ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1 disables speculation — engine law); TEXT-ONLY and censored base, both accepted tradeoffs; no server-side sampler, so the thinking recipe rides the wire -- [[frank-dragoon-flash-server-v2-swift15]] — the ACTIVE frank-dragoon runbook - since 2026-10-06 (supersedes halogen 0.1.4): halogen-flash-server 0.16.2 - serving the abliterated Swift1.5 Qwen3.8-Flash-Next v2 (.hgn) +- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook + since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2 + abliterated** conversion of orcarouter's pinned BF16 + (`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) with the card's + memory-saving `HALOGEN_KV_POOL_POSITIONS=262144` (7.2 GiB, one pool behind + both slots) and the stock peonist n-gram/vision companions **hardlinked** + in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP + always-on, vision ON, no server-side sampler change +- [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17 + frank-dragoon runbook (rollback target — payload + fetch script on disk): + halogen-flash-server 0.16.2 serving the abliterated **Swift1.5 finetune** + of the Qwen3.8-Flash-Next v2 `.hgn`, two slots over a 524,288-position + pool; owns the 400-zone law and the sampler/prompt-cache contract - [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05 (rollback target; payload, fork build and build script on disk): the pinned kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/