docs(frank-dragoon): the 401,408-position pool is live; every figure re-keyed to the boot ledger
Deployed 2026-10-18 (e2e 33/33). The boot ledger prints `reserving the KV pool, 401408 positions (about 11.0 GiB)` and `62.9 GiB of weights locked … 11.0 GiB of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`, so the ~10.8 GiB that every page quoted from the ~28 KiB/position arithmetic is replaced by the reservation the engine actually makes (11.0) and its preflight bill (11.3). Host MemAvailable fell 26 -> 22 GiB across the raise: the FLAGS.md collision (pool RAM is n-gram page cache) paid in full, so the warm prompt-cache ratio is flagged as the number to watch. All "committed but not yet deployed" clauses retired: AGENTS.md topology row, KV-pool law and memory ledger, frank-dragoon.env lore, both index pages, the iommu page, and the runbook chronology (third production boot recorded). The sweep also caught the LLM index still describing frank-zealot as Bonsai-2-27B at 2 x 200,704 in its intro and topology diagram - re-keyed to the live 9B: 8 full-attention + 24 Gated DeltaNet + blk.32 MTP, 3 x 262,144, tensor type 141.
This commit is contained in:
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
|||||||
| Target | Service | Runtime | Model | Port |
|
| Target | Service | Runtime | Model | Port |
|
||||||
|--------|---------|---------|-------|------|
|
|--------|---------|---------|-------|------|
|
||||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 and DEPLOYED 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; **11.0 GiB reserved, measured on that deploy** (the preflight formula bills 11.3) where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +151 MiB of recurrent state for the three DeltaNet cells — the SPEC-ARMED arms print 452 MiB, because MTP depth 2 bills three seqs per slot), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Production boot 17,922 used / 6,203 free; **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7) and **~300 t/s** aggregate across three concurrent streams (102.8 each); 2,877 t/s prefill at 106,418 was the trial arm. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for **vision**, on the RTX 3090, **THREE 262,144-token slots** (`--ctx-size 786432 --parallel 3` = every conversation at the trained window), **q8_0 K + q8_0 V** under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; **f16 KV needs 26,112 MiB and cannot boot this card**; +151 MiB of recurrent state for the three DeltaNet cells — the SPEC-ARMED arms print 452 MiB, because MTP depth 2 bills three seqs per slot), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, not because the head is absent**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Production boot 17,922 used / 6,203 free; **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7) and **~300 t/s** aggregate across three concurrent streams (102.8 each); 2,877 t/s prefill at 106,418 was the trial arm. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
|
||||||
|
|
||||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||||
@@ -407,10 +407,10 @@ MAXIMUM-length conversations DO coexist. The PER-REQUEST ceiling stays
|
|||||||
`HALOGEN_CTX=200704`: the checkpoint's native window is 262,144 and the engine's
|
`HALOGEN_CTX=200704`: the checkpoint's native window is 262,144 and the engine's
|
||||||
own 400 text refuses to exceed it without `HALOGEN_ROPE_YARN`, so "more context"
|
own 400 text refuses to exceed it without `HALOGEN_ROPE_YARN`, so "more context"
|
||||||
bought here is CONCURRENCY, not a longer single conversation. ~28 KiB/position ⇒
|
bought here is CONCURRENCY, not a longer single conversation. ~28 KiB/position ⇒
|
||||||
the ledger reserves "about 10.8 GiB", where 200,704 cost 5.5, 262,144 cost 7.2
|
the 2026-10-18 deploy RESERVES **11.0 GiB** (`reserving the KV pool, 401408
|
||||||
and the 2026-10-06 shape's 524,288 measured 14.6; the engine's preflight FORMULA
|
positions (about 11.0 GiB)`; the preflight formula bills 11.3 with block scratch
|
||||||
bills slightly more than the reservation (block scratch + ~115 MiB/slot of O(1)
|
+ ~115 MiB/slot of O(1) state), where 200,704 cost 5.5, 262,144 cost 7.2 and
|
||||||
state) — quote the reservation.
|
the 2026-10-06 shape's 524,288 measured 14.6 — quote the reservation.
|
||||||
`HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
`HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
||||||
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
|
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
|
||||||
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
|
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
|
||||||
@@ -428,10 +428,12 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content +
|
|||||||
"video".
|
"video".
|
||||||
|
|
||||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): weights
|
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): weights
|
||||||
62.9 (trunk 62.11 + vision 0.84) + KV ~10.8 + working 5.7
|
62.9 (trunk 62.11 + vision 0.84) + KV **11.0** + working 5.7
|
||||||
(`3.8 + 8.4 × MAX_TOK/16384`) ≈ **79.4 GiB in all**, the ~1.0 GiB prompt cache
|
(`3.8 + 8.4 × MAX_TOK/16384`) = **79.6 GiB in all** (boot ledger, the
|
||||||
booked separately by the ledger, + `HALOGEN_HOST_RESERVE_GIB=20` → ~22 GiB
|
401,408-position deploy of 2026-10-18), the **1.3 GiB** prompt cache (12 entries
|
||||||
margin over baseline co-tenancy (the pool-equals-ctx boot printed 74.1 GiB with
|
× 111 MiB at ctx 200704) booked separately by the ledger, + `HALOGEN_HOST_
|
||||||
|
RESERVE_GIB=20` → ~20 GiB margin over baseline co-tenancy (`MemAvailable` read
|
||||||
|
22 GiB right after the raise; the pool-equals-ctx boot printed 74.1 GiB with
|
||||||
~30 GiB of margin). The 51.2 GB n-gram table rides as mmap'd page cache beside
|
~30 GiB of margin). The 51.2 GB n-gram table rides as mmap'd page cache beside
|
||||||
the checkpoint (ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why
|
the checkpoint (ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why
|
||||||
every pool position is a double cost: **RAM the pool takes is RAM that table
|
every pool position is a double cost: **RAM the pool takes is RAM that table
|
||||||
|
|||||||
+15
-12
@@ -68,10 +68,11 @@
|
|||||||
# ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the
|
# ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the
|
||||||
# checkpoint's native window is 262,144 and the engine refuses to go past it
|
# checkpoint's native window is 262,144 and the engine refuses to go past it
|
||||||
# without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position
|
# without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position
|
||||||
# (FLAGS.md) ⇒ ~10.8 GiB, where pool == ctx (200,704) cost 5.5, 262,144 cost
|
# (FLAGS.md) ⇒ the DEPLOYED 2026-10-18 boot RESERVES 11.0 GiB (`reserving the
|
||||||
# 7.2 and the 2026-10-06 shape's 524,288 measured 14.6. The engine's preflight
|
# KV pool, 401408 positions (about 11.0 GiB)`), where pool == ctx (200,704)
|
||||||
# formula bills a little more than the reservation (block scratch + ~115
|
# cost 5.5, 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6.
|
||||||
# MiB/slot of O(1) state) — quote the RESERVATION. FLAGS.md law: "RAM the pool
|
# The preflight formula bills 11.3 (block scratch + ~115 MiB/slot of O(1)
|
||||||
|
# state) — quote the RESERVATION. FLAGS.md law: "RAM the pool
|
||||||
# takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this
|
# takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this
|
||||||
# much page cache, so the warm prompt-cache ratio is what to re-measure.
|
# much page cache, so the warm prompt-cache ratio is what to re-measure.
|
||||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||||
@@ -82,14 +83,16 @@
|
|||||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the boot
|
# mmap'd page cache beside the checkpoint, not a pinned charge — the boot
|
||||||
# ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"):
|
# ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"):
|
||||||
# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~10.8 +
|
# weights 62.9 GiB locked (0.8 of them the vision tower) + KV 11.0 +
|
||||||
# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 79.4 GiB engine-side, with the
|
# working 5.7 (HALOGEN_MAX_TOK=8192) = 79.6 GiB engine-side (the boot
|
||||||
# prompt cache (~1.0 at this ctx; 1.3 was measured at 262144) booked
|
# ledger of the 401408-position deploy, 2026-10-18), with the 1.3 GiB
|
||||||
# separately by the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
# prompt cache (12 entries x 111 MiB at ctx 200704) booked separately by
|
||||||
# honest. Against the 121.2 GiB box that leaves ~22 GiB over the reserve for
|
# the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host honest. Against
|
||||||
# baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot
|
# the 121.2 GiB box that leaves ~20 GiB over the reserve (MemAvailable
|
||||||
# ledger is the referee — read it after deploy and re-size if the engine
|
# read 22 GiB right after the raise) for baseline co-tenancy
|
||||||
# prints a clamp.
|
# (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot ledger is the
|
||||||
|
# referee — read it after deploy and re-size if the engine prints a
|
||||||
|
# clamp.
|
||||||
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
||||||
# turns the MTP head off for whole requests after low acceptance and
|
# turns the MTP head off for whole requests after low acceptance and
|
||||||
# HALVES decode as a result. Frank accepted the head always-on trade
|
# HALVES decode as a result. Frank accepted the head always-on trade
|
||||||
|
|||||||
@@ -202,10 +202,9 @@ boot ledger is unchanged to the MiB — 62.9 GiB weights (0.8 of them vision) +
|
|||||||
GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP
|
GiB KV + 5.7 GiB working = 74.1 GiB, two slots over ONE 200,704-position pool, MTP
|
||||||
depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all
|
depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafana all
|
||||||
answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
|
answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
|
||||||
slot parity was COMMITTED — `2b12716` — but as of 2026-10-18 NOT yet
|
slot parity was COMMITTED (`2b12716`) and **deployed the same evening**: the
|
||||||
redeployed: the live boot still reserves ONE 200,704-position pool (5.5 GiB),
|
boot now reserves `401408 positions (about 11.0 GiB)` and the engine-side total
|
||||||
host `frank-dragoon.env` included. When it lands, the reservation is ~10.8
|
is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working),
|
||||||
GiB and the engine-side total ~79.4 GiB;
|
|
||||||
see [[frank-dragoon-flash-server-v2-native]].)
|
see [[frank-dragoon-flash-server-v2-native]].)
|
||||||
|
|
||||||
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
|
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
|
||||||
|
|||||||
@@ -94,9 +94,9 @@ the pool: **two long conversations can no longer share the endpoint** (two of
|
|||||||
≤100k can). That is acceptable here because frank-dragoon carries the `slow`
|
≤100k can). That is acceptable here because frank-dragoon carries the `slow`
|
||||||
(medium-thinking) role and the `security-reviewer` subagent, not the
|
(medium-thinking) role and the `security-reviewer` subagent, not the
|
||||||
interactive workhorse (that is frank-archon).
|
interactive workhorse (that is frank-archon).
|
||||||
Frank lifted exactly that concurrency limit on 2026-10-18 — see *Pool
|
Frank lifted exactly that concurrency limit on 2026-10-18 and the lift is
|
||||||
doubled for slot parity* below; that lift is COMMITTED but not yet deployed,
|
**deployed** (section below): the pool is again `slots × ctx`, so two
|
||||||
so the sentences above remain the live truth until the next redeploy.
|
maximum-length conversations fit the endpoint at once.
|
||||||
|
|
||||||
**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot
|
**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot
|
||||||
printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
|
printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
|
||||||
@@ -115,7 +115,7 @@ that long prompts read.
|
|||||||
`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk
|
`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk
|
||||||
stays pinned. Disk-backing the weights is not a lever on this format.
|
stays pinned. Disk-backing the weights is not a lever on this format.
|
||||||
|
|
||||||
## Pool doubled for slot parity — committed 2026-10-18, awaiting redeploy
|
## Pool doubled for slot parity — committed AND deployed 2026-10-18
|
||||||
|
|
||||||
`HALOGEN_KV_POOL_POSITIONS` **200,704 → 401,408** (commit `2b12716`), Frank's
|
`HALOGEN_KV_POOL_POSITIONS` **200,704 → 401,408** (commit `2b12716`), Frank's
|
||||||
call: frank-dragoon's total in-flight context was **half** frank-zealot's —
|
call: frank-dragoon's total in-flight context was **half** frank-zealot's —
|
||||||
@@ -133,24 +133,28 @@ conversation had to stay under ~100k (commit message;
|
|||||||
(`scripts/serve-frank-dragoon.sh:85-95`); 401,408 is `slots × ctx`, the top
|
(`scripts/serve-frank-dragoon.sh:85-95`); 401,408 is `slots × ctx`, the top
|
||||||
of the legal band, and exactly frank-zealot's pool number — parity, not
|
of the legal band, and exactly frank-zealot's pool number — parity, not
|
||||||
approximation.
|
approximation.
|
||||||
- Expected reservation **~10.8 GiB** at ~28 KiB/position (preflight formula,
|
- Measured reservation **11.0 GiB** — the boot ledger prints `reserving the KV
|
||||||
`scripts/frank-dragoon.env:70-72`); the measured points beside it are 5.5
|
pool, 401408 positions (about 11.0 GiB)`. The preflight KV-budget line bills
|
||||||
GiB at 200,704 (boot ledger 2026-10-18), 7.2 GiB reserved / 7.4 GiB billed
|
**11.3 GiB** (`~28 KiB/position incl. block scratch + ~115 MiB/slot`) plus
|
||||||
at 262,144, and 14.6 GiB at the 2026-10-06 shape's 524,288
|
1.3 GiB of prompt cache. The measured points beside it: 5.5 GiB at 200,704,
|
||||||
|
7.2 GiB at 262,144, and 14.6 GiB at the 2026-10-06 shape's 524,288
|
||||||
(`scripts/frank-dragoon.env:49-52`, `:74-76`).
|
(`scripts/frank-dragoon.env:49-52`, `:74-76`).
|
||||||
- That RAM comes out of the n-gram table's page cache — FLAGS.md law: RAM the
|
- That RAM comes out of the n-gram table's page cache — FLAGS.md law: RAM the
|
||||||
pool takes is RAM that table loses.
|
pool takes is RAM that table loses.
|
||||||
|
|
||||||
**Deploy state as of 2026-10-18: NOT yet deployed.** The live container
|
**Deployed 2026-10-18** (`./scripts/deploy.sh dankclaw --service
|
||||||
(booted 2026-10-18 01:25) still runs `HALOGEN_KV_POOL_POSITIONS=200704` —
|
frank-dragoon`, Playwright gate 33/33). The boot ledger confirms the shape:
|
||||||
verified in its `flash_serve` `/proc/<pid>/environ` and in the host-deployed
|
`reserving the KV pool, 401408 positions (about 11.0 GiB)`,
|
||||||
`frank-dragoon.env` — and the 2026-10-18 boot ledger reserves `the KV pool,
|
`2 slots over ONE 401408-position KV pool, each request up to 200704 (it
|
||||||
200704 positions (about 5.5 GiB)` with `74.1 GiB in all`
|
reserves prompt + max_tokens)`, and the sum
|
||||||
(`/home/dank/logs/frank_dragoon.log`). Until the next
|
`62.9 GiB of weights locked in RAM (0.8 of them the vision tower's), 11.0 GiB
|
||||||
`./scripts/deploy.sh dankclaw --service frank-dragoon`, the boot ledger keeps
|
of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`. Prompt cache ON (12
|
||||||
printing `2 slots over ONE 200704-position KV pool`; the first boot at the
|
entries × 111 MiB at ctx 200704 = the 1.3 GiB in the KV-budget line), n-gram
|
||||||
new pool must show `ONE 401408-position KV pool` and the ~10.8 GiB
|
47.68 GiB still disk-backed (`Pin::None`), no clamp line. Host `MemAvailable`
|
||||||
reservation — read that line before quoting 401,408 as live anywhere.
|
fell 26 → 22 GiB across the raise, which is the FLAGS.md collision being paid
|
||||||
|
in page cache. **Not re-measured since the raise:** the warm prompt-cache ratio
|
||||||
|
(136.7× on the pre-raise boot) — that is the number to watch if a long warm
|
||||||
|
turn starts costing full prefill again.
|
||||||
|
|
||||||
## Measured A/B — same battery, both boots (2026-10-17)
|
## Measured A/B — same battery, both boots (2026-10-17)
|
||||||
|
|
||||||
@@ -307,9 +311,17 @@ restart; `webui.db.bak-20261017b` holds the pre-cutover row).
|
|||||||
**What the cut cost:** with pool == ctx one maximum-length request fills the
|
**What the cut cost:** with pool == ctx one maximum-length request fills the
|
||||||
pool, so two 200k conversations never coexist (two of ≤100 k do), and the
|
pool, so two 200k conversations never coexist (two of ≤100 k do), and the
|
||||||
harness window is now 155,648 — the 180 k compaction target of the 262,144 era
|
harness window is now 155,648 — the 180 k compaction target of the 262,144 era
|
||||||
is unreachable because the window itself is smaller. (The pool == ctx
|
is unreachable because the window itself is smaller. (That pool == ctx
|
||||||
concurrency limit is superseded — on redeploy only — by *Pool doubled for slot
|
concurrency limit IS superseded: the doubling deployed the next day — see
|
||||||
parity* above; the window math stands, the pool never moves the ceiling.)
|
*Pool doubled for slot parity* above. The window math stands, though: the pool
|
||||||
|
never moves the per-request ceiling.)
|
||||||
**What it bought:** 1.7
|
**What it bought:** 1.7
|
||||||
GiB back as page cache for the 47.68 GiB n-gram table (9.1 GiB against the
|
GiB back as page cache for the 47.68 GiB n-gram table (9.1 GiB against the
|
||||||
2026-10-06 shape's 524,288 pool), and a smaller prompt-cache entry.
|
2026-10-06 shape's 524,288 pool), and a smaller prompt-cache entry.
|
||||||
|
|
||||||
|
**Third production boot — the `401408` pool, deployed 2026-10-18.** Same ctx,
|
||||||
|
same working set, same 111 MiB prompt-cache entry; the pool line reads
|
||||||
|
`reserving the KV pool, 401408 positions (about 11.0 GiB)` and the total
|
||||||
|
`79.6 GiB in all`. Ledger and the measured cost in
|
||||||
|
*Pool doubled for slot parity* above; host `MemAvailable` after that boot:
|
||||||
|
22 GiB.
|
||||||
|
|||||||
@@ -22,14 +22,15 @@ and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
|||||||
digest serving the NATIVE v2 conversion of the abliterated
|
digest serving the NATIVE v2 conversion of the abliterated
|
||||||
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||||
200,704-position KV pool at ctx 200704 — the 2 × ctx = 401,408 raise is
|
401,408-position KV pool — 2 × ctx, raised and DEPLOYED 2026-10-18 — at ctx
|
||||||
committed 2026-10-18 but NOT yet deployed, vision ON)** and
|
200704, vision ON)** and
|
||||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||||
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
|
Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
|
||||||
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
|
Gated DeltaNet layers + the blk.32 MTP block), THREE 262,144-token
|
||||||
|
conversations, MTP present in the file but OFF by measurement,
|
||||||
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
||||||
the publisher's own Q8_0 Qwen3-VL mmproj, on the RTX 3090 — and the custom
|
the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
|
||||||
tensor type 142 + `prism.hadamard.*` metadata mean no other loader can open the
|
tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
|
||||||
file)**.
|
file)**.
|
||||||
|
|
||||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||||
@@ -58,11 +59,11 @@ gfx1151 iGPU 114 GiB GTT]
|
|||||||
E[flash-server container gfx1151, digest-pinned image
|
E[flash-server container gfx1151, digest-pinned image
|
||||||
frank-dragoon, port 7770]
|
frank-dragoon, port 7770]
|
||||||
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
||||||
ONE 200704-position KV pool at ctx 200704 (401408 committed, awaiting deploy), 2 slots, MTP always-on, image input]
|
ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image input]
|
||||||
G[llama.cpp PrismML fork CUDA build
|
G[llama.cpp PrismML fork CUDA build
|
||||||
frank-zealot, port 7771]
|
frank-zealot, port 7771]
|
||||||
H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj
|
H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
|
||||||
2 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
3 x 262144 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
||||||
E2 --> E --> F
|
E2 --> E --> F
|
||||||
E2 --> G --> H
|
E2 --> G --> H
|
||||||
end
|
end
|
||||||
@@ -94,13 +95,10 @@ frank-zealot, port 7771]
|
|||||||
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
|
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
|
||||||
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
||||||
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
||||||
in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 =
|
in-flight context and **deployed the same evening**: the boot ledger reserves
|
||||||
7.2 GiB. **Deploy state 2026-10-18: the running container and the
|
`401408 positions (about 11.0 GiB)` with `79.6 GiB in all`
|
||||||
host-deployed env still carry the 200,704-position pool** (boot ledger
|
(`scripts/frank-dragoon.env:70-75`). It bought CONCURRENCY, not a longer
|
||||||
`reserving the KV pool, 200704 positions (about 5.5 GiB)`; host
|
window: the per-request ceiling stays 200,704, because 262,144 is the
|
||||||
`frank-dragoon.env` `HALOGEN_KV_POOL_POSITIONS=200704`) — the doubling lands
|
|
||||||
on the next redeploy (commit `2b12716`).
|
|
||||||
longer window: the per-request ceiling stays 200,704, because 262,144 is the
|
|
||||||
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
||||||
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||||
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
|
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
|
||||||
|
|||||||
@@ -314,8 +314,9 @@ The LLM inference fleet on the two hosts. Full index:
|
|||||||
since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2
|
since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2
|
||||||
abliterated** conversion of orcarouter's pinned BF16
|
abliterated** conversion of orcarouter's pinned BF16
|
||||||
(`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a
|
(`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a
|
||||||
**401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, ~10.8 GiB — one
|
**401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, 11.0 GiB
|
||||||
pool behind both slots, raised to 2 × ctx on 2026-10-18 so TWO maximum-length
|
reserved — one pool behind both slots, raised to 2 × ctx and deployed
|
||||||
|
2026-10-18 so TWO maximum-length
|
||||||
requests coexist) and the stock peonist n-gram/vision companions **hardlinked**
|
requests coexist) and the stock peonist n-gram/vision companions **hardlinked**
|
||||||
in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP
|
in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP
|
||||||
always-on, vision ON, no server-side sampler change
|
always-on, vision ON, no server-side sampler change
|
||||||
|
|||||||
@@ -2190,3 +2190,39 @@ every restart since the cutover.
|
|||||||
200,704). Note the columns are `params`/`meta` JSON — the table has no
|
200,704). Note the columns are `params`/`meta` JSON — the table has no
|
||||||
`context_length`/`description` columns.
|
`context_length`/`description` columns.
|
||||||
|
|
||||||
|
|
||||||
|
## 2026-10-18 — frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot
|
||||||
|
|
||||||
|
- `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33. The
|
||||||
|
drop-in check the zealot incident bought ran FIRST: `DropInPaths` empty, so
|
||||||
|
the repo env governed. Boot ledger: `reserving the KV pool, 401408 positions
|
||||||
|
(about 11.0 GiB)`, `2 slots over ONE 401408-position KV pool, each request up
|
||||||
|
to 200704`, and `62.9 GiB of weights locked … 11.0 GiB of KV pool, 5.7 GiB of
|
||||||
|
working memory, 79.6 GiB in all`; prompt cache ON (12 entries × 111 MiB at
|
||||||
|
ctx 200704 = 1.3 GiB), n-gram 47.68 GiB still `Pin::None`, no clamp line.
|
||||||
|
- The price is exactly the FLAGS.md collision: host `MemAvailable` fell
|
||||||
|
**26 → 22 GiB**, because every pool position is RAM the n-gram table's page
|
||||||
|
cache loses. The warm prompt-cache ratio (136.7× measured pre-raise) was NOT
|
||||||
|
re-measured — recorded in the runbook as the number to watch.
|
||||||
|
- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the
|
||||||
|
engine reserves **11.0 GiB** and its preflight bills **11.3**. Re-keyed in
|
||||||
|
AGENTS.md (topology row, KV-pool law, memory ledger), the `frank-dragoon.env`
|
||||||
|
lore block, both index pages, the iommu page and the runbook, and every
|
||||||
|
"committed but not yet deployed" clause was retired with the measurement.
|
||||||
|
- The same sweep caught the LLM index still describing frank-zealot as
|
||||||
|
**Bonsai-2-27B at 2 × 200,704** in its intro paragraph and topology diagram —
|
||||||
|
the 9B swap of the same day had not reached those two spots. Re-keyed: 8
|
||||||
|
full-attention + 24 Gated DeltaNet + the `blk.32` MTP block, 3 × 262,144,
|
||||||
|
tensor type **141**.
|
||||||
|
- PTQ1_0 provenance corrected: `scripts/download-frank-zealot-bonsai2-ptq1.sh`
|
||||||
|
IS committed, so the armed-but-never-measured arm that hijacked the zealot
|
||||||
|
deploy was a **repo-tracked candidate**, not an orphan. The runbook now states
|
||||||
|
the decision it still owes: measure it in a real window, or record it
|
||||||
|
declined.
|
||||||
|
- OpenWebUI: both decoration rows re-keyed in the live DB. The values live
|
||||||
|
inside JSON columns — `params.context.length` and `meta.description` — not in
|
||||||
|
fields named for them, and there is no `sqlite3` binary on the host, so
|
||||||
|
`python3` stdlib + `Connection.backup()` (`webui.db.bak-20261018`). Zealot
|
||||||
|
200,704 → **262,144** with the 9B description; dragoon 262,144 → **200,704**,
|
||||||
|
which had kept the pre-lowering window while its own description said
|
||||||
|
200,704.
|
||||||
|
|||||||
Reference in New Issue
Block a user