docs: frank-large halogen runbook, AGENTS.md + indices for the swap

New runbook frank-large-halogen27b-dflash2.md; LLM + vault indices
re-pointed, ROCmFPX q6agent runbook marked RETIRED (rollback target);
AGENTS.md topology row, detail section, image-capability paragraph
(two endpoints accept images now), bench-law prose and runbook index
updated.

AGENTS.md

@@ -12,8 +12,8 @@ merge absorbed the former `vllmScripts` repo, history preserved).
   Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
 - **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
   via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (ROCmFPX llama.cpp
-  fork on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
+  the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
 - **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
   (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
-| Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 |
+| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (33.4 GiB; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (the sibling engine measured 41 → 14 t/s), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot is a 400 naming the limit, never a truncation; `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` is a 413. Cutover 2026-10-05|7770|
 | Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
 
 Retired endpoints (`frank-npu`, and every prior runtime) are in
@@ -223,7 +223,7 @@ Scratch ports are not used on LLM hosts — a
 test process cannot share the single GPU anyway; validate against the service
 itself.
 
-(A gufo-era law; frank-large is ROCmFPX llama.cpp since 2026-10-02, but the rollback tag keeps it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
+(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
 Measured 2026-09-28 on frank-large at depth 32768: its corpus
 reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
 draft widths, i.e. the same highly-predictable text every run, and 93 % of
@@ -249,7 +249,7 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
 merely wedged kernel usually still answers ARP.
 
 1. Stop and mask (self-healing would restart it):
-   - Dankclaw `frank-large` (root unit, vLLM podman on the iGPU):
+   - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
      `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
    - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
      `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
@@ -285,9 +285,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-10-02 **all three endpoints accept images** (`input: [text, image]`):
-`frank-large` through the BF16 mmproj that ships inside its GGUF payload,
-`frank-primary` through a BF16 mmproj loaded explicitly (this build never
+of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
+image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
+ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
+`frank-primary` takes them through a BF16 mmproj loaded explicitly (this build never
 autoloads projectors), and `frank-small` through
 `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` passed as `--mmproj` — the MiMo
 GGUF itself carries no vision tensors. `frank-small`'s `contextWindow` is
@@ -304,84 +305,59 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
 audio encoder, and left again with the Gemma4 retirement on 2026-10-01
 (the MiMo tower is vision-only).
 
-### frank-large — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (Dankclaw, ACTIVE)
+### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)
 
-The **kingjones30/ROCmFPX llama.cpp fork** at commit
-`36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built for gfx1151 by
-`scripts/build-rocmfpx.sh` into `/home/dank/GitPackages/kingjones-ROCmFPX/build`,
-as a **root systemd unit** (`scripts/frank-large.service`, `User=dank`,
-`Restart=always`) running a bare `llama-server` (no container) on port 7770
-from the Strix Halo iGPU. Boot to healthy: **~15 s**.
+The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
+digest-pinned, published tag 0.1.4) as a podman container driven by
+`scripts/serve-frank-large.sh` under a **root systemd unit**
+(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman
+stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart
+policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to
+`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state,
+any clamp) lives there, not in the journal**. Serving
+`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,880,329,102 B
+≈ 33.4 GiB, p1 weights / w4 activations / d2 KV) + `frank-large.tokenizer.gguf`
+(1,293,343 B). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
+([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
+chosen for the publisher's 31.7 t/s DFlash2 mean.
 
-**The runtime is not interchangeable — it is chosen by the FILE.**
-`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF`'s protected-heads
-rung, `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` (24,856,607,008 B,
-sha256 `e966deb3…`, ftype 114 ≈ 7.44 bpw), stores **339 tensors of type 102,
-`GGML_TYPE_Q6_0_ROCMFPX`, plus 159 type-103 `Q8_0_ROCMFPX`** — quantizations
-that exist in no other loader: stock llama.cpp and gufo both refuse the file at
-load (`unknown tensor type`). The AGENT construction keeps 64 output-head
-tensors at plain precision (the protected-heads design of the aligned repo's
-Q8 file, one quant rung down), and the uncensored tuning is what Frank swapped
-to on 2026-10-03. Vision comes from the BF16 `mmproj-Qwen3.8-27B-BF16.gguf`
-(931,145,888 B — hardlinked from the aligned payload dir, byte-identical).
+**The format is the lock.** `.hgn` loads in nothing else — the runtime is
+chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
+2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
+halogen-server repo restarted versioning at 0.1.x.
 
-**The pin is load-bearing.** The 2026-08-25 ROCmFPX builds segfaulted in
-`ggml_compute_forward_set_rows` on exactly this file; `build-rocmfpx.sh`
-checks the commit out by SHA, so a fork-side force-push cannot silently
-re-break the endpoint.
+**Two product regressions, both accepted by Frank before deploy:** the
+endpoint is **text-only** (README honest limits: the vision encoder is
+ignored — frank-small owns images now) and the checkpoint is a finetune of the
+**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
+commit-revert away with the AGENT payload still on disk).
 
-**Spec decode is the performance feature, and it is measured.** The draft is
-the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (1,680,271,648 B, sha256 `051a1764…`,
-`nextn_predict_layers = 1`), pinned to the same card:
-`--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-n-min 0
---spec-draft-p-min 0.0 --spec-draft-device ROCm0 --spec-draft-ngl 99`.
-On the ALIGNED Q8 file (2026-10-02, bench windows, 2,202-token prompt, temp 0):
-**MTP 17.00–17.14 t/s vs 7.60–7.66 with the draft stripped — +123 %**,
-acceptance 0.462. On the UNCENSORED Q6 file (2026-10-03, production service,
-prose at 4,813-token depth): **27.57–28.10 t/s decode, draft acceptance
-396/456 = 0.868** (mean acceptance length ≈ 4.4 — drafting at nearly the full
-n-max 4), schema-echo traffic 31.44 t/s at 639/639 = 1.000. The drafter is
-materially better on this file; treat the card's 0.95 as a file property, not a
-throughput promise. **`ngram-mod` chained onto MTP was measured and REJECTED
-(2026-10-03):** `--spec-type draft-mtp,ngram-mod` cost −15 % on prose
-(23.40 t/s) and −7 % on verbatim block-copy even with the matcher demonstrably
-firing — the boot log says why: `recurrent state rollback is not compatible
-with 'ngram-mod' - disabling rollback support`, and the hybrid's MTP
-verification needs that rollback. Keep `draft-mtp` alone.
+**Engine laws that shape the config** (all in `scripts/frank-large.env` with
+evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
+drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
+ceiling (400 naming the limit, never truncation); `models.yml`
+`contextWindow: 200700` must match or the harness discovers the wall as a 400.
+`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
+pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 413).
+`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
+temperature is greedy, so the thinking recipe exists only in `models.yml`
+`extraBody`. `response_format` is unsupported (no constrained decoding).
+Thinking lands in **`reasoning_content`** (harness field unchanged); the same
+Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
+`models.yml` fold stays mandatory.
 
-**Shape and budget:** ONE session at the native 262,144 window, unquantized
-f16 KV (~64 KiB/token → ~16 GiB, affordable inside the iGPU's ~114 GiB GTT
-aperture), `--fit off --flash-attn on --direct-io --ctx-checkpoints 16
---cache-ram 8192 --cache-idle-slots`. Measured GTT ~61 GiB under live load
-(host baseline 16.7 GiB). Prompt cache verified (Q8 era): a repeat pass over
-the same 6k prompt replays in 0.5 s.
+Fetch: `scripts/download-frank-large-halogen.sh` into
+`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + sha256 +
+`--verify-only`, no token on the command line). The deployer and serve script
+both refuse a **symlinked or directory tokenizer** — an HF snapshot layout
+passes `test -f` but fails the engine's open at request time.
 
-**Client contract (llama.cpp, live-verified 2026-10-02):** `--jinja` is on
-(without it `chat_template_kwargs` are silently ignored), tools are native
-(`get_weather` → `{"city":"Seattle"}`), thinking lands in
-**`reasoning_content`**, `reasoning_effort` accepts ONLY
-`xhigh|medium|low` (anything else is HTTP 500 — `models.yml` folds `high` and
-`max` to `xhigh`), and thinking is enabled server-side with the card's
-recipe (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0) plus
-`--reasoning-format deepseek --reasoning-budget -1`. Two traps: a request
-whose whole budget is consumed by reasoning returns `content: null` (bench
-clients must fall back to `reasoning_content` or they crash *after*
-measuring), and the fork logs
-`MTP boundary missing for seq_id=… ; resyncing` under concurrent draft
-resets — self-recovering, not a fault.
-
-Fetch: `scripts/download-frank-large-kingjones-uncensored.sh` (hardlinks the
-byte-identical MTP head and projector from the aligned payload dir when
-present, otherwise fetches; `-f` + byte count + LFS sha256 verified,
-`--verify-only`, no token on the command line).
-
-Runbook: `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md`.
-Rollback: the ALIGNED Q8 file (the previous shape) remains on disk at
-`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
-2026-10-03 swap commit (unit/env/serve trio) and redeploy. The older gufo tag
-`frank-large-gufo-q4kxl-20261001` is now a **re-download** rollback: its
-Q4_K_XL payload and DFlash2 drafter were removed from disk 2026-10-03
-(`scripts/build-gufo.sh` stays for the rebuild).
+Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
+Rollback: revert the swap commit and redeploy — the ROCmFPX payload
+(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
+and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
+The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
+rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
 
 ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
 
@@ -1005,7 +981,7 @@ Active:
 
 | Service | Runbook |
 |---------|---------|
-| frank-large (ROCmFPX llama.cpp fork, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP4 + vision, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
+| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
 | frank-small (llama.cpp CUDA, MiMo-V2.6-Distill-Qwen-9B Q6_K, 2×200,704 f16 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md` |
 | frank-primary (tcclaviger vLLM, Qwen3.8-Flash-Next FP8, Dankserver) | `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
@@ -1015,6 +991,7 @@ Retired (rollback targets and history):
 
 | Runtime | Host | Window | Runbook |
 |---------|------|--------|---------|
+| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
 | ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
 | gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
 | vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |