Merge origin/main into docs/radiance-prefill-decode-census

Two conflicts, both from the 2026-10-19 vault compaction on main:

* frontmatter description — main shortened it to satisfy OKF DESC
  (max 120 chars); this branch's longer wording would have failed the
  seam. Kept main's shape and folded the new §9 in by naming
  step-shape attribution instead of profiler attribution.
* the Related bullet for [[vLLM Porting and Compile Triage]] — that
  page was destroyed on main, so the link is dropped rather than
  resurrected; this branch's [[frank-archon]] row is kept.

Index rows regenerated with scripts/gen-vault-index.py.

dankclaw/voiceservice/app.py

@@ -7,7 +7,10 @@ Consumers (the omp harness, OpenWebUI, open-notebook) speak the OpenAI
 directory under ``$VOICES_DIR`` and ``reference.wav`` inside it is what makes
 the name real. A name that has no reference is not a voice — the shim answers
 400 with the list that is, instead of silently narrating in a fallback voice
-(silent fallback is how a wrong-voice endpoint stays wrong for weeks).
+(silent fallback is how a wrong-voice endpoint stays wrong for weeks). A request
+that names NO voice at all takes ``$DEFAULT_VOICE``: that is a spoken choice in
+the env file rather than a surprise in the code, and the journal marks it
+``(default)`` so a name nobody asked for is never mistaken for one that was.
 
 Two backends sit behind the one surface, chosen by ``TTS_BACKEND``:
 
@@ -70,7 +73,7 @@ class Config:
 
     def __init__(self, voices_dir, backend, upstream, upstream_model, llama_bin,
                  llama_model, llama_mmproj, llama_lang, llama_threads, port,
-                 max_chars, timeout):
+                 max_chars, timeout, default_voice=""):
         self.voices_dir = Path(voices_dir)
         self.backend = backend
         self.upstream = upstream
@@ -83,6 +86,7 @@ class Config:
         self.port = port
         self.max_chars = max_chars
         self.timeout = timeout
+        self.default_voice = default_voice
 
     @classmethod
     def from_env(cls, environ=None):
@@ -100,6 +104,7 @@ class Config:
             port=_int(env, "SHIM_PORT", 7772),
             max_chars=_int(env, "SHIM_MAX_CHARS", 15000),
             timeout=float(_int(env, "SHIM_TIMEOUT", 600)),
+            default_voice=(env.get("DEFAULT_VOICE") or "").strip(),
         )
 
 
@@ -271,8 +276,13 @@ class SpeechHandler(BaseHTTPRequestHandler):
                                    "endpoint serves wav and mp3" % fmt))
             return
 
-        # `voice_id` is the alias the omp harness forwards for `voice`.
-        voice = body.get("voice") or body.get("voice_id")
+        # `voice_id` is the alias the omp harness forwards for `voice`. A request
+        # that names nothing takes the declared default — the harness has no
+        # setting that feeds a voice, so voice-less calls are the normal shape
+        # there, not a client bug.
+        requested = body.get("voice") or body.get("voice_id")
+        defaulted = not requested and bool(cfg.default_voice)
+        voice = requested or cfg.default_voice
         reference = resolve_voice(cfg.voices_dir, voice)
         if reference is None:
             available = list_voices(cfg.voices_dir)
@@ -297,8 +307,9 @@ class SpeechHandler(BaseHTTPRequestHandler):
 
         content_type = MP3_CONTENT_TYPE if fmt == "mp3" else WAV_CONTENT_TYPE
         self._bytes(200, audio, content_type)
-        self.log_message("speech voice=%s format=%s chars=%d -> %d bytes %s",
-                         voice, fmt, len(text), len(audio), content_type)
+        self.log_message("speech voice=%s%s format=%s chars=%d -> %d bytes %s",
+                         voice, " (default)" if defaulted else "", fmt,
+                         len(text), len(audio), content_type)
 
     # -- backends ----------------------------------------------------------
     def _speech_http(self, cfg, model, text, voice, reference):

report/npu-tts-runtime-landscape-2026-10-19.md

@@ -0,0 +1,319 @@
+# XDNA2 NPU inference runtimes on Linux (Strix Halo) — can any of them serve TTS?
+
+Research slice, 2026-10-19. Worker: `NpuRuntimeLandscape`. Research only, no code.
+Question: every way to execute model inference on an AMD XDNA2 NPU on Linux as of
+2026-10, and whether any of them can serve a speech-synthesis model.
+
+> **Status, same night — the spike that followed this slice settled it.** NO-GO on the
+> NPU (the loader declines the graph, not the silicon); the actionable door is the gfx1151
+> iGPU behind `-DGGML_VULKAN=ON`. Measured contract, exoneration chain and host gates:
+> `tts-npu-spike-verdict-2026-10-19.md`. That page supersedes §8–§9 below.
+
+Convention: vendor claims carry URL + the version/date they hold for.
+**[host]** = Dankclaw state verified on the live machine 2026-10-19 (Main), not a
+vendor claim. `[INFERENCE]` = my deduction from the cited artifacts.
+
+## 0. Answers to the three questions Main ranked
+
+**(1) A runtime that already runs TTS on XDNA2 today: it exists at AMD, not on our
+host.** Ryzen AI Software **1.8.0** (docs `latest` == 1.8.0; released 2026-07-23)
+lists `neutts-air` and `neutts-nano` as model type **TTS**, variant
+*full-optimized* (NPU-only) — <https://ryzenai.docs.amd.com/en/latest/llm_list.html>;
+release note "New TTS model support: neutts-air, neutts-nano" —
+<https://ryzenai.docs.amd.com/en/latest/relnotes.html>. Pre-compiled artifacts are
+published: `amd/neutts-air_rai_1.8.0_npu_4K` (arch `qwen2`, lastModified
+2026-07-23, Apache-2.0, "AWQ / Group 128 / Asymmetric / BFP16 activations / UINT4
+Weights") and `amd/neutts-nano_rai_1.8.0_npu_4K` (arch `llama`).
+**Neither artifact is a TTS engine.** Both file lists are LM-only (`model.onnx`,
+`genai_config.json`, BPE tokenizer, AIE-ML compiler metastate: `dd_metastate_*`
+`.ctrlpkt/.super/.state`, `cache/txn_bins.zip`) with **no codec, no vocoder, no
+reference-audio encoder**. [INFERENCE from HF file lists]
+
+**(2) FastFlowLM: no TTS, and we cannot add one** — see §2. Halogen: no TTS entry
+**[host]**. So "something installed already serves TTS on the NPU" is FALSE.
+
+**(3) Our own ONNX graph compiled for the NPU on Linux:** possible for the plain
+ONNX flows (they compile on Linux), **impossible for a new LLM/OGA model** — model
+generation is Windows-only in 1.8.0 — and gated by an operator table where BF16
+lacks Softmax/LayerNorm/ReLU/Gelu (see §4).
+
+## 1. Landscape: every way to reach the XDNA2 array on Linux
+
+| # | Path | License | Strix Halo + Linux, 2026-10 | Speech synthesis |
+|---|------|---------|------------------------------|------------------|
+| 1 | Ryzen AI SW → ONNX Runtime + Vitis AI / Ryzen AI EP; compile INT8 (`X2`) or BF16 (`VAIML`) ONNX at session creation | AMD proprietary (tarball) | supported: our device ID/rev is in AMD's STX table; compile+run works on Linux (§3, §4) | user-authored ONNX only; no TTS recipe/demo |
+| 2 | Same EP driven by **OGA** (`onnxruntime-genai`), "full fusion" NPU-only LLM flow; runner = AMD `model_benchmark` / OGA C++ API | AMD + MS OGA | Linux: **vendor-precompiled models only**, no hybrid flow | **neutts-air / neutts-nano = TTS** (LM half) |
+| 3 | **FastFlowLM** `flm` | MIT + free proprietary kernels | Strix Halo "Supported"; Ubuntu 24.04/25.10/**26.04**, Arch | none (ASR input only) |
+| 4 | **IRON / mlir-aie** + Peano → hand-written AIE-ML design → XRT/`amdxdna` | Apache-2.0 + LLVM exception | explicitly documented "AMD Ryzen™ AI on Linux"; same driver/XRT stack | anything, hand-written; nothing ships TTS |
+| 5 | Vendor-curated AIE-ML graph sets (halogen `HALOGEN_NPU_MODELS`, FLM packs) | vendor | present **[host]** | none **[host]** |
+| 6 | Windows-only: DirectML, Ryzen AI Windows EP (`ryzenai-llm`), Studio Effects, `whispercpp` NPU | — | not usable | (per Lemonade matrix: Windows only) |
+| 7 | ggml/llama.cpp/**qwentts.cpp**, whisper.cpp, sherpa-onnx, Piper | — | no XDNA backend in ggml/ORT-mainline | CPU / iGPU only |
+| 8 | IREE, TVM, ONNX Runtime mainline EPs (OpenVINO = Intel NPU, MIGraphX/ROCm = AMD GPU) | — | no XDNA2 target; AMD's NPU EP is only "Xilinx Vitis-AI (preview)" in the ORT EP list and ships inside AMD's own build | n/a |
+
+Sources: §2–§6 below.
+
+## 2. FastFlowLM (`/usr/bin/flm` 1.0.1 **[host]**) — catalog + HTTP surface
+
+* Repo moved into AMD's ROCm org **2026-08-11** (`github.com/ROCm/FastFlowLM`,
+  v1.0.0). "Supports all Ryzen™ AI Series chips with XDNA2 NPUs (Strix, Strix Halo,
+  Kraken, and Gorgon Point)."
+  <https://github.com/ROCm/FastFlowLM> (README fetched 2026-10-19).
+* Linux support landed 2026-03; distros: Ubuntu 24.04, 25.10, **26.04**, Arch,
+  generic. Needs NPU firmware ≥ 1.1.0.0, kernel **7.0+** with `amdxdna` (or
+  `amdxdna-dkms`), XRT stack (`ppa:lemonade-team/stable` → `libxrt-npu2`,
+  `amdxdna-dkms`); `flm validate` reads the DRM path (`/dev/accel/accel0`), `flm
+  run` goes through **XRT** and fails with `No such device with index '0'` unless
+  `xrt-smi examine` lists the NPU; `memlock` must be raised.
+  <https://fastflowlm.com/docs/install_lin/> (page dated 2026-03-05, fetched
+  2026-10-19). Static build bundles **XRT v2.21.75** — the same XRT series as the
+  distro package on the host **[host]**.
+* HTTP surface: `flm serve <model>` → OpenAI-compatible REST, **port 52625**, no
+  auth. Maintained endpoints: `v1/models`, `v1/chat/completions`,
+  `v1/audio/transcriptions`, `v1/embeddings`. **There is no
+  `/v1/audio/speech`.**
+  <https://fastflowlm.com/docs/instructions/server/openapi/>
+* Catalog 2026-10-19 (<https://fastflowlm.com/docs/models/>): LLaMA, DeepSeek,
+  Qwen, Gemma, MedGemma, TranslateGemma, gpt-oss, LiquidAI/LFM, Phi, Nanbeige,
+  **Whisper — card states "Type: Speech-to-Text (ASR)"**
+  (<https://fastflowlm.com/docs/models/whisper/>), EmbeddingGemma, SmolVLA,
+  Hy-MT2, MiniCPM. No speech-synthesis entry.
+* Engines are **prebuilt, closed, downloaded from Hugging Face per architecture**
+  ("Internet access to HuggingFace is required to download the optimized model
+  kernels"). The only in-tree engine build is Gemma-4E
+  (`cmake --preset linux-default -DFLM_BUILD_GEMMA4E=ON`).
+  → **A new TTS architecture cannot be added to `flm` by us.**
+
+## 3. AMD Ryzen AI Software 1.8.0 on Linux — install source, device support
+
+* **Install channel (important correction to the pypi.amd.com evidence).** The pip
+  index is Windows-only for the EP runtime (`ryzenai_dynamic_dispatch-1.7.0-…
+  win_amd64.whl`, `onnxruntime_vitisai-1.23.2-…win_amd64.whl`; `flexmlrt` has a
+  single **linux_aarch64** wheel) — but AMD's Linux stack is not on pip. It ships
+  as a tarball that is, as of 2026-10-19, **publicly reachable without an AMD
+  account**: `https://download.amd.com/opendownload/RyzenAI/1.8.0/ryzen_ai-1.8.0.tgz`
+  → HTTP 206 on a range probe, `Content-Range: bytes 0-8/**7819506855**`
+  (**≈7.8 GiB**, magic `1f 8b 08` = gzip). (The docs link it behind the
+  EULA form `account.amd.com/...ryzenai-eula-public-xef.html`; the CDN path
+  serves it directly. Verified by me with curl 2026-10-19. I did not finish
+  downloading 7.8 GiB, so the tarball's *contents* list is unverified —
+  `[INFERENCE]` only where noted.)
+* Driver/XRT package (public, 90,016,137 bytes, `application/zip`):
+  `https://download.amd.com/opendownload/RyzenAI/Driver/RAI_1.8_Linux_NPU_XRT.zip`
+  contains `xrt_202620.2.25.37_24.04-amd64-{base,base-dev,npu}.deb` +
+  `xrt_plugin.2.25.260102.56.release_24.04-amd64-amdxdna.deb` (verified by
+  download + unpack). **The plugin deb ships firmware for
+  `/usr/lib/firmware/amdnpu/17f0_11/npu.dev.sbin`** — exactly the directory our
+  NPU loads **[host]** — plus `1502_00`, `17f0_10`, `17f1_10`, `17f1_13`,
+  `libvxdna.so`, `libxrt_driver_xdna.so.2.25.260102.56`, and the DKMS driver
+  source tarball. **No xclbin/PDI and no compiler binary in this package**: it is
+  driver + XRT shim + firmware only.
+* **Our device is officially STX.** AMD's APU table
+  (<https://ryzenai.docs.amd.com/en/latest/app_development.md#apu-types>):
+  `0x1022:0x17F0 rev 0x11 → STX` (0x00/0x10 also STX, 0x20 = KRK, `0x1502` rev 0
+  = PHX/HPT). Release notes: "Strix and **Strix Halo** as STX", and the STX/KRK
+  column enables CNN INT8, CNN BF16, NLP BF16, LLM (OGA).
+* Vendor driver source (same deb, `share/amdxdna/amdxdna.tar.gz`, source dated
+  2026-06-29) maps `0x17f0 rev 0x11 → dev_npu5_info`, `.fw_path =
+  "amdnpu/17f0_11/"`, `default_vbnv = "RyzenAI-npu5"`, and the revision→name table
+  has `{ AIE2_DEV_REVISION_HALO, "NPU Strix Halo" }`; `xrt-smi examine` therefore
+  reports the device as **NPU Strix Halo**.
+* Linux install recipe (1.8.0 docs
+  <https://ryzenai.docs.amd.com/en/latest/linux.html>): Ubuntu 24.04 packages,
+  **Python 3.12** (`apt install python3.12 python3.12-venv`),
+  `libboost-filesystem1.74.0`, dkms; then `install_ryzen_ai.sh -a yes -p <venv>`;
+  `LD_LIBRARY_PATH=…:${RYZEN_AI_INSTALLATION_PATH}/onnxruntime/lib/`;
+  `quicktest.py` "picks up a simple CNN model, **compiles it** and runs on AMD's
+  Neural Processing Unit" — i.e. **compilation runs on Linux** for the non-LLM
+  flows. "Linux currently supports NPU only flow" (no iGPU EP).
+  Host friction: our box is Ubuntu 26.04 + **Python 3.14** **[host]**, so the venv
+  must be built with a 3.12 interpreter and the packages are 24.04-built
+  `[INFERENCE]`.
+* LLM flow on Linux
+  (<https://ryzenai.docs.amd.com/en/latest/llm_linux.html>): prequantized
+  ready-to-run models from HF, `deployment/lib`,
+  `RYZENAI_EP_PATH=$PWD/deployment/lib/libonnxruntime_providers_ryzenai.so`,
+  run via the `model_benchmark` binary; **"Linux does not support Hybrid flow"**;
+  **"Model generation is not supported in current release"** (repeated in
+  relnotes 1.8 Known Issues: "Models generated on Windows are compatible with
+  Linux"); needs `memlock unlimited`.
+  Sample throughput in that doc: **49.1 tokens/s decode**, 753 tokens/s prefill
+  (Phi-3.5-mini, 4K model).
+* Third-party confirmation of the OS split: Lemonade's backend matrix
+  (<https://raw.githubusercontent.com/lemonade-sdk/lemonade/main/README.md>,
+  fetched 2026-10-19) — `flm`/npu/XDNA2 = **Windows, Linux**;
+  `ryzenai-llm`/npu = **Windows only**; `whispercpp`/npu (STT) = **Windows only**;
+  text-to-speech engines `kokoro` and `acestep` = metal / cuda / vulkan / rocm,
+  **no npu device at all**. Lemonade is the "Server Interface" layer of AMD's own
+  Ryzen AI stack (docs index), so this matrix is the vendor-sanctioned view.
+
+## 4. Can we compile our own graph on Linux? (the no-recipe path)
+
+Mechanics from <https://ryzenai.docs.amd.com/en/latest/modelrun.md> (1.8.0):
+ONNX only, opset 17 recommended; the EP compiles on first `InferenceSession` and
+caches (VitisAI EP cache, or ORT **EP context cache** — `ep.context_enable`, and
+"precompile the EP context model using Python and deploy it using a C++ program"
+is explicitly supported); EP auto-partitions unsupported nodes to CPU;
+BF16 = `target VAIML` (`preferred_data_storage: unvectorized` for transformers),
+BF16 **STX/KRK-or-newer only**; INT8 = `target=X2` with XINT8/A8W8/A16W8 and
+"general asymmetric quantization … to enable third-party quantized models";
+**deployment EP cannot compile BF16 on the fly** — ship a pre-compiled model
+(<https://ryzenai.docs.amd.com/en/latest/app_development.md>);
+cache dirs are not portable across EP or NPU-driver versions.
+
+Operator coverage is the hard gate
+(<https://ryzenai.docs.amd.com/en/latest/ops_support.md>, "operators supported in
+Ryzen AI 1.8.0"; "Y" = broad coverage, some configurations may still fail):
+
+* Good for TTS front-ends/vocoders: `ConvTranspose` **Y in BF16/A16W8/A8W8**
+  (absent only from XINT8), `Conv`, `Resize`, `Upsample`, `Pad`, `LSTM` (BF16),
+  `CumSum` (BF16/XINT8), `STFT` (XINT8 only).
+* **Blockers for a decoder-style transformer in the plain flows:** in **BF16** the
+  table has **no Softmax, no LayerNormalization, no Relu/Gelu**; LayerNorm is
+  XINT8-only and Softmax is A16W8/A8W8/XINT8-only. So attention either lives in
+  the **LLM/OGA flow** (whose op set is exactly
+  `SimplifiedLayerNormalization`, `SkipSimplifiedLayerNormalization`, `MatMulNBits`
+  (W4ABF16/W4ABFP16), `RotaryEmbedding`, **`GroupQueryAttention`**, `Sigmoid`,
+  `Mul`, `Add`) or must be decomposed into a quantized INT8 flow.
+* Missing entirely: `GRU`, `Trilu`, `OneHot`, `EyeLike`, `Multinomial`,
+  `NonMaxSuppression`, `CastLike`, `MelWeightMatrix`/`MelFilterGroup`,
+  `RMSNormalization` → graphs that emit them must be rewritten before they
+  partition onto the NPU. [INFERENCE from absence in the table.]
+* Quantization is a separate tool, not in the driver package: **AMD Quark**
+  (`quark.torch` / `quarkonnx`), which documents `ConvTranspose2d`, `conv1d`,
+  softmax/LayerNorm/BFP quantization (relnotes 1.3/1.2/1.1/0.9) — Quark is *not*
+  on pypi.org under the obvious names **[host probe]** and ships from AMD's
+  gated/pip-private channel.
+* **New LLM-shaped model (our Qwen-Talker analogue): Windows-only today.**
+  OGA model generation is not available on Linux in 1.8.0 (§3). So a *new*
+  speech-LM cannot be self-served onto the NPU on Linux without AMD (or a
+  Windows box) in the loop.
+* Practical shape cost: the neutts artifacts carry **one compiled variant per
+  padded sequence length** (128/256/512/768/1024/1536/2048/2304/3072/4096 for
+  prefill; 256…4096 for token-phase attention mask) — i.e. per-shape compile
+  artifacts are the norm for the LLM flow. [INFERENCE from the HF file list.]
+
+## 5. Does the AMD TTS artifact actually produce speech? (cloned-voice survival)
+
+NeuTTS is **LM + separate neural codec** ("Simple LM + codec architecture"),
+clone from ~3 s of reference audio, and the codec is **NeuCodec (50 Hz, single
+codebook)**: `neuphonic/neucodec` (PyTorch, encode+decode),
+`neucodec-onnx-decoder{,-int8}` (**decode only**), "Decoder-only (ONNX) codecs
+require **pre-encoded references** — these ship with the repo for all bundled
+voices." Phoneme input; 2048-token context ≈ 30 s of audio; NeuTTS-Air ~360 M
+active / 552 M with embeddings; upstream benchmarks are **CPU-only llama.cpp**
+(119 tok/s prefill/decode mix on a Ryzen 9 HX 370) and explicitly exclude the
+codec. <https://raw.githubusercontent.com/neuphonic/neutts/main/README.md>
+(fetched 2026-10-19); `neuphonic/neutts-air` HF metadata (lastModified
+2026-08-26, `qwen2`, GGUF + safetensors).
+
+Consequences for the fleet spike:
+
+1. AMD's NPU artifact = LM only; **waveform decode and reference encoding are CPU
+   work either way**. Our `reference.wav` voices survive only if we run the
+   NeuCodec **encoder** (PyTorch, per the codec table there is no ONNX encoder)
+   once per voice and then feed pre-encoded codes. That is a one-time offline
+   step per voice; after that the compiled graph takes codes/prompt prefix as
+   ordinary inputs, so nothing about the compiled graph forbids a caller-supplied
+   speaker prefix. `[INFERENCE]`
+2. Real-time math: 50 Hz codec means **≥50 tokens/s decode** for 1× RT. AMD's own
+   Linux sample output for a 4K Phi-3.5-mini is **49.1 tok/s** — so budget for
+   roughly RT-1× on one 4K "full fusion" graph, before the codec's CPU cost.
+   `[INFERENCE — different model; must be measured.]`
+3. Host RAM: the same doc reports **peak working set 3.54 GB** for that 4K model.
+   NeuTTS-Air is ~552 M params at W4A-BFP16, so expect a smaller footprint than
+   3.5 GB `[INFERENCE]`; still non-trivial against 14 GiB available **[host]**.
+4. AMD ships **no TTS demo, runner, or server** on any OS: the examples page has
+   only CNN/BF16/DistilBERT/super-resolution/OGA-LLM/VLM/RAG
+   (<https://ryzenai.docs.amd.com/en/latest/examples.md>); the RyzenAI-SW repo
+   tree (669 paths, `main`, fetched 2026-10-19) contains speech demos **only under
+   `Demos/ASR/{Whisper,Parakeet-TDT}` and `Transformer-examples/ASR/Whisper-AI`**
+   — no TTS directory. Serving TTS therefore means writing our own OGA host
+   (or driving `model_benchmark`-style C++ code) plus phonemizer + NeuCodec glue.
+
+## 6. Two graphs on one NPU, and the iGPU hazard
+
+* **Concurrency is a first-class driver feature.** The vendor `amdxdna` source
+  (in `xrt_plugin.2.25.260102.56`, 2026-06-29) sets `.hwctx_limit = 16` for
+  npu4/npu5 — i.e. **16 hardware contexts** on our device — and allocates each
+  context a set of AIE columns with QoS `opc = col_opc (4096) × num_col`
+  (`aie2_ctx.c`, `aie2_solver.h`), with `frame_boundary_preempt` /
+  `fill_preempt_slot` in `aie2_pci.h` (firmware preemption at frame boundaries).
+  So halogen's armed models and a TTS graph can hold separate contexts
+  simultaneously; they contend for columns and OPC share, they are not mutually
+  exclusive.
+* AMD's user-visible history agrees: 1.4 "Support for up to **16 hardware
+  contexts** on Ryzen AI 300 series"; 1.5 "Application concurrency: improves the
+  resource distribution across applications"; 1.3 "Idle detection and suspension
+  of contexts", "Rebalance feature for AIE hardware resource optimization";
+  1.0 known issue "Support for multiple applications is limited to up to eight"
+  (all in <https://ryzenai.docs.amd.com/en/latest/relnotes.html>).
+* **Package conflict, the real risk** [INFERENCE]: the vendor stack wants AMD's
+  XRT 2.25.37 + `xrt_plugin…amdxdna` deb **[host: distro `libxrt-npu2
+  1:2.21.75+dfsg-4` installed]**, and firmware must match the loaded `amdxdna.ko`
+  — the vendor README warns that a stale/mismatched `npu.sbin` causes
+  `ERT_CMD_ABORT`/mailbox timeouts, and that a distro **in-tree** `amdxdna` older
+  than the XRT SHIM makes `GET_ARRAY`/`QUERY_TELEMETRY` return
+  `EOPNOTSUPP`/`EINVAL` (<https://github.com/amd/xdna-driver> README,
+  fetched 2026-10-19). Installing AMD's debs over the distro XRT changes the ABI
+  under FastFlowLM/halogen, so the spike must decide the XRT version **once**, for
+  the whole host.
+* **iGPU / fabric clock:** no vendor document supports a fabric-clock pin, and no
+  vendor document addresses NPU+iGPU concurrency hangs. AMD's Strix Halo
+  optimization doc (ROCm docs, dated 2026-02-20, and its successor "AMD RDNA3.5
+  system optimization", ROCm 10.1.0) covers GPUVM/GART/GTT, the TTM `pages_limit`
+  knob, `amd-ttm`, BIOS VRAM sizing, and minimum kernel versions
+  (<https://rocm.docs.amd.com/en/docs-7.2.0/how-to/system-optimization/strixhalo.html>,
+  <https://rocm.docs.amd.com/en/latest/reference/system-optimization/rdna3-5.html>).
+  The `pp_dpm_fclk` level-7 pin and the GPU+NPU hang observation stay
+  fleet-observed **[host]**, i.e. `[INFERENCE]` territory — no citation exists.
+
+## 7. Paths I checked and ruled out (so nobody re-walks them)
+
+* **Windows-only doors**: DirectML; Ryzen AI Windows EP; Lemonade's
+  `ryzenai-llm` and `whispercpp`-NPU engines (matrix above); AMD's OGA model
+  generation. Not usable on Dankclaw.
+* **Mainline ONNX Runtime** has no AMD-NPU EP: its EP summary lists AMD ROCm and
+  AMD MIGraphX (GPU) and "Xilinx Vitis-AI (*preview*)" only
+  (<https://onnxruntime.ai/docs/execution-providers/>, 2026-10-19); the working
+  EP is AMD's own build (`libonnxruntime_providers_ryzenai.so` in the tarball's
+  deployment dir, per llm_linux).
+* **ggml / llama.cpp / qwentts.cpp**: no XDNA backend → the current engine
+  (`tts-server` on GGUF, CPU-only build **[host]**) cannot be "turned on" for the
+  NPU; migration means a different runtime, not a flag.
+* **`flm`**: catalog/engines are closed per-architecture (§2) — the cheapest door
+  by install cost, and it is not a door for TTS.
+* **IRON / mlir-aie** is the only open toolchain that documents running on the
+  Ryzen AI NPU from Linux ("A close-to-metal Python API for programming AMD Ryzen™
+  AI NPUs, built on an open-source MLIR-based compiler toolchain"; "Getting
+  Started for AMD Ryzen™ AI on Linux", Ubuntu 24.04/24.10, XRT from
+  `ppa:amd-team/xrt`: `libxrt2 libxrt-npu2 … amdxdna-dkms`, `render` group
+  required) — <https://raw.githubusercontent.com/Xilinx/mlir-aie/main/README.md>,
+  fetched 2026-10-19. This is how FLM's kernels were made ("Low-level kernels
+  optimized using IRON + AIE-MLIR", FLM README) — i.e. real, but it is
+  hand-written per-op tile programming, not a model compiler: writing a TTS model
+  here is a multi-month engineering project, not a migration.
+* **Embedded** Ryzen AI P100 support in the same Linux doc is irrelevant to STX.
+
+## 8. Recommendation implied by the above (for Main's spike)
+
+Spike the AMD path as decided, with these constraints planned in: Python 3.12 venv
++ 24.04-built wheels on Ubuntu 26.04; one host-wide XRT decision (AMD 2.25.37 vs
+distro 2.21.75) with firmware/driver pairing checked via `xrt-smi examine` →
+"NPU Strix Halo"; 7.8 GiB tarball staged off the public CDN URL (§3); pre-encode
+each `reference.wav` with the **PyTorch** NeuCodec encoder because the ONNX codec
+is decoder-only; keep phonemizer + codec on CPU; measure decode tokens/s against
+the 50 Hz budget and peak RSS against the ~14 GiB headroom; expect to write the
+serving shell yourself (AMD ships `model_benchmark`, no TTS demo).
+
+## 9. Loose ends (not verified; I stopped on Main's stand-down)
+
+* Contents of `ryzen_ai-1.8.0.tgz` (7.8 GiB) — whether the Linux tarball contains
+  the AIE-ML compiler (`vaiml`, `aiecompiler_client`) for the *BF16/INT8* flows
+  and which Python wheels it carries. My statement that non-LLM compilation works
+  on Linux rests on AMD's own Linux doc ("quicktest.py … compiles it and runs on
+  … NPU"), not on the package contents. Download was killed mid-flight.
+* Whether AMD's `neutts` artifacts run correctly under the Linux `model_benchmark`
+  runner (the TTS rows are listed without OS qualifiers; no Linux TTS run is
+  documented anywhere in 1.8.0 docs).
+* Exact AIE column count/total `ncols` exposed for Strix Halo (driver reads it
+  from AIE metadata: `ndev->total_col = min(XRS_MAX_COL, ndev->aie.metadata.cols)`).

report/qwen3-tts-payload-portability-2026-10-19.md

@@ -0,0 +1,438 @@
+# Qwen3-TTS payload + qwentts.cpp runtime: architecture and portability
+
+Research deliverable (no code). Slice: *what the fleet's TTS payload actually is, and how
+portable it is off the CPU ggml path.* Companion slice: `NpuRuntimeLandscape` (what NPU
+runtimes exist on Dankclaw).
+
+Evidence date: 2026-10-19. Everything below is grounded in:
+
+- `github.com/ServeurpersoCom/qwentts.cpp` @ `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`
+  (2026-10-16, upstream HEAD on 2026-10-19) and its ggml submodule
+  `ServeurpersoCom/ggml` @ `cc5df49a6b878db2b68b9a7b45976e279594627d` (2026-10-16).
+- The two GGUF files as they sit on Dankclaw (`/home/dank/models/Qwen3-TTS-GGUF/`), read
+  byte-for-byte from the live host.
+- Upstream Qwen3-TTS model cards / release material.
+
+Status: COMPLETE for the assigned slice. ANSWER C was confirmed on the live host the same
+night — the running `frank-tts-engine` container carries `libggml-base.so.0.27.0` +
+`libggml-cpu.so.0.27.0` and nothing else, and our own
+`dankclaw/voiceservice/Containerfile.engine` writes `-DGGML_VULKAN=OFF` out explicitly, so
+the iGPU change is a flag in a file the fleet owns. Verdict, measured contract and the NPU
+exoneration chain: `tts-npu-spike-verdict-2026-10-19.md`.
+
+---
+
+## 1. What the payload is
+
+Two files, both read byte-for-byte off the live host (`/home/dank/models/Qwen3-TTS-GGUF/`,
+owned `dank:dank`, mtimes 2026-10-18 18:31), both from
+`https://huggingface.co/Serveurperso/Qwen3-TTS-GGUF` (community converter, not Qwen):
+
+| file | bytes | GGUF | tensors | arch string | file_type |
+|---|---|---|---|---|---|
+| `qwen-talker-1.7b-base-Q4_K_M.gguf` | 1,219,245,248 | v3, kv=64 | 480 | `qwen3-tts` | `Q4_K_M` |
+| `qwen-tokenizer-12hz-Q8_0.gguf` | 291,150,624 | v3, kv=51 | 398 | `qwen3-tts-tokenizer` | `Q8_0` |
+
+`general.name` of the talker file is `Qwen3-TTS-12Hz-1.7B-base`; the header sizes add up
+to the file sizes (sum of tensor bytes + header + alignment padding), so nothing in these
+files is unaccounted for.
+
+### 1.1 Talker file (480 tensors)
+
+Three models in one container, per `docs/ARCHITECTURE.md` §GGUF layout
+(qwentts.cpp @ pinned commit) and confirmed by the tensor census:
+
+| prefix | tensors | bytes | what it is |
+|---|---|---|---|
+| `talker.*` | 316 | 1,061.9 MB | 28-layer Qwen3 decoder (hidden 2048, FFN 6144, 16 Q heads / 8 KV heads, head_dim 128, RoPE θ=1e6, RMS ε=1e-6, ctx 32768), text stream + codec stream summed, `codec_head` → 3072 codebook-0 logits |
+| `code_pred.*` | 88 | 114.8 MB | 5-layer Qwen3 MTP head (hidden 1024, FFN 3072, 15 acoustic codebooks × 2048), own KV cache |
+| `spk_enc.*` | 76 | 36.6 MB | ECAPA-TDNN speaker encoder (x-vector, base checkpoints only; mel → conv0 → 3 SE-Res2Net → MFA → ASP → 2048-d vector) |
+
+Dtype split (exact tensor byte sums): `q4_K` 250 tensors / 864.1 MB, `q6_K` 17 / 312.0 MB,
+`f32` 176 / 25.8 MB, `f16` 37 / 11.4 MB.
+
+Which tensors carry which quant is the load-bearing portability fact:
+
+- `q6_K` (17): `talker.text_embd.weight [2048, 151936]` (255.3 MB), `talker.codec_embd.weight
+  [2048, 3072]`, and the 15 `code_pred.codec_embd.<g>.weight [2048, 2048]` embedding tables.
+  These are the llama.cpp `Q4_K_M` "embedding/output rows go to Q6_K" rule, applied to
+  embedding tables rather than to an `lm_head`.
+- `q4_K` (250): every attention/FFN projection of the talker and of the code predictor.
+  Distinct matmul shapes: `[2048,2048]`×73, `[2048,1024]`×62, `[2048,6144]`×56,
+  `[6144,2048]`×28, `[1024,2048]`×20, `[1024,3072]`×10, `[1024,1024]`×10, `[3072,1024]`×5,
+  `[2048,3072]`×2, `[2048,151936]`×1. Every one is 2-D and ≥ 1024 on both axes — good
+  GEMM shapes, nothing exotic.
+- `f32` (176) / `f16` (37): all of `spk_enc` (conv weights F16 because K=1/3/5 rows are not
+  block-alignable, per the converter's `should_quantize` policy in `tools/quantize.cpp` +
+  `quantize.sh`; every bias/norm/LayerScale vector F32), plus the talker/predictor norms and
+  the `text_proj` biases.
+
+Metadata the runtime actually reads (talker file): `qwen3-tts.num_code_groups=16`,
+`talker.vocab_size=3072`, `text_vocab_size=151936`, `text_hidden_size=2048`,
+`position_id_per_seconds=13`, `rope.mrope_section=[24,20,20]`, `mrope_interleaved=1`,
+the 7 codec special ids (pad 2148 / bos 2149 / eos 2150 / think 2154 / nothink 2155 /
+think_bos 2156 / think_eos 2157), `language_names` (10: chinese, english, german, italian,
+portuguese, spanish, japanese, korean, french, russian) with their codec ids, the text-side
+special ids (im_start 151644, im_end 151645, tts_pad 151671, tts_bos 151672, tts_eos 151673),
+the whole `generation_config.json` (do_sample, top_k 50, top_p 1.0, temperature 0.9,
+repetition_penalty 1.05, `subtalker_*` mirrors, max_new_tokens 8192), and an embedded
+**Qwen2 BPE tokenizer** (`tokenizer.ggml.model=gpt2`, 151,676 tokens, 151,291 merges,
+eos 151643) — that BPE block is 5.95 MB of the header, i.e. 49 % of the header is vocabulary.
+
+Two live-file details that contradict the upstream doc text (worth knowing when reading
+`docs/ARCHITECTURE.md`: the doc example block is stale against the shipped file):
+`mrope_interleaved` is **1** (doc shows `false`), and `spk_enc.sample_rate` is **24000**
+(doc says 16000). Doc also says "404 tensors on the 1.7B voice_design build" — the base
+build shipped here has 480 because it carries `spk_enc.*`.
+
+### 1.2 Tokenizer/codec file (398 tensors)
+
+One neural audio codec, 12.5 frames/s over 24 kHz mono, hop 1920, 16 codebooks × 2048
+entries (1 semantic + 15 acoustic). Dtype split: `f16` 50 / 143.6 MB, `q8_0` 110 / 76.9 MB,
+`f32` 238 / 70.7 MB. `tok_dec.*` 253 tensors / 201.3 MB, `tok_enc.*` 145 / 89.9 MB.
+
+Composition (metadata + tensor census, both files agree):
+
+- **Encoder** (used only to make voice references, not on the synthesis hot path): SEANet
+  conv stack (init k=7, 4 stages with ratios 8/6/5/4 → cumulative 960×, 64→512 ch, final
+  k=3), an 8-layer transformer (hidden 512, 8 heads, head_dim 64, FFN 2048, RoPE 1e4,
+  LayerNorm+bias, GELU, LayerScale), a downsample conv k=4 s=2 (25 → 12.5 Hz), then the RVQ
+  argmin loop. Metadata records `encoder.num_quantizers=32` with
+  `encoder_valid_num_quantizers=16` — 16 codebooks are used, 16 are dead weight carried from
+  the upstream config.
+- **Decoder** (the hot path: every generated frame): 16 codebook lookups + per-split
+  `output_proj` (256→512), `conv_pre`, an 8-layer transformer (hidden 512, 16/16 heads,
+  head_dim 64, FFN 1024, **sliding window 72**, RMSNorm, SwiGLU, LayerScale, input_proj
+  1024→512 / output_proj 512→1024), a ConvNeXt upampler (`upsampling_ratios=[2,2]`, causal
+  transposed convs + depthwise k=7), then a DAC-style decoder `dec.0..3`
+  (`upsample_rates=[8,5,4,3]`, channels 1536→768→384→192→96, `decoder_dim=1536`) whose blocks
+  are SnakeBeta → causal transposed conv (k = 2·stride) → 3 residual units (dilations 1/3/9)
+  → snake_post → conv_post k=7 → 96→1.
+- All 32 codebook tensors are **F32, 67.1 MB total** (16 encode + 16 decode, `[256, 2048]`),
+  deliberately unquantized: the converter's `should_quantize` keeps RVQ codebooks and their
+  wrapping projections at F32 because nearest-neighbour lookup is sensitive to per-row
+  quantisation noise (`docs/ARCHITECTURE.md` "Quantisation policy"; mirrored in
+  `quantize.sh`). 49 of the codec's 55 3-D conv tensors are **F16** with non-alignable K
+  (7, 3, 2, 1, 16, 10, …) — they cannot be block-quantized at all.
+- **No flow matching, no diffusion, no vocoder-with-ODE.** It is a pure RVQ codec
+  (Mimi/SoundStream/Mimi-style: SEANet + transformer + DAC decoder). That matters for the
+  NPU question: every component is convolution, attention, or embedding lookup.
+
+### 1.3 Where the payload came from, and what the official weights look like
+
+Official upstream (`huggingface.co/api/models`, retrieved 2026-10-19):
+
+| repo | last modified | license | weights |
+|---|---|---|---|
+| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | 2026-01-23 | apache-2.0 | `model.safetensors` BF16, 1,928,677,440 params, plus an embedded `speech_tokenizer/model.safetensors` |
+| `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | 2026-01-29 | apache-2.0 | same layout, named-speaker tables |
+| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | 2026-01-29 | apache-2.0 | 1.7B only |
+| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` / `-0.6B-CustomVoice` | 2026-01-29 | apache-2.0 | narrow variant (hidden 1024) |
+| `Qwen/Qwen3-TTS-Tokenizer-12Hz` | 2026-01-29 | apache-2.0 | codec alone, F32, 170,557,441 params |
+
+So the GGUF is **not** the only public form: the safetensors are the canonical form
+(Apache-2.0, no separate community license risk), and the GGUF is a lossy re-quantization of
+them produced by the same author who wrote the runtime (`Serveurperso`), via
+`checkpoints.sh` → `convert.py` (F32 GGUF) → `quantize.sh` (BF16/Q8_0/Q4_K_M).
+Sizes to remember for any re-export: the talker container is 1.93 B params in BF16
+(that count includes the embedded codec), the codec alone is 170.6 M params in F32.
+
+**Portability consequence #1 — the GGUF is a dead end off-ggml.** Q4_K/Q6_K are ggml
+block formats (256-element super-blocks, F16 sub-block scales, 144 B / 210 B per block). No
+ONNX graph, no Vitis-AI/AIE-ML compiler, and no vendor NPU runtime consumes them; AMD's own
+quantizer works from ONNX (`ryzenai-quantize`, QDQ/QOP power-of-two — Ryzen AI Software
+1.8.0 release notes, https://ryzenai.docs.amd.com/en/latest/relnotes.html). Moving the
+payload to a non-ggml stack therefore means starting from the safetensors again, not
+converting the GGUF.
+
+**Portability consequence #2 — the quantization split is semantic, not cosmetic.** The
+converter keeps RVQ codebooks, `input_proj`/`output_proj` around them, and every 1-D and
+non-alignable-conv tensor out of the k-quants precisely because quantizing them breaks voice
+fidelity. Any re-quantization path (GPTQ/AWQ/QDQ-INT8/INT4 for an NPU compiler) inherits that
+constraint: the talker backbone is requantizable, the codec's codebook lookups and SnakeBeta
+convs are not, at least not without re-measuring voice quality.
+
+## 2. What the runtime is
+
+`qwentts.cpp` @ `5916bf7c` (2026-10-16, MIT; upstream model Apache-2.0) is a C++17 ggml
+application — 43 files under `src/` + `tools/`, three binaries (`qwen-tts`, `qwen-codec`,
+`tts-server`) over one public plain-C99 ABI (`src/qwen.h`, `qt_*`, `QT_ABI_VERSION`
+range-checked; `tests/abi-c.c` is compiled `-std=c99 -Wall -Werror -pedantic` on every
+build). It is not a llama.cpp derivative: it links ggml directly and builds its own graphs.
+
+Five graph families, and their shapes are the whole portability story:
+
+| graph | batch/time axis | notes |
+|---|---|---|
+| talker prefill | T = prompt length (hundreds–thousands) | summed text+codec embedding streams, 28-layer GQA decoder, KV cache |
+| talker decode | **T = 1 per audio frame** | 12.5 frames/s → 80 ms of audio per step; `codec_head` → 3072 logits |
+| code predictor (MTP) | **T = 1, 16 steps per frame** (2 prefill + 14 decode, cache reset every frame) | 5-layer decoder, 15 heads × 2048 |
+| speaker encoder | one pass per voice registration | ECAPA-TDNN with mel fused into the graph |
+| codec decode | offline: T = 300-frame chunks; streaming: **T = 1 frame** with persistent conv/KV-ring state | the streaming graph is built once with fixed topology so the CUDA graph cache stays in pure replay |
+
+Per generated frame the host therefore runs ~1 + 16 + 1 tiny graph computes for 80 ms of
+audio, and `tts-server` serializes synthesis FIFO (`--max-batch` defaults to 1, and the
+fleet runs it at 1). This is the archetypal latency-bound, bandwidth-bound,
+batch-1 workload; the only big-M matmuls in the whole system are the prefill and the
+chunked (buffered) codec decode.
+
+Op contract actually issued to ggml (`src/`, counted): `mul_mat` 77, `cont` 70, `add` 57,
+`mul` 40, `rms_norm` 22, `clamp` 15, `rope_ext` 12, `get_rows` 12, `set_rows` 8,
+`scale_bias` 6, `swiglu_split` 5, `sqr` 5, `mean` 5, `elu` 4, `col2im_1d` 4, `soft_max` 3,
+`conv_1d` 3, `im2col` 2, `cumsum` 2, `sin`, `sigmoid`, `tanh`, `div`, `argsort`, `transpose`,
+`permute`, `reshape/view`, plus the CPU-side Philox multinomial sampler.
+
+**No custom ggml op is required today.** The repo doc still says the fork "provides the two
+custom ops the codec needs: `GGML_OP_SNAKE` and `GGML_OP_COL2IM_1D`"; that is stale at the
+pinned commit. Verified 2026-10-19:
+
+- the submodule `ServeurpersoCom/ggml` @ `cc5df49` is upstream `ggml-org/ggml` master
+  (`a05c992`, 2026-10-15) **+2 commits, 0 behind** (GitHub compare API), and
+  `include/ggml.h` is byte-identical to upstream master (`diff` empty);
+- the 2 commits are `193e339` (2026-06-19) "cpu: add fused snake activation
+  (mul -> sin -> sqr -> mul -> add)" — a **CPU subgraph fusion**, not an op (the runtime
+  builds snake with `ggml_sin`/`ggml_sqr`/`ggml_mul`/`ggml_add` in `src/dac-decoder-v2.h:222`),
+  and `cc5df49` "sycl: fix get_rows on reordered quantized weights";
+- `ggml_col2im_1d` is an **upstream ggml API** (same declaration, line 2097, in both trees);
+  the Vulkan backend fuses the snake chain through ggml's generic subgraph fuser
+  (`ggml-vulkan.cpp:14548 ggml_vk_can_fuse_snake`, `vulkan-shaders/snake.comp`).
+
+Consequence: any stock ggml build (upstream, or llama.cpp's vendored ggml) can execute this
+payload; only the CPU snake fusion is fork-specific, and it is a performance nicety.
+
+Backend/device support: every tool takes `--device <name>` (`CUDA0`, `Vulkan0`, `MTL0`,
+`CPU…`, default best available), resolved through the ggml backend registry; the deployed
+container turns CUDA/HIP/Vulkan/Metal/SYCL/RPC/BLAS all OFF and compiles the native-CPU
+backend with OpenMP, with CPU threads = `hardware_concurrency()/2` (`src/backend.h`).
+The registry in this ggml knows CPU, CUDA, Metal, SYCL, Vulkan, WebGPU, ZDNN, OpenCL,
+**Hexagon**, BLAS, RPC, VirtGPU, **CANN**, ZenDNN, **OpenVINO**, ET — and nothing AMD-NPU
+(no `xdna`/`aie`/`vitis`/`ryzen` string anywhere in the ggml tree).
+
+Voice handling (what the fleet depends on): the registry is **process RAM only**; a restart
+empties it, which is why the unit has a boot warmer. `POST /v1/audio/voices` accepts either
+`{name, ref_text, wav_b64}` (server extracts) or `{name, ref_text, spk_b64, rvq_b64}`
+(pre-extracted latents verbatim); `ref_text` present ⇒ ICL clone mode, absent ⇒ x-vector-only
+mode. `.spk` = the ECAPA x-vector; `.rvq` = 16 codes/frame packed 11 bits LSB-first
+(`src/rvq-file.h`). `wav` and `pcm` `response_format` are different audio at the same seed
+(buffered chunked decode vs stateful streaming decode) — the fleet already encodes that rule.
+
+Parity expectations for any port: the upstream harness matches the PyTorch reference per
+stage (embed cos 0.999988, all 28 hidden taps ≥ 0.999893, logits 0.999824) but explicitly
+reports free-run divergence (`CodesFull exact 4.96 %`, `Audio cos 0.095`) and explains why —
+one argmax tie at FP epsilon diverts the AR trajectory. So a migrated endpoint must be
+accepted by listening + per-stage cosine, never by waveform diff against today's audio.
+
+---
+
+## 3. The three answers Main asked for
+
+### ANSWER A — Qwen3-TTS GGUF → ONNX: no, and it does not matter
+
+There is **no GGUF→ONNX path** and there will not be a useful one: GGUF is a weight
+container for ggml, its Q4_K/Q6_K block layouts have no ONNX consumer, and nothing in the
+GGUF carries the *graph* (the talker/predictor/codec structure lives in C++ code, only the
+op-level structure exists in ggml at runtime). But the GGUF is not the payload's only form —
+official Apache-2.0 safetensors are upstream, and **four independent ports of the full
+pipeline already exist**:
+
+| port | form | what it proves |
+|---|---|---|
+| `xkos/Qwen3-TTS-12Hz-1.7B-ONNX` (HF, updated 2026-03-03, apache-2.0) | FP16+FP32 ONNX, fp16/voice_clone + fp16/voice_design splits: `talker_decode.onnx`, `code_predictor.onnx`, `code_predictor_kv.onnx`, `code_predictor_embed_g0..15.onnx`, `text_embedding.onnx`, `codec_embedding.onnx`, shared `speaker_encoder.onnx`, `speech_tokenizer_encoder.onnx`, `speech_tokenizer_decoder.onnx`, `model_cache.npz` | every component of this architecture exports to ONNX, codec decoder included |
+| `zhangluoyang/qwen3-tts-onnxruntime` | export tooling + Python inference for Base/CustomVoice/VoiceDesign | the export is reproducible from the official checkpoints |
+| `Echo9Zulu/Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO`, `aurora2035/Qwen3-TTS-12Hz-0.6B-Base-OpenVINO-INT8` (2026-03) | OpenVINO IR, **INT8**: `talker`, `code_predictor`, `codec_embedding`, `text_model`, `openvino_speaker_encoder_model`, `speech_tokenizer/speech_decoder` | the talker and predictor INT8-quantize for an NPU-targeting stack |
+| `litert-community/Qwen3-TTS-12Hz-0.6B-Base` (2026-10-17, apache-2.0) | `qwen3_tts_int8.litertlm` / `_int4.litertlm` (LiteRT-LM `OmniEngine`), legacy split graphs `talker_int4.tflite` (prefill_32/prefill_128/decode signatures, KV 1024, blockwise-32 OCTAV INT4), `mtp_folded_int8.tflite` (16 inner steps × 5 layers folded into one graph, in-graph argmax+gather, GPTQ int8, ~5× faster), `codec_partA` (transformer, fp32) + `codec_partB` (conv upsampler, XNNPACK FORCE_FP16, ~2.5×, "ASR-identical") | an NPU/mobile compiler accepts the whole thing — at the price of graph splitting, static prefill buckets, and folding the per-frame inner loop |
+
+What the ports had to concede, and what any port inherits:
+
+- the generation structure does not fit existing engine loops: LiteRT-LM's Engine "does not
+  support this generation structure yet", so their first shape was **three graphs + a
+  host-side loop**, and the MTP step was "invoked 17× per frame" until they folded it;
+- per-step dispatch cost dominates: even on a Snapdragon 8 Elite Gen 5 they keep the MTP
+  loop, text projection, embedding gathers and the codec on CPU "avoiding 16 synchronous
+  CPU↔GPU round-trips per 80 ms frame", putting only the 28-layer talker on the GPU;
+- RTF on that hardware is **2.57–2.97 (Galaxy S26) and 5.65–8.48 (Pixel 10 Pro)** for the
+  0.6 B model, i.e. 2.6–8× slower than real time. Dankclaw's current CPU engine measures
+  **warm RTF 0.55–0.67 with TTFA 94–112 ms** on the 1.7 B model (fleet trial, 2026-10-18).
+  An NPU port is not "faster by default" for this workload; it has to beat that.
+- quantize the LM (int8/int4: proven), keep the codec's codebook lookups and SnakeBeta convs
+  in fp16/fp32 (the LiteRT split does exactly this).
+
+So: the answer to "can the fleet's payload be exported" is *yes — from the safetensors, not
+from the GGUF*, and the work is a pipeline rewrite (multi-graph host loop, per-frame fold,
+static prefill buckets), not a conversion. That is the same conclusion NeuTTS-Air's
+precompiled-graph route encodes: on AMD's Linux stack the graph is vendor-curated, so the
+fleet chooses the model that AMD already compiled rather than compiling Qwen3-TTS.
+
+### ANSWER B — is there any ggml backend for an AMD NPU? Not upstream; three out-of-tree, none for TTS
+
+Checked against the trees today (2026-10-19), because this was the crux:
+
+- **Nothing in-tree.** `ggml-org/llama.cpp` master (4,237 paths, GitHub tree API) contains no
+  `xdna`/`aie`/`vitis`/`ryzen` file or directory; the ggml backend set is CPU, BLAS, CUDA,
+  HIP, Metal, Vulkan, SYCL, OpenCL, WebGPU, MUSA, RPC, CANN (Ascend), **Hexagon** (Qualcomm),
+  OpenVINO (Intel CPU/GPU/NPU), ZenDNN, ZDNN, VirtGPU, ET. The deployed fork's registry
+  matches. The feature request "Feature Request: XDNA backend" (llama.cpp issue **#21725**,
+  opened 2026-04-10) is still **open**, 34 comments; the older #1499 was closed stale
+  2025-04-07. So Main's premise is confirmed: no ggml XDNA backend exists to flip a flag on.
+- **`Cyronius/ggml-xdna`** — a real ggml backend (`XDNA0`) loaded into *stock* llama.cpp via
+  `GGML_BACKEND_PATH`, no fork. It does bfp16 (8-value blocks sharing an exponent) GEMMs from
+  GGUF weights, and is **prefill-only**: it takes only matmuls in prompt chunks ≥ 1024 tokens
+  (`GGML_XDNA_MIN_BATCH` default 1024, `GGML_XDNA_MIN_MFLOP` 256), attention and every decode
+  token stay on the Vulkan device; pinned to llama.cpp release b10944; tested on Strix Point
+  (HX 370) only — Strix Halo "should pass, but nobody has run it on one yet"; Linux support
+  new and untested on hardware by the author; needs `libxrt-npu2`, `/dev/accel/accel0`,
+  unlimited `memlock`. Best measured prefill gain 1.88× on a 27 B model; models under
+  2,048-wide **lose** (0.77× at width 1,536).
+- **`BrandedTamarasu-glitch/OllamaAMDNPU`** — a llama.cpp fork with `-DGGML_XDNA=ON`,
+  explicitly targeting **Ryzen AI MAX / XDNA2 on Linux** (`/dev/accel/accel0`, XRT 2.21.75+,
+  amdxdna, unlimited memlock). It implements **only `GGML_OP_MUL_MAT`**, dispatched to
+  mlir-aie-compiled `.xclbin`s built per fixed tile shape (e.g. `M=2048 K=2048 N=64`,
+  int8 in / int32 out, per-row weight scales cached at first use), with a second slot for a
+  different K. Its own numbers on a Ryzen AI MAX 385 with Llama-3.1-8B Q4_K_M: NPU prefill
+  12–19 t/s vs **Vulkan iGPU 600–930 t/s**, NPU decode **0.65–1.40 t/s** ("Decode does not
+  use the NPU … M=1 per token, no xclbin covers it"), 0.6 % AIE utilization, and the stated
+  conclusion "**For LLaMA 8B on Strix Halo: use Vulkan exclusively**". Single-maintainer,
+  AI-assisted, self-measured — treat as a proof of concept, but it is the closest published
+  picture of what XDNA2 graph execution costs.
+- Also referenced in the same thread: `Atomic-Germ/OpenFlowLM-Next` (whole-model NPU engine,
+  LLM-only — and slower than the iGPU in the same comparison), `GaTmaNnes/xdna2-`,
+  `hardWorker254/Npu-Embeddings-XDNA1` (adds Whisper ASR ops, NPU1/Linux). None is a TTS path.
+
+Why none of them would carry this workload even if installed: the TTS hot loop is **batch-1
+by construction** (one talker step + 16 predictor steps + one codec frame per 80 ms), which
+sits under every dispatch gate those backends define (≥1024 tokens / ≥32768 M·N·K / tile M
+covering only prefill), and the codec half of the pipeline is conv, col2im, gather and
+elementwise ops — not `MUL_MAT` — so it would fall back to CPU op-by-op. The NPU case for
+this host is therefore *not* "run Qwen3-TTS on the NPU"; it is "run a vendor-compiled AR TTS
+graph on the NPU", which is exactly the NeuTTS-Air spike.
+
+### ANSWER C — would `qwentts.cpp` build for the gfx1151 iGPU? Yes, via **Vulkan**, not HIP
+
+- Upstream ships the build scripts and claims the backends: `buildvulkan.sh` ("AMD/Intel GPU
+  (Vulkan)"), `buildcuda.sh`, `buildsycl.sh`, `buildcpu.sh`, `buildall.sh` (all backends,
+  runtime dynamic loading), and README line 6 "runs on CPU, CUDA, Metal, Vulkan". CMake knobs
+  `GGML_VULKAN` / `GGML_HIP` / `GGML_SYCL` are the same ones the fleet's Containerfile turns
+  OFF, so the engine rebuild is a flag change: `-DGGML_VULKAN=ON` plus
+  `--device Vulkan0` on the same command line the unit runs today.
+- The codec's exotic ops are covered by the Vulkan backend, which is the thing that decides
+  whether a GPU build is coherent: `vulkan-shaders/col2im_1d.comp`, `snake.comp`,
+  `swiglu*.comp`, `cumsum*.comp` exist and `supports_op` handles `GGML_OP_COL2IM_1D` /
+  `CONV_TRANSPOSE_1D` / `CUMSUM`. By contrast `COL2IM_1D` appears in **no** HIP, OpenCL,
+  WebGPU, Hexagon or CANN source — a **HIP** (ROCm) build would shard the ConvNeXt/DAC
+  transposed convs back to the CPU per op. On an AMD iGPU the portable path is RADV/Vulkan.
+- `--device` also accepts a multi-device split (`-dev`-style naming is ggml's), and the
+  runtime's documented FP16 guard (`--clamp-fp16`) exists for sub-Ampere CUDA accumulation;
+  nothing in the tree is NVIDIA-specific.
+- Unverified by me: any vendor statement pairing this fork with gfx1151. Supporting evidence
+  that the hardware/driver combination is viable: this very host already runs a Vulkan
+  endpoint on the gfx1151 iGPU (fleet ground truth, 2026-10-19), and Mesa's Vulkan
+  implementation is officially supported on Radeon Software for Linux
+  (AMD release notes RN-AMDGPU-UNIFIED-LINUX-25.20.3). A gfx1151 Vulkan build of this fork is
+  a build-and-measure spike, not an unsupported port.
+- The real constraint is the one already in the fleet docs: the iGPU belongs to another
+  endpoint, and GPU+NPU concurrency has hung Strix Halo machines. An iGPU TTS build competes
+  with the text endpoints for the same silicon; the NPU route's only genuine advantage is
+  that it does not (the XDNA backend authors make the same argument: "Use NPU when the iGPU
+  is busy … no GPU contention").
+
+---
+
+## 4. What any replacement must reproduce, and what the voices cost
+
+Endpoint contract in production today (from the live unit/env, not the doc):
+`tts-server --model qwen-talker-1.7b-base-Q4_K_M.gguf --codec qwen-tokenizer-12hz-Q8_0.gguf
+--alias frank-tts-engine --host 0.0.0.0 --port 8880 --lang auto --max-batch 1`, no published
+port (the façade on :7772 dials the private bridge name), `GET /health`, `GET /v1/models`,
+`GET|POST /v1/audio/voices`, `DELETE /v1/audio/voices/{name}`, `POST /v1/audio/speech` with
+`response_format` wav|pcm (different audio at equal seed) and optional `seed`,
+`max_new_tokens`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `subtalker_*`.
+Voice registry is RAM-only, so boot warming is mandatory, and `--alias` keeps the quantization
+choice out of every consumer's config.
+
+Live voice census on Dankclaw (2026-10-19, `/home/dank/voices/`): three packs
+(`narrator1..3`), each carrying `reference.wav` (24 kHz mono crop), `reference16k.wav`,
+`manifest.json` (keys: name, kind, generated_utc, pipeline, source, forbidden_region_guard,
+silence_scan, crop, loudnorm, audio, acceptance, acceptance_passed) and ffmpeg analysis
+artifacts. **No `.spk` and no `.rvq` exist on the live host** — the repo's
+`build-voice-pack.sh` treats them as required-but-they-are-not-there, so today the engine
+re-extracts the x-vector (and the ICL codes when a transcript is supplied) from the WAV at
+registration time.
+
+That is the portable part, and it is the same shape AMD's NeuTTS flow wants
+(`encode_reference(reference.wav)` + a reference transcript): the fleet's voice identity is
+*the WAV plus its transcript*, processed at request/registration time by the model's own
+speaker encoder + codec — **never a trained per-voice artifact**. Qwen3-TTS cloning is
+inference-time conditioning (`generate_voice_clone(ref_audio=…, ref_text=…, x_vector_only_mode=…)`;
+`create_voice_clone_prompt` exists purely to avoid recomputing prompt features). So any
+replacement that consumes a WAV (+transcript) keeps all three narrators alive with no
+re-derivation, and if it wants precomputed latents instead, the LiteRT precedent shows what
+that looks like: a 4 KB x-vector `.npy` per voice, extracted offline
+(`extract_speaker_embedding.py`) — for the 1.7 B base checkpoint that vector is 2048-dim
+(`qwen3-tts.spk_enc.embedding_length = 2048` in the shipped GGUF; the LiteRT 0.6 B bundle
+ships a 1024-dim one).
+
+One trap for a mixed-runtime future: `.spk`/`.rvq` files, if they are ever written, are
+engine-specific encodings (ECAPA x-vector layout; 11-bit-packed RVQ stream) and are **not**
+interchangeable between runtimes. Keep `reference.wav` as the pack's source of truth and
+regenerate anything derived per engine — which is what the NeuTTS re-clone plan does anyway.
+
+## 5. Sources
+
+Runtime/payload (all read locally at pinned versions, 2026-10-19):
+
+- `github.com/ServeurpersoCom/qwentts.cpp` @ `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`
+  (commit date 2026-10-16; also upstream HEAD today): `README.md`, `docs/ARCHITECTURE.md`
+  (860 lines), `src/*`, `tools/*`, `docs/DOCKER.md`, `buildvulkan.sh`/`buildcpu.sh`/`buildall.sh`.
+- submodule `ServeurpersoCom/ggml` @ `cc5df49a6b878db2b68b9a7b45976e279594627d` (2026-10-16);
+  delta vs `ggml-org/ggml` master `a05c992` (2026-10-15) = ahead 2 / behind 0, per
+  https://github.com/ServeurpersoCom/ggml/compare/ggml-org:master...HEAD ;
+  `include/ggml.h` identical to upstream
+  https://raw.githubusercontent.com/ggml-org/ggml/master/include/ggml.h .
+- Live payload bytes: `/home/dank/models/Qwen3-TTS-GGUF/qwen-talker-1.7b-base-Q4_K_M.gguf`
+  (1,219,245,248 B) and `qwen-tokenizer-12hz-Q8_0.gguf` (291,150,624 B), headers parsed from
+  the host; sizes/sha256 match `scripts/download-frank-tts-engine-qwentts.sh`.
+- Live units/env/packs: `dankclaw/voiceservice/{Containerfile.engine,tts-engine-start.sh,tts-engine-warm.sh,quadlet/*}`,
+  `scripts/frank-tts-engine.env`, `scripts/frank-tts.env`, `/home/dank/voices/*`.
+
+Model provenance:
+
+- `https://huggingface.co/api/models/Qwen/Qwen3-TTS-12Hz-1.7B-Base` (+ CustomVoice,
+  VoiceDesign, 0.6B variants, `Qwen3-TTS-Tokenizer-12Hz`), apache-2.0, last-modified
+  2026-01-23 / 2026-01-29; safetensors param counts 1,928,677,440 (BF16, incl. embedded
+  `speech_tokenizer/`) and 170,557,441 (F32 codec).
+- `https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base/raw/main/README.md` (clone API:
+  `generate_voice_clone(ref_audio, ref_text, x_vector_only_mode)`,
+  `create_voice_clone_prompt`; reference stack = PyTorch + `device_map="cuda:0"` + bf16 +
+  flash-attn-2, Python 3.12 recommended).
+- `https://github.com/QwenLM/Qwen3-TTS`.
+
+Portability precedents:
+
+- `https://huggingface.co/xkos/Qwen3-TTS-12Hz-1.7B-ONNX` (2026-03-03) ·
+  `https://github.com/zhangluoyang/qwen3-tts-onnxruntime` ·
+  `https://huggingface.co/Echo9Zulu/Qwen3-TTS-12Hz-CustomVoice-1.7B-INT8-OpenVINO` (2026-03-22) ·
+  `https://huggingface.co/aurora2035/Qwen3-TTS-12Hz-0.6B-Base-OpenVINO-INT8` (2026-03-18) ·
+  `https://huggingface.co/litert-community/Qwen3-TTS-12Hz-0.6B-Base` (2026-10-17) incl.
+  `legacy/README.md` (graph splits, MTP fold, codec partA/partB, RTF table).
+
+AMD NPU / ggml backend status:
+
+- `https://api.github.com/repos/ggml-org/llama.cpp/git/trees/master?recursive=1` (no
+  XDNA/AIE/Vitis backend, 2026-10-19) ·
+  `https://github.com/ggml-org/llama.cpp/issues/21725` (open since 2026-04-10) ·
+  `https://github.com/ggml-org/llama.cpp/issues/1499` (closed stale 2025-04-07).
+- `https://github.com/Cyronius/ggml-xdna` (prefill-only, gates, Linux requirements, results
+  measured 2026-10-05/06 on HX 370) ·
+  `https://github.com/BrandedTamarasu-glitch/OllamaAMDNPU` (XDNA2 ggml backend on Strix Halo,
+  phases 1–12, "use Vulkan exclusively", measured on Ryzen AI MAX 385).
+- `https://github.com/amd/RyzenAI-SW` (main, 669 paths, 2026-08-18): audio content is ASR
+  only (`Demos/ASR/Whisper`, `Demos/ASR/Parakeet-TDT`, `Transformer-examples/ASR/Whisper-AI`);
+  **no TTS example anywhere in the official Ryzen AI Software tree**.
+- `https://ryzenai.docs.amd.com/en/latest/relnotes.html` (Ryzen AI SW 1.8.0: ONNX-format
+  models, ONNX Runtime as the loader, power-of-two QDQ/QOP quantization).
+- Mesa/RADV on AMD iGPUs: `https://www.amd.com/en/resources/support-articles/release-notes/RN-AMDGPU-UNIFIED-LINUX-25-20.3.html`
+  (Mesa Vulkan driver officially supported).
+
+Status: COMPLETE for the assigned slice (payload architecture, runtime architecture,
+portability). `[INFERENCE]` marks: none above except the Mesa-version caveat in ANSWER C,
+which is flagged there.

report/tts-npu-spike-verdict-2026-10-19.md

@@ -0,0 +1,180 @@
+# TTS on the Strix Halo NPU — spike verdict (2026-10-19)
+
+Question: **how can the fleet's TTS engine run on hardware rather than CPU?** Answer in
+one line: not on the NPU — the loader, not the silicon, refuses it; the door that exists is
+the gfx1151 iGPU behind a build flag we already own.
+
+Spike evidence lives in `report/npu-tts-runtime-landscape-2026-10-19.md` (which runtimes can
+reach XDNA2 at all) and `report/qwen3-tts-payload-portability-2026-10-19.md` (what the
+payload is and where it can be moved). This page owns only the **measured spike result** and
+the **model-side contract** neither slice has. Throwaway work ran in containers under
+`/home/dank/spikes/npu-tts` on Dankclaw; nothing in the fleet was changed — `frank-tts` and
+`frank-tts-engine` still serve `narrator1|2|3` on CPU.
+
+Legend: **[host]** live census 2026-10-19 · **[spike]** measured in the spike containers ·
+**[vendor]** primary vendor source, with the version it holds for.
+
+## 1. Verdict
+
+**NO-GO on the NPU, and the wall is a loader.** The XDNA2 device is healthy by AMD's own
+instruments; the runtime that owns this model family opens the device and then declines the
+graph, silently. One run loaded successfully; byte-identical runs never reproduced it, and a
+path that cannot be reproduced is not a foundation. **[spike]**
+
+**Go on the iGPU, and the lever is in this repo.**
+`dankclaw/voiceservice/Containerfile.engine` builds `qwentts.cpp` (commit `5916bf7`) with
+`-DGGML_VULKAN=OFF` written out deliberately beside CUDA/HIP/METAL/SYCL/RPC — the running
+container carries `libggml-base.so.0.27.0` + `libggml-cpu.so.0.27.0` + `tts-server` and
+nothing else. So: `-DGGML_VULKAN=ON` + `--device Vulkan0`, plus `/dev/dri` passthrough in
+the engine quadlet (the ComfyUI pattern: `AddDevice` + `GroupAdd=keep-groups` + a
+world-writable udev rule). The target shape is **co-tenancy** — the engine resident on the
+gfx1151 *beside* `frank-dragoon`, not a handover. §5 carries the census and both gates:
+the **heap gate is PASSED** — RADV's `DEVICE_LOCAL` heap is 77.33 GiB with 65.16 GiB of
+budget while dragoon is loaded, so memory is not the constraint — which leaves the
+contention gate (dragoon's TTFT with a synthesis in flight) and one unproven piece:
+whether qwentts.cpp's codec ops have Vulkan kernels for every op. **[host]** **[spike]**
+
+Why the iGPU is worth doing at all, given CPU already works: the deployed payload needs
+only **12.5 codes/s** (§2), so this is latency headroom and freed cores, not reviving a
+dead service. The independent Strix Halo measurement in the portability slice ("use Vulkan
+exclusively"; NPU decode under 1.5 t/s for an 8B) points the same way.
+
+## 2. The two arms do not share a real-time bar
+
+|Arm|Talker|Codec|Codes per second of audio|Bar|
+|---|---|---|---|---|
+|**Deployed** (`frank-tts-engine`)|`qwen-talker-1.7b-base-Q4_K_M.gguf`|`qwen-tokenizer-12hz-Q8_0.gguf`|**12.5**|~13 tok/s **[host]**|
+|**Spike** (`amd/neutts-air_rai_1.8.0_npu_4K`)|NuTTS-Air, 24-layer Qwen2|NeuCodec `24000 / hop 480`|**50.00**|~50 tok/s **[spike]**|
+
+Measured from the staged NPU artifact — the contract any future arm must satisfy:
+
+|Fact|Value|
+|---|---|
+Vocab|`genai_config.json` `vocab_size` **217,652**; base BPE 151,643; specials 151,644–151,670 **[spike]**|
+Speech tokens|**65,536** `<|speech_N|>` entries, ids **151,671–217,206** contiguous; the vendor template maps a codec index straight to its id, `"".join(f"<|speech_{i}|>")`. A 10,001-code assumption silently truncates the space — encoded references really do reach the 60k range. **[spike]**|
+Stop token|`<|SPEECH_GENERATION_END|>` = 151,670, but the export's `model.eos_token_id` is `[151645, 151643]` (the *chat turn* stop) and `search.eos_token_ids` is unset — pass 151,670 explicitly or speech ends after one token **[spike]**|
+Input ceiling|4096 tokens **total**, input + output **[vendor]**|
+Frame rate|50.00 codes/s confirmed across three references (49.96 / 49.99 / 50.00) **[spike]**|
+Reference encode|`librosa.load(sr=16000, mono=True)` → `[1,1,T]` float → `codec.encode_code(audio_or_path=…)` (vendor `NeuTTS.encode_reference`); CPU RTF 0.28–0.38 on 8 threads, so codes are a cacheable one-time cost, never a per-request charge. `neucodec` also needs `torchtune`; install both `--no-deps --target <dir>` so a second torch never shadows the venv's. **[spike]**|
+Pre-encoded voices|`/spike/logs/codes-narrator{1,2,3}.npy` — 994 / 1,249 / 1,250 codes for 19.90 / 24.98 / 25.00 s **[spike]**|
+
+**Gap that any arm inherits:** the voice packs carry no transcript. `manifest.json` holds
+provenance and crop geometry (`crop.source_speech_run` 98.739–118.636 s of a 22,569 s
+audiobook) and no text, and Dankclaw has no ASR at all — `whisper`, `whisper.cpp`,
+`faster-whisper` all absent. NuTTS conditioning needs `ref_text`; do not invent it. **[host]**
+
+## 3. Exoneration chain — why the blame is not the hardware
+
+Every row is an instrument, not an inference.
+
+|Probe|Result|
+|---|---|
+`xrt-smi examine`|`[0000:c6:00.1] RyzenAI-npu5 aie2p 6x8`, XRT 2.25.37 (vendor extract), NPU fw 1.1.2.65, `amdxdna 7.0.0-38-generic` **[spike]**|
+`xrt-smi validate`|**PASSED**, including `throughput` **[spike]**|
+`pyxrt.device(0)` in-container|constructs — XRT enumerates the device **[spike]**|
+`strace -f -e trace=openat` on the failing load|`genai_config.json` read → `model.onnx` opened → `libxrt_driver_xdna.so.2` loaded → `/dev/accel/accel0` opened `O_RDWR` → GEM ioctls succeed → **EP declines, before touching `dd_metastate_*` or `cache/txn_bins.zip`** **[spike]**|
+`dmesg`, `fuser /dev/accel/accel0`|silent since boot, no holder — the decline never reaches the kernel **[spike]**|
+Hardlinked mirror dirs of the model tree (`rundir*`)|retired, no change — the "metastate shadowing" theory is dead **[spike]**|
+Every logging lever in that build: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT`|**zero lines** — the decline path is silent **[spike]**|
+Vendor `quicktest`|`Session successfully initialized` — proves the XRT/driver/device path, **not** NPU residency: that build falls back to CPU silently. Prove residency with `/sys/class/accel/accel0/device/mem_usage` during generation. **[spike] [vendor]**|
+
+No NPU token rate was ever measured, so none is quoted here. The single successful load
+(13:17, `[load] 0.84s`, finite logits, XRT banner at *generate* time) has not been
+reproduced by the byte-identical command since, with the model files pristine by inode and
+mtime. **[spike]**
+
+## 4. Host gates the spike proved (so they are not re-litigated)
+
+- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render` 0660): the host
+  gid never enters the userns map. `--group-add keep-groups`, `--userns=keep-id`,
+  `--userns=keep-id:gid=990`, `--group-add render`, `--group-add gid:990` — all fail.
+  Doors: rootful podman (separate storage), or a world-writable udev rule, which the fleet
+  already does for `kfd`/`renderD*`. **[spike]**
+- **Never `dpkg -i` the vendor XRT deb** — it installs a DKMS `amdxdna` over the distro
+  driver. `dpkg-deb -x` the debs and bind-mount the tree at `/opt/xilinx/xrt`; the plugin
+  deb is what carries `libxrt_driver_xdna.so`. **[spike]**
+- In-tree `amdxdna 0.7.0-38` + `iommu=pt` is enough for context creation; a missing device
+  node is *not* the modern failure mode. **[host]**
+- `xrt-smi` inside a container needs `libboost-filesystem1.83.0` +
+  `libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`);
+  boost is needed by `xrt-smi` only, never by the OGA/EP path. **[spike]**
+- Headroom is tight and it is host RAM, not a separate pool — see §5 for tonight's census.
+  A multi-GB NPU partition plus an AIE compile workspace does not fit beside
+  `frank-dragoon`. Benign-looking `Failed to alloc dev bo memory, ret -28` is that
+  pressure. **[host]**
+
+## 5. Target shape: the TTS engine shares the gfx1151 with `frank-dragoon`
+
+The deployment Frank wants is **both resident at once**, not a handover. Measured 2026-10-19
+(read-only, nothing loaded or unloaded):
+
+|Measure|Value|Reading|
+|---|---|---|
+iGPU GTT in use|**18,523 MiB of 116,733 MiB**|≈96 GiB of aperture free; this is the real budget (fleet law) **[host]**|
+DRM "VRAM" carve-out|2,048 MiB total, 174 MiB used|what KFD/DRM calls dedicated — **not** the budget **[host]**|
+Host RAM|121 GiB total, 17 GiB available, 79 GiB buff/cache|**GTT pages are system pages**: an iGPU allocation is host RAM consumed, so the two tenants share one pool **[host]**|
+TTS engine today|3.84 GiB RSS, CPU|its device-resident replacement lands in that same pool **[host]**|
+
+**Two gates, in this order.**
+
+1. **Heap gate — PASSED.** RADV does **not** hand a container tenant the 2 GiB DRM
+   carve-out. `vulkaninfo` inside the engine's own base image (`ubuntu:24.04` plus
+   `mesa-vulkan-drivers vulkan-tools`, `--device /dev/dri`, rootless) sees
+   `Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, and reports:
+
+   |Heap|size|budget with dragoon loaded|flags|
+   |---|---|---|---|
+   |0|38.67 GiB|32.58 GiB|host-visible, not device-local|
+   |1|**77.33 GiB**|**65.16 GiB**|`DEVICE_LOCAL`|
+
+   The heaps sum to the ~116 GiB aperture in Mesa's ⅓/⅔ APU split — the same RAM twice:
+   a Vulkan allocation is a GTT mapping, so a budget is a ceiling on mappings, not a pile
+   of free silicon memory. ggml's Vulkan backend asks for `DEVICE_LOCAL`, which is heap 1,
+   and the TTS engine needs roughly 4 GiB of it. **Memory stops being the question.** **[spike]**
+   Probe traps: `vulkaninfo --batch` is not a flag in this `vulkan-tools` (it prints the
+   help page and exits 1 — use `vulkaninfo --text`), and the container has **no NVIDIA
+   ICD**, so Vulkan device 0 is the iGPU here. Do not trust that ordering — select by
+   device *name*, or an image that grows an `nvidia_icd.json` silently points TTS at the
+   3090 slot.
+2. **Contention gate.** `frank-dragoon` is the endpoint the operator session runs on, so the
+   honest measurement with both loaded is dragoon's own TTFT and decode with a synthesis in
+   flight — one request each, no ladder. Expect compute-unit sharing rather than memory
+   pressure: the aperture has room, the shaders do not.
+
+**Change surface** — the whole edit, so nobody re-derives it:
+
+1. `Containerfile.engine` build stage: add `libvulkan-dev glslang-dev`. It installs only
+   `ca-certificates cmake g++ git make` today, and ggml's Vulkan backend compiles its
+   shaders at build time — the flag alone fails at configure.
+2. `-DGGML_VULKAN=OFF` → `ON` in the same cmake block; the flag list is written out
+   explicitly, so this is a one-word diff.
+3. Runtime stage: add `mesa-vulkan-drivers`. Without a Vulkan **ICD inside the image** the
+   device node buys nothing — there is no `radeon_icd.json` in the container today.
+4. `frank-tts-engine.container`: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` — **not**
+   `renderD128`. On this hybrid box that node is the NVIDIA card; the iGPU is
+   `renderD129` (`readlink -f /sys/class/drm/renderD129/device/driver` → `amdgpu`).
+5. Engine argv: `--device Vulkan0` — qwentts.cpp's tools take `--device <name>` at the
+   pinned commit.
+
+Gate 1 recipe, already run and **PASSED**: `mesa-vulkan-drivers` + `vulkan-tools` in a
+throwaway container off the same base, `--device /dev/dri`, then `vulkaninfo --text` and read
+the `VkPhysicalDeviceMemoryProperties` heap sizes — no model, no load, no compute. The probe
+trap: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` — it prints the help
+page and exits 1, which reads like a driver failure. Every package involved has a candidate on
+the host's apt (`vulkan-tools` 1.4.341, `libvulkan-dev` 1.4.341, `glslang-dev`/`glslang-tools`
+16.2.0); the build stage runs on the digest-pinned Ubuntu 24.04 base, whose versions are older
+but present. Script: `/home/dank/spikes/npu-tts/vk_probe.sh` on Dankclaw.
+
+Constraints that do not relax for a co-tenant: `/dev/dri` passthrough needs the ComfyUI
+pattern (`AddDevice` + `GroupAdd=keep-groups` + the world-writable udev rule) because
+systemd user-unit spawns carry no supplementary groups; the 64 GiB swap at `/swap.img`
+exists to protect co-tenants, and paging a KV pool is worse than the OOM it prevents;
+`pp_dpm_fclk` reads `auto` (unheld), which only becomes a hazard if the NPU ever joins.
+
+## 6. Re-visit condition
+
+Retry the NPU when **a Linux loader claims the graph**: an `onnxruntime-genai-ryzenai`
+build whose RyzenAI adapter takes the NuTTS-Air Full-Fusion graph on `RyzenAI-npu5`, or that
+model appearing in a runtime's own supported catalog with a Linux runner. Then the
+prerequisites are already mapped and priced: the gated NeuCodec encoder for cloned voices,
+the `ref_text` transcripts, the 4096-token prompt budget, and 50 codes/s as the bar.

scripts/autoresearch-frank-archon.sh

@@ -29,10 +29,10 @@
 # The scratch mirrors the production container contract exactly (serve-frank-
 # primary.sh): --network host, --init, seccomp=unconfined (io_uring for the
 # n-gram table), memlock unbounded (the ~132 GiB pinned ledger), /dev/kfd +
-# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
-# differences: the container name (arbench, so prod's identity is untouched)
-# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
-# prefill is cold — the prod tier at kvcache/frank-archon is left intact.
+# /dev/dri, the .rad ro, the tune cache mount. The ONLY differences: the
+# container name (arbench, so prod's identity is untouched) and the prefix cache
+# switched OFF (--prefix-cache-mib 0), so every prefill is cold — the tier is
+# host RAM now, there is no per-run disk directory to wipe.
 set -uo pipefail
 
 ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-archon.env}"
@@ -60,9 +60,8 @@ set -a; . "$ENV_FILE"; set +a
 sudo systemctl stop frank-archon >/dev/null 2>&1 || true
 podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
 
-# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
-FRESH_KV="$FRANK_ARCHON_STATE_DIR/kvcache/$SCRATCH"
-rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
+# --- the prefix cache is switched off below, so there is no scratch KV tree ---
+# to wipe; the cold prefill comes from --prefix-cache-mib 0.
 mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
 
 # --- boot the scratch: the production run line, scratch name + fresh KV ------
@@ -73,7 +72,6 @@ podman run -d --name "$SCRATCH" \
   --ulimit memlock=-1:-1 \
   --device /dev/kfd --device /dev/dri \
   -v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
-  -v "$FRANK_ARCHON_STATE_DIR":/data \
   -v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
   -v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
   "$FRANK_ARCHON_IMAGE" \
@@ -87,12 +85,9 @@ podman run -d --name "$SCRATCH" \
   --placement "$FRANK_ARCHON_PLACEMENT" \
   --host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
   --gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
-  --vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
-  --vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
   --kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
-  --prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
-  --prefix-cache-dir "/data/kvcache/$SCRATCH" \
-  --prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
+  --prefix-cache-mib 0 \
+  --busy-prefill-chunk "${FRANK_ARCHON_BUSY_PREFILL_CHUNK:-auto}" \
   --ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
   --num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
   --max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \

scripts/axe-measurements.py

@@ -0,0 +1,164 @@
+#!/usr/bin/env python3
+"""Destroy measurement ledgers in the surviving pages.
+
+Point of use (AGENTS.md → Documentation Policy): a rate/throughput figure for
+anything not running and not being compared is destroyed, not filed. This pass
+deletes
+
+  1. any section whose heading is a bench/throughput/rate/ladder/verdict ledger,
+  2. any table row or standalone bullet that REPORTS a rate (`123.4 t/s`,
+     `tok/s`, `ms/token`, `TTFT`, acceptance %, `llama-bench`, a bench arm),
+
+and it never touches a shape number (ctx, slots, MiB budgets, ports) or a bullet
+under a *Laws* / *Wire contract* / *Rollback* heading, where the sentence is the
+payload and the number is decoration. Dry run by default; `--apply` writes.
+
+Usage: axe-measurements.py [--apply] [--verbose]
+"""
+import pathlib
+import re
+import sys
+
+VAULT = pathlib.Path("serverAppConfigs-docs")
+RATE = re.compile(
+    r"\d[\d,.]*\s*(t/s|tok/s|tokens?/s|tok s|tps|ms/token|µs/token|GB/s|GiB/s)\b|"
+    r"\b(llama-bench|bench (arm|ladder|window)|decode ladder|prefill ladder)\b|"
+    r"\bacceptance (rate|of)\b|\bdraft_n\b|\bttft\b|\bServer-Timing\b",
+    re.I,
+)
+# A heading is a ledger only when the noun LEDS it and nothing instructional
+# follows: `## Throughput` yes, `## 6. The context ladder (find the max bootable
+# window by measurement)` no — that one is the method, which is point-of-use.
+LEDGER_HEAD = re.compile(
+    r"^#{2,4}\s+(?:\d+[.) ]\s*)?(bench(es|mark|marks)?|throughput|results?|timings?|"
+    r"measured (figures|numbers|results|rates|rates)|rates|verdict|acceptance|ladder)"
+    r"\b[^|]{0,40}$",
+    re.I,
+)
+INSTRUCTIONAL = re.compile(r"how|why|law|policy|method|procedure|find|by measurement|execution|flip|checklist", re.I)
+PROTECTED = re.compile(
+    r"^#{2,4}\s+.*(law|wire|rollback|contract|shape|eras|files|health|see also|related|"
+    r"boot|preflight|environment)",
+    re.I,
+)
+def blank_rate_cells(text):
+    """Kill throughput COLUMNS wherever they appear: a cell that reports a rate
+    becomes an em-dash, so the row keeps whatever else it records (the payload
+    name, the reason, the rollback) and loses only the dead number."""
+    out = []
+    for line in text.split("\n"):
+        st = line.strip()
+        if st.startswith("|") and st.endswith("|") and RATE.search(line):
+            cells = line.split("|")
+            for i, c in enumerate(cells):
+                if RATE.search(c):
+                    cells[i] = " — "
+            line = "|".join(cells)
+        out.append(line)
+    return "\n".join(out)
+
+
+def collapse_eras(text):
+    """One Eras row per era is fleet law; the NUMBERS in the old rows are not.
+    Keep the current era verbatim, fold everything older into one row that says
+    the numbers were destroyed and where the rollback actually lives (git)."""
+    lines = text.split("\n")
+    out = []
+    in_eras = False
+    ncol = 0
+    folded = 0
+    for line in lines:
+        if line.startswith("#"):
+            in_eras = bool(re.match(r"^#{2,4}\s+.*eras\b", line, re.I))
+            if folded:
+                out.append("| " + " | ".join(
+                    [f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured "
+                         "number destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1)))
+                folded = 0
+            out.append(line)
+            continue
+        st = line.strip()
+        if in_eras and st.startswith("|"):
+            cells = [c.strip() for c in st.strip("|").split("|")]
+            if set("".join(cells)) <= set(":- "):
+                ncol = len(cells)
+                out.append(line)
+                continue
+            if ncol == 0:
+                ncol = len(cells)
+            if re.search(r"2026-1[01]|current|active|running", st, re.I):
+                out.append(line)
+            else:
+                folded += 1
+            continue
+        if folded and not st.startswith("|"):
+            out.append("| " + " | ".join(
+                [f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured number "
+                     "destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1)))
+            folded = 0
+            ncol = 0
+        out.append(line)
+    if folded:
+        out.append("| " + " | ".join(
+            [f"≤ 2026-09 — {folded} earlier eras destroyed 2026-10-19"] + ["—"] * max(0, ncol - 1)))
+    return "\n".join(out)
+
+
+def main(argv):
+    apply = "--apply" in argv
+    verbose = "--verbose" in argv
+    total_lines = 0
+    pages = 0
+    for page in sorted(VAULT.rglob("*.md")):
+        s = str(page)
+        if ".AGENTTRASH" in s or ".obsidian" in s:
+            continue
+        text = page.read_text(errors="replace")
+        before = len(text.split("\n"))
+        if s.startswith("serverAppConfigs-docs/LLM/"):
+            text = collapse_eras(blank_rate_cells(text))
+        lines = text.split("\n")
+        out = []
+        skip = False
+        under_protected = False
+        cut = 0
+        for line in lines:
+            if line.startswith("#"):
+                under_protected = bool(PROTECTED.search(line))
+                skip = (bool(LEDGER_HEAD.match(line)) and not under_protected
+                        and not INSTRUCTIONAL.search(line))
+                if skip:
+                    cut += 1
+                    continue
+            if skip:
+                cut += 1
+                continue
+            stripped = line.strip()
+            out.append(line)
+            if under_protected:
+                # Eras rows are the rollback record, Law bullets are the payload:
+                # the number is decoration inside them, and the row/bullet is not
+                # a ledger entry. Leave the whole section alone.
+                continue
+            if stripped.startswith(("|", "-", "*")) and RATE.search(line):
+                # Destroy only when the line EXISTS to report a number: a table
+                # row, or a bullet whose verb is measurement.
+                if stripped.startswith("|") or re.search(
+                        r"\b(measured|measures|bench|benched|ran at|runs at|clocked|hit|"
+                        r"landed at|came in at|peak|peaks at|sustained|wall.clock)\b", line, re.I):
+                    out.pop()
+                    cut += 1
+        final = re.sub(r"\n{3,}", "\n\n", "\n".join(out))
+        saved = before - len(final.split("\n"))
+        if saved or cut:
+            pages += 1
+            total_lines += max(saved, cut)
+            print(f"  {s.replace('serverAppConfigs-docs/', '')}: -{max(saved, cut)}")
+            if apply:
+                page.write_text(final.rstrip("\n") + "\n")
+    print(f"{'APPLIED' if apply else 'dry run'}: {pages} pages, {total_lines} lines of ledger")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/check-okf.py

@@ -0,0 +1,293 @@
+#!/usr/bin/env python3
+"""OKF v0.1 conformance linter for a docs bundle (default serverAppConfigs-docs).
+
+Checks the hard rules of the Open Knowledge Format spec plus the maintenance
+guidance, so "conforms to OKF" is a test and not a claim:
+
+  FRONTMATTER  every non-reserved .md has parseable YAML frontmatter with a
+               non-empty `type` from the spec's value set
+  TYPE         the value is one of reference / howto / tutorial / explanation
+  DESC         `description` is present, non-blank, not the title, <= 120 chars,
+               and does not restate a fleet quantity
+  TIMESTAMP    `timestamp` present, ISO 8601, not in the future
+  LENGTH       lines between 1 and 500 inclusive
+  INDEX-FM     no index.md may carry frontmatter (index.md is reserved)
+  LOG-DATE     log.md date headings are `## YYYY-MM-DD` or `## YYYY-MM-DD — Title`,
+               one entry per date, newest first
+  INDEX-COVER  every index.md links each of its sibling pages; the root index
+               links every page in the bundle
+  INDEX-NUM    an index table row restates a fleet quantity — a digit run of
+               three or more, a digits-plus-unit figure, a port, or a
+               multiplier. Names (Qwen3.6, RDNA4, RTX 3090, PICO-8, P4) are
+               not quantities and are allowed
+  CITATIONS    informational: pages with factual claims and no Citations
+               section are counted, never failed
+
+  LENGTH       lines between 1 and 500 inclusive. A page may opt out of the
+               ceiling with `okf-exempt: length` in its frontmatter: AGENTS.md
+               makes one page per endpoint permanent, and an endpoint page with
+               a full Eras table cannot split without creating a second owner
+               of the same shape (tests/docs_completeness.bats enforces that)
+
+Exit 0 when there are no violations.
+"""
+import datetime
+import re
+import sys
+from pathlib import Path
+
+TYPES = {"reference", "howto", "tutorial", "explanation"}
+RESERVED = {"index.md", "log.md", "AGENTS.md"}
+
+# A quantity, as opposed to a name: a comma-grouped figure, a run of 3+ digits,
+# a unit-bearing figure, a port (":7770"), or a multiplier ("2x", "3 *").
+QUANTITY = re.compile(
+    r"\d{1,3}(?:,\d{3})+"          # 262,144
+    r"|\d{3,}"                     # 200000, 7770, 3090
+    r"|\d+(?:\.\d+)?\s*(?:GiB|MiB|GB|MB|TB|tok/s|t/s|x\b|%)"
+    r"|[:×]\s*\d"
+    r"|\d+\s*[x×]\s*\d"
+)
+# Product / model / standard names that legitimately contain digits.
+NAMES = re.compile(
+    r"Qwen[\d.]*[-\w]*|Gemma\s?\d[\w.]*|MiMo[-\w.\d.]*|Ornith[-\w.\d]*"
+    r"|Bonsai[-\w.\d]*|Ling[-\w.\d.]*|DeepSeek[-\w.\d]*|GLM[-\w.\d]*"
+    r"|PICO-\d+|RDNA\s?\d+|R\d{4}|RTX\s?\d+|XDNA\d|gfx\d+|CUDA\s?[\d.]+"
+    r"|\d+\s?GbE|Wi-?Fi\s?\d|v?\d+\.\d+(?:\.\d+)?|b\d+|P\d\b|A\d+\b"
+    r"|frank-\w*2026-\d\d-\d\d|2026-\d\d-\d\d",
+    re.I,
+)
+DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}")
+FM_KEY = re.compile(r"^([A-Za-z][\w-]*):\s*(.*)$")
+
+
+def split_frontmatter(text):
+    if not text.startswith("---"):
+        return None, text
+    end = text.find("\n---", 3)
+    if end < 0:
+        return None, text
+    body_start = text.find("\n", end + 1)
+    return text[3:end], text[body_start + 1 :] if body_start >= 0 else ""
+
+
+def fields(raw):
+    out = {}
+    if not raw:
+        return out
+    key = None
+    for line in raw.splitlines():
+        if not line.strip():
+            continue
+        m = FM_KEY.match(line)
+        if m:
+            key, val = m.group(1), m.group(2).strip().strip("\"'")
+            out[key] = val
+        elif line.startswith(" ") and key:
+            out[key] = (out[key] + " " + line.strip()).strip()
+    return out
+
+
+def strip_names(value):
+    return NAMES.sub(" ", value)
+
+
+def check_file(path: Path, root: Path, length_exempt=frozenset()):
+    v = []
+    rel = str(path.relative_to(root))
+    text = path.read_text(errors="replace")
+    lines = text.splitlines()
+    name = path.name
+    raw, body = split_frontmatter(text)
+    fm = fields(raw)
+
+    if name not in RESERVED:
+        if raw is None:
+            v.append(("FRONTMATTER", rel, "no parseable frontmatter"))
+        else:
+            t = fm.get("type", "")
+            if not t:
+                v.append(("TYPE", rel, "missing type"))
+            elif t not in TYPES:
+                v.append(("TYPE", rel, f"type '{t}' not in {sorted(TYPES)}"))
+            d = fm.get("description", "")
+            if not d:
+                v.append(("DESC", rel, "missing description"))
+            else:
+                if len(d) > 120:
+                    v.append(("DESC", rel, f"description is {len(d)} chars (max 120)"))
+                if d.strip("# ").lower() == fm.get("title", "\0").strip("# ").lower():
+                    v.append(("DESC", rel, "description equals title"))
+                if QUANTITY.search(strip_names(d)):
+                    v.append(("DESC", rel, "description restates a quantity"))
+            ts = fm.get("timestamp", "")
+            if not ts:
+                v.append(("TIMESTAMP", rel, "missing timestamp"))
+            else:
+                try:
+                    stamp = datetime.date.fromisoformat(ts[:10])
+                    if stamp > datetime.date.today():
+                        v.append(("TIMESTAMP", rel, f"timestamp {ts} is in the future"))
+                except ValueError:
+                    v.append(("TIMESTAMP", rel, f"unparseable timestamp {ts!r}"))
+
+    if name == "index.md" and raw is not None:
+        v.append(("INDEX-FM", rel, "index.md must not carry frontmatter"))
+
+    if name == "log.md":
+        dates = []
+        for line in lines:
+            m = re.match(r"^##\s+(\S.*?)\s*$", line)
+            if not m:
+                continue
+            head = m.group(1)
+            if not DATE_RE.match(head):
+                continue
+            dates.append(head[:10])
+            # OKF requires the heading to USE the date format; a title after an
+            # em-dash is allowed and costs zero lines, so `## 2026-08-17 — Why the
+            # pair was rebuilt` is conformant and `## 2026-08-17Rebuilt` is not.
+            if head != head[:10] and not re.match(r"^\d{4}-\d{2}-\d{2}\s+—\s+\S", head):
+                v.append(("LOG-DATE", rel,
+                          f"date heading must be the date alone or `date — Title`: '## {head}'"))
+        if len(dates) != len(set(dates)):
+            v.append(("LOG-DATE", rel, "duplicate date heading"))
+        if dates != sorted(dates, reverse=True):
+            v.append(("LOG-DATE", rel, "date headings not newest first"))
+
+    if raw is not None and name not in RESERVED:
+        n = len(lines)
+        body_lines = len([ln for ln in body.splitlines() if ln.strip()])
+        opted_out = "length" in fm.get("okf-exempt", "").lower()
+        if n > 500:
+            if rel in length_exempt:
+                v.append(("LENGTH-OK", rel,
+                          f"{n} lines — over the hint, pinned by the "
+                          "one-page-per-endpoint law"))
+            elif opted_out:
+                v.append(("LENGTH-OK", rel,
+                          f"{n} lines — over the hint, opted out by the page"))
+            else:
+                v.append(("LENGTH", rel, f"{n} lines (max 500)"))
+        elif body_lines < 15:
+            v.append(("LENGTH", rel, "stub: fewer than 15 body lines"))
+
+    # A second `type:` key inside the body means the page carries two copies of
+    # itself with a frontmatter block in the middle — the shape a naive
+    # frontmatter rewrite produces. It doubles the length, splits the wikilinks
+    # and makes the page's facts ambiguous, so it is a hard failure.
+    if raw is not None and name not in RESERVED:
+        dupes = [i + len((raw or "").splitlines()) + 2
+                 for i, ln in enumerate(body.splitlines())
+                 if re.match(r"^(type|title|description):", ln)]
+        if dupes:
+            v.append(("DOUBLED", rel,
+                      f"frontmatter keys inside the body at line(s) "
+                      f"{', '.join(str(d) for d in dupes[:5])} — the page "
+                      "contains a duplicated copy of itself"))
+
+    return v
+
+
+def index_rows(text):
+    rows = []
+    for line in text.splitlines():
+        if line.startswith("|") and "[[" in line:
+            rows.append(line)
+    return rows
+
+
+def check_links(root: Path):
+    v = []
+    pages = [
+        p for p in sorted(root.rglob("*.md"))
+        if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
+    ]
+    for index in [p for p in pages if p.name == "index.md"]:
+        text = index.read_text(errors="replace")
+        links = set(re.findall(r"\[\[([^\]|#]+)", text))
+        if index.parent == root:
+            targets = [p for p in pages if p.name not in RESERVED]
+        else:
+            targets = [p for p in index.parent.glob("*.md") if p.name != "index.md"]
+        for target in targets:
+            if target.stem not in links and target.name not in links:
+                v.append(("INDEX-COVER", str(index.relative_to(root)),
+                          f"does not link sibling {target.name}"))
+    for index in [p for p in pages if p.name == "index.md"]:
+        for row in index_rows(index.read_text(errors="replace")):
+            if QUANTITY.search(strip_names(row)):
+                v.append(("INDEX-NUM", str(index.relative_to(root)),
+                          "row restates a quantity: " + row[:80]))
+    return v
+
+
+def live_endpoints(root: Path):
+    """Pages pinned by AGENTS.md's one-page-per-endpoint law.
+
+    Derived, never hardcoded — the same derivation as
+    tests/docs_completeness.bats: an endpoint is live when scripts/ holds both
+    its env and its unit. Those pages carry the Eras table for their whole
+    history, so the OKF 500-line split hint cannot apply to them without
+    creating a second owner of the same shape.
+    """
+    scripts = root.parent / "scripts"
+    out = set()
+    if scripts.is_dir():
+        for envf in scripts.glob("frank-*.env"):
+            if (scripts / f"{envf.stem}.service").exists():
+                out.add(f"LLM/{envf.stem}.md")
+    return out
+
+
+def retired_era_pages(root: Path):
+    """Era pages of a retired endpoint: campaign ledgers, one owner each.
+
+    AGENTS.md lets a campaign ledger that exists nowhere else live on the page
+    that measured it, and a retired endpoint keeps exactly one page per era.
+    Splitting such a ledger to satisfy the 500-line hint would fragment one
+    campaign across two owners, so the tag opts the page out.
+    """
+    out = set()
+    for path in root.rglob("*.md"):
+        s = str(path)
+        if ".AGENTTRASH" in s or ".obsidian" in s:
+            continue
+        raw, _ = split_frontmatter(path.read_text(errors="replace"))
+        if raw is not None and "retired-endpoint" in fields(raw).get("tags", ""):
+            out.add(str(path.relative_to(root)))
+    return out
+
+
+def main(argv):
+    root = Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
+    exempt = frozenset(live_endpoints(root)) | retired_era_pages(root)
+    violations = []
+    notes = []
+    citation_gaps = 0
+    pages = 0
+    for path in sorted(root.rglob("*.md")):
+        s = str(path)
+        if ".AGENTTRASH" in s or ".obsidian" in s:
+            continue
+        pages += 1
+        for code, rel, detail in check_file(path, root, exempt):
+            (notes if code == "LENGTH-OK" else violations).append((code, rel, detail))
+        text = path.read_text(errors="replace")
+        if len(text.splitlines()) > 40 and not re.search(
+                r"^##\s+(Citations|Sources)\b", text, re.M):
+            citation_gaps += 1
+    violations += check_links(root)
+
+    for code, rel, detail in violations:
+        print(f"  {code:12s} {rel}: {detail}")
+    for code, rel, detail in notes:
+        print(f"  {code:12s} {rel}: {detail}")
+    print(f"{pages} concept pages, {len(violations)} OKF violations"
+          f" ({citation_gaps} pages without a Citations section — soft;"
+          f" {len(notes)} length exemptions honoured)")
+    return 1 if violations else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/check-vault-links.py

@@ -31,7 +31,12 @@ def strip_md(path: str) -> str:
 
 
 def build_index(vault: pathlib.Path):
-    pages = sorted(vault.rglob("*.md"))
+    # `.AGENTTRASH` is deleted-but-held state, not vault content: it must not be
+    # scanned (its relative links resolve against the trash dir, not the vault)
+    # and must not answer as a link target — a live page pointing into the trash
+    # is an unresolved link, not a resolved one.
+    pages = sorted(p for p in vault.rglob("*.md")
+                   if ".AGENTTRASH" not in str(p))
     stems: dict[str, list[pathlib.Path]] = {}
     for page in pages:
         stems.setdefault(page.stem.casefold(), []).append(page)

scripts/destroy-vault.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Destroy the vault's dead weight: era ledgers, retired runtimes, dead experiments.
+
+`AGENTS.md` was treating a measurement as a record worth keeping forever. It is not: a
+throughput figure for a payload that no longer runs, on a runtime that no longer exists,
+is a liability — the next agent spends context on a number that AI progress has already
+obsolete. This pass moves those pages to `.AGENTTRASH/` (repo law: never `rm`) and then
+deletes every reference to them from the surviving pages, including the sentence that
+carried the link when the link was the sentence's only content.
+
+Usage: destroy-vault.py <list-file>
+List file: one vault-relative path per line; a directory path destroys the whole tree.
+"""
+import pathlib
+import re
+import shutil
+import sys
+
+VAULT = pathlib.Path("serverAppConfigs-docs")
+TRASH = VAULT / ".AGENTTRASH" / "20261019-destruction"
+
+
+def targets(list_file):
+    out = []
+    for line in pathlib.Path(list_file).read_text().splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        p = VAULT / line
+        if p.is_dir():
+            out += [f for f in sorted(p.rglob("*.md"))]
+        elif p.is_file():
+            out.append(p)
+        else:
+            print(f"  SKIP (missing): {line}")
+    return out
+
+
+def main(argv):
+    if len(argv) < 2:
+        print(__doc__)
+        return 2
+    doomed = targets(argv[1])
+    # Stems come from the trash too, so re-running the pass (or running it after a
+    # partial destruction) still knows which names are dead and keeps scrubbing.
+    stems = {p.stem for p in doomed}
+    if TRASH.is_dir():
+        stems |= {p.stem for p in TRASH.rglob("*.md")}
+    moved = 0
+    for p in doomed:
+        dest = TRASH / p.relative_to(VAULT)
+        dest.parent.mkdir(parents=True, exist_ok=True)
+        shutil.move(str(p), str(dest))
+        moved += 1
+    print(f"destroyed {moved} pages ({sum(len(p.read_text(errors='replace').splitlines()) for p in TRASH.rglob('*.md'))} lines staged in trash)")
+
+    # Repoint-or-delete: a link to a destroyed page is itself dead information.
+    link_wl = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|([^\]]*))?\]\]")
+    link_md = re.compile(r"\[[^\]]*\]\(([^)#]*\.md)(?:#[^)]*)?\)")
+    pages_cleaned = links_dropped = lines_dropped = 0
+    for page in sorted(VAULT.rglob("*.md")):
+        s = str(page)
+        if ".AGENTTRASH" in s or ".obsidian" in s:
+            continue
+        text = page.read_text(errors="replace")
+        lines = text.split("\n")
+        out = []
+        hits = 0
+
+        def dead(target):
+            # NOT pathlib.Path(...).stem: that strips a real suffix off names like
+            # `Lemonade and sd.cpp`, so a link to a destroyed page survives and the
+            # vault then fails its own link check.
+            t = target.strip()
+            if t.endswith(".md"):
+                t = t[:-3]
+            return t in stems or pathlib.PurePosixPath(t).name in stems
+
+        for line in lines:
+            found = [t for t in link_wl.findall(line) if dead(t[0])]
+            for m in link_md.finditer(line):
+                if dead(m.group(1)):
+                    found.append(m.group(1))
+            if not found:
+                out.append(line)
+                continue
+            hits += len(found)
+            stripped = link_wl.sub(
+                lambda m: m.group(2) if (m.group(2) and not dead(m.group(1))) else "",
+                line)
+            stripped = link_md.sub(
+                lambda m: "" if dead(m.group(1)) else m.group(0), stripped)
+            # What is left of the line after the dead links are gone?
+            residue = re.sub(r"[\s\-*•·|:;,.\u2014\u2013]+", "", stripped)
+            if residue:
+                out.append(re.sub(r"\s+$", "", re.sub(r"^(\s*[-*]\s*)[\s:;,.\u2014-]+", r"\1", stripped)))
+            else:
+                lines_dropped += 1
+        if hits:
+            page.write_text("\n".join(out))
+            pages_cleaned += 1
+            links_dropped += hits
+    print(f"cleaned {pages_cleaned} surviving pages: {links_dropped} dead links dropped, "
+          f"{lines_dropped} lines left empty and deleted")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/frank-archon.env

@@ -1,306 +1,131 @@
-# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving the
-# PUBLISHER's 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
-# (MODEL_FILE below, fetched by scripts/download-frank-archon-iq4r.sh) on
-# BOTH R9700s (TP=2), dankserver :7770.
-# 2026-10-15 PAYLOAD SWAP: the model moved from the in-house w4dn8 conversion
-# of the Swift1.5 finetune to the engine author's published container
-# qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, sha256-verified at fetch —
-# the w4dn8 measured 116,416 MiB, so this shape carries over with 94 MiB to
-# spare). NO argument changed: every number below is the 2026-10-13 shape,
-# kept verbatim. The swap commit is the rollback (the w4dn8 payload stays on
-# disk).
-
+# frank-archon.env — radiance (StillDeadcode) on BOTH R9700s (TP=2),
+# Dankserver :7770, publisher's 4-bit-codebook container of the official
+# Qwen3.8-Flash-Next.
 #
-# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
-# with the per-card budget spent on resident experts: 25,600 MiB of slab +
-# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
-# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
-# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
-# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
-# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
-# +25 % at the full window) while KV beyond the contract buys nothing.
-# Consequence, deliberate: a FOURTH full-height conversation does not fit this
-# pair, and the engine refuses it — it never truncates. See the KV block for the
-# arithmetic and the measured dtype law. The w4dn8 payload is the swap
-# rollback and keeps its trio under the w4dn8 runbook; the fp8-native payload
-# that came before it was DELETED from disk 2026-10-15 (Frank's call, to free
-# the volume for this download — rebuildable via the fetch + build scripts
-# named in the fp8-native runbook).
-#
-# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
-# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
-# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
-# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
-# frank-archon-vllm-shapeG-20261003).
-#
-# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
-# gfx1201, drives `podman run` directly from the digest-pinned
-# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
-# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
-# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
-# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
-# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
-# THE 4-bit-codebook publisher shape is the DEPLOYED payload since
-# 2026-10-15. The card's own run line (--host-pool-mib 12288,
-# --expert-vs-cache-ratio 0.82, seqs 8, spec 3) is the publisher's
-# suggestion on 32 GB cards; this file's shape is the fleet's measured one.
-#
-# PAYLOAD: /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad — the
-# official Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's
-# own author (recipe qwen4exp-w4nl64-i8-hc8m: experts as 4-bit codes into a
-# 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation,
-# GPTQ + 3 coordinate-descent sweeps on a 10M-token calibration, ten
-# bf16-protected experts per layer, int8 W8A8 trunk + lm_head, fp8 HC
-# mixtures, PLE n-gram fp8 fixed-scale, MTP head, bf16 vision tower; card KL
-# 0.0789 all-tokens / top-1 92.85 % vs bf16). Fetched byte-exact + sha256 by
-# scripts/download-frank-archon-iq4r.sh; the recipe/LICENSE stay in the
-# staging dir beside it as provenance. ROLLBACK: revert the swap commit — the
-# in-house qwen4exp-swift15-w4dn8-i8-hc8m.rad (scripts/convert-swift15-w4dn8.sh,
-# KLD-gated) stays on disk. Behind it, the fp8-native
-# qwen4exp-fp8-native.rad was DELETED from disk 2026-10-15 (Frank's call,
-# one-time rm, to free the volume for this payload); rebuild =
-# scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh — a
-# NO-RECIPE rad-convert copy of the official 131-shard checkpoint under the
-# descriptor law (no shipped plugin claims (qwen4exp, "fp8"), so the staging
-# config.json drops quantization_config; per-weight encodings are data, not
-# the descriptor).
-#
-# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
-# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
-#   then engine.serve()) — during load the port refuses connections, so a
-#   bare curl /health gate is honest. /health == /ping: 200 JSON while
-#   serving, 503 only while stopping (core/server/admin.cpp:385).
-# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
-#   `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
-#   the chat format. Both need no parser flags.
-# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
-#   typical_p, presence_penalty, frequency_penalty, repetition_penalty,
-#   repeat_last_n, chat_template_kwargs, reasoning_effort,
-#   enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
-#   models.yml's extraBody set is fully inside it. Unknown params are
-#   refused (400), which is why nothing unverified rides the wire.
-# - MTP: the .rad carries the model's mtp.* head (the publisher's recipe
-#   names the mtp.fc_hidden/fc_embedding/draft_head rules; card default
-#   depth 3). Served at the depth named by FRANK_ARCHON_SPEC_TOKENS below —
-#   the depth this shape's decode ladder was measured at.
-# - The fused TP all-reduce is declared over max-num-seqs x
-#   (1 + --num-speculative-tokens) rows; 3 x (1+2) = 9 rows, under the 25-row
-#   exactness bound of the wht6 wire.
-# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
-# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
-# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
-# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
-# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
-# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
-# Acceptance on the greedy harness is deterministic (the control
-# reproduced the recorded 0.755 exactly), so the drop is a real numerics
-# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
-# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
-# The vram-pair translation itself is exact (1.2.0 boot ledger took
-# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
-# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
-# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
-# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
-# file already uses the pair, which BOTH pins accept (rollback symmetry).
-FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
-# The .rad container FILE (mounted under /models) and the state dir for the
-# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
-# .rad files; the raw HF download and its staging dir live beside it and
-# are never mounted.
+# ONE OWNER: shape, payload identity, eras, laws and every measured number live
+# in serverAppConfigs-docs/LLM/frank-archon.md. This file carries the knobs plus
+# only the reason a reader editing THIS line cannot recover from the page.
+# Where the two disagree, this file wins on values; the boot ledger arbitrates
+# the engine shape.
+FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed
+# 1.8.1 (2026-10-18), pinned by digest. Two CLI breaks sit between it and the
+# 1.0.13 line this endpoint ran for two eras, and both are boot refusals, not
+# warnings: --vram-weights-mib/--vram-kv-mib were REMOVED at 1.3.0 (the card is
+# now split by the engine's budget resolver — see below), and the prefix cache
+# lost its disk tier. Draft acceptance has to be re-scored on any pin past 1.2.0:
+# 1.2.0's MoE weight rotation measured 0.7123 against the 0.7549 control, and the
+# suspect is untangled (the page's Dead ends section names it).
+# The .rad file mounts under /models; the tune cache mounts separately below.
+# The model dir holds ONLY .rad files — the HF download and staging stay beside it.
 FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
 FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
 FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
-# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
-# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
-# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
-# cache must ride a host mount — serve script mounts it, see the runbook's
-# "Tuning levers").
-# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
-# name across systemctl and podman ps. The runtime changes; the name does
-# not. The legacy list retires every predecessor identity that could hold
-# the GPUs or the port.
+# state dir's tune/ subdir carries rad-tune's kernel-selection cache; the engine
+# reads $RADIANCE_HOME/tune/<machine>.tune and that defaults to /opt/radiance
+# (inside the image), so the cache only survives on a host mount.
+# Container identity is the SERVICE name: one name across systemctl and
+# `podman ps`; the runtime changes, the name does not. The legacy list retires
+# every predecessor identity that could hold the GPUs or the port.
 FRANK_ARCHON_CONTAINER=frank-archon
 FRANK_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
-# Served model id — Protoss name (wire rename 2026-10-09). The container,
-# unit, scripts and this file keep the frank-archon name above; only the
-# alias on the wire moved. Radiance validates the requested model name, so
-# every client (harness, bench scripts, probes) MUST send this value.
+# Served id. Radiance VALIDATES the requested model name, so every client must
+# send exactly this value (unlike llama.cpp and halogen, which ignore it).
 FRANK_ARCHON_SERVED_NAME=frank-archon
 FRANK_ARCHON_PORT=7770
-# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
-# repo's root chat_template.jinja, vendored here; the checkpoint's own
-# template is baked into the .rad by rad-convert and this overrides it).
-# Radiance derives the reasoning/tool-call reply format from it.
+# Frank's chat-template call: froggeric's fixed Qwen3.8 template, vendored in
+# the repo; it overrides the template baked into the .rad, and radiance derives
+# the reasoning and tool-call reply format from it.
 FRANK_ARCHON_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
 
 # --- engine shape (the boot ledger arbitrates every number below) ---------
-# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
-# x8/x8 on root ports) made collectives legal on this host; radiance's TP
-# wire is libr4d's own, not RCCL.
+# TP=2 needs the 2026-09-20 board (x8/x8 on root ports): a collective cannot be
+# dispatched on a card whose PCIe chain lacks atomics. Radiance's wire is
+# libr4d's own, not RCCL.
 FRANK_ARCHON_TP=2
-# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
-# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
-# engine default.
+# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice). It
+# stays EXACT for single-sequence decode and any step of <= 25 rows — see
+# MAX_SEQS x (1 + SPEC_TOKENS) below.
 FRANK_ARCHON_TP_WIRE=wht6
-# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
-# concurrent conversations of --max-model-len fit the KV pool (see the KV
-# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
-# tcclaviger vLLM fork's torch.compile shape specialization on
-# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
-# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
-# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
-# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
-# not speed, so a slot costs KV, never decode.
-# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
-# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
-# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
-# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
-# steps are lossy).
-# The pool is sized to this contract and nothing more (see the KV block):
-# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
-# the resident expert slab. The trade is explicit and it is Frank's call:
-# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
-# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
-# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
-# traffic that wants four full-height sessions is a --vram-kv-mib change, not
-# a code change.
-FRANK_ARCHON_MAX_SEQS=3
-# Served window. Native is 262,144; 200,000 is the publisher's production
-# value (compose) and what models.yml's contextWindow must quote — a longer
-# prompt is refused by the engine, never truncated. The engine treats
-# `kv group tokens >= max-model-len` as a HARD limit ("ONE request longer
-# than N tokens is accepted by the API and then refused as too large to ever
-# fit — this is a hard limit, not a backing ratio"), so the pool is sized
-# for seqs x maxlen, not for one sequence.
+# Decode slots. A slot costs KV, never decode; the pool is sized to this
+# contract and nothing more. At 1.8.1 the budget resolver caps the cache at
+# 556,348 fp8 tokens — 2.8 sessions at 200,000 — so a THIRD full-height
+# conversation answers 400 while /health and every shorter chat work fine
+# (3 x 50k streams run at 77.6 t/s aggregate). Frank's trade: declare the two
+# slots the engine can hold and let the freed ~1.1 GiB go to the expert plane,
+# instead of advertising a third slot that only exists at shorter prompts.
+FRANK_ARCHON_MAX_SEQS=2
+# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD
+# limit, and models.yml contextWindow MUST quote this number (over-length is a
+# refusal, not a truncation). The pool is sized for seqs x maxlen, not one
+# sequence.
 FRANK_ARCHON_MAX_MODEL_LEN=200000
 FRANK_ARCHON_PLACEMENT=expert_tiered
-# Pinned host pool for the expert rows that do not fit VRAM. vram_weights and
-# this pool serve the per-rank resident set TOGETHER: at the fp8-KV pair
-# (26071/3543) the ledger showed the VRAM slab filled to 259 KiB of slack,
-# i.e. the pair was exactly sized. Cutting vram-weights by 3.8 GiB to pay for
-# the bf16 KV pool SPILLS ~3.8 GiB/rank here, so the pool moves with it —
-# under-sizing it is a boot failure (36864 refused the boot in the fp8 era).
-# The boot ledger's coverage line remains the referee, not this arithmetic.
+# Pinned host pool for expert rows that do not fit VRAM. Under the budget
+# resolver this is ALSO the lever that decides which experts stay on the card:
+# the card's leftover goes to the expert plane first and the KV cache takes the
+# rest, so growing the pool moves experts off the card and shrinking it spills
+# them onto VRAM the cache wanted. Under-sizing it is a boot failure (36,864
+# refused a boot). The ledger's coverage line is the referee, not this file.
+# Measured, not guessed: 45,568 (PR #30) moved host_pool to 45.09 GiB and left
+# vram_kv byte-identical at 4.20 GiB / 556,348 tokens on both ranks — extra
+# host RAM above what the expert plane's footprint needs buys no cache. Back at
+# 44,928, the size that boots clean.
 FRANK_ARCHON_HOST_POOL_MIB=44928
-# VRAM left unclaimed per card (publisher default 96 MiB).
+# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle
+# floor is built from it.
 FRANK_ARCHON_GPU_HEADROOM_MIB=96
-# VRAM split of claimed memory per card, MiB (0 = derive from the card).
-# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
-# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
-# neither budgeted nor accounted — the driver's context, the activation arena
-# and --gpu-headroom-mib come out of it; anything past those is throughput on
-# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
-# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
-# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
-# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
-# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
-# the engine's; the pair below spends the difference and lands 458 MiB above
-# the engine's floor. `weights + kv <= 30,969` is the gate.
-# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
-# expert cache's spillover (`mover: 2275 slots can be lent out of the KV
-# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
-# so a bigger KV pool steals weight residency from the slab AND from the
-# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
-# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
-# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
-# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
-# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
-# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
-# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
-# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
-# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
-# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
-# The margin is thin by design (608,027 predicted tokens against the 600,000
-# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
-# once` line is the referee for THIS pair and the concurrent probe is its
-# regression test.
-# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
-# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
-# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
-# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
-# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
-# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
-# pair landed inside noise of EACH OTHER, which is what made it read as a
-# plateau; two points are not a curve.
-# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
-# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
-# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
-# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
-# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
-FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
-FRANK_ARCHON_VRAM_KV_MIB=5120
-# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
-# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
-# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
-# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
-# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
-# 2026-10-08, bf16 boot 2026-10-11):
-#   kv_attn     13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
-#   kv_qsa_bkey 13 layers, 4-token blocks, page  3.25 KiB at BOTH dtypes —
-#               the QSA block key is an index, not attention state, and does
-#               NOT follow --kv-cache-dtype
-#   => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
-#               not 2x, precisely because the bkey plane is dtype-fixed.
-# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
-# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
-# checkpoint region 8 x 55.99 = 448 MiB. So:
-#   pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
-#   3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
-#   and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
-#   three-slot contract with 1.3 % slack, every MiB above 5,063 having been
-#   handed to the expert slab instead. The ledger line is the referee ("backs N
-#   sequences all at 200000 tokens at once") — a pair this tight is read off the
-#   boot, never assumed.
-#   At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
-#   the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
-#   11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
-#   slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
-#   tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
-#   sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
-# Secondary saving: the prefix-cache entry shrinks with the attention plane
-# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
-# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
-# many conversations as it did at bf16 — no re-size needed to park three slots.
+# There is no VRAM pair any more. --vram-weights-mib/--vram-kv-mib were removed
+# at 1.3.0: the engine takes capacity - already-held - GPU_HEADROOM - arena,
+# puts the static (non-expert) weights off the top, then divides the rest between
+# the expert plane and the KV cache — experts first when the cache has no idle
+# blocks to lend, and never less than ONE MAX_MODEL_LEN sequence for the cache.
+# What moves the split from here is HOST_POOL_MIB (which experts stay resident),
+# MAX_MODEL_LEN/MAX_SEQS (the cache bound) and GPU_HEADROOM_MIB. The slab size
+# the prefill-slope law cares about is a boot-ledger line now, not a knob: read
+# it after every resize instead of asserting it here.
+# fp8 is the ONLY rung below bf16 on this engine: one kv_dtype for the whole
+# cache, so llama.cpp's exact-K/quantized-V split (frank-zealot) is unservable
+# here. bf16 costs 1.89x fp8, not 2x — the QSA block-key plane is dtype-fixed.
 FRANK_ARCHON_KV_DTYPE=fp8
-# Spec decode: the checkpoint's own MTP head, served at depth 2 — the
-# publisher's card ships depth 3 as its peak; this fleet's ladder was
-# measured at 2 and the swap keeps every argument, so 2 stays (rows
-# 3 x (1+2) = 9, inside the wht6 25-row exactness bound).
+# The checkpoint's own MTP head at depth 2: this fleet's ladder was measured at
+# 2, and 3 x (1+2) = 9 rows stays inside the wht6 25-row exactness bound.
 FRANK_ARCHON_SPEC_TOKENS=2
 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
-# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
-# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
-# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
-# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
-# parked set fits with the tier as it stands — at RAM the host has (104 GiB
-# available with the service loaded).
-FRANK_ARCHON_PREFIX_HOST_MIB=8192
-FRANK_ARCHON_PREFIX_DISK_MIB=65536
-# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
-# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
-# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
+# Step sharing. A prefill chunk that shares a step with decoding sequences holds
+# the device for the chunk's whole time, and every decode row in that step waits
+# that long for its next token (the census: [[LLM Performance Diagnosis]] §9).
+# `auto` leaves each decode row behind the chunk a row plus its draft window and
+# keeps a share for whatever is waiting to be admitted; a number pins the chunk's
+# cap; `off` is the pre-1.4 behaviour this endpoint ran for two eras, where the
+# chunk took the step and admission never ran. Interleave inserts N decode-only
+# steps after a chunk step taken beside decoders: it protects their cadence and
+# holds a waiting request out of those steps, so it stays 0 until the post-migration
+# decode-vs-prefill buckets say it is wanted.
+FRANK_ARCHON_BUSY_PREFILL_CHUNK=auto
+FRANK_ARCHON_PREFILL_INTERLEAVE=0
+# Prefix cache: pinned host memory only. The disk tier was removed outright
+# (--prefix-cache-dir and --prefix-cache-disk-mib are refusals), so what used to
+# be restored from NVMe is now re-prefilled. The tier is sized in ENTRIES and one
+# entry tracks the cache dtype (58.50 KiB at fp8). Watch prefix_hit_rate after
+# this change: a drop is extra prefill, and prefill is exactly the traffic that
+# costs decode its cadence.
+FRANK_ARCHON_PREFIX_CACHE_MIB=8192
+# Frank's pin law: the PLE n-gram table serves from pinned host RAM the cards
+# read. `ram` REFUSES the start if memlock is too small — the unit's
+# LimitMEMLOCK=infinity is the other half of this law.
 FRANK_ARCHON_NGRAM_PLACEMENT=ram
-# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
-# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
-# pages_limit (the current kernel exposes NO pages_max; the cap function
-# takes the min over the knobs that exist). Below the plan the load dies
-# mid-pin with an allocator error instead of this named refusal.
-# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
-# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
-# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
-# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
-# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
-# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
-# root unit is privileged; the knob is a 0644 root module parameter), so the
-# recovery is a boot, not a visit to the machine.
+# TTM preflight. The pinned ledger rides TTM's shared pages, and the kernel
+# RESETS pages_limit to half of RAM on every boot; below the plan the load dies
+# mid-pin with an anonymous allocator error. ttm-preflight.sh raises the knob
+# itself before refusing by name, so recovery is a boot, not a visit.
+# Floor = 2 x HOST_POOL + 48,824 (static weights + arena) + PREFIX_CACHE + MNBT
+# per the bats ledger test; at the 44,928 pool that computes to 148,920 MiB.
 FRANK_ARCHON_TTM_MIN_MIB=150000
 FRANK_ARCHON_TTM_TARGET_MIB=204800
-# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
-# multi-flag values as one string; the serve script expands it unquoted on
-# purpose.
+# Escape hatch for experiments. Unset -> no extra argv; the serve script expands
+# this unquoted on purpose, so keep multi-flag values as one string.
 FRANK_ARCHON_EXTRA_ARGS=
-# The .rad's minimum size in MiB (deployer gate; a truncated fetch is a
-# guaranteed long-timeout boot instead of a fast failure). The deployed
-# publisher container measures 116,322 MiB (121,969,901,568 B); the
-# in-house w4dn8 rollback measured 116,416 MiB. Floor sits just under both.
+# Deployer gate: a truncated fetch must be a fast refusal, not a long-timeout
+# boot. Floor sits under the live payload and under every rollback payload
+# (see the page's Eras table for what is on disk and what needs a re-download).
 FRANK_ARCHON_RAD_MIN_MIB=110000

scripts/frank-dragoon.env

@@ -1,209 +1,82 @@
 # frank-dragoon.env — tunables for the frank-dragoon endpoint on Dankclaw's
 # Strix Halo iGPU (gfx1151), host port 7770.
 #
-# RUNTIME (2026-10-17, Frank): halogen-flash-server 0.16.2 — the
-# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
-# digest-pinned below) serving Quat3rnion/halogen-qwen3.8-flash-next-v2-
-# abliterated — the NATIVE v2 conversion (2026-10-11) of orcarouter's pinned
-# BF16 Qwen3.8-Flash-Next, abliterated — as a podman container under a root
-# unit, driven by scripts/serve-frank-dragoon.sh. This CUTS OVER the
-# 2026-10-06 Swift1.5-finetune shape; rollback = revert the swap commit and
-# redeploy: the swift15 payload stays on disk at
-# /home/dank/models/halogen-flashnext-v2/, the 27B-era payload beside it.
+# ONE OWNER: payload identity, shape, eras, laws and every measured number live
+# in serverAppConfigs-docs/LLM/frank-dragoon.md. This file carries the knobs
+# plus only the reason a reader editing THIS line cannot recover from the page.
+# Where the two disagree, this file wins on values; the boot ledger arbitrates
+# the memory figures.
 #
-# MEASURED A/B, both boots on this box, identical battery traffic (engine
-# tools with the service stopped, behavior probes against it, 2026-10-17):
-# NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls and the
-# red-PNG vision probe identical, needle hit at both depths of a 21.5k-token
-# haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s,
-# wikitext-2 PPL 3.968 vs 3.877 (the finetune's edge is gone: +2.3 %) and
-# thinking tokens 1,476 vs 784 mean — the finetune compressed reasoning, the
-# base thinks ~1.9x longer per answer.
-#
-# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered ctx to
-# 200,704 (RAM + the harness's compaction budget) and, on 2026-10-18, raised the
-# pool to 2 x ctx = 401,408 so BOTH slots can carry a full-length request at
-# once; the rates above are the ceiling-262,144 record and the production ledger
-# is the ctx-200,704 one.
-#
-# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
-# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
-# .hgn loads ONLY in the flash-server line (the old halogen engine is for
-# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
-# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
-# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
-# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
-# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
-# red-PNG probe verified on the card; the 27B engine ignored its encoder),
-# tool calls verified, and the native 262,144 context — served at 200,704 by
-# choice from 2026-10-17 (see SHAPE: the ceiling is a RAM decision, not a
-# capability limit).
-#
-# SHAPE — the knobs that are NOT free, with the arithmetic:
-#   HALOGEN_CTX=200704 is the PER-REQUEST ceiling and the per-conversation
-#   hard limit — a prompt over it is a 400 naming the limit, never a silent
-#   truncation. LOWERED from 262,144 on 2026-10-17 at Frank's direction (the
-#   checkpoint's native window is 262,144 and the engine serves anything
-#   below it): every position costs KV pool RAM and shrinks the prompt-cache
-#   entry (111 MiB at 262,144, ~85 MiB here). The 400-zone law is INVARIANT
-#   to the value — it says the harness window must sit ~45k UNDER the engine
-#   ceiling, because the harness clamps max_tokens to contextWindow − est −
-#   4096, its chars/4 est UNDER-counts the rendered prompt by ~3 % on
-#   code/agent traffic, and compaction only fires at 85 % (measured 2026-10-07
-#   at the old 200,192 ceiling: est 152,938, server 157,388, over by 354).
-#   Harness contract (models.yml): contextWindow 155648 (= ceiling − 45056,
-#   the gap that absorbs the estimate error), maxTokens 45056.
-#   Compaction fires at 85 % = 132,301 est ≈ 136k measured — the 180k target
-#   of the 262,144 era is unreachable at this ceiling: the window itself is
-#   155,648. Worst-case request: prompt + budget <= 155648 − 4096 + 3 %
-#   est-error ≈ 156.2k <= 200,704 — ~9.5× the 3 % error of margin (robust to a
-#   ~25 % under-count). Compaction rides on frank-zealot: input ≈ threshold −
-#   keepRecent(20k) ≈ 112k, inside its 200,704 slot.
-#   HALOGEN_KV_POOL_POSITIONS=401408 = SLOTS x CTX (2 x 200,704), the top of the
-#   legal band (the seam enforces ctx <= pool <= slots * ctx). Still ONE pool
-#   behind HALOGEN_KV_SLOTS=2 (ledger form: "2 slots over ONE <pool>-position KV
-#   pool, each request up to HALOGEN_CTX"), but sized so BOTH slots can hold a
-#   MAXIMUM-length request at the same time — the parity with frank-zealot's
-#   401,408-position pool that Frank ordered on 2026-10-18. The PER-REQUEST
-#   ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the
-#   checkpoint's native window is 262,144 and the engine refuses to go past it
-#   without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position
-#   (FLAGS.md) ⇒ the DEPLOYED 2026-10-18 boot RESERVES 11.0 GiB (`reserving the
-#   KV pool, 401408 positions (about 11.0 GiB)`), where pool == ctx (200,704)
-#   cost 5.5, 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6.
-#   The preflight formula bills 11.3 (block scratch + ~115 MiB/slot of O(1)
-#   state) — quote the RESERVATION. FLAGS.md law: "RAM the pool
-#   takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this
-#   much page cache, so the warm prompt-cache ratio is what to re-measure.
-#   HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
-#   drafter is NOT slot-locked on this engine (batching and speculation
-#   coexist — DFlash-era slots>1 law does NOT carry over; the card's own
-#   quickstart ships 2 slots WITH the head), so the second slot is
-#   concurrent capacity, not a speculation tax.
-#   Memory ledger against the box's 121.2 GiB unified (the n-gram table is
-#   mmap'd page cache beside the checkpoint, not a pinned charge — the boot
-#   ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"):
-#     weights 62.9 GiB locked (0.8 of them the vision tower) + KV 11.0 +
-#     working 5.7 (HALOGEN_MAX_TOK=8192) = 79.6 GiB engine-side (the boot
-#     ledger of the 401408-position deploy, 2026-10-18), with the 1.3 GiB
-#     prompt cache (12 entries x 111 MiB at ctx 200704) booked separately by
-#     the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host honest. Against
-#     the 121.2 GiB box that leaves ~20 GiB over the reserve (MemAvailable
-#     read 22 GiB right after the raise) for baseline co-tenancy
-#     (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot ledger is the
-#     referee — read it after deploy and re-size if the engine prints a
-#     clamp.
-#   HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
-#   turns the MTP head off for whole requests after low acceptance and
-#   HALVES decode as a result. Frank accepted the head always-on trade
-#   (temp 1.0 traffic costs acceptance; the card still ships the recipe).
-#
-# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
-# ~/.omp/agent/models.yml):
-#   * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
-#     the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
-#     an absent temperature is no longer greedy. models.yml extraBody
-#     carries the same values — consistent, not load-bearing.
-#   * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
-#     max_tokens; the harness sends at most its maxTokens 45056. Requests
-#     over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
-#     queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
-#     worst case needs > 6550 s; 7200 honors it. Do not move one alone.
-#   * max_tokens INCLUDES thinking; a reasoning reply over a small budget
-#     returns empty content with finish_reason "length".
-#   * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
-#     100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
-#     unlike the 27B engine, auto-sizing fits this box's ledger.
-#   * Thinking lands in reasoning_content (same as the 27B engine); the
-#     v2 chat template rejects reasoning_effort outside xhigh|medium|low,
-#     so the models.yml fold (high->xhigh) still applies.
-#   * Vision: OpenAI image_url content parts only (the fleet's standard
-#     contract). >=256 px probe images or the model answers "video".
-#   * Concurrency at depth is tight now: the KV pool is ONE 200,704-position
-#     budget behind 2 slots — exactly one maximum-length request fills it. Two
-#     conversations coexist only while they stay under ~100k each. The
-#     PER-REQUEST ceiling moved WITH the pool (262144 -> 200704), so the harness
-#     contract moves too: contextWindow 155648 = ceiling − 45k, never higher.
-#   * response_format json_object/json_schema unsupported in this line
-#     (no constrained decoding) — unchanged from the 27B engine.
-#   * The engine's token protocol has no auth and binds loopback INSIDE
-#     the container; only the frontend port (8731) is published. Same as
-#     before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
-#     amdgpu.noretry=0 (check at deploy).
-
-# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
-# is what was qualified). 0.16.2 = the version the model card names as its
-# quickstart pin; RepoDigest captured at pull on 2026-10-06.
+# The .hgn format is the lock: it loads ONLY in the flash-server line, so the
+# engine moves with the payload, never ahead of it.
+# Image digest-pinned (a tag pull silently re-points; the digest is what was
+# qualified — 0.16.2 is the version the model card names as its quickstart pin).
 FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
-# Container identity: kept from the 27B era (unit ExecStop names it).
+# Container identity kept from the 27B era: the unit's ExecStop names it.
 FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
-# Old quadlet-era container identity; the serve script clears stale names.
+# Predecessor identity the serve script clears if it is still holding the GPUs.
 FRANK_DRAGOON_LEGACY_CONTAINERS=frank-dragoon
-# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
-# default); the engine port (8730) is never published.
+# host:container — the frontend listens on 8731 inside; the engine's own port
+# (8730) has no auth and is NEVER published.
 FRANK_DRAGOON_PUBLISH=7770:8731
 
 FRANK_DRAGOON_MODEL_DIR=/home/dank/models/halogen-v2-abliterated
-# The checkpoint: Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated
-# qwen38-flash-next-v2-abliterated.hgn — the NATIVE 2026-10-11 v2 conversion
-# of orcarouter's pinned BF16 Qwen3.8-Flash-Next, abliterated. NOT the writer
-# transplant this filename used to carry: 66,687,678,432 B where the swift15
-# r2 measured 66,687,678,464 B — same filename family, different artifact.
-# Hub LFS sha256 verified at fetch by scripts/download-frank-dragoon-v2abl.sh.
+# Same filename family, DIFFERENT artifact: this is the native v2 conversion,
+# 32 bytes smaller than the writer transplant that used to carry the name.
+# Never infer the payload from the filename — this size and sha256 are the gate.
 FRANK_DRAGOON_CHECKPOINT=qwen38-flash-next-v2-abliterated.hgn
 FRANK_DRAGOON_CHECKPOINT_BYTES=66687678432
 FRANK_DRAGOON_CHECKPOINT_SHA256=b5bd925fa2b44343be4a2cd328cae6b57ce7ffa2fe2cc56281717377e5a1ce8a
-# The n-gram lookup table: REQUIRED (the engine reads it beside the
-# checkpoint; the card's quickstart names it explicitly). From
-# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
-# The card does NOT establish this stock table as matched to the NATIVE
-# conversion and no source-matched table is published, so the engine's own
-# tools are the referee: NIAH 18/18 and PPL 3.968 on it (2026-10-17). It is
-# HARDLINKED from the swift15 payload dir (a hardlink is a regular file; the
-# serve gate refuses symlinks), i.e. byte-identical to the file that era
-# served.
+# The n-gram table is REQUIRED (the engine reads it beside the checkpoint) and
+# disk-backed: mmap'd page cache, never pinned. It is a stock peonist companion
+# hardlinked from the previous payload dir — a hardlink is a regular file, and
+# the serve gate refuses symlinks — with the engine's own NIAH/PPL verdicts on
+# the page as the referee.
 FRANK_DRAGOON_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
 FRANK_DRAGOON_NGRAM_BYTES=51200246144
 FRANK_DRAGOON_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
-# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
-# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
+# Vision tower: optional on the engine, ENABLED here (image input).
 FRANK_DRAGOON_VISION_TOWER=qwen38-flash-next-vision.hgn
 FRANK_DRAGOON_VISION_BYTES=897916416
 FRANK_DRAGOON_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
-# The tokenizer is a FLAT DIRECTORY, $FRANK_DRAGOON_MODEL_DIR/tokenizer,
-# mounted whole at /tokenizer. The two repos ship the same one; fetched
-# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
-# the 27B-era copy; the rest are plain git blobs — size-checked only). The
-# serve gate and the deployer refuse a missing file or ANY symlink: an HF
-# cache snapshot symlinks each file into blobs/, and those links dangle
+# The tokenizer is a FLAT DIRECTORY ($FRANK_DRAGOON_MODEL_DIR/tokenizer) mounted
+# whole. The serve gate and the deployer refuse a missing file or ANY symlink:
+# an HF cache snapshot symlinks each file into blobs/ and those links dangle
 # inside the container mount.
 FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
 
 # --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
 # stays at the image default, which is what was qualified) ------------------
-# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
-# scripts and this file keep the frank-dragoon name. OpenWebUI api_configs
-# model_ids for the :7770 backend MUST name this value.
+# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it.
 HALOGEN_MODEL_ID=frank-dragoon
+# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a
+# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on
+# the page) — a window equal to the ceiling makes every long request a 400.
 HALOGEN_CTX=200704
+# ONE pool behind both slots, sized so BOTH can carry a maximum-length request
+# at once. Pool positions buy concurrency; the per-request window never moves
+# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and
+# RAM the pool takes is RAM the n-gram table's page cache loses.
 HALOGEN_KV_POOL_POSITIONS=401408
+# Two concurrent conversations: the MTP drafter is not slot-locked on this
+# engine, so the second slot is capacity, not a speculation tax.
 HALOGEN_KV_SLOTS=2
 HALOGEN_HOST_RESERVE_GIB=20
-# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
-# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
-# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
+# The MTP head is INSIDE the checkpoint — do NOT set HALOGEN_MTP_HEAD.
+# 0 is mandatory: the adaptive default disarms the head for whole requests after
+# low acceptance and halves decode. Depth 2 is the card's best-all-round default.
 HALOGEN_SPEC_ADAPT=0
-# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
-# the GPU, the timeout bounds how long the NEXT client waits. Do not move
-# one alone (see the client-contract note above).
+# Coupled trio: MAX_TOK is only the default for requests that omit max_tokens;
+# the cap bounds how long ONE request holds the GPU and the queue timeout bounds
+# how long the NEXT client waits. Move all three together or none.
 HALOGEN_MAX_TOK=8192
 HALOGEN_MAX_TOKENS_CAP=65536
 HALOGEN_QUEUE_TIMEOUT=7200
-# Server-side sampler = the card's thinking recipe (the engine now HAS one;
-# requests may still override per-call). Matches models.yml extraBody.
+# The engine has a server-side sampler (an absent temperature is NOT greedy
+# here); models.yml extraBody carries the same recipe.
 HALOGEN_TEMPERATURE=1.0
 HALOGEN_TOP_P=0.95
 HALOGEN_TOP_K=20
-# Card's belt-and-braces: the engine pins weights by default
-# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
+# Weights stay pinned: the engine pins by default and only the retired w4b build
+# honored HALOGEN_FLASH_PIN_TRUNK=0, which this payload rejects.
 HALOGEN_WEIGHTS_LOCK=1

scripts/frank-tts.env

@@ -57,6 +57,16 @@ TTS_BACKEND=omni-http
 TTS_UPSTREAM=http://frank-tts-engine:8880
 TTS_UPSTREAM_MODEL=voxcpm2
 
+# THE DEFAULT VOICE — what a request that names NO voice speaks. The omp harness
+# has no setting that feeds a voice to its speech device (`voice_id` is a
+# per-call argument; `speech.voice` belongs to the TUI's Kokoro speak-aloud and
+# never reaches an HTTP request), so a harness voiceover arrives voice-less and
+# would otherwise be a 400 for every caller that does not name one. An UNKNOWN
+# name still 400s with the list that exists — this covers the absent case only —
+# and the journal marks such a line `(default)` so an unrequested voice never
+# reads as a requested one. Any pack directory in /home/dank/voices is legal.
+DEFAULT_VOICE=narrator1
+
 # CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not
 # configured": the endpoint answers a 500 that says so instead of running
 # `nice env -m ...` with an empty binary name.

scripts/frank-zealot.env

@@ -1,141 +1,72 @@
-# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving
-# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file**
-# on the RTX 3090, :7771.
+# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090,
+# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file.
 #
-# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export
-# at THREE 262,144-token conversations (Frank: "if you can possibly fit a third
-# context, do it"). The 9B bills HALF the KV of the 27B per token, which is what
-# funds 3 x the native window on one 24,123 MiB card. Speculative decoding ships
-# OFF — measured, not assumed; see the MTP law below, which is the whole story
-# and the reason the publisher's "depth 2 works best" line does not transfer.
-#
-# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
-# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata
-# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at
-# load and a loader that ignored the Hadamard metadata would dequant WRONG. On
-# top of that lock, MTP beside a quantized V cache needs fork build >= b10776,
-# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to
-# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback
-# runtime for the 27B payload.
+# ONE OWNER: shape, payload identity, eras, laws and every measured number
+# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the
+# knobs and only the reason a reader editing THIS line cannot recover
+# elsewhere. Where the two disagree, this file wins on values.
 LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server
-# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by
-# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft
-# head ships INSIDE the checkpoint, so MTP availability is a property of the
-# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag
-# value; loading this file instead of that one is the only change that matters.
+# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.*
+# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard
+# metadata dequantizes wrong, silently.
 MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf
-# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference:
-# the publisher's card says the BF16 mmproj "can crash llama-server when several
-# image requests run at once; Q8_0 is not affected", and this endpoint runs
-# --parallel 3, i.e. exactly that shape. The projector's worst case lands AFTER
-# the engine's own fit check, so it is never inside a projection number.
+# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server
+# under concurrent image requests, which is exactly --parallel 3.
 MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf
-# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export
-# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it
-# matched, NOT to this 9B file — if a refusal ever surfaces there is no
-# one-line swap here; the 9B dealigned export would have to be fetched first.
-#
-# Served model id — the harness/family name. The SERVICE, unit, scripts and
-# this env file keep the frank-zealot name; only the alias on the wire is the
-# model id. Consumers that key on the served id: OpenWebUI (model row,
-# access_grant, api_configs model_ids) and the harness models.yml.
+# Served id — the harness/family name, not a fact about the payload.
+# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml.
 ALIAS=frank-zealot
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7771
-# CTX_SIZE is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144 per
-# conversation — each slot at the model's TRAINED window, which is the point of
-# the swap (the 27B era had to cut each conversation to 200,704 to fit two).
-# MUST stay divisible by PARALLEL (the seam enforces it) and the harness
-# contextWindow quotes the PER-SLOT number — a longer prompt is answered with a
-# 400, never truncated. The ceiling cannot go past 3 x 262,144: that is the
-# native wall (no YaRN in this path) and the arithmetic agrees — a 4th slot
-# would push each below the compaction contract.
-CTX_SIZE=786432
+# CTX_SIZE is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
+# conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
+# models.yml contextWindow quotes the PER-SLOT number — a longer prompt is a
+# 400, never a truncation. The pool is bought, not granted: K+V at 25,088
+# B/token is 14,406.00 MiB here and the MTP draft context bills its own f16 KV
+# over the WHOLE pool (4,096 B/position ⇒ 2,352.00 MiB), so the old 786,432
+# pool cannot carry a drafter at any KV dtype — 18,816 + 3,072 of cache alone
+# against the 24,123 MiB the driver leaves addressable.
+CTX_SIZE=602112
 PARALLEL=3
-BATCH_SIZE=4096
-# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this
-# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a
-# second job here: it sets the compute-buffer size, and the compute buffers are
-# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm.
-UBATCH_SIZE=2048
-# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with
-# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet;
-# 4 KV heads × 256 dim = 8,192 elements per token per cache. q8_0 bills 8,704 B
-# per cache ⇒ 17,408 B/token = 17 KiB, so the 786,432 pool is 13,056.00 MiB —
-# the ledger said exactly that. HALF the 27B's 34,816 B/token, which is the
-# whole reason three native windows fit. f16 would bill 26,112 MiB at this pool
-# (against a 24,123 MiB card) and dies at cudaMalloc; the 24 DeltaNet layers
-# bill a fixed recurrent state per SLOT instead of per token — the PRODUCTION
-# ledger prints "RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)"
-# (~50 MiB a cell), while the SPEC-ARMED arms print 452.25 MiB, because MTP
-# depth 2 bills three seqs per slot (9 cells) — which is why pool arithmetic
-# alone under-prices the shape.
-CACHE_TYPE_K=q8_0
+# The logical batch is VRAM- and throughput-neutral at a fixed ubatch (the law
+# on [[frank-zealot]]); 1024×6 keeps it a clean multiple of the ubatch.
+BATCH_SIZE=6144
+# Ubatch is a prefill knob (decode never moves with it) and it sets the
+# compute-buffer size, which decides whether a drafter fits. 1024 is the rung
+# every measured MTP arm here ran on (213.5 t/s against 109.2 at 2048). Its
+# VRAM credit is SHAPE-specific: the 786,432-pool arm census showed ~1,306 MiB
+# back, this boot measured 2,089.88 MiB of target compute against 2,496.13 at
+# ubatch 2048 — 406 MiB back, with the split moving (CUDA0 +257, CUDA_Host
+# −663). The drafter is funded by CACHE_TYPE_V below, not by this knob.
+UBATCH_SIZE=1024
+# K exact, V in the 8-bit class: 16,384 + 8,704 = 25,088 B/token (8,192
+# elements per token per cache × 2 B for f16, × 1.0625 for q8_0 — there is no
+# true fp8 KV dtype in this build). That 7,680 B/token saved against exact V is
+# what funds BOTH the third conversation and the drafter. Quantized V is
+# illegal without --flash-attn, which the serve script pins. Rate law and the
+# arithmetic: [[frank-zealot]] → Laws.
+CACHE_TYPE_K=f16
 CACHE_TYPE_V=q8_0
-# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter
-# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it
-# still loses, because the draft context bills its OWN f16 KV over the WHOLE
-# pool: 4,096 B/position × 786,432 = 3,072.00 MiB, plus its compute buffers,
-# 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18):
-#   3 × 262,144  spec OFF  ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s
-#   3 × 262,144  spec ON   ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s
-#   3 × 262,144  spec ON   ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s
-#   2 × 262,144  spec ON   ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s
-#   1 × 262,144  spec ON   ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s
-# i.e. MTP is worth +28 % decode whenever the card has room, and at the 3-slot
-# pool it costs 1.3 GiB of compute buffers that ubatch 2048 refuses to give up —
-# the squeeze, not the drafter, is what halves decode. The arm that WOULD run
-# MTP at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (decode 213.5,
-# 3-stream 250.8 aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a
-# card whose projector worst-case lands after the fit check. This endpoint's
-# load is the fleet's COMPACTION HUB (prefill-bound, ~170k in) plus 2-3
-# concurrent subagents, so the deployed arm takes the +17 % prefill
-# (2,877 vs 2,395 t/s at 106,418 tokens) and the +12 % three-stream aggregate
-# (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free.
-# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*):
-# flags move as a set or the arm measures a crash instead of an absence.
-SPEC_TYPE=none
+# Speculation ON (Frank, 2026-10-19): the head is in the file
+# (qwen35.nextn_predict_layers = 1, blk.32.nextn.*) and rides the MAIN model's
+# weights, so there is no --spec-draft-model. Its bill is pool × 4,096 B of
+# draft KV plus ~0.45 GiB of buffers and a tripled recurrent state — paid for
+# by CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024 above. Build gate: quantized V
+# beside MTP needs fork build >= b10776; LLAMA_BIN is 10777.
+# SPEC_TYPE=none is the rollback and strips the WHOLE set (--spec-type and
+# every --spec-draft-*); half a set has segfaulted at load.
+SPEC_TYPE=draft-mtp
 SPEC_DRAFT_N_MAX=2
 SPEC_DRAFT_N_MIN=0
 SPEC_DRAFT_P_MIN=0
-# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp
-# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the
-# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty
-# is the ONE recipe value with no GGUF metadata key, so it must reach the argv.
-# PRESENCE_PENALTY="" is the documented opt-out.
+# The sampler recipe lives in the checkpoint (general.sampling.*) and
+# llama.cpp reads it itself. Presence penalty is the one recipe value with no
+# GGUF metadata key, so it must reach argv. "" is the documented opt-out.
 PRESENCE_PENALTY=1.0
 NGL=99
-# Boot ledger (2026-10-18 PRODUCTION boot under the unit, verbosity 4):
-#   17,922 MiB used / 6,203 MiB free of 24,123 (nvidia-smi settled)
-#   CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB
-#   CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions)
-#   CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs)
-#   compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf)
-#   mmproj worst-case estimate 868.32 MiB (248.10 CUDA + 24.93 CPU compute
-#   meta) — charged AFTER the engine's own fit check, as always
-#   no clamp line, no cudaMalloc error
-# Measured on this boot (/completion timings, greedy, ignore_eos, 512 tokens):
-# 168.7 t/s decode (three runs 168.4-168.7) and ~300 t/s aggregate across
-# THREE concurrent streams (102.8 each). The trial arm's record (167.2 decode,
-# 280.2 aggregate, 2,877 t/s prefill at 106,418) stands; the 27B it replaces
-# gave 76.6 / 109.4 / 691.7 at 166k. The FIRST request after a boot is a
-# warmup artifact (37.7 t/s class) and a natural generation that stops early
-# makes a wall-clock aggregate read 4x low - measure with ignore_eos and read
-# the engine's own timings.
-# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV on the
-# 27B era measured 16,392 used / 7,733 free at 2 × 200,704; the same drop here
-# frees ~6.5 GiB at three windows. UBATCH_SIZE=1024 is the cheaper lever
-# (−1,306 MiB measured on this shape).
-# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
-# and graph capture land in the journal, which is what the VRAM-fit forensics
-# read. Levels above 4 flood the journal.
+# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and
+# that is what VRAM forensics read. Above 4 floods the journal.
 VERBOSITY=4
-
-# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's
-# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so
-# this service has held the critical LlmEndpointDown alert for as long as the
-# metric has history (verified 2026-10-18: the oldest retained sample of
-# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl
-# http://host.containers.internal:7771/metrics, lastError "server returned HTTP
-# status 501 Not Implemented"). 0 opts out and re-breaks the scrape.
+# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads
+# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape.
 METRICS=1

scripts/frank-zealot.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 3x262144 slots over one 786432 pool, q8_0 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
+Description=llama.cpp (PrismML build 10777) Ternary-Bonsai-2-9B PQ2_0-MTP GGUF (qwen35 hybrid, 2x262144 slots over one 524288 pool, unquantized f16 K+V, MTP off, mmproj Q8_0 vision) on the RTX 3090 (dankclaw, frank-zealot)
 After=network-online.target
 Wants=network-online.target
 

scripts/gen-vault-index.py

@@ -0,0 +1,211 @@
+#!/usr/bin/env python3
+"""Regenerate serverAppConfigs-docs/index.md from page frontmatter.
+
+The root index is a locator (AGENTS.md -> *Documentation Policy*): one row per
+page, one clause, no numbers. Hand-maintaining it is how rows go missing, and
+the docs seam fails when a page is not linked from it, so it is generated:
+
+  * preamble     everything above the first `## ` in the existing file is kept
+                 verbatim (that is the curated statement of the vault's law)
+  * section      from the page's directory, overridden by its frontmatter tags
+                 (`endpoint` / `retired-endpoint` split the LLM pages)
+  * row text     the page's frontmatter `description`, verbatim
+
+A page with no description is a hard error rather than an empty cell, so the
+generator cannot silently produce a silent index. Sections with no pages are
+skipped. Deterministic: the same vault always yields the same bytes.
+
+Sub-directory indices are hand-grouped (they carry curated prose), so they are
+synced rather than regenerated: dead rows dropped, row text refreshed from the
+page's description, uncovered siblings appended.
+"""
+import pathlib
+import re
+import sys
+
+RESERVED = {"index.md", "log.md"}
+SECTIONS = [
+    ("Inference endpoints", "endpoint"),
+    ("Retired endpoints", "retired-endpoint"),
+]
+DIRECTORY_SECTIONS = [
+    ("Dankclaw", "Dankclaw services"),
+    ("Dankserver", "Dankserver services"),
+    ("FrankBook-Pro", "FrankBook-Pro"),
+    ("Desktop", "Workstations"),
+    ("Infrastructure", "Cross-host infrastructure"),
+    ("Architecture", "Repo architecture and policy"),
+    ("Operations", "Operations and incidents"),
+    ("Agent", "Agent / harness"),
+    ("Research", "Research and methodology"),
+]
+PIPELINE_SECTION = "Pipeline Parallelism (series)"
+LLM_FALLBACK = "LLM technique and hardware references"
+CHANGE_SECTION = "Change history"
+ORDER = [t for t, _ in SECTIONS] + [t for _, t in DIRECTORY_SECTIONS] + \
+        [PIPELINE_SECTION, LLM_FALLBACK, CHANGE_SECTION]
+
+
+def frontmatter_fields(text):
+    if not text.startswith("---"):
+        return {}
+    end = text.find("\n---", 3)
+    if end == -1:
+        return {}
+    raw = text[3:end]
+    fields = {}
+    for key in ("description", "type"):
+        m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
+        if m:
+            fields[key] = m.group(1).strip().strip("\"'")
+    tags = []
+    m = re.search(r"^tags:\s*\[(.*?)\]$", raw, re.M)
+    if m:
+        tags = [t.strip().strip("\"'") for t in m.group(1).split(",") if t.strip()]
+    elif re.search(r"^tags:\s*$", raw, re.M):
+        block = re.search(r"^tags:\s*\n((?:\s+-\s+.*\n?)+)", raw, re.M)
+        if block:
+            tags = [l.strip().lstrip("-").strip().strip("\"'")
+                    for l in block.group(1).splitlines() if l.strip()]
+    fields["tags"] = tags
+    return fields
+
+
+def section_for(page, vault, fields):
+    rel = page.relative_to(vault)
+    if len(rel.parts) == 1:
+        return CHANGE_SECTION if page.name == "log.md" else "Uncategorised"
+    top = rel.parts[0]
+    if top == "LLM":
+        if PIPELINE_SECTION_DIR in rel.parts:
+            return PIPELINE_SECTION
+        for title, tag in SECTIONS:
+            if tag in fields.get("tags", []):
+                return title
+        return LLM_FALLBACK
+    for prefix, title in DIRECTORY_SECTIONS:
+        if top == prefix:
+            return title
+    return "Uncategorised"
+
+
+ROW = re.compile(r"^(\| )\[\[([^\]|]+)\]\]( \|)( .*?)( \|$)")
+BULLET = re.compile(r"^(- )\[\[([^\]]+)\]\]( — )(.*?)$")
+
+
+def sync_subindex(index: pathlib.Path, vault: pathlib.Path):
+    """Keep a hand-grouped sub-index honest without erasing its grouping.
+
+    Rows are locators, so their text is the linked page's own description; a
+    row whose page is gone is a dead promise and is dropped; a direct sibling
+    with no row is appended under `## Pages` (that is what the link-integrity
+    seam checks). Curated headings and prose are untouched.
+    """
+    here = index.parent
+    known = {p.stem for p in here.rglob("*.md")
+             if ".AGENTTRASH" not in str(p) and p.name != "index.md"}
+    lines = index.read_text().split("\n")
+    out, dropped, refreshed = [], 0, 0
+    for line in lines:
+        # A row whose first cell is empty is a description with no page: the link
+        # was deleted when the page was destroyed, so the row is a dead promise.
+        if re.match(r"^\|\s*\|", line.strip()):
+            dropped += 1
+            continue
+        for pat, wrap in ((ROW, "| [[{}]] |{} {} |"), (BULLET, "- [[{}]] — {}")):
+            m = pat.match(line)
+            if not m:
+                continue
+            stem = m.group(2).strip()
+            if stem not in known:
+                dropped += 1
+                line = None
+                break
+            cand = next((p for p in here.rglob(f"{stem}.md")
+                         if ".AGENTTRASH" not in str(p)), None)
+            desc = frontmatter_fields(cand.read_text(errors="replace")).get("description") \
+                if cand else None
+            if desc:
+                current = m.group(4).strip()
+                if current != desc:
+                    line = wrap.format(stem, " " if pat is ROW else "", desc) \
+                        if pat is ROW else wrap.format(stem, desc)
+                    refreshed += 1
+            break
+        if line is not None:
+            out.append(line)
+    text = "\n".join(out)
+    # Presence must be the wikilink itself: a bare stem is a substring of other
+    # page names (`frank-medium` lives inside `frank-medium-rocmfpx`), and a
+    # sibling matched that way silently never gets a row.
+    missing = sorted(
+        (p.stem, frontmatter_fields(p.read_text(errors="replace")).get("description", ""))
+        for p in here.glob("*.md")
+        if p.name != "index.md" and f"[[{p.stem}]]" not in text
+    )
+    if missing:
+        block = ["", "## Pages", "", "| Page | Owns |", "|---|---|"]
+        block += [f"| [[{stem}]] | {desc} |" for stem, desc in missing]
+        text = text.rstrip("\n") + "\n" + "\n".join(block) + "\n"
+    if text != index.read_text():
+        index.write_text(text)
+    return dropped, refreshed, len(missing)
+
+
+PIPELINE_SECTION_DIR = "Pipeline Parallelism"
+
+
+def main(argv):
+    vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
+    index = vault / "index.md"
+    if not index.is_file():
+        print(f"missing {index}")
+        return 2
+    existing = index.read_text()
+    preamble = existing.split("\n## ", 1)[0].rstrip("\n")
+
+    pages = [p for p in vault.rglob("*.md")
+             if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
+             and p.name != "index.md" and p != index]
+    buckets, errors = {}, []
+    for page in pages:
+        fields = frontmatter_fields(page.read_text(errors="replace"))
+        if page.name != "log.md":
+            desc = fields.get("description")
+            if not desc:
+                errors.append(str(page.relative_to(vault)))
+                continue
+        else:
+            desc = "Chronological incident and law log for the whole bundle."
+        buckets.setdefault(section_for(page, vault, fields), []).append((page.stem, desc))
+    if errors:
+        print("pages with no description (fix the frontmatter, not the index):")
+        for e in sorted(errors):
+            print("  " + e)
+        return 1
+
+    out = [preamble, ""]
+    for title in ORDER + sorted(set(buckets) - set(ORDER)):
+        rows = sorted(buckets.get(title, []), key=lambda r: r[0].lower())
+        if not rows:
+            continue
+        out += [f"## {title}", "", "| Page | Owns |", "|---|---|"]
+        out += [f"| [[{stem}]] | {desc} |" for stem, desc in rows]
+        out.append("")
+    out.append("<!-- generated by scripts/gen-vault-index.py; edit page frontmatter instead -->")
+    index.write_text("\n".join(out) + "\n")
+    touched = 0
+    for sub in sorted(vault.rglob("index.md")):
+        if sub == index or ".AGENTTRASH" in str(sub):
+            continue
+        dropped, refreshed, added = sync_subindex(sub, vault)
+        if dropped or refreshed or added:
+            touched += 1
+            print(f"  {sub}: -{dropped} dead rows, {refreshed} refreshed, +{added} added")
+    print(f"{index}: {sum(len(v) for v in buckets.values())} rows in "
+          f"{len([t for t in ORDER if buckets.get(t)])} sections, {touched} sub-indices synced")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/port-census.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""Port census: prove a merge/collapse lost no fact.
+
+For every line that existed in the OLD set and not in the NEW set, check that
+each of its significant tokens (numbers, long identifiers, paths) still occurs
+somewhere in the NEW corpus. A line whose every token survives is a reworded
+line; a line with an unheld token is a candidate fact loss and gets printed.
+
+  port-census.py <new-glob-dir> <old-file> [<old-file> ...]
+
+Old files may be `git-rev:path` (read from git) or a path (e.g. a trash copy).
+"""
+import pathlib
+import re
+import subprocess
+import sys
+
+
+def toks(line):
+    out = []
+    for n in re.findall(r"\d[\d,]*(?:\.\d+)?", line):
+        if len(n.replace(",", "").split(".")[0]) >= 3 or (
+            "." in n and len(n.split(".")[-1]) >= 2 and len(n.replace(",", "")) >= 4
+        ):
+            out.append(n)
+    # Identifiers, flags, paths and backticked spans are the facts worth
+    # tracking; ordinary prose words are not — they reword on every rewrite.
+    # A hyphen alone is not evidence of an identifier (`long-standing`), so a
+    # dashed word counts only with two hyphens (`frank-tts-engine`) or with a
+    # digit, dot, slash, colon or `--` inside it.
+    for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line):
+        core = w.strip(".,;:!?\"')]")
+        if (core.startswith("--") or core.isupper()
+                or re.search(r"[\d_/:.]", core) or core.count("-") >= 2):
+            out.append(core)
+    out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3]
+    return out
+
+
+def held(tok, corpus):
+    t = tok.replace(",", "")
+    alt = rf"\b{re.escape(t)}\b"
+    for c in corpus:
+        if tok in c or t in c:
+            return True
+        if re.search(alt, c.replace(",", "")):
+            return True
+    return False
+
+
+def main(argv):
+    new_dir = pathlib.Path(argv[1])
+
+    def corpus_excluding(exclude):
+        """Every vault page except the one under test.
+
+        The corpus is what must still hold a fact AFTER the file is gone, so a
+        file that lives inside the scanned directory has to be removed from it —
+        otherwise every token is "held" by the file itself and the census proves
+        nothing (it reports 0 losses for every page, which is the bug this
+        guards).
+        """
+        out = []
+        for p in new_dir.rglob("*.md"):
+            s = str(p)
+            if ".AGENTTRASH" in s or ".obsidian" in s:
+                continue
+            if exclude is not None and p.resolve() == exclude:
+                continue
+            out.append(p.read_text(errors="replace"))
+        return out
+
+    lost = checked = 0
+    for spec in argv[2:]:
+        if spec.startswith("git-"):
+            ref, path = spec[4:].split(":", 1)
+            old = subprocess.run(["git", "show", f"{ref}:{path}"],
+                                 capture_output=True, text=True).stdout
+            label = f"{ref}:{path}"
+            exclude = None
+        else:
+            old = pathlib.Path(spec).read_text(errors="replace")
+            label = spec
+            exclude = pathlib.Path(spec).resolve()
+        blob = "\n".join(corpus_excluding(exclude))
+        for line in old.splitlines():
+            s = line.strip().strip("->*|# ").strip()
+            if len(s) < 30 or s.startswith("```"):
+                continue
+            checked += 1
+            missing = [t for t in toks(s) if not held(t, [blob])]
+            if missing:
+                lost += 1
+                print(f"  {label}\n    L: {s[:150]}\n    unheld: {', '.join(missing[:6])}")
+    print(f"{checked} old lines checked, {lost} with unheld tokens")
+    return 1 if lost else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/repair-damage.py

@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+"""Repair the two artifacts the destruction passes introduced.
+
+1. The link stripper could eat the space after a `- ` marker (`-`X``) and leave a
+   dangling `see .` where a wikilink used to be.
+2. The Eras collapse could swallow a table's HEADER row (it looked like just
+   another non-current row), leaving a separator with no header above it.
+
+Usage: repair-damage.py [--apply]
+"""
+import pathlib
+import re
+import sys
+
+VAULT = pathlib.Path("serverAppConfigs-docs")
+ERAS_HEADER = "| Window | Payload | Runtime | Shape | Replaced because | Rollback |"
+
+
+def main(argv):
+    apply = "--apply" in argv
+    marker = dangling = header = 0
+    for p in sorted(VAULT.rglob("*.md")):
+        s = str(p)
+        if ".AGENTTRASH" in s or ".obsidian" in s:
+            continue
+        lines = p.read_text(errors="replace").split("\n")
+        out = []
+        # Only a pointer whose target was deleted reads as `… see .` — never touch
+        # a line merely because it wraps on a comma. (An earlier version of this
+        # rule stripped the trailing comma off 660 wrapped prose lines, which
+        # `scripts/restore-punctuation.py` then had to undo from HEAD.)
+        for line in lines:
+            new = line
+            if re.match(r"^(\s*)[-*](?![-* ])[$`[(*_]", new):
+                new = re.sub(r"^(\s*)[-*]", r"\1- ", new)
+                marker += 1
+            if re.search(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", new):
+                new = re.sub(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", "", new)
+                dangling += 1
+            out.append(new)
+        # A separator row whose line above is not a header gets its header back.
+        fixed = []
+        for i, line in enumerate(out):
+            if re.match(r"^\|-+", line.strip()) and (not fixed or not fixed[-1].strip().startswith("|")):
+                fixed.append(ERAS_HEADER)
+                header += 1
+            fixed.append(line)
+        if apply and "".join(fixed) != "".join(lines):
+            p.write_text("\n".join(fixed))
+    print(f"{'APPLIED' if apply else 'dry run'}: marker spacing {marker}, "
+          f"dangling pointers {dangling}, restored table headers {header}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/restore-punctuation.py

@@ -0,0 +1,66 @@
+#!/usr/bin/env python3
+"""Undo trailing-punctuation damage by restoring lines that match HEAD exactly
+except for a stripped trailing `,` / `;` / removed trailing word.
+
+The repair pass over-reached: its "dangling pointer" cleanup also matched every
+wrapped prose line ending in a comma, and deleted the comma. A line whose text
+is otherwise byte-identical to its HEAD version, differing only by trailing
+punctuation, is not an edit — it is damage, so HEAD's line goes back.
+
+Usage: restore-punctuation.py [--apply]
+"""
+import pathlib
+import subprocess
+import sys
+
+REPO = pathlib.Path(".")
+VAULT = REPO / "serverAppConfigs-docs"
+
+
+def head_text(rel):
+    r = subprocess.run(["git", "show", f"HEAD:{rel}"], capture_output=True, text=True)
+    return r.stdout if r.returncode == 0 else None
+
+
+def main(argv):
+    apply = "--apply" in argv
+    restored = 0
+    pages = 0
+    for p in sorted(VAULT.rglob("*.md")):
+        s = str(p)
+        if ".AGENTTRASH" in s or ".obsidian" in s:
+            continue
+        rel = s
+        head = head_text(rel)
+        if head is None:
+            continue
+        cur = p.read_text(errors="replace").split("\n")
+        # Key on (length, prefix) of the HEAD line minus trailing punctuation so a
+        # damaged line can find its original.
+        index = {}
+        for hl in head.split("\n"):
+            h = hl.rstrip()
+            if h.endswith((",", ";")) or True:
+                key = (len(h.rstrip(",;. ")), h.rstrip(",;. ")[:60])
+                index.setdefault(key, h)
+        touched = 0
+        for i, cl in enumerate(cur):
+            c = cl.rstrip()
+            if not c or c.endswith((".", ":", "|", "`", ")", ","  , ";")):
+                continue
+            key = (len(c), c[:60])
+            hl = index.get(key)
+            if hl and hl != c and hl.startswith(c):
+                cur[i] = hl
+                touched += 1
+        if touched:
+            pages += 1
+            restored += touched
+            if apply:
+                p.write_text("\n".join(cur))
+    print(f"{'APPLIED' if apply else 'dry run'}: {restored} lines restored across {pages} pages")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

scripts/serve-frank-archon.sh

@@ -15,12 +15,13 @@
 # log states every budget before anything is allocated (weights vs KV
 # split, the expert host pool's coverage of the worst case, the PLE pin,
 # the P2P answer per pair) — read it after every resize.
-# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
-# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
-# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
-# own budget table makes claimable, spent on the slab because prefill is
-# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
-# 8 GiB + disk 64 GiB, PLE pinned in RAM.
+# Deployed shape (2026-10-19, radiance 1.8.1): THREE decode slots x 200,000
+# tokens, fp8 KV, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache in
+# pinned host RAM, PLE pinned in RAM. The per-card split between the expert slab
+# and the KV pool is NO LONGER an operator number — 1.3.0 removed the VRAM pair
+# and the budget resolver derives it from headroom, the cache bound and the host
+# pool, so the slab size and the pool's token count are LEDGER lines: after every
+# boot read them out of the log and check the pool still backs 3 x 200,000.
 #
 # The root unit (scripts/frank-archon.service, User=dank,
 # LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
@@ -56,13 +57,11 @@ set +a
 : "${FRANK_ARCHON_PLACEMENT:?FRANK_ARCHON_PLACEMENT must be set}"
 : "${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
 : "${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
-: "${FRANK_ARCHON_VRAM_WEIGHTS_MIB:?FRANK_ARCHON_VRAM_WEIGHTS_MIB must be set}"
-: "${FRANK_ARCHON_VRAM_KV_MIB:?FRANK_ARCHON_VRAM_KV_MIB must be set}"
+: "${FRANK_ARCHON_BUSY_PREFILL_CHUNK:?FRANK_ARCHON_BUSY_PREFILL_CHUNK must be set (auto|off|N)}"
 : "${FRANK_ARCHON_KV_DTYPE:?FRANK_ARCHON_KV_DTYPE must be set}"
 : "${FRANK_ARCHON_SPEC_TOKENS:?FRANK_ARCHON_SPEC_TOKENS must be set}"
 : "${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
-: "${FRANK_ARCHON_PREFIX_HOST_MIB:?FRANK_ARCHON_PREFIX_HOST_MIB must be set}"
-: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
+: "${FRANK_ARCHON_PREFIX_CACHE_MIB:?FRANK_ARCHON_PREFIX_CACHE_MIB must be set}"
 : "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
 : "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
 : "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
@@ -103,19 +102,12 @@ if [ ! -f "$FRANK_ARCHON_TEMPLATE" ]; then
     exit 1
 fi
 
-mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
-# The engine's create is NOT recursive: with only the state dir mounted, its
-# "cannot create /data/kvcache/frank-archon: No such file or directory"
-# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
-# the full path on the host so the tier opens on first boot.
-
 mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
 # rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
 # the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
 # $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
 # the cache must ride a host mount or it dies with the --rm container
-# that wrote it. The dir is pre-created for the same reason as kvcache
-# (the engine's create is not recursive).
+# that wrote it. The engine's create is not recursive, so the dir is pre-created.
 
 if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
     echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
@@ -148,6 +140,15 @@ done
 LOG=/home/dank/logs/frank_archon.log
 mkdir -p "$(dirname "$LOG")"
 
+# Interleave is opt-in: 0 is the engine's default and passing it anyway would put
+# a flag in argv that changes nothing but the record. The sharing cap is always
+# explicit because `auto` vs `off` is the difference between decode waiting one
+# chunk and waiting the whole step.
+PREFILL_ARGS=()
+if [ "${FRANK_ARCHON_PREFILL_INTERLEAVE:-0}" -gt 0 ]; then
+    PREFILL_ARGS=(--prefill-interleave "$FRANK_ARCHON_PREFILL_INTERLEAVE")
+fi
+
 # Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
 # --network host (the server binds the host's addresses directly, no NAT),
 # --init (the engine releases the cards on SIGTERM), seccomp=unconfined
@@ -156,8 +157,9 @@ mkdir -p "$(dirname "$LOG")"
 # makes for host memory), memlock unbounded (the pinned ledger — legal
 # because the root unit grants LimitMEMLOCK=infinity to this dank process),
 # /dev/kfd + /dev/dri (render/video groups are moot for a root container).
-# The template rides a single-file mount; the .rad dir is ro; /data holds
-# the prefix-cache disk tier.
+# The template rides a single-file mount and the .rad dir is ro. The prefix cache
+# has no disk tier any more, so the only host state this container writes is the
+# rad-tune cache under /opt/radiance/share/radiance/tune.
 podman run -d --name "$FRANK_ARCHON_CONTAINER" \
   --network host \
   --init \
@@ -165,7 +167,6 @@ podman run -d --name "$FRANK_ARCHON_CONTAINER" \
   --ulimit memlock=-1:-1 \
   --device /dev/kfd --device /dev/dri \
   -v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
-  -v "$FRANK_ARCHON_STATE_DIR":/data \
   -v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
   -v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
   "$FRANK_ARCHON_IMAGE" \
@@ -179,15 +180,13 @@ podman run -d --name "$FRANK_ARCHON_CONTAINER" \
   --placement "$FRANK_ARCHON_PLACEMENT" \
   --host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
   --gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
-  --vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
-  --vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
   --kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
-  --prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
-  --prefix-cache-dir /data/kvcache/frank-archon \
-  --prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
+  --prefix-cache-mib "$FRANK_ARCHON_PREFIX_CACHE_MIB" \
   --ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
   --num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
   --max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
+  --busy-prefill-chunk "$FRANK_ARCHON_BUSY_PREFILL_CHUNK" \
+  "${PREFILL_ARGS[@]}" \
   ${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
   > "$LOG" 2>&1
 

scripts/serve-frank-zealot.sh

@@ -3,33 +3,47 @@
 # in the file) as frank-zealot on Dankclaw's RTX 3090 via llama.cpp (PrismML
 # vendor build 10777, CUDA sm_86).
 #
-# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export at THREE
-# 262,144-token conversations (--ctx-size 786432 --parallel 3), q8_0 K + q8_0 V.
-# The 9B bills 17,408 B/token against the 27B's 34,816 (8 full-attention layers
-# instead of 16), and that halved rate is what funds three NATIVE windows where
-# the 27B could only afford two cut to 200,704. Verbosity 4 ships so the
-# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the
-# journal.
+# 2026-10-18 (Frank): Bonsai-2 27B PQ2_0 → the 9B MTP export, then the same
+# evening TWO 262,144-token conversations (--ctx-size 524288 --parallel 2) over
+# an UNQUANTIZED f16 KV cache.
+#
+# 2026-10-19 (Frank): THREE 200,704-token conversations over a 602,112 pool
+# (--ctx-size 602112 --parallel 3), K left at the engine's own default (f16),
+# V dropped to q8_0, batch 1024×6 / ubatch 1024, and the MTP drafter ARMED
+# (--spec-type draft-mtp). The 9B bills 8,192 elements per token per cache (8
+# full-attention layers instead of the 27B's 16), so f16 K is 16,384 B/token and
+# q8_0 V is 8,704: 25,088 B/token = 14,406.00 MiB at the 602,112 pool. That
+# 7,680 B/token against exact V is exactly what funds the third conversation
+# AND a draft context that bills f16 KV over the whole pool. Verbosity 4 ships
+# so the VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in
+# the journal.
 #
 # Architecture (engine ledger, not the marketing page): qwen35 hybrid, 32
 # layers with qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated
 # DeltaNet, 4 KV heads × 256 dim, native 262,144. The KV pool scales with the 8
-# full layers (13,056.00 MiB at the 786,432 pool, ledger-exact); the DeltaNet
-# layers cost a fixed recurrent state per SLOT — 452.25 MiB for 3 cells, ~151
-# MiB each. The runtime is a hard lock, not a preference: custom GGML type 142
-# plus prism.hadamard.* metadata, which only the vendor build registers, and
-# MTP beside quantized V needs build ≥ b10776 — so the prebuilt moved 10754 →
-# 10777 with the payload. Vision comes from the publisher's Q8_0 projector (the
-# BF16 one can crash the server on concurrent images, and this endpoint runs
-# three slots), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
-# the documented text-only opt-out.
+# full layers (14,406.00 MiB of f16 K + q8_0 V at the 602,112 pool); the
+# DeltaNet layers cost a fixed recurrent state per SLOT — ~50.25 MiB a cell, so
+# 150.75 MiB for three, and an armed depth-2 drafter triples that again. The
+# runtime is a hard lock, not a preference: custom GGML type 142 plus
+# prism.hadamard.* metadata, which only the vendor build registers — and the
+# b10776 gate that moved the prebuilt 10754 → 10777 (MTP beside a quantized V)
+# is LOAD-BEARING again on this shape, not moot as it was under exact KV.
+# Vision comes from the publisher's Q8_0 projector (the BF16 one can crash the
+# server on concurrent images, and this endpoint runs three slots), so --mmproj is
+# what gives this endpoint eyes; MMPROJ_FILE="" is the documented text-only
+# opt-out.
 #
-# Speculative decoding ships OFF, measured: the head loads and accepts well
-# (0.798/0.617 per position, mean length 2.41) but the draft context bills its
-# own f16 KV over the WHOLE pool — 4,096 B/position × 786,432 = 3,072 MiB plus
-# buffers — and at ubatch 2048 that squeeze HALVES decode (167.2 → 109.2 t/s).
-# frank-zealot.env carries the six-arm table and the arm that turns it back on
-# (--ubatch-size 1024, decode 213.5 t/s) with its price.
+# Speculative decoding ships ON (Frank, 2026-10-19). The head is IN the file and
+# accepts well (0.798/0.617 per position, mean length 2.41, measured 2026-10-18);
+# what made it lose that evening was the bill, not the drafts — the draft context
+# is built against the target's WHOLE pool, so blk.32 adds 4,096 B/position
+# (2,352.00 MiB at 602,112) plus 480.97 MiB of its own compute buffers on the
+# card, and it triples the recurrent state per slot (452.25 MiB for three cells
+# against 50.25 a cell with the drafter off). CACHE_TYPE_V=q8_0 is what pays
+# that: 7,680 B/token cheaper than exact V, 4,410 MiB at this pool. The ubatch
+# drop to 1024 contributes ~406 MiB here, NOT the ~1,306 the 786,432-pool census
+# showed — that figure does not transfer between shapes. SPEC_TYPE=none is
+# the rollback and strips the whole flag set.
 #
 # Sampling: the checkpoint carries the publisher's thinking recipe as GGUF
 # metadata (general.sampling.* = temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
@@ -56,43 +70,46 @@ fi
 : "${ALIAS:=frank-zealot}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7771}"
-# ctx-size is the POOL llama.cpp divides across slots: 786,432 / 3 = 262,144
-# per conversation — every slot at the model's trained window. The harness
-# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
-# with a 400, never truncated.
-: "${CTX_SIZE:=786432}"
+# ctx-size is the POOL llama.cpp divides across slots: 602,112 / 3 = 200,704 a
+# conversation. The harness contextWindow MUST quote the PER-SLOT number — a
+# longer prompt is answered with a 400, never truncated. The pool is priced by
+# the KV bill, not by the trained window: at 25,088 B/token (f16 K + q8_0 V) the
+# 602,112 pool costs 14,406.00 MiB and the armed drafter adds its own f16 KV over
+# the whole pool (2,352.00 MiB). Exact V at this pool (18,816 MiB) or the old
+# 786,432 pool at any dtype cannot boot inside the 24,123 MiB the driver leaves
+# addressable.
+: "${CTX_SIZE:=602112}"
 : "${PARALLEL:=3}"
 # Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
-# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
-# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
-# throughput-neutral at a fixed ubatch. It ALSO sets the compute-buffer size,
-# which is the term that decides whether an MTP draft context fits beside a
-# 786,432-position pool: 1024 hands back 1,306 MiB and is the arm that runs the
-# drafter at three slots (frank-zealot.env carries the table).
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=2048}"
-# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 3 × 262,144 on this
-# card. 17 KiB/token at this geometry (8 full-attention layers × 4 KV heads ×
-# 256 dim); 13,056.00 MiB at the 786,432 pool, ledger-exact. f16 would bill
-# 26,112 MiB against a 24,123 MiB card and dies at cudaMalloc; q4_0 is the
-# fallback rung. Explicit either way so the contract is visible in the argv,
-# and --flash-attn stays required for quantized V (no true fp8 KV dtype exists
-# in this build).
-: "${CACHE_TYPE_K:=q8_0}"
+# The logical batch is VRAM- and throughput-neutral at a fixed ubatch, so 6144
+# (1024×6) is free; the ubatch is what sets the compute-buffer size, and 1024 is
+# the rung every measured MTP arm here ran on (213.5 t/s at 1024 against 109.2
+# at 2048) — worth ~406 MiB at THIS shape, not the ~1,306 the old pool showed.
+: "${BATCH_SIZE:=6144}"
+: "${UBATCH_SIZE:=1024}"
+# Asymmetric KV: f16 K (the engine's own default, spelled out so the contract is
+# visible in argv) + q8_0 V. 8,192 elements per token per cache (8 full-attention
+# layers × 4 KV heads × 256 dim) ⇒ 16,384 B/token for K and 8,704 for V (q8_0 is
+# 1.0625 B/element — llama.cpp has no true fp8 KV dtype), 25,088 B/token and
+# 14,406.00 MiB at the 602,112 pool. That is the halfway door between exact KV
+# (32,768 B/token, which costs the third conversation and the drafter both) and
+# the fully quantized pair (17,408 B/token, which costs retrieval fidelity in the
+# keys); --flash-attn stays pinned because quantized V is illegal without it.
+: "${CACHE_TYPE_K:=f16}"
 : "${CACHE_TYPE_V:=q8_0}"
-# Speculative decoding defaults OFF, measured on this card and shape: the head
-# IS in the checkpoint (qwen35.nextn_predict_layers = 1 — MTP is a property of
-# the FILE, so the plain PQ2_0 sibling could not run it at any flag value), it
-# loads, and it accepts well (mean accepted length 2.41) — but the draft context
-# bills its own f16 KV over the WHOLE pool (4,096 B/position, 3,072 MiB here)
-# and at ubatch 2048 that squeeze halves decode: 167.2 → 109.2 t/s. There is no
-# --spec-draft-cache-type knob in this build to make the draft cheap, and no
-# --spec-draft-model either: the draft rides the main model's weights. Card gate:
-# quantized V alongside MTP needs fork build >= prism-b10776, which is why
-# LLAMA_BIN moved to the 10777 prebuilt; on an older build the pair is illegal.
-# SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 is the documented MTP arm (213.5 t/s);
-# SPEC_TYPE=none strips the whole set (flags move together).
-: "${SPEC_TYPE:=none}"
+# Speculative decoding defaults ON: the head IS in the checkpoint
+# (qwen35.nextn_predict_layers = 1 — MTP is a property of the FILE, so the plain
+# PQ2_0 sibling could not run it at any flag value) and the draft context rides
+# the MAIN model's weights, so there is no --spec-draft-model. Its bill is the
+# target's whole pool at 4,096 B/position plus ~0.45 GiB of buffers at this
+# ubatch and a tripled recurrent state; CACHE_TYPE_V=q8_0 and UBATCH_SIZE=1024
+# above are what pay for it. The draft cache dtype IS a further knob
+# (--spec-draft-type-k / --spec-draft-type-v, both default f16 — verified in this
+# build's --help) and stays at its default here. Card gate: quantized V alongside
+# MTP needs fork build >= prism-b10776, which is why LLAMA_BIN is the 10777
+# prebuilt; the type-142 loader lock keeps it pinned regardless.
+# SPEC_TYPE=none is the rollback and strips the whole set (flags move together).
+: "${SPEC_TYPE:=draft-mtp}"
 : "${SPEC_DRAFT_N_MAX:=2}"
 : "${SPEC_DRAFT_N_MIN:=0}"
 : "${SPEC_DRAFT_P_MIN:=0}"
@@ -104,7 +121,7 @@ fi
 : "${NGL:=99}"
 # Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
 # 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
-# the VRAM-fit forensics need — the q8_0 KV contract and the draft-KV bill are
+# the VRAM-fit forensics need — the exact-KV size and the draft-KV bill are
 # only provable from this log. Levels above 4 flood the journal.
 : "${VERBOSITY:=4}"
 # /metrics needs the flag: llama.cpp returns 501 unless the server starts with
@@ -169,11 +186,11 @@ fi
 # open after boot; make sure it exists before the server needs it.
 nvidia-modprobe -u -c=0 2>/dev/null || true
 
-# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
-# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
-# the q8_0 KV contract in frank-zealot.env is a real path — without this
-# flag or that build option the engine silently falls back and the
-# contract becomes a lie.
+# Flash attention stays pinned on. Exact f16 KV runs fine with or without it,
+# but the documented fallback rung (quantized V) is ILLEGAL without it, and the
+# build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17) so that rung is a
+# real path — dropping the flag here would leave the retreat route silently
+# falling back to something the env file does not describe.
 ARGS=(
     --model "$MODEL_FILE"
     --alias "$ALIAS"

serverAppConfigs-docs/Agent/Agent Docs and Edits.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Agent Docs and Edits
-description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each.
+description: "Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each."
 tags: [agents, docs, markdown, hashline, edit-recovery]
 timestamp: 2026-10-18T00:00:00Z
 ---
@@ -21,7 +21,7 @@ Damage catalogue:
    no longer frontmatter (title/tags/description become body text).
 2. A rewrite **appended** instead of merged → duplicate headings and a stray
    second `---` mid-file.
-3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`,
+3. Literal model artifacts in the body: `<|channel>}<tool_call|>...`
    `+- `, `- - `.
 4. Wrapped bullets re-emitted with a `- ` prefix on every physical line.
 5. Line-range replacement that dropped a sentence tail → dangling fragment,
@@ -46,7 +46,7 @@ Repair order (matters):
 2. If it merely duplicated: **salvage unique content first** — cut the
    one-of-a-kind section, paste it into the copy you keep, THEN delete the
    duplicate ranges. Never delete first.
-3. Re-add missing sentence tails (insert before the orphaned fragment),
+3. Re-add missing sentence tails (insert before the orphaned fragment)
    keeping list continuation lines indented.
 4. Re-run all checks above.
 
@@ -57,7 +57,7 @@ safe, but its snapshot tag changes on every edit — re-read before each
 subsequent edit. `.AGENTTRASH/` is gitignored (`.gitignore:25`), so NEW trash
 stays local — but do not read that as "the repo has none of it": on 2026-10-18
 `git ls-files | grep -c AGENTTRASH` returns **37** tracked paths
-(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`,
+(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`
 `tests/.AGENTTRASH/`), because gitignore never untracks what was committed
 before the rule. Trash is a temporary hold, not an archive.
 

serverAppConfigs-docs/Agent/Command Center Dotfiles.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Command Center Dotfiles
-description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas.
+description: "How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas."
 tags: [dotfiles, omp, deploy, rsync, blankpad]
 timestamp: 2026-10-05T00:00:00-07:00
 ---

serverAppConfigs-docs/Agent/OMP Harness.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: OMP Harness
-description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean.
+description: "The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits."
 tags: [omp, harness, compaction, models-yml, context, samplers]
 timestamp: 2026-10-18T00:00:00Z
 ---
@@ -14,19 +14,19 @@ registry; single source of truth — there is no copy on the servers) and
 this file is repo-managed; edits are harness-side only.
 
 **Model ids are the StarCraft unit names (official 2026-10-08, on the wire
-since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
-`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
-:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`,
+since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and
+`frank-zealot` (Dankclaw) — engine, port and card behind each id live on
+[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`
 `frank-zealot`. The SERVICES keep the old names (units, scripts, env
 files, runbook pages); pages written in the old era name the service,
 the harness names the model. The wire now matches: each endpoint's
 `/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
 picker/grants, the homepage docs and the raw API all use one id per model
-(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
-note on radiance echoing the requested name).
+(see [[OpenWebUI]] for the grant re-key and the [[frank-archon]]
+note on the endpoint echoing the requested name).
 
 Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
-192.168.0.143:7771 each answer with the unit name (`frank-archon`,
+192.168.0.143:7771 each answer with the unit name (`frank-archon`
 `frank-dragoon`, `frank-zealot`).
 
 ## Compaction chain arithmetic (verify before ANY window change)
@@ -40,20 +40,21 @@ breaks compaction.
 Where the contract lives:
 
 - `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL:
-  `http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`),
-  `methodOrder` (`shake → snapcompact → soft → handoff → remote`,
+  `http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`)
+  `methodOrder` (`shake → snapcompact → soft → handoff → remote`
   `config.yml:77-82`), `thresholdPercent` (**95**, `config.yml:84`), idle
   settings (`idleEnabled`, `idleThresholdTokens: 100000`)
-- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`,
+- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`
   `compactionModel` references
 - Implementation: omp plugin
   `~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
 
 The arithmetic (the coupled contract every endpoint's maxTokens comment
-encodes). Recomputed from the live files 2026-10-18 — contextWindows
-200,000 (archon) / 155,648 (dragoon) / 200,704 (zealot) at
-`models.yml:48,131,206`; maxTokens 16,384 / 45,056 / 16,384 at
-`models.yml:56,143,212`:
+encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
+endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
+follows is a **worked example** recomputed from the live files on 2026-10-18 —
+a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
+maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
 
 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
    95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
@@ -103,16 +104,16 @@ input. Boot-shape OOM laddering is a different problem
 ## Per-model sampler config (`compat.extraBody`)
 
 `models.yml` has **no per-model sampler keys**. The global knobs are
-`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`,
+`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`
 `repetitionPenalty`, and a knob reaches the wire only when it is set — the
-bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`),
+bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`)
 and this workstation's `config.yml` sets **none** of them. The 2026-10-05 note
 that the unset default is `-1`/"not sent" is retired: there is no `-1`
 sentinel, an unset key is simply absent. The per-model hook is
 `compat.extraBody`:
 
 **Why it works:** `applyOpenAIExtraBody` runs **last** in the
-openai-completions builder — after `applyChatCompletionsCompatPolicy` — via
+openai-completions builder — after `applyChatCompletionsCompatPolicy` —
 `Object.assign(params, extraBody)`. So extraBody overrides global settings
 and compat-policy output.
 
@@ -141,28 +142,25 @@ Use **wire names**, not OMP camelCase. Two traps:
 2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
    also sets it — `Object.assign` is shallow and replaces the whole object,
    silently dropping the `reasoning_effort` kwarg the compat policy set.
-   Top-level `preserve_thinking` is honoured by halogen. Live state 2026-10-18:
-   frank-archon's radiance entry carries `chat_template_kwargs:
-   {enable_thinking, preserve_thinking}` in `extraBody` (`models.yml:89-91`)
-   and nothing else sets that object at that endpoint, so the shallow-merge
-   hazard is latent rather than firing; the FreeToken-era "verify-and-prune at
-   the next swap" note is retired with that swap (2026-10-05,
-   `scripts/serve-frank-archon.sh:9-10`).
+   Top-level `preserve_thinking` is honoured by frank-dragoon
+   ([[frank-dragoon]]). Live state 2026-10-18: frank-archon's `extraBody`
+   carries `chat_template_kwargs: {enable_thinking, preserve_thinking}`
+   (`models.yml:89-91`) and nothing else sets that object at that endpoint, so
+   the shallow-merge hazard is latent rather than firing.
 
-**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier flash-server
+**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier frank-dragoon
 builds carried **no server-side sampler defaults**, so an absent
 `temperature` decoded greedy while the endpoint looked configured. That era is
-over: `HALOGEN_TEMPERATURE/TOP_P/TOP_K` now set the card's thinking recipe
-(1.0 / 0.95 / 20) as engine defaults (`scripts/frank-dragoon.env:98-103`,
-exported at `scripts/serve-frank-dragoon.sh:186-188`), so an absent temperature
-is no longer greedy. frank-dragoon's `extraBody` still carries the same trio —
-consistent, not load-bearing. Treat any older "do not drop it or you get
-greedy" wording as history.
+over — the endpoint now ships its own defaults ([[frank-dragoon]]), so an
+absent temperature is no longer greedy. frank-dragoon's `extraBody` still
+carries the same trio — consistent, not load-bearing. Treat any older "do not
+drop it or you get greedy" wording as history.
 
-**Reasoning-effort folding.** Qwen3.8's template accepts only
-`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
-else ("Unexpected reasoning effort high"). Every endpoint's
-`reasoningEffortMap` folds all harness levels onto that set; a role whose
+**Reasoning-effort folding.** The set `reasoning_effort` accepts is a property
+of the endpoint, not of the harness: [[frank-dragoon]] and [[frank-zealot]]
+raise on anything outside `xhigh|medium|low`, while [[frank-archon]] does not
+gate the set. Every endpoint's `reasoningEffortMap` folds all harness levels
+onto whichever set that endpoint ships; a role whose
 level is unmapped fails on its FIRST request, and the failure names Jinja,
 not the model.
 
@@ -193,7 +191,7 @@ assistant turn carrying a long `reasoning_content`, then compare
 `usage.prompt_tokens` with the flag true vs false. A large delta means the
 field changed template rendering; zero delta means it was ignored.
 
-Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`),
+Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`)
 never `rm`.
 
 ## Static-preamble audit ("the context feels bloated")
@@ -207,7 +205,7 @@ What loads statically every session (measure each, don't guess):
    ```bash
    python3 - <<'EOF'
    import glob, re, os
-   for root in [os.path.expanduser("~/.omp/agent/managed-skills"),
+   for root in [os.path.expanduser("~/.omp/agent/managed-skills")
                 os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]:
        n=tot=0
        for sk in glob.glob(root+"/*/SKILL.md"):
@@ -307,4 +305,4 @@ standalone; fold go/no-go arithmetic skills into their probe skill
 - [[Agent Docs and Edits]] — editing vault docs and hashline targets safely
 - [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides
 - [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit
-- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes
+-  measuring the endpoints this config routes

serverAppConfigs-docs/Agent/index.md

@@ -1,23 +1,12 @@
----
-type: index
-description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home.
-tags: [agent, index]
-timestamp: 2026-10-05T00:00:00-07:00
----
-
 # Agent
 
 Pages about the agent's own tooling and context — how omp is configured, how
 docs get edited safely, and how the agent's home is deployed.
 
-- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via
-  `compat.extraBody`, preamble and skill-library audits
-- [[Agent Docs and Edits]] — doc-structure audits after parallel edits;
-  hashline edit recovery
-- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync
-  transport, override packages
+- [[OMP Harness]] — The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits.
+- [[Agent Docs and Edits]] — Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each.
+- [[Command Center Dotfiles]] — How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas.
 
 ## Related
 
-- [[Testing Policy]] — the seams these pages' checks ride
-- [LLM](../LLM/index.md) — the fleet these agents operate
+-  the fleet these agents operate

serverAppConfigs-docs/Architecture/Container Network Namespaces.md

@@ -1,8 +1,7 @@
 ---
 type: reference
 title: Container Network Namespaces
-description: Which local services are addressable from which Podman network namespace on
-  Dankclaw, and why bridge-network containers cannot dial host-netns loopback listeners.
+description: "How Podman builds container networks here: the network inventory, the netns sharing mechanism, and the isolation limits."
 tags:
   - networking
   - podman
@@ -48,7 +47,7 @@ network namespace**:
 
 `Network=` values are read from the live Quadlet sources in
 `/home/dank/.config/containers/systemd/*.container`; runtime network names on the
-host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`,
+host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`
 `systemd-searxng`, `systemd-open-webui`.
 
 Two consequences that are easy to miss:
@@ -84,7 +83,7 @@ podman exec <name> cat /proc/net/route          # default route → gateway IP
 podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?'
 ```
 
-Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`,
+Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`
 `10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns,
 published) and `:9119` (its own published port) connect; `:8080` and `:8787`
 are **refused instantly** even though both listeners are bound to `0.0.0.0` in

serverAppConfigs-docs/Architecture/Deploy State Triage.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Deploy State Triage
-description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
+description: "Reading deploy-state markers against live containers and deciding, safely, whether to redeploy."
 tags: [deployment, deploy-state, systemd, podman, triage]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -22,8 +22,10 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
 "behavior pending".
 
 **Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
-whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
-a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
+whose marker lags. On the big-payload endpoints that is a full model reload
+measured in tens of minutes — each endpoint's boot budget is on its own page
+([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon
+(dankserver:7770) is also the
 harness's own model backend, so a blind deploy is a self-inflicted outage for
 possibly zero behavioral change.
 
@@ -34,7 +36,7 @@ possibly zero behavioral change.
 2. Probe the target (ssh as `dank`, passwordless sudo present):
    - `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
    - `git -C ~/serverAppConfigs rev-parse HEAD`
-   - `systemctl is-active/is-enabled <svc>`;
+   - `systemctl is-active/is-enabled <svc>`
      `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
    - the on-host env file already carries the new keys (it landed via the
      ff-pull).
@@ -46,9 +48,8 @@ possibly zero behavioral change.
    - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
    - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
      tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
-   - confirm the launcher maps each tunable to its flag
-     (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
-     --enable-prefix-caching`.
+  - confirm the launcher maps each tunable to its flag
+    (`grep scripts/launch.sh`).
 
 **Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
 live — **do NOT restart**; reconcile the marker (below). Only if the live
@@ -118,8 +119,8 @@ each, kill stale ones by explicit PID.
 A killed deploy leaves behind:
 
 - the unit **stopped/failed** (`Restart=always` does NOT fire after an
-  explicit stop — start it yourself);
-- the state marker **pending** (pre-deploy or in-flight revision);
+  explicit stop — start it yourself)
+- the state marker **pending** (pre-deploy or in-flight revision)
 - the e2e gate un-run (the deployer runs it last).
 
 Reconcile the marker ONLY after the service itself is verified healthy — a
@@ -141,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts
 numbers — which looks exactly like your change having no effect. Key on
 container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
 cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
-See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
+See  for the full capture
 procedure.
 
 ## Related
 
 - [[Deployment Flow]] — the deploy pipeline itself
 - [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
-- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
+-  journal capture

serverAppConfigs-docs/Architecture/Deployment Flow.md

@@ -1,5 +1,5 @@
 ---
-type: guide
+type: howto
 title: Deployment Flow
 description: Service-scoped Git deployment for the two homelab hosts.
 tags:
@@ -46,7 +46,7 @@ graph LR
 ./scripts/deploy.sh dankserver
 ./scripts/deploy.sh frankbook-pro --service qbittorrent   # podman-machine target
 ./scripts/deploy.sh dankclaw --service searxng
-./scripts/deploy.sh dankserver --service frank-medium
+./scripts/deploy.sh dankserver --service frank-archon
 ./scripts/deploy.sh dankclaw --dry-run
 ```
 
@@ -62,13 +62,13 @@ service on the other host.
 |---|---|---|
 | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
 | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
-| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
-| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
-| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
+| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw iGPU inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-dragoon]]. |
+| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw 3090 inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-zealot]]. |
+| `frank-archon` | systemd + `systemctl` (root unit) | `deploy_llm_archon` plans from repo paths and gates on the port; it does NOT see host-side unit drift (see [[frank-archon]] → Deploy / verify). |
 | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
 
 The shared chat template maps to the inference service on the selected target.
-The `frank-npu` endpoint is retired (2026-09-11) and has no deploy handler;
+The `frank-npu` endpoint is retired () and has no deploy handler
 an unrelated large-model deployment never touches it.
 
 ## Caddy safety
@@ -125,6 +125,5 @@ routing. See [[Testing Policy]].
 ## See Also
 
 - [[Testing Policy]]
-- [LLM deployment](../LLM/deployment.md)
 - [[Caddy]]
 - [[SearXNG]]

serverAppConfigs-docs/Architecture/E2E Suite.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "`tests/e2e/` drives the **live fleet** through a real browser: fresh contexts"
+description: "The Playwright suite that drives the live fleet through a real browser, its project set, and the FrankBook conditional."
 tags:
   - architecture
   - e2e
@@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior.
   *arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
   sleeping Mac mini must never fail an unrelated deploy. The project joins only
   when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
-  trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
+  trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`
   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
   vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
   that touches the arr links must therefore be paired with a `19-frankbook` run

serverAppConfigs-docs/Architecture/Network Architecture.md

@@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
 
 As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
 Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
-else), and accepted traffic returns to Caddy's internal listener on `:8016`,
+else), and accepted traffic returns to Caddy's internal listener on `:8016`
 which proxies every request whole to the Authentik server with the Host
 header preserved; Authentik's embedded outpost routes on that Host against
 each proxy provider's `external_host`, serves the login flow in place for
@@ -67,12 +67,12 @@ graph TB
 ```
 
 Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`):
-homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`),
+homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`)
 prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta
 (`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.`
 (the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP
 clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass
-unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP),
+unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP)
 books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts
 that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own
 auth"). Grafana is **not** exempt even though it signs in through authentik as
@@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through
 `Network=searxng.network` (runtime network `systemd-searxng`), while
 `searxng.container`, `marreta.container` and `marreta-browser.container` join
 the tunnel the modern way — `Network=gluetun.container`
-(`dankclaw/searxng/quadlet/searxng.container:18`,
-`dankclaw/marreta/quadlet/marreta.container:34`,
+(`dankclaw/searxng/quadlet/searxng.container:18`
+`dankclaw/marreta/quadlet/marreta.container:34`
 `dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet
 equivalent of the retired compose `network_mode: service:gluetun`. **Five**
 containers share the VPN exit, not three, and Marreta's outbound fetches leave
 through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of
-exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`);
+exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`)
 everything else on Dankclaw uses the normal LAN route. This matches KB Pattern
 2 in [[Gluetun]].
 See [[Container Network Namespaces]] for which addresses are reachable from that

serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Quadlet Lifecycle
-description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth.
+description: "The Quadlet discipline: ownership audits, compose cutover, drift and crash-loop triage, generator quirks, log retrieval."
 tags: [podman, quadlet, systemd, boot-persistence, compose, drift]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -36,14 +36,14 @@ done
 ## 2. Cutover: compose service → Quadlet unit
 
 Checklist (precedents to copy:
-`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
+`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
 `dankclaw/observability/quadlet/`):
 
 1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
    line for line: `Image=` (`localhost/` prefix for locally built images +
    `Pull=never`), `ContainerName=` (Caddy and scrape configs address
    containers by name), `Network=`, `User=0:0` when the image's own user
-   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
+   cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved)
    `Exec=` for compose `command:`, `PublishPort=`, health keys,
    `[Service] Restart=always` + `RestartSec=10`, `[Install]
    WantedBy=default.target`. **Never `systemctl --user enable` the generated
@@ -56,7 +56,7 @@ Checklist (precedents to copy:
    `systemctl --user restart <name>.service` → `wait_for_health`; retire any
    legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
    `~/.config/systemd/user/<svc>.service` that shells out to
-   `podman compose up` collides BY NAME with the generated unit);
+   `podman compose up` collides BY NAME with the generated unit)
    `podman rm -f <container>` so the unit's container can take the name.
 4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
    AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
@@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever.
 
 Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
 = the §2 cutover. **Verify the capability, not the status:** `podman exec
-<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
+<name> python3 -c "import torch; print(torch.cuda.is_available())"`
 `ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
 2026-09-16, stale device-less container, 9,146 restarts. History repeats:
 2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
@@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
   unhealthy but logs look fine, compare the baked
   `{{.Config.Healthcheck.Test}}` array against the intended command before
   believing the status.
-- **Probe a new key before shipping:** drop a throwaway `probe.container`,
+- **Probe a new key before shipping:** drop a throwaway `probe.container`
   `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
   "unsupported key"`, `systemctl --user cat probe.service` (shows the
   generated ExecStart verbatim), then remove + reload.

serverAppConfigs-docs/Architecture/Stale Signal Audit.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Stale Signal Audit
-description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time.
+description: "The audit battery for surfaces that rot without failing: dead probes, vacuous assertions, gates that never run."
 tags: [testing, observability, audit, signals, gates]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -34,8 +34,9 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
 `/metrics` that is disabled (Gitea 404).
 
 - A scrape cannot express "is this HTTP service answering?" — the right probe
-  is the **blackbox exporter** in probe mode (which also gives guarded vhosts
-  a real uptime signal, since Prometheus cannot solve an Anubis challenge).
+  is the **blackbox exporter** in probe mode
+  (`/probe?target=…&module=http_2xx`), which also gives guarded vhosts
+  a real uptime signal, since Prometheus cannot solve an Anubis challenge.
 - **Fix in the direction of honesty, never of silence:** unreachable target →
   mark the homepage card `wip: true` with the reason in `description` (the
   checker skips `wip`; `wip` is the existing convention for "real service,
@@ -43,8 +44,8 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
   database port on the LAN to make a badge green, and never delete a scrape
   job — an absent scrape and a healthy one are indistinguishable.
 - Alert rules must not depend on always-red signals — scope them to services
-  that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
-  exactly that reason).
+  that can actually be healthy; the `frank-*` scope `LlmEndpointDown` carries
+  is recorded on [[Prometheus]].
 - A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
   `connection refused` to `unsupported Content-Type "text/html"` means the
   service came up and only the scrape is wrong. Read the error, not just

serverAppConfigs-docs/Architecture/Testing Policy.md

@@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con
 ## See Also
 
 - [[Deployment Flow]]
-- [[index]]

serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md

@@ -1,6 +1,6 @@
 ---
 type: howto
-description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next."
+description: "Proven Qwen-Image text-to-image workflows on Dankclaw: the recipes, the API call shape, and which models to add next."
 tags:
   - dankclaw
   - comfyui
@@ -56,8 +56,8 @@ sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed
 fixed in the file (`1930` in the art workflow, `2026` in the text one) —
 read from `dankclaw/comfyui/workflows/*.json` node `9`.
 
-Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`,
-`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`,
+Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`
+`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`
 `SaveImage` (ids 3–11).
 
 | Workflow | File | Shape | Steps | Made for |
@@ -113,9 +113,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file
 Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats`
 reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats`
 2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and
-all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`,
-`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`,
-`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`,
+all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`
+`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`
+`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`
 `huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk
 yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18).
 
@@ -126,7 +126,7 @@ yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18)
 | 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) |
 | 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) |
 
-Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`,
+Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`
 #3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`.
 Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count +
 sha256 checks) or a plain `hf download` with the token in the environment —

serverAppConfigs-docs/Dankclaw/ComfyUI.md

@@ -43,7 +43,7 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not
 
 ## GPU device access (rootless), and why the unit exists
 
-The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`,
+The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`
 `dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses
 `GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman
 cannot map the group name into the user namespace. The companion udev rule
@@ -115,7 +115,7 @@ generated units.
 The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/`
 2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats`
 reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The
-health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`,
+health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`
 bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 (`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs
 `http://127.0.0.1:8188/` inside the container, and the container reports
@@ -135,5 +135,5 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd`
 - [[Dankclaw Stack]]
 - [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
 - [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions
-- [[Lemonade and sd.cpp]] — the ggml-family alternative
+-  the ggml-family alternative
 - Knowledge base: [[ComfyUI]] (tools)

serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md

@@ -13,11 +13,10 @@ timestamp: 2026-08-26
 # Dankclaw Stack
 
 Dankclaw (192.168.0.143) runs the AI, search, development, and observability
-stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX
-llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
-the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
-FLM v1.0.1) serves sight+sound models. The completion endpoint
-`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235).
+stack. Its LLM inference endpoints are bare-metal host services, not part of
+the container stack; each has its own page — shape, payload, eras and wire
+contract live there, nowhere else. See [[frank-dragoon]] and [[frank-zealot]]
+(both on this host) and [[frank-archon]] (Dankserver, 192.168.0.235).
 
 ```mermaid
 graph LR
@@ -28,9 +27,8 @@ graph LR
     C --> F[ComfyUI]
     C --> D[Grafana]
     S --> V[Gluetun VPN]
-    O --> L[llama.cpp dankserver :7770]
+    O --> L[LLM endpoints]
     O --> H[Hermes]
-    O --> E[frank-dragoon :7770]
 ```
 
 ## Services
@@ -48,18 +46,16 @@ graph LR
 | Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
 | Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
 | Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
-| frank-dragoon (host service) | — | `7770` | [[deployment]] |
-| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
-| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
+| frank-dragoon (host service) | — | `7770` | [[frank-dragoon]] |
+| frank-zealot (host service) | — | `7771` | [[frank-zealot]] |
+| frank-archon (host service, Dankserver) | — | `7770` | [[frank-archon]] |
 
-The LLM rows are bare-metal systemd services, not containers. This
-repo owns their scripts, units, and knowledge base (see
-[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
-`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
-on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
-`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
-`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
-kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver.
+The LLM rows are bare-metal systemd services, not containers, and this repo
+owns their scripts, units and knowledge base (see
+). Which payload and runtime sits on
+each port is era state and lives on the endpoint page, not here. `frank-npu`
+(the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon`
+on the Dankserver port) are both retired — see  and
 
 ## Boot Persistence
 
@@ -75,7 +71,7 @@ Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-
 
 ## See Also
 
-- [[index]] — repository index
+-  repository index
 - [[Network Architecture]] — routing and proxy
 - [[Observability]] — Grafana, Prometheus, logs
-- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
+-  the model-serving endpoints on this host

serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker, on a private podman bridge behind frank-tts — repo-owned, not yet deployed."
+description: "The engine behind the fleet's speech endpoint: its runtime, private bridge, and the surface it publishes."
 tags:
   - dankclaw
   - tts
@@ -14,7 +14,7 @@ timestamp: 2026-10-18
 
 `frank-tts-engine` is the **arm A text-to-speech engine** on Dankclaw: [qwentts.cpp](https://github.com/ServeurpersoCom/qwentts.cpp)'s resident `tts-server` HTTP process, holding the Qwen3-TTS **1.7 B base talker (Q4_K_M)** and the **12 Hz tokenizer (Q8_0)** in RAM once and synthesizing from there. It listens on container port **8880** and is **deliberately not a fleet endpoint** — the endpoint consumers address is `frank-tts` on **7772**, which resolves voice names, re-encodes to mp3 and owns the timeout budget. This page is the engine runbook; the endpoint's contract lives with its own page.
 
-**Nothing here is deployed.** This page documents repo-owned configuration — the Quadlet unit and network, the image build, the weight fetcher, the boot warmer, the deployer path — and the measured facts those files were written against. No container, image, weight file or network exists on Dankclaw yet, and the engine publishes no port at all, so there is nothing for the homepage checker or a browser to reach. Deploying it is a deliberate next step, in the order at the bottom of this page.
+**Deployed 2026-10-18.** The chain ran in the order at the bottom of this page — fetch (1.4 GiB, byte + LFS oid verified, 18 s), image build against the pinned commit, unit + network install, container-health gate, warm pass — and the live state still reads the way this page describes it: unit `active` with no drop-ins, container healthy 31 s after the build, **4.0 GiB** resident on a host that kept 17 GiB available, `3 voice(s) registered and warm` in the journal, and no published port anywhere. Two things the first real boot taught: the warm hook's registration body must reach curl through a **file**, not an argument (Troubleshooting), and `STEP 3/13 RUN useradd` is the first honest proof of the image's non-root user.
 
 **Host:** Dankclaw (`192.168.0.143`), rootless Podman, user `dank` — **Port:** 8880, **never published** — **Network:** the private `systemd-frank-tts-engine` bridge (10.89.5.0/24) and no other — **Deployer:** `frank-tts-engine` — **Consumers:** `frank-tts` (7772) and, through it, every voice request in the fleet including [[Open Notebook]]'s podcast renders.
 
@@ -37,7 +37,7 @@ timestamp: 2026-10-18
 
 ## Why this engine, and not the alternatives
 
-The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in [[Local TTS and Voice Cloning]], which is the source of these numbers):
+The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in , which is the source of these numbers):
 
 | Measure | Resident `tts-server` (arm A, chosen) | Mainline per-request CLI path (arm B) |
 |---|---|---|
@@ -96,25 +96,27 @@ The **façade** on 7772 is the pair's public surface and carries the probe. Note
 |---|---|
 | `dankclaw/voiceservice/Containerfile.engine` | two-stage build of the pinned commit, proven cmake line, digest-pinned base, ships both scripts |
 | `dankclaw/voiceservice/tts-engine-start.sh` | image entrypoint (`/app/start.sh`): env → flags, refuses to boot without weights |
-| `dankclaw/voiceservice/tts-engine-warm.sh` | boot warmer (`/app/warm-voices.sh`): register from `reference.wav`, speak one sentence, report cold/warm |
+| `dankclaw/voiceservice/tts-engine-warm.sh` | boot warmer (`/app/warm-voices.sh`): register from `reference.wav` (body via a temp file), speak a fixed sentence twice, report cold/warm |
 | `dankclaw/voiceservice/quadlet/frank-tts-engine.container` | the unit: bridge only, no publish, env file, two read-only mounts, health probe, warm hook, `Restart=always` |
 | `dankclaw/voiceservice/quadlet/frank-tts-engine.network` | the private 10.89.5.0/24 bridge, with the subnet registry |
-| `scripts/frank-tts-engine.env` | paths and numbers only, with the thread law and the cold-voice law in its header |
+| `scripts/frank-tts-engine.env` | paths and numbers only, with the thread law and the registry law in its header |
 | `scripts/download-frank-tts-engine-qwentts.sh` | weight fetcher: no resume, byte count + LFS oid check, `--verify-only` |
 | `scripts/deploy-remote.sh` → `deploy_frank_tts_engine` | weights gate → build → quadlet → restart → container health → warm |
 | `tests/frank_tts_engine.bats` | the seam: generator output, entrypoint argv, warmer against a stub engine, planner and routing maps |
 
-## Deploy order (host steps, in order)
+## Deploy record (host steps, in the order they ran)
 
-Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblocks the podcast pipeline.
+Run 2026-10-18 as `./scripts/deploy.sh dankclaw --service frank-tts-engine`, then
+`--service frank-tts`. Steps 1–6 are this service and the façade; only step 7 is
+still open.
 
 1. **Fetch the weights** — `scripts/download-frank-tts-engine-qwentts.sh` (1.4 GiB, byte + sha256 verified, idempotent; `--verify-only` is what the deployer calls).
-2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). Builds the image (needs github.com once), installs the unit and the network file, restarts, gates on the container's health status, then warms.
+2. **Deploy the engine** — `./scripts/deploy.sh dankclaw --service frank-tts-engine` (`--service` is the only way to name one service; `deploy-remote.sh` is the remote half and takes `<target> <git-revision> [service]...`, so never call it by hand with a bare service name). It builds the image (needs github.com once), installs the unit and the network file, gates on the container's health status, then warms.
 3. **Confirm the endpoint is alive** — `podman exec frank-tts-engine curl -fsS http://127.0.0.1:8880/health` → `{"status":"ok"}`, and `systemctl --user status frank-tts-engine` showing the unit `active` with the warm pass behind it.
-4. **Confirm the voices are known** — the warmer's journal lines must name every pack; once the façade is up, `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
-5. **Prove the cold-voice law is paid at boot, not by a listener** — two identical synthesis requests for one voice, back to back, from inside the engine's namespace; the **second** must answer well under a second (`warm=` in the journal is already that number; the manual pair is the confirmation on the real artifact).
-6. Then the façade on **7772** (its own page): nothing may be published there until the engine answers, because open-notebook's render fails the moment the façade is up and the engine is not.
-7. Nothing on 7772 exists yet — that is why open-notebook's podcast render cannot run. See [[Open Notebook]].
+4. **Confirm the voices are known** — the warmer's journal lines name every pack, and once the façade is up `curl http://127.0.0.1:7772/v1/audio/voices` is the authoritative list (the façade has exactly three routes: `/health`, `/v1/audio/voices`, `/v1/audio/speech`).
+5. **Measure the ladder instead of trusting the trial** — two identical requests for one voice back to back. The design expected 6.2-8.0 s then 0.5 ms; the deployed container gave **1,883 ms then 1,226-1,283 ms**, so the hook buys ~0.6 s and stays only for the registry reason (law 1). The retired numbers must not be quoted.
+6. **Then the façade on 7772** (its own page), after the engine answered — that order is the whole point of step 6: publishing the façade in front of a dead engine would make every consumer fail at the first voice line.
+7. **Open:** [[Open Notebook]] is still not installed, so nothing renders a podcast yet; the voice leg it waits on now exists.
 
 ## Troubleshooting
 
@@ -126,6 +128,6 @@ Nothing below has been run. Steps 1–5 are this service; 6–7 are what unblock
 | Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) |
 | `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work |
 | A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design |
-| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] |
+| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see  |
 | The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this |
 | A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces |

serverAppConfigs-docs/Dankclaw/Frank TTS.md

@@ -1,11 +1,13 @@
 ---
 type: reference
-description: "frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in front of frank-tts-engine; repo-owned, not yet deployed."
+description: "The fleet's cloned-voice speech endpoint and the one voice surface consumers address; wired as the harness speech role."
 tags:
+ - endpoint
  - dankclaw
  - tts
  - voice
  - quadlet
+timestamp: 2026-10-19
 ---
 
 # Frank TTS
@@ -16,20 +18,26 @@ audio in that cloned voice. Everything else in the voice stack sits behind it
 the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers
 never learn a path.
 
-**Nothing here is deployed.** No host was touched while this was written: no
-image built, no container started, nothing listening on 7772. That is also why
-[[Open Notebook]] cannot render a podcast episode yet — its pipeline calls this
-endpoint.
+**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`
+both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
+the same change. Probes on the deployed endpoint: `/health` answers
+`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`
+and two different voices rendered the same sentence into **different** bytes
+(24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
+pass-through of one default. The operator harness routes its `speech` role here;
+see [[#The harness speech role]]. What still waits is [[Open Notebook]]: its
+render needs the app installed, not this endpoint.
 
 | | |
 | --- | --- |
 | Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
-| Port | **7772**, published (`PublishPort=7772:7772`); the family is 7770 dragoon, 7771 zealot, 7772 tts |
+| Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] |
 | Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only |
 | Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
 | Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
 | Voice packs | `/home/dank/voices` mounted read-only at `/voices` — host state, never repo state |
 | Upstream | `http://frank-tts-engine:8880` (`scripts/frank-tts.env`, `TTS_UPSTREAM`) |
+| Default voice | `DEFAULT_VOICE` (`scripts/frank-tts.env`) answers a request that names none; the journal marks that line `(default)`, and an **unknown** name is still a 400 |
 | Rollback | `TTS_BACKEND=llama-tts-cli` runs a CLI binary per request and needs no upstream at all |
 
 ## Naming a voice
@@ -37,7 +45,9 @@ endpoint.
 A voice **is** a directory name. `voice=narrator1` resolves to
 `/voices/narrator1/reference.wav`, and that file — not a manifest, not a config
 entry — is what makes the name real. omp's TTS device may send `voice_id`; the
-shim accepts either.
+shim accepts either. A call that sends **neither** — the harness's normal shape,
+because no harness setting carries a voice for this transport — speaks
+`$DEFAULT_VOICE`; empty default makes it a 400 again.
 
 The usable shape is `^[a-z0-9][a-z0-9._-]*$`: lowercase, starting with a letter
 or digit, then letters, digits, `.`, `_`, `-`, at most 64 characters. No slash,
@@ -101,6 +111,21 @@ own `/health` answers 200 with the backend name and the voice list it resolved
 without dialing the engine, so "façade healthy, engine down" is a real state: it
 shows up as a failed synthesis, never as a down probe.
 
+## Where the engine computes
+
+CPU, and only CPU is compiled in: `Containerfile.engine` builds `qwentts.cpp` with
+`-DGGML_VULKAN=OFF` written out explicitly, so the container carries `libggml-cpu` and no
+other backend. The realistic accelerator is the gfx1151 iGPU behind that flag — plus
+`/dev/dri` passthrough, the ComfyUI pattern — on the same payload and the same pinned
+commit, and the intended shape is **co-tenancy**: the engine resident on the card beside
+`frank-dragoon`, not a handover. Memory is not what decides that; shader contention is, and
+the census plus both gates are in `report/tts-npu-spike-verdict-2026-10-19.md` §5.
+**The Strix Halo NPU is not a door at all**: XDNA2 executes only precompiled AIE
+partition programs, and AMD's own Linux runtime opens `/dev/accel/accel0`, succeeds at the
+ioctls, then declines this model family's graph without emitting a single log line. The
+measured record and the model-side contract live in `report/` at the repo root
+(`tts-npu-spike-verdict-2026-10-19.md`).
+
 ## Files
 
 | Path | What |
@@ -109,7 +134,7 @@ shows up as a failed synthesis, never as a down probe.
 | `dankclaw/voiceservice/Containerfile` | python 3.12-alpine pinned by digest + ffmpeg, uid 10000 |
 | `dankclaw/voiceservice/quadlet/frank-tts.container` | the unit: two networks, `Restart=always`, `HealthCmd`, `[Install]` |
 | `dankclaw/voiceservice/quadlet/frank-tts-engine.network` | the shared bridge, named after the engine unit and attached by both |
-| `scripts/frank-tts.env` | backend, upstream, port, limits — paths and numbers, no credentials |
+| `scripts/frank-tts.env` | backend, upstream, port, limits, and the default voice — paths and numbers, no credentials |
 | `tests/frank_tts_unit.bats` | the shim against stub engines over real sockets |
 | `tests/frank_tts_engine.bats` | the pair's wiring: generator output, planner, homepage, deployer |
 
@@ -134,12 +159,55 @@ types: [[Prometheus]] records that the existing HTML probes (`comfyui`, the two
 `services` targets) sit **down** with
 `received unsupported Content-Type … and no fallback_scrape_protocol specified`.
 A `/health` probe job would therefore be structurally never-green, and because
-`LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, a permanently red
-`frank-tts` target would page on Signal every few minutes for an endpoint that
-is fine — the frank-archon lesson pointed the other way. The engine cannot be
-scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
-`podman healthcheck run`, the two unit journals, and the homepage card, which is
-gated `wip: down` until the deploy and probes `127.0.0.1:7772`.
+`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
+lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
+Signal every few minutes for an endpoint that is fine. The engine cannot be
+scraped at all: it publishes nothing. Status lives in `systemctl --user status`
+`podman healthcheck run`, the two unit journals, and the homepage card, which
+probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
+2026-10-18 deploy.
+
+## The harness speech role
+
+The operator harness reaches this endpoint through
+`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`
+`baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api
+`openai-speech` on the **model** row (naming a runner api at *provider* level
+fails the whole file's validation and disables every custom provider, which is
+how the fleet's chat endpoints disappear) — plus `config.yml`
+`modelRoles.speech: frank-tts (dankclaw)/frank-tts`. Four laws, read out of the
+harness bundle and measured the day of the flip:
+
+- **The only voice the harness can send is the `voice_id` argument of the call.**
+  The transport builds `{model, input, response_format, voice?}` and adds `voice`
+  **only** when the caller passed `voice_id`; no setting feeds it.
+  `tts.localVoice` is the *local Kokoro backend's* voice (it is what answered
+  `voice=kokoro/bm_george, backend=local` in the control test), and
+  `speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list,
+  and it never reaches an HTTP request. So a harness call arrives voice-less by
+  default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared
+  pack answers a voice-less request and the journal marks that line `(default)`
+  while an **unknown** name still 400s with the list that exists.
+- **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not
+  select a pack; it hands the vocalizer a value outside its enum. (The first
+  draft of this section said the opposite and was reverted the same day, after a
+  fresh session's call came back `unknown voice None`.)
+- **There is no silent fallback.** An explicitly configured `modelRoles.speech`
+  yields exactly one candidate, so a dead 7772 fails the speech device with
+  *"No available speech model matches the speech role"* instead of quietly
+  degrading to Kokoro. That is the price of `auth: none` (which makes a dead
+  endpoint read as available) and it is the wanted shape: a narration job should
+  fail loudly rather than ship a voice Frank never asked for.
+- **A session older than the models.yml row cannot see it.** `models.yml` loads
+  at process start while `config.yml` is read live, so a long-lived session
+  carrying the flipped role fails exactly like a dead endpoint while a fresh
+  process succeeds in the same minute. Measured 2026-10-18: the open session
+  failed twice (provider/id form, then the bare id), `local/kokoro` succeeded in
+  that same session, and a fresh `omp -p --tools tts` returned `Saved 180524
+  bytes (model=frank-tts (dankclaw)/frank-tts, voice=narrator1, codec=wav)`
+  while the façade logged the matching `speech voice=narrator1 format=wav
+  chars=51 -> 180524 bytes` from the podman bridge. Rule the stale registry out
+  before blaming the service.
 
 ## Troubleshooting
 
@@ -150,3 +218,5 @@ gated `wip: down` until the deploy and probes `127.0.0.1:7772`.
 | Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
 | First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
 | The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
+| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
+---

serverAppConfigs-docs/Dankclaw/Gitea.md

@@ -1,5 +1,5 @@
 ---
-type: configuration
+type: reference
 title: Gitea
 description: Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service.
 tags:
@@ -33,7 +33,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net
 | `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) |
 | `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* |
 
-The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)
+The LLM fleet knowledge base:
 
 ## Management
 

serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md

@@ -30,7 +30,7 @@ entrypoint does not start the dashboard. The dashboard is a separate process, so
 it runs in its own Quadlet unit
 ([`dankclaw/hermes/quadlet/hermes-dashboard.container`](../../dankclaw/hermes/quadlet/hermes-dashboard.container):
 image `localhost/hermes-agent:latest`, `Pull=never`, `Exec=hermes dashboard
---host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`,
+--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`
 `RestartSec=5`). All three Hermes containers mount the same `/data` bind mount.
 
 Unlike its two siblings — `hermes.container` and `hermes-webui.container` both

serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md

@@ -1,7 +1,7 @@
 ---
-type: configuration
+type: reference
 title: Hermes-WebUI (Hermex server)
-description: The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH tunnel, from the whole LAN (it binds all interfaces), and from https://hermes.franksdank.cloud (internal-only vhost).
+description: "The iPhone companion app for Hermes on Dankclaw and the ways it is reached from the LAN."
 tags:
   - hermes
   - hermes-webui
@@ -40,7 +40,7 @@ from another LAN host: `/health` → 200, `/` → 302) — **Vhost:**
 
 ## Why a container, not a host process
 
-The gateway's state files are owned by the container-mapped uid (`109999`,
+The gateway's state files are owned by the container-mapped uid (`109999`
 from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` →
 `dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 =
 109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless
@@ -82,12 +82,12 @@ the sibling dashboard on `9119` is not.
 
 - Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994,
   shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only
-  *(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
+  - (unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable
   without `sudo`)*; the account can do nothing except
   authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel
   while the transport stays up for port forwarding.
 - sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31)
-  scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`,
+  scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`
   `X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are
   present under `Match User tunnel` (`cat` 2026-10-18).
 - Key options on the authorized key pin the blast radius:
@@ -130,7 +130,7 @@ only if inactive (it is never restarted on deploy, so in-flight browser
 sessions survive), re-tags the image alias, then restarts
 `hermes.service`, `hermes-dashboard.service` and
 `hermes-webui.service` in the **user** manager of `dank` (linger must be on for
-them to come up at boot — `loginctl show-user dank -p Linger` → `yes`,
+them to come up at boot — `loginctl show-user dank -p Linger` → `yes`
 2026-10-18), so gateway/dashboard/webui restarts stay isolated from
 inference deploys.
 Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live

serverAppConfigs-docs/Dankclaw/Hermes.md

@@ -18,7 +18,7 @@ bindings, 2026-10-18). Note the **container name drift**: the unit is
 `podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so
 `podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no
 `ContainerName=` key even though its header comment says "container name:
-hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via
+hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached
 `hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A
 record; the name resolves from the Pi-hole hosts file,
 `dankclaw/pihole/addn-hosts:39`) — **Docs:**
@@ -37,7 +37,7 @@ designed wiring; the remedy for both halves is in the log line
 
 **Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted
 at `/data` in all three Hermes containers (gateway, webui, dashboard —
-`hermes.container:36`, `hermes-webui.container:46`,
+`hermes.container:36`, `hermes-webui.container:46`
 `hermes-dashboard.container:24`). Each
 container additionally gets an anonymous volume at `/opt/data`, which is
 re-created empty on every container recreation — anything that lives only there
@@ -78,9 +78,9 @@ is not durable. The tracked, sanitized configuration baseline is
   `/data/config.yaml` is not readable without `sudo`):
   `dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
   `frank-dragoon` — verified from the host with `curl
-  http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len
-  200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is
-  a rootless pasta publish and is IPv4-only);
+  http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
+  `127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
+  and is IPv4-only)
   `dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
   2026-10-18, that endpoint serves only `frank-archon`** (`GET
   http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
@@ -88,7 +88,8 @@ is not durable. The tracked, sanitized configuration baseline is
   2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
   if the live `config.yaml` still names `frank-medium` the provider is dead
   weight and requests will 404 on a nonexistent model. Diff the live config
-  before trusting either name.
+  before trusting either name. Which payload sits behind each port, and the
+  served id it answers to, is on [[frank-dragoon]] and [[frank-archon]].
 - The live `config.yaml` contains secrets and user state and is not committed;
   the tracked example is the sanitized backup of its non-secret settings.
 
@@ -114,7 +115,7 @@ per session; the browser itself lives in a dedicated daemon container, the
 digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host
 network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows
 `127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image
-`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`,
+`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`
 `dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal
 gateway and the webui server — run in the host network namespace (`podman
 inspect` → `net=host` for both, 2026-10-18), so they can
@@ -227,9 +228,10 @@ resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
 
 What actually breaks, and the fix:
 
-- **`response_format` (json_object / json_schema) is a hard 400 on every
-  local backend here** — neither FreeToken nor halogen implements constrained
-  decoding. Config cannot override it: `call_llm` merges the task's
+- **`response_format` (json_object / json_schema) is a hard 400 on the local
+  backends this task was written against** — what each endpoint accepts on the
+  wire is on [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Config
+  cannot override it: `call_llm` merges the task's
   `extra_body` first and then `update()`s the caller's, so the hardcoded one
   wins. Only temperature and max_tokens have reactive retry paths.
 - **Fix: move that one task to the Responses transport**, which never
@@ -246,9 +248,8 @@ What actually breaks, and the fix:
           effort: none                 # mandatory, see below
   ```
 
-  Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw
-  `:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider
-  name is recorded against — answers only `frank-archon`. The provider/model
+  Before copying this: which id is served on which port is endpoint state —
+  [[frank-zealot]] and [[frank-archon]]. The provider/model
   pairing above is the shape that was landed in 2026; re-derive it from the live
   config before reusing it.
 
@@ -281,7 +282,7 @@ Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth
 on match, 401 on miss. No username field. `/health` answers 200
 unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with
 `{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w
-%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via
+%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth
 those**; probe
 login with a deliberately wrong password (expect 401) and the real one
 (expect 200). Details on the webui surface: [[Hermes-WebUI]].
@@ -310,7 +311,7 @@ login with a deliberately wrong password (expect 401) and the real one
    instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18;
    off-network ingress is SSH tunnel only —
    user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth
-   *(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*,
+   - (unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*
    forward remote end
    localhost:8787).
 

serverAppConfigs-docs/Dankclaw/Marreta.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Marreta on Dankclaw
-description: The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet pair that shares SearXNG's Gluetun netns for egress and sits behind the Anubis + Authentik guard chain at marreta.franksdank.cloud.
+description: "The content-cleaning proxy SearXNG borrows egress from: a quadlet pair in the VPN netns behind the guard chain."
 tags:
   - dankclaw
   - marreta
@@ -19,7 +19,7 @@ returns a stripped, readable version of the page — no paywall chrome, no
 tracking, no cookie walls. It is a Laravel 13 app on the
 `serversideup/php:8.4-fpm-nginx` image (`composer.json` requires
 `php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` —
-<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>),
+<https://raw.githubusercontent.com/manualdousuario/marreta/main/composer.json>)
 with an optional headless-browser companion for JavaScript-heavy pages.
 
 **Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained)
@@ -31,7 +31,7 @@ with an optional headless-browser companion for JavaScript-heavy pages.
 - Fetches a target URL with a cascade of strategies: plain HTTP (curl), a
   headless browser (Lightpanda) for JS pages, and the Internet Archive
   (Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php`
-  files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`,
+  files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`
   `WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`).
 - Strips paywall/consent/tracking markup and re-renders clean HTML.
 - Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` /
@@ -69,7 +69,7 @@ graph LR
 | `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) |
 
 Both units are boot-persistent (`WantedBy=default.target`) and self-healing
-(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`,
+(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`
 `marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live
 2026-10-18 both report `systemctl --user is-enabled` → `generated` and
 `is-active` → `active`, and `podman inspect` reports marreta **healthy**
@@ -87,14 +87,14 @@ The fix is a single env var, verified against the serversideup source:
 
 - The image sets `ENV NGINX_HTTP_PORT="8080"` and the site template is
   `listen ${NGINX_HTTP_PORT} default_server;`.
-- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build);
+- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build)
   the entrypoint `10-init-webserver-config.sh` runs `envsubst` over every
   `NGINX_*` variable **at container start**.
 - So `Environment=NGINX_HTTP_PORT=8085` in the quadlet relocates the listen
   port off the collision with no custom build.
 
 Marreta then publishes **nothing** of its own — the netns owner (Gluetun)
-publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`),
+publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`)
 exactly as it publishes `8081:8080` for SearXNG. Caddy dials
 `192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the
 relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085`
@@ -127,7 +127,7 @@ on the mechanism: `BrowserFetchStrategy.php` only calls
    (`dankclaw/Caddyfile:159-163`).
 2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies
    whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`).
-3. **Post-auth routing** — authenticated traffic returns to `http://:8017`,
+3. **Post-auth routing** — authenticated traffic returns to `http://:8017`
    where `@marreta host marreta.franksdank.cloud` routes to
    `reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`).
 
@@ -164,7 +164,7 @@ Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0
 — that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and
 this container sets no `TRUSTED_PROXIES`, so the default is what runs
 (`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL`
-starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`,
+starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`
 `URL::forceScheme('https')`). With
 `APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded
 scheme and client IP through the whole chain, so there is no redirect loop
@@ -234,7 +234,7 @@ netns down and would otherwise orphan it).
 Deployed via `deploy.sh dankclaw --service gluetun --service marreta --service
 caddy` (gluetun first to recreate the tunnel with the new `8082:8085` publish,
 which also cycles SearXNG's netns ref). Authentik `ProxyProvider` `marreta`
-registered (pk 13, `external_host=https://marreta.franksdank.cloud/`,
+registered (pk 13, `external_host=https://marreta.franksdank.cloud/`
 `internal_host=http://caddy:8017`). Results:
 
 - Unauth edge → Anubis challenge page (not 502). TLS cert issued via DNS-01.
@@ -245,14 +245,14 @@ registered (pk 13, `external_host=https://marreta.franksdank.cloud/`,
   (`GET 127.0.0.1:9222/json/version` → `Lightpanda/1.0`).
 - SearXNG unaffected: `:8081` → 200 after the tunnel recreate.
 
-> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`,
+> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`
 > which 500s every app route (see Secrets above). Fixed by re-keying with the
 > `base64:` prefix before any encrypted data was persisted.
 
 ## Citations
 
 - Marreta upstream: <https://github.com/manualdousuario/marreta> (README,
-  `compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`,
+  `compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`
   `lang/` directory listing).
 - serversideup/php fpm-nginx port mechanism: `serversideup/docker-php`
   `src/variations/fpm-nginx/Dockerfile` (`NGINX_HTTP_PORT="8080"`) and

serverAppConfigs-docs/Dankclaw/Open Notebook.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out — script written by frank-zealot, voices rendered by frank-tts, no cloud provider anywhere in the path."
+description: "The fleet's podcast generator: notes in, multi-speaker episode out, script and voices both served in-house."
 tags:
   - dankclaw
   - open-notebook
@@ -13,7 +13,7 @@ timestamp: 2026-10-18
 
 open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook)) is the notebook layer that turns existing notes and sources into a **spoken episode**: an outline, then a dialogue transcript, then one synthesized clip per speaker turn, concatenated with ffmpeg into an MP3. In the fleet it is the consumer that closes the loop Frank's audio stack was built for — the script is written by **frank-zealot**, the voices are rendered by **frank-tts**, and nothing leaves the LAN.
 
-**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Deploying it is a deliberate next step, not a pending one.
+**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Re-censused 2026-10-19 and unchanged: no `*notebook*` unit in `systemctl --user list-unit-files`, nothing installed under `~/.config/containers/systemd/` by that name, no container in `podman ps` — the repo's three quadlet files have never been installed on the host. Deploying it is a deliberate next step, not a pending one.
 
 **Host:** Dankclaw (rootless Podman, user `dank`) — **Ports:** 8502 UI and 5056 API, published on the LAN, **no vhost and no Caddy route** — **Deployer:** `open-notebook` (one service, two units) — **Docs:** https://github.com/lfnovo/open-notebook
 
@@ -59,9 +59,9 @@ podman exec open-notebook curl -sS http://frank-tts:7772/health
 podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/models
 ```
 
-**The blocker to state plainly:** frank-tts is built, named and tested but **not deployed** — nothing answers on 7772 (`ss -ltn`, Dankclaw, 2026-10-18; [[Local TTS and Voice Cloning]] records the same census). open-notebook deployed today writes scripts and fails at the first voice line. frank-tts ships first, or podcast generation waits.
+**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
 
-**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
+**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). The decision rests on each endpoint's documented request surface ([[frank-zealot]], [[frank-dragoon]]): zealot's is a known-good target for that call shape, dragoon's structured-output behaviour is **unverified**, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
 
 ## Speakers and voices: 1–4, and the IDs are free text
 

serverAppConfigs-docs/Dankclaw/OpenWebUI.md

@@ -39,28 +39,26 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve
 `frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
 `shared/homepage/services.json`)*.
 
-| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host |
-|---|---|---|---|---|---|
-| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
-| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
-| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 3×262,144 per `scripts/frank-zealot.env`) | Dankclaw |
+| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
+|---|---|---|---|---|
+| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
+| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
+| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
 
 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
 bridge move; on this network they are wrong — see the reachability gotcha below, which
-is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771`
-closed the same day (production unit stopped 03:38 per the repo Benchmarking
-Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
-PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the
-picker row below was re-keyed to it (see the next paragraph).
+is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
+that day's payload swap (see [[frank-zealot]] for what was deployed and when).
 
 The `model` table carries exactly two decoration rows, `frank-zealot` and
 `frank-dragoon`. Both the context length and the description live **inside
 JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
 `{"description": …}`; there is no `context_length` or `description` column, so
-`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18:
-`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon`
-→ 200,704 (its `params` had kept the pre-lowering 262,144 window from
-2026-10-17 while its own description already said 200,704). These rows are
+`sqlite3` one-liners against those names fail. `params.context.length` is a copy
+of the endpoint's per-slot window, so the authoritative number lives on the
+endpoint page ([[frank-zealot]], [[frank-dragoon]]) and the row must follow it —
+the 2026-10-18 re-key existed because `frank-dragoon`'s row still carried the
+window its payload had before that day's swap. These rows are
 live DB state, so a payload swap must re-key the database too, not only the
 repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
 stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
@@ -104,16 +102,16 @@ Two gotchas make the wiring non-obvious:
 The Ollama connection is disabled (`ollama.enable=false`).
 
 > All three endpoints are bare-metal systemd services managed by this
-> repository. Model swaps, context sizing, and spec tuning happen there. See
-> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only
-> hides it from this picker — the endpoint itself keeps serving every other
-> client (the harness, Hermes, API consumers).
+> repository; their shape, swaps and tuning live on
+> [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Disabling a
+> connection here only hides it from this picker — the endpoint itself keeps
+> serving every other client (the harness, Hermes, API consumers).
 
 ## Container introspection (DB and backend python)
 
 - Working dir for imports: `/app/backend`; package name `open_webui`
   (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
-  BEFORE importing any `open_webui` module (import-time hard requirement);
+  BEFORE importing any `open_webui` module (import-time hard requirement)
   the secret lives at `/app/backend/.webui_secret_key`.
 - **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
   module (`podman exec open-webui python3 -c "import sqlite3;
@@ -124,8 +122,8 @@ The Ollama connection is disabled (`ollama.enable=false`).
   pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
   `status`/`enabled` attribute — use `is_active` from the DB.
 - **API auth for verification:** `create_token` lives in
-  `open_webui.utils.auth` (NOT `open_webui.utils.utils`);
-  `get_verified_user` requires an `id` claim (not just `sub`);
+  `open_webui.utils.auth` (NOT `open_webui.utils.utils`)
+  `get_verified_user` requires an `id` claim (not just `sub`)
   `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
   container:
 
@@ -143,14 +141,14 @@ The Ollama connection is disabled (`ollama.enable=false`).
    models. Response key is `items`, not `data`.
 2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
    with exactly one word: pong") → each returns content `pong`. Reasoning
-   models need a generous budget: frank-zealot (Qwen3.5 template) returns
-   empty content + `finish_reason: "length"` at `max_tokens 200` (the
-   thinking block eats it) and answers `pong` at 1500.
+   endpoints need a generous budget: thinking consumes `max_tokens`, so a short
+   one returns empty content + `finish_reason: "length"` (see
+   [[frank-zealot]]); `frank-zealot` answers `pong` at 1500.
 3. A 500 "Server Connection Error" → `podman logs open-webui` for
    `ClientConnectorError`: a network-reachability problem (host LAN IP vs
    `host.containers.internal`), not a model problem.
 
-**Durability:** the model table + connections are DB-only (not git-managed);
+**Durability:** the model table + connections are DB-only (not git-managed)
 a fresh data volume loses them — including the public access grant above.
 For reproducibility, add an idempotent first-start seed script or document
 the setup in the service doc.
@@ -158,7 +156,7 @@ the setup in the service doc.
 ## API access (auth-gated)
 
 Every `/api/*` endpoint requires an authenticated session (SSO login JWT)
-or a per-user API key (`auth.enable_api_keys=true`; keys are minted per
+or a per-user API key (`auth.enable_api_keys=true`; keys are minted
 account under Profile → API Keys). Anonymous web visitors get the `pending`
 role (`ui.default_user_role=pending`), which cannot chat. Verified
 2026-10-07 from loopback `:3005` and over the public vhost:
@@ -168,13 +166,12 @@ bogus bearer token); only `/api/config`, `/api/version` and `/health`
 `POST /api/chat/completions` all return **401** with no key (also with a bogus bearer
 token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of
 them expose the model list. An external program needs an account + API key — there is
-no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`,
+no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`
 `/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and
 `POST /api/chat/completions` → 401; `/api/config` additionally publishes
 `features.enable_login_form=false`, `features.enable_signup=false` and
 `oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside.
 
-
 ## Web search
 
 Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
@@ -188,7 +185,7 @@ Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
   The Anubis→Authentik guard chain is a browser mechanism (proof-of-work +
   session flow); a plain HTTP client cannot solve it, and the internal port
   bypasses Caddy entirely.
-- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`,
+- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`
   a hardcoded Python constant — no config knob). The `pass_ip` list is checked
   FIRST and short-circuits the whole botdetection chain, and OpenWebUI's bridge
   egress (`10.89.x`) is in `pass_ip`, so the cap never applies. Verified: an
@@ -251,7 +248,6 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
   restore the password form: set `ui.enable_login_form` back to `true` in the
   Admin Panel (or DB) and drop the two env vars from the quadlet.
 
-
 ## Boot persistence (Quadlet)
 
 The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
@@ -292,6 +288,41 @@ carries base64 image data or a URL.
 Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
 on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
 
+## Text-to-speech (frank-tts)
+
+TTS is the fleet's one OpenAI-compatible speech endpoint —
+`http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url`
+in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That
+port is the rootless quadlet `frank-tts` (this repo:
+`dankclaw/voiceservice/quadlet/frank-tts.container`
+`scripts/frank-tts.env`), and its shim resolves `voice=<name>` against
+`/home/dank/voices/<name>/reference.wav` — a name without a reference is a
+400 listing the voices that exist, never a silent fallback.
+
+**The 29-voice `af_* … bm_fable` list is NOT frank-tts.** That enum belongs
+to **Kokoro.js running in the browser**: the deployed front end (0.11.4)
+ships an in-page Kokoro implementation (chunk carrying a frozen voice table
+`{af_heart:{name:"Heart",…}}` and `_validate_voice`, model
+`onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype*
+setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
+`browser-kokoro` is chosen the page synthesizes locally and rejects any
+voice outside its fixed pack — so `narrator1` fails with
+`Voice: "narrator1" not found. Should be one of: af_heart, …` **without a
+single request reaching the server**. Personal audio settings are
+browser: no `user.settings` row and no container env carries an audio
+override, yet the toast still appears ⇒ the choice lives in that browser's
+own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**,
+which hands synthesis to the admin config (frank-tts); picking `af_heart`
+instead keeps Kokoro, and Kokoro cannot clone — fixed pack only.
+
+Verified 2026-10-18 from inside the container with the exact OpenWebUI body
+(`{"model":"frank-tts","input":…,"voice":"narrator1",
+"response_format":"mp3","speed":1.0}`) → **200, `audio/mpeg`, 7,629 B**,
+while `podman logs open-webui` recorded **zero** `audio/speech` requests for
+the failing click — that contrast is the instrument: a server-path failure
+leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves
+neither.
+
 ## Retired instances
 
 - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
@@ -302,8 +333,8 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
 
 - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
 - The default admin account is created on first login; secure it immediately.
-- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
-- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
+- If the backend is loading, OpenWebUI shows connection errors until the endpoint is ready; each endpoint's boot budget is on its own page ([[frank-dragoon]], [[frank-zealot]], [[frank-archon]]).
+- Repeated from *The LLM backends*: the on-host endpoints are reached via the **host LAN IP**, never `host.containers.internal`.
 
 ## History
 
@@ -321,7 +352,7 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
   began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
   id-keyed objects had to move in the same window as the alias flip:
   `model.id`+`name` (`frank-zealot` → `frank-zealot`, `params`
-  `{"context":{"length":200704}}` and the `meta.description` kept),
+  `{"context":{"length":200704}}` and the `meta.description` kept)
   `access_grant` (`resource_type='model'`, `resource_id`, principal
   `user`/`*`, permission `read`), and `openai.api_configs` per-backend
   `model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1
@@ -349,6 +380,7 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
 ## See Also
 
 - [[Dankclaw Stack]]
+- [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)*
 - [[Hermes]] — the agent OpenWebUI can talk to
 - [[SearXNG]] — the search engine on this host
-- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
+- [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]] — the endpoints behind the picker

serverAppConfigs-docs/Dankclaw/Postgres.md

@@ -88,7 +88,7 @@ never enter this repo. See [[Gitea]] and [[Authentik]].
 
 ## Boot persistence (Quadlet)
 
-Both units carry `Restart=always` (`gitea-postgres.container:28`,
+Both units carry `Restart=always` (`gitea-postgres.container:28`
 `authentik-postgres.container:38`) and `[Install] WantedBy=default.target`
 (`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed
 under `~/.config/containers/systemd/`, and report `generated` / `active` with

serverAppConfigs-docs/Dankclaw/SearXNG Providers.md

@@ -143,7 +143,7 @@ engines:
     disabled: false
 ```
 
-Key per-engine keys (each overwrites the engine-module default; see
+Key per-engine keys (each overwrites the engine-module default
 `settings_engines.html`):
 
 | Key | Effect |
@@ -164,7 +164,7 @@ Key per-engine keys (each overwrites the engine-module default; see
 *different* category. When the primary `google` gets CAPTCHA'd it suspends, but
 `google2` still returns results, and vice-versa.
 
-**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
+**Parser caveat (read before fiddling with `parsers`):** the modern `google`
 `duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
 XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
 [scrapy_html]` under `name: google` mostly does nothing. To make Google
@@ -376,14 +376,14 @@ done
 
 **2. `/config` REST endpoint** lists every engine + enabled/disabled +
 categories — an inventory of what is live. It answers 200 on this instance today
-(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`,
+(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`
 `categories`, `timeout`, `shortcut`, and it is the source of the engine counts on
 [[SearXNG]]). `?engines=<name>` isolates a single
 provider; `/debug` adds engine diagnostics.
 
 **3. `check.searx.space`** — the hosted official checker probes every engine of
 your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
-plain HTTP probe misses) and posts results on searx.space. Add your instance via
+plain HTTP probe misses) and posts results on searx.space. Add your instance
 a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
 to reach the SearXNG port.
 
@@ -417,10 +417,10 @@ misleading).
    already replaced — checked 2026-10-18; see [[SearXNG]]). It is
    also immune to the shared-VPN-exit-IP blocking this page is about: it
    authenticates, so reputation of the exit IP is irrelevant.
-5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
+5. **Verify first** — `/json` + `?engines=google` and the searx-checker to
    exactly which engines are down and why. Confirm the running SearXNG version
    (check whether `parsers`/`rate_limit` exist in the pinned build).
-6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
+6. **Use conservative request tuning** — Dankclaw now allows 5 seconds
    upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds
    (`:27`), and retries once (`:32`). This addresses
    modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only

serverAppConfigs-docs/Dankclaw/SearXNG.md

@@ -108,7 +108,7 @@ engines:
     disabled: true
 ```
 
-The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas);
+The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas)
 the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked
 `secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is
 censused below.
@@ -154,7 +154,7 @@ what keeps the `trusted_proxies` entry meaningful.
 
 ## Unblocking Google & Search Providers
 
-The current request policy allows a modest amount of VPN latency (`5.0s` per
+The current request policy allows a modest amount of VPN latency (`5.0s`
 request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting
 retries to one (`:32`). This can recover slow
 providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
@@ -169,10 +169,10 @@ upstream request volume. SearXNG **suspends** offending engines — which is wha
 up as "Google went unavailable". The default suspension is shorter than the folklore:
 see the table in [[SearXNG Providers]]. This is upstream, not local.
 
-Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
+Fixes (cheapest first): tune the `outgoing` block (`request_timeout`
 `retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
 fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
-if Google still re-blocks — route the Google engines at a clean-IP proxy via
+if Google still re-blocks — route the Google engines at a clean-IP proxy
 per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
 root cause, the suspend-model table, the `engines:` config, and the
 `/json` verification recipe.
@@ -212,17 +212,16 @@ root cause, the suspend-model table, the `engines:` config, and the
   (`searxng.container:21`), and that is one of the variables SearXNG substitutes at
   load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a
   config change, out of scope for docs.
-- Keep the valkey cache container on the network SearXNG can actually reach — see
+- Keep the valkey cache container on the network SearXNG can actually reach —
   [[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's
   netns), so the network that matters is the one Gluetun and `searxng-valkey` share.
 
-
 ## Kagi (paid API engine)
 
 Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native
-`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`),
+`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`)
 which sets `Authorization: Bearer <key>` (`kagi.py:103`) and issues
-`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`),
+`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`)
 and declares `require_api_key: True` (`kagi.py:63`).
 
 - **Enabled**: `weight: 2` alongside bing/duckduckgo, categories
@@ -231,10 +230,10 @@ and declares `require_api_key: True` (`kagi.py:63`).
   `enabled: true`, categories `general, web` (2026-10-18).
 - **Cost control**: `paging: false` — exactly one API credit per aggregated
   query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0)
-  (`shared/searxng/settings.yml:74-77`; upstream paging default is `True`,
+  (`shared/searxng/settings.yml:74-77`; upstream paging default is `True`
   `kagi.py:67`, so the `false` here is a real override).
-- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`;
-  `deploy-remote.sh` bakes the real key into the live settings.yml via
+- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`
+  `deploy-remote.sh` bakes the real key into the live settings.yml
   `scripts/searxng-render-settings.py`, reading `/opt/searxng/.env`
   (owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are
   the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`).
@@ -262,15 +261,15 @@ and declares `require_api_key: True` (`kagi.py:63`).
 
 - 262 engines in the build, **75 enabled**, categories of the enabled set led by
   `other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7).
-- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`,
-  `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`,
-  `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`,
-  `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`,
+- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`
+  `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`
+  `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`
+  `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`
   `youtube`, `vimeo`, `dailymotion`, `bing videos`.
-- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`,
+- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`
   `brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`.
 - The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target
-  engines that **no longer exist in this build** — `/config` lists no `startpage`,
+  engines that **no longer exist in this build** — `/config` lists no `startpage`
   `startpage images` or `startpage news`, so those overrides are no-ops kept as
   intent. *(labelled 2026-10-18)*
 

serverAppConfigs-docs/Dankclaw/Seerr.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line)."
+description: "The request manager for the *arr stack — the maintained successor to the archived Overseerr line."
 tags:
   - dankclaw
   - seerr
@@ -40,8 +40,8 @@ the next outpost reload), and the missing app bounced post-login to
 authentik's own user UI instead of Seerr. Both backfilled live: the three
 standard flows copied from a known-good provider, plus an application
 (`seerr-guard`) linked to the provider. Verified by the hardened SSO sweep
-(`tests/e2e/specs/13-sso.spec.ts`) — see [[Authentik]] for the two
-provisioning rules this incident established.
+(`tests/e2e/specs/13-sso.spec.ts`) — see [[Authentik Outpost Reference]] for the
+two provisioning rules this incident established.
 
 ## *arr wiring
 

serverAppConfigs-docs/Dankclaw/Swap Configuration.md

@@ -1,7 +1,7 @@
 ---
-type: configuration
+type: reference
 title: Dankclaw Swap Configuration
-description: The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit. Covers why it was raised from 8 GiB, the safe resize ordering that avoids an OOM on a host whose MemAvailable is inflated by pinned model weights, and the swappiness caveat.
+description: "The host swapfile: why it is sized as it is, the resize ordering that avoids an OOM, and the swappiness caveat."
 tags:
   - dankclaw
   - host
@@ -36,14 +36,14 @@ it is not invisible.
 ## Why 64 GiB
 
 Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
-**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1`
-working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose
-working set was ~4.3 GiB. That is a decision-time reading, not today's: on
+**103.5 GiB of the 121.2 GiB** host, leaving roughly **7.1 GiB** for the then-28
+co-tenant containers whose working set was ~4.3 GiB. How that total is split
+between weights, KV pool and working memory is the engine's own ledger — it lives
+on [[frank-dragoon]]. That is a decision-time reading, not today's: on
 2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
 (`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
 `frank-dragoon-halogen` container accounts for **70.33 GiB**
-(`podman stats --no-stream`, same day) on the current flash-server payload —
-a different model and KV shape than the one measured in September.
+(`podman stats --no-stream`, same day).
 The engine itself warned at the time that when the headroom runs out,
 frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
 no disk activity and no output". Swap is the valve that turns an OOM kill into slow
@@ -59,7 +59,7 @@ like this, **never `swapoff` before the replacement is online.** The old 8 GiB
 held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was
 genuinely free — the pinned model weights are counted as reclaimable page
 cache, so the headline number is a lie by roughly the size of the model
-(see [[halogen-flash-server]]). Swapping off into that apparent space, with
+(see [[frank-dragoon]]). Swapping off into that apparent space, with
 nothing else to receive the pages, is how you invite the OOM killer.
 
 The ordering that avoids it — create and enable the new swap **first**, so the
@@ -113,10 +113,10 @@ frank-dragoon `health=200` throughout.
 
 ## Caveats
 
-- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are
-  held deliberately; paging them is a stall, not a relief. More swap does not
-  make the doubled KV pool affordable — see the reserve/pool ceiling in
-  [[halogen-flash-server]].
+- **Swap cannot help the pinned weights.** frank-dragoon's weights are pinned
+  deliberately; paging them is a stall, not a relief. More swap does not make a
+  larger KV pool affordable — see the reserve/pool ceiling in
+  [[frank-dragoon]].
 - **`vm.swappiness=60` is untested against this workload.** Under real
   pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
   pool and working set *are* anonymous. Swapping the KV pool would be far worse
@@ -144,13 +144,13 @@ frank-dragoon `health=200` throughout.
 The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at
 `dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging
 directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only
-`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`,
-`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`,
+`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`
+`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`
 `20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`.
 The old swapfile has therefore been purged; nothing on the host still holds it.
 
 ## See Also
 
-- [[halogen-flash-server]] — the memory model that motivated this
+- [[frank-dragoon]] — the memory model that motivated this
 - [[Dankclaw System Specs]] — the Strix Halo box
 - [[Dankclaw Improvements]] — other host-level changes

serverAppConfigs-docs/Dankclaw/Voice Packs.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material gate scripts/build-voice-pack.sh enforces, how to add a fourth voice, and what to do when a pinned engine bump changes the extractor's output sizes."
+description: "How a voice pack is built: the files every pack must carry, the source-material gate, adding a voice, engine-bump drift."
 tags:
   - dankclaw
   - frank-tts
@@ -23,7 +23,7 @@ voice is how a wrong-voice endpoint stays wrong for weeks.
 state that stays out of the repo, mounted read-only at `/voices` by the façade
 unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in
 `scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:**
-arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]].
+arm A `qwentts.cpp`, decided in .
 
 Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade
 container runs as uid 10000 and reads the mount from outside its own uid map.
@@ -137,7 +137,7 @@ invariant to every crop in the tracked pack list.
 1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop
    offset is a claim about one exact file, so the digest goes in the env with it.
 2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once:
-   `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`,
+   `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`
    build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest
    on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the
    `.rvq` falls under the floor.
@@ -188,15 +188,15 @@ loud, not a bug. Work through it in order:
 1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec`
    prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , …
    codebooks @ … Hz` on every run, and the pinned tree's
-   `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`;
+   `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`
    `codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`.
 2. Decide whether the new number is intended. A changed `enc_dim` means the old
    `.spk` files are unreadable by the new engine, so **every** pack must be
    rebuilt, not just the failing one.
 3. Update the four arithmetic lines in `scripts/voice-packs.env`
-   (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`,
+   (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`
    and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and**
-   `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`,
+   `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`
    `VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes
    without moving the commit is the mistake to avoid.
 4. Rebuild all three packs with `--force` (the retired directories keep the old
@@ -226,19 +226,19 @@ loud, not a bug. Work through it in order:
   narrator1 and narrator2). The builder's first real run is therefore a
   migration, and it should be a `--force` run per voice so the old directories
   are retired rather than left half-updated.
-- The three source files (`/home/blank/audiosample.mp3`,
+- The three source files (`/home/blank/audiosample.mp3`
   `audiosample2.m4b`, `audiosample3.m4b`, digests pinned in
   `scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the
   builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so
   they have to be copied there once before a build.
 - `VOICE_PACK_TOOL` points at the trial's build tree
   (`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
-  at exactly `5916bf7c`), and the two GGUFs at
-  `/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's
-  download script puts them — that directory does not exist yet, so until it
-  does the models have to come from the trial's `models/` directory.
-- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs
-  is possible on that host while serving them is not.
+  at exactly `5916bf7c`), and the two GGUFs live at
+  `/home/dank/models/Qwen3-TTS-GGUF/`, the engine slice's fetch target — that
+  path and its fetch state belong to [[Frank TTS Engine]].
+- Whether the façade (7772) and the engine (8880) are up and serving is deploy
+  state on [[Frank TTS]] / [[Frank TTS Engine]] — read it there, not from this
+  census.
 
 ## Known gaps, on purpose
 

serverAppConfigs-docs/Dankserver/Calibre.md

@@ -80,7 +80,7 @@ image's own nginx (the desktop), so the content server runs on internal 8090.
 **Auth model (calibre 9.x):** users live in a SQLite userdb whose `pw` column
 is **plaintext** — the file is mode 600. The single admin account is
 bootstrapped once by the deploy handler
-(`deploy_calibre_server` in `scripts/deploy-remote.sh`) via
+(`deploy_calibre_server` in `scripts/deploy-remote.sh`)
 `calibre-server --manage-users -- add`, the password piped on stdin with
 `printf` (a trailing newline from `echo` lands in the stored password and
 breaks every login). The desktop's basic auth (`CUSTOM_USER`/`CUSTOM_PASSWORD`)

serverAppConfigs-docs/Dankserver/Copyparty.md

@@ -83,7 +83,7 @@ The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum co
 
 ## Volumes
 
-Live deployment: the whole LUKS array is served at URL `/w` via
+Live deployment: the whole LUKS array is served at URL `/w`
 `Volume=/mnt/entropy:/w:ro` in the quadlet unit ([[Entropy Volume]]) —
 read-only **at the mount level**, so no containerized consumer can write into
 the array through this path regardless of account permissions. The ac edition
@@ -111,7 +111,7 @@ convention; see [[#Command-line interface (ac edition)]] above.
 ```
 
 `${COPAY_ACCOUNT}`/`${COPAY_PASSWORD}` are **never** written into the unit:
-they live in the host-local `/home/dank/Containers/copyparty/env` (0600),
+they live in the host-local `/home/dank/Containers/copyparty/env` (0600)
 loaded via `[Service] EnvironmentFile=` so systemd expands them into
 `Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD}` at start. The password still
 lands in the container argv — visible to anyone who can run
@@ -143,13 +143,13 @@ Copyparty v1.20.9+ supports WOPI (Web Application Open Platform Interface) for e
 
 ## Security Advisories
 
-| Version | CVE | Issue |
-|---------|-----|-------|
-| v1.20.19 | GHSA-phv8-wgjp-g4p9 | FTP upload outside volume |
-| v1.20.17 | GHSA-x5pq-m9p8-f4vx | dirkey/filekey confusion |
-| v1.20.9 | CVE-2026-27948 / GHSA-62cr-6wp5-q43h | XSS via file upload |
+| Version | Advisory | CVE | Issue |
+|---------|----------|-----|-------|
+| v1.20.19 | GHSA-phv8-wgjp-g4p9 | CVE-2026-104102 | FTP upload outside the defined volumes (limited path traversal) |
+| v1.20.17 | GHSA-x5pq-m9p8-f4vx | CVE-2026-70657 | dirkey/filekey confusion |
+| v1.20.9 | GHSA-62cr-6wp5-q43h | CVE-2026-27948 | reflected XSS via the `?setck=` URL parameter (not the upload path — the separate upload/DOM-XSS advisory is GHSA-m2jw-cj8v-937r) |
 
-Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`.
+Upgrade to v1.20.19+ to cover all known advisories. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`. Each row's patched version is the advisory's own `first_patched_version`.
 
 ## Gotchas
 
@@ -174,3 +174,10 @@ Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters
 - [[Dankserver Stack]]
 - [[Caddy]] — TLS termination and authentication in front
 - [[ARR-Stack]] — Copyparty as a viewer alongside automated download management
+
+## Citations
+
+- GitHub Security Advisories, `9001/copyparty` —
+  https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19)
+  each row's patched version read from `first_patched_version` in the
+  `https://api.github.com/advisories/<GHSA-id>` response.

serverAppConfigs-docs/Dankserver/Dankserver Stack.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "Dankserver (192.168.0.235) runs the media and library stack."
+description: "Service inventory and routing overview for the media and library host."
 tags:
   - dankserver
 timestamp: 2026-09-03
@@ -54,5 +54,5 @@ The following services ran on Dankserver and are now retired from this stack:
 
 ## See Also
 
-- [[index]] — repository index
+-  repository index
 - [[Network Architecture]] — routing and proxy

serverAppConfigs-docs/Dankserver/Jellyfin.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Jellyfin
-description: Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and hardware-transcode status.
+description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status."
 tags: [jellyfin, dankserver, media-server]
 timestamp: 2026-09-03T00:00:00Z
 ---
@@ -14,7 +14,7 @@ serving music, movies, and shows over LAN. It replaced the earlier research
 brief; it runs alongside [[Plex]], which remains the primary server for some
 clients.
 
-**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`,
+**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`
 public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain →
 [[Authentik]] outpost (post-auth route proxies `localhost:8096`).
 
@@ -67,7 +67,7 @@ user to verify the password.
 - LDAP provider `Jellyfin LDAP` (Base DN `dc=franksdank,dc=cloud`, bind/search
   mode `direct`) + application `Jellyfin`.
 - LDAP outpost `Jellyfin LDAP Outpost` → container `authentik-outpost-ldap`
-  (quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`),
+  (quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`)
   publishing `389`/`636` on the Dankclaw host for cross-host binds.
 - Service account `svc-jellyfin-ldap` (DN
   `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud`).
@@ -80,7 +80,7 @@ user to verify the password.
 **Jellyfin side** (plugin config, set via admin → Plugins → LDAP-Auth):
 - Server `192.168.0.143:389` (plain LDAP over the trusted LAN), Base DN
   `dc=franksdank,dc=cloud`, bind user = the service-account DN, search
-  attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`,
+  attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`
   `CreateUsersFromLdap=true`, all folders enabled.
 
 **Gotcha — the bind credential is a PASSWORD, not a token.** The LDAP bind is

serverAppConfigs-docs/Dankserver/Plex.md

@@ -30,7 +30,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
 | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
 | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
-| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` |
+| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to [[frank-archon]] |
 | Deploy | `./scripts/deploy.sh dankserver --service plex` |
 | Docs | https://support.plex.tv |
 
@@ -60,7 +60,7 @@ root runs as the `dank` user, and the deployer adopts the data root with
 ## Integration points
 
 - **Caddy:** Plex is exempt from the Anubis bot challenge
-  (plain-HTTP legacy clients cannot solve JS challenges) — see
+  (plain-HTTP legacy clients cannot solve JS challenges) —
   [[Anubis]] "Exempt by construction".
 - **Tautulli** ([[Tautulli]]) is the companion: it polls PMS on the appnet
   for statistics and notifications and makes no sense without Plex.

serverAppConfigs-docs/Dankserver/RomM-Scraping.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver."
+description: "Metadata providers, cover art, ROM naming and the scraping workflow behind the RomM library."
 tags:
   - dankserver
   - romm

serverAppConfigs-docs/Dankserver/RomM.md

@@ -32,7 +32,7 @@ Anubis → Authentik chain ([[Caddy]] routes it post-auth to
 ## Architecture
 
 Current RomM releases require MariaDB plus Redis (or Valkey) for the task
-queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit),
+queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit)
 not valkey.
 
 ```mermaid
@@ -114,7 +114,7 @@ scan:
 ## Deployment (Quadlet)
 
 Since 2026-09-01 the trio runs as repo-tracked quadlet units, deployed with
-`./scripts/deploy.sh dankserver --service romm` (any of `romm`,
+`./scripts/deploy.sh dankserver --service romm` (any of `romm`
 `romm-mariadb`, `romm-redis` triggers the full ordered cutover):
 
 - Units: `dankserver/romm/quadlet/{romm,romm-mariadb,romm-redis}.container`

serverAppConfigs-docs/Dankserver/mariadb.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: MariaDB (RomM Backend)
-description: MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and backup path.
+description: "The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path."
 tags: [mariadb, romm, dankserver]
 timestamp: 2026-09-03T00:00:00Z
 ---

serverAppConfigs-docs/Dankserver/redis.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Redis (RomM Cache / Queue)
-description: Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup path.
+description: "The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path."
 tags: [redis, romm, dankserver]
 timestamp: 2026-09-03T00:00:00Z
 ---

serverAppConfigs-docs/Desktop/Noctalia and Niri.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Noctalia and Niri
-description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning.
+description: "The noctalia shell on niri across workstations: config layers, remote IPC, custom palettes, output rotation and pinning."
 tags: [desktop, noctalia, niri, blankpad, workstation, ghostty]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -45,14 +45,14 @@ Verified: `msg status` returns live JSON state; unknown panel ids return an
 error listing all valid panel ids (clipboard, control-center, launcher,
 polkit, session, setup-wizard, test, tray-drawer, wallpaper).
 
-Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`,
-`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`,
-`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`,
+Verified v5 msg verb grammar (keybinds-safe): `panel-toggle <id>`
+`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`
+`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`
 `media next|previous|playPause` (graceful no-op, rc 0, with no active
-player), `brightness-up [target] [step]`, `brightness-down`,
-`settings-open [context]`, `settings-toggle [context]`, `config-reload`,
-`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`,
-`color-scheme-get|set`, `theme-mode-get`, `templates-apply`,
+player), `brightness-up [target] [step]`, `brightness-down`
+`settings-open [context]`, `settings-toggle [context]`, `config-reload`
+`wallpaper-get [connector]`, `wallpaper-set <path>`, `wallpaper-next`
+`color-scheme-get|set`, `theme-mode-get`, `templates-apply`
 `screenshot-fullscreen`.
 
 **Wallpaper directory semantics:** empty/unset `[wallpaper] directory`
@@ -81,9 +81,9 @@ Custom palettes live at `~/.config/noctalia/palettes/<Name>.json`
 synced. Missing/invalid file → silent fallback to the builtin palette.
 
 Palette JSON shape = the community file's shape: `dark`/`light` maps of
-`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`,
+`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`
 `mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block
-(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`,
+(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`
 `cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme
 -o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the
 *generator* format, not this one. Copy an existing community JSON and mutate
@@ -114,7 +114,7 @@ Procedure:
    `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with
    `color-scheme-get`.
 5. Re-render every configured template: `noctalia msg templates-apply`.
-   Regenerates `~/.config/ghostty/themes/noctalia`,
+   Regenerates `~/.config/ghostty/themes/noctalia`
    `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm,
    starship, etc. Symlinked-into-dotfiles paths show up as modified files
    worth committing.
@@ -216,7 +216,7 @@ Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel).
 scale, transform, logical pos/size) and `niri --version`. Output **names are
 connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings —
 the EDID strings printed before them are decoration. On blankpad the live
-config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`,
+config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`
 and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the
 repo path (that is what gets committed); check with `ls -ld`.
 

serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: "*arr Stack"
-description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every container on the gluetun WireGuard netns (all egress VPN): topology, wiring, auth posture, and gotchas."
+description: "The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas."
 tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver]
 timestamp: 2026-09-04T00:00:00Z
 ---
@@ -38,7 +38,7 @@ live call at a time instead of guessing payload shapes:
 - `'Base Url' must not be empty` when keys were camelCase → **PascalCase is
   required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys.
 - `DefaultMetadataProfileId must be > 0` → fetch real ids from
-  `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`);
+  `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`)
   `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠
   profiles.
 - `targetType null` → configContract not registered in that build (this
@@ -87,7 +87,7 @@ settings values live in `prowlarr.db` → `Applications.Settings` JSON (the
    pipeline hides a miss and you conclude "empty field" when the grep itself
    failed.
 4. Decision framing: when posture = gated-with-zero-users, present it as a
-   decision menu (None behind the front-door wall vs External/authentikarr),
+   decision menu (None behind the front-door wall vs External/authentikarr)
    not a silent change — first-account-wins means whoever logs in first owns
    the instance.
 

serverAppConfigs-docs/FrankBook-Pro/index.md

@@ -7,10 +7,10 @@ Gluetun WireGuard netns. The host overview lives at [[FrankBook-Pro]]
 
 ## *arr Stack
 
-- [[ARR-Stack]] — the full *arr stack: indexers, managers, and the download client
-- [[sonarr]] — TV show management
-- [[radarr]] — movie management
-- [[lidarr]] — music management
-- [[readarr]] — ebook/audiobook management
-- [[prowlarr]] — indexer management
-- [[qbittorrent]] — torrent download client
+- [[ARR-Stack]] — The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas.
+- [[sonarr]] — Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts.
+- [[radarr]] — Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts.
+- [[lidarr]] — Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts.
+- [[readarr]] — The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client.
+- [[prowlarr]] — Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas.
+- [[qbittorrent]] — The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe.

serverAppConfigs-docs/FrankBook-Pro/prowlarr.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Prowlarr
-description: Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and verified live gotchas.
+description: "Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas."
 tags: [prowlarr, arr-stack, frankbook-pro]
 timestamp: 2026-09-03T00:00:00Z
 ---
@@ -70,7 +70,7 @@ http://prowlarr:9696/{indexerId}/download?link=...&file=...   (torrent handoff)
 The global API key is in `config.xml` (`<ApiKey>`); per-application keys are
 only used by Prowlarr's own sync calls, NOT by the feeds. The arr download
 clients authenticate to qBittorrent with the permanent WebUI password stored
-in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — see
+in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver —
 [[qBittorrent]] for why that password must be pinned in the conf file.
 
 **Readarr caveat:** Readarr is abandoned upstream; its qBittorrent client
@@ -97,13 +97,13 @@ successor; the indexer row itself is healthy.
 - Categories in the indexer row must be a subset of what the feed's `caps`
   advertises; extras log "does not support the following categories" and
   some feeds reject them.
-- **A freshly created indexer row defaults to all-off.** If `enableRss`,
+- **A freshly created indexer row defaults to all-off.** If `enableRss`
   `enableAutomaticSearch`, or `enableInteractiveSearch` aren't set, the
   app logs `Searching indexers for [X]. 0 active indexers` (Lidarr,
   verified). Search proves TV + music chains 2026-08-30: Sonarr returned
   76 live releases for an anime term, Lidarr 18 for YOASOBI – THE BOOK,
   all via the Prowlarr-Nyaa feed.
-- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`);
+- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`)
   GET responses carry `fields: []` — real settings values live in
   `prowlarr.db` → `Applications.Settings` JSON (read-only inspection).
 - Instance name/`InstanceName=Prowlarr` in `config.xml` does NOT prefix the

serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: qBittorrent
-description: qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns, PBKDF2 WebUI credential recipe.
+description: "The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe."
 tags: [qbittorrent, arr-stack, frankbook-pro, gluetun]
 timestamp: 2026-09-03T00:00:00Z
 ---
@@ -13,7 +13,7 @@ qBittorrent is the torrent download client for the *arr stack. The stack runs on
 not on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr,
 and Readarr via category tagging.
 
-**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`,
+**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`
 published by the [[Gluetun]] unit because qBittorrent shares that container's
 network namespace (a netns-sharer publishes no ports of its own).
 
@@ -121,7 +121,7 @@ Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for
 `podman machine ssh`.
 
 **Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112
-(NOT core — `core` is the podman-machine VM user, reached only via
+(NOT core — `core` is the podman-machine VM user, reached only
 `podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped
 2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a
 low MaxAuthTries: when password-bootstrapping use `-o

serverAppConfigs-docs/FrankBook-Pro/readarr.md

@@ -1,18 +1,19 @@
 ---
 type: reference
 title: Readarr
-description: Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB download client.
+description: "The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client."
 tags: [readarr, arr-stack, frankbook-pro]
 timestamp: 2026-09-03T00:00:00Z
 ---
 
 # Readarr
 
-> **RETIRED UPSTREAM (May 2026).** The Servarr Team retired Readarr: its
-> metadata became unusable and the community migration to Open Library stalled.
-> There is no upstream development. Alternatives: the Bookshelf community fork,
-> or the Calibre content server. This instance still runs on a pinned image
-> tag; see below.
+> **RETIRED UPSTREAM (announced 2025-06-27).** The Servarr Team retired
+> Readarr: its metadata became unusable and the community migration to Open
+> Library stalled. There is no upstream development — the GitHub repository is
+> archived and its last push is 2025-06-27. Alternatives: the Bookshelf
+> community fork, or the Calibre content server. This instance still runs on a
+> pinned image tag; see below.
 
 Readarr manages ebooks and audiobooks in the *arr stack. The stack runs on
 **FrankBook-Pro** (the Mac mini) as quadlet units inside its podman machine —
@@ -69,9 +70,9 @@ Restart=always
 
 ## Gotchas
 
-- Readarr is abandoned upstream (retired May 2026; last line was 0.4.x before
-  the final tags). The pinned `amd64-1.0.980` tag exists precisely because
-  `:latest` does not exist for this image on LSIO.
+- Readarr is abandoned upstream (retired 2025-06-27; last line was 0.4.x
+  before the final tags). The pinned `amd64-1.0.980` tag exists precisely
+  because `:latest` does not exist for this image on LSIO.
 - Calibre integration is folder-based, not API-based. Readarr drops files into
   the books root which [[Calibre-Web]] serves.
 - Not all indexers carry all formats (epub, mobi, pdf). Goodreads/StoryGraph
@@ -81,3 +82,10 @@ Restart=always
 
 - [[ARR-Stack]] — full *arr stack documentation
 - [[Calibre-Web]] — ebook web frontend that serves Readarr downloads
+
+## Citations
+
+- Servarr wiki, *Readarr* (Retired) — "Announcement: Retirement of Readarr
+  2025-06-27" — https://wiki.servarr.com/readarr (accessed 2026-10-19)
+- `Readarr/Readarr` on GitHub: `archived: true`, last push 2025-06-27 —
+  https://api.github.com/repos/Readarr/Readarr (accessed 2026-10-19)

serverAppConfigs-docs/Infrastructure/Alertmanager.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through..."
+description: "Receives firing rules from Prometheus, groups and dedupes them, and forwards each group to the host-side Signal relay."
 tags:
   - infrastructure
 timestamp: 2026-09-17

serverAppConfigs-docs/Infrastructure/Anubis.md

@@ -93,7 +93,7 @@ and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
 (`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
 (`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
 it is exempt by absence — and therefore unauthenticated on the LAN at
-`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
+`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`
 `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
 (`dankclaw/Caddyfile:283-423`).
 
@@ -140,7 +140,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
    instead of allowed, the browser's fetch throws and it renders its own
    error page (iOS/macOS browsers phrase it as "the request failed and the
    interceptors did not return an alternative response") instead of the
-   login flow. Verify with a JS-capable client end-to-end — mint via
+   login flow. Verify with a JS-capable client end-to-end — mint
    `make-challenge`, solve, submit with browser-exact query params — before
    declaring a challenge chain healthy.
 7. **Single-file bind mount + git pull = stale policy.** The policy is
@@ -158,7 +158,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at
    answers any request lacking `X-Real-Ip` with a 500 whose log line reads
    `[misconfiguration] X-Real-Ip header is not set`. The real chain always
    sets it (every guarded Caddy listener carries `header_up X-Real-Ip
-   {remote_host}` — the bats seam enforces the line), so users never see
+   {remote_host}` — the bats seam enforces the line), so users never
    this. But any direct probe of `:8923` (scratch container, curl, a new
    reverse-proxy hop) MUST set the header or it looks like a policy failure.
    Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`.

serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md

@@ -0,0 +1,230 @@
+---
+type: reference
+description: "Outpost, proxy-provider, OAuth2, LDAP and flow configuration for the homelab's Authentik SSO."
+tags:
+  - infrastructure
+  - authentik
+timestamp: 2026-10-19
+---
+
+# Authentik Outpost Reference
+
+Configuration lookup for the [[Authentik]] deployment: the live proxy-provider
+inventory behind the guarded vhosts, the provisioning rules the embedded outpost
+enforces, this fork's OAuth2 relying-party quirks, the LDAP provider objects, the
+self-serve user-settings flow, and the live database shape. What the service is,
+how it is deployed, wired and operated is [[Authentik]]; the three core
+containers are [[Authentik Processes]].
+
+## Live proxy-provider inventory
+
+Live provider inventory (read from the `authentik-postgres` container with
+read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
+`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
+members in `authentik_outposts_outpost_providers`), all with
+`internal_host = http://caddy:8017`, all carrying the **same three flow
+UUIDs** (`default-authentication-flow`
+`default-provider-authorization-implicit-consent`
+`default-provider-invalidation-flow`), and each backed by an Application
+(the `slug` column below):
+
+| Provider pk | name | external_host | app slug |
+|---|---|---|---|
+| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
+| 2 | search-proxy | `https://search.franksdank.cloud` | search |
+| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
+| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
+| 6 | files-proxy | `https://files.franksdank.cloud` | files |
+| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
+| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
+| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
+| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |
+
+That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
+the lockstep rule in [[Authentik]] holds today. Two corrections against the 2026-09-02 table:
+**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
+**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
+returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
+exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
+not proxy providers, so the proxy outpost does not deserialize them: pk 9
+`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
+(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
+below rejects), and pk 14 `Open WebUI`
+(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
+`external_host` (the runbook-recipe style) and bare origins both intercept
+correctly — the outpost matches on hostname.
+
+## Provisioning rules for proxy providers
+
+Two provisioning rules, each verified live by an outage:
+
+1. **All three flows are required.** A ProxyProvider created via raw API/ORM
+   with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
+   null wedges the embedded outpost's config fetch: the Rust side rejects the
+   whole config JSON (serde: `invalid type: null, expected a formatted UUID
+   string`), logs `Failed to fetch outpost from API, retrying in 3 seconds`
+   forever, and **everything behind the outpost dies at the next outpost
+   reload** — including hosts whose providers were fine (verified 2026-09-02
+   when three providers provisioned without any flows took down every guarded
+   vhost after a server restart). Backfill all three flows (copy from pk 1:
+   `default-authentication-flow`
+   `default-provider-authorization-implicit-consent` (the implicit-consent
+   authorization flow, `default-invalidation-flow`) and the retry loop
+   recovers on its next 3-second cycle. The silent killer: while the outpost
+   process keeps its last good in-memory config, everything works — the
+   outage only surfaces on restart/reload.
+2. **Every proxy provider needs an attached Application.** The outpost
+   resolves the post-login redirect through the provider's application; a
+   provider with no app completes the login (authentik session established)
+   but bounces to authentik's own user UI (`/if/user/#/library`) instead of
+   the app. No error is logged anywhere — the SSO "works" and the app never
+   appears. Create the Application with `provider_id` set (the UI does this
+   automatically; raw API/ORM drives must do it explicitly).
+
+## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
+
+The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
+verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
+implementation diverges from upstream in ways that bit us; all verified live:
+
+- **Application row required.** `OAuth2Provider.objects.create()` does not
+  create the single-table-inheritance `Application` row; without it the flow
+  executor can never complete an authorize request. Create both rows.
+- **`grant_types` defaults to empty.** Until the provider's `grant_types`
+  contains `"authorization_code"` (enum value with underscore), the token
+  endpoint rejects every exchange with `invalid_request`.
+- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
+  `parse_token_request` see no `grant_type` → `unsupported_grant_type`
+  POST-body client secrets are not accepted — authenticate the confidential
+  client with `Authorization: Basic base64(client_id:client_secret)`.
+- **Provider resolution is by `client_id` query param** on the global
+  `/application/o/authorize/` route; per-slug authorize paths do not exist in
+  this build. Token/userinfo live at `/application/o/token/` and
+  `/application/o/userinfo/?api=1`.
+- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
+  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
+  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
+  becomes the *expression source* instead of the context; every call raises a
+  swallowed `PropertyMappingExpressionException` and the claim is skipped.
+  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
+  provider FK (linkage lives in the `ProviderPropertyMapping` through model)
+  so `filter(provider=...)` always yields an empty queryset and tokens asked
+  to carry scopes get zero. Net effect: userinfo/introspection answers with
+  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
+  build until the evaluator is patched or upgraded. Clients consuming this
+  provider must key identity off `sub`.
+- Scope mappings for this provider exist — `grafana-email-scope` and
+  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
+  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
+  the evaluator bug is fixed; no config change is needed then.
+
+## LDAP provider objects (provisioned via `ak shell` ORM)
+
+The LDAP outpost's own deployment — unit, image digest, published ports, token
+env var — is in [[Authentik]]. The objects it serves:
+
+Read from the live database (read-only `SELECT`s against the
+`authentik-postgres` container, 2026-10-18):
+
+| Object | Value |
+|---|---|
+| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
+| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
+| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — an earlier revision's "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
+| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — an earlier revision recorded `intent=verification`; the live value is `app_password` |
+| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
+| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
+| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (an earlier revision said slug `jellyfin`, which does not exist) |
+
+Hard-won rules (the first is now contested by the live database):
+
+- **Token intent: contested, do not trust either value blindly.** An earlier
+  revision asserted that a bind token with `intent=app_password` is rejected by
+  the flow's password stage with `Invalid password` and only
+  `intent=verification` passes. The live token for `svc-jellyfin-ldap` is
+  `app_password` (`authentik_core_token`, 2026-10-18), so either the rule is
+  wrong or the current Jellyfin bind path is not that token. Nothing observable
+  settles it today: the LDAP outpost's retained log holds only its 5-minute
+  `initialized direct binder` / `Update providers` config cycles and no bind
+  activity *(unverified 2026-10-18: no bind was performed — reproducing it
+  requires the bind secret)*. Re-verify with a real bind before changing the
+  token's intent either way.
+- **Search needs `search_full_directory`.** Without the RBAC grant the
+  outpost returns only the bind user itself — a `(memberOf=…)` filter then
+  matches zero entries and the plugin logs `Found no users matching X`.
+  The permission is object-level on the provider; grant it via an RBAC role
+  and retest.
+- `ak shell` import path for the provider model in 2026.8:
+  `from authentik.providers.ldap.models import LDAPProvider`.
+  - (unverified 2026-10-18: importing it outside a configured Django shell fails
+  with `ImproperlyConfigured`, and no `ak shell` was driven against the live
+  auth service for this check)*
+
+## Self-serve user-settings flow (password change)
+
+Users change their own account details through the `default-user-settings-flow`
+(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the
+target of the homepage footer's "account" link. It is a `stage_configuration`
+designated flow, so it renders every bound `ConfigurableStage` for the logged-in
+user.
+
+Stages, in execution order (the `FlowStageBinding.order` field, not the M2M
+insertion order):
+
+| order | stage | type | what it does |
+|---|---|---|---|
+| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** |
+| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user |
+
+**Password change is a prompt field, not a stage.** The `UserWriteStage`
+writes whatever the preceding `PromptStage` collected: for the key `password`
+it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix
+is a `password` field on the `default-user-settings` prompt
+(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a
+`PasswordStage`. A `PasswordStage` is an *authentication* stage: its
+`validate_password` runs `authenticate()` against the backends to check the
+*current* password; it never sets a new one. (A `PasswordStage` was added on
+2026-09-25 and removed the same day for exactly this reason.)
+
+Two ORM gotchas: `Flow.stages.add()` fails on this through model
+(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
+MTI), so create the `FlowStageBinding` row directly with an explicit `order`
+and a prompt field is a `Prompt` row linked through `PromptStageField`
+(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).
+
+**A blank password must not set an empty password.** `set_password("")` would
+make the account log in with no password, so a validation policy
+(`default-user-settings-no-empty-password`, bound to the prompt
+`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
+is `""` and allows it when set. Verified with the `PolicyEvaluator`
+(empty → deny, set → allow, absent → allow) and end-to-end by driving
+`UserWriteStageView.update_user` on a throwaway user (new password valid, old
+invalid).
+
+## Live database shape
+
+For live DB truth prefer the postgres container, and it works without any
+credential — the local socket is trusted, so a read-only audit is just
+`podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
+with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
+before guessing: 2026.x renamed both tables and columns, and the models are
+MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
+three `*_flow_id` columns; the proxy table is
+`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
+really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
+table in 2026.8), the LDAP table is
+`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`
+`authentik_core_application` attaches to a provider through `provider_id`
+(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
+is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
+is the same story from the ORM side: `type` lives in the subclass tables.
+Never select secret columns (`client_secret`, `key`, `password`) into a log.
+
+## Cross-references
+
+- [[Authentik]] — what the service is, its deployment, wiring, operations and
+  failure modes
+- [[Authentik Processes]] — the postgres, server and worker containers
+- [[Caddy]] — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners
+- [[Jellyfin]] — consumes the LDAP outpost via its LDAP-Auth plugin
+- [[Grafana]] — guarded vhost *and* OAuth2 relying party of this provider

serverAppConfigs-docs/Infrastructure/Authentik Processes.md

@@ -0,0 +1,76 @@
+---
+type: reference
+description: "The three core Authentik containers — database, server and worker — with their units, images, roles and probes."
+tags:
+  - infrastructure
+  - authentik
+  - postgres
+timestamp: 2026-10-19
+---
+
+# Authentik Processes
+
+The three core containers of the [[Authentik]] stack on Dankclaw. The two
+outpost units are not covered here — they live on [[Authentik]] and
+[[Authentik Outpost Reference]]. Server and worker run the same image; the
+`command` selects the role (`server` / `worker`). Data persists under
+`/home/dank/Containers/authentik/` as bind mounts (repo policy — no named
+volumes), moved from `/home/dank/authentik/` in the 2026-09-01 quadlet
+standardization; the retired stubs this page replaces recorded the pre-move
+paths `/home/dank/authentik/data` (server and worker) and
+`/home/dank/authentik/postgres` (database).
+
+## authentik-postgres
+
+The PostgreSQL 16 database for Authentik — the only stateful store in the stack
+(users, flows, sessions, events). Sessions, caching, and WebSockets moved to
+PostgreSQL in 2025.10 (Redis removed).
+
+- **Image:** `docker.io/library/postgres:16-alpine`
+- **Data:** `/home/dank/Containers/authentik/postgres` (bind mount; no named
+  volumes)
+- **Health:** `pg_isready -d authentik -U authentik`
+- **Retired wording:** the stub called this a "Compose service … in
+  `dankclaw/compose.yaml`"; the compose stack is gone and the unit is a quadlet.
+- The side-by-side with `gitea-postgres` (network, published ports, secret
+  source, boot persistence) is owned by [[Postgres]].
+
+## authentik-server
+
+The Authentik server — the core, the admin API, and the embedded Caddy outpost.
+It also serves the login page at `https://auth.franksdank.cloud`.
+
+- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container`
+  (`authentik-server.service`)
+- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (`command: server`)
+- **Port:** `9000` (published) — the outpost, the admin API and the login UI all
+  live here
+- **Data:** `/home/dank/Containers/authentik/data` (bind mount, shared with the
+  worker)
+- **Health:** recorded by the retired stub as
+  `http://localhost:9000/-/health/`; superseded — [[Authentik]] → Operations
+  owns the live probe.
+- **Retired wording:** the stub described "the Caddy `forward_auth` vhosts"
+  proxying here; the chain Caddy actually runs is [[Authentik]] → Caddy
+  integration.
+
+## authentik-worker
+
+The Authentik worker — background tasks: event processing, provider
+synchronization, scheduled flows, and certificate renewal.
+
+- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container`
+  (`authentik-worker.service`)
+- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (`command: worker`)
+- **Runs as:** `0:0` (rootless: container root maps to the host user that owns
+  the data dir)
+- **Data:** `/home/dank/Containers/authentik/data` (bind mount, shared with the
+  server)
+- **No port** — the worker has no HTTP endpoint.
+
+## Cross-references
+
+- [[Authentik]] — what the service is, its deployment, wiring, operations and
+  failure modes
+- [[Authentik Outpost Reference]] — provider, flow and token configuration
+- [[Postgres]] — both PostgreSQL clusters on Dankclaw, side by side

serverAppConfigs-docs/Infrastructure/Authentik.md

@@ -15,26 +15,33 @@ application-level authentication with its proxy outpost. Not wrapped: Gitea,
 OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
 they handle their own session. Grafana is the exception that is both: it is a
 guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
-relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
-`dankclaw/Caddyfile:374-377`; live provider table below).
+relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`
+`dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]).
 
-- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
-  the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
+- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`
+  the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`
   2026-10-18)
 - **Host:** dankclaw (`192.168.0.143`)
 - **Name:** `auth.franksdank.cloud`
-- **Stack: five quadlet units, no compose.** `authentik-postgres`,
-  `authentik-server`, `authentik-worker` (`deploy_authentik`,
+- **Stack: five quadlet units, no compose.** `authentik-postgres`
+  `authentik-server`, `authentik-worker` (`deploy_authentik`
   `scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
-  (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
+  (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`
   `deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
-  `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
+  `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`
   `scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
   `active`, `Restart=always`, `WantedBy=default.target`, with linger on
   (`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
   pre-quadlet era, retired.
 - **LDAP outpost:** serves LDAP for Jellyfin — see the section below.
 
+**Split off from this page:** [[Authentik Processes]] owns the per-container
+facts for `authentik-postgres`, `authentik-server` and `authentik-worker`
+[[Authentik Outpost Reference]] owns the provider/flow/token lookup material —
+the live proxy-provider inventory, the proxy-provider provisioning rules, the
+OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow
+design, and the live database shape.
+
 ## Why Authentik
 
 The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was
@@ -100,23 +107,10 @@ contract:
   The chain terminates there — nothing dials back into Authentik from
   `:8017`.
 
-## The three compose services
+## The stack's processes
 
-| Service | Role | Image |
-|---|---|---|
-| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
-| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.3` |
-| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.3` |
-
-- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
-  moved to PostgreSQL in 2025.10 (Redis removed).
-- **Server** and **worker** run the same image; the `command` selects the role
-  (`server` / `worker`).
-- The worker runs as `0:0` (rootless: container root maps to the host user that
-  owns the data dir).
-- Data persists under `/home/dank/Containers/authentik/{data,postgres}`
-  (bind mounts, repo policy — no named volumes; moved from `/home/dank/authentik/`
-  in the 2026-09-01 quadlet standardization).
+Per-container facts — unit path, image and role, port, run-as, data directory,
+health probe — are owned by [[Authentik Processes]].
 
 ## Caddy integration
 
@@ -142,110 +136,15 @@ routing table, and the Authentik proxy-provider list must be the same set of
 hosts. A guarded host with no provider 404s at login time; a provider with no
 DNS record breaks off-LAN clients.
 
-Live provider inventory (read from the `authentik-postgres` container with
-read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
-`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
-members in `authentik_outposts_outpost_providers`), all with
-`internal_host = http://caddy:8017`, all carrying the **same three flow
-UUIDs** (`default-authentication-flow`,
-`default-provider-authorization-implicit-consent`,
-`default-provider-invalidation-flow`), and each backed by an Application
-(the `slug` column below):
-
-| Provider pk | name | external_host | app slug |
-|---|---|---|---|
-| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy |
-| 2 | search-proxy | `https://search.franksdank.cloud` | search |
-| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus |
-| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm |
-| 6 | files-proxy | `https://files.franksdank.cloud` | files |
-| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
-| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
-| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
-| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |
-
-That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
-the lockstep rule holds today. Two corrections against the 2026-09-02 table:
-**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
-**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
-returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
-exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
-not proxy providers, so the proxy outpost does not deserialize them: pk 9
-`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
-(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
-below rejects), and pk 14 `Open WebUI`
-(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
-`external_host` (the runbook-recipe style) and bare origins both intercept
-correctly — the outpost matches on hostname.
-
-Two provisioning rules, each verified live by an outage:
-
-1. **All three flows are required.** A ProxyProvider created via raw API/ORM
-   with any of `authentication_flow`/`authorization_flow`/`invalidation_flow`
-   null wedges the embedded outpost's config fetch: the Rust side rejects the
-   whole config JSON (serde: `invalid type: null, expected a formatted UUID
-   string`), logs `Failed to fetch outpost from API, retrying in 3 seconds`
-   forever, and **everything behind the outpost dies at the next outpost
-   reload** — including hosts whose providers were fine (verified 2026-09-02
-   when three providers provisioned without any flows took down every guarded
-   vhost after a server restart). Backfill all three flows (copy from pk 1:
-   `default-authentication-flow`,
-   `default-provider-authorization-implicit-consent` (the implicit-consent
-   authorization flow, `default-invalidation-flow`) and the retry loop
-   recovers on its next 3-second cycle. The silent killer: while the outpost
-   process keeps its last good in-memory config, everything works — the
-   outage only surfaces on restart/reload.
-2. **Every proxy provider needs an attached Application.** The outpost
-   resolves the post-login redirect through the provider's application; a
-   provider with no app completes the login (authentik session established)
-   but bounces to authentik's own user UI (`/if/user/#/library`) instead of
-   the app. No error is logged anywhere — the SSO "works" and the app never
-   appears. Create the Application with `provider_id` set (the UI does this
-   automatically; raw API/ORM drives must do it explicitly).
-
-
-## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
-
-The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
-verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
-implementation diverges from upstream in ways that bit us; all verified live:
-
-- **Application row required.** `OAuth2Provider.objects.create()` does not
-  create the single-table-inheritance `Application` row; without it the flow
-  executor can never complete an authorize request. Create both rows.
-- **`grant_types` defaults to empty.** Until the provider's `grant_types`
-  contains `"authorization_code"` (enum value with underscore), the token
-  endpoint rejects every exchange with `invalid_request`.
-- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes
-  `parse_token_request` see no `grant_type` → `unsupported_grant_type`;
-  POST-body client secrets are not accepted — authenticate the confidential
-  client with `Authorization: Basic base64(client_id:client_secret)`.
-- **Provider resolution is by `client_id` query param** on the global
-  `/application/o/authorize/` route; per-slug authorize paths do not exist in
-  this build. Token/userinfo live at `/application/o/token/` and
-  `/application/o/userinfo/?api=1`.
-- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
-  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
-  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
-  becomes the *expression source* instead of the context; every call raises a
-  swallowed `PropertyMappingExpressionException` and the claim is skipped.
-  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
-  provider FK (linkage lives in the `ProviderPropertyMapping` through model),
-  so `filter(provider=...)` always yields an empty queryset and tokens asked
-  to carry scopes get zero. Net effect: userinfo/introspection answers with
-  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
-  build until the evaluator is patched or upgraded. Clients consuming this
-  provider must key identity off `sub`.
-- Scope mappings for this provider exist — `grafana-email-scope` and
-  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
-  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
-  the evaluator bug is fixed; no config change is needed then.
+The live proxy-provider inventory (the per-vhost `external_host` → app mapping)
+the two provisioning rules the outpost enforces, and this fork's OAuth2
+relying-party quirks are reference lookup: [[Authentik Outpost Reference]].
 
 ## LDAP outpost (Jellyfin auth, live 2026-09-01)
 
 The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
 → `authentik-outpost-ldap.service`, one of the five units in this stack —
-serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
+serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`
 `Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
 single-platform digest**
 `ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
@@ -260,51 +159,16 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
 .digest'` and pin that digest.
 
 - Ports **389** (ldap) and **636** (ldaps) publish on the LAN
-  (`PublishPort=389:3389`, `636:6636`,
+  (`PublishPort=389:3389`, `636:6636`
   `dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
   shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
   them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
   (2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN`
   from the host `dankclaw/.env` (git-ignored).
 
-### Authentik-side objects (provisioned via `ak shell` ORM)
-
-Read from the live database (read-only `SELECT`s against the
-`authentik-postgres` container, 2026-10-18):
-
-| Object | Value |
-|---|---|
-| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
-| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
-| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
-| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — the page previously recorded `intent=verification`; the live value is `app_password` |
-| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
-| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
-| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (the page previously said slug `jellyfin`, which does not exist) |
-
-Hard-won rules (the first is now contested by the live database):
-
-- **Token intent: contested, do not trust either value blindly.** This page
-  long asserted that a bind token with `intent=app_password` is rejected by the
-  flow's password stage with `Invalid password` and only `intent=verification`
-  passes. The live token for `svc-jellyfin-ldap` is `app_password`
-  (`authentik_core_token`, 2026-10-18), so either the rule is wrong or the
-  current Jellyfin bind path is not that token. Nothing observable settles it
-  today: the LDAP outpost's retained log holds only its 5-minute
-  `initialized direct binder` / `Update providers` config cycles and no bind
-  activity *(unverified 2026-10-18: no bind was performed — reproducing it
-  requires the bind secret)*. Re-verify with a real bind before changing the
-  token's intent either way.
-- **Search needs `search_full_directory`.** Without the RBAC grant the
-  outpost returns only the bind user itself — a `(memberOf=…)` filter then
-  matches zero entries and the plugin logs `Found no users matching X`.
-  The permission is object-level on the provider; grant it via an RBAC role
-  and retest.
-- `ak shell` import path for the provider model in 2026.8:
-  `from authentik.providers.ldap.models import LDAPProvider`.
-  *(unverified 2026-10-18: importing it outside a configured Django shell fails
-  with `ImproperlyConfigured`, and no `ak shell` was driven against the live
-  auth service for this check)*
+The objects behind it — LDAP provider, bind DN, service account and bind token,
+RBAC grant, groups, application slug — and the hard-won bind rules are in
+[[Authentik Outpost Reference]].
 
 ## Configuration (secrets)
 
@@ -332,7 +196,7 @@ every outpost session (see the ORM playbook).
    core quadlets, starts postgres health-gated (`wait_container_healthy
    authentik-postgres 120`), then the server (300 s health wait) and the worker.
    The outposts are separate handlers: `--service authentik-outpost` and
-   `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
+   `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`
    dispatch at `:1579-1588`).
 2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
    creates the superuser (email + password) and generates the DB schema.
@@ -340,12 +204,13 @@ every outpost session (see the ORM playbook).
    Applications** → add an application + a **Proxy** provider whose
    `external_host` is exactly the browser-facing origin
    (`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
-   flows set (see the provisioning pitfall above). Attach the provider to the
+   flows set (see the provisioning rules in
+   [[Authentik Outpost Reference]]). Attach the provider to the
    embedded outpost.
 4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
    the app with the correct redirect URI, and paste the `client_id` /
    `client_secret` into the app's SSO settings. The relying parties that exist
-   today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
+   today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14)
    **Gitea has no provider row at all** (live provider table, 2026-10-18) — it
    authenticates itself and stays off the chain by design
    (`dankclaw/Caddyfile:110-120`).
@@ -373,53 +238,12 @@ every outpost session (see the ORM playbook).
   the LDAP section) — bumping only the tag would leave the LDAP outpost behind.
   Authentik migrates the schema on startup.
 
-## Self-serve user-settings flow (password change)
-
-Users change their own account details through the `default-user-settings-flow`
-(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the
-target of the homepage footer's "account" link. It is a `stage_configuration`
-designated flow, so it renders every bound `ConfigurableStage` for the logged-in
-user.
-
-Stages, in execution order (the `FlowStageBinding.order` field, not the M2M
-insertion order):
-
-| order | stage | type | what it does |
-|---|---|---|---|
-| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** |
-| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user |
-
-**Password change is a prompt field, not a stage.** The `UserWriteStage`
-writes whatever the preceding `PromptStage` collected: for the key `password`
-it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix
-is a `password` field on the `default-user-settings` prompt
-(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a
-`PasswordStage`. A `PasswordStage` is an *authentication* stage: its
-`validate_password` runs `authenticate()` against the backends to check the
-*current* password; it never sets a new one. (A `PasswordStage` was added on
-2026-09-25 and removed the same day for exactly this reason.)
-
-Two ORM gotchas: `Flow.stages.add()` fails on this through model
-(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is
-MTI), so create the `FlowStageBinding` row directly with an explicit `order`;
-and a prompt field is a `Prompt` row linked through `PromptStageField`
-(`prompt_stage` + `prompt`; the order comes from `Prompt.order`).
-
-**A blank password must not set an empty password.** `set_password("")` would
-make the account log in with no password, so a validation policy
-(`default-user-settings-no-empty-password`, bound to the prompt via
-`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]`
-is `""` and allows it when set. Verified with the `PolicyEvaluator`
-(empty → deny, set → allow, absent → allow) and end-to-end by driving
-`UserWriteStageView.update_user` on a throwaway user (new password valid, old
-invalid).
-
 ## Outpost claim cache (`proxysession`) — 403 after a group add
 
 The embedded outpost does not re-evaluate group membership per request. Each
 completed authorize writes one row to `authentik_providers_proxy_proxysession`
 whose `session_data` JSON holds the claims (`groups` included) that get served
-for that browser session until the row expires (observed TTLs 1–24 h per
+for that browser session until the row expires (observed TTLs 1–24 h
 row). A session authorized **before** a user was added to a group keeps
 serving the old group list — Caddy-side RBAC header gates then 403 despite
 correct membership, and no authentik event is generated (the 403 happens at
@@ -457,7 +281,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
    (the inventory table carries both bare origins and trailing slashes;
    routing matches on hostname, verified 2026-09-02).
    `internal_host = http://caddy:8017`, all three flows set, and an
-   Application attached — the two provisioning rules above.
+   Application attached — the two provisioning rules in [[Authentik Outpost Reference]].
 2. **Route flip in ONE commit/deploy**: the vhost becomes
    `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
    inside the `http://:8017` listener add `@<svc> host <vhost>` +
@@ -485,7 +309,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
 000 (connection refused) — the post-auth listener lives in the caddy
 container's netns; probe inside it or grep the config. Non-browser clients
 cannot solve the JS challenge — before flipping, audit service-to-service
-traffic: ingestion/push paths must target internal addresses (`loki:3100`),
+traffic: ingestion/push paths must target internal addresses (`loki:3100`)
 never a public hostname of a guarded service. Exempt-by-construction
 categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
 endpoints, services with their own complete auth — document the exemption
@@ -512,31 +336,18 @@ EOF
   `sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
   the exact banner text and the 229-object count — no `ak shell` was driven
   against the live auth service for this audit)*.
-- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
+- Import paths (each wrong guess costs a failed exec): `User`, `Group`
   `Application`, `ApplicationEntitlement`, `PropertyMapping` →
   `authentik.core.models`; `ProxyProvider` →
-  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
-  `LDAPProvider` → `authentik.providers.ldap.models`;
+  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`)
+  `LDAPProvider` → `authentik.providers.ldap.models`
   `authentik.core.models.application` does NOT exist as an import path; there
   is no `authentik.policies.group` module in this build.
-  *(unverified 2026-10-18: these module paths were not re-executed — importing
+  - (unverified 2026-10-18: these module paths were not re-executed — importing
   outside a configured Django shell fails with `ImproperlyConfigured`)*
-- For live DB truth prefer the postgres container, and it works without any
-  credential — the local socket is trusted, so a read-only audit is just
-  `podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
-  with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
-  before guessing: 2026.x renamed both tables and columns, and the models are
-  MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
-  three `*_flow_id` columns; the proxy table is
-  `authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
-  really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
-  table in 2026.8), the LDAP table is
-  `authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
-  `authentik_core_application` attaches to a provider through `provider_id`
-  (its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
-  is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
-  is the same story from the ORM side: `type` lives in the subclass tables.
-  Never select secret columns (`client_secret`, `key`, `password`) into a log.
+- The live database's table and column names (the 2026.x renames, the MTI parent
+  keys, the read-only `psql` audit recipe) are reference lookup:
+  [[Authentik Outpost Reference]].
 
 ## User provisioning playbook (user-level account)
 
@@ -565,7 +376,7 @@ downstream —
   `category: user` panels from `services.json`.
 - `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
   **"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
-  `Open WebUI` application (`authentik_core_applicationentitlement`,
+  `Open WebUI` application (`authentik_core_applicationentitlement`
   2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
   claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
   expression were not read; the mapping itself is not visible under that name
@@ -581,7 +392,7 @@ downstream —
   artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
   and every one carries the `<user>@franksdank.cloud` address the template
   prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
-  *(unverified 2026-10-18)*. Template: `username` +
+  - (unverified 2026-10-18)*. Template: `username` +
   `<user>@franksdank.cloud` + display name + `is_active=True` + the password
   set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
   `openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
@@ -589,7 +400,7 @@ downstream —
   the row carried no MFA device, token, entitlement, or proxy session, and
   events survive a user delete because `Event.user` is a denormalized JSON
   field, not a FK).
-- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
+- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`
   `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
 - Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
   Plex, Seerr, Marreta) need no per-user group: any authenticated user
@@ -618,8 +429,8 @@ More field gotchas (each cost a failed exec): `ApplicationEntitlement.applicatio
 can be null (guard with `getattr`); `LDAPProvider` has
 `search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
 `organization` — the backing table
-(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
-`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
+(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`
+`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`
 `uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
 (`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
 (scopes are linked through `ProviderPropertyMapping`, which is why
@@ -673,5 +484,5 @@ podman run --rm --network container:<container> --entrypoint /bin/busybox \
 - [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
 - [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
 - Research: `.docaudit/research/auth.md` — the full provider comparison
-  *(the file is gone from the repo and from the host checkout as of
+  - (the file is gone from the repo and from the host checkout as of
   2026-10-18)*

serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md

@@ -47,7 +47,7 @@ torrent.
     WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s
 
 The private key is hashed into the VM env file and never enters the repo.
-NAT-PMP inbound forwarding is available on Proton (gluetun supports it via
+NAT-PMP inbound forwarding is available on Proton (gluetun supports it
 `FIREWALL_INPUT_PORTS` + `NATPMP=true`) — deliberately deferred: the e2e
 gate passes outbound-only, and swarm peers needing an inbound port is a
 seeding-era concern, not a grab-only one.
@@ -86,7 +86,7 @@ are unaffected.
   got `Unauthenticated`); the live conf was patched and
   `scripts/deploy-remote.sh` fixed to write the `_PBKDF2` key. Gotcha
   (live, 2026-09-01): after qB 5.x ran, its conf rewrite left a hash line
-  4.6.7 rejects (`invalid credentials` for every login, including the arrs);
+  4.6.7 rejects (`invalid credentials` for every login, including the arrs)
   rewriting the same `@ByteArray` line fresh (same password, 100k
   iterations) restored login. Any future version flip: regenerate the line.
 - `HostHeaderValidation=false` + `ServerDomains=*`: the socat forwarder
@@ -98,7 +98,7 @@ are unaffected.
   path (no auto-TMM), so the default itself points at `Finished Torrents`:
   every grab lands there regardless of client behavior; human review moves
   files onward; nothing writes to the entropy volume.
-- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`,
+- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`
   `tv-sonarr` (all → `Finished Torrents`).
 - **Download clients** (gateway `10.88.0.1:8080`, user `admin`): all five test
   green against live qB — sonarr (`tv-sonarr`), radarr, lidarr, readarr
@@ -109,12 +109,12 @@ are unaffected.
   as "Authentication failure" — that kills runtime auth too, not just API
   creates; the fix was pinning qB itself (below), after which readarr tests
   200 like the rest.
-- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`,
+- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`
   not `latest`): `latest` resolved to 5.x, whose `204` login breaks
   Readarr's frozen client; 4.6.7 answers the legacy `200` + `"Ok."` that
   every client accepts. Pinned in
   `frankbook-pro/qbittorrent/quadlet/qbittorrent.container`.
-- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`,
+- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`
   gateway ports). Readarr deliberately omitted: prowlarr's indexer-push
   test to readarr's `/api/v1/indexer/test` 500s (EOL schema drift).
   Legacy `Prowlarr-*` indexer rows carried in the migrated dankserver DBs

serverAppConfigs-docs/Infrastructure/Grafana.md

@@ -46,7 +46,7 @@ The Prometheus datasource is declared in git, not clicked into `grafana.db`:
 address Caddy proxies to, never the loopback-published port — and carries
 `timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
 scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
-alongside it by `dashboards/provider.yml` (provider `fleet`,
+alongside it by `dashboards/provider.yml` (provider `fleet`
 `updateIntervalSeconds: 30`, `allowUiUpdates: true`).
 
 A Loki datasource is still **not** provisioned — the tree contains exactly one
@@ -69,8 +69,8 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
   client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
   Basic client auth, form-encoded token body, and the fork's userinfo-claims
   bug that leaves identity at the `sub` hash only) is in
-  [[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana:
-  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
+  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
+  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
   `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
   users land as Viewer with `login = <sub hex>`; no email/groups/role claims
   reach Grafana until the evaluator bug is fixed — do not write role mappings
@@ -79,7 +79,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
   the host without the database or an admin password. *(unverified 2026-10-18)*)
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
   file is a gluetun env file and it carries `HOME=/root`; systemd imports it
-  for the service, podman then looks for its configuration under `/root`,
+  for the service, podman then looks for its configuration under `/root`
   cannot read it, and the unit dies at start with
   `stat /root/.config: permission denied`. The unit reads
   `/home/dank/Containers/grafana/env`, which the deployer seeds with exactly

serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Homepage Status Probes
-description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
+description: "How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions."
 tags: [homepage, status, probes, e2e, homepage-status]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -51,7 +51,7 @@ See [[Container Host Reachability]] for the netns rules.
    published nothing → structurally unprobeable from the host).
 3. Probe from the **correct vantage**: from a bridge-net container the host
    is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
-   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
+   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`)
    which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
    bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
    they expand **inside** `podman exec bash -c`.
@@ -83,7 +83,7 @@ python3 -m http.server 8899 --bind 127.0.0.1
 ```
 
 Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
-(panels render after the async fetch; checking immediately returns 0 panels),
+(panels render after the async fetch; checking immediately returns 0 panels)
 extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
 `.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
 plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for

serverAppConfigs-docs/Infrastructure/Homepage.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation."
+description: "The password-protected service list on Dankclaw: what each card shows and where its data comes from."
 tags:
   - infrastructure
   - homepage
@@ -22,7 +22,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
 **Docs page shipped (2026-09-17):** `/docs` serves a static user documentation
 page (`shared/homepage/docs.html`) — access walkthrough, the running-model
 guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
-remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`,
+remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`
 `frank-zealot`) moved from admin-only to the **user** page with live model
 descriptions and a `docs` link to `/docs#models`.
 
@@ -59,7 +59,7 @@ without which the picker showed "No results found" for every family account
 "equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
 intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra
 (same era)", frank-zealot "equivalent to last year's frontier level
-(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
+(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic)
 pinned in `homepage_config.bats`.
 
 **Protoss rename + Model Information component (2026-10-08):** the model
@@ -91,7 +91,7 @@ same renames, api-id mapping, supervision and compaction notes.
 app card now carries hex-clipped art beside its name (`services.json` `art`
 field, files under `img/apps/`): 19 brand marks tinted amber from the
 Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323
-initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard),
+initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard)
 node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy,
 Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model
 cards render in the order zealot → dragoon → archon (registry order is the
@@ -143,8 +143,8 @@ per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki
 indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red,
 Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow,
 Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service
-fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`),
-`warn` on OpenWebUI (searches ride the Seerr→SearXNG path),
+fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`)
+`warn` on OpenWebUI (searches ride the Seerr→SearXNG path)
 `model.compaction` (the amber "set this as compaction model" callout
 frank-zealot carries). frank-dragoon's card portrait is the SC2 unit
 portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole
@@ -159,10 +159,9 @@ Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
 entirely — admin tooling; Seerr is presented as the front door for
 requests. Each fleet model gained a Hugging Face source link: card field
 `model.hf` rendered as the cyan `model on huggingface ↗` line under the
-role, and the docs `Model:` bullets link the checkpoint names (zealot →
-HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon →
-Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon →
-Qwen/Qwen3.8-Flash-Next-FP8).
+role, and the docs `Model:` bullets link the checkpoint names — which
+checkpoint each endpoint runs is payload state on
+[[frank-dragoon]], [[frank-archon]] and [[frank-zealot]].
 
 **Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
 wave):** the amber CRT palette is retired. The page is now deep space
@@ -176,7 +175,7 @@ titles are chamfered plates with the diamond glyph inside the plate.
 Fonts: body is **Share Tech Mono** (legible retro terminal), headings
 are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all
 three pages load the pair from Google Fonts. Four tiles that had
-reverted to initials got pictograms: Calibre-Web an open book (orange),
+reverted to initials got pictograms: Calibre-Web an open book (orange)
 RomM a purple GameBoy, Copyparty a blue folder with upload arrow,
 Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`).
 frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching
@@ -198,7 +197,7 @@ state — the only animation left is the speaking-voice pulse. The result
 is flat navy panels on crisp 1px borders: same Protoss direction, half
 the pixels.
 
-**Model ids official (2026-10-08):** the card ids `frank-zealot`,
+**Model ids official (2026-10-08):** the card ids `frank-zealot`
 `frank-dragoon`, `frank-archon` are now the harness's official model ids
 in `models.yml`/`config.yml` (see [[OMP Harness]]); the
 systemd services keep the `frank-archon`/`frank-dragoon`/
@@ -206,7 +205,7 @@ systemd services keep the `frank-archon`/`frank-dragoon`/
 one name per model.
 
 **Api ids renamed on the wire (2026-10-09):** the promise in
-`docs.html` is kept — the served model ids are now `frank-zealot`,
+`docs.html` is kept — the served model ids are now `frank-zealot`
 `frank-dragoon`, `frank-archon`, so the docs page no longer carries
 `api id:` annotations, and the curl/opencode/Pi examples send
 `frank-zealot`. The systemd services keep the old names. `docs.html`
@@ -217,7 +216,7 @@ ids, which did not change.
 clip-paths replaced with `border-radius` (the clipped corners read as
 gaps in the 1px border); section-title diamonds moved from absolute
 positioning to a fixed-width inline glyph (aligned, wrap-safe); closed
-WIP items grey out (dim type, desaturated art, muted border via
+WIP items grey out (dim type, desaturated art, muted border
 `:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track
 (`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms
 beat; the two SC2 source oggs are trashed); privacy notes became
@@ -235,11 +234,10 @@ house-DNS advice; the bullet now just says the names resolve on the
 home network.
 
 **Archon card runs the fleet's own model (2026-10-08):** the
-`frank-archon` homepage card and docs Model bullet now say the payload
-is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link
-our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)
-in place of the FP8-era base-model link; capacity line says three
-submissions (`--max-num-seqs 3`).
+`frank-archon` homepage card and docs Model bullet swapped the FP8-era
+base-model link for our own HF publication, and the capacity line for the
+endpoint's own concurrency figure; the payload, its publication and the slot
+count all live on [[frank-archon]].
 
 ## Design decision — 2026-08-22
 

serverAppConfigs-docs/Infrastructure/LAN-DNS.md

@@ -188,7 +188,7 @@ sudo systemctl restart systemd-resolved && resolvectl flush-caches
 ```
 
 The `~` (route-only) prefix is load-bearing: it routes only those domains to
-Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
+Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`
 `resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
 
 **Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
@@ -236,7 +236,7 @@ Endpoints (all POST, JSON body):
   `.content.records[]` as some docs claim — parse with
   `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
 - Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
-- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`
   `type`, `content`, `ttl`, `prio`).
 
 Quirks that bite (verified live 2026-08-31):

serverAppConfigs-docs/Infrastructure/Loki.md

@@ -55,7 +55,7 @@ The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `
 The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
 
 - `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
-- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
+- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`
   `chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
   `find /opt/loki/data -type f` counted **8 files**, all WAL segments,
   cluster-seed and delete-request bookkeeping, **no chunk objects**

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Observability Provisioning
-description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
+description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
 tags: [observability, prometheus, grafana, provisioning, deployment]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -21,7 +21,7 @@ committed.
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
 
-Targets and datasources reference **container names** (`grafana:3000`,
+Targets and datasources reference **container names** (`grafana:3000`
 `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
 loopback-published ports; `alloy` is **not scraped at all** (no job in the
 config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
@@ -47,7 +47,7 @@ from `deploy_observability`:
 - `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
 
 `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
-`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
+`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
 `shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
 `shared/grafana/**` to `grafana` (`:261-264`); changes under
 `dankclaw/observability/` select the touched unit and its dependency consumer
@@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint.
 
 **Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
 Current delivered set: **4 alert rules + 2 recording rules** —
-`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
+`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`
 `LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
-`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
+`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`)
 confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
 reports 4 alerting + 2 recording. Rules reload with the prometheus service.
 
@@ -101,7 +101,7 @@ curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.to
 
 The two Prometheus queries were run on 2026-10-18 and are what every number on
 this page comes from: 4 alerting + 2 recording rules, and `up` for
-`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
+`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
 `frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
 listed in [[Prometheus]]. The Grafana query needs the local admin password and
 was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
@@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect).
 
 ## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
 
-Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
+Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
 loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
 unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
 via `host.containers.internal`) → signal-cli, addressed by number
 `+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
-2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
+2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
 `curl -s http://192.168.0.143:18093/health` → `ok`, and
 `journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
 lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
@@ -130,13 +130,14 @@ the blackbox exporter. See [[Prometheus]] → Targets.
 **A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
 `up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
 the oldest sample, 2026-10-11, is already 0 — while the service itself answered
-`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented**
-on `/metrics` unless the server is started with `--metrics`, so the scrape was
-refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing:
+`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
+is started with the flag, so the scrape was refused at the protocol layer (the
+flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
+stayed firing:
 the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
 -u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
 on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
-`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target
+metrics opt-in landed that day; the target
 reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
 How many payload generations it spanned is not readable from any host
 artifact. *(unverified 2026-10-18)*
@@ -147,11 +148,12 @@ state lives in Prometheus' process, not on disk — a restart re-stamps
 `activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
 evidence of a fresh fault. (Alertmanager by contrast **does** persist:
 `--storage.path=/alertmanager` on a host volume keeps silences and the
-notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
+notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
 `/opt/alertmanager/data`.)
 
 Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
-metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
+metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
+([[frank-dragoon]]), which is why
 `frank-dragoon` has always been green. The active-alert count is a
 moment-in-time value and should never be documented as a constant:
 `/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
@@ -167,8 +169,8 @@ arm (`systemctl is-active frank-zealot` → `failed` after
 `stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
 `TimeoutStopSec`, the known shutdown shape for these engines). So
 `up{job="frank-zealot"} == 1` that day measured the trial, not the production
-shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408
-PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract.
+shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
+and it is the repo contract.
 The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
 reinstalls and starts it, restoring boot persistence and self-healing. Never
 quote a candidate arm's flags or VRAM as fleet state — the process and its shape

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: "Observability on Dankclaw"
-description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured."
+description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
 tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
 timestamp: 2026-09-11
 ---
@@ -51,13 +51,13 @@ Dankserver runs one member of this family, node-exporter, as its own unit
 ## Exposure
 
 The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
-shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
+shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
 `127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
 inside, and several carry lifecycle or query APIs. Browser access goes through
 Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
 `anubis:8923`, and the post-auth handlers route by container name on
-`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
-`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
+`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
+`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
 `prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
 Alertmanager has no vhost at all. node-exporter is the exception that must live
 in the host network namespace, and it binds exactly two addresses rather than
@@ -70,13 +70,13 @@ and the podman bridge subnet `10.89.0.0/16` before the DROP
 
 ## Deployment
 
-`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
+`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
 which installs the unit, copies that service's config
-(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
+(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
 `copy_alloy_config`, `copy_grafana_provisioning`, plus
 `install_signal_relay` with the alertmanager deploy), then waits on the loopback
-health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
-`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
+health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
+`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
 `/node-exporter`: `/metrics`). Startup order follows the data dependencies —
 `grafana` after `prometheus`, `alloy` after `loki`
 (`grafana.container:32-33`, `alloy.container:38-39`). See

serverAppConfigs-docs/Infrastructure/Prometheus.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the provisioned Grafana datasource. Down targets today are structural, not outages."
+description: "Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural."
 tags:
   - infrastructure
   - prometheus
@@ -25,7 +25,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
 | Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
 | Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
-| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
+| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
 
 ## Targets
 
@@ -37,10 +37,10 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
 | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
-| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works |
-| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) |
+| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] |
+| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] |
 
-The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18).
+The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
 
 **Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
 
@@ -53,15 +53,14 @@ The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint
 
 Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
 
-**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service.
+**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
 
 Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
 
-A scrape cannot express "is this HTTP service answering?" — that is what a
-probe is for, and the remedy is the **blackbox exporter** in probe mode
-(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
-uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
-is deployed, these three stay red for a structural reason: deleting them would
+The remedy for a probe-shaped question is the **blackbox exporter**; the
+classification rule and the honesty law for it live in [[Stale Signal Audit]].
+Until the exporter is deployed, these three stay red for a structural reason:
+deleting them would
 hide a real service, and fixing them by publishing databases or enabling metrics
 would trade a wrong signal for a real exposure. Targets are `host:port`, never
 URLs.

serverAppConfigs-docs/Infrastructure/authentik-postgres.md

@@ -1,24 +0,0 @@
----
-type: reference
-description: "The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /"
-tags:
-  - infrastructure
-  - authentik
-  - postgres
-timestamp: 2026-08-28
----
-
-# authentik-postgres
-
-The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /
-OIDC provider. It is the only stateful store in the stack (users, flows,
-sessions, events). Redis was removed in Authentik 2025.10; sessions and caching
-now live here.
-
-- **Compose service:** `authentik-postgres` (in `dankclaw/compose.yaml`)
-- **Image:** `docker.io/library/postgres:16-alpine`
-- **Data:** `/home/dank/authentik/postgres` (bind mount; no named volumes)
-- **Health:** `pg_isready -d authentik -U authentik`
-
-See [Authentik.md](Authentik.md) for the full architecture, setup, backups
-(`pg_dump`), and failure modes.

serverAppConfigs-docs/Infrastructure/authentik-server.md

@@ -1,24 +0,0 @@
----
-type: reference
-description: "The Authentik server — the core, the admin API, and the embedded Caddy outpost."
-tags:
-  - infrastructure
-  - authentik
-timestamp: 2026-09-16
----
-
-# authentik-server
-
-The Authentik server — the core, the admin API, and the embedded Caddy outpost.
-This is the service the Caddy `forward_auth` vhosts proxy to
-(`authentik-server:9000`). It also serves the login page at
-`https://auth.franksdank.cloud`.
-
-- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` (`authentik-server.service`)
-- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (command: `server`)
-- **Port:** `9000` (published; the outpost + admin + login all live here)
-- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker)
-- **Health:** `http://localhost:9000/-/health/`
-
-See [Authentik.md](Authentik.md) for the full architecture, the Caddy
-forward_auth pattern, first-time setup, and failure modes.

serverAppConfigs-docs/Infrastructure/authentik-worker.md

@@ -1,23 +0,0 @@
----
-type: reference
-description: "The Authentik worker — background tasks: event processing, provider"
-tags:
-  - infrastructure
-  - authentik
-timestamp: 2026-09-16
----
-
-# authentik-worker
-
-The Authentik worker — background tasks: event processing, provider
-synchronization, scheduled flows, and certificate renewal. It runs the same
-image as the server; the `command: worker` selects the role.
-
-- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` (`authentik-worker.service`)
-- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (command: `worker`)
-- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir)
-- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server)
-- **No port** — the worker has no HTTP endpoint.
-
-See [Authentik.md](Authentik.md) for the full architecture, setup, and failure
-modes.

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

@@ -14,7 +14,7 @@ timestamp: 2026-09-03
 
 DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.
 
-DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.
+DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both publish draft models for the Qwen3.5/Qwen3.6 families (table below); what any of them pairs with on this fleet is an era on [[frank-dragoon]] or [[frank-zealot]].
 
 ## How DFlash Works
 
@@ -100,7 +100,7 @@ The DFlash drafter proposes, then ngram-mod runs as an additional implementation
 `--spec-type` takes a comma list, and the gate is `ctx_dft` in
 `common/speculative.cpp` (`common_speculative_init_result`): **draftless**
 types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
-unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
+unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`
 `draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
 
 - A chain may hold ONE context-backed drafter plus any number of ngram types.
@@ -115,12 +115,12 @@ unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
 - Internal order is fixed (ngram types first, then drafters); CLI order is
   irrelevant.
 - **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
-  on the qwen35 hybrid (frank-dragoon era) measured −15 % prose / −7 %
-  block-copy (2026-10-03): adding the ngram layer logs `recurrent state
-  rollback is not compatible with 'ngram-mod' - disabling rollback support`,
-  and MTP verification depends on that rollback, so acceptance drops on ALL
-  traffic even where the matcher fires. Measure chains per architecture,
-  never by analogy.
+  on the qwen35 hybrid loses on both traffic shapes (2026-10-03): adding the
+  ngram layer logs `recurrent state rollback is not compatible with
+  'ngram-mod' - disabling rollback support`, and MTP verification depends on
+  that rollback, so acceptance drops on ALL traffic even where the matcher
+  fires. The measured size of each loss is [[frank-dragoon]]'s (Laws). Measure
+  chains per architecture, never by analogy.
 - ngram-mod prints NO startup banner — the load log shows only the drafter's
   `adding speculative implementation` line. Never conclude "chain not
   engaged" from the load log; identify behaviourally (`draft_n` > 0 only
@@ -198,7 +198,10 @@ Compatible MTP-only draft GGUFs on Hugging Face:
 
 Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.
 
-On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
+On the mid-August 2026 Dankclaw shape `draft-dflash` was the benchmarked winner
+at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those
+three figures are that retired llama.cpp era's property ([[frank-dragoon]] owns
+the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see ).
 
 ## Draft Models
 
@@ -213,13 +216,12 @@ DFlash draft model sizes for the models on these servers:
 
 The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.
 
-## Deployment on Dankserver (NVIDIA RTX 3090)
+## DFlash on an RTX 3090 (Ampere) — the archived vLLM path
 
-The Dankserver llama.cpp deployment (`frank-zealot`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
-
-> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-zealot` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
-
-The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
+DFlash was tried on the 3090 line, lost there to the ngram chain, and was never
+kept deployed; that card's payloads, shapes and rates are [[frank-zealot]]'s
+(its Eras rows are the llama.cpp arms the draft was tested against). What
+transfers is the Ampere arithmetic — compute capability 8.6, so:
 
 - FP8 KV cache is valid on CUDA 12.
 - FP8 weights compute through a fallback path; no native FP8 tensor cores.
@@ -248,11 +250,15 @@ docker run --rm --name vllm-qwen36-dflash \
     --gpu-memory-utilization 0.95
 ```
 
-## Deployment on Dankclaw (AMD Strix Halo gfx1151)
+## DFlash and DSpark on Strix Halo (gfx1151) — what killed them
 
-`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
-
-History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
+Both context-backed drafters died the same way on the iGPU: under parallel load.
+DSpark took `ROCm error: an illegal memory access` inside the shared draft
+context with two concurrent 16k prompts; DFlash corrupted multi-slot into
+premature EOS and was retired first. The timeline, the crash class, the
+"concurrency is the trigger, not size" finding and the `PARALLEL=1` workaround
+are on ; the shape they were measured on is
+[[frank-dragoon]]'s mid-August 2026 era.
 
 ## A/B protocol on a live llama.cpp endpoint
 
@@ -297,7 +303,7 @@ Protocol:
 
 Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
 `predicted_per_second`; cumulative `/metrics`
-`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
+`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`
 per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
 while tokens flow = the proposer is NOT engaged, whatever the flags say. A
 zero-draft config is a useful negative control: it proves the other config's
@@ -315,7 +321,7 @@ not switch without re-measuring".
 
 The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
 slot split, quant rung — not just spec flags) is the transient systemd
-drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
+drop-in procedure in [[frank-dragoon]] and [[frank-zealot]].
 
 ## Is it worth it? (arithmetic before deployment)
 
@@ -359,24 +365,26 @@ pay for K sequential forwards.
 Trained heads are target-specific: EAGLE/DFlash consume the TARGET's hidden
 states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
 
-**Step 4 — acceptance reality checks.** Fleet datum: a trained z-lab DFlash2
-at depth 7 accepted ~41 % on its matched target (`draft-dflash 7 → 43.5
-tok/s`); treat 40–60 % as the honest band for a matched pair. Cheapest proxy
-for `p` at temp 0 when both models are greedy and share a tokenizer: greedy
-continuation agreement on identical prompts, two API calls, no weights moved.
-llama.cpp can also synthesize the projection without any draft artifact:
+**Step 4 — acceptance reality checks.** Treat 40–60 % as the honest band for a
+matched pair; the arm that anchors the bottom of it on this fleet is the
+depth-7 `draft-dflash` row of 's spec table
+(accepted share, rate and VRAM for every depth are in that table). Cheapest
+proxy for `p` at temp 0 when both models are greedy and share a tokenizer:
+greedy continuation agreement on identical prompts, two API calls, no weights
+moved. llama.cpp can also synthesize the projection without any draft artifact:
 `--spec-synth-rates P0,P1,...` / `--spec-synth-len L`. Speculation pays IN
-PROPORTION TO HOW SLOW THE TARGET IS — the same 41 % gave 2.3x on an
-18.8 tok/s layer-split target and a loss on a 25 tok/s one; fast targets are
-usually sparse MoE, the worst case for drafting.
+PROPORTION TO HOW SLOW THE TARGET IS — that band's midpoint was worth 2.3× on
+the slow two-card layer-split target and a LOSS on the 25 tok/s target of the
+worked example above; fast targets are usually sparse MoE, the worst case for
+drafting.
 
 **Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
 engines refuse external drafters by design: halogen-flash-server's
 `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
-name is a deliberate 400 (a silent downgrade would make comparisons lie);
+name is a deliberate 400 (a silent downgrade would make comparisons lie)
 there is no device routing in its loopback token protocol, so a drafter can
 never sit on another card. Pure llama.cpp DOES route the draft:
-`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
+`--spec-draft-device/-devd` (default follows `--device`), `-ngld`
 `-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
 One process sees only the devices its backends were built with — CUDA+Vulkan
 is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
@@ -403,20 +411,16 @@ measurement that would settle it.
 
 - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
 - DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
-- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
+-The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by .
 - `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
 - Use `rejection_sample_method` default `strict` for lossless output.
 
 ## See Also
 
 - [[Quantization by Hardware Architecture]]
-- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
-- [[Running Qwen3.6 27B FP8 on Dankclaw]]
 - [[Dankclaw System Specs]]
 - [[Dankserver System Specs]]
-- [[VLLM Overview]]
-- [[deployment]]
 - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
 - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
-- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
-- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
+-  where the drafter's KV actually lands
+- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle

serverAppConfigs-docs/LLM/Dankclaw Improvements.md

@@ -17,66 +17,59 @@ Dates: researched 2026-08. Hardware is verified on the live machine unless
 marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
 
 **Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
-iGPU behind :7770 — is retired. Dankclaw now serves
-[[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2, port
-7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on
-the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18
-03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18
-([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of
-the era they were taken in; the `ttm` ceiling and the NVMe scheduler rows were
-re-checked 2026-10-18 and still hold.
+iGPU — is retired; what runs on this box now (both endpoints, their engines and
+ports) is [[frank-dragoon]]'s and [[frank-zealot]]'s pages, and both run under their
+repo units today — no drift open as of 2026-10-19 (`frank-zealot` active since
+2026-10-18 21:06, the deployer's 2-slot reslice; `frank-dragoon` active). The OS facts
+are this page's: `iommu=pt` since 2026-10-18 ([[dankclaw-iommu-translation-off]]), and
+the `ttm` ceiling and NVMe scheduler rows below were re-checked 2026-10-18 and
+still hold. Every benchmark number below is a measurement of the era it was
+taken in.
 
 ## Corrections (re-verified 2026-08-15/16)
 
-- **Backend**: that era's production was **Vulkan0**, not ROCm. Measured on this box
-  (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
-  Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
-  302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
-  hold on the 8060S. See [[frank-dragoon-benchmark]].
+- **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm
+  wins prefill and Vulkan wins decode, and the decode gap widens with context:
+  the 64k A/B is  §4, the 262k A/B is
+   §High-Context Backend Comparison. The "HIP
+  out-performs Vulkan for decode" claim did not hold on the 8060S.
 - **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
   does not exist on this machine (checked 2026-08-15). GPU DPM is already
   `performance`, forced level `auto`. This item is N/A.
-- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). The law
-  still holds on `frank-zealot` (`CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` —
-  `scripts/frank-zealot.env:55-56`).
+- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What
+  `frank-zealot` runs now is its own Current shape — this row is not it.
 - **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
-  (ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. Today `frank-zealot`
-  is configured with speculation **off** (`SPEC_TYPE=none` — `scripts/frank-zealot.env:67`).
-- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and
-  persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that
-  once held the service at ~15 GiB is gone.
+  (ngram-mod) with draft-mtp first — timeline on ; the
+  spec setting each endpoint runs today is on its own page.
+- **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded
+  once on [[Dankclaw System Specs]] (Memory).
 
 ## Verified Memory Picture (measured 2026-08-11)
 
 | Figure | Value | Source |
 | ------ | ----- | ------ |
 | OS physical RAM | 30.5 GiB | `/proc/meminfo` |
-| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` |
 | ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve |
 | ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
 
 That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
 "dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
-recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT). The OS now accounts
-**121 GiB** (`MemTotal 127108080 kB`, 2026-10-18) and `rocm-smi
---showmeminfo vram` reports **2.0 GiB** (2,147,483,648 B) of dedicated VRAM —
-the carve is gone and the GPU budget is the GTT aperture, not the SMI figure
-([[Dankclaw System Specs]]).
-The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and
-`/etc/modprobe.d/ttm.conf` still carries the `options ttm` line (both read
-2026-10-18).
+recorded in  §Why Vulkan + GTT; the 76.2 GiB row is an
+era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`
+the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated
+carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read
+2026-10-18.
 
 This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
-limits documented in [[frank-dragoon-benchmark]].
+limits documented in [[frank-dragoon]].
 
 ## 1. Self-Hosted Agentic Tools
 
-All of these connect to the local OpenAI-compatible endpoint
-(`http://127.0.0.1:7770/v1`). Today that port is **halogen-flash-server**, not a
-llama.cpp server, and the address must be the IPv4 literal: `curl
-http://localhost:7770/v1/models` returns nothing while
-`http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18 — the rootless
-podman publish is IPv4-only). None need NVIDIA.
+All of these connect to the local OpenAI-compatible endpoint on
+`:7770` (which engine owns that port is [[frank-dragoon]]'s to say), and the
+address must be the IPv4 literal: `curl http://localhost:7770/v1/models` returns
+nothing while `http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18
+— the rootless podman publish is IPv4-only). None need NVIDIA.
 
 | Tool | Use | Endpoint config | Priority |
 | ---- | --- | --------------- | -------- |
@@ -90,12 +83,13 @@ podman publish is IPv4-only). None need NVIDIA.
 | Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
 
 The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
-with the endpoint on 7770 and need no extra GPU. The 7770 endpoint today is
-halogen-flash-server, not llama.cpp.
+with the endpoint on 7770 and need no extra GPU.
 
 ## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
 
-Verified against the running service and llama.cpp source.
+Verified against that era's running service and llama.cpp source; the era is
+retired ([[frank-dragoon]]), so this table is history and a method, not a live
+TODO list.
 
 | Change | Recommendation | Priority |
 | ------ | -------------- | -------- |
@@ -104,8 +98,8 @@ Verified against the running service and llama.cpp source.
 | Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
 | Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
 | Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
-| DFlash n_max | Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput. | Verify |
-| ngram layering | Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled. | Keep off |
+| DFlash n_max | Depth 8 (block 16) was high; a shallower draft cuts rejection cost at the same throughput. Sweep depth per the A/B protocol in [[DFlash DSpark Speculative Decoding]]. | Verify |
+| ngram layering | Layering ngram on the DFlash draft measured slower than the draft alone at 4k — figures on [[DFlash DSpark Speculative Decoding]]. | Keep off |
 
 ## 3. Server / OS / Hardware Settings
 
@@ -113,7 +107,7 @@ All require root and a reboot for the modprobe change.
 
 | Setting | Action | Effect | Priority |
 | ------- | ------ | ------ | -------- |
-| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done |
+| TTM GPU pool | Raised 2026-08-25; the knob, its value and its persistence file are [[Dankclaw System Specs]]'s (Memory row) | Lifted the ~15 GiB kernel-page ceiling so a full 262k KV plus a ~30 GB payload fits one slot | Done |
 | Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ |
 | NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done |
 | BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
@@ -130,16 +124,14 @@ All require root and a reboot for the modprobe change.
 
 ## Sources
 
-- [[frank-dragoon-benchmark]] — measured backend/spec numbers
-- [[deployment]] — deploy and update process
+- [[frank-dragoon]] — measured backend/spec numbers
+-  deploy and update process
 - AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
-- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
+- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`
   `common/arg.cpp`, `common/common.h`
 - NotebookCheck Ryzen AI MAX+ 395 analysis
 
 ## See Also
 
-- [[frank-dragoon-benchmark]]
+- [[frank-dragoon]]
 - [[DFlash DSpark Speculative Decoding]]
-- [[deployment]]
-- [[index]]

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -10,7 +10,7 @@ timestamp: 2026-09-03
 
 ## Server Overview
 
-Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[ciru-activefpx-promptforge]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
+Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
 ## Hardware Specifications
 
 ### CPU
@@ -32,7 +32,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
 | Device ID | 0x1586 |
 | Architecture | gfx1151 (Strix Halo) |
 | GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `mem_info_gtt_total` = 122,402,963,456 B read 2026-10-18; rocm-smi's smaller SVM carve-out is not the budget) |
-| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ([[ling-3.0-flash-trial]] §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
+| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ( §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
 
 ### NPU
 
@@ -44,9 +44,9 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
 | Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) |
 | Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) |
 | Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
-| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] |
+| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See  |
 
-The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot-llama-prism-bonsai2-pq20]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot-gemma4-12b]], 2026-09-17 → 10-01).
+The NPU endpoint is retired (2026-09-11; ). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see ); the Gemma4-12B QAT rung it once pointed at is retired history (, 2026-09-17 → 10-01).
 
 ### ROCm
 
@@ -66,6 +66,7 @@ The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name
 | System RAM | 121 GiB (shared CPU + GPU; `free -g` total 121, `MemTotal 127108080 kB`, 2026-10-18) |
 | GPU-Visible (ROCm) | ~114 GiB GTT aperture (`mem_info_gtt_total` 122,402,963,456 B — the budget) |
 | Swap | 64 GB `/swap.img` (`swapon`, 2026-10-18; `free` reports 63 GiB) |
+| TTM page cap | `ttm.pages_limit = 29883536` (114 GiB of 4 KiB pages) and `page_pool_size=29883536`, persisted in `/etc/modprobe.d/ttm.conf`, applied live 2026-08-25 and re-read 2026-10-18 (`/sys/module/ttm/parameters/pages_limit`). The kernel default is half of RAM, so in the firmware-VGM era (~30 GiB visible) the cap was ~15 GiB and a 28 GB payload plus a 262k KV cache could not load — this is the row every other page should cite |
 | /dev/shm | 61 GiB tmpfs (half of RAM, `df /dev/shm`, 2026-10-18) |
 
 ### Storage
@@ -79,7 +80,7 @@ The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name
 | Interface | Type | Speed | IP | Notes |
 |-----------|------|-------|----|-------|
 | eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carried the RPC path to Dankserver during the 2026-08-17 pipeline-parallelism trial; no RPC listener runs today (`ss -ltn` shows no :50052, 2026-10-18). Default route via 192.168.0.1. |
-| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. |
+| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See  section 0. |
 | docker0 | bridge | - | 172.17.0.1 | Usually down. |
 
 The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at
@@ -95,21 +96,19 @@ cable between claw and server is required for 10G.
 | Kernel | 7.0.0-38-generic (`uname -r`, 2026-10-18) |
 | Boot line | `iommu=pt` since 2026-10-18 (`/proc/cmdline`, no `amd_iommu` token); eras before: `amd_iommu=off` 2026-09-11 → 10-18, IOMMU default-on before that — [[dankclaw-iommu-translation-off]] |
 | Python | 3.14.4 (system, `python3 --version`, 2026-10-18). The 3.12.13 CIRU venv left with that runtime (retired 2026-08-22) |
-| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[ciru-activefpx-promptforge]]); not the serving runtime |
+| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[frank-dragoon]]); not the serving runtime |
 | uv | 0.12.8 (`~/.local/bin/uv --version`, 2026-10-18) |
 
 ## Runtime Relevance
 
 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
 - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
-- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon-flash-server-v2-native]]) — and `frank-zealot` on 7771 ([[frank-zealot-llama-prism-bonsai2-pq20]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon-gufo-qwen38-27b]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
+- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
 - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
 
 ## See Also
 
-- [[ciru-activefpx-promptforge]]
+- [[frank-dragoon]]
 - [[Shared Memory Architecture Dankclaw]]
-- [[frank-redo-2026-08-17]]
-- [[FastFlowLM NPU Inference]]

serverAppConfigs-docs/LLM/Dankserver System Specs.md

@@ -1,7 +1,7 @@
 ---
-type: configuration
+type: reference
 title: Dankserver System Specs
-description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium.
+description: "Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture."
 tags:
   - dankserver
   - hardware
@@ -16,7 +16,7 @@ timestamp: 2026-09-20
 ## Server Overview
 
 Dankserver is a media/library server and the `frank-archon` LLM host (radiance
-1.0.13, 2×R9700 tensor-parallel — [[frank-archon-radiance-qwen38-iq4r]]; the
+1.0.13, 2×R9700 tensor-parallel — [[frank-archon]]; the
 `frank-medium` name this page carried was retired 2026-09-20, see §Active
 Services). The host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that
 served the FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and
@@ -30,7 +30,7 @@ read 2026-10-18).
 | Property | Value |
 |----------|-------|
 | Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
-| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
+| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see ) |
 | GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3` (both bridges: `current_link_width=8`, `current_link_speed=32.0 GT/s`, read 2026-10-18); ~31.5 GB/s per card. Each card sits behind an on-board PCIe switch (`01:00.0`/`02:00.0`, `05:00.0`/`06:00.0` — "Navi 10 XL Upstream/Downstream Port of PCI Express Switch"), so the GPU endpoint itself trains **x16** Gen5 while the CPU-facing leg stays x8 |
 | Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board (2026-09-20 census; not re-readable today — `lspci -vv` needs root) *(unverified 2026-10-18)* |
 | Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
@@ -55,7 +55,7 @@ read 2026-10-18).
 | Architecture | gfx1201 (RDNA4) |
 | VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
 | Link | PCIe 5.0, endpoint-trained **x16** each behind the on-board switch; the CPU-facing leg is x8 (`00:01.1` / `00:01.3`, both `current_link_width=8` at 32.0 GT/s, read 2026-10-18). No XGMI. Separate IOMMU groups — `03:00.0` → group **15**, `07:00.0` → group **20** (`/sys/bus/pci/devices/.../iommu_group`, read 2026-10-18) |
-| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see [[Dual-R9700-Cooperation]] |
+| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see  |
 | PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
 | iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2 (rocm-smi GPU[2], 512 MiB). The retired llama.cpp trios hid it with `HIP_VISIBLE_DEVICES=0,1`; the live radiance engine instead enumerates all three AQL devices and ignores it ("3 devices [0:gfx1201 1:gfx1201 2:gfx1036], using 2 of 3, ignoring gfx1036") |
 
@@ -101,11 +101,10 @@ RCCL-backed rebuild needs no new host packages.
 | Kernel | 7.0.0-38-generic, `iommu=pt` (`uname -r`, `/proc/cmdline`, read 2026-10-18) |
 | Python | 3.14.4 |
 | ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
-| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon-radiance-qwen38-iq4r]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
+| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
 
 The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
-hijack `libhsa-runtime64` were purged on 2026-09-15 — see
-[[frank-medium-rocmfpx]].
+hijack `libhsa-runtime64` were purged on 2026-09-15 —
 
 ### Active Services
 
@@ -113,16 +112,16 @@ The active inference service is `frank-archon.service` (`systemctl is-active`
 `active`, `is-enabled` → `enabled`, read 2026-10-18): radiance 1.0.13 in a
 host-network podman container answering `curl 127.0.0.1:7770/health` →
 `{"status":"ok","model":"frank-archon",…}`. See
-[[frank-archon-radiance-qwen38-iq4r]].
+[[frank-archon]].
 
 The services this page once called active are gone: `frank-medium.service`
 (llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung
-ran 2026-09-16 → 09-20 and is documented in [[frank-medium-llamacpp-dflash2]];
+ran 2026-09-16 → 09-20 and is documented in 
 `frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU
 swap. Its archived unit was moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`
 and the vllm-radiance replica trio to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`
 — neither path exists today (checked 2026-10-18), so the archive location is
-unverified. See [[deployment]].
+unverified. See .
 
 ## Hardware Relevance
 
@@ -134,7 +133,7 @@ unverified. See [[deployment]].
   tokens at 32:33, ~6.3/~5.2 GiB headroom) is history. Either way a batch or
   context change needs a re-fit and not a guess.
 - 246 GiB system RAM is **not** headroom: the live shape pins a 43.88 GiB host
-  pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`,
+  pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`
   `scripts/frank-archon.env:184`) plus the 47.68 GiB n-gram/PLE table in RAM,
   and `free -g` showed **90 GiB available** with the engine loaded (read
   2026-10-18). The frank-medium claim that "nothing in the serving path is
@@ -146,7 +145,7 @@ unverified. See [[deployment]].
 
 ## See Also
 
-- [[frank-archon-radiance-qwen38-iq4r]] — the active serving runbook
-- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
-- [[deployment]] — the deploy flow
+- [[frank-archon]] — the active serving runbook
+-  why the PCIe path matters, and what the swap changed
+-  the deploy flow
 - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory

serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md

@@ -1,330 +0,0 @@
----
-type: research
-title: Dual R9700 Cooperation
-description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed.
-tags:
-  - frank-medium
-  - rocm
-  - rccl
-  - pcie-atomics
-  - tensor-parallelism
-  - dankserver
-timestamp: 2026-09-16
----
-
-# Dual R9700 Cooperation
-
-Dankserver's two R9700s run **one tensor-parallel engine across both cards** —
-today that is `frank-archon` under radiance 1.0.13, TP=2
-([[frank-archon-radiance-qwen38-iq4r]]). When this page was written the endpoint
-was `frank-medium`: first llama.cpp's layer split
-([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20
-one vLLM-radiance process at `tensor_parallel_size 2`
-([[frank-medium-radiance-mxfp4]], 2026-09-20 → 09-26). This page records why
-true tensor parallelism was impossible on the previous board (TUF GAMING B650-E
-WIFI), and what the 2026-09-20 board swap changed.
-
-## The short answer
-
-**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose
-bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden
-hostcall-buffer parameter and ROCr refuses to supply it to a device without
-atomics, so *no* RCCL collective or P2P operation could be dispatched on that
-card — see the [measured matrix](#measured-matrix) and the
-[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below.
-
-On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which
-bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
-recommended, at B850 rather than X870E money. Both cards now hang off CPU root
-ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
-`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
-the exact precondition this page's own test plan named, and it is now
-[measured](#measured-verdict-2026-09-20) rather than assumed.
-
-What that bought:
-
-- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted
-  and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp
-  `--split-mode tensor` booted and served against an RCCL-backed build — so a
-  collective HAS been dispatched on this host since the swap.
-- **TP=2 is the production shape.** It wins on every axis measured, by 2.1x on
-  decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day
-  of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is
-  retired 2026-09-26, and what runs today is the iq4r publisher under radiance
-  1.0.13 ([[frank-archon-radiance-qwen38-iq4r]]).
-- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
-  could not carry the DFlash2 drafter, which is the honest caveat on the
-  comparison below.
-- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
-  card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
-  ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
-  and M.2_2 is free.
-
-## The gate, in source
-
-```cpp
-void Device::checkAtomicSupport() {                    // rocclr/device/rocm/rocdevice.cpp:181
-  std::vector<amd::Device::LinkAttrType> link_attrs;
-  link_attrs.push_back(std::make_pair(LinkAttribute::kLinkAtomicSupport, 0));
-  if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) {
-    if (link_attrs[0].second == 1) {
-      info_.pcie_atomics_ = true;                      // no else branch
-    }
-  }
-}
-```
-
-The flag is a **queried hardware property**, consumed at
-`rocvirtual.cpp:3954` (the hostcall/AQL dispatch path), in `rocblit.cpp` (SDMA
-copy paths) and exported as `hipDeviceProp.hostNativeAtomicSupported`. There is
-no environment variable, module parameter, or sysfs toggle around it — verified
-by grepping ROCclr for `getenv`/`HSA_*`/`ROCR_*`/`AMD_*` in the gate's
-neighborhood: none. Writing the endpoint's Device Control 2 bit 6 by hand
-(`setpci -s 0b:00.0 CAP_EXP+28.w=0440`) does nothing because the value is
-consumed at driver probe time.
-
-What has to be true upstream of it: every PCIe bridge on the GPU's path to the
-CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path
-includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability
-bit**, so no software can enable it.
-
-### The same gate, as this board reports it (2026-09-16)
-
-`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` →
-`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` →
-`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`:
-
-| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
-|---|---|---|---|---|
-| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 |
-| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 |
-| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 |
-| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` |
-| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) |
-| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 |
-
-Three things fall out of that table, none of them in the source alone:
-
-- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the
-  chipset, not the platform, which is why a slot on CPU lanes is the fix.
-- **The endpoint's own bit differs between the cards, and that is the kernel's
-  doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses
-  to set Requester Enable when any bridge cannot route atomics. ROCclr's
-  `pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it
-  makes `lspci -vv -s <bdf> | grep AtomicOpsCtl` a one-command diagnosis: on
-  this host card 0 reads `ReqEn+` and card 1 `ReqEn−`.
-- **Card 1's effective link is x1**, not the x16 its own capability block
-  advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a
-  bandwidth fact rather than an atomics one, and it only matters for
-  collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
-  x4. The layer split's boundary copy does not care.
-
-## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO)
-
-`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` →
-`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` →
-`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only
-LAN/SATA/WiFi:
-
-| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
-|---|---|---|---|---|
-| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 |
-| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 |
-| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 |
-| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
-| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 |
-| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 |
-| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 |
-| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
-
-- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the
-  kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no
-  `Routing-` bridge. Same one-command diagnosis, opposite answer.
-- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
-  by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
-  card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
-- `rocm-smi --showtopo` reported `PCIE` between the cards (Navi 48 has no XGMI),
-  they were in separate IOMMU groups (15 and 20 — re-read 2026-10-18: still 15
-  and 20), and both root ports had ACS `ReqRedir+`, so this page inferred P2P
-  would hairpin at the root complex. **That inference is superseded:** the live
-  engine says the opposite about the same pair — "I device: the 2 cards reach
-  each other's memory directly (peer to peer)" (`podman logs frank-archon`,
-  2026-10-15 boot). Plain P2P traffic measured 45.27 GB/s bidirectional on the
-  *old* board — DMA was never the gate, atomics were.
-- Kernel 7.0.0-31-generic with `iommu=pt` then; this host runs 7.0.0-38-generic
-  with `iommu=pt` today (`/proc/cmdline`, read 2026-10-18). Both were ruled out
-  as causes ([above](#ruled-out-empirically-do-not-re-derive)) and this host's
-  IOMMU posture is unchanged.
-
-**Still unproven (as of 2026-09-20).** The gate that matters is ROCclr's
-`checkAtomicSupport()` reading the `kLinkAtomicSupport` link attribute; the ReqEn
-bit is the same cause seen from the kernel side, but no RCCL kernel had been
-launched since the swap. Until the 2-rank probe runs — or a `--split-mode row`
-boot that does *not* log `internal AllReduce init failed ... falling back to
-meta-backend butterfly` — "collectives work here" was a strong expectation, not a
-measurement. **RCCL never got that measurement**; what settled the operational
-question instead is production traffic: radiance's TP=2 wire moves cross-rank
-all-reduces over the direct P2P path and reports its own precision — "tp wire:
-LOSSY (--tp-wire wht6) at 128 KiB a message and above … Steps of at most 25 rows
-are exact" (`podman logs frank-archon`, 2026-10-15 boot).
-
-## Measured matrix
-
-Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
-model. Rank 0 = card #1 (CPU lanes), rank 1 = card #2 (chipset):
-
-| Operation | rank 0 (card #1) | rank 1 (card #2) |
-|---|---|---|
-| `broadcast` | OK | **FAIL** `DistBackendError` |
-| `all_reduce` | OK | **FAIL** |
-| `barrier` | OK | **FAIL** |
-| `send` / `recv` (P2P) | blocked, then timeout | **FAIL** |
-| `all_gather` (after failure) | group unusable | — |
-
-The `AMD_LOG_LEVEL=3` trace names the cause: `Pcie atomics not enabled, hostcall
-not supported` → `AQL dispatch failed!` → `hipLaunchKernel` returns
-`hipErrorIllegalState` for `ncclDevKernel_Generic_4(...)`. Nothing in vLLM or
-RCCL mentions atomics; the runtime log is the only place it appears.
-
-**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism
-as an option, because vLLM's PP stage handoff is exactly that primitive.
-
-## Software escapes: closed, and the last one is now moot
-
-Every row below was written under the old board. The last row — the only door
-that was ever open — stopped mattering on 2026-09-20, because the hardware gate
-it existed to route around no longer applies to this host.
-
-| Idea | Finding |
-|---|---|
-| vLLM's IPC custom allreduce | **Gated off by a table keyed to CUDA sm versions** — `{9.0, 10.0, 10.3, 10.7}`; ROCm reports capability `12.0` for gfx1201, so the gate is `False`. `custom_ar` itself is compiled into the image. |
-| ...and if that table were patched | Still insufficient: `broadcast`, `barrier`, and the TP logits `all_gather` have no non-RCCL path. |
-| The radiance fork's own allreduce | Already present and *on by default* (`RADIANCE_USE_R4D_AR`, size gate `RADIANCE_AR_MAX_KB`): a one-shot P2P push allreduce wrapping `CudaCommunicator.all_reduce`, exchanging IPC handles over the **gloo** group. With the gate raised it covers prefill and decode, and `ncclDevKernel` disappears from the profile. But it does not intercept `PyNcclCommunicator.__init__`'s **warmup allreduce**, nor `pynccl_comm.all_gather` — both direct RCCL calls. |
-| NCCL symmetric memory | Unavailable: the shipped RCCL advertises NCCL 2.30.4 but exposes no `ncclDevComm*` API, so torch's symmetric-memory path cannot run. |
-| Prefill/decode disaggregation | Needs a KV-transfer connector plus coordination collectives; both out. |
-| gloo for GPU work | CPU/TCP collectives — orders of magnitude too slow for per-layer TP. |
-| **Rebuild RCCL so its kernels do not declare a hostcall buffer** | **OPEN — unverified.** The claim is that the hostcall requirement is triggered by device-side `printf`/`assert`, and that `-mprintf-kind=buffered` avoids it. Counter-evidence: RCCL's device source at `src/device/` contains **no live** `printf`, `assert`, or `__assertfail` (checked), so what makes its kernels declare `HiddenHostcallBuffer` is unexplained. The claim traces to search-result summaries, not a primary source. Cheap to settle once a host is available: recompile librccl with `-mprintf-kind=buffered`, dump a kernel's declared arguments (`roc-obj-extract` + `llvm-readelf --notes`) and look for `hidden_hostcall_buffer`; if it is gone, a 2-rank probe should dispatch. **Stop and mask the service first** — see the note below. |
-
-## Bandwidth: why x4 is enough
-
-A 27B TP=2 allreduce at batch 4 moves roughly
-$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$
-per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**,
-or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8
-~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound.
-That is why the cheap option (the M.2 riser) and the expensive one (the board
-swap) were both considered adequate on bandwidth alone — and why the board that
-landed, at x8/x8, clears the arithmetic with room to spare.
-
-## Ruled out empirically (do not re-derive)
-
-- Kernel versions 7.0.0-28 / -30 / -31 — byte-identical failure.
-- `iommu=pt`, and `amd_iommu=off` is unrelated (that was a Dankclaw prefill knob).
-- P2P hardware: 45.274 GB/s bidirectional between the cards works fine — plain
-  memory traffic is not the problem; *atomic* routing is.
-- A from-source librccl rebuild: identical failure at the identical call path.
-- Every NCCL transport/algorithm knob.
-- A one-card test proves nothing: NCCL short-circuits at `world_size=1`, which
-  is exactly why the two replicas work at all.
-- Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 —
-  different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure.
-
-## Measured verdict (2026-09-20)
-
-Both cooperation modes were run on this host on 2026-09-20 — the same salted
-10-prompt battery (`scripts/bench-conversation.py`), the service stopped and
-masked between arms. Decode/prefill are geometric means over the ten rows;
-burst-4 is four concurrent copies of one prompt, reported as aggregate tokens/s.
-
-| Shape | Cross-device path | Decode | TTFT | Prefill | burst-4 |
-|---|---|---|---|---|---|
-| **radiance TP=2 + DFlash2 — deployed** | R4D one-shot allreduce | **95.8 t/s** | **0.21 s** | **1493.6 t/s** | **259.4 t/s (65.6/stream)** |
-| llama.cpp layer split 32:33 + DFlash2 — predecessor | none, boundary activation copy | 45.7 t/s | 0.72 s | ~530 t/s | 71.7 t/s |
-| llama.cpp `--split-mode tensor` 1:1 | RCCL allreduce | 30.5 t/s | 0.71 s | n/r | 54.3 t/s |
-
-Row 3 carries a caveat that no flag can remove: llama.cpp's tensor mode could not
-load the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight
-pre-allocated on the non-primary device has no backend once `n_copies == 1`), so
-it decoded **without** speculation while the other two rows decoded with it. The
-comparison that decides anything is row 1 against row 2, and some of that gap is
-the 4-bit MXFP4 checkpoint rather than the parallelism.
-
-Two numbers elsewhere on this page are therefore not interchangeable:
-
-- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce,
-  with RCCL off, on the old board. It was never a TP decode number.
-- 26.3 t/s from the same table is that butterfly tensor's decode. Row 3's 30.5 is
-  an RCCL tensor decode a day later: same shape family, different collective path.
-
-RCCL needs PCIe atomics on *every* participating GPU — the gate the old board
-failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is
-why TP=2 runs today rather than merely "being unblocked": the image ships TheRock
-RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so
-radiance could not have used RCCL on this host even with atomics routed. Ubuntu's
-RCCL 2.27 does carry gfx1201 kernels, and that is what row 3 linked against.
-
-**Rule for any future multi-GPU collective experiment: stop and mask the
-production service first**, per the benchmarking policy, or run it on a host that
-is not serving users. The production service is `frank-archon`, a **root** unit,
-so on Dankserver that is `sudo systemctl stop frank-archon && sudo systemctl
-mask frank-archon` (`scripts/frank-archon.service`; never `disable` — that
-deletes the unit file, and `mask` is what the bench policy prescribes). The
-2026-09-20 wording below — `systemctl --user stop frank-medium && systemctl
---user mask frank-medium`, frank-medium being a rootless quadlet — is retired
-with that service, which today reports `not-found` (read 2026-10-18).
-
-This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live
-service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a
-power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the
-GPU driver is a plausible cause and this host has precedent for going
-unreachable until a power cycle.
-
-
-## The third path (production in the layer-split era): llama.cpp's layer split
-
-Everything above is about *collectives*, and vLLM cannot proceed without them
-in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage
-handoff is a p2p `send`/`recv` of the hidden state, the primitive this host
-also fails. **llama.cpp's `--split-mode layer` needs no collective at all**:
-layers are distributed across the cards and the boundary activation is a
-plain per-backend copy, once per forward pass. No RCCL kernel is ever launched
-on the second card, so the atomics gate never comes up.
-
-That is what `frank-medium` ran from 2026-09-16 to 09-20
-([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards,
-262144 context. Measured against tensor split
-on this host: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 /
-26.3 tensor (butterfly fallback, no RCCL). **That tensor number is not a
-prediction for an RCCL-backed build** — it is what tensor split costs when the
-allreduce degrades to a meta-backend butterfly, and it is the number the
-board-swap path exists to retire.
-
-The ranking therefore changed twice. On 2026-09-16: software already gets both
-cards onto one model, so the hardware fix stopped being a capability question
-and became a performance one — *tensor* parallelism, ~40 % decode, plus one
-pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
-hardware half of that question was answered by the board swap; the llama.cpp
-measurement half never ran, and `GGML_HIP_RCCL` still defaults `OFF` in
-`scripts/build-llama-rdna4.sh:41` (re-read 2026-10-18) for exactly the reason
-this page documents — moot while no llama.cpp shape is deployed on this host.
-
-## What would change the answer
-
-- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot
-  since 2026-09-20.** That door only mattered while the hardware gate could not
-  be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis
-  and its counter-evidence stay in the escape table for the record.
-- The symmetric-kernel path inside RCCL, if a release ever ships the device API
-  (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock
-  nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does
-  not.
-- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic
-  routing~~ — arrived as new silicon instead.
-- A GPU-side workaround: none found in ROCclr, and the gate is per-device,
-  queried once at probe.
-
-## See Also
-
-- [[Pipeline Parallelism/index|Pipeline Parallelism]] — two-machine research (the PP variant of this problem)
-- [[Dankserver System Specs]] — hardware inventory
-- [[Observability]] — the scrape/dashboard/alert stack around these endpoints

serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

@@ -1,436 +0,0 @@
----
-type: research
-title: FastFlowLM NPU Inference
-description: FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.
-tags:
-  - fastflowlm
-  - npu
-  - xdna2
-  - dankclaw
-  - research
-timestamp: 2026-09-03
----
-
-> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
-> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
-> The `frank-zealot` name and port 52625 then belonged to the vLLM
-> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
->
-> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today
-> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625
-> (`ss -ltn`, read 2026-10-18). The runtime is still installed
-> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`,
-> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is
-> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
-> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
-> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
-> the RTX 3090 ([[frank-zealot-llama-prism-bonsai2-pq20]]). See [[frank-npu]]
-> for the retirement record. Everything below is the 2026-08 research record.
-
-# FastFlowLM NPU Inference
-
-FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.
-
-## Verdict (2026-08-16)
-
-- The NPU is a complement, not a replacement. It fits dense models up to about 9B.
-- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
-- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
-- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
-- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
-- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`.
-
-## What FastFlowLM Is
-
-FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.
-
-Key properties, per the project README and docs:
-
-- Runs fully on the NPU. No GPU or CPU load during inference.
-- CLI mode (`flm run <tag>`) and server mode (`flm serve <tag>`).
-- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
-- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
-- Context lengths up to 256k tokens on listed models.
-- Vision, audio, embedding, and MoE model support.
-- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
-- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`.
-
-## NPU Hardware on Dankclaw
-
-Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS (vendor spec: "NPU TOPS — Up to 50 TOPS", [amd.com product page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)). It sits beside the Radeon 8060S iGPU and shares the same unified memory pool — 30 GiB exposed to the OS in the 2026-08 reading below; the box presents **121 GiB** today (`free -g` total 121, read 2026-10-18).
-
-Live verification on Dankclaw, 2026-08-16:
-
-```
-$ flm validate
-[FLM]  Fetching models from: /opt/fastflowlm/share/flm/model_list.json
-[Linux]  Kernel: 7.0.0-29-generic
-[Linux]  NPU: /dev/accel/accel0 with 8 columns
-[Linux]  NPU FW Version: 1.1.2.65
-[Linux]  amdxdna version: 0.7
-[ERROR]  Memlock limit is too low (8MB). Please raise the limit or set to infinity.
-```
-
-- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18).
-- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
-- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`).
-- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
-- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".
-
-## Memory Model
-
-The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.
-
-Budget math for this fleet:
-
-| Machine | RAM | NPU budget (approx. half) |
-|---------|-----|---------------------------|
-| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB |
-| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
-| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
-
-2026-10-18 re-read: Dankclaw's OS now reports **121 GiB** (`free -g`) and
-Dankserver **246 GiB** (`free -g`), so the half-of-RAM rule would price the NPU
-budget at ~60 GiB and the 7700X box still has no NPU. The 30 GiB / 62 GB rows
-are the 2026-08-16 state; the table is moot while no NPU endpoint is deployed.
-
-Weight sizes (approximate, Q4-class):
-
-| Model | Quant | Weights | Fits ~15 GiB budget? |
-|-------|-------|---------|----------------------|
-| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes |
-| Qwen3-8B | Q4_1 | ~5 GB | yes |
-| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge |
-| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB |
-| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists |
-
-Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.
-
-## Supported Models (Qwen family, plus gpt-oss)
-
-All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.
-
-| Tag | Base model | Context max | Notes |
-|-----|-----------|-------------|-------|
-| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle |
-| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle |
-| `qwen3:4b` | Qwen3-4B | 32k | tool calling |
-| `qwen3:8b` | Qwen3-8B | 32k | tool calling |
-| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking |
-| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling |
-| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision |
-| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | |
-| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision |
-| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle |
-| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling |
-| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling |
-| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling |
-| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling |
-| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM |
-
-Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.
-
-## Published Benchmarks (FLM, vendor numbers)
-
-All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.
-
-Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:
-
-| Model | 1k | 2k | 4k | 8k | 16k | 32k |
-|-------|-----:|-----:|-----:|-----:|-----:|-----:|
-| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 |
-| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 |
-| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 |
-| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 |
-
-Prefill t/s for the same run:
-
-| Model | 1k | 2k | 4k | 8k | 16k | 32k |
-|-------|-----:|-----:|-----:|-----:|-----:|-----:|
-| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 |
-| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 |
-| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 |
-| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 |
-
-Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**:
-
-| Metric | 1k | 2k | 4k | 8k | 16k | 32k |
-|--------|-----:|-----:|-----:|-----:|-----:|-----:|
-| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 |
-| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 |
-
-Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology.
-
-The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.
-
-## Fit to the Frank Program
-
-| Our model | NPU verdict |
-|-----------|-------------|
-| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. |
-| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. |
-| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. |
-| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. |
-
-Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.
-
-## Install State on Dankclaw (verified 2026-08-16)
-
-- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
-- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
-- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.)
-- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
-- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
-- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
-- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
-- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
-
-## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
-
-Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.
-
-| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % |
-|-----|-----------:|----------:|-------:|------------:|-----------:|---------:|
-| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 |
-| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 |
-| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a |
-
-Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.
-
-Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.
-
-## Model Bakeoff (measured 2026-08-16, 4 models)
-
-All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
-
-| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
-|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
-| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 |
-| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 |
-| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** |
-| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 |
-
-Notes:
-
-- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
-- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
-- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
-- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.
-
-## Promotion (2026-08-16)
-
-`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
-
-### Boot policy (2026-08-17)
-
-The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
-
-```
-sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
-sudo systemctl daemon-reload
-sudo systemctl enable frank-zealot
-```
-
-Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
-
-The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
-
-Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
-
-Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept.
-
-## Parallel Streams (measured 2026-08-16)
-
-The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
-
-Concurrent-request test, gpt-oss-20b, 128-token cap per stream:
-
-| Streams | Wall s | Aggregate t/s | Per-stream wall s |
-|--------:|-------:|--------------:|-------------------|
-| 1 | 10.6 | 12.1 | 10.6 |
-| 2 | 21.1 | 12.0 | 10.8 / 21.1 |
-| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 |
-
-The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.
-
-Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.
-
-## Test Runbook
-
-This runbook targets a retired endpoint; keep it as the procedure if the NPU is
-ever revived. Do not run large model loads while the production service holds
-most of the unified pool — 30 GiB exposed to the OS in the 2026-08 reading,
-**121 GiB** today (`free -g`, read 2026-10-18). Check free RAM first, or stop
-the service the way the bench policy requires (`sudo systemctl stop
-frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
-otherwise restart it). The NPU shares the unified pool with everything else on
-the box.
-
-```mermaid
-graph TD
-    A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
-    B --> C{NPU visible to XRT?}
-    C -- no --> D[Fix driver or firmware path]
-    C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
-    E --> F{Generates tokens?}
-    F -- no --> D
-    F -- yes --> G[4. flm serve on a scratch port, never 7770]
-    G --> H[5. Bench decode + prefill t/s]
-    H --> I{Speed acceptable?}
-    I -- no --> J[Record results, stop]
-    I -- yes --> K[6. Attempt gpt-oss:20b]
-    K --> L{Loads in ~15 GiB?}
-    L -- no --> J
-    L -- yes --> M[Bench, record]
-    M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
-    N --> O[Record, decision]
-```
-
-1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead.
-2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device.
-3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit.
-4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
-5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
-6. Step up in size: 9B, then gpt-oss-20b.
-7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
-
-## Registering packs absent from the catalog (NPU2)
-
-Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
-to the FLM registry. Registry facts: user registry
-`/home/dank/.config/flm/model_list.json`, shape
-`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
-file_url, size, flm_min_version, files[], vlm, default_context_length,
-max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
-resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
-env var set lists custom tags (`⏬` = not downloaded).
-
-1. Back up the registry JSON first (`cp -a`).
-2. Insert the entry via a python json edit + atomic `os.replace`. Set
-   `files` to the FULL HF file list — **including `vision_weight.q4nx` even
-   for text-only packs**: the qwen3.5-family loader opens it unconditionally
-   and a missing file makes the first request HANG silently (no error log).
-3. **Do not use `flm pull <tag>` for custom entries** — it throws
-   `json.exception out_of_range key '<tag>:<size>' not found` while building
-   the download list. Place files manually into
-   `/home/dank/.config/flm/models/<PackName>/` with
-   `curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
-   — plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
-   weights; verify byte counts against the HF tree API before trusting a
-   download.
-4. Kernel packs: copy from the same-shape family pack (shape-match
-   `config.json` on hidden_size/num_hidden_layers/head_dim first) into
-   `/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
-   (root-owned dir).
-5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
-   the new process loads with all files present.
-6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
-   `/v1/models` alone proves registration, not loadability.
-
-Hard constraints: only ONE `flm serve` process can hold the NPU hardware
-context — a second concurrent serve process fails every load with
-`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
-0x2000003); never run scratch/verify instances while production holds the
-NPU. Tag switching happens INSIDE one serve process; the first request
-after a model switch pays ~6–11 s cold-load cost (visible in
-`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
-pattern present in your own command line — use bracket-guarded patterns
-(`[q]wen3.5:4b`) or exact PIDs.
-
-## Tag load-failure diagnosis (200 with an error body)
-
-`POST /v1/chat/completions` answers **HTTP 200** with a flat
-`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
-returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
-200. A status-code-only client reads this as a successful empty completion —
-**always check for the `error` key**. Two distinct causes, distinguished via
-`journalctl -u <flm-service> --full`:
-
-1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
-   Correct behavior, not a bug.
-2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
-   downloads missing weight files on the fly (first request hangs ~90–110 s),
-   then fails `Failed to load model: No such file
-   '/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
-   attempts fail fast (~2 s). Root cause: the serve script exports
-   `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
-   kernel lookup resolves under the user dir, which only contains symlinks
-   someone linked. **A tag listed in `/v1/models` is NOT proof it is
-   loadable on this host.** Fix: one symlink per missing pack
-   (`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
-   /home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
-   symlink the first request loaded and answered in 7.4 s wall.
-
-Switch-cost behavior: the `model` field routes per-request (no restart
-needed); a request whose model differs from the resident one triggers a full
-reload (7–11 s wall for an 8-token completion on ~4B-class packs);
-alternating tags pays the swap each time. Single-lane NPU: concurrent
-requests queue FIFO behind the current lock holder (`NPU Locked!` →
-`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
-the journal for `Failed to load model` (the quoted xclbin path names the
-missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
-installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
-
-## Incident: qwen3.6-moe unreachable (2026-08-16)
-
-Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
-
-1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
-2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
-3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
-
-Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
-
-### Unknown-tag fallback (FLM behavior)
-
-When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`:
-
-- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
-- FLM then loads the 1B model in place of the requested model.
-- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.
-
-**Correction (live-probed 2026-08-30):** the fallback no longer fires on the
-current serve path — an unknown tag now answers HTTP 200 with a flat
-`{"error":"Failed to load <tag> model!"}` body and nothing is loaded or served;
-`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200.
-
-Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model.
-
-### Request compatibility: the `developer` role (found 2026-08-16)
-
-pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection:
-
-- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template.
-- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`.
-- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug.
-- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`.
-- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error.
-
-## Risks and Open Questions
-
-- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
-- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
-- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot
-  of Dankclaw killed it. The fleet law today is the opposite: every deployed
-  container/service ships a boot-persistent, self-healing unit, and the retired
-  NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it
-  cannot come back at boot.
-- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
-- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
-- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then
-  served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the
-  `amd_iommu=off` decision that day and the unit is masked today. The remaining
-  NPU content (benchmarks, runbook steps) is research on a device that is again
-  usable since 2026-10-18 but serves nothing.
-
-## Citations
-
-1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
-2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
-3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
-4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
-5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
-6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
-7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
-8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`.
-9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection.

serverAppConfigs-docs/LLM/FreeToken Builds.md

@@ -1,97 +0,0 @@
----
-type: reference
-title: FreeToken Builds
-description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
-tags: [freetoken, build, dankclaw, vllm, mtp]
-timestamp: 2026-10-05T00:00:00-07:00
----
-
-# FreeToken Builds
-
-FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
-vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
-different dependency traps. Both were verified end-to-end 2026-09-11/12.
-
-## Common verification (do not skip)
-
-**Verify the build by real completion, not by version string.** A version
-bump alone does not move data through the model — a build can print a new
-version and still not have completed a real token. Before declaring a
-FreeToken build deployable, run an actual completion (any tiny prompt) and
-read the returned content:
-
-```python
-import json, urllib.request
-req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
-    json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
-    "content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
-print(json.load(urllib.request.urlopen(req, timeout=600)))
-```
-
-A 200 with a real `choices[0].message.content` is the only build green.
-
-## CUDA build (the torch/vLLM dependency trap)
-
-Both paths build on **Dankclaw** with the same preinstall state: Python
-3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
-download.pytorch.org/whl/nightly/cu130, installed with
-`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
---index-url https://download.pytorch.org/whl/nightly/cu130` — use
-`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
-version). Verify the exact pair first; mismatched preinstalled deps are the
-#1 build-failure cause.
-
-Then install the **matching vllm dev wheel**:
-
-```bash
-uv pip install --index-strategy unsafe-best-match \
-  --index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
-  --extra-index-url https://wheels.vllm.ai/nightly/cu130
-```
-
-The wheel URL pattern is fixed:
-`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
-— read it back from vLLM's CI-run page (`run details →
-build_and_install_package` job, "install the vllm wheel" step). **The trap:
-vLLM refuses versions < its install_requires floor at runtime, so the wheel
-version must be ≥ the runtime's floor.** A wheel built at a lower base
-(0.5.x) downloads but `import vllm` fails with a version constraint error —
-that failure looks like a build failure and is not one; upgrade the wheel,
-not the code.
-
-If `uv pip install -e . --no-deps` refuses (some repos pin the
-`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
-after `python -m venv --copies .venv && source .venv/bin/activate` and
-`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
-re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
-
-## ROCm build (gfx1151 via TheRock nightly)
-
-Same venv discipline, ROCm wheels from TheRock's nightly index:
-
-```bash
-uv pip install --index-strategy unsafe-best-match \
-  --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
-  "torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
-```
-
-Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
-repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
-prebuilt extension wheels into the tree (the repo's `setup.py` skips
-compiling them when found). The ROCm build also needs
-`install-fastrak-gfx1151.sh` from the repo's scripts and
-`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
-x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
-`franksdank/freetoken-ops-gfx1151` — fetch it once and point
-`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
-the extension API changed.
-
-`--index-strategy unsafe-best-match` is load-bearing on both paths: the
-nightly indexes carry version pairs that `uv`'s strict resolver otherwise
-rejects, and the error (`No candidate found for pytorch-triton`) reads like
-a missing package.
-
-## Related
-
-- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
-- [[Llama Cpp Serving Sizing]] — the harness measurement discipline

serverAppConfigs-docs/LLM/GGUF Compatibility.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: GGUF Compatibility and Runtime Lock-in
-description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
+description: "Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can."
 tags: [gguf, llama.cpp, forks, quantization, speculative]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -14,7 +14,7 @@ Studio refuse custom types at load (`unknown tensor type`); a fork-private type
 means exactly one engine can serve the file. Measured on this fleet repeatedly:
 `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
 (`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
-tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
+tables live in  and
 (ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
 
 ## 1. Read the header, do not guess
@@ -69,14 +69,14 @@ filenames lie about quantization and about MTP/vision heads.
    ```
 
    Validate before trusting: the parse is right iff the tensor-info block ends
-   somewhere sane and names look real (`blk.0.attn_q.weight`,
+   somewhere sane and names look real (`blk.0.attn_q.weight`
    `token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
    offset — do not work around it.
 
 3. **Diff what engines actually bind on**: `general.architecture`; tensor
    count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
-   KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
-   `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
+   KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`
+   `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`
    `.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
    (`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
    tool usually comes out byte-identical in every one of these — that is the
@@ -117,7 +117,8 @@ re-download.
 
 ## Related
 
-- [[Llama Cpp Runtimes]] — which fork ships which custom types
-- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
-- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
+-  which fork ships which custom types
+-  the enum tables and code-level differences
+*  the era that served the ftype-115 file, and
+  [[frank-dragoon]] for what runs there now
 - [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads

serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Heterogeneous Split Triage
-description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
+description: "Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys."
 tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -93,7 +93,7 @@ of bytes), so it is tempting. Before proposing it:
   (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
   must be a head **trained for that target** (EAGLE/DFlash/MTP are
   conditioned on the target's hidden states and are not portable between
-  models);
+  models)
 - a plain drafter costs **K sequential forwards per round**, so it must be
   10–50× faster than the target. Break-even with `r = t_draft/t_target`:
   `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
@@ -107,10 +107,10 @@ of bytes), so it is tempting. Before proposing it:
   (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
 
 The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device`
-(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268),
+(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268)
 `--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v`
 (`-ctkd`/`-ctvd`, 4111/4124) all exist in
-`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp),
+`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)
 read 2026-10-18).
 
 ## 6. Expressing a split (llama.cpp, the usual vehicle)
@@ -122,8 +122,8 @@ read 2026-10-18).
 
 - buffer types match by name at load (`ggml_backend_buft_name`), so device
   names work as `-ot` values;
-- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`,
-  `common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`;
+- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`
+  `common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`
 - several backends in one binary is fine (CUDA + Vulkan is the practical
   pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs
   do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the
@@ -155,7 +155,7 @@ Rank and report; prefer the boring answer when it wins:
 1. **Two independent endpoints** (zero shared traffic) — often the real
    answer to "what's a useful combination", especially over a narrow link.
 2. **One process, weights-resident, KV co-located with attention on the
-   fast pool** — promising and quantifiable (+% decode at long context);
+   fast pool** — promising and quantifiable (+% decode at long context)
    needs a dual-backend build and a baseline measurement.
 3. **Hot-expert cache / per-token weight streaming** — affordable only if
    the miss set per token is small (a few experts; check
@@ -169,5 +169,5 @@ Rank and report; prefer the boring answer when it wins:
 ## Related
 
 - [[PCIe Link and Atomics]] — the link and collective walls underneath
-- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
+-  the deployed split shapes
 - [[Llama Cpp Serving Sizing]] — KV and pool arithmetic

serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md

@@ -1,222 +0,0 @@
----
-type: reference
-description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing."
-tags:
-  - llm
-timestamp: 2026-08-28
----
-
-# Hybrid CPU GPU Processing Dankserver
-
-## Overview
-
-Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]).
-
-## Architecture
-
-### Model Weights
-
-In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2).
-
-### KV Cache
-
-KV cache moves between GPU and CPU:
-
-1. Active KV cache pages stay in GPU VRAM
-2. Inactive pages swap to CPU RAM
-3. Swapped pages write to disk swap space if RAM is full
-4. Pages reload to GPU when needed
-
-### Compute Distribution
-
-- GPU handles all matrix multiplication and attention computation
-- CPU handles tokenization, detokenization, and input processing
-- CPU handles KV cache swap operations
-- GPU worker processes run on CPU cores via multiprocessing
-
-## Optimal Settings for Dankserver
-
-### GPU Memory Utilization
-
-Set `--gpu-memory-utilization` based on your GPU:
-
-| GPU | Recommended Value |
-|-----|-------------------|
-| 24 GB (RTX 3090/4090) | 0.7-0.8 |
-| 40 GB (A100) | 0.8-0.9 |
-| 80 GB (A100 80GB) | 0.85-0.95 |
-| 80 GB (H100) | 0.9-0.95 |
-
-Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s.
-The live engine does not use this knob at all — it claims an explicit per-card
-pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`,
-`scripts/frank-archon.env:230-231`).
-
-Lower values leave more GPU memory for KV cache. This reduces swap frequency.
-
-### Swap Space / KV Offload
-
-Recent vLLM releases removed the `--swap-space` engine argument — `CacheConfig` on `main` has no `swap_space` field ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/cache.py), read 2026-10-18). Use `--kv-offloading-size` (`kv_offloading_size`, same file) to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
-
-| System RAM | Recommended Swap |
-|------------|------------------|
-| 32 GB | 8-16 GB |
-| 64 GB | 16-32 GB |
-| 128 GB | 32-64 GB |
-| 256 GB | 64-128 GB |
-
-More swap space allows longer contexts and more concurrent requests.
-
-### Tensor Parallelism
-
-Use tensor parallelism for large models:
-
-```
-vllm serve meta-llama/Llama-3.1-70B-Instruct \
-    --tensor-parallel-size 4 \
-    --gpu-memory-utilization 0.85
-```
-
-### KV Cache Data Type
-
-Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache).
-
-```
-vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
-    --kv-cache-dtype fp8 \
-    --gpu-memory-utilization 0.9
-```
-
-FP8 KV cache reduces memory usage by 50% compared to BF16.
-
-## NUMA Optimization
-
-Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies:
-
-```
-vllm serve model \
-    --tensor-parallel-size 4 \
-    --numa-bind
-```
-
-This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.
-
-For explicit NUMA binding:
-
-```
-vllm serve model \
-    --tensor-parallel-size 4 \
-    --numa-bind \
-    --numa-bind-nodes 0 0 1 1
-```
-
-## CPU Resource Requirements
-
-VLLM needs CPU cores for:
-
-- 1 API server process
-- 1 engine core process
-- N GPU worker processes (one per GPU)
-
-Minimum physical cores = 2 + N (where N is GPU count).
-
-For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read
-2026-10-18):
-
-- Minimum 4 physical CPU cores by the 2 + N rule
-- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off)
-
-## Performance Tuning
-
-### Chunked Prefill
-
-Chunked prefill is enabled by default. Tune with:
-
-```
-vllm serve model \
-    --max-num-batched-tokens 8192
-```
-
-Higher values improve throughput. Lower values improve latency.
-
-### CUDA Graphs
-
-CUDA graphs improve throughput. Disable for faster startup:
-
-```
-vllm serve model \
-    --enforce-eager
-```
-
-### Compilation Cache
-
-Reuse torch.compile artifacts:
-
-```
-export VLLM_CACHE_ROOT=~/.cache/vllm
-vllm serve model
-```
-
-Copy cache between boots for faster startup.
-
-### Skip Memory Profiling
-
-Skip memory profiling on repeated boots:
-
-```
-vllm serve model \
-    --kv-cache-memory <value_from_first_boot_log>
-```
-
-## Monitoring
-
-Check GPU memory:
-
-```
-rocm-smi
-```
-
-Check for preemption warnings in VLLM logs.
-
-Check CPU utilization:
-
-```
-htop
-```
-
-## Example Commands
-
-### Single GPU, Medium Model
-
-```
-vllm serve Qwen/Qwen3.6-35B-A3B \
-    --gpu-memory-utilization 0.85 \
-    --max-num-batched-tokens 4096 \
-    --max-num-seqs 128
-```
-
-### Single GPU, FP8 Model
-
-```
-vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
-    --kv-cache-dtype fp8 \
-    --gpu-memory-utilization 0.9 \
-    --max-num-batched-tokens 8192
-```
-
-### Multi-GPU, Large Model
-
-```
-vllm serve meta-llama/Llama-3.1-70B-Instruct \
-    --tensor-parallel-size 4 \
-    --gpu-memory-utilization 0.9 \
-    --max-model-len 8192
-```
-
-## See Also
-
-- [[VLLM Overview]]
-- [[VLLM Index]]
-- [[RAM CPU Offloading in VLLM]]
-- [[Shared Memory Architecture Dankclaw]]
-- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]

serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md

@@ -1,271 +0,0 @@
----
-type: reference
-title: LLM Bench Windows and Measurement
-description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
-tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
-timestamp: 2026-10-05T00:00:00-07:00
----
-
-# LLM Bench Windows and Measurement
-
-Fleet law first: **STOP the production service before any GPU experiment**
-(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
-GPU and corrupts results; an unisolated GPU experiment has precedent for
-wedging a host until power cycle. Reference harnesses:
-`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
-`scripts/autoresearch/frank-archon-arms.sh`.
-
-## 1. The window structure (one config per arm, a driver owns the window)
-
-1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
-   no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
-   EXITED containers and can never clear — `rm -f` the exited one first) AND
-   port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
-   `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
-   then polls `/health` gets **200 from the still-running PRODUCTION engine**
-   — the arm "passes" with production's boot log, then `podman exec` dies
-   with `container state improper`.
-2. **Boot the arm as the service, not a scratch port** (no scratch ports on
-   LLM hosts — one card, one server). The serve script takes its config via
-   env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
-   pointing at a staged copy; the checkout is never mutated.
-3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
-   the loop — a refused boot dies in one poll instead of 20 min. Capture the
-   serve script's FULL stdout on failure (a `tail -8` loses engine-init root
-   causes; raise happens above the tail).
-4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
-   SpecDecoding lines) — they survive measure crashes.
-5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
-   `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
-   before the next boot; restore production from the EXIT trap and verify the
-   **conjunction oracle**: health 200 + `is-active` + `is-enabled` +
-   mask-symlink clean. A bare `systemctl start` is not a restore oracle.
-   (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
-   enough: systemd never auto-restarts an explicitly stopped unit.)
-6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
-   mixed speeds.
-
-**Destructive tails (2026-10-01):** a scratch script whose tail does
-`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
-probe loop survived the driver kill — it can destroy a just-deployed
-production engine (signature: deploy exits green, then seconds-to-minutes
-later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
-crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
-and kill strays by explicit PID; killing the dev-side job does NOT kill the
-host-side `deploy-remote.sh` (two of them once managed the same unit).
-
-Two failure shapes inside the window that read as "the bench died":
-
-- **Stale reads from a stopping container.** `podman stop` answers while the
-  engine inside is still flushing; a `podman logs` taken in that window
-  returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
-  `podman wait`) before reading logs, or you will write "no results" into
-  the table for a run that has them.
-- **A D-state worker wedges the stop.** A bench child that wedged in the
-  kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
-  then the container is `Stopping` forever. Kill by container first
-  (`podman kill <ctr>` → SIGKILL path), verify the process is gone from
-  `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
-  window script proceed to a redeploy with a zombie bench worker holding
-  VRAM.
-
-## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
-
-**What llama-bench cannot express — decide BEFORE designing:** no
-`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
-activation buffers fit beside the DEPLOYED KV — only the real unit answers
-that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
-(`tg` numbers are the drafterless baseline, useful as a control only), and a
-warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
-cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
-~zero depth and meaningless for long conversations.
-
-**The separator trap:** `/` separates devices WITHIN one configuration, `,`
-separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
-single value 36 → everything on one card. `--tensor-split 36,29` (server) →
-`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
-
-**Harness shape:** one invocation per cell (an infeasible cell calls
-`exit(1)` and kills every later cell of a cross-product invocation — and
-infeasibility is usually the question); TSV row per cell as it completes +
-`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
---showmeminfo vram` VRAM trace per cell (a cell that allocated less than
-intended looks fast); resume by skipping non-empty cell outputs; stop the
-service in the harness, restore from `trap … EXIT`, verify afterwards. Match
-the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
-
-**Run under a transient unit, not a shell:**
-
-```bash
-sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
-     --property=LimitMEMLOCK=infinity --property=User=dank \
-     /home/dank/serverAppConfigs/scripts/<harness>.sh
-```
-
-Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
-8 MiB and fails the load for an unrelated reason. This also survives ssh
-disconnects. **Boot rungs under the production environment, not a bench
-shell** — a rootless shell probe booted 3×262144 bf16 that production refused
-(systemd `User=` + memlock change what the pinned-expert path holds):
-`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
-Restart=no env VARFILE=… /path/serve.sh`.
-
-**Verify before burning hours:** dry-run against a STUB binary with
-`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
-including one forced-`exit 3` cell — prove the TSV row, failure capture,
-resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
-nonexistent file. **Reading:** bracket the sweep (reference cell first AND
-last); check per-cell `stddev_ts` (a difference smaller than spread is not a
-result); single-sequence decode is usually flat across batch/ubatch (prefill
-knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
-smaller-batch cells); report what the grid cannot prove (drafter, slots, real
-context).
-
-## 3. HTTP benches of server-side parameters
-
-`llama-bench` benches the bench binary, not the running server — server-side
-params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
-HTTP:
-
-- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
-  `prompt_tokens / wall_time`.
-- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
-  spec acceptance all stabilize over thousands of tokens); measure
-  `predicted_per_second` from `timings` or `completion_tokens / wall_time`;
-  temperature 0.
-
-Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
-`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
-sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
-reasoning parser streams zero deltas — use `/v1/completions`.
-
-## 4. The measurement law (under speculative decoding this is absolute)
-
-**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
-mapping; chunk counting reported 8.3 tok/s for a config whose server counters
-said 23–26 — a 3x false regression that nearly abandoned the change that
-worked. Always:
-
-```python
-decode_tok_s = usage.completion_tokens / (wall - ttft)
-# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
-```
-
-vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
-measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
-them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
-acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
-temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
-(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
-repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
-before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
-nearly shipped the wrong setting.
-
-## 5. Honest methodology
-
-- **Match the corpus to the workload.** Token soup under-reports speculative
-  decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
-  settings; acceptance, not step time, moves throughput. Build the corpus
-  from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
-  `--custom-output-len`.
-- **Salt every run** with a unique per-arm token in the first bytes — it is
-  the prefix-cache key (and a proxy routing key). Without it arm 2 measures
-  arm 1's warm cache.
-- **Two workload shapes need two clients:** distinct prompts (batch
-  behaviour, prefill parallelism) and sequential turns with growing history
-  (TTFT of the first streamed chunk per turn — the only measurement that
-  exposes prefix-cache behaviour). These pull in opposite directions: a cold
-  batch spreads prefill across replicas, a conversation's next turn only pays
-  if it lands where its prefix is cached.
-- **Read the engine's ledger, not the card:** `Available KV cache memory` /
-  `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
-  request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
-  is a LIFETIME average across every prompt the replica ever saw — send the
-  same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
-  → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
-  full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
-  chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
-  three points before treating a knob as a lever.
-- **Soak before believing:** 15–30 min of concurrent families, count failures
-  AND latency; report hit/miss split and medians with min/max — the median of
-  a bimodal distribution describes neither mode.
-- **Verification that is not the benchmark:** run the client the fleet
-  actually uses; and a deployed unit is not proven until an unattended reboot
-  brings it back.
-- **Record the "no" answers** with numbers and mechanism — placement balance,
-  concurrency admission and chunk-size tweaks all measured flat or worse
-  here; the wall was arithmetic (cached prefixes + live sequences share one
-  pool).
-
-## 6. Sweep hygiene (the arms that produced wrong verdicts)
-
-- **Prefetch uniformity:** every arm in a sweep carries the same
-  expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
-  prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
-  12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
-  verdict.
-- **Knob-landed preflight:** for every `-e` knob, confirm
-  `podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
-  produces a "parity" arm that is really the default (v1 of the gather sweep
-  lost all 3 arms this way; serve-script hooks are additive and can drop
-  vars).
-- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
-  deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
-  compare against default-off, not "incumbent + one flag".
-- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
-  autofit (remainder) so pool numbers are reproducible; compute need =
-  tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
-  states — measure, don't assume 2x).
-- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
-  deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
-  one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
-- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
-  `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
-
-## 7. Bash laws that ate windows (each cost an arm)
-
-1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
-   `SPEC=3: command not found`, nothing runs. Pass literal assignments:
-   `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
-2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
-   an if/else it aborts ONLY that compound statement — the script CONTINUES,
-   every arm tears down clean with zero measurements. Detection law: `ls`
-   the per-arm `.measure.txt` files before trusting any summary; arm echoes
-   are not evidence.
-3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
-   it aborts AFTER the harness stopped production. Two statements.
-4. `sed -i` renames the file; already-running bash keeps the OLD inode —
-   mid-flight patches miss arms launched in the preceding minutes. Patch
-   early or run a make-up window.
-5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
-   (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
-   fails silently. Same trap in `git commit -m`: backticks become command
-   substitution.
-6. No comments INSIDE a backslash-continued command (the continuation joins
-   the comment); verify multi-line command construction with `bash -n` AND a
-   render test.
-
-## 8. Results location and recovery (frank-archon arms)
-
-| copy | path |
-|---|---|
-| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
-| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
-| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
-| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
-| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
-
-`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
-constants 99.9/88.8 identical across arms by construction; **identical
-numbers across arms == stub or bug, never a measurement.** After compaction,
-re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
-logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
-window.
-
-## Related
-
-- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
-- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
-- [[LLM Performance Diagnosis]] — profiling and attribution methods
-- [[Remote Job Runner]] — detached execution of long sweeps
-- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history

serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md

@@ -1,9 +1,9 @@
 ---
 type: reference
 title: LLM Performance Diagnosis
-description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
+description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution."
 tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
-timestamp: 2026-10-05T00:00:00-07:00
+timestamp: 2026-10-19T00:00:00-07:00
 ---
 
 # LLM Performance Diagnosis
@@ -20,8 +20,8 @@ Sweep decode across context depth. **Flat** ⇒ attention over history is not
 the cost; the problem is per-token expert movement and KV/attention tuning is
 ruled out entirely. A genuinely context-bound server shows a clear downward
 slope. Record cold vs warm separately — a small gap means the cache is
-working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
-to 16,095 tokens.
+working. The canonical flat run (fp8 expert-offload shape, a few hundred to
+sixteen thousand tokens of context) is in 's profile.
 
 ## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
 
@@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
 
 No amount of link arithmetic would have produced those two conclusions: the
 suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
-whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
+whose own boot log names the cause (`FULL_DECODE_ONLY` override). It
 predicted a negative result in advance (spec decode cannot help a shape whose
 per-token cost scales with the token count) which matched an unexplained flat
 MTP measurement.
@@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing.
 - One `max_tokens=200` request is production traffic and fine; anything that
   pins the card is a bench → stop-and-mask first. Check WHICH instance you
   are reading (`systemctl status | head -12`; task ids jump across
-  restarts). The engine's own ledger (`KV self size`, `n_slots`,
+  restarts). The engine's own ledger (`KV self size`, `n_slots`
   `n_ctx_slot`) outranks your model of it.
 
 ## 5. The fork's data path (before blaming the link)
@@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as
 draft depth grows. Check the threshold's premise: a default tuned at
 12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
 several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
-does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
+does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required)
 check for the skip line before concluding autotuning ran. Gather-geometry
 knobs read at import time → each arm is a full reload.
 
@@ -161,7 +161,7 @@ of the same RAM, not extra memory.
 ## 7. IBS model-access attribution (AMD CPUs)
 
 Real CPU access patterns for mmap'd weights, live, no stop (established on
-frank-archon, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
+[[frank-archon]], 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
 `scripts/frank-archon-expert-activity.py` (self-contained dump + sample +
 analyze; `--analyze-only`).
 
@@ -189,7 +189,7 @@ analyze; `--analyze-only`).
 Measure with a **sequential-turn client**, not only a batch benchmark — a
 batch of distinct prompts parallelises prefill across replicas and hides the
 problem. The two failure modes: round-robin loses every other cache hit
-(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
+(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned)
 pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
 pinned). **The rule that gets both: route by request shape** — a request
 carrying more than one message is a continuation (agent loop re-sending its
@@ -209,13 +209,79 @@ is not a per-request verdict (probe directly: same prompt twice to an idle
 replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
 `vllm bench serve` needs `--tokenizer <local path>`; verify metric names
 against the live endpoint before dashboards. Test shape: drive the routing
-function directly with synthetic bodies (stdlib-only policy module);
+function directly with synthetic bodies (stdlib-only policy module)
 conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
 hashes differently per turn and pinning silently never happens.
 
+## 9. Decode cadence under a prefill (the step-shape census)
+
+"Decode turns to sludge the moment a long prompt arrives, then recovers" is not a
+link or a gather story — on a chunked-prefill engine the unit of interference is
+**the step**. A step that carries a prefill chunk takes the chunk's time, and every
+decode row sharing that step waits that long for its next token. Chunk size is
+`min(remaining, --checkpoint-interval, --max-num-batched-tokens)`, so on a hybrid
+model the interval caps it before the budget does.
+
+Census it without touching the model — the engine's own counters (`/stats`,
+`/metrics`, cumulative since boot):
+
+```
+decode-only step = step_ns_decode / steps_decode
+shared step      = (step_ns - step_ns_decode) / (steps_prefill + steps_mixed)
+```
+
+Then bucket a day of `decode_tokens_per_second` by `prefill_tokens_per_second`
+(Prometheus `query_range`, step 120 s) and **check the lag**: correlate decode with
+prefill at lag 0 and again at 2-14 minutes. Lag-0-only ⇒ step-time contention, the
+chunk is holding the device. A lagged tail ⇒ residency churn — the prefill's working
+set evicted something decode needs (expert slab, KV) and decode pays after the
+prefill is over. `preemptions` separates the third shape: a preempted-and-recomputed
+sequence stops producing entirely and then re-prefills from token 0, which is a
+stall, not sludge.
+
+Knobs, in the order worth trying them (radiance GUIDE §4.5):
+
+- `--prefill-interleave N` — after a chunk step taken beside decoders, the next N
+  steps carry decode rows only and admit nothing. Decoders keep their cadence; the
+  prompt pays that in TTFT. Counts as `engine_steps_held_total`.
+- `--busy-prefill-chunk MODE` — the chunk's share of a step that has other work:
+  `auto` (default) reserves a row plus the draft window per decode row behind it and
+  an eighth of the budget per waiting prefill/request, `off` lets the chunk take the
+  whole step (admission never runs while it does), `N` pins the share. Never under
+  the prefill stager's smallest pass — the startup line names it, and below it every
+  routed layer reads its pooled experts synchronously, which costs the prefill far
+  more than sharing saves. Counts as `engine_prefill_chunks_capped_total`.
+- `--max-num-batched-tokens` — the blunt rung, and the only one older pins have. The
+  vendor's curve on a 48-layer MoE at 200k on two cards: chunk 512 → 1750 t/s, 1024
+  → 2200, 2048 → 2706 (a 185k prompt: 138 s against 91 s), so halving the stall
+  costs ~19 % of prefill. Raising it past the interval buys nothing and loses expert
+  residency to the arena.
+- `--prefill-span N` — the inverse trade: a *lone* prefill carries N intervals a step
+  while every shared step is planned at a smaller shared budget, which on an
+  expert-tiered box lends that arena back to the expert slab. Vendor TP2
+  expert-tiered TTFT deltas vs a plain 2048 budget: −6.1 % at 2k, +2.3 % at 18k,
+  +49.1 % at 250k — long prompts only.
+
+Two traps. **The arena/slab coupling makes a shared step doubly expensive on an
+expert-tiered shape**: the full-plan arena takes VRAM the resident expert slab would
+have had (320 MiB/card at TP2 for a 4096 budget), so the decode rows inside that step
+also read experts over the link — a smaller shared budget helps twice. And **re-check
+`prefix_hit_rate` / `prefix_evictions` after any chunk change**: the arena and the KV
+pool are the same memory, and the failure looks like turn two taking as long as turn
+one.
+
+Availability is a version question, not a config question: `--busy-prefill-chunk` and
+`--prefill-interleave` are absent from older pins — prove it on the running container
+(`podman exec <c> <binary> --help | grep -iE "prefill|chunk"`), never from the vendored
+tree, and check the source clone's depth before dating a flag from `git log -S`
+(a `--depth 1` clone reports its own tip for every search).
+
+Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = interval
+2048) — the measured census and the verdict for this fleet's shape are that page's.
+
 ## Related
 
-- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
+-  the window discipline and usage-counter law
 - [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
-- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
+- [[frank-archon]] — the endpoint where §9's step-shape census was run

serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md

@@ -1,287 +0,0 @@
----
-type: reference
-title: Llama Cpp Multi-Device Splits
-description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
-tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
-timestamp: 2026-10-05T00:00:00-07:00
----
-
-# Llama Cpp Multi-Device Splits
-
-## 1. Expressibility law — classify the arrangement before running anything
-
-llama.cpp expresses exactly two things across devices, and they are not
-interchangeable:
-
-| arrangement | how | expressible? |
-|---|---|---|
-| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
-| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
-
-Upstream anchors, re-read 2026-10-18 against `master`:
-`src/llama-context.cpp:517-526` computes `pipeline_parallel` with
-`!model.has_tensor_overrides()` as a required term;
-`ggml/src/ggml-backend.cpp:1089` carries the verbatim abort
-`pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)`
-(the `:941` anchor above is where it sat in the `37b53fd` build this was
-measured on); and `common/arg.cpp` defines `-sm/--split-mode` (2821),
-`-fit/--fit` (2882), `--spec-type` (4295), `--spec-draft-device` (4259),
-`--spec-draft-n-max` (4161), `-fa/--flash-attn` (1758) — every flag in §4 is
-real upstream, not fork-only
-([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)).
-
-Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
-with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
-cross-backend pair → model load fails with only `failed to load model`;
-`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
-is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
-a card can be the trunk OR the offload host, never the complement of an
-AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
-override, not the driver mix.
-
-If the plan needs tensors on a specific device: stop, the answer is no for
-any vendor mix. Expressible alternatives, in order: (1) two independent
-endpoints, one model per device; (2) single-vendor layer split; (3) device +
-CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
-was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
-land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
-
-Layer-split arithmetic to check before promising: a whole layer carries its
-experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
-KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
-non-repeating tensors (PLE tables) land on the main device and cannot move —
-they cap how small the primary can get.
-
-## 2. No collectives? The atomics wall and the layer-split answer
-
-Decide with one line before designing:
-
-```bash
-sudo lspci -vv -s <card2> | grep AtomicOpsCtl   # ReqEn- ⇒ no collectives, ever
-lspci -tv                                        # a Routing- bridge is a hardware wall
-```
-
-`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
-device-backend `send`/`recv` of the hidden state
-(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
-fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
-collective: layers distribute in order, only the boundary activation crosses
-PCIe once per forward pass. Full probe/repair detail:
-[[PCIe Link and Atomics]].
-
-## 3. Build: pin by feature, bundle the runtime at the exact depth
-
-- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
-  `ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
-  (`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
-- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
-  segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
-  mode degrades to the butterfly fallback; layer split unaffected).
-- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
-  directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
-  (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
-  the engine loads every GiB, answers `/health`, and dies on the FIRST request
-  with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
-  arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
-  `patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
-- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
-  coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
-  `glslc` + libvulkan headers. **Device names are not what you expect:**
-  `--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
-  `Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
-  llama.cpp print the valid set and exit (self-diagnosing).
-- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
-  CMakeCache.
-
-## 4. Serve flags that matter on a split
-
-```bash
-HIP_VISIBLE_DEVICES=0,1 llama-server \
-  --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
-  --gpu-layers 999 --fit off --flash-attn on \
-  --ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
-  --jinja --metrics --cache-prompt \
-  --spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
-  --spec-draft-device ROCm1 --spec-draft-n-max 7
-```
-
-- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
-  on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
-  is the concurrency policy, concurrent agent loops queue.
-- Pin the drafter to the card holding the target's LAST layers
-  (`--spec-draft-device`); under tensor mode an unpinned draft joins the
-  meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
-  be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
-- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
-  low/medium/high/xhigh and map harness levels before declaring ready.
-
-## 5. Split balancing: measure, never interpolate
-
-**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
-qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
-a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
-layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
-single step; the best integer split can sit 0.2 GiB from balance while its
-neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
-overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
-a gate; treat a NEGATIVE fitted constant as a unit bug).
-
-Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
-buffers are **~2x heavier on the output/drafter card** — 512→1536 added
-~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
-`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
-only the distribution moves; use that to sanity-check a suspicious row.
-
-**Procedure (one split per load):**
-
-```bash
-WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
-export FRANK_MEDIUM_ENV_FILE=$WORK
-for ts in 39,26 38,27 37,28 36,29 35,30; do
-  sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
-  nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
-  # poll /health (or process exit), then:
-  rocm-smi --showmeminfo vram | grep 'VRAM Total Used'   # sample ~6 s after health
-  pkill -x llama-server; sleep 6
-done
-```
-
-Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
-and poll from a separate call (a load is 30–90 s). Pick the smallest
-inter-card gap that clears the per-card floor with room to spare; one `-ts`
-step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
-not to solve. Prefer balance: the busy card's headroom bounds the context
-(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
-card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
-`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
-not prevent a boundary-copy fault.
-
-## 6. Drafter KV: sized from the TARGET, bounded by its window policy
-
-`common/speculative.cpp` builds the draft context with
-`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — `common/speculative.cpp:2663` on
-`master`, re-read 2026-10-18 — **no flag caps it.** So:
-
-```
-draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
-```
-
-…unless the drafter declares a sliding window. Measured at 262,144 target
-ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
-with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
-balance point, and swapping them invalidated an already-balanced split.
-
-**Failure signature:** the engine loads the whole model, answers `/health`,
-and dies on the FIRST draft decode with `ROCm error: out of memory` /
-`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
-a sizing failure, not a broken drafter; under a self-healing unit it presents
-as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
-`key_length` from the drafter's GGUF header first. A compact reader (the
-three format traps: no KV alignment — v3 alignment applies only to the tensor
-section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
-FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
-is the v2/v3 file order, NOT gguf.h's 2/3):
-
-```python
-import struct, sys
-def rd(path):
-    f = open(path, 'rb'); assert f.read(4) == b'GGUF'
-    struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
-    nk = struct.unpack('<Q', f.read(8))[0]
-    def s():
-        n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
-    SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
-    FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
-    keys = {}
-    for _ in range(nk):
-        k = s(); t = struct.unpack('<I', f.read(4))[0]
-        if t == 8: keys[k] = s()
-        elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
-        else:
-            et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
-            if et == 8: keys[k] = [s() for _ in range(n)]
-            elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
-            else: raise Exception('unsupported arr elem type %d' % et)
-    return keys
-```
-
-## 7. Heterogeneous placement triage (go/no-go with arithmetic)
-
-- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
-  Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
-  vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
-  almost never the constraint; **hop count is** — every boundary is a host
-  sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
-  `graph splits` from the loader log.
-- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
-  context` (attention reads the whole cache every token). Highest-leverage
-  placement: **KV on the fastest-memory device together with its attention
-  compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
-  ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
-  VRAM`.
-- **What placement never buys:** prefill is compute-bound and expert weights
-  are read once per batch — moving them buys nothing and adds a serial stage.
-  Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
-  gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
-  `1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
-  points (assumes sequential stages — true at batch 1).
-- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
-  carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
-  window means the "GPU memory" IS the same LPDDR: adding a stage there adds
-  a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
-- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
-  but needs the exact tokenizer for a plain drafter or a head trained for
-  that target; a plain drafter costs K sequential forwards (break-even needs
-  p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
-  already committing ~1.7 tok/round on its own MTP head is a high bar); and
-  check the runtime accepts external drafters at all (closed engines 400
-  unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
-- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
-  ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
-  capture can abort on unusual graph shapes; make the FIRST measurement a
-  single-device baseline of the runtime on the new device — a split's
-  arithmetic is worthless if the baseline loses to the incumbent; a split
-  consumes the fast device entirely — check what it was serving.
-
-**Verdict ranking:** (1) two independent endpoints — often the real answer;
-(2) one process, weights-resident, KV co-located with attention on the fast
-pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
-overlaps — no mainstream runtime implements it; (4) name what does NOT help:
-TP across unequal devices, KV on the far device, loading over the link,
-prefill-motivated placements.
-
-## 8. Harness syntax and verification
-
-`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
-`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
-`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
-takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
-any placement-dependent number needs an independent memory check
-(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
-conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
-bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
-
-Verification battery: `/health`; both cards' memory ≈ half the weights each
-(busier card also carries the drafter; compare `model_bytes/2 + drafter +
-ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
-N·bytes/2 — neither card's total may equal the model file (that arithmetic
-rules out replication); `spec_decode_num_drafts_total` > 0 and a real
-completion; a deep prompt (140k) proves a raised window is REAL, not
-declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
-`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
-`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
-(default journal output truncates these lines).
-
-Ops traps: a server started over ssh outlives stopping the runner (keeps the
-port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
-sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
-the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
-treating a grep count as a live fault.
-
-## Related
-
-- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
-- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
-- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
-- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
-- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history

serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Llama Cpp Serving Sizing
-description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
+description: "Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache."
 tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -19,18 +19,19 @@ guesswork.
    `n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
    `kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
    state buffer — slot count is a memory term, not free.
-2. **Two boots on the real card** (production stopped, per the bench law),
+2. **Two boots on the real card** (production stopped, per the bench law)
    each near the target; record `nvidia-smi` used/total plus the boot log's
    `n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
    `non-KV = used − (tokens × KiB/token)`; the two rows separate
-   weights/buffers/graphs from the new demand (a 918 MB projector bills
-   ~1.4 GiB).
+   weights/buffers/graphs from the new demand; a projector is billed after the
+   fit at roughly 1.5x its file size ([[frank-zealot]] Contract has that
+   measurement).
 3. **Solve the band:** `KV_tokens ≤ (card − non-KV − headroom_low) /
    KiB/token`, round the pool DOWN to a multiple of `PARALLEL × 4096` so every
    slot gets an equal integer window. Boot and measure; if outside the band,
    walk one step — never re-derive from a formula.
 4. **Propagate the per-slot number — it is the client contract:**
-   `models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
+   `models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window)
    or long conversations die with `400 exceed_context_size_error` instead of
    compacting; compaction/summary budgets sized to the old window get
    re-checked; endpoint docs and the rollback table move too.
@@ -54,8 +55,8 @@ First decide which kind of memory is in question:
 | device | kind | read it with |
 |---|---|---|
 | RTX 3090 (Dankclaw) | discrete GDDR6X | `nvidia-smi --query-gpu=memory.total,memory.used,memory.free` |
-| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` (114 GiB) vs `mem_info_vram_*` (2 GiB carve-out) |
-| R9700s (Dankserver) | discrete | `rocm-smi --showmeminfo vram` → 34,208,743,424 B = 32,624 MiB (31.86 GiB) per card, read 2026-10-18 — **not** the 64 GiB "SVM-backed" figure this row once carried ([[Dankserver System Specs]]) |
+| gfx1151 iGPU (Dankclaw) | **UMA** — "VRAM" is host RAM | `/sys/class/drm/card1/device/mem_info_gtt_{total,used}` vs `mem_info_vram_*`; the aperture and carve-out values are [[Dankclaw System Specs]]'s |
+| R9700s (Dankserver) | discrete | `rocm-smi --showmeminfo vram`; the per-card byte/MiB value (and the stale "64 GiB SVM-backed" figure it replaced) is [[Dankserver System Specs]]'s |
 
 Per-process truth on NVIDIA:
 `nvidia-smi --query-compute-apps=pid,process_name,used_memory`. Never add the
@@ -63,14 +64,15 @@ iGPU's GTT to host RAM as a separate pool; `free -g` is the arbiter for
 co-tenants.
 
 - **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
-  contiguous blocks — the honest fragmentation signal: 240 blocks/12.3 GiB
-  after reboot vs 475/14.7 GiB before means the model did not come back into
-  the same hole), device budget, KV pool/working split, "left for everything
+  contiguous blocks — the honest fragmentation signal: a block census that drops
+  across a restart means the model did not come back into the same hole; the
+  before/after census this was learned on is that era's, on [[frank-dragoon]])
+  device budget, KV pool/working split, "left for everything
   else". llama.cpp prints them at `llama-model.cpp:1827` (per-backend model buffer)
   and `llama-context.cpp:692` (compute buffer) on the era's build; on today's
   `master` the same `LLAMA_LOG_INFO` lines sit at `llama-model.cpp:2009`
   ("model buffer size =") and `llama-context.cpp:842` ("compute buffer size =")
-  ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp),
+  ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp)
   re-read 2026-10-18) — but **`llama-bench`
   suppresses loader INFO and a systemd unit did not put them in the journal**
   (verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
@@ -83,14 +85,13 @@ co-tenants.
   graph/compute pool).
 - **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
   "move layers to the card" converts reclaimable cache into device memory and
-  frees nothing for co-tenants. A served model's footprint ≠ weight size
-(a retired frank-zealot shape: 22.6 of 24.1 GiB for an 11.29 GiB model; the card
-today reads 24,576 MiB = 24.0 GiB total, `nvidia-smi --query-gpu=memory.total`,
-2026-10-18, and the deployed payload is the PQ2_0 pair
-[[frank-zealot-llama-prism-bonsai2-pq20]]). Do the capacity
-  subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
-  28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
-- **Context lengths: derive, never remember.** `llama-bench` computes
+  frees nothing for co-tenants. A served model's footprint is not its weight
+  size: the MiMo Q5_K_M era booted a 6.40 GiB payload to **20,530 MiB used** of the
+  3090's addressable 24,576 — the Eras row and its addressable ≠ advertised law are on
+  [[frank-zealot]]. What the card reports used is weights + KV + compute buffers +
+  projector, never weights alone. Do
+  the capacity subtraction before designing, on the card number the owner page
+  quotes, not the advertised one.
   `n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
   flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
   576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
@@ -112,10 +113,10 @@ them from the binary (`llama-server --help`) before sizing on them.
 **State law** (compute before trusting any cap): a state is the serialized KV
 of the whole sequence + constant recurrent state + the draft context's KV:
 `per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
-draft_per_token`. Measured on frank-medium (qwen3.5 hybrid, f16, MTP draft):
-16×4×256×2×2 = 64 KiB/token + 4 KiB draft = **68 KiB/token**; a full
-196,608-token slot state ≈ 12.8 GiB. Cross-check against the journal's
-`prompt state size` lines — the two must agree.
+draft_per_token`. The worked qwen35-hybrid instance of that formula (full layer
+count, f16 KV, MTP draft) and the slot-size table it prices are on
+; cross-check any such arithmetic against the
+journal's `prompt state size` lines — the two must agree.
 
 **The journal is the instrument:** `exceeds cache size limit … skipping` =
 cap binding in production; `making room … removing oldest` / `cache size
@@ -162,13 +163,13 @@ Only the charlie12345/ROCmFPX build-qwen38 (PR #27742 qwen4exp merged) takes
 it; CIRU, upstream and stock builds do not. Validate live: load with
 `--flash-attn on`, health 200, a real completion finishing `"stop"`, journal
 clean of `GGML_ASSERT`/`cudaMalloc`/`missing tensor`, and flash-attn engaged
-in the timings. Context cap for full-model residency on that shape stayed
-32,768.
+in the timings. The context cap that build reached for full-model residency is
+part of that era's record on [[frank-dragoon]].
 
 ## Related
 
 - [[GGUF Compatibility]] — what a payload is before it is priced
-- [[frank-zealot-llama-mimo-q6k-2x200704]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot-llama-prism-bonsai2-pq20]])
-- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
+- [[frank-zealot]] — the endpoint whose Eras rows are the two-boot priced shapes this method produced
+-  the state-law worked example behind section 3
 - [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
-- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
+- /  — the multi-device side

serverAppConfigs-docs/LLM/Model Swap and Rollback.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Model Swap and Rollback
-description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
+description: "The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert."
 tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -40,9 +40,9 @@ curl -s <health>                                      # then a REAL completion
 ```
 
 Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
-confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
-FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
-quant by size fit, then accept the compute path the card supports. Before
+confirm with a real `/v1/chat/completions`. On Ampere an FP8 file buys memory
+and no compute ([[Quantization by Hardware Architecture]]): choose the quant by
+size fit, then accept the compute path the card supports. Before
 choosing a smaller quant, state WHICH of the two you are buying — headroom,
 or a design change (a smaller quant does not change what the runtime can
 express; per-tensor cross-device placement stays refused under
@@ -51,17 +51,17 @@ express; per-tensor cross-device placement stays refused under
 ## 2. Ground truth first
 
 - Read the HF model card (runtime pin, hard constraints, recommended args)
-  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+  + the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`
   `scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
-- Check the host before downloading anything: `ls /home/dank/models/`,
+- Check the host before downloading anything: `ls /home/dank/models/`
   `df -h /home`, `du -sh` — bench-era or prior-deployment files often already
   exist.
 
 ## 3. Fetch and verify (HF payloads)
 
 **Inventory by QUANT NAME, never by extension** — large quants ship as
-`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
-`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
+`.partNofM` fragments, so an `endswith(".gguf")` filter makes a whole rung look
+nonexistent (AGENTS.md carries the worked fragment ledger). Check both
 mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
 repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
 return 403 even with a valid token until the account accepts the gate —
@@ -129,7 +129,7 @@ arms possible — keep it intact.
   `images` is silently ignored; probe with ≥256 px).
 - vLLM specifics: first request after boot can take ~18 s (deferred Triton
   loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
-  the contract ([[vLLM Boot Forensics and Config Interrogation]]).
+  the contract ().
 
 ## 7. systemd handoff — THE TRAP
 
@@ -150,7 +150,7 @@ recoverable, drift is not.
    entry in `serverAppConfigs-docs/log.md`.
 2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
    context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
-   the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
+  the pool; e.g. a 524288 pool over 2 slots = 262144 per slot); vLLM reports
    `max-model-len` and its concurrency line. Overselling causes silent
    client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
    engine's cap, `input` capabilities (text/image), and
@@ -205,7 +205,7 @@ Pure measurement sessions that changed no state need only pin 1.
   ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
   MMLU-Pro vs Q8_0).
 - GGUF parsing gotcha: expert tensors are
-  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
+  `blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`
   a `len(parts) == 3` filter matches zero layers with no parse error. Filter
   on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
 
@@ -214,7 +214,7 @@ Pure measurement sessions that changed no state need only pin 1.
 When a host must serve two models at once (an iGPU runtime plus a second
 card), or one endpoint is moving and a new one takes the freed hardware.
 
-**0. Decide the shape before touching hardware.** One provider per
+**0. Decide the shape before touching hardware.** One provider
 HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
 — a second endpoint is a new provider block, not a second model in the
 existing one. Read the budget as an inequality: for unified memory the
@@ -235,7 +235,7 @@ suffix — and check what pointed at them: bench harnesses, a chat template
 only the old runtime read, and `scripts/test-changed.sh` mappings are the
 usual victims (a mapping to a retired test file fails the pre-commit
 wrapper silently for weeks). One bats file per service asserting
-INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
+INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation)
 not constants; a negative grep must strip comments first
 (`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
 the very thing you removed.
@@ -252,23 +252,21 @@ small/fast + image roles at the new endpoint; move the vision role WITH the
 vision tower (one owner per capability). `shared/homepage/services.json`: a
 card per endpoint, probed from the monitor's own host. Prometheus: only add
 a job if the runtime actually exposes Prometheus text — and re-measure rather
-than repeat, because that answer moves. This bullet once said halogen serves
-`/health`, `/cache`, `/v1/*` "and nothing else"; measured 2026-10-18 on
-Dankclaw, `curl -s http://127.0.0.1:7770/metrics` returns Prometheus text
-(`# HELP llamacpp:prompt_tokens_total …`, content-type
-`text/plain; version=0.0.4`) while an invented path 404s — so the surface has
-grown and a scrape target is expressible again.
+than repeat, because that answer moves: halogen's HTTP surface has grown twice
+since this page was written, and the endpoint list that matters for a scrape
+target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire
+contract.
 
 **4. Cutover, then verify by driving it.** `deploy.sh <target> --service
-<large> --service <small>` (order matters); per endpoint: `/health`,
+<large> --service <small>` (order matters); per endpoint: `/health`
 `/v1/models` reports the alias, a chat completion returns content AND
 `reasoning_content`, a `tools` request returns `tool_calls`, an image
 request is answered (or refused with a named reason), the runtime's own
 speculation counters move, and the unit is `enabled` and `active`. All three LLM
-endpoints are **root** units today (`User=dank` + `WantedBy=multi-user.target` —
-`scripts/frank-archon.service:30,71`, `scripts/frank-dragoon.service:8,31`,
-`scripts/frank-zealot.service:8,26`), so linger is not what gates them; linger
-matters for rootless quadlets. Read the engine's boot ledger for a silent clamp. Finally spend one
+endpoints are **root** units today (`scripts/frank-archon.service`
+`scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is
+not what gates them; linger matters for rootless quadlets. Read the engine's boot
+ledger for a silent clamp. Finally spend one
 real request through the harness per changed role — routing labels lie; the
 servers' own journals do not.
 
@@ -284,5 +282,6 @@ predecessor "stopped" (the port race above).
 - [[GGUF Compatibility]] — which runtime can load a payload at all
 - [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
-- [[LLM Bench Windows and Measurement]] — measuring the new deployment
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon-radiance-qwen38-iq4r]]
+-  measuring the new deployment
+* [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era
+  (2026-09-29 → 10-05) is a row in its Eras table

serverAppConfigs-docs/LLM/PCIe Link and Atomics.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: PCIe Link and Atomics
-description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
+description: "Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives."
 tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
 timestamp: 2026-10-05T00:00:00-07:00
 ---
@@ -61,17 +61,15 @@ H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
 Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
 LPDDR5X on the Strix Halo — 128 GB LPDDR5x-8000 on a 256-bit bus,
 [amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)).
-**If the link is >10x slower than the memory behind it (x4 is 31x here),
+**If the link is >10x slower than the memory behind it (x4 is 31x here)
 streaming loses by construction and no tuning rescues it; doubling the link
 only doubles a losing number.**
 
-Worked example (Dankclaw 2026-09-16): Gen4 x4 measured H2D **6.72 / D2H 6.60
-GB/s** (85 % of spec) — pinned H2D ≈ async, pageable within 1 %, i.e.
-link-limited not staging-limited. A 3 MiB slab (the expert-streaming
-granularity) = 474 µs. A3B MoE at Q4_K_M (~1.70 GiB active/token) → 272
-ms/token ⇒ **3.7 tok/s** (4.6 with 20 % resident) vs measured **20.46 tok/s**
-with experts computed on the CPU, 25–26.5 on the iGPU. The streaming design
-loses 3–5x.
+Worked example: the Dankclaw Gen4 x4 run of 2026-09-16 — measured H2D/D2H
+rates, the 3 MiB slab cost, and the streaming-vs-compute comparison it decided
+— is recorded on  §The link, measured. Read it once and
+apply the rule above: on that link the streaming design lost by 3–5x, and the
+x8 ceiling would only have halved the loss.
 
 ## 3. Measure the link: two probes, same mechanics
 
@@ -90,7 +88,7 @@ time, so encode them:
   + `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
   segfaulting process loses block-buffered stdout and hides the crash point.
 - Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
-  --offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
+  --offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`
   never `| tail -N` (truncation hid a missing `#include <cstring>`).
 - `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
 - Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
@@ -111,28 +109,28 @@ static double one(size_t bytes, bool pinned, int iters) {
     if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
     if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
     else        { host = malloc(bytes); }
-    memset(host, 0x5a, bytes);
-    hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
-    for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
-    hipDeviceSynchronize();
-    hipEventRecord(a, 0);
-    for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
-    hipEventRecord(b, 0); hipEventSynchronize(b);
-    float ms = 0; hipEventElapsedTime(&ms, a, b);
+    memset(host, 0x5a, bytes)
+    hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b)
+    for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice)
+    hipDeviceSynchronize()
+    hipEventRecord(a, 0)
+    for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice)
+    hipEventRecord(b, 0); hipEventSynchronize(b)
+    float ms = 0; hipEventElapsedTime(&ms, a, b)
     double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
-    if (!pinned) free(host); else hipHostFree(host);
-    hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
+    if (!pinned) free(host); else hipHostFree(host)
+    hipFree(dev); hipEventDestroy(a); hipEventDestroy(b)
     return gb;
 }
 int main(int argc, char **argv) {
-    setvbuf(stdout, NULL, _IONBF, 0);
+    setvbuf(stdout, NULL, _IONBF, 0)
     if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
     size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
     for (int i = 0; i < 6; i++) {
         size_t b = sz[i];
         int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
         printf("  %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
-               b/1048576.0, one(b,true,iters), one(b,false,iters));
+               b/1048576.0, one(b,true,iters), one(b,false,iters))
     }
     return 0;
 }
@@ -150,9 +148,9 @@ int main(int argc, char **argv) {
   (the usual "pageable costs 2x" rule does not apply at this link).
 - Small-transfer penalty (2.345 MiB: ~106 µs vs ~78 µs ideal, ~21 %) is
   per-transfer overhead — recoverable only by BATCHING transfers, not by a
-  faster link. (Dankserver 2026-09-29: bulk 22.7 GB/s at the 2.344 MiB slab =
-  91 % of Gen5 x8, while the engine's effective read path ran ~9.5 GB/s — the
-  gap is the fork's gather software path, not the wire.)
+  faster link. The engine-vs-wire version of that reading on Dankserver
+  (bulk slab rate against the engine's effective gather rate, same
+  granularity) is [[frank-archon]]'s tcclaviger-era record.
 - Both GPUs symmetric ⇒ no cross-device rebalance is available; measure both.
 - Don't quote a derived line from inside a probe without hand-checking it —
   a printf format/argument mismatch in one of these printed a nonsense
@@ -186,7 +184,8 @@ sudo lspci -vv -s <bridge> | grep AtomicOpsCap                   # want Routing+
 `Routing-` is a read-only hardware capability. Typical AM5 layout: one x16
 slot on CPU root ports (atomics fine), the rest behind the chipset (usually
 not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
-batch 4 ≈ 1.4 ms at Gen4 x4 — compute-bound, not interconnect-bound.
+batch 4 is a millisecond-scale figure at x4 — compute-bound, not
+interconnect-bound (full derivation:  §Bandwidth).
 
 **Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
 timeout) — one probe answers several design questions:**
@@ -215,14 +214,13 @@ reporting an outage (`gitea` vs `gitea-server`).
 
 1. CPU-attached M.2 → PCIe x4 riser (~$30) — puts the card on the CPU root
    complex; x4 suffices for TP.
-2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — both
-   R9700s on root ports `00:01.1`/`00:01.3`, every bridge `Routing+`, both
-   endpoints `ReqEn+`). Re-checked 2026-10-18 without root: both root ports
-   report `current_link_width=8` at `32.0 GT/s`
+2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — the
+   per-device chain census, including both endpoints reading `ReqEn+`, is
+   's). Re-checked 2026-10-18 without root: both root
+   ports report `current_link_width=8` at `32.0 GT/s`
    (`/sys/bus/pci/devices/0000:00:01.{1,3}/current_link_width`), and the live
-   engine states direct P2P between the cards
-   ([[Dual-R9700-Cooperation]]).
-3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
+   engine states direct P2P between the cards.
+3. llama.cpp `--split-mode layer` — no collectives at all (); costs the GGUF conversion plus paged attention /
    continuous batching / spec decode.
 4. Independent TP=1 replicas + routing proxy — cards never share one model.
 
@@ -232,7 +230,7 @@ is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
 `CUSTOM_ALL_REDUCE_MAX_SIZES` in
 `vllm/distributed/device_communicators/all_reduce_utils.py`, whose keys on
 today's `main` are exactly `"9.0"`, `"10.0"`, `"10.3"`, `"10.7"`
-([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py),
+([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py)
 read 2026-10-18); nothing keys on a ROCm-reported `12.0`. — and patching it is
 insufficient because TP init's `broadcast`/`barrier`/logits `all_gather` have
 no non-RCCL path. A
@@ -251,6 +249,6 @@ kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
 
 ## Related
 
-- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
-- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
+-  the layer-split path around the wall
+- /  — measured history
+- [[frank-archon]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)

serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md

@@ -1,530 +0,0 @@
----
-type: reference
-title: Ethernet Bandwidth Math
-description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.
-tags:
-  - pipeline-parallelism
-  - research
-timestamp: 2026-08-17
----
-
-# Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
-
-## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17)
-
-Everything in this section was measured on the live deployment
-([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on
-both ends. The theoretical tables below use H=4096/L=32; the real model
-is larger (see constants), which raises the per-token bytes but not the
-conclusions.
-
-### 0.1 Verified constants (from the GGUF + env, not guessed)
-
-| Constant | Value | Source |
-| -------- | ----- | ------ |
-| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) |
-| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
-| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
-| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
-| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
-| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
-| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
-| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
-| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured |
-| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured |
-| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured |
-
-The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more
-jitter (mdev 0.334 vs 0.211 ms).
-
-### 0.2 Measured wire bytes (the number that matters)
-
-Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**.
-The *measured* bytes per position per direction are ~4x that:
-
-| Direction | Measured | Raw hidden |
-| --------- | -------- | ---------- |
-| head -> worker | 39.8 KB/position | 10.0 KB |
-| worker -> head | 38.4 KB/position | 10.0 KB |
-
-The extra ~30 KB/position is the serialized RPC graph (per-position
-nodes for all 45 worker blocks: attention, KV append, SSM ops)
-plus protocol framing. It scales with position count just like the
-payload, so it does not shrink away.
-
-Per ubatch (256 positions) the measured crossing is **20.5 MB**
-(10.4 MB head->worker + 10.1 MB worker->head).
-
-### 0.3 Measured throughput and the validated model
-
-Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE
-path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition:
-
-| Term | 2.5 GbE (measured anchor) |
-| ---- | ------------------------- |
-| wire per ubatch (20.5 MB) | 70 ms |
-| compute + overhead per ubatch (residual) | 489 ms |
-| ubatch total | 559 ms = 2.18 ms/token |
-
-Using that residual and only swapping the wire time per link speed:
-
-| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty |
-| ---- | ----------- | ----------------- | ------------ | -------------------- |
-| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% |
-| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% |
-| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% |
-
-Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**.
-
-### 0.4 The KV snapshot: where the link really saturates
-
-On task release, `llama-server` saves the conversation to the prompt
-cache. The KV for the 11 FA layers living on the worker GPU must be
-copied **worker -> head** for that snapshot, and the slot is BLOCKED
-while it runs (`PARALLEL=1` = one user queues behind it).
-
-Measured for a 98,100-token conversation: 3,454 MB copied (model check:
-98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg,
-308 MB/s peak).
-
-| Link | snapshot duration (slot blocked) |
-| ---- | -------------------------------- |
-| 1 GbE | ~31 s |
-| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") |
-| 10 GbE | ~5 s |
-
-This is the only phase that runs the link at ~100% utilization for a
-long time. Prefill runs at 12-26% of line rate in bursts (per ubatch)
-and ~6-16% on average; decode runs at ~2% or less.
-
-### 0.5 Decode (MTP verify step, 5 positions)
-
-| Link | net time per step | share of 164 ms step | decode t/s |
-| ---- | ----------------- | -------------------- | ---------- |
-| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 |
-| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) |
-| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 |
-
-Decode is latency-touched but bandwidth-trivial. Average wire rate
-~19 Mbit/s in each direction.
-
-### 0.6 Verdict for the 10 GbE USB-C adapter question
-
-- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for
-  free: the claw's native 2.5 GbE `eno1` now carries the RPC path
-  (the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the
-  path, the RPC TCP re-attached on the new interface).
-- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking.
-  It requires the L2 device at 192.168.0.1 to be 10G-capable, or a
-  direct cable between the two machines (dedicated link, second IP
-  pair, or a lower-metric static route).
-- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable.
-
-## 1. Bandwidth Math: Bytes per Token per Layer Boundary
-
-### 1.1 Pipeline Parallelism (PP)
-
-During autoregressive generation, each transformer layer produces a hidden state of shape `(batch_size, hidden_size)`. At a pipeline stage boundary, this tensor must be communicated between nodes.
-
-**Forward pass only (inference):**
-```
-bytes_per_boundary = batch_size × hidden_size × 2   [fp16 = 2 bytes]
-```
-
-**Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length.
-
-**For a 27B dense model (e.g., Qwen3.6-27B):**
-- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures).
-  Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120
-  (GGUF-verified); see section 0, which supersedes these theoretical
-  numbers with measurements.
-- Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB`
-- With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok`
-
-**For a 36B MoE model (e.g., Qwen3.6-35B-A3B):**
-- Hidden size: H = 6144 (common for this scale)
-- Layers: L = 48 (typical)
-- Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB`
-- With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok`
-
-**Full example — 27B model, 2 PP stages (1 boundary), batch=1:**
-```
-1 boundary × 4096 × 2 bytes = 8,192 bytes/tok
-```
-
-**Example — same model, batch=64:**
-```
-1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok
-```
-
-**Example — 36B model, 2 PP stages, batch=1:**
-```
-1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok
-```
-
-### 1.2 Tensor Parallelism (TP) — for comparison
-
-In tensor parallelism, each MLP and attention layer performs an all-reduce on the hidden state:
-
-```
-bytes_per_layer = batch_size × hidden_size × 2   [one all-reduce per layer]
-total_bytes_per_tok = L × batch_size × hidden_size × 2   [L all-reduces total]
-```
-
-**For 27B, H=4096, L=32, TP=2, batch=1:**
-```
-32 layers × 4096 × 1 × 2 bytes = 262,144 bytes = 256 KB/tok
-(16 KB per layer all-reduce × 32 layers)
-```
-
-**For 36B, H=6144, L=48, TP=2, batch=1:**
-```
-48 layers × 6144 × 1 × 2 bytes = 589,824 bytes = 576 KB/tok
-(12 KB per layer all-reduce × 48 layers)
-```
-
-### 1.3 Scaling with batch size
-
-| Model | H | L | tok/s | Batch | PP boundary bytes/tok | TP total bytes/tok |
-|-------|-----|-----|-------|-------|----------------------|--------------------|
-| 27B | 4096 | 32 | 10 | 1 | 8 KB | 256 KB |
-| 27B | 4096 | 32 | 50 | 1 | 8 KB | 256 KB |
-| 27B | 4096 | 32 | 100 | 1 | 8 KB | 256 KB |
-| 27B | 4096 | 32 | 10 | 16 | 128 KB | 4,096 KB |
-| 27B | 4096 | 32 | 50 | 16 | 128 KB | 4,096 KB |
-| 27B | 4096 | 32 | 100 | 16 | 128 KB | 4,096 KB |
-| 27B | 4096 | 32 | 10 | 64 | 512 KB | 16,384 KB |
-| 27B | 4096 | 32 | 50 | 64 | 512 KB | 16,384 KB |
-| 27B | 4096 | 32 | 100 | 64 | 512 KB | 16,384 KB |
-| 36B | 6144 | 48 | 10 | 1 | 12 KB | 576 KB |
-| 36B | 6144 | 48 | 50 | 1 | 12 KB | 576 KB |
-| 36B | 6144 | 48 | 100 | 1 | 12 KB | 576 KB |
-| 36B | 6144 | 48 | 10 | 16 | 192 KB | 9,216 KB |
-| 36B | 6144 | 48 | 50 | 16 | 192 KB | 9,216 KB |
-| 36B | 6144 | 48 | 100 | 16 | 192 KB | 9,216 KB |
-| 36B | 6144 | 48 | 10 | 64 | 768 KB | 36,864 KB |
-| 36B | 6144 | 48 | 50 | 64 | 768 KB | 36,864 KB |
-| 36B | 6144 | 48 | 100 | 64 | 768 KB | 36,864 KB |
-
----
-
-## 2. Required Bandwidth vs. Ethernet Capabilities
-
-### 2.1 Ethernet link capacities
-
-| Link | Raw Rate | Usable TCP (approx.) | MB/s usable |
-|------|----------|---------------------|-------------|
-| 1 GbE | 1,000 Mbit/s | ~940 Mbit/s | ~118 MB/s |
-| 2.5 GbE | 2,500 Mbit/s | ~2,350 Mbit/s | ~294 MB/s |
-| 10 GbE | 10,000 Mbit/s | ~9,400 Mbit/s | ~1,175 MB/s |
-
-### 2.2 Required bandwidth (Mbps) by model, batch, and throughput
-
-**27B model (H=4096), 1 PP boundary:**
-
-| tok/s | Batch=1 | Batch=16 | Batch=64 |
-|-------|---------|----------|----------|
-| 10 | 0.13 Mbit/s | 2.05 Mbit/s | 8.19 Mbit/s |
-| 50 | 0.65 Mbit/s | 10.24 Mbit/s | 40.96 Mbit/s |
-| 100 | 1.31 Mbit/s | 20.48 Mbit/s | 81.92 Mbit/s |
-
-**36B model (H=6144), 1 PP boundary:**
-
-| tok/s | Batch=1 | Batch=16 | Batch=64 |
-|-------|---------|----------|----------|
-| 10 | 0.20 Mbit/s | 3.07 Mbit/s | 12.29 Mbit/s |
-| 50 | 0.98 Mbit/s | 15.36 Mbit/s | 61.44 Mbit/s |
-| 100 | 1.97 Mbit/s | 30.72 Mbit/s | 122.88 Mbit/s |
-
-**27B model, TP (L=32 layers all-reduce):**
-
-| tok/s | Batch=1 | Batch=16 | Batch=64 |
-|-------|---------|----------|----------|
-| 10 | 2.05 Mbit/s | 32.77 Mbit/s | 131.07 Mbit/s |
-| 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s |
-| 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s |
-
-### 2.3 Feasibility matrix
-
-Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).
-
-**27B model — PP (1 boundary), batch=1:**
-
-| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) |
-|---|---|---|---|
-| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s |
-| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s |
-| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s |
-
-**27B model — PP (1 boundary), batch=16:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
-| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
-| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
-
-**27B model — PP (1 boundary), batch=64:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s |
-| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s |
-| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s |
-
-**27B model — TP (32 layers), batch=1:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
-| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
-| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
-
-**27B model — TP (32 layers), batch=16:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s |
-| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s |
-| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s |
-
-**27B model — TP (32 layers), batch=64:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s |
-| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s |
-| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s |
-
-**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE**
-
-**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s ≈ 2.5 GbE usable (2500 Mbit/s) — ~52% saturated**
-
-**Key finding:** PP is feasible across all ethernet classes for all batch sizes up to at least 100 tok/s. TP becomes bottlenecked on 1 GbE at batch=64 and 100 tok/s.
-
----
-
-## 3. Why PP is the Only Realistic Option over Ethernet
-
-### 3.1 Communication cost comparison
-
-| Scheme | Communication per tok per rank | 27B, H=4096, batch=1 | 36B, H=6144, batch=1 |
-|--------|------|---|---|
-| PP (Nₚₚ boundaries) | Nₚₚ × H × 2 | Nₚₚ × 8 KB | Nₚₚ × 12 KB |
-| TP (L layers all-reduce) | L × H × 2 | 32 × 8 KB = 256 KB | 48 × 12 KB = 576 KB |
-
-PP communication per token is **L/Nₚₚ times smaller** than TP communication per token (factor of ~16× for 2 PP stages).
-
-### 3.2 Latency-bound nature of autoregressive generation
-
-During autoregressive generation, each layer computes one token at a time. In TP, every layer requires an all-reduce (2× ring latency or 1× tree latency per layer). For L=32 layers, that's 32 synchronization points per token.
-
-In PP, activations cross the stage boundary only once per layer (or L/Nₚₚ times). The critical path is:
-
-```
-Stage 0 computes → send hidden state across boundary → Stage 1 computes → ...
-```
-
-This is a single bottleneck per token, with latency proportional to network RTT × number of stages, not L.
-
-### 3.3 Bandwidth utilization
-
-With PP, each boundary crossing transfers `H × 2` bytes. For 8 KB at 1 GbE, the transfer completes in ~68 µs. This is negligible compared to compute time (milliseconds per layer on a GPU). The link is underutilized.
-
-With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 KB total at 1 GbE, that's ~2 ms of pure transfer time — potentially comparable to compute time on a fast GPU.
-
-### 3.4 Real-world evidence
-
-- Petals uses PP exclusively over the public internet and achieves usable inference at 1–2 tok/s even with 100 Mbit/s links [1].
-- llama.cpp RPC splits layers across nodes in pipeline order, not tensor sharding [2].
-- exo uses TP but achieves speedup only with RDMA over Thunderbolt (~50 Gbit/s) [3].
-- Petals paper Table 3: inference performance does **not depend much on bandwidth** (1 Gbit/s vs 100 Mbit/s gives nearly identical results) but **degrades significantly with higher latency** [1].
-
-**Conclusion:** For gigabit-class ethernet, PP is the only practical multi-node parallelism strategy for inference. TP can work on 10 GbE for small batches but saturates at 1 GbE with moderate batch sizes.
-
----
-
-## 4. Real-World Multi-Node Inference Benchmarks
-
-### 4.1 Summary table
-
-| System | Model | Hardware | # Nodes | Network | tok/s (batch=1) | Notes |
-|--------|-------|----------|---------|---------|-----------------|-------|
-| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 1 Gbit/s, <5 ms RTT | 1.71 | PP, 22 stages, quantized [1] |
-| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, <5 ms RTT | 1.66 | ~3% slowdown vs 1 Gbit/s [1] |
-| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, 100 ms RTT | 1.23 | Latency matters more than BW [1] |
-| Petals (wan) | BLOOM-176B | 14 consumer GPUs | 14 | Real internet, 100–1000 Mbit/s | 0.83 | Europe+N. America [1] |
-| Petals | Llama 2 70B | consumer GPUs | various | Internet | up to 6 | Interactive chat [4] |
-| Petals | Falcon 180B | consumer GPUs | various | Internet | up to 4 | Interactive chat [4] |
-| llama.cpp RPC | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | Ethernet (2.5 GbE) | ~20 | Scales down with more nodes [3] |
-| llama.cpp RPC | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~25 | Scales down with more nodes [3] |
-| llama.cpp RPC | Kimi K2 (1T) | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~30 | A32B sparse, 32B active [3] |
-| Exo (RDMA) | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | 32 | TP, scales UP with more nodes [3] |
-| Exo (RDMA) | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | ~30 | 671B dense TP [3] |
-| vLLM PP | Llama-3-70B | 2× A100 80GB | 2 | NVLink + Ethernet | ~15 | PP across nodes, TP within [5] |
-
-### 4.2 Detailed notes
-
-**Petals [1]:**
-- Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
-- For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
-- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
-- Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
-- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
-- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)
-
-**llama.cpp RPC [2]:**
-- Model weights and KV cache distributed across nodes proportionally to memory
-- Default: pipeline-style distribution of layers (first layers on node A, last on node B)
-- Each node runs `ggml-rpc-server`, main host uses `--rpc` flag
-- RDMA transport supported (RoCEv2), negotiated automatically with libibverbs
-- Over ethernet, performance decreases with more nodes (each extra node adds a hop)
-- Over RDMA/TB5, performance scales up with more nodes (less latency per hop)
-
-**Exo [3]:**
-- Uses tensor parallelism with MLX distributed backend
-- With RDMA over Thunderbolt 5: up to 1.8× speedup on 2 devices, 3.2× on 4 devices
-- Topology-aware auto-parallelism considers latency/bandwidth between links
-- Thunderbolt 5 provides ~50–60 GbE real-world throughput (RDMA)
-
-**Key insight from all systems:** Ethernet (1 Gbit/s) is sufficient for PP inference of 20–40B models because per-token activation traffic is small (8–12 KB/boundary). The bottleneck is **latency** (RTT between nodes), not bandwidth.
-
----
-
-## 5. Overheads
-
-### 5.1 TCP/IP overhead
-
-| Overhead | Amount |
-|----------|--------|
-| Ethernet header + trailer | 18 bytes (14 + 4 FCS) |
-| IPv4 header (no options) | 20 bytes |
-| TCP header (no options) | 20 bytes |
-| Total per packet | ~58 bytes overhead |
-| MTU (jumbo frames 9000) | 9000 bytes payload → 0.6% overhead |
-| MTU (1500 standard) | 1500 bytes payload → 3.9% overhead |
-
-**Practical impact:** For 8 KB transfers (27B PP boundary), use jumbo frames where possible to minimize TCP overhead. For small transfers (< 1.5 KB), a single TCP packet is used and the overhead is 58/8192 = 0.7% per transfer — negligible.
-
-### 5.2 RPC/serialization overhead
-
-- llama.cpp RPC: Protobuf serialization adds ~10–20% overhead for small tensors
-- Petals: PyTorch tensor serialization over hivemind/gRPC, adds serialization time
-- vLLM PP: gRPC with protobuf, similar overhead
-- Serialization scales with tensor size: negligible for 8 KB+ tensors
-
-### 5.3 Pipeline bubble overhead
-
-With Nₚₚ pipeline stages and microbatching, the pipeline bubble fraction is:
-
-```
-bubble_fraction = (Nₚₚ - 1) / (Nₚₚ + microbatches - 1)
-```
-
-For 2 stages, 1 microbatch (single tok): bubble = 1/2 = 50% idle time.
-For 2 stages, 8 microbatches: bubble = 1/9 = 11%.
-
-In autoregressive generation with batch=1 and no microbatching, each stage processes one token sequentially:
-- Stage 0 finishes, sends hidden state, waits for Stage 1
-- Stage 1 processes, returns result, Stage 0 processes next token
-- Effective utilization ≈ 1/Nₚₚ if no pipelining overlap
-
-**Mitigation:** Process multiple tokens before waiting. Send next token to Stage 0 while Stage 1 is computing — pipelining reduces bubble.
-
-### 5.4 Per-hop LAN latency
-
-| Network | Typical RTT | One-way latency |
-|---------|------------|-----------------|
-| 1 GbE (Cat6) | 30–50 µs | 15–25 µs |
-| 2.5 GbE (Cat6) | 30–50 µs | 15–25 µs |
-| 10 GbE (Cat6a) | 20–30 µs | 10–15 µs |
-| RoCEv2 (IB/25GbE) | 1–5 µs | <5 µs |
-| RDMA (Thunderbolt 5) | 10–30 µs round-trip end-to-end | — |
-
-**Impact on PP inference:** With Nₚₚ = 2 stages and batch=1, one token takes:
-```
-t_compute + t_roundtrip + t_compute
-```
-
-At 30 µs RTT and ~2 ms layer compute time per GPU: the network adds ~30 µs, which is 1.5% of total. Negligible at LAN speeds.
-
-With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute time.
-
-### 5.5 Quantization overhead
-
-- Petals uses dynamic blockwise 8-bit quantization: halves hidden state bandwidth from 16-bit to 8-bit [1]
-- For 27B: 8 KB → 4 KB per boundary with 8-bit quantization
-- Accuracy impact: <1% on standard benchmarks [1]
-- Quantization/dequantization adds ~1–5 ms per stage on modern GPUs (negligible vs. compute)
-
----
-
-## 6. Conclusions and Recommendations
-
-### For the 27B / 36B models on gigabit-class ethernet:
-
-1. **1 GbE is sufficient for PP inference** up to at least 100 tok/s at batch=64 (81.92 Mbit/s required, well under 1000 Mbit/s capacity). Even with TCP/IP and RPC overheads, margin is >90%.
-
-2. **Pipeline parallelism is the correct choice** over ethernet — PP sends 8–12 KB per boundary vs. 256–576 KB for TP, a 32× advantage for 2 PP stages.
-
-3. **Latency matters more than bandwidth** — the Petals paper demonstrates that dropping from 1 Gbit/s to 100 Mbit/s only reduces throughput by ~3%, while increasing RTT from 5 ms to 100 ms reduces it by ~28%.
-
-4. **With 8-bit quantization**, bandwidth requirements halve, providing additional safety margin.
-
-5. **Jumbo frames recommended** to minimize TCP overhead on larger transfers.
-
-6. **For batch sizes ≥ 16**, 1 GbE remains viable for PP but consider 2.5 GbE for headroom and faster model loading.
-
----
-
-## References
-
-[1] Borzunov et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL 2023. arXiv:2209.01188. Available at: https://arxiv.org/pdf/2209.01188 — Primary benchmark source: Table 3 shows inference throughput at various bandwidths and latencies for BLOOM-176B. Key finding: inference insensitive to bandwidth, sensitive to latency. Recommended 25 Mbit/s bidirectional bandwidth.
-
-[2] ggml-org/llama.cpp RPC documentation. Available at: https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md — Architecture for distributed LLM inference via RPC. Supports RDMA transport (RoCEv2), automatic negotiation, local tensor caching.
-
-[3] exo-explore/exo README. Available at: https://github.com/exo-explore/exo — Multi-node LLM inference on Apple Silicon with TP and RDMA over Thunderbolt 5. Benchmarks with Qwen3-235B, DeepSeek v3.1, Kimi K2 on 4× M3 Ultra Mac Studio.
-
-[4] Petals chatbot demo and health monitor. Available at: https://chat.petals.dev, https://health.petals.dev — Real-world performance of Llama 2 70B (up to 6 tok/s) and Falcon 180B (up to 4 tok/s) on public swarm.
-
-[5] vLLM pipeline parallel documentation. Available at: https://docs.vllm.ai/en/stable/dev/engine/llm_engine.html — vLLM supports PP across nodes using gRPC with CUDA-graph support for low-latency stage transitions.
-
----
-
-## Appendix: Full bandwidth formulae
-
-### Pipeline Parallelism (generation, batch=1)
-
-```
-bytes_per_boundary_per_tok = hidden_size × 2   [fp16]
-total_bytes_per_tok = num_boundaries × hidden_size × 2
-bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
-```
-
-### Pipeline Parallelism (generation, batch=N)
-
-```
-bytes_per_boundary_per_tok = hidden_size × N × 2
-total_bytes_per_tok = num_boundaries × hidden_size × N × 2
-```
-
-### Tensor Parallelism (generation, batch=1)
-
-```
-bytes_per_layer_allreduce = hidden_size × 2
-total_bytes_per_tok = num_layers × hidden_size × 2
-bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
-```
-
-### Tensor Parallelism (generation, batch=N)
-
-```
-total_bytes_per_tok = num_layers × hidden_size × N × 2
-```
-
-### With 8-bit quantization (like Petals)
-
-```
-# Simply divide all fp16 byte counts by 2
-bytes_per_boundary_per_tok = hidden_size × 1   [int8]
-```