On main: peer frank-small WIP held for frank-medium deploy

The 320k pool (163840 per conversation over two slots) costs ~4.3 GiB of KV,
and the BF16 projector (888 MiB) lands on ROCm0. Fit at the deployed point,
computed, not estimated: ROCm0 29.19 / ROCm1 26.04 GiB of 31.86 -- 2.67 / 5.82
GiB free, both above the 1.5 GiB floor.

36:29 is one layer over the 35:30 balance on purpose: at this context the
balance point is 34:31 (0.11 GiB gap), and the operator moved a layer to ROCm0
to buy headroom on the drafter card. The gap is 3.15 GiB by construction, so
the seam's slack is that gap rounded up (3225 MiB) with the trade written down,
and the ctx cap moves 262144 -> 327680 with the rope caveat (the GGUF carries
no scaling; positions past the trained 262144 extrapolate on the base rope).

Vision: NO_MMPROJ=0, the serve script now refuses to start without the
projector when vision is on (a missing file is a deploy error, not a silent
text-only server), the download script fetches and byte-verifies it, and the
deploy handler's preflight checks for it. The projector is already on the host,
byte- and sha256-verified against the Hub.

.AGENTTRASH/frank-default-large.env.dflash2-vulkan-2026-08-21

@@ -1,60 +0,0 @@
-# Tunables for frank-default-large (upstream llama.cpp PR #27342 DFlash2,
-# Vulkan backend, dankclaw, Strix Halo gfx1151).
-#
-# Runtime: llama.cpp PR #27342 built with GGML_VULKAN=ON serving
-# unsloth/Qwen3.8-27B-UD-Q8_K_XL (target) with a z-lab DFlash2 drafter
-# (Qwen3.8-27B-DFlash2-Q4_K_M). Target at 31.4 GB, DFlash2 drafter at 2.0 GB.
-# Alias frank-default-large, port 52625.
-#
-# DFlash2 runs on the Vulkan (RADV) backend. Its candidate selector needs
-# TOP_K + ARGMAX, which the HIP backend does not implement. This replaced
-# the CIRU HIP runtime on 2026-08-21. See docs/dflash2-large-vulkan.md.
-#
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
-# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
-# incident). This deployment uses Vulkan, not HIP, but the rule stays for
-# portability.
-
-# Upstream llama.cpp DFlash2 binary (PR #27342, Vulkan build).
-LLAMA_BIN=/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server
-
-# Model files. The target (UD-Q8_K_XL) was already on disk; the DFlash2
-# drafter was added for this deployment. Both live in /home/dank/models.
-MODEL_DIR=/home/dank/models
-MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf
-DRAFT_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
-
-ALIAS=frank-default-large
-BIND_ADDR=0.0.0.0
-LISTEN_PORT=52625
-
-# Serving: full native 262144 context, 1 slot. The Vulkan device reports
-# 113942 MiB; KV at 262144 x 64 KiB/cell is ~16 GiB. rocm-smi reports the
-# 64 GiB budget; trust the runtime for the addressable SVM pool.
-CTX_SIZE=262144
-PARALLEL=1
-BATCH_SIZE=4096
-UBATCH_SIZE=4096
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=f16
-CTX_CHECKPOINTS=16
-CACHE_RAM=8192
-
-# DFlash2 speculative decoding. The drafter's trained block size is 8;
-# SPEC_DRAFT_N_MAX clamps to it. TOP_K selector (top 16 candidates).
-SPEC_TYPE=draft-dflash
-SPEC_DRAFT_N_MAX=8
-SPEC_DRAFT_N_MIN=0
-SPEC_DRAFT_P_MIN=0.0
-SPEC_DRAFT_P_SPLIT=0.10
-
-# Sampling (model card defaults, thinking-general).
-TEMPERATURE=0.7
-TOP_P=0.95
-
-# Thinking enabled. deepseek format moves the thinking reasoning into
-# message.reasoning_content (OpenAI-compatible).
-REASONING=on
-REASONING_FORMAT=deepseek
-
-VERBOSITY=4

.AGENTTRASH/frank-default-small.env.llama.cpp

@@ -0,0 +1,71 @@
+# Tunables for frank-default-small (Tiel-Coder-35B-A3B UD-Q6_K_XL,
+# qwen35moe hybrid MoE, Dankserver, RTX 3090).
+#
+# Runtime: upstream llama.cpp (latest master, CUDA build with
+# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86) on RTX 3090
+# 24 GB; Ryzen 7 7700X; 62 GB RAM.  Replaced the ik_llama.cpp Ornith
+# CRACK Q8_0 deployment on 2026-09-01.
+#
+# Model: peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF, UD-Q6_K_XL quant
+# (30 GB).  Architecture: qwen35moe — 40 layers, full_attention_interval=4
+# (10 attention + 30 SSM/Mamba), 256 experts / 8 used, 2 KV heads × 256
+# dim, embedding 2048, rope dimension_count 64, rope freq_base 10 000 000.
+# Text-only (no mmproj).  Built-in chat template in the GGUF.
+#
+# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
+# Non-expert weights (~1.6 GB) fit entirely on the 24 GB card.
+#
+# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per slot
+# × 3 parallel) and 10 attention layers, the KV cache is ~2 GB — the
+# requested 2 GB VRAM buffer.
+#
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
+# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
+# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
+
+# Model file (UD-Q6_K_XL, 30 GB, text-only MoE hybrid).
+MODEL_DIR=/home/dank/models/tiel-coder-35b
+MODEL_FILE=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf
+
+# Upstream llama.cpp llama-server (CUDA build, latest master).
+# Built with -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86
+# using the staged CUDA 13.2 toolkit at /home/dank/cuda13-stage.
+LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
+
+ALIAS=frank-default-small
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
+
+# Total context budget across all parallel slots.  200700 / 3 = 66 900
+# per slot.  The model's context_length is 262 144, so 66 900 is well
+# within range.
+CTX_SIZE=200700
+PARALLEL=3
+
+# Expert offload: all MoE expert weights pinned to CPU RAM.
+CPU_MOE=1
+
+# All non-expert layers on GPU (auto-fit will determine the exact count;
+# with --cpu-moe the non-expert footprint is ~1.6 GB, leaving ample room).
+N_GPU_LAYERS=999
+
+# KV cache: Q8_0 for both K and V.  ~2 GB total for 200700 ctx × 3 slots
+# × 10 attention layers × (2 KV heads × 256 dim × 1 byte) × 2 (K+V).
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=q8_0
+
+VERBOSITY=2
+
+# Prefill batch sizing.
+BATCH_SIZE=4096
+UBATCH_SIZE=1024
+
+# Sampling.
+TEMPERATURE=1.0
+TOP_K=20
+TOP_P=0.95
+MIN_P=0.0
+
+# Compute threads lowered so other processes keep a core.
+THREADS=7
+THREADS_HTTP=8

.AGENTTRASH/frank-default-small.service.llama.cpp

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Ornith-1.5-35B-A3B CRACK Q6_K ngram-mod + Qwen-Sharp template (dankserver, frank-default-small)
+Description=llama.cpp Tiel-Coder-35B-A3B Q6_K_XL qwen35moe hybrid MoE (dankserver, frank-default-small)
 After=network-online.target
 Wants=network-online.target
 

.AGENTTRASH/frank-large.env.llama.cpp

@@ -0,0 +1,126 @@
+# Tunables for frank-large (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
+# STRIX-GGUF on the kingjones30/ROCmFPX llama.cpp runtime, Dankclaw,
+# Strix Halo gfx1151, unified memory).
+#
+# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
+# qwen4exp-on-rocmfpx-d3ca537 patch applied — the model card's pinned
+# runtime: ROCmFP4 tensor types AND the qwen4exp architecture in one
+# tree; stock llama.cpp and charlie's fork alone cannot load this GGUF).
+# Built per the card recipe:
+#   cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
+# at /home/dank/GitPackages/kingjones-ROCmFPX/build.
+# Replaced the agentionai ROCmFP4-FAST imatrix deployment (charlie's
+# build-qwen38 runtime) on 2026-08-30.
+#
+# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
+# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
+# at layer 1), Q4_0_ROCMFP4_STRIX (file_type 105), 113.47 GiB single
+# file. STRIX is the top quality tier: PLE table at Q8_0 (50.66 GiB),
+# token embeddings + lm head at Q6_K. The PLE table NEVER enters GPU
+# memory — it streams from the file through the OS page cache. Two hard
+# card constraints:
+#  1. mmap MUST stay enabled (default) — --no-mmap forces the 50.66 GiB
+#     table into anonymous memory and the cgroup killer takes the
+#     process with nothing in the server log (only dmesg).
+#  2. Do NOT pin the PLE tensor to CPU with -ot — measured gen fell
+#     ~23 -> 13.4 tok/s; the kernel streams it better.
+# NOTE: the card's own build_attn_qsa still calls ggml_fill on the
+# F16-cast kq mask (the path that aborted our charlie build with
+# --flash-attn on before scripts/patches/qwen4exp-qsa-f32-mask.patch).
+# If a clean kingjones-tree build aborts graph build with
+# GGML_ASSERT(a-type == GGML_TYPE_F32), port that fix (F32 base mask +
+# cast at the attention call) into this tree and rebuild.
+#
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
+# incident). llama.cpp reads --host/--port/--temp as CLI flags, never as
+# env vars.
+
+# kingjones30/ROCmFPX runtime binary (card recipe build).
+LLAMA_BIN=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server
+
+# Model package from
+# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF
+# Layout: single unsharded GGUF, text-only (the repo ships no mmproj;
+# the vision tower below is borrowed from the agentionai FAST package —
+# same base model, f16 — kept only if it loads clean, drop
+# MMPROJ_PATH if it does not).
+MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
+MODEL_FILE=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf
+MMPROJ_PATH=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf
+
+# Qwen-Sharp chat template (tracked in the repo, staged on the host). The
+# Sharp template drives thinking on by default (medium effort); the server
+# emits reasoning in DeepSeek format so clients receive reasoning_content
+# separately instead of literal thinking tags in content.
+CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
+CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
+
+# Alias + network.
+ALIAS=frank-large
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
+
+# Serving: 401408-token context, TWO slots (parallel 2), batch 4096 /
+# ubatch 512, 16+16 threads, f16 KV, 16 context checkpoints, 8 GiB
+# prompt cache. --fit off: auto-fit segfaults on this model's
+# hyper-connection/indexer config; -ngl 999 puts every layer on the
+# SVM-backed GPU pool. ubatch 512 matches the card's re-measurement
+# (-ub 1024/2048/4096 showed no effect; -t 32 vs 16 is noise).
+#
+# CTX_SIZE=401408 / PARALLEL=2: carried over from the FAST deployment
+# where it loaded clean (~74 s, health 200). The card measures the full
+# 262144 window at ~71.7 GiB GTT at load on the same 128 GB board; QSA
+# caps KV growth so the extra window is cheap. KV footprint scales
+# with the 12 attention layers only. Verify load on first STRIX boot.
+CTX_SIZE=401408
+PARALLEL=2
+
+BATCH_SIZE=4096
+UBATCH_SIZE=512
+NUM_THREADS=16
+THREAD_BATCH=16
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+CTX_CHECKPOINTS=16
+CACHE_RAM=8192
+SLOT_SAVE_PATH=/home/dank/serverAppConfigs/scripts/slot-state
+SPLIT_MODE=none
+FIT=off
+NGL=999
+FLASH_ATTN=on
+
+# Sampling (card instruct table, non-thinking).
+TEMPERATURE=0.7
+TOP_K=20
+TOP_P=0.8
+MIN_P=0.0
+PRESENCE_PENALTY=1.5
+
+# Reasoning on (Sharp template drives it; preserve_thinking + preserve_reasoning
+# keep the reasoning trail in the conversation). DeepSeek format projects the
+# reasoning into the API's reasoning_content field instead of literal thinking
+# tags.
+REASONING=on
+REASONING_FORMAT=deepseek
+
+# Strix Halo unified memory + native gfx1151.
+HSA_OVERRIDE_GFX_VERSION=11.5.1
+
+VERBOSITY=4
+
+
+# Speculative decoding: disabled. The card's Usage block ships no spec
+# flags. The Flash-Next MTP heads do not load on this loader (requires
+# per-layer blk.N.hc_attn_norm; every head carries only blk.48 —
+# verified 3x on build-qwen38). DRAFT_MODEL_* point at the retired
+# agentionai FAST head and are unused while SPEC_TYPE=none.
+DRAFT_MODEL_DIR=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST
+DRAFT_MODEL_FILE=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf
+SPEC_TYPE=none
+SPEC_DRAFT_N_MAX=3
+SPEC_DRAFT_N_MIN=0
+SPEC_DRAFT_P_MIN=0.75
+SPEC_DRAFT_NGL=999
+SPEC_DRAFT_DEVICE=ROCm0
+SPEC_DRAFT_TYPE_K=f16
+SPEC_DRAFT_TYPE_V=f16

.AGENTTRASH/frank-large.service.llama.cpp

@@ -1,5 +1,5 @@
 [Unit]
-Description=CIRU llama.cpp Qwen3.8-27B ROCmFP4 STRIX MTP + Sharp template (dankclaw, frank-default-large)
+Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-large)
 After=network-online.target
 Wants=network-online.target
 
@@ -7,7 +7,7 @@ Wants=network-online.target
 Type=simple
 User=dank
 Group=dank
-Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-default-large.env"
+Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
 ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
 Restart=on-failure
 RestartSec=10

.AGENTTRASH/frank-npu.env.flm-npu

@@ -0,0 +1,23 @@
+# Tunables for frank-npu (FastFlowLM NPU endpoint, Dankclaw)
+# Runtime: FLM 1.0.1 (/usr/bin/flm) on the XDNA2 NPU (50 TOPS). Boot tag is
+# nuextract3:4b (NuExtract3-4B-NPU2, Qwen3.5-4B skeleton, structured-extraction
+# model); serve routes every request by the exact FLM registry tag in the
+# request's model field, so any registered tag (gemma4-it:e4b, qwythos:9b, ...)
+# is reachable on this port regardless of the boot tag. The NPU is a
+# single-lane device: generations serialize FIFO, and the first request after a
+# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
+# pool shared with frank-large (iGPU).
+FLM_MODEL=nuextract3:4b
+FLM_HOST=0.0.0.0
+FLM_PORT=7771
+# turbo = max NPU power mode (xrt-smi configure --pmode turbo at serve start).
+# No other tenant uses the NPU on this host, so the top mode costs nothing to
+# neighbours; drop to `performance` if iGPU power-budget contention ever shows.
+FLM_PMODE=turbo
+# FLM_ASR=1 additionally loads Whisper-V3-Turbo-NPU2 (+0.62 GB) and exposes
+# /v1/audio/transcriptions. NuExtract3 answers text-only; audio stays off.
+FLM_ASR=0
+# -1 = no --ctx-len flag: every pack loads with its own trained context
+# (nuextract3:4b -> 32768, gemma4-it:e4b -> 131072). A global override would
+# over-allocate KV for the small packs.
+FLM_CTX_LEN=-1

.AGENTTRASH/frank-npu.service.flm-npu

@@ -0,0 +1,17 @@
+[Unit]
+Description=FastFlowLM NPU sight+sound inference (frank-npu) on Dankclaw
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+Environment="FRANK_NPU_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-npu.env"
+ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-npu.sh
+LimitMEMLOCK=infinity
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target

.AGENTTRASH/serve-frank-large.sh.dflash2-vulkan-2026-08-21

@@ -1,104 +0,0 @@
-#!/usr/bin/env bash
-# Serve unsloth/Qwen3.8-27B-UD-Q8_K_XL as frank-default-large with a
-# z-lab DFlash2 drafter (Qwen3.8-27B-DFlash2-Q4_K_M) on the Vulkan backend.
-#
-# Runtime: upstream llama.cpp PR #27342 (DFlash2 support) built with
-# GGML_VULKAN=ON for the Strix Halo gfx1151 (RADV). DFlash2 selects its
-# candidates through TOP_K + ARGMAX ops that the HIP backend does not
-# implement, so the whole stack runs on Vulkan0.
-#
-# The target and the drafter share the Qwen3.8 arch. DFlash is lossless
-# under strict rejection sampling. --spec-draft-n-max clamps to the drafter's
-# trained block size (8).
-#
-# Tunables come from frank-default-large.env in this directory unless
-# FRANK_LARGE_ENV_FILE overrides that path.
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models}"
-: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
-: "${DRAFT_FILE:=Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
-: "${ALIAS:=frank-default-large}"
-: "${BIND_ADDR:=0.0.0.0}"
-: "${LISTEN_PORT:=52625}"
-: "${CTX_SIZE:=262144}"
-: "${PARALLEL:=1}"
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=4096}"
-: "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
-: "${CTX_CHECKPOINTS:=16}"
-: "${CACHE_RAM:=8192}"
-: "${SPEC_TYPE:=draft-dflash}"
-: "${SPEC_DRAFT_N_MAX:=8}"
-: "${SPEC_DRAFT_N_MIN:=0}"
-: "${SPEC_DRAFT_P_MIN:=0.0}"
-: "${SPEC_DRAFT_P_SPLIT:=0.10}"
-: "${TEMPERATURE:=0.7}"
-: "${TOP_P:=0.95}"
-: "${REASONING:=on}"
-: "${REASONING_FORMAT:=deepseek}"
-: "${VERBOSITY:=4}"
-
-MODEL="$MODEL_DIR/$MODEL_FILE"
-DRAFT="$MODEL_DIR/$DRAFT_FILE"
-
-if [[ ! -f "$MODEL" ]]; then
-    echo "missing required file: $MODEL" >&2
-    exit 1
-fi
-if [[ ! -f "$DRAFT" ]]; then
-    echo "missing DFlash2 drafter: $DRAFT" >&2
-    exit 1
-fi
-if [[ ! -x "$LLAMA_BIN" ]]; then
-    echo "missing llama-server binary: $LLAMA_BIN" >&2
-    exit 1
-fi
-
-exec "$LLAMA_BIN" \
-    --model "$MODEL" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    -dev Vulkan0 \
-    -ngl 999 \
-    -c "$CTX_SIZE" \
-    -b "$BATCH_SIZE" \
-    -ub "$UBATCH_SIZE" \
-    -fa on \
-    -ctk "$CACHE_TYPE_K" \
-    -ctv "$CACHE_TYPE_V" \
-    -np "$PARALLEL" \
-    -ctxcp "$CTX_CHECKPOINTS" \
-    --cache-ram "$CACHE_RAM" \
-    --cache-prompt \
-    --metrics \
-    --no-mmap \
-    --spec-type "$SPEC_TYPE" \
-    --spec-draft-model "$DRAFT" \
-    --spec-draft-device Vulkan0 \
-    --spec-draft-ngl 999 \
-    --spec-draft-type-k f16 \
-    --spec-draft-type-v f16 \
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
-    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
-    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
-    --spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \
-    --temp "$TEMPERATURE" \
-    --top-p "$TOP_P" \
-    --reasoning "$REASONING" \
-    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget -1 \
-    -lv "$VERBOSITY"

.AGENTTRASH/serve-frank-large.sh.llama.cpp

@@ -0,0 +1,180 @@
+#!/usr/bin/env bash
+# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as
+# frank-large via the kingjones30/ROCmFPX llama.cpp runtime on
+# Dankclaw (Strix Halo gfx1151, unified memory).
+#
+# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
+# model card's qwen4exp-on-rocmfpx patch) built at
+# /home/dank/GitPackages/kingjones-ROCmFPX/build (HIP, gfx1151,
+# ROCm 7.2.4, card recipe). The ROCmFP4 quant types plus the qwen4exp
+# architecture must be in one tree; stock llama.cpp cannot load this
+# GGUF.
+#
+# Model: Qwen3.8-Flash-Next base (qwen4exp MoE: 512 experts / 10 used,
+# 48 layers = 12 attention + 36 SSM, QSA indexer, hyper-connections, PLE
+# at layer 1) quantized Q4_0_ROCMFP4_STRIX, 113.47 GiB single file. The
+# 50.66 GiB PLE n-gram table stays OUT of GPU memory and streams via
+# mmap from the OS page cache — never add --no-mmap or -ot for it.
+#
+# Profile: card sampling table (temp 0.7, top_p 0.80, top_k 20, min_p 0.0,
+# presence 1.5 — non-thinking instruct) mixed with the validated serving
+# profile: ctx 401408 x 2 slots, b 4096 / ub 512, t 16 / tb 16, f16 KV,
+# fa on (if the graph build aborts on the QSA F16 mask, see the note in
+# the env file: port scripts/patches/qwen4exp-qsa-f32-mask.patch into
+# this tree and rebuild), cont-batching, prompt cache + 8 GiB cache-ram
+# + idle-slot cache, 16 ctx checkpoints, --fit off (auto-fit segfaults
+# on qwen4exp hyper-connection config). Speculative decoding off: the
+# Flash-Next MTP heads do not load on this loader; the card ships no
+# spec flags.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${LLAMA_BIN:=/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server}"
+: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX}"
+: "${MODEL_FILE:=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf}"
+: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
+: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
+: "${CHAT_TEMPLATE_KWARGS:={}}"
+: "${ALIAS:=frank-large}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7770}"
+: "${CTX_SIZE:=401408}"
+: "${PARALLEL:=2}"
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=512}"
+: "${NUM_THREADS:=16}"
+: "${THREAD_BATCH:=16}"
+: "${CACHE_TYPE_K:=f16}"
+: "${CACHE_TYPE_V:=f16}"
+: "${CTX_CHECKPOINTS:=16}"
+: "${CACHE_RAM:=8192}"
+: "${SLOT_SAVE_PATH:=$SCRIPT_DIR/slot-state}"
+: "${SPLIT_MODE:=none}"
+: "${FIT:=off}"
+: "${NGL:=999}"
+: "${FLASH_ATTN:=on}"
+: "${TEMPERATURE:=0.7}"
+: "${TOP_K:=20}"
+: "${TOP_P:=0.8}"
+: "${MIN_P:=0.0}"
+: "${PRESENCE_PENALTY:=1.5}"
+: "${REASONING:=on}"
+: "${REASONING_FORMAT:=deepseek}"
+: "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}"
+: "${SPEC_TYPE:=none}"
+: "${DRAFT_MODEL_DIR:=/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST}"
+: "${DRAFT_MODEL_FILE:=Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf}"
+: "${SPEC_DRAFT_N_MAX:=3}"
+: "${SPEC_DRAFT_N_MIN:=0}"
+: "${SPEC_DRAFT_P_MIN:=0.75}"
+: "${SPEC_DRAFT_NGL:=999}"
+: "${SPEC_DRAFT_DEVICE:=ROCm0}"
+: "${SPEC_DRAFT_TYPE_K:=f16}"
+: "${SPEC_DRAFT_TYPE_V:=f16}"
+
+MODEL="$MODEL_DIR/$MODEL_FILE"
+for f in "$LLAMA_BIN" "$MODEL" "$CHAT_TEMPLATE_FILE"; do
+    if [[ ! -r "$f" ]]; then
+        echo "missing or unreadable required artifact: $f" >&2
+        exit 1
+    fi
+done
+if [[ ! -x "$LLAMA_BIN" ]]; then
+    echo "missing executable runtime: $LLAMA_BIN" >&2
+    exit 1
+fi
+
+# --- speculative decoding (MTP draft head) ---------------------------------
+# The MTP draft head is a separate GGUF sidecar loaded via --spec-draft-model
+# (alias --model-draft).  ngram-only specs do not need a draft file.
+# Set SPEC_TYPE=none in the env to disable speculative decoding entirely.
+SPEC_ARGS=()
+if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
+    SPEC_ARGS=(
+        --spec-type "$SPEC_TYPE"
+        --spec-draft-n-max "$SPEC_DRAFT_N_MAX"
+        --spec-draft-n-min "$SPEC_DRAFT_N_MIN"
+        --spec-draft-p-min "$SPEC_DRAFT_P_MIN"
+    )
+    if [[ "$SPEC_TYPE" == *draft-mtp* ]]; then
+        DRAFT_MODEL="$DRAFT_MODEL_DIR/$DRAFT_MODEL_FILE"
+        if [[ ! -r "$DRAFT_MODEL" ]]; then
+            echo "missing or unreadable draft model: $DRAFT_MODEL" >&2
+            exit 1
+        fi
+        SPEC_ARGS+=(
+            --spec-draft-model "$DRAFT_MODEL"
+            --spec-draft-ngl "$SPEC_DRAFT_NGL"
+            --spec-draft-device "$SPEC_DRAFT_DEVICE"
+            --spec-draft-type-k "$SPEC_DRAFT_TYPE_K"
+            --spec-draft-type-v "$SPEC_DRAFT_TYPE_V"
+        )
+    fi
+    if [[ -n "${SPEC_DRAFT_P_SPLIT:-}" ]]; then
+        SPEC_ARGS+=(--spec-draft-p-split "$SPEC_DRAFT_P_SPLIT")
+    fi
+fi
+
+# The STRIX repo ships no mmproj; this is the agentionai FAST package's
+# f16 vision tower for the same base model. Passed only when readable.
+MMPROJ_ARGS=()
+if [[ -n "${MMPROJ_PATH:-}" && -r "${MMPROJ_PATH:-}" ]]; then
+    MMPROJ_ARGS=(--mmproj "$MMPROJ_PATH")
+fi
+
+SERVER_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
+mkdir -p "$SLOT_SAVE_PATH"
+ROCM_PATH="${ROCM_PATH:-/opt/rocm}"
+LIBRARY_PATH="$SERVER_DIR:$ROCM_PATH/lib:$ROCM_PATH/lib/rocm_sysdeps/lib:$ROCM_PATH/lib/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
+
+exec /usr/bin/env \
+    ROCM_PATH="$ROCM_PATH" \
+    HIP_VISIBLE_DEVICES=0 \
+    LD_LIBRARY_PATH="$LIBRARY_PATH" \
+    HSA_OVERRIDE_GFX_VERSION="$HSA_OVERRIDE_GFX_VERSION" \
+    "$LLAMA_BIN" \
+    --model "$MODEL" \
+    --alias "$ALIAS" \
+    --chat-template-file "$CHAT_TEMPLATE_FILE" \
+    --chat-template-kwargs "$CHAT_TEMPLATE_KWARGS" \
+    --host "$BIND_ADDR" \
+    --port "$LISTEN_PORT" \
+    --jinja \
+    --device ROCm0 \
+    --gpu-layers "$NGL" \
+    --split-mode "$SPLIT_MODE" \
+    --parallel "$PARALLEL" \
+    --ctx-size "$CTX_SIZE" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --fit "$FIT" \
+    --flash-attn "$FLASH_ATTN" \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --threads "$NUM_THREADS" \
+    --threads-batch "$THREAD_BATCH" \
+    --ctx-checkpoints "$CTX_CHECKPOINTS" \
+    --cache-ram "$CACHE_RAM" \
+    --slot-save-path "$SLOT_SAVE_PATH" \
+    --cache-prompt \
+    --cache-idle-slots \
+    --metrics \
+    --temp "$TEMPERATURE" \
+    --top-p "$TOP_P" \
+    --top-k "$TOP_K" \
+    --min-p "$MIN_P" \
+    --presence-penalty "$PRESENCE_PENALTY" \
+    --reasoning "$REASONING" \
+    --reasoning-format "$REASONING_FORMAT" \
+    --reasoning-budget -1 \
+    "${MMPROJ_ARGS[@]}" \
+    "${SPEC_ARGS[@]}"

.AGENTTRASH/serve-frank-npu.sh.flm-npu

@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+# Serve the FastFlowLM (FLM) sight+sound endpoint — frank-npu.
+#
+# Runs `flm serve <tag>` on the XDNA2 NPU (50 TOPS, Dankclaw). The NPU is a
+# single-lane device: generations serialize in FIFO order, so this is a
+# background/triage endpoint, not an interactive multi-user chat. The NPU
+# draws from the unified RAM pool shared with frank-large (the iGPU
+# llama.cpp service); see serverAppConfigs-docs/LLM/frank-npu.md
+# for the memory budget and the model-selection tradeoff.
+#
+# NOTE: FLM requires unlimited memlock. The unit sets LimitMEMLOCK=infinity.
+# NOTE: an unknown model tag does NOT 404 — /v1/chat/completions answers HTTP 200 with a
+# flat {"error":"Failed to load <tag> model!"} body and no choices; /v1/completions returns a
+# nested OpenAI-shaped server_error (code 500) also under HTTP 200. Client model IDs must be the exact FLM registry
+# tag (e.g. gemma4-it:e4b), never a service alias. Custom tags only resolve
+# when FLM_CONFIG_PATH points at the user registry that carries them and their
+# kernels are symlinked into /home/dank/.config/flm/xclbins/<PackName>.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_NPU_ENV_FILE:-$SCRIPT_DIR/frank-npu.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -------------------
+: "${FLM_BIN:=/usr/bin/flm}"
+: "${FLM_MODEL:=gemma4-it:e4b}"
+: "${FLM_HOST:=0.0.0.0}"
+: "${FLM_PORT:=7771}"
+: "${FLM_PMODE:=turbo}"
+: "${FLM_ASR:=0}"
+: "${FLM_CTX_LEN:=131072}"
+
+# The user registry is a superset of the system registry (it carries the
+# custom tags too), and its model_path resolves relative to the registry file,
+# so exports keep a tag swap in the env file from needing script edits.
+export FLM_CONFIG_PATH="${FLM_CONFIG_PATH:-/home/dank/.config/flm/model_list.json}"
+export FLM_XCLBIN_PATH="${FLM_XCLBIN_PATH:-/home/dank/.config/flm/xclbins}"
+
+if [[ ! -x "$FLM_BIN" ]]; then
+    echo "missing flm binary: $FLM_BIN" >&2
+    exit 1
+fi
+
+# --- optional flags ----------------------------------------------------------
+OPT_ARGS=()
+if [[ "${FLM_ASR}" != "0" ]]; then
+    OPT_ARGS+=(--asr "${FLM_ASR}")
+fi
+if [[ "${FLM_CTX_LEN}" != "-1" ]]; then
+    OPT_ARGS+=(--ctx-len "${FLM_CTX_LEN}")
+fi
+
+exec "$FLM_BIN" serve "$FLM_MODEL" \
+    --host "$FLM_HOST" \
+    -p "$FLM_PORT" \
+    --pmode "$FLM_PMODE" \
+    "${OPT_ARGS[@]}"

.AGENTTRASH/serve-frank-small.sh.llama.cpp

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Serve peculiar-ragdoll/Tiel-Coder-35B-A3B (UD-Q6_K_XL) as
+# frank-default-small via stock llama.cpp on Dankserver (RTX 3090).
+#
+# Runtime: upstream llama.cpp (latest master, CUDA build with
+# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86).
+# The model is a 35B-total / 8-active MoE hybrid (qwen35moe arch) with
+# SSM/Mamba layers (full_attention_interval=4: 10 attention + 30 SSM
+# layers out of 40).  256 experts, 8 used per token.
+#
+# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
+# Non-expert weights (attention, SSM, shared expert, embeddings, output)
+# are ~1.6 GB and fit entirely on the 24 GB RTX 3090.
+#
+# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per
+# slot × 3 parallel) and 10 attention layers (2 KV heads × 256 dim),
+# the KV cache is ~2 GB — the requested 2 GB VRAM buffer.
+#
+# Model is loaded with mmap (default).  The 30 GB Q6_K_XL file is
+# memory-mapped; only accessed pages enter RAM.
+#
+# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
+# overrides that path.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-default-small.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -----------------
+: "${MODEL_DIR:=/home/dank/models/tiel-coder-35b}"
+: "${MODEL_FILE:=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf}"
+: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
+: "${ALIAS:=frank-default-small}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7770}"
+: "${CTX_SIZE:=200700}"
+: "${PARALLEL:=3}"
+: "${CACHE_TYPE_K:=q8_0}"
+: "${CACHE_TYPE_V:=q8_0}"
+: "${N_GPU_LAYERS:=999}"
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=1024}"
+: "${VERBOSITY:=2}"
+: "${CPU_MOE:=1}"
+: "${TEMPERATURE:=1.0}"
+: "${TOP_K:=20}"
+: "${TOP_P:=0.95}"
+: "${MIN_P:=0.0}"
+: "${THREADS:=7}"
+: "${THREADS_HTTP:=8}"
+
+MODEL="$MODEL_DIR/$MODEL_FILE"
+
+if [[ ! -f "$MODEL" ]]; then
+    echo "missing required file: $MODEL" >&2
+    exit 1
+fi
+if [[ ! -x "$LLAMA_SERVER" ]]; then
+    echo "missing llama-server binary: $LLAMA_SERVER" >&2
+    exit 1
+fi
+
+exec "$LLAMA_SERVER" \
+    --model "$MODEL" \
+    $([ "$CPU_MOE" = "1" ] && printf -- "--cpu-moe") \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --ctx-size "$CTX_SIZE" \
+    --parallel "$PARALLEL" \
+    --gpu-layers "$N_GPU_LAYERS" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --temp "$TEMPERATURE" \
+    --top-k "$TOP_K" \
+    --top-p "$TOP_P" \
+    --min-p "$MIN_P" \
+    --threads "$THREADS" \
+    --threads-http "$THREADS_HTTP" \
+    --alias "$ALIAS" \
+    --host "$BIND_ADDR" \
+    --port "$LISTEN_PORT"

.githooks/pre-commit

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+# Repository pre-commit hook: run only tests selected by staged paths.
+set -euo pipefail
+
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+exec "$REPO_ROOT/scripts/test-changed.sh"

.gitignore

@@ -5,10 +5,20 @@
 # LLM fleet tunables (paths and numbers, never credentials)
 !scripts/*.env
 
+# LLM fleet runtime state (slot save/restore files written by the server)
+scripts/slot-state/
+
+# Playwright e2e: local creds, browser storage states, node artifacts
+tests/e2e/.env.local
+tests/e2e/.auth/
+tests/e2e/node_modules/
+tests/e2e/test-results/
+tests/e2e/playwright-report/
+
 # pi subagent artifacts
 .pi-subagents/
 
-# doc-audit scratch (transient, synthesized into docs/ then trashed)
+# doc-audit scratch (transient, synthesized into serverAppConfigs-docs/ then trashed)
 .docaudit/
 
 # Agent trash

AGENTS.md

@@ -2,347 +2,729 @@
 
 ## Overview
 
-This repository is the single source of truth for the two homelab servers. It
+This repository is the single source of truth for the two homelab servers and
-tracks both the Podman container stacks and the LLM inference fleet. One repo,
+the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM
-one checkout per host, one deploy script.
+inference fleet. One repo, one checkout per host, one deploy script. All
+services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
+merge absorbed the former `vllmScripts` repo, history preserved).
 
 - **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
-  Calibre-Web, the *arr stack, Jellyfin) as bare containers, plus the
+  Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
-  `frank-default-small` LLM service.
+- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
-- **Dankclaw** (192.168.0.143) — AI/search/development/observability stack
+  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
-  (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus,
+  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
-  Loki, Alloy, Pi-hole) plus the `frank-default-large` LLM service. Caddy on
+  the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
   Dankclaw reverse-proxies both hosts.
+- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
+  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
+  its podman machine; all egress rides the Gluetun WireGuard netns. Deployed
+  via the `frankbook-pro` handler.
 
-**2026-08-22 merge:** the former `vllmScripts` repository (LLM fleet scripts,
+## Fleet topology (ACTIVE)
-units, env tunables, and runbook) is merged into this repo. Its full git
-history is preserved (two-parent import commit under the retired `llm/`
-prefix, reorganized in follow-up commits). The retired target checkouts
-(`/home/dank/vLLM` on Dankserver, `/home/dank/vllmScripts` on Dankclaw) are
-archived. All services now run from `/home/dank/serverAppConfigs` on each
-host.
 
-### LLM fleet topology (2026-08-22)
+| Target | Service | Runtime | Model | Port |
-
-| Target | Runtime | Service | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
+| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (35:30), 262144 ctx over 2 slots, batch 6144x512, text only | 7770 |
-| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 7770 |
+| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
+| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
 
-History: the 2026-08-17 redo retired the pipeline-parallel split and the
+Retired endpoints (`frank-npu`, and every prior runtime) are in
-FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
+the [runbook index](#runbook-index). The full swap history lives in
-Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU,
+`serverAppConfigs-docs/LLM/index.md` — do not grow it here.
-then was reverted the same day for speed. On 2026-08-22 the service briefly
-swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
-(operator decision). Runbooks: [docs/LLM/index.md](docs/LLM/index.md).
-Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
 
 ## Repository Layout
 
 - `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
 - `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
 - `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
-- `scripts/` — `deploy.sh` (unified deployer), LLM serve scripts, systemd
+- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
-  units, env tunables, bench harnesses
+  scripts, systemd units, env tunables, bench harnesses
-- `old/` — retired PP/NPU-era LLM scripts (reference only)
+- `old/` — retired LLM scripts (reference only)
-- `docs/` — knowledge base (Obsidian vault). `docs/LLM/` is the LLM fleet
+- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format)
-- `tests/` — bats test suite
+- `tests/` — bats suite + Playwright e2e
 
 ## Git Workflow
 
-All edits flow through git. Do not edit files on the remote machine directly.
+All edits flow through git. Never edit files on a remote machine directly.
 
-### Protocol
+1. **Source** the live config from the target machine
-
+2. **Commit** the fetched version here (redact secrets into `.env`)
-1. **Source the config from the target machine** — fetch the live
+3. **Edit** locally
-   compose/Caddyfile/serve script from the server where it runs
+4. **Test** — `./tests/run.sh` must pass
-2. **Place it in this repo** — commit the fetched version (redact secrets
+5. **Commit and push** — origin is Gitea on Dankclaw
-   into `.env`)
+6. **Pull** on the target (`deploy.sh` does this)
-3. **Edit here** — make all changes locally in the working tree
+7. **Deploy** — `scripts/deploy.sh` plans and dispatches only changed services
-4. **Run the tests** — `./tests/run.sh` must pass
-5. **Commit and push** — push to origin (Gitea on Dankclaw)
-6. **Pull on the target** — `git pull` on the remote machine
-7. **Restart** — `scripts/deploy.sh` pulls, restarts the stack and the LLM
-   service, and checks health
 
 ## Deployment
 
-Use `scripts/deploy.sh` to deploy committed changes to a target. One script
-covers both workloads per host: the podman stack and the LLM service.
-
 ```
-./scripts/deploy.sh dankclaw        # AI stack + frank-default-large (:7770)
+./scripts/deploy.sh dankclaw
-./scripts/deploy.sh dankserver      # media stack + frank-default-small (:7770)
+./scripts/deploy.sh dankserver
-./scripts/deploy.sh dankclaw --dry-run   # preview without changes
+./scripts/deploy.sh dankclaw --service searxng
+./scripts/deploy.sh dankserver --service frank-medium
+./scripts/deploy.sh dankclaw --dry-run
 ```
 
-Set `FRANK_SUDO_PW` for headless runs on a host without passwordless sudo
+The planner maps changed paths to services; the remote runner restarts only
-(remote sudo reads it from stdin via `sudo -S`). The value is never stored
+the selected service and records its revision outside the checkout.
-in this repo.
+Documentation-only changes are a no-op. LLM services are selected only by
+their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
+and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
+container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
+runs on a host without passwordless sudo; the value never enters the repo.
+
+### Deployment completeness (MANDATORY)
+
+A deploy is **not complete** when `deploy.sh` exits green. It requires:
+
+1. `./tests/run.sh` — all seams green, and
+2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
+   the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
+   behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
+   only with `--skip-e2e` in an emergency.
 
 ```mermaid
 graph LR
-    A[Edit locally] --> B[Tests pass]
+    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
-    B --> C[Commit]
+    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
-    C --> D[Push to origin]
+    E --> F[Dispatch selected service] --> G[Per-service health]
-    D --> E[deploy.sh: pull on target]
+    G --> H[Playwright e2e gate] --> I{green?}
-    E --> F[restart podman stack]
+    I -->|yes| J[Deploy complete]
-    F --> G[relink + restart LLM unit]
+    I -->|no| K[DEPLOY INCOMPLETE]
-    G --> H[health: homepage + LLM]
 ```
 
+## Boot Persistence & Self-Healing (MANDATORY)
+
+Every deployed, working Podman container MUST have a self-healing,
+boot-persistent Quadlet unit. A container that runs only from `compose.yaml`
+or a manual `podman run` dies silently at reboot — rootless Podman does not
+auto-start bare containers, and `restart: unless-stopped` does not survive the
+explicit stop that shutdown performs. Never report such a service as "up".
+
+Unit requirements:
+
+- **Boot persistence** — `[Install] WantedBy=default.target`, installed to
+  `~/.config/containers/systemd/<name>.container`, enabled in the user
+  manager, with linger on for the deploy user (`loginctl show-user dank -p
+  Linger` → `Linger=yes`).
+- **Self-healing** — `Restart=always` (or `on-failure`) with a short
+  `RestartSec`; quadlet health check where the service has one.
+- **Repo provenance** — the `.container` file lives in this repo
+  (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by
+  the quadlet test seam. The host-side unit is generated state; the repo file
+  is the source of truth.
+
+Before declaring a service deployed: `systemctl --user is-enabled <name>` →
+`enabled`, `is-active` → `active`, and the container answers its health check.
+A running container with no unit is a gap: write the unit, test it, deploy it.
+
+This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
+bare compose container never returned; 2026-09-11: five observability
+services had no units). The observability services now have units under
+`dankclaw/observability/quadlet/` and their compose definitions were removed —
+one owner per container name. The stack is **configured as of 2026-09-16**: a
+Grafana datasource and the `llm-fleet` dashboard are provisioned from
+`shared/grafana/provisioning/`, and `shared/prometheus/rules/llm-fleet.yml`
+loads 5 alert rules + 1 recording rule. Nothing is *delivered* — there is no
+Alertmanager, so a firing alert reaches nobody (an operator decision, not an
+oversight). Five Prometheus targets read down for structural reasons
+(HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter.
+See `serverAppConfigs-docs/Infrastructure/Observability.md`.
+
 ## Testing
 
-Tests are mandatory before commit. `bats` must be installed on the
+Tests are mandatory before commit (`bats` required on the dev machine):
-development machine.
 
 ```
 ./tests/run.sh
 ```
 
-Test seams: compose validity, env consistency (no leaked secrets), Caddyfile
+Seams: compose validity, env consistency (no leaked secrets), Caddyfile
-validity, quadlet units, docs completeness, script syntax (all of `scripts/`,
+validity, quadlet units, docs completeness **including vault link integrity**,
-including the LLM serve scripts), LLM unit integrity (units point at files in
+script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
-this repo), LLM secret hygiene, deployment flow. See
+deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
-[docs/Architecture/Testing Policy.md](docs/Architecture/Testing%20Policy.md).
+and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
+no project, because such a spec silently never runs.
+See `serverAppConfigs-docs/Architecture/Testing Policy.md`.
 
 ## Secret Rules
 
-- Never commit real secrets. The live Gitea database password and the live
+- Never commit real secrets. The live Gitea DB password and sudo password are
-  sudo password are test-enforced as leak checks.
+  test-enforced leak checks.
-- `.env` files are git-ignored. Commit only `.env.example` with placeholders.
+- `.env` files are git-ignored; commit only `.env.example` with placeholders.
-  The LLM env tunables in `scripts/` are tracked by exception (`!scripts/*.env`).
+  Exception: `scripts/*.env` LLM tunables (paths and numbers, never
-  They hold paths and numbers, never credentials.
+  credentials).
-- Do not document passwords in this repo, including sudo passwords. SSH keys
+- Do not document passwords anywhere in the repo. SSH keys are the auth
-  are the auth mechanism. Headless sudo passes the password via the
+  mechanism; `FRANK_SUDO_PW` covers headless sudo and never enters the repo.
-  `FRANK_SUDO_PW` environment variable. The value never enters the repo.
+- **A token passed as `curl -H "Authorization: Bearer …"` is world-readable.**
+  It lands in `/proc/<pid>/cmdline`, so any `ps`/`pgrep -a` on the host prints
+  it — that leaked an HF token into a session transcript on 2026-09-16, while
+  merely checking whether a download was still running. Pass secrets out of
+  band (`curl -K <mode-600 file>` carrying the header, or `HF_TOKEN` in the
+  environment for `hf`/`hf_transfer`) and never `pgrep -a` a fetch command that
+  carries an auth header.
+
+## External Exposure Policy (MANDATORY)
+
+Every Caddy-proxied service is either behind the auth chain (Anubis :8923 →
+Authentik embedded outpost) or explicitly internal-only — no public DNS name,
+no vhost. New services default to internal-only until guarded.
+
+When guarding a new vhost — provider first, then route:
+
+1. Create the outpost proxy provider FIRST (`external_host` = exact vhost URL
+   with trailing slash; `internal_host = http://caddy:8017`). If the vhost
+   flips first, authenticated traffic hits the outpost error page.
+2. Flip the vhost to `reverse_proxy anubis:8923` with
+   `header_up X-Real-Ip {remote_host}`, and add the post-auth
+   `@<svc> host <vhost>` handler on the `http://:8017` listener routing to
+   the real upstream.
+3. Anubis guard set, Caddy TLS edge, and outpost provider list change
+   together in one deploy.
+4. Service-to-service traffic targets internal addresses (`loki:3100`), never
+   the public hostname — non-browser clients cannot solve challenges.
+
+Exempt categories (document the reason in the vhost comment): the IdP itself
+(`auth.`), git smart-HTTP endpoints, services with their own complete auth.
 
 ## Deletion & Trash Policy (MANDATORY)
 
-Never run `rm` to delete files. The command deletes permanently. Move every
+Never `rm`. Move deletions into trash so they stay recoverable:
-item you delete into trash instead. The trash keeps the deletion recoverable
-and reviewable.
 
-1. In-repo files: move the item to `.AGENTTRASH/` beside its directory.
+1. In-repo files → `.AGENTTRASH/` beside the item's directory.
-2. Home and other paths: create a dated staging directory
+2. Elsewhere → dated staging dir `~/.AGENTTRASH/YYYYMMDD-topic`.
-   `~/.AGENTTRASH/YYYYMMDD-topic` and move each item there.
+3. List staged items and sizes; wait for explicit user approval to purge.
-3. List the staged items and their sizes for the user.
-4. Wait for explicit user permission before you empty the trash.
 
-Rules:
+`rm`, `rm -rf`, and `git clean -f` are forbidden without explicit permission.
-
+`git * --no-verify` is forbidden. The trash is a temporary hold, not an
-- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user
+archive.
-  permission.
-- `git * --no-verify` is forbidden.
-- A moved item stays in the trash until the user approves its purge.
-- The trash is a temporary hold, not a long-term archive.
 
 ## Benchmarking Policy (MANDATORY)
 
-**STOP the production service BEFORE running any benchmark on a target.**
+**STOP the production service BEFORE benchmarking** — these servers serve
-These servers serve real users (about 3 people). Running a bench while the
+real users, and a bench beside a live service starves the GPU and corrupts
-service is up starves the GPU and corrupts results. Restart the service when
+results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm
-the bench finishes.
+bench` (vLLM), and halogen-flash-server's own `bench`/`sweep` modes for
+frank-large, which is the active runtime there again. Rolling your own bench
+requires asking first.
+Scratch ports are not used on LLM hosts — a
+test process cannot share the single GPU anyway; validate against the service
+itself.
 
-1. Stop the service before any benchmark:
+This covers **any GPU experiment, not just benchmarks** — including a bare
-   `sudo systemctl stop frank-default-small` (Dankserver) or
+2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
-   `sudo systemctl stop frank-default-large` (Dankclaw).
+against the live service "because it only needs 200 MB of VRAM"; the host went
-2. Run the bench on a scratch port (never the LLM service port 7770).
+hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no
-3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
+recovery — a power cycle was required). A hung AQL dispatch wedging the GPU
-4. Verify health before reporting:
+driver is the plausible mechanism, and this host has precedent for going
-   `curl http://192.168.0.235:7770/health` or
+unreachable until a power cycle. If an experiment needs GPUs, stop and mask
-   `curl http://192.168.0.143:7770/health`.
+first, or use a host that serves nobody. Telling an off machine from a wedged
+one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
+merely wedged kernel usually still answers ARP.
 
-## Pi Model Config (`~/.pi/agent/models.json`)
+1. Stop and mask (self-healing would restart it):
+   - Dankclaw `frank-large` (rootless Quadlet again since 2026-09-16):
+     `systemctl --user stop frank-large`, then move its
+     `~/.config/containers/systemd/frank-large.container` aside and
+     `systemctl --user daemon-reload` — the quadlet generator re-creates that
+     unit from the file on every reload, so the source is the lever, not `mask`.
+   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
+     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
+   - Dankserver `frank-medium` (system unit):
+     `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
+2. Bench.
+3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
 
-The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`.
+**CRITICAL:** never `systemctl disable` a system unit on these hosts — it
-This file lives outside the repository. Each provider is one host endpoint.
+deletes the unit file from `/etc/systemd/system/`, not just the symlink.
-Each model `id` is the served alias. The two aliases are
+`mask` symlinks to `/dev/null` and is safe/reversible.
-`frank-default-small` (Dankserver `:7770`) and `frank-default-large`
-(Dankclaw `:7770`).
 
-The model `id` values are stable. The file
+## Model Config (harness)
-`/home/dank/.pi/agent/settings.json` references them. Two resolution rules
-apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`):
 
-- The default model resolves as a `defaultProvider` + `defaultModel` pair.
+The harness reaches the fleet through `.omp/agent/models.yml` in the
-  The `defaultProvider` must name the provider that serves the `defaultModel`.
+operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
-- A subagent override uses a bare model `id`. A bare `id` resolves only when
+where the harness runs; there is no copy on the servers. Each provider is one
-  it is unique across all providers.
+HOST:PORT; each model `id` is the served alias: `frank-medium` →
+`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
+`192.168.0.143:7771`. Per-role routing lives beside it in `config.yml` under
+`modelRoles`.
 
-When you rename a service or move a model to a new host, update
+Resolution rules: the default model resolves as a `defaultProvider` +
-`models.json`. Update the provider key, the model `id`, `contextWindow`, and
+`defaultModel` pair (the provider must serve the model); a subagent override
-`input`. If the default model moved hosts, also update `defaultProvider` in
+uses a bare model `id`, which resolves only when unique across providers. Two
-`settings.json`. Set `contextWindow` to the per-slot context (both hosts
+ports on one host need two providers — the file cannot express "same host,
-`262144`). The full schema is in the agent knowledge base page
+different port".
-`~/.pi/agent/docs/pi/models.md`.
 
-## Runtime & Troubleshooting
+When renaming a service, retiring an endpoint, or moving a model: update the
+provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
+`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
+a role pointing at a provider the file does not define fails at call time. As
+of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
+`frank-large` is text-only because its halogen deployment loads no vision
+tower. There is **no audio** input anywhere in the fleet: that capability left
+with the FLM NPU endpoint on 2026-09-11.
 
-### v3 CIRU ActiveFPX PromptForge llama.cpp (Dankclaw, ACTIVE)
+### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
 
-The v3 runtime serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3`
+halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16**
-from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`
+(`dankclaw/frank-large/quadlet/frank-large.container`), serving
-(PromptForge v2.3 + v3 Output-K8 runtime patch). The v3 profile:
+`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + a
-`PROMPTFORGE_MODE=m2048_fused_tail1476`, full-W8 companion views, Output-K8
+quality overlay that re-fetched to 2.40 GiB on 2026-09-16) on port 7770 from
-MTP draft proxy, BF16 mmproj vision, TEMPERATURE 0.7 / TOP_P 0.95,
+the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384,
-batch 4096 / ubatch 4096, 16 context checkpoints, reasoning on (deepseek,
+2 slots), greedy default, reasoning extracted to `reasoning_content`, and **no
-budget -1), `--no-mmap`. Five-file payload (~36.7 GiB) in
+vision tower** — that moved to frank-small. It replaced ik_llama.cpp, which had
-`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. Full runbook:
+replaced it for two days.
-[docs/LLM/ciru-activefpx-promptforge.md](docs/LLM/ciru-activefpx-promptforge.md).
 
-### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22)
+The engine **fetches its own weights on first start** (`HALOGEN_DOWNLOAD`, into
+the `/home/dank/models/halogen` bind mount): clearing that directory and
+restarting the unit is the supported way to pick up upstream updates, and it
+needs the directory to exist **empty** — a missing directory fails the
+container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure` until `mkdir -p` brings
+it back. The sidecars are versioned in place, so a stale copy is invisible from
+the file name (both overlays were ~95 MB larger upstream than the copies this
+host had been serving).
 
-The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge`
+Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit):
-(pinned tag `kairic-edge-qwen38-27b-v1`). It was deployed and reverted to v3
+device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so
-the same day (operator decision). The runtime routes qualified prompt and
+the configured pool was **not clamped**; 68.0 GiB of weights locked + 11.0 GiB
-verification shapes through the RDNA 3.5 native IU4 matrix instruction.
+of KV pool + 16.5 GiB of working memory = 95.4 GiB in all, leaving 14.7 GiB
+(475 contiguous 2 MiB blocks) for everything else: frank-small on the 3090 and
+~28 co-tenant containers. The engine printed its co-tenant stall NOTE and
+logged 4,350 compaction stalls (285 failed) while reserving the pool, so read
+that as "tight and fragmented", not "out of memory".
 
-**Release gap — CK `config.h`:** the pinned CK commit ships only
+Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
-`config.h.in`. The release script and guide never generate
+Rollback: [[frank-large-ik-uncensored]].
-`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK
-CMake configure and copy it in before building (full runbook in
-[docs/LLM/kairic-edge.md](docs/LLM/kairic-edge.md)). The file is untracked
-in CK git. A `git clean` removes it. Local rebuilds are byte-identical
-(`d7625a0c...`).
 
-**Target greedy argmax fast path:** the pinned runner sets
+### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
-`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy
-requests (400) and the card's own chat sampling is non-greedy, so the
-production attempt ran it at 0. It still rejects reasoning budgets unless
-compatibility mode is used. Do not re-deploy Kairic without re-testing chat
-first.
 
-**Rollback target:** the v3 model files remain in
+Mainline llama.cpp master `37b53fd` as a **root systemd unit** (repo
-`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in
+`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
-`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
+serving `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` at IQ3_S — the `-mtp` build, so
-Restored to v3 on 2026-08-22 (commit `c60156f`).
+the model's own Multi-Token Prediction head drives speculative decoding — with
+the BF16 mmproj, on port 7771, 131,072 ctx / 1 slot, f16 KV, greedy default.
+Deployed flags are in `scripts/frank-small.env`; the serve script refuses to
+start if the weights, the projector or the binary are missing.
 
-### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21)
+Verified at deploy (2026-09-16): `/v1/models` reports `frank-small`; a chat
+completion returns content plus `reasoning_content`; a tool call returns
+`tool_calls`; the slot timing lines show `draft acceptance = 0.66–0.94, mean
+len ≈ 3` (MTP engaged); an image request is answered from the projector while
+the halogen side refuses images with a named reason; 22,362 MiB of the card's
+24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`.
 
-A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was
+### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
-reverted the same day for speed. Its serve script, env, and the build
-worktree are archived in `.AGENTTRASH/` and documented in
-[docs/LLM/dflash2-large-vulkan.md](docs/LLM/dflash2-large-vulkan.md).
 
-### FastMTP patched llama.cpp (Dankserver, retired 2026-08-18)
+Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
+systemd unit `scripts/frank-medium.service`) serving
+`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
+(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
+draftless **ngram-mod** matcher, in **one process across both R9700s**,
+`--split-mode layer` at a **35:30 tensor split** (the balance point for the
+deployed batch: the drafter rides ROCm1, so it takes fewer target layers),
+262144 ctx over **2 slots** of 131072, batch 6144x512, text only.  DSpark was
+put up in the drafter's place the same day and reverted: it declares no sliding
+window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
+layers of the split.
 
-Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18
+**Layer split is why both cards now serve ONE model.** Tensor parallelism is
-(files in `.AGENTTRASH/`). The build notes are kept for reference. The
+impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
-HauhauCS FastMTP draft sidecar required a llama.cpp build from commit
+its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact
-`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
+primitive that cannot be dispatched on card #2. llama.cpp's layer split needs
-`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
+no collective at all: layers are distributed in order and the boundary
-`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with a
+activation is a per-backend copy, once per forward pass. Measured on this host
-stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not
+(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor.
-rebuild from another commit without re-applying the patch.
 
-**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`.
+This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
-Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls
+routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
-back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
+live, with their units moved to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
-vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
-for the base recipe.
 
-### Env var naming rule — HIP runtime collision
+Files: `scripts/build-llama-rdna4.sh` (builds the runtime, gfx1201 only, private
+ROCm 7.14 toolchain bundled beside the binary), `scripts/download-frank-medium-gguf.sh`
+(resumable, byte-verified payload), `scripts/frank-medium.env`,
+`scripts/serve-frank-medium.sh`, `scripts/frank-medium.service`. The deploy
+handler refuses to start a unit whose runtime or payload is missing, retires
+the vLLM quadlets, waits for 7770 to be released (the retired proxy held it) and
+gates on `/health`.
 
-The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`,
+The ref is pinned by two features and must not float: **DFlash2**
-and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with
+(ggml-org/llama.cpp#27342) and **qwen35 MTP**. The ROCmFPX fork the endpoint
-`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 `TEMP=1.0`
+used earlier that day has neither.
-made HIP use the path `1.0` for its JIT kernel cache, then segfaulted every
-model load. Commit `227ce01` fixed it by renaming the sampling variable to
-`TEMPERATURE`.
 
-Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for
+Runbook: `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md`.
-collisions with this command:
+Retired runtimes: [[frank-medium-radiance-mxfp4]] (vLLM replicas, same day),
+[[frank-medium-rocmfpx]] (llama.cpp fork, earlier that day).
 
-```
+## Key Lessons Learned
-strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"
+Durable operational lessons from the fleet's history. Full detail in the
-```
+linked runbooks.
 
-### Crash-loop diagnostic procedure
+**PCIe / multi-GPU**
 
-When a llama service segfaults during model load, follow these steps.
+- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
+  kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
+  them when the device has no atomics, and the failure surfaces as
+  `hipErrorIllegalState` ("the operation cannot be performed in the present
+  state") from `hipLaunchKernel`, not as anything that names atomics. The
+  value is consumed at driver probe time, so writing the AtomicOp Requester
+  bit with `setpci` at runtime changes nothing. Check the *chain*:
+  `lspci -vv -s <gpu>` and every upstream bridge must show
+  `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
+  endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
+  6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
+  and the rest behind the chipset (usually not) — that asymmetry is what
+  blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
+  short-circuits at `world_size=1`.
+- **The fix is physical — and cheap on this board.** Two paths put both cards
+  on CPU root ports, which restores the atomics routing and therefore TP=2: an
+  M.2→PCIe x4 riser into a *CPU-attached* M.2 slot (on dankserver's TUF
+  GAMING B650-E WIFI that is M.2_2, which is free — per ASUS's spec page;
+  `PCIEX16_2`, where card #2 sits today, is chipset and x1-mode), or a board
+  whose CPU x16 bifurcates to x8/x8. Bandwidth is not the constraint: a 27B
+  TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4.
+  RCCL's P2P `send`/`recv` fails on such a card too, so pipeline parallelism is
+  not an escape either. Full evidence matrix, the open rebuild hypothesis, and
+  the test plan: [[Dual-R9700-Cooperation]].
+- **When a GPU collective dies, read the runtime, not the framework.** The
+  message that identified the cause came from `AMD_LOG_LEVEL=3`
+  (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
+  failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
+  prints the failing HIP call with its return code.
+- **The one software path that lets both cards serve ONE model: llama.cpp's
+  layer split.** vLLM cannot do it here in any of its parallelisms — TP needs
+  an allreduce, PP needs the same p2p primitive, and both die on the card that
+  cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no
+  collective: layers are distributed in order and only the boundary activation
+  crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
+  18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
+  fallback). That is what `frank-medium` runs — see
+  [[frank-medium-llamacpp-dflash2]].
+- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
+  combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
+  split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
+  held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
+  depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
+  across devices: `--override-tensor <group>=<device>` sets
+  `has_tensor_overrides()`, a REQUIRED term of the scheduler's
+  `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
+  that reads it), and that flag is what sets `n_copies` in
+  `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
+  non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
+  reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
+  target that works, because the CPU backend is last in the list. Pure Vulkan
+  changes nothing here — the gate is the override, not the driver mix. See
+  [[x4-pair-experiment]].
+- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
+  constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
+  services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
+  ceiling — with async no better than sync and pageable within 1 % of pinned,
+  i.e. the copies are link-limited rather than staging-limited (the usual
+  "pageable costs 2x" rule does not apply here). The card is x16-capable and the
+  Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
+  `00:03.1`), so the x4 lives in the adapter or cable; the link idles at
+  2.5 GT/s x4 and trains up under load. Consequence for any engine that
+  *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
+  model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
+  20 % resident cache), against **20.46 tok/s** for the same model with the
+  experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
+  carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
+  computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
+  this streaming shape — also loads only modelopt block-FP8 safetensors for
+  `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
+  the question. See [[x4-pair-experiment]].
 
-```mermaid
-graph TD
-    A[Service SEGVs at load] --> B[Read journal + kernel segfault line]
-    B --> C{Segfault in libamdhip64?}
-    C -- yes --> D[Bisect script vs raw binary]
-    C -- no --> E[Check config, memory, model files]
-    D --> F[Run gdb for backtrace]
-    F --> G[Find root cause]
-```
 
-1. Read the service journal with `journalctl -u <service> -n 100`. Then read
+**ROCm / GPU**
-   the kernel segfault line with `journalctl -k`. A segfault at address 8 in
-   `libamdhip64` means a NULL unref in the HIP runtime.
-2. Run the serve script directly. Then run the raw binary with the same
-   flags. A different outcome isolates the env file as the cause.
-3. Run gdb for the backtrace with this command:
-   ```
-   gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh
-   ```
-   `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost.
-   Trust gdb, not the log.
-4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root
-   cause. Investigate other causes before assuming memory pressure.
 
-### Operational quirks
+- **Layer split is the deployed shape on both multi-device llama.cpp services,
+  for two different reasons.** On Dankserver it is the *only* way both R9700s
+  serve one model (no collective can be dispatched on card #2). On a
+  RCCL-less fork build it is also the fast one: tensor parallel falls back to
+  a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
+  Pin the spec-decode drafter to one device — an unpinned draft under tensor
+  mode aborts the KV build, and on a layer split the natural home is the card
+  that ends up holding the target's last layers.
+- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
+  trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
+  one directory level. rocBLAS and hipBLASLt look for their device code at a
+  fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
+  and the flattened copy fails at *request* time, not load time: the engine
+  reads the weights, answers `/health` and dies on the first completion with
+  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
+  : gfx1201`. Declare the destination, then copy the contents.
+- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
+  ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
+  only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
+  `-ot` (23 → 13.4 t/s).
+- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve
+  scripts export env files with `set -a`, so a tunable named `TEMP` leaks
+  into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every
+  load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable
+  these; check with
+  `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
+- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash
+  attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+
+  t/s, measured 2026-08-17).
+- **Flash-Next MTP heads do not load on any current loader** — they need full
+  per-layer `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified
+  3x). Don't re-attempt without new loader support.
+- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
+  qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
+  the agentionai FAST layer set; the kingjones STRIX set works unpatched.
+- **GPU budget on Dankclaw is the rocm-smi figure** (64 GiB, SVM-backed).
+  `free`/`dmidecode` show the CPU-side carve-out — intentional, do not
+  change.
 
-- Both hosts allow passwordless sudo for the deploy user (`(ALL) NOPASSWD:
+**Load / crash diagnosis**
-  ALL`, verified 2026-08-22). Use plain `sudo` in committed scripts. Do not
-  use `echo password | sudo -S` in committed scripts. The `FRANK_SUDO_PW`
-  env var covers headless runs on a host without that.
-- Never run `pkill -f` with a pattern that appears in your own command line.
-  It kills your ssh session. Use `pkill -x <exact-name>`.
-- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM
-  (`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw v3
-  loads the ~36.7 GiB five-file payload in ~18 s with a warm page cache,
-  a few minutes cold. Health returns 503 while loading. Do not kill the
-  process. `deploy.sh` waits up to 20 minutes on both targets (see
-  `LLM_WAIT`).
-- A client abort and a server release look different in the journal. A
-  client abort logs `W srv stop: cancel task, id_task = N`. A natural stop
-  (EOS or limit) logs only `slot release` with `truncated = 0`. The server
-  never preempts a running generation. When all slots are busy, new tasks
-  defer into a queue. See the 2026-08-13 incident in
-  [docs/LLM/deployment.md](docs/LLM/deployment.md).
-- ComfyUI on Dankclaw runs CPU-only (`--cpu`). ROCm wheels do not support
-  gfx1151. Do not use CUDA images.
-- SearXNG outbound traffic must stay on the Gluetun tunnel. Do not move
-  its ports off the gluetun service.
-- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash
-  premature-EOS bug are historical. See
-  [docs/LLM/dankserver-vram-growth.md](docs/LLM/dankserver-vram-growth.md)
-  and [docs/LLM/qwen3.8-spec-matrix.md](docs/LLM/qwen3.8-spec-matrix.md).
-  The FastFlowLM NPU endpoint is retired. See
-  [docs/LLM/FastFlowLM NPU Inference.md](docs/LLM/FastFlowLM%20NPU%20Inference.md).
 
-### Dankclaw memory model
+- Model load is slow (minutes cold); health returns 503 while loading; do
+  not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On
+  halogen, the FIRST request after a (re)start can go silent ~45–60 s
+  (health `engine_unresponsive`, 503s) and then self-recovers — do not
+  restart over it.
+- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` + kernel line via
+  `journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in
+  HIP); (2) run serve script, then raw binary — different outcome isolates
+  the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash
+  scripts/serve-<svc>.sh` — llama-server buffers stdout and loses the log
+  tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed`
+  spam is a symptom, not a root cause.
+- Journal signatures: client abort logs `W srv stop: cancel task, id_task =
+  N`; natural stop logs only `slot release` with `truncated = 0`. The server
+  never preempts a running generation; excess tasks queue when slots are
+  busy.
 
-The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported by
+**Host memory / swap**
-rocm-smi (SVM-backed). The docs' older 96 GiB figure was the addressable
+
-SVM pool. The GPU node reports `local_mem_size 0`, so all allocations are
+- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
-GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or
+  of the old one, or live pages must land in RAM that `MemAvailable`
-`dmidecode` for the GPU budget. They show the CPU-side carve-out. This is
+  overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
-intentional. Do not change it.
+  the co-tenants, not frank-large — pinned weights don't swap, and paging
+  the KV pool is worse than the OOM it prevents.
+- Host swap state is invisible to `deploy.sh` (no repo file) — check
+  `swapon --show` before reasoning about headroom.
+
+**Runtime-specific guards**
+
+- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
+  `third_party/composable_kernel/include/ck/config.h` generated manually
+  (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
+  non-greedy requests and reasoning budgets. Re-test chat before production.
+- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
+  stock binaries fail the draft load with a tensor-shape mismatch.
+
+**llama.cpp serving**
+
+- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
+  carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
+  `--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
+  the MAIN model's weights (`shares_model`), so there is no second copy of a
+  12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
+  tensors at all, and pointing `--model-draft` at the file would load it twice.
+- **What can be chained, and what cannot.** `--spec-type a,b` builds every
+  enabled implementation and chains drafting through them, in a fixed internal
+  order (ngram types first, then the drafters — CLI order is irrelevant). The
+  gate is `ctx_dft`: **draftless** types (the ngram family) are added
+  unconditionally, while `draft-mtp`, `draft-dflash`, `draft-dspark` and
+  `draft-eagle3` all require a draft CONTEXT. So a chain may hold one
+  context-backed drafter plus any number of ngram types — `draft-dflash,ngram-mod`
+  is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-
+  identical to it on prose at the fleet's 48/64 window) — but
+  `draft-mtp,draft-dflash` dies at load: *"context type MTP requested but model
+  doesn't contain MTP layers"*, because MTP builds its context against whatever
+  model the draft context holds. Two context-backed types cannot share one
+  process, and MTP's head only loads when MTP **is** the draft model.
+- **A drafter's KV scales with the TARGET's context, and its window policy
+  decides how much that costs.** The draft context is built with
+  `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
+  no flag to cap it — so a drafter that declares NO sliding window caches the
+  whole window on whatever device `--spec-draft-device` names. Swapping DFlash2
+  (`dflash.attention.sliding_window = 2048`, ~40 MiB of draft KV) for DSpark
+  (no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position,
+  so **5.0 GiB at 262144**) put ROCm1 at 33.9 GB of 34.2 GB and killed the
+  FIRST draft graph capture with `ROCm error: out of memory` — after the model
+  had loaded and `/health` had answered. Read `sliding_window` out of the
+  drafter's GGUF header before sizing any spec-decode deployment, and prefer a
+  skewed `-ts` over a context cut when the full window matters. The pair was
+  swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round
+  trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
+  36:29), which is the practical size of the effect. See
+  [[frank-medium-llamacpp-dflash2]].
+- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
+  compute buffers land about TWICE as heavy on the card that holds the output
+  layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
+  shape (262144 over two slots, a 64k prefill measured at 64k depth,
+  `scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068
+  at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280,
+  1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the
+  SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs
+  ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical
+  `--batch-size` does not matter: 2048 through 8192 land within 2 % of each
+  other at a fixed ubatch and use identical VRAM. Decode never moves at all
+  (15.88 tok/s at depth 65536 on every cell). Because the buffers are
+  asymmetric, a split balanced at one batch is unbalanced at another: 512 ->
+  1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30.
+  Re-measure the balance after changing the batch. (8192/2048 failed to allocate
+  its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
+  free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
+  model — the sweep's or the seam's — can see it.)
+- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
+  size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
+  worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
+  (0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both
+  measured on frank-medium. Balance by measurement, never by interpolating
+  between two splits; an affine fit reproduces the deployed point and overstates
+  the gaps in between. `tests/frank_medium_unit.bats` encodes that fit as a
+  gate. See [[frank-medium-llamacpp-dflash2]].
+- **ngram-mod prints no startup banner.** The load log shows the drafter's
+  `adding speculative implementation 'draft-dflash'` and nothing at all for the
+  matcher, so a chain can look unengaged while it is working. Identify it
+  behaviourally (`draft_n` > 0 only where text repeats) or by the per-task
+  `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB
+  host-RAM hash table) and its window decides which traffic it can see: too
+  short and it proposes on ordinary prose fragments, which costs verification
+  without acceptance.
+- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
+  `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
+  then served frank-large: every tool-using agent on that endpoint was broken
+  and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
+  `--jinja` on and the serve script pins it; halogen also accepts `tools` and
+  returns `tool_calls`. Check tool support before routing a role at an endpoint.
+- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
+  option and the same ~12x prefill collapse described in the ROCm note above.
+  f16 KV needs nothing, which is why frank-small runs f16.
+- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
+  `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
+  configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
+  values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
+  everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
+  conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
+  bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
+  table's own `dev` column was printing the truth the whole time. Any
+  placement-dependent number needs an independent check: sample the other
+  device's memory (`nvidia-smi`) while the run is loaded.
+- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
+  template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
+  raises on anything else — HTTP 500, not a clamp — so the harness's levels
+  must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
+  A role whose level is unmapped fails on its FIRST request, and the failure
+  names Jinja, not the model: every subagent on `frank-small` died with
+  `Unexpected reasoning effort high` until `high: xhigh` was added on
+  2026-09-16. Halogen accepts the wider set, so the two entries legitimately
+  differ.
+
+**Ops**
+
+- Never `pkill -f` a pattern that appears in your own command line — it kills
+  your ssh session. Use `pkill -x <exact-name>`.
+- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
+  committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
+  without it.
+- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
+  `AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
+  world-writable — user-unit spawns carry no supplementary groups). It must NOT
+  pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
+  2026-09-16, when a stale device-less container was found restarted 9,146
+  times (`RuntimeError: No HIP GPUs are available`).
+- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
+  ports off the gluetun service.
+- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
+  2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
+  needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
+  in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
+- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
+  only the CURRENT instance; history across recreations lives in
+  `journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
+  (per-user storage — `sudo podman logs` finds nothing).
+- **A dead rootless podman pause process breaks ALL container management on the
+  host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
+  command fails with `invalid internal status … could not find any running
+  process`, every container start fails binding podman's bridge addresses
+  (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
+  assign requested address`), and every `Restart=always` quadlet loops. Container
+  storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
+  token, not a pid) and the rootless netns holding the bridges — deleting the
+  stale token is not enough, and `podman system migrate` segfaults in
+  `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
+  quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
+  first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
+  trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
+  had no other word on it). Treat the pause process as host state that can die
+  for its own reasons, and reach for the reboot when it does.
+- **A container cannot reach a `Network=host` service at the host's LAN IP.**
+  netavark DNATs *published* ports only: from a bridge container
+  `192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
+  gateway `10.89.1.1` are both refused. Use `host.containers.internal`
+  (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
+  upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
+  was open from that bridge — it never was).
+- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
+  `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
+  carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
+  + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
+  IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
+  sha256, concatenate in numeric order — that is how the deployed 111 GiB file
+  was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
+  Q4_K_S look like it does not exist.
+- **A fetch script must `-f` and verify, or it reports success on an access
+  error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
+  as the "artifact": a gated repo yielded three 153-byte shards with identical
+  sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
+  `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
+  size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
+  already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
+  `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
+  return 403 even with the host's HF token until that account accepts the gate.
+
+## Runbook Index
+
+Active:
+
+| Service | Runbook |
+|---------|---------|
+| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
+| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
+| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
+| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
+| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
+
+Retired (rollback targets and history):
+
+| Runtime | Host | Window | Runbook |
+|---------|------|--------|---------|
+| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
+| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
+| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
+| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
+| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
+| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
+| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
+| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
+| FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
+| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
+| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
+
+Retired runbooks are under `serverAppConfigs-docs/LLM/`. Retired scripts live
+in `old/` and `.AGENTTRASH/`.
 
 ## Why
 
-Direct edits on the remote drift from version control. Future agents will not
+Direct edits on the remote drift from version control, and future agents
-know what changed. Git is the single source of truth. The two workloads share
+won't know what changed. Git is the single source of truth. The two workloads
-two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
+share two hosts, one Caddy, and one Gitea; one repo keeps them in one
-
+history.
-> User request → Fetch from remote → Commit here → Edit → Test → Commit → Push → Pull on remote → Deploy + restart
 
 ## Summary
 
 | Context | Action |
 |---------|--------|
-| Editing a compose/Caddyfile/serve script | **Work through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote directly. |
+| Editing a compose/Caddyfile/serve script | **Through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote. |
-| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
+| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before editing. |
-| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
+| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
-| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls on the target, restarts the stack and the LLM service, and checks both health endpoints. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
+| Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
-| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
+| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
+| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. |
 | Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
 | Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
-| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |
+| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |

README.md

@@ -12,8 +12,8 @@ one deploy script.
 
 | Host | User | IP | Podman stack | LLM service | LLM port |
 |------|------|----|--------------|-------------|----------|
-| dankserver | dank | 192.168.0.235 | media and library | frank-default-small (llama.cpp, Qwen3.6-35B-A3B) | 7770 |
+| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
-| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-default-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
+| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 |
 
 ## Active Services
 
@@ -22,7 +22,7 @@ one deploy script.
 | RomM | dankserver | romm.dankserver.local | 8085 |
 | Copyparty | dankserver | files.dankserver.local | 3923 |
 | Calibre-Web | dankserver | books.dankserver.local | 8083 |
-| frank-default-small | dankserver | — (bare endpoint) | 7770 |
+| frank-medium | dankserver | — (bare endpoint) | 7770 |
 | SearXNG | dankclaw | search.dankclaw.local | 8081 |
 | ComfyUI | dankclaw | comfyui.dankclaw.local | 8188 |
 | OpenWebUI | dankclaw | chat.dankclaw.local | 3000 |
@@ -30,33 +30,34 @@ one deploy script.
 | Gitea | dankclaw | gitea.dankclaw.local | 3001 |
 | Grafana | dankclaw | grafana.dankclaw.local | 3002 |
 | Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
-| frank-default-large | dankclaw | — (bare endpoint) | 7770 |
+| frank-large | dankclaw | — (bare endpoint) | 7770 |
 
 ## Structure
 
 - `dankserver/` — compose.yaml + Caddyfile + .env.example for the Dankserver stack
 - `dankclaw/` — compose.yaml + Caddyfile + .env.example for the Dankclaw stack
 - `shared/` — homepage, prometheus config, deploy helpers
-- `scripts/` — `deploy.sh` (unified: stack + LLM), LLM serve scripts, systemd units, env tunables, bench harnesses
+- `scripts/` — `deploy.sh` and `deploy-remote.sh` (service-scoped deployment), the planner, LLM serve scripts, systemd units, env tunables, and bench harnesses
 - `old/` — retired PP/NPU-era LLM scripts (reference only)
-- `docs/` — project knowledge base (Obsidian vault, wikilinks). `docs/LLM/` is the LLM fleet
+- `serverAppConfigs-docs/` — project knowledge base (Obsidian vault, wikilinks). `serverAppConfigs-docs/LLM/` is the LLM fleet
 - `tests/` — bats test suite (see [[Testing Policy]])
 
 ## Deployment
 
-See [docs/Architecture/Deployment Flow.md](docs/Architecture/Deployment%20Flow.md)
+See [serverAppConfigs-docs/Architecture/Deployment Flow.md](serverAppConfigs-docs/Architecture/Deployment%20Flow.md)
-and [docs/LLM/deployment.md](docs/LLM/deployment.md). Run
+and [serverAppConfigs-docs/LLM/deployment.md](serverAppConfigs-docs/LLM/deployment.md). Run
-`./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes.
+`./scripts/deploy.sh <dankclaw|dankserver>` after committing changes. It
-It pushes, pulls on the target, restarts the podman stack and the LLM
+plans and deploys only the changed services. Use `--service NAME` for an
-service, and checks both health endpoints.
+explicit single-service retry; inference is restarted only when its own files
+(or the shared chat template) changed.
 
 ## Knowledge Base
 
-The `docs/` directory is an Obsidian vault. Start at
+The `serverAppConfigs-docs/` directory is an Obsidian vault. Start at
-[docs/index.md](docs/index.md). The LLM fleet knowledge base lives at
+[serverAppConfigs-docs/index.md](serverAppConfigs-docs/index.md). The LLM fleet knowledge base lives at
-[docs/LLM/index.md](docs/LLM/index.md).
+[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
 
 ## Testing
 
 Run `./tests/run.sh` before every push. See
-[docs/Architecture/Testing Policy.md](docs/Architecture/Testing%20Policy.md).
+[serverAppConfigs-docs/Architecture/Testing Policy.md](serverAppConfigs-docs/Architecture/Testing%20Policy.md).

dankclaw/.env.example

@@ -20,16 +20,18 @@ WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s
 # Gitea database
 GITEA_DB_PASSWORD=changeme
 
-# Grafana admin
+# Grafana OAuth2 client secret (authentik OAuth2Provider "grafana", pk 9).
-GRAFANA_ADMIN_USER=admin
+# Created via ak shell on authentik-server; the live .env must hold the
-GRAFANA_ADMIN_PASSWORD=changeme
+# provider's generated client_secret verbatim. Rotation = rotate the provider
+# secret in authentik, restart grafana after updating the live .env value.
+GRAFANA_OAUTH_CLIENT_SECRET=changeme
 
 # Caddy (the homelab's single reverse proxy, on this host)
 # bcrypt hash (caddy hash-password). deploy.sh renders it into
-# /home/dank/Containers/caddy/Caddyfile.
+# /home/dank/Containers/caddy/Caddyfile, which the container mounts.
-# CADDY_BASICAUTH_HASH: the shared family password, still used by the
+# CADDY_BASICAUTH_HASH: the shared family password, now used only by the
-#   vhosts that carry basicauth rather than Authentik forward_auth
+#   pihole vhost (grafana moved behind the Authentik guard chain 2026-09-01).
-#   (grafana, pihole). Username: dankuser.
+#   Username: dankuser.
 CADDY_BASICAUTH_HASH=changeme
 # (The homepage's per-user hashes were retired: the homepage and its /admin/
 #  page are now behind Authentik forward_auth, not basicauth.)
@@ -40,6 +42,11 @@ SIGNAL_ACCOUNT=+15551234567
 # Comma-separated E.164 numbers allowed to message the agent.
 SIGNAL_ALLOWED_USERS=+15551234567
 
+# Hermes web UI (Hermex server) login password. Required: the webui binds to
+# loopback and is reached only through the SSH tunnel, but Hermex still signs
+# in with this password. Set a strong value in the live .env.
+HERMES_WEBUI_PASSWORD=changeme
+
 # Porkbun API (franksdank.cloud + franksdank.casa zones). Caddy uses these
 # for DNS-01 Let's Encrypt wildcard cert issuance. Porkbun API keys are
 # account-wide; rotate them here rather than in the dashboard.
@@ -54,3 +61,16 @@ AUTHENTIK_PG_PASS=changeme
 # AUTHENTIK_SECRET_KEY: the server's secret key (64+ chars).
 #   Generate: openssl rand -base64 48
 AUTHENTIK_SECRET_KEY=changeme
+
+# Marreta (content-cleaning proxy, behind the Authentik guard chain).
+# deploy-remote.sh seeds /home/dank/Containers/marreta/env from these three
+# values (only if that file does not already exist) and the marreta quadlet
+# maps them to the app's APP_KEY / ADMIN_EMAIL / ADMIN_PASSWORD.
+# MARRETA_APP_KEY: Laravel encryption key. Generate:
+#   openssl rand -base64 32
+# Changing it after first boot invalidates encrypted session/cookie data.
+MARRETA_APP_KEY=changeme
+# Admin panel login (marreta.franksdank.cloud/admin). The app creates this
+# account on first boot from these values.
+MARRETA_ADMIN_EMAIL=admin@franksdank.cloud
+MARRETA_ADMIN_PASSWORD=changeme

dankclaw/Caddyfile

@@ -6,18 +6,32 @@
 #     published LAN port (quadlet services on isolated networks)
 #   - dankserver services (192.168.0.235): by published LAN port
 #
-# Naming: flat franksdank.cloud scheme (see docs/Infrastructure/LAN-DNS.md).
+# Naming: flat franksdank.cloud scheme (see serverAppConfigs-docs/Infrastructure/LAN-DNS.md).
 # DNS for these names is served by the Pi-hole quadlet on this host
 # (dankclaw/pihole/addn-hosts — every name resolves to 192.168.0.143)
 # and handed to LAN clients by the SURFboard G36 router.
 #
-# Authentication: Caddy forward_auth (Authentik SSO) protects the vhosts that
+# Authentication chain (2026-08-29): the guarded vhosts (homepage, SearXNG,
-# do not have their own application-level auth (homepage, SearXNG, Hermes,
+# Prometheus, RomM, Copyparty) proxy EVERYTHING through the Anubis
-# Prometheus, RomM, Copyparty). The basicauth hash is a placeholder; deploy.sh
+# bot-challenge proxy first, then the whole request is proxied THROUGH
-# renders the real hash (CADDY_BASICAUTH_HASH from the host .env, git-ignored)
+# Authentik (2026.8 proxy-provider mode: this build serves no forward_auth
-# into /home/dank/Containers/caddy/Caddyfile, which the container mounts.
+# endpoint; the authentication flow runs inline — unauthenticated clients get
-# Apps with their own auth (Grafana, Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole)
+# the login UI served in place, and once a session exists Authentik proxies
-# are not wrapped in forward_auth — they handle their own login.
+# the request on to the app via the provider's internal_host). Chain: client →
+# Caddy (TLS, public listener) → Anubis :8923 (verify: DENY known AI scrapers,
+# CHALLENGE everyone else) → Caddy's internal listener http://:8016 (auth
+# decision; Host headers survive every hop) → authenticated traffic is
+# forwarded back to http://:8017 and routed straight to the app.
+# Apps with their own auth (Gitea, OpenWebUI, Jellyfin, *arr, Pi-hole) and the
+# Authentik IdP itself (auth.franksdank.cloud — its login flow must never sit
+# behind a challenge) are exempt by construction: their vhosts never route
+# through Anubis. ComfyUI left this exempt set on 2026-08-31 and Grafana left
+# it on 2026-09-01 (OAuth2 relying-party mode against authentik): both now sit
+# behind the guard chain; Grafana authenticates users itself via the OAuth2
+# code flow, so no identity header is forwarded to it.
+# The pihole vhost keeps its basicauth edge; the hash placeholder below is
+# rendered by deploy.sh from CADDY_BASICAUTH_HASH (host .env, git-ignored) into
+# /home/dank/Containers/caddy/Caddyfile, which the container mounts.
 #
 # TLS: real Let's Encrypt certs. Site addresses carry no http:// prefix,
 # so Caddy serves HTTPS (and auto-redirects HTTP). Certs are issued via
@@ -28,7 +42,7 @@
 {
 	# Admin stays ON: it binds to 127.0.0.1:2019 inside the container and the
 	# port is not published, so nothing outside the container can reach it.
-	# It powers the config file watcher, so a rendered Caddyfile change
+	# It powers the config file watcher, so a rendered-config change
 	# hot-reloads without a container restart. 'admin off' once left a broken
 	# basicauth hash sitting on disk unapplied for hours (2026-08-22).
 	email admin@franksdank.cloud
@@ -44,59 +58,38 @@
 
 # --- Bare-IP access (health checks, convenience) ---
 :80 {
-	redir https://franksdank.cloud/ 302
+	redir https://www.franksdank.cloud/ 308
 }
 
-# --- Homepage (Authentik SSO) ---
+# --- Homepage (Anubis → Authentik SSO) ---
-franksdank.cloud www.franksdank.cloud {
+# www holds the homepage; the naked domain redirects onto it at the TLS edge.
-	root * /srv/homepage
+# A redirect needs no challenge, so unauthenticated naked traffic never enters
+# the auth chain. The proxy provider's external_host must match the host that
+# actually carries traffic (https://www.franksdank.cloud) or Authentik matches
+# zero providers and answers its error page instead.
+franksdank.cloud {
+	redir https://www.franksdank.cloud/ 308
+}
 
-	route {
+www.franksdank.cloud {
-		# Authentik outpost (login UI + session management).
+	reverse_proxy anubis:8923 {
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
+		header_up X-Real-Ip {remote_host}
-
-		# Forward auth: check the Authentik session before serving the page.
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-
-		# Operator-only admin page. Auth is checked on the original path,
-		# before any rewrite, so the path matcher is reliable. 'handle'
-		# preserves order.
-		handle /admin* {
-			try_files /admin.html
-			file_server
-		}
-
-		# Shared data + assets: fetched by both the user page and the admin
-		# page. With forward_auth, any authenticated user can access them
-		# (the browser's Authentik session covers these fetches).
-		@data path /services.json /status.json /app.js /style.css /deployed.js
-		handle @data {
-			file_server
-		}
-
-		# Everything else (the user page).
-		handle {
-			try_files /index.html
-			file_server
-		}
 	}
 }
 
-# --- Legacy name: the old homepage name redirects to the naked domain ---
+# --- Legacy name: the old homepage name redirects straight onto www ---
 dankclaw.franksdank.cloud {
-	redir https://franksdank.cloud/ 302
+	redir https://www.franksdank.cloud/ 308
 }
 
 # --- Legacy name: the media host has no Caddy of its own anymore ---
 dankserver.franksdank.cloud {
-	redir https://franksdank.cloud/ 302
+	redir https://www.franksdank.cloud/ 308
 }
 
 # --- Authentik (SSO / OIDC provider) ---
+# Exempt by construction: the IdP login flow must be reachable without a
+# challenge, so this vhost never routes through Anubis.
 auth.franksdank.cloud {
 	reverse_proxy authentik-server:9000
 }
@@ -107,69 +100,66 @@ auth.franksdank.cloud {
 
 # --- SearXNG (quadlet; behind the Gluetun tunnel, published 8081) ---
 search.franksdank.cloud {
-	route {
+	reverse_proxy anubis:8923 {
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
+		header_up X-Real-Ip {remote_host}
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-		reverse_proxy 192.168.0.143:8081
 	}
 }
 
-# --- ComfyUI ---
-comfyui.franksdank.cloud {
-	reverse_proxy comfyui:8188
-}
-
 # --- OpenWebUI (quadlet on slirp4netns; published 3005) ---
-chat.franksdank.cloud {
+ai.franksdank.cloud {
 	reverse_proxy 192.168.0.143:3005
 }
 
-# --- Hermes ---
-hermes.franksdank.cloud {
-	route {
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-		reverse_proxy hermes:8090
-	}
-}
-
 # --- Gitea (quadlet; published 3000) ---
+# Exempt by construction: Gitea has its own login, and git smart-HTTP
+# clients (git push/fetch over HTTPS) cannot solve JS challenges. The
+# policy would need a bypass for /info/refs + /git-*-pack paths if this
+# ever moves behind Anubis.
 gitea.franksdank.cloud {
 	reverse_proxy 192.168.0.143:3000
 }
 
-# --- Grafana ---
+# --- Grafana (guarded: Anubis → Authentik outpost gate; login inside Grafana) ---
+# Auth model = OAuth2 relying party against authentik (OAuth2Provider "grafana",
+# confidential client). The guard chain gates unauthenticated traffic; the
+# outpost carries no provider token in this deployment, so identity headers
+# never reach Grafana — RP mode needs none and works regardless of topology.
+# The basicauth edge this replaces (CADDY_BASICAUTH_HASH) is retired.
 grafana.franksdank.cloud {
-	basicauth {
+	reverse_proxy anubis:8923 {
-		dankuser REPLACE_WITH_HASH
+		header_up X-Real-Ip {remote_host}
 	}
-	reverse_proxy grafana:3000
 }
 
 # --- Prometheus ---
 prometheus.franksdank.cloud {
-	route {
+	reverse_proxy anubis:8923 {
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
+		header_up X-Real-Ip {remote_host}
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-		reverse_proxy prometheus:9090
 	}
 }
 
-# --- Loki ---
+# --- Loki (guarded: Anubis → Authentik; loki-proxy provider, internal_host caddy:8017) ---
 loki.franksdank.cloud {
-	reverse_proxy loki:3100
+	reverse_proxy anubis:8923 {
+		header_up X-Real-Ip {remote_host}
+	}
+}
+
+# --- Seerr (guarded: Anubis → Authentik; seerr provider, internal_host caddy:8017) ---
+seerr.franksdank.cloud {
+	reverse_proxy anubis:8923 {
+		header_up X-Real-Ip {remote_host}
+	}
+}
+
+# --- Marreta (guarded: Anubis → Authentik; marreta provider, internal_host caddy:8017) ---
+# Content-cleaning proxy. Any authentik-authenticated user may use it (no
+# group gate). Egress rides the searxng gluetun tunnel; reached via the
+# host-published port 8082 (gluetun netns), not by container name.
+marreta.franksdank.cloud {
+	reverse_proxy anubis:8923 {
+		header_up X-Real-Ip {remote_host}
+	}
 }
 
 # --- Pi-hole (quadlet on slirp4netns; published 8083) ---
@@ -180,6 +170,32 @@ pihole.franksdank.cloud {
 	reverse_proxy 192.168.0.143:8083
 }
 
+# --- Hermes WebUI (Hermex webui; host-netns container, published 8787) ------
+# Exempt by construction: the webui authenticates itself (HERMES_WEBUI_PASSWORD),
+# so no Anubis challenge and no Authentik proxy provider is required — nothing
+# here enters the http://:8016 auth chain. Internal-only by DNS absence: there is
+# no public Porkbun A record for this name, so only LAN clients (Pi-hole
+# addn-hosts → 192.168.0.143) can resolve it at all; DNS-01 issuance via the
+# Porkbun module needs no public record. Upstream is `host.containers.internal`
+# — netavark's host address, 169.254.1.2 — NOT the host LAN IP: hermes-webui is
+# `Network=host`, so 8787 is a host-netns bind and not a published port, and
+# netavark DNATs only published ports (192.168.0.143:9119, the dashboard below,
+# answers from this bridge; 192.168.0.143:8787 does not). Measured live
+# 2026-09-16 inside the Caddy container: 192.168.0.143:8787 and 10.89.1.1:8787
+# both REFUSED, host.containers.internal:8787 → 200.
+hermes.franksdank.cloud {
+	reverse_proxy host.containers.internal:8787
+}
+
+# --- Hermes dashboard (admin UI; published 9119 on dankclaw_default) --------
+# Exempt by construction: the app enforces its own complete auth — verified live
+# from Caddy's own network, an unauthenticated GET / returns 302 → /login?next=%2F
+# (uvicorn), and basic_auth.username + a scrypt password_hash are set in the
+# dashboard's /data/config.yaml. No outpost provider needed.
+hermesadmin.franksdank.cloud {
+	reverse_proxy 192.168.0.143:9119
+}
+
 # ═══════════════════════════════════════════════════════════════
 # Services on dankserver (192.168.0.235) — published LAN ports
 # ═══════════════════════════════════════════════════════════════
@@ -190,11 +206,24 @@ pihole.franksdank.cloud {
 # Let's Encrypt cert for it. It stays plain-HTTP legacy; the canonical
 # name is plex.franksdank.cloud (HTTPS, below).
 http://plex.franksdank.casa {
-	reverse_proxy 192.168.0.235:32400
+	reverse_proxy 192.168.0.235:32400 {
+		transport http {
+			tls_insecure_skip_verify
+		}
+	}
 }
 
+# PMS requires TLS from every non-loopback client: plain-HTTP off-loopback
+# is read and silently closed (diagnosed 2026-08-26), which surfaced as an
+# empty-body 502 here. Dial the upstream over TLS; PMS's self-signed cert is
+# skipped on this pinned-LAN hop only (client-facing LE TLS is untouched).
+# The web UI lives at /web/; / returns the XML MediaContainer index.
 plex.franksdank.cloud {
-	reverse_proxy 192.168.0.235:32400
+	reverse_proxy 192.168.0.235:32400 {
+		transport http {
+			tls_insecure_skip_verify
+		}
+	}
 }
 
 # --- Calibre-Web ---
@@ -204,62 +233,170 @@ books.franksdank.cloud {
 
 # --- RomM ---
 romm.franksdank.cloud {
-	route {
+	reverse_proxy anubis:8923 {
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
+		header_up X-Real-Ip {remote_host}
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-		reverse_proxy 192.168.0.235:8085
 	}
 }
 
 # --- Copyparty ---
 files.franksdank.cloud {
-	route {
+	reverse_proxy anubis:8923 {
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
+		header_up X-Real-Ip {remote_host}
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-		reverse_proxy 192.168.0.235:3923
 	}
 }
 
 # --- Jellyfin ---
-jellyfin.franksdank.cloud {
+jelly.franksdank.cloud {
-	reverse_proxy 192.168.0.235:8096 {
+	reverse_proxy 192.168.0.235:8096
-		transport http {
+}
-			versions h2c 1.1
+
-		}
+
-	}
+# --- *arr Stack (exempt by construction: own auth) ---
+
+prowlarr.franksdank.cloud {
+	reverse_proxy 192.168.0.112:9696
 }
 
-# --- *arr Stack ---
 radarr.franksdank.cloud {
-	reverse_proxy 192.168.0.235:7878
+	reverse_proxy 192.168.0.112:7878
 }
 
-sonarr.franksdank.cloud {
+# --- Sonarr: internal-only by construction (retired 2026-09-04) -----------
-	reverse_proxy 192.168.0.235:8989
+# No vhost and no public DNS record for sonarr.franksdank.cloud: the TV-library
-}
+# admin UI is reachable only on the LAN at 192.168.0.112:8989 (users request TV
+# through Seerr, which sits behind the guard chain; admins reach Sonarr's own
+# UI directly). A brief unguarded public vhost lived here and served the
+# zero-auth UI to the internet via manual DNS — retired.
 
 lidarr.franksdank.cloud {
-	reverse_proxy 192.168.0.235:8686
+	reverse_proxy 192.168.0.112:8686
 }
 
 readarr.franksdank.cloud {
-	reverse_proxy 192.168.0.235:8787
+	reverse_proxy 192.168.0.112:8787
-}
-
-prowlarr.franksdank.cloud {
-	reverse_proxy 192.168.0.235:9696
 }
 
 qbittorrent.franksdank.cloud {
-	reverse_proxy 192.168.0.235:8080
+	reverse_proxy 192.168.0.112:8080
+}
+
+# --- Internal listener: auth decision layer ---------------------------------
+# Anubis dials http://caddy:8016 after verifying a client (TARGET in
+# dankclaw/anubis/quadlet/anubis.container). Host headers survive the hop.
+# Every request here is proxied whole-request to Authentik with Host
+# preserved. This build's embedded outpost routes ONLY on the ORIGINAL Host
+# against each proxy provider's external_host — meta headers are ignored and
+# cross-host absolute next= redirects are rejected by the flow's open-redirect
+# guard ("Request has been denied: Invalid next URL"), verified live.
+# Unauthenticated clients get the login flow served in place; authenticated
+# sessions are forwarded by the provider's internal_host back to
+# http://:8017 and routed straight to the app. Every host that may enter this
+# listener MUST have a proxy provider registered with the embedded outpost —
+# the Anubis guard set, the Caddy TLS edge, and the provider list stay in
+# lockstep. Exempt vhosts never enter this block.
+http://:8016 {
+	reverse_proxy authentik-server:9000 {
+		header_up X-Forwarded-Proto https
+	}
+}
+
+# --- Post-auth listener: where Authentik forwards authenticated traffic ----
+# The proxy provider's internal_host is http://caddy:8017, so every request
+# that passes authentication lands here (Host header preserved) and routes
+# straight to its app. This is the end of the chain — nothing dials back
+# into Authentik from here, so the chain cannot loop.
+http://:8017 {
+	@home host www.franksdank.cloud
+	handle @home {
+		root * /srv/homepage
+		# RBAC (2026-09-01): /admin requires membership in the homepage-admins
+		# authentik group. In proxy-provider mode the outpost injects
+		# X-Authentik-Groups on its upstream request to this listener and strips
+		# any client-supplied X-Authentik-* headers first, so browsers cannot
+		# spoof it. Regex is a substring match, insensitive to the group-list
+		# separator.
+		handle /admin* {
+			@adminok header_regexp X-Authentik-Groups homepage-admins
+			handle @adminok {
+				try_files /admin.html
+				file_server
+			}
+			respond "Forbidden: homepage admins only" 403
+		}
+		@data path /services.json /status.json /app.js /style.css /deployed.js
+		handle @data {
+			file_server
+		}
+		handle {
+			try_files /index.html
+			file_server
+		}
+	}
+
+	@search host search.franksdank.cloud
+	handle @search {
+		reverse_proxy 192.168.0.143:8081
+	}
+
+	@loki host loki.franksdank.cloud
+	handle @loki {
+		reverse_proxy loki:3100
+	}
+
+	@seerr host seerr.franksdank.cloud
+	handle @seerr {
+		reverse_proxy seerr:5055
+	}
+
+	@grafana host grafana.franksdank.cloud
+	handle @grafana {
+		reverse_proxy grafana:3000
+	}
+
+	@prometheus host prometheus.franksdank.cloud
+	handle @prometheus {
+		reverse_proxy prometheus:9090
+	}
+
+	@romm host romm.franksdank.cloud
+	handle @romm {
+		reverse_proxy 192.168.0.235:8085
+	}
+
+	@files host files.franksdank.cloud
+	handle @files {
+		reverse_proxy 192.168.0.235:3923
+	}
+
+	@prowlarr host prowlarr.franksdank.cloud
+	handle @prowlarr {
+		reverse_proxy 192.168.0.112:9696
+	}
+
+	@radarr host radarr.franksdank.cloud
+	handle @radarr {
+		reverse_proxy 192.168.0.112:7878
+	}
+
+	@lidarr host lidarr.franksdank.cloud
+	handle @lidarr {
+		reverse_proxy 192.168.0.112:8686
+	}
+
+	@readarr host readarr.franksdank.cloud
+	handle @readarr {
+		reverse_proxy 192.168.0.112:8787
+	}
+
+	@qbittorrent host qbittorrent.franksdank.cloud
+	handle @qbittorrent {
+		reverse_proxy 192.168.0.112:8080
+	}
+
+	@marreta host marreta.franksdank.cloud
+	handle @marreta {
+		reverse_proxy 192.168.0.143:8082
+	}
 }
 
 # --- Optional: Jackett ---

dankclaw/anubis/policy.json

@@ -0,0 +1,25 @@
+{
+  "bots": [
+    {
+      "name": "deny AI scrapers",
+      "user_agent_regex": "(?i)^(GPTBot|ChatGPT-User|OAI-SearchBot|CCBot|anthropic-ai|ClaudeBot|Claude-Web|PerplexityBot|PerplexityUser|Google-Extended|Applebot-Extended|Bytespider|Amazonbot|meta-externalagent|DuckAssistBot|cohere-ai|YouBot|KimiBot)",
+      "action": "DENY"
+    },
+    {
+      "name": "serve anubis own endpoints",
+      "path_regex": "^/\\.within\\.website",
+      "action": "ALLOW"
+    },
+    {
+      "name": "serve authentik outpost transport",
+      "path_regex": "^/outpost\\.goauthentik\\.io/",
+      "action": "ALLOW"
+    },
+    {
+      "name": "challenge everything else",
+      "path_regex": ".",
+      "action": "CHALLENGE"
+    }
+  ],
+  "dnsbl": false
+}

dankclaw/anubis/quadlet/anubis.container

@@ -0,0 +1,70 @@
+# Quadlet unit: Anubis bot-challenge proxy (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/anubis.container
+# Unit name: anubis.service (container name: anubis)
+#
+# Two-tier chain: the public Caddy listeners for the guarded vhosts proxy
+# everything to this container (port 8923, internal only — no published port,
+# so metrics on :9090 stay reachable only inside dankclaw_default); Anubis
+# verifies the client (challenge/DENY per policy) and forwards accepted
+# traffic back to Caddy's internal listener (TARGET below), where the normal
+# per-vhost routing — including Authentik forward_auth — resumes. Host headers
+# survive both hops, so vhost matching on the internal listener still works.
+#
+# Image pin: the image is pulled from ghcr.io/techarohq/anubis (source of
+# truth), but the unit runs it under a localhost/-aliased tag: on this host's
+# podman 5.7 quadlet path, registry-prefixed refs fail to resolve from local
+# storage at container create time ("image not known") even when present; the
+# localhost alias resolves (verified live 2026-08-28). v1.27.0 pinned 2026-09-02: --help
+# verified on the image itself (cookie-domain, ed25519-private-key-hex,
+# cookie-expiration-time default 168h, difficulty default 4); JSON policy
+# schema unchanged; X-Real-Ip is now REQUIRED (missing header => HTTP 500) —
+# the Caddy chain always sets it (bats seam enforces the header_up line).
+# Upgrading = pull new tag, re-tag the localhost alias, restart the unit.
+#
+# No secrets in this repo. COOKIE_DOMAIN makes the challenge cookie valid for
+# all *.franksdank.cloud vhosts: solve the proof-of-work once (e.g. on www)
+# and every guarded vhost honors it until expiry (default 168h, aligned with
+# authentik's 7-day session) or re-login. ED25519_PRIVATE_KEY_HEX pins the
+# JWT signing key (host file /home/dank/Containers/anubis/env, mode 0600,
+# gitea-style ${VAR} expansion via the [Service] EnvironmentFile): without
+# it every restart invalidates everyone's solved challenges. See
+# serverAppConfigs-docs/Infrastructure/Anubis.md.
+
+[Unit]
+Description=Anubis bot-challenge proxy container
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=anubis
+Image=localhost/techarohq/anubis:v1.27.0
+Pull=never
+Network=dankclaw_default
+# flagenv maps env names to flag names (TARGET -> --target, POLICY_FNAME ->
+# --policy-fname). TARGET points back at Caddy's internal listener; the
+# container DNS name resolves on dankclaw_default.
+Environment=TARGET=http://caddy:8016
+Environment=POLICY_FNAME=/policy/policy.json
+Environment=COOKIE_DOMAIN=franksdank.cloud
+EnvironmentFile=/home/dank/Containers/anubis/env
+Environment=ED25519_PRIVATE_KEY_HEX=${ED25519_PRIVATE_KEY_HEX}
+Volume=/home/dank/serverAppConfigs/dankclaw/anubis/policy.json:/policy/policy.json:ro
+# The image ships no HEALTHCHECK (distroless ko-build: entrypoint is
+# /ko-app/anubis, no shell in the image), so the probe is declared here.
+# JSON exec form on purpose: this quadlet generator joins HealthCmd into a
+# single \x20-escaped argv element and podman then runs it as shell-form —
+# which needs a shell the distroless image does not have. A JSON array is
+# passed through verbatim and executes /ko-app/anubis directly.
+HealthCmd=["/ko-app/anubis","--healthcheck"]
+HealthInterval=15s
+HealthTimeout=5s
+HealthRetries=3
+
+[Service]
+EnvironmentFile=/home/dank/Containers/anubis/env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/authentik/quadlet/authentik-outpost-ldap.container

@@ -0,0 +1,40 @@
+# Quadlet unit: Authentik LDAP outpost (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/authentik-outpost-ldap.container
+# Unit name: authentik-outpost-ldap.service (podman 5.x drops the container- prefix)
+#
+# Serves the authentik LDAP provider ("Jellyfin LDAP", Base DN
+# dc=franksdank,dc=cloud) so dankserver's Jellyfin can authenticate its users
+# via the LDAP-Auth plugin. The type-specific ghcr.io/goauthentik/ldap image
+# listens on 3389 (ldap) / 6636 (ldaps) inside the container; published to
+# host 389/636 for cross-host binds from dankserver. The outpost token lives
+# in the host .env (AUTHENTIK_LDAP_OUTPOST_TOKEN), never in this repo. The
+# outpost dials core over websocket at AUTHENTIK_HOST; auth. is guard-chain
+# exempt (IdP).
+
+[Unit]
+Description=Authentik LDAP outpost container
+After=network.target authentik-server.service
+Wants=authentik-server.service
+
+[Container]
+ContainerName=authentik-outpost-ldap
+Network=dankclaw_default
+# Image=ghcr.io/goauthentik/ldap:2026.8.0 (amd64 digest; podman 5.7.0 refuses
+# the tag's nested manifest list, so pin the single-platform digest).
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Image=ghcr.io/goauthentik/ldap@sha256:fa33bc8f522113830e1662f4c034352f9eb08e0a7d3da2ddadad9a4b33b4cbf4
+Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
+Environment=AUTHENTIK_INSECURE=false
+Environment=AUTHENTIK_TOKEN=${AUTHENTIK_LDAP_OUTPOST_TOKEN}
+PublishPort=389:3389
+PublishPort=636:6636
+User=0:0
+
+[Service]
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/authentik/quadlet/authentik-outpost.container

@@ -0,0 +1,44 @@
+# Quadlet unit: Authentik outpost (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/authentik-outpost.container
+# Unit name: authentik-outpost.service (podman 5.x drops the container- prefix)
+#
+# In authentik 2026.8.x the proxy-provider URL space (/outpost.goauthentik.io/*)
+# is served by the Rust worker binary, not the server — verified live: with a
+# tenant-matching Host header, /if/flow/... answers 200 on the server while
+# /auth/caddy still 404s. This container runs the same image as the server and
+# worker units; Exec=worker selects the Rust worker role, and the
+# AUTHENTIK_HOST/AUTHENTIK_TOKEN env flips it into outpost mode: it dials back
+# to core over websocket (ws/outpost/<uuid>/) and serves forward-auth for Caddy
+# on its own 9000. No published port — Caddy reaches it by container DNS on
+# dankclaw_default; the token lives in the host .env, never in this repo.
+
+[Unit]
+Description=Authentik outpost container
+After=network.target authentik-postgres.service authentik-server.service
+Wants=authentik-server.service
+
+[Container]
+ContainerName=authentik-outpost
+Image=ghcr.io/goauthentik/server:2026.8.0
+Exec=worker
+Network=dankclaw_default
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Environment=AUTHENTIK_POSTGRESQL__HOST=authentik-postgres
+Environment=AUTHENTIK_POSTGRESQL__NAME=authentik
+Environment=AUTHENTIK_POSTGRESQL__USER=authentik
+Environment=AUTHENTIK_POSTGRESQL__PASSWORD=${AUTHENTIK_PG_PASS}
+Environment=AUTHENTIK_SECRET_KEY=${AUTHENTIK_SECRET_KEY}
+Environment=AUTHENTIK_HOST=https://auth.franksdank.cloud
+Environment=AUTHENTIK_INSECURE=false
+Environment=AUTHENTIK_TOKEN=${AUTHENTIK_OUTPOST_TOKEN}
+Volume=/home/dank/Containers/authentik/data:/data
+User=0:0
+
+[Service]
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/authentik/quadlet/authentik-postgres.container

@@ -0,0 +1,42 @@
+# Quadlet unit: Authentik PostgreSQL 16 (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/authentik-postgres.container
+# Unit name: authentik-postgres.service (podman 5.x drops the container- prefix)
+#
+# Replaces the compose service retired when this stack went boot-persistent
+# (rootless Podman does not auto-start bare containers at boot). Secrets stay
+# out of this file: systemd expands ${AUTHENTIK_PG_PASS} from the git-ignored
+# host .env at start time (same pattern as gitea-postgres).
+
+[Unit]
+Description=Authentik PostgreSQL container
+After=network.target
+
+[Container]
+ContainerName=authentik-postgres
+Image=docker.io/library/postgres:16-alpine
+Network=dankclaw_default
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Environment=POSTGRES_DB=authentik
+Environment=POSTGRES_USER=authentik
+Environment=POSTGRES_PASSWORD=${AUTHENTIK_PG_PASS}
+Volume=/home/dank/Containers/authentik/postgres:/var/lib/postgresql/data
+# Unquoted argv: this quadlet generator space-splits HealthCmd into the
+# healthcheck's argv array, so a quoted probe would arrive mangled (the old
+# `/bin/sh -c "pg_isready..."` form arrived as /bin/sh + broken args and
+# failed with exit 1 forever — the health status was a false negative while
+# the DB itself accepted connections). Bare argv works under both exec forms.
+HealthCmd=pg_isready -d authentik -U authentik
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=5
+HealthStartPeriod=20s
+
+[Service]
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+TimeoutStartSec=900
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/authentik/quadlet/authentik-server.container

@@ -0,0 +1,45 @@
+# Quadlet unit: Authentik core + embedded Caddy outpost (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/authentik-server.container
+# Unit name: authentik-server.service (podman 5.x drops the container- prefix)
+#
+# The embedded outpost serves /outpost.goauthentik.io/* including the
+# /auth/caddy forward_auth endpoint the Caddyfile chains to — no outpost token
+# exists for this deployment by design. Resolved by name (A2A DNS) from every
+# container on dankclaw_default, including caddy. The 9000 publish is
+# loopback-only on purpose: LAN clients reach Authentik through Caddy, never
+# around it; the loopback publish exists so host-side tooling can probe
+# /-/health/. Runs as the image default user (same userns mapping the compose
+# era used, so the existing /data ownership carries over unchanged).
+
+[Unit]
+Description=Authentik server container
+After=network.target authentik-postgres.service
+Wants=authentik-postgres.service
+
+[Container]
+ContainerName=authentik-server
+Image=ghcr.io/goauthentik/server:2026.8.0
+Exec=server
+Network=dankclaw_default
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Environment=AUTHENTIK_POSTGRESQL__HOST=authentik-postgres
+Environment=AUTHENTIK_POSTGRESQL__NAME=authentik
+Environment=AUTHENTIK_POSTGRESQL__USER=authentik
+Environment=AUTHENTIK_POSTGRESQL__PASSWORD=${AUTHENTIK_PG_PASS}
+Environment=AUTHENTIK_SECRET_KEY=${AUTHENTIK_SECRET_KEY}
+PublishPort=127.0.0.1:9000:9000
+# Probe must contain no quotes: this host's quadlet v1 generator space-splits
+# HealthCmd into the healthcheck argv array, and quoted probes arrive mangled
+# (verified live 2026-08-28). `python /manage.py check --database default`
+# exits nonzero iff Django (incl. the DB connection) cannot serve.
+HealthCmd=python /manage.py check --database default
+
+[Service]
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+TimeoutStartSec=900
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/authentik/quadlet/authentik-worker.container

@@ -0,0 +1,37 @@
+# Quadlet unit: Authentik worker — background tasks, email, webhook/log outbox
+# (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/authentik-worker.container
+# Unit name: authentik-worker.service (podman 5.x drops the container- prefix)
+#
+# Same image as the server; the Exec= command selects the worker role. Runs as
+# user 0:0 (rootless: maps to the invoking host user, matching the compose-era
+# semantics). No published ports — the worker talks to PostgreSQL in-network
+# and has no listener of its own.
+
+[Unit]
+Description=Authentik worker container
+After=network.target authentik-postgres.service authentik-server.service
+Wants=authentik-server.service
+
+[Container]
+ContainerName=authentik-worker
+Image=ghcr.io/goauthentik/server:2026.8.0
+Exec=worker
+Network=dankclaw_default
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Environment=AUTHENTIK_POSTGRESQL__HOST=authentik-postgres
+Environment=AUTHENTIK_POSTGRESQL__NAME=authentik
+Environment=AUTHENTIK_POSTGRESQL__USER=authentik
+Environment=AUTHENTIK_POSTGRESQL__PASSWORD=${AUTHENTIK_PG_PASS}
+Environment=AUTHENTIK_SECRET_KEY=${AUTHENTIK_SECRET_KEY}
+Volume=/home/dank/Containers/authentik/data:/data
+User=0:0
+
+[Service]
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/caddy/Containerfile

@@ -4,7 +4,7 @@
 # directly through the Porkbun API, so the wildcard cert for
 # *.franksdank.cloud / *.franksdank.casa can be issued and renewed without
 # any port 80/443 reachability from the internet (see the Caddyfile in
-# ../Caddyfile and docs/Infrastructure/LAN-DNS.md).
+# ../Caddyfile and serverAppConfigs-docs/Infrastructure/LAN-DNS.md).
 #
 # Build on dankclaw (podman rootless, same as every other container here):
 #   podman build -t franksdank/caddy-porkbun:latest -f dankclaw/caddy/Containerfile .

dankclaw/caddy/env.example

@@ -0,0 +1,7 @@
+# Caddy environment (placeholders only). The deploy writes the live file at
+# /home/dank/Containers/caddy/env from the checkout's .env; the quadlet reads
+# it via EnvironmentFile. Consumed by the Caddyfile's acme_dns block
+# (DNS-01 wildcard cert issuance). PORKBUN_API_SECRET_KEY is the name the
+# caddy-dns/porkbun module expects.
+PORKBUN_API_KEY=changeme
+PORKBUN_API_SECRET_KEY=changeme

dankclaw/caddy/quadlet/caddy.container

@@ -0,0 +1,54 @@
+# Quadlet unit: canonical Caddy reverse proxy + ACME (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/caddy.container
+# Unit name: caddy.service (container name: caddy)
+#
+# The homelab's single Caddy: it fronts every service on both hosts and issues
+# Let's Encrypt wildcard certs via DNS-01 (Porkbun). It left compose.yaml on
+# 2026-08-26 after a reboot exposed that the bare compose container never came
+# back: rootless Podman has no boot-time auto-start for bare containers, and
+# `restart: unless-stopped` does not survive the explicit stop that shutdown
+# performs. See AGENTS.md "Boot Persistence & Self-Healing" and
+# serverAppConfigs-docs/Infrastructure/Caddy.md.
+#
+# It joins the existing compose network (dankclaw_default) so the Caddyfile's
+# container-name upstreams (authentik-server, grafana, ...) still resolve; if
+# no compose service has created that network yet at boot, quadlet creates it
+# first and compose reuses it by name later. The image is built locally from
+# dankclaw/caddy/Containerfile and exists only on this host, so Pull=never: a
+# boot must not stall on a registry that does not have it. Porkbun credentials
+# come from /home/dank/Containers/caddy/env (deploy writes it from the
+# checkout's .env; secrets never enter the repo).
+
+[Unit]
+Description=Caddy reverse proxy + ACME (Porkbun DNS-01) container
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=caddy
+Image=franksdank/caddy-porkbun:latest
+Pull=never
+Network=dankclaw_default
+PublishPort=0.0.0.0:80:80/tcp
+PublishPort=0.0.0.0:443:443/tcp
+EnvironmentFile=/home/dank/Containers/caddy/env
+Volume=/home/dank/Containers/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
+Volume=/home/dank/Containers/caddy/data:/data
+Volume=/home/dank/Containers/caddy/config:/config
+# The homepage: the host's /srv/homepage (deploy-managed, not the repo's
+# shared/homepage).
+Volume=/srv/homepage:/srv/homepage:ro
+# The bare-IP catch-all answers 302 (curl -f passes on <400); this proves
+# Caddy itself is serving without touching any upstream.
+HealthCmd=curl -sf -o /dev/null http://localhost:80/
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/comfyui/Containerfile

@@ -1,55 +0,0 @@
-# ComfyUI on ROCm for the AMD Strix Halo APU (gfx1151).
-#
-# The official `rocm/comfyui` and pytorch.org ROCm wheels compile kernels for
-# datacenter archs (gfx942/gfx950) and omit gfx1151. This base — AMD's
-# gfx1151-tagged PyTorch build (torch 2.9.1+rocm7.12.0rc1) — is the verified
-# working GPU path on Dankclaw (2026-08-14: avail=True, Radeon 8060S, cap (11,5)).
-#
-# The base is heavier than a dedicated ComfyUI image (it carries vLLM), but it
-# is the only prebuilt image that ships gfx1151 torch kernels. See
-# docs/Research/ComfyUI ROCm and Vulkan.md.
-
-FROM docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0
-
-# ComfyUI source + runtime requirements (non-GPU deps install from PyPI).
-ENV PYTHONUNBUFFERED=1 \
-    COMFYUI_DIR=/opt/ComfyUI
-
-WORKDIR /opt
-
-# git ships in the base; clone the stable ComfyUI tag.
-RUN git clone --depth 1 --branch v0.3.6 \
-        https://github.com/Comfy-Org/ComfyUI.git "$COMFYUI_DIR"
-
-WORKDIR "$COMFYUI_DIR"
-
-# Install ComfyUI Python requirements. Do NOT reinstall torch/torchvision —
-# keep the gfx1151-enabled torch from the base. The requirements pin torch,
-# so install everything else from a deps-only pass.
-RUN /opt/python/bin/pip install --no-cache-dir \
-        -r requirements.txt \
-        --no-deps 2>/dev/null || true
-
-# Ensure the managed deps that are missing get installed without touching torch.
-# (First pass is best-effort; a reinstall of torch is deliberately avoided.)
-RUN /opt/python/bin/python -m pip install --no-cache-dir \
-        requests==2.32.3 \
-        aiohttp==3.10.10 \
-        yarl==1.13.1 \
-        pillow==10.4.0 \
-        numpy>=1.24 \
-        huggingface-hub>=0.24 \
-        einops \
-        comfyui-frontend-package>=1.0 \
-        model-management>=1.0.3 \
-        opaque>=0.0.10 \
-        pydantic>=2.6 \
-        websockets>=15 \
-        imageio
-
-# Run ComfyUI as the host user so bind mounts /opt/comfyui are writable.
-USER 1000:1000
-
-EXPOSE 8188
-
-ENTRYPOINT ["/opt/python/bin/python", "main.py", "--listen", "0.0.0.0", "--port", "8188"]

dankclaw/comfyui/comfyui-healthcheck.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+"""ComfyUI container healthcheck: the UI must answer on :8188.
+
+Runs INSIDE the container, where the UI listens (the unit publishes that port on
+the host loopback for the homepage status card). ComfyUI only answers once
+torch, the custom nodes and the graph are loaded, so a 200 here means the
+service is usable rather than merely a live process — the distinction that
+matters, because the failure this replaced was a container that stayed "up"
+while dying on every start.
+
+HealthCmd in the quadlet splits on spaces into an argv array (podman 5.x v1
+generator), which is why this is a file and not an inline `python3 -c`.
+"""
+import sys
+import urllib.request
+
+try:
+    with urllib.request.urlopen("http://127.0.0.1:8188/", timeout=10) as resp:
+        sys.exit(0 if resp.status == 200 else 1)
+except Exception:
+    sys.exit(1)

dankclaw/comfyui/quadlet/comfyui.container

@@ -0,0 +1,68 @@
+# Quadlet unit: ComfyUI (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/comfyui.container
+# Unit name: comfyui.service (container name: comfyui)
+#
+# WHY THIS FILE EXISTS. ComfyUI ran from `dankclaw/compose.yaml` through a
+# hand-written `~/.config/systemd/user/comfyui.service` that called
+# `podman compose up -d`. Its container was created 2026-09-15 14:12, BEFORE the
+# GPU declarations were added to the compose service, and `restart:
+# unless-stopped` then kept restarting that stale container — so it started
+# without /dev/kfd, without /dev/dri and without keep-groups, died on
+# `RuntimeError: No HIP GPUs are available`, and had restarted 9,146 times when
+# this was found. A recreate via compose would have fixed the container; a unit
+# fixes the class: one owner per container name, repo-tracked, boot-persistent
+# through the generator.
+#
+# GPU ACCESS. /dev/kfd + /dev/dri plus keep-groups. A literal `render` group
+# entry cannot work under rootless Podman (the group name does not resolve
+# inside the user namespace), and the companion udev rule
+# (dankclaw/udev/99-comfyui-gpu-access.rules) keeps those nodes MODE 0666 so a
+# user-unit spawn — which carries no supplementary groups — can still open them.
+# Both halves are asserted by tests/comfyui_rocm.bats.
+#
+# EXPOSURE. Loopback only. ComfyUI has no authentication and executes arbitrary
+# node graphs, and it has no Caddy vhost, so the repository's rule applies:
+# an unguarded service is internal-only. The homepage status card probes
+# 127.0.0.1:8188, which this satisfies. To reach the UI from another machine,
+# guard it behind the auth chain and give it a vhost — do not widen the bind.
+#
+# DATA. Models, input, output and custom_nodes live under
+# /home/dank/comfyui-data (host state, not in the repo).
+[Unit]
+Description=ComfyUI (gfx1151 ROCm image generation)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=comfyui
+Image=localhost/comfyui-rocm:gfx1151
+Pull=never
+Network=dankclaw_default
+PublishPort=127.0.0.1:8188:8188
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+GroupAdd=keep-groups
+Volume=/home/dank/comfyui-data/models:/opt/ComfyUI/models
+Volume=/home/dank/comfyui-data/input:/opt/ComfyUI/input
+Volume=/home/dank/comfyui-data/output:/opt/ComfyUI/output
+Volume=/home/dank/comfyui-data/custom_nodes:/opt/ComfyUI/custom_nodes
+Volume=/home/dank/serverAppConfigs/dankclaw/comfyui/comfyui-healthcheck.py:/usr/local/bin/comfyui-healthcheck.py:ro
+Exec=python main.py --listen 0.0.0.0 --port 8188
+# The probe lives in a file because the podman 5.x v1 generator splits
+# HealthCmd on spaces into an argv array; an inline `python3 -c "..."` becomes
+# a puzzle of quoting that fails silently.
+HealthCmd=python3 /usr/local/bin/comfyui-healthcheck.py
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+# First start loads torch and the node graph; give it room before the probe
+# counts against it.
+HealthStartPeriod=300s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/compose.yaml

@@ -1,315 +0,0 @@
-# Dankclaw stack — AI, search, development & observability services.
-# Deploy: cd /home/dank/serverAppConfigs && podman-compose -f dankclaw/compose.yaml up -d
-# See ../docs/Dankclaw/ and ../docs/Infrastructure/ for service documentation.
-
-services:
-  # --- SearXNG stack (gluetun + searxng + valkey) --------------------------
-  # These three services are now boot-persistent Quadlet units, not compose
-  # services. They live in searxng/quadlet/ and install to
-  # ~/.config/containers/systemd/ on the host. searxng reuses gluetun's netns
-  # (Network=gluetun.container) so its traffic exits the WireGuard tunnel.
-  # Deploy with: scripts/deploy-searxng-quadlet.sh dankclaw
-  # See ../docs/Dankclaw/SearXNG.md, Gluetun.md, Valkey.md.
-
-  # --- AI image generation (ROCm GPU backend, gfx1151) ---------------------
-  # Built from Containerfile.comfyui on AMD's gfx1151 torch base. The GPU is
-  # reached via the /dev/kfd + /dev/dri passthrough (checked host-side; the run
-  # user is already in the render group). Only the 'video' group is added:
-  # rootless podman cannot map the render gid into the container namespace.
-  # Models + data persist under /home/dank/comfyui-data. See ../docs/Dankclaw/ComfyUI.md.
-  comfyui:
-    build:
-      context: .
-      dockerfile: Containerfile.comfyui
-    image: comfyui-rocm:gfx1151
-    container_name: comfyui
-    restart: unless-stopped
-    command: ["python", "main.py", "--listen", "0.0.0.0", "--port", "8188"]
-    ports:
-      - "8188:8188"
-    devices:
-      - /dev/kfd:/dev/kfd
-      - /dev/dri:/dev/dri
-    group_add:
-      - video
-    volumes:
-      - /home/dank/comfyui-data/models:/opt/ComfyUI/models
-      - /home/dank/comfyui-data/input:/opt/ComfyUI/input
-      - /home/dank/comfyui-data/output:/opt/ComfyUI/output
-      - /home/dank/comfyui-data/custom_nodes:/opt/ComfyUI/custom_nodes
-    healthcheck:
-      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8188/')"]
-      interval: 30s
-      timeout: 10s
-      retries: 5
-      start_period: 180s
-
-  # --- Chat UI (open-webui) ------------------------------------------------
-  # Now a boot-persistent Quadlet unit (quadlet, slirp4netns, port 3005), not
-  # a compose service. See open-webui/quadlet/open-webui.container and
-  # ../docs/Dankclaw/OpenWebUI.md.
-
-  hermes:
-    image: docker.io/nousresearch/hermes-agent:latest
-    container_name: hermes
-    restart: unless-stopped
-    # The image's default s6 entrypoint runs the interactive TUI, which exits
-    # without a TTY and crash-loops the container. Override it to run the
-    # headless gateway (the documented Docker mode). The gateway drops to the
-    # unprivileged hermes user (uid 10000), so the data dir must be writable by it.
-    entrypoint: ["bash", "-c", "hermes gateway run --force -v"]
-    environment:
-      HERMES_DATA_DIR: /data
-      # Point hermes's config/.env at the mounted data dir (persistent).
-      HERMES_HOME: /data
-      # Signal adapter -> the host signal-cli daemon. The slirp host alias
-      # (host.containers.internal) can drop at runtime; the host LAN IP is the
-      # stable container->host path. 127.0.0.1 inside the container is not the host.
-      SIGNAL_HTTP_URL: "http://192.168.0.143:8080"
-      SIGNAL_ACCOUNT: "${SIGNAL_ACCOUNT}"
-      SIGNAL_ALLOWED_USERS: "${SIGNAL_ALLOWED_USERS}"
-    ports:
-      - "8090:8090"
-    volumes:
-      - /home/dank/hermes-data:/data
-    # No healthcheck: 'hermes gateway run' starts the messaging gateway (Signal
-    # SSE), not the :8090 web dashboard, so there is no HTTP endpoint to probe.
-    # The container's running state + the daemon's SSE connection are the signal.
-
-  # --- Hermes web dashboard (management UI: config, API keys, sessions) -----
-  # A separate process from the Signal gateway (the gateway entrypoint does not
-  # start the dashboard). Bound to the LAN with basic auth: the username + a
-  # scrypt password_hash live in the git-ignored /home/dank/hermes-data config.
-  # Port 9119 is published so a browser on the LAN can reach it.
-  hermes-dashboard:
-    image: docker.io/nousresearch/hermes-agent:latest
-    container_name: hermes-dashboard
-    restart: unless-stopped
-    entrypoint: ["bash", "-c", "hermes dashboard --host 0.0.0.0 --port 9119 --no-open --skip-build"]
-    environment:
-      HERMES_DATA_DIR: /data
-      HERMES_HOME: /data
-    ports:
-      - "9119:9119"
-    volumes:
-      - /home/dank/hermes-data:/data
-
-  # --- Git hosting (Gitea) -------------------------------------------------
-  # Now boot-persistent Quadlet units (private bridge network, DNS enabled),
-  # not compose services. See gitea/quadlet/ and ../docs/Dankclaw/Gitea.md.
-
-  # --- Observability -------------------------------------------------------
-  prometheus:
-    image: docker.io/prom/prometheus:latest
-    container_name: prometheus
-    restart: unless-stopped
-    # Rootless: container root maps to the invoking user (dank), which
-    # owns /opt/prometheus/data. The image's default user (nobody) maps to
-    # a subuid that cannot write it.
-    user: "0:0"
-    command:
-      - --config.file=/etc/prometheus/prometheus.yml
-      - --storage.tsdb.path=/prometheus
-      - --storage.tsdb.retention.time=30d
-      - --web.enable-lifecycle
-    ports:
-      - "9090:9090"
-    volumes:
-      - /opt/prometheus/config:/etc/prometheus
-      - /opt/prometheus/data:/prometheus
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  grafana:
-    image: docker.io/grafana/grafana:latest
-    container_name: grafana
-    restart: unless-stopped
-    # Rootless: container root maps to the invoking user (dank), which
-    # owns /opt/grafana/data. The image's default user (472) maps to a
-    # subuid that cannot write it.
-    user: "0:0"
-    environment:
-      GF_SERVER_ROOT_URL: "https://grafana.franksdank.cloud"
-      GF_SECURITY_ADMIN_USER: "${GRAFANA_ADMIN_USER}"
-      GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
-    ports:
-      - "3002:3000"
-    volumes:
-      - /opt/grafana/data:/var/lib/grafana
-      - /opt/grafana/provisioning:/etc/grafana/provisioning:ro
-    depends_on:
-      - prometheus
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  loki:
-    # Local build: grafana/loki + static busybox (healthcheck).
-    image: franksdank/loki:latest
-    container_name: loki
-    restart: unless-stopped
-    # Rootless: container root maps to the invoking user (dank), which
-    # owns /opt/loki/data. The image's default user (10001) maps to a
-    # subuid that cannot write it.
-    user: "0:0"
-    command: ["-config.file=/etc/loki/loki-config.yaml"]
-    ports:
-      - "3100:3100"
-    volumes:
-      - /opt/loki/config:/etc/loki
-      - /opt/loki/data:/loki
-    healthcheck:
-      test: ["CMD", "/usr/local/bin/busybox", "wget", "--spider", "-q", "http://localhost:3100/ready"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  alloy:
-    # Local build: grafana/alloy + static busybox (healthcheck).
-    image: franksdank/alloy:latest
-    container_name: alloy
-    restart: unless-stopped
-    volumes:
-      - /opt/alloy/config:/etc/alloy
-      - /run/systemd/journal/socket:/run/systemd/journal/socket:ro
-      - /var/log:/var/log:ro
-      - /var/lib/containers:/var/lib/containers:ro
-    depends_on:
-      - loki
-    healthcheck:
-      test: ["CMD", "/usr/local/bin/busybox", "wget", "--spider", "-q", "http://localhost:12345/-/healthy"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  node-exporter:
-    image: docker.io/prom/node-exporter:latest
-    container_name: node-exporter
-    restart: unless-stopped
-    network_mode: host
-    pid: host
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-    command:
-      - --path.procfs=/host/proc
-      - --path.sysfs=/host/sys
-      - --path.rootfs=/rootfs
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9100/metrics"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-
-  # --- Authentik (SSO / OIDC provider) -------------------------------------
-  # 3-service stack: PostgreSQL + server + worker. Redis was removed in
-  # 2025.10 (sessions/caching moved to PostgreSQL). The Caddyfile's
-  # forward_auth vhosts proxy to authentik-server:9000 (compose network).
-  # See ../docs/Infrastructure/Authentik.md.
-  authentik-postgres:
-    image: docker.io/library/postgres:16-alpine
-    container_name: authentik-postgres
-    restart: unless-stopped
-    environment:
-      POSTGRES_DB: authentik
-      POSTGRES_USER: authentik
-      POSTGRES_PASSWORD: ${AUTHENTIK_PG_PASS}
-    volumes:
-      - /home/dank/authentik/postgres:/var/lib/postgresql/data
-    healthcheck:
-      test: ["CMD-SHELL", "pg_isready -d authentik -U authentik"]
-      interval: 30s
-      retries: 5
-      start_period: 20s
-
-  authentik-server:
-    image: ghcr.io/goauthentik/server:2026.8.0
-    container_name: authentik-server
-    restart: unless-stopped
-    command: server
-    environment:
-      AUTHENTIK_POSTGRESQL__HOST: authentik-postgres
-      AUTHENTIK_POSTGRESQL__NAME: authentik
-      AUTHENTIK_POSTGRESQL__USER: authentik
-      AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_PG_PASS}
-      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
-    ports:
-      - "9000:9000"
-    volumes:
-      - /home/dank/authentik/data:/data
-    shm_size: 512mb
-    depends_on:
-      authentik-postgres:
-        condition: service_healthy
-    healthcheck:
-      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:9000/-/health/')"]
-      interval: 30s
-      timeout: 10s
-      retries: 5
-      start_period: 30s
-
-  authentik-worker:
-    image: ghcr.io/goauthentik/server:2026.8.0
-    container_name: authentik-worker
-    restart: unless-stopped
-    command: worker
-    environment:
-      AUTHENTIK_POSTGRESQL__HOST: authentik-postgres
-      AUTHENTIK_POSTGRESQL__NAME: authentik
-      AUTHENTIK_POSTGRESQL__USER: authentik
-      AUTHENTIK_POSTGRESQL__PASSWORD: ${AUTHENTIK_PG_PASS}
-      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
-    user: "0:0"
-    volumes:
-      - /home/dank/authentik/data:/data
-    depends_on:
-      authentik-postgres:
-        condition: service_healthy
-
-  # --- Reverse proxy + homepage (the homelab's single Caddy) ---------------
-  # The Caddyfile mounted here is the RENDERED copy (basicauth hash filled in
-  # from the host .env by scripts/deploy.sh); the repo copy carries the
-  # REPLACE_WITH_HASH placeholder. Certs/config live under
-  # /home/dank/Containers/caddy (rootless-writable).
-  caddy:
-    # Locally built: stock Caddy + the Porkbun DNS provider
-    # (dankclaw/caddy/Containerfile). See that file for build instructions.
-    image: franksdank/caddy-porkbun:latest
-    container_name: caddy
-    restart: unless-stopped
-    ports:
-      - "80:80"
-      - "443:443"
-    environment:
-      # Consumed by the Caddyfile's acme_dns block (DNS-01 wildcard certs).
-      PORKBUN_API_KEY: ${PORKBUN_API_KEY}
-      PORKBUN_API_SECRET_KEY: ${PORKBUN_API_SECRET_KEY}
-    volumes:
-      - /home/dank/Containers/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
-      - /home/dank/Containers/caddy/data:/data
-      - /home/dank/Containers/caddy/config:/config
-      # The homepage: the host's /srv/homepage. The deploy copies the static
-      # files there; the status checker writes status.json and the deploy
-      # writes deployed.js there. (Not the repo's shared/homepage, which
-      # lacks the generated files.)
-      - /srv/homepage:/srv/homepage:ro
-    depends_on:
-      - authentik-server
-      - comfyui
-      - hermes
-      - grafana
-      - prometheus
-      - loki
-    healthcheck:
-      # The bare-IP catch-all answers 302 (curl -f passes on <400); this
-      # proves Caddy itself is serving without touching any upstream.
-      test: ["CMD", "curl", "-sf", "-o", "/dev/null", "http://localhost:80/"]
-      interval: 30s
-      timeout: 10s
-      retries: 3

dankclaw/frank-large/quadlet/frank-large.container

@@ -0,0 +1,219 @@
+# Quadlet unit: frank-large (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
+# Unit name: frank-large.service (container name: frank-large)
+#
+# The large inference service. The halogen-flash-server runtime (since
+# 2026-09-11, restored 2026-09-16 after a two-day ik_llama.cpp hybrid
+# interlude) serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
+# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
+# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
+# prefill (the dominant cost of any real prompt) on this exact SKU. The RTX
+# 3090 it briefly shared with frank-large is now frank-small's card.
+#
+# SHAPE. The image runs both halves in one container: the engine on container
+# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
+# front-end on :8731, published here as host 7770 so the fleet's clients,
+# services.json probe and docs keep the port they have always used. Same for
+# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
+# every response, so no client had to be renamed. Halogen never rejects a
+# request for naming a different model, it just serves it.
+#
+# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
+# node. Both are 0666 on this host, so the rootless user namespace can open
+# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
+# the image's documented requirements; memlock must be unlimited.
+#
+# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
+# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
+#
+# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
+# configuration the published decode numbers were measured with, and `free`,
+# MemAvailable and every monitor built on them will overstate free memory by
+# the size of the model (~68 GiB) because the kernel counts the locked weights
+# as reclaimable page cache. Halogen prints the corrected figure at startup.
+# The pinned trunk means this container holds most of the host; a co-tenant
+# that then takes the remainder can stall both for minutes at full CPU with no
+# disk activity and no output. That is documented behaviour, not a crash, and
+# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
+# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
+# stays on as the safety net: if the machine refuses the configured pool the
+# engine lowers it itself and prints what it settled on. Read that ledger on
+# every restart — it is the ground truth for sizing, not this comment.
+#
+# POOL RETUNED 2026-09-16: 400,384 positions -- exactly two 200,192-token
+# conversations -- beside a per-conversation ceiling of 200,192 and two slots.
+# It replaces the 401,408 the 2026-09-13 trim left behind, which held one full
+# native-262k conversation beside a ~139k one, or three at 131k.
+#
+# MEASURED AT 401,408 (2026-09-13 deploy ledger): 68.0 weights + 11.0 KV +
+# 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB blocks left,
+# engine need printed ~31.5 against the modelled ~33.4, and NO stall-warning
+# NOTE. Fit-check budget here: 121.2 - 67.7 - 20 = 33.5. That boot is the
+# baseline the 09-16 retune is measured against: the vision tower (1.97 GiB)
+# and roughly 4.2 GiB of arena come back to the host, which is what pays for
+# the co-tenant below.
+#
+# CO-TENANT since 2026-09-16: this host also serves frank-small on the RTX
+# 3090 (llama.cpp, ~12.2 GiB of weights that load through the page cache and
+# then live in VRAM, plus a CUDA context). It is the first real co-tenant this
+# service has had, which is why the arena was halved and the reserve raised.
+#
+# BOOT NOTE: on the first request after a (re)start the engine can go silent
+# for ~45-60 s ("has not answered PING") while it first-touches the freshly
+# reserved pool; /health reads engine_unresponsive during it and recovers by
+# itself. The quadlet health check's HealthStartPeriod=300s covers it; do not
+# restart the unit over it.
+#
+# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
+# (the entrypoint checks whether the checkpoint is absent) into the bind mount
+# below, which must therefore be read-WRITE. huggingface_hub resumes partial
+# files natively, and the quadlet runs the container with --replace --rm, so a
+# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
+# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
+# tower and the tokenizer. The repo is public and ungated, so no token is
+# needed; later starts open no outbound connections at all.
+#
+# VISION. Deliberately NOT set since 2026-09-16. The tower is a 1.97 GiB
+# charge against the same device budget as the pool and the arena, and the
+# fleet's image path moved to frank-small (llama.cpp on the RTX 3090, loaded
+# with --mmproj mmproj-Qwen3.8-27B-BF16). This server refuses image content
+# parts, and models.yml declares frank-large text-only to match.
+#
+# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
+# There are no server-side sampler defaults: a request that sends no
+# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
+# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
+# returns finish_reason "length" with empty content and the partial reasoning
+# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
+# silent truncation.
+#
+# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
+# path cannot resolve registry-prefixed refs from local storage at create
+# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
+# restart. Pull=never keeps boots off the registry.
+[Unit]
+Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
+After=network-online.target
+
+[Container]
+ContainerName=frank-large
+Image=localhost/peonist-ai/halogen-flash-server:0.5.8
+# Published port stays 7770: Hermes, open-webui, the harness and the homepage
+# probe all point here. 8730 (the engine) is deliberately not published.
+PublishPort=7770:8731
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+# Read-write: the first start downloads into it. Contains only weights.
+Volume=/home/dank/models/halogen:/models
+# Fetch the weights on first start only (absent checkpoint gates it).
+Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
+# Fleet identity: the id at /v1/models, /health and every response body.
+Environment=HALOGEN_MODEL_ID=frank-large
+# Per-conversation ceiling: exactly half the pool, so two conversations of the
+# full window are always resident. 200,192 rather than a round 200,000 is
+# pool/2. The model's native window is 262,144; a conversation past this
+# ceiling is refused, never silently truncated. Clients declare 200192 (see
+# the frank-large contextWindow in models.yml).
+Environment=HALOGEN_CTX=200192
+# The KV pool: 400,384 positions = 2 x 200,192 (the 2026-09-16 retune, down
+# from 401,408). ~28 KiB per position including block scratch, so ~11.0 GiB
+# held -- the 401,408 ledger entry measured exactly 11.0. The two slots still
+# OVERSUBSCRIBE the pool for any real request, because admission reserves
+# prompt + max_tokens: two full-window conversations each holding an output
+# budget are pool-bound, and the second waits in arrival order for the first
+# to release its region. Read the `kv pool:` lines on every boot;
+# HALOGEN_KV_POOL_FIT lowers the pool itself if the budget refuses the
+# configured one, and a green health check does not mean the pool you asked
+# for is the pool you got.
+Environment=HALOGEN_KV_POOL_POSITIONS=400384
+# Concurrency cap: two, one per half of the pool. Each slot is 111.0 MiB of
+# O(1) state, so this hands ~111 MiB back against the three set on 2026-09-12
+# for the fleet size of the time.
+Environment=HALOGEN_KV_SLOTS=2
+# Sizing is budget arithmetic, all of it. ONE budget covers the KV pool, the
+# per-call prefill arena and the vision tower together:
+#
+#   budget = 121.2 host - 67.7 resident weights - reserve
+#   need   = KV (~28 KiB/position) + arena + vision
+#   arena  = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
+#
+# The arena term is linear in max_tok, and it reproduces five live restarts
+# AND upstream's three published "on the device" figures to within 0.1 GiB —
+# note those published figures are pool+arena totals, not pool alone, which is
+# the misreading that first made a doubling look affordable here. Corners this
+# box offered at a reserve of 24, at the old 262,144-position pool:
+#
+#   16384 -> need 28.6 vs 29.5  FITS, 0.9 spare   (doubling, 8-9% prefill cost)
+#   32768 -> need 37.0 vs 29.5  POOL CLAMPED to 262144, deploy still green
+#   65536 -> need 46.5 vs 29.5  OOM, will not boot; unit still reads ACTIVE
+#
+# LOWERED 32768 -> 24576 on 2026-09-16 for the frank-small co-tenant on this
+# same host. The arena's variable term goes 8.40 x 2 -> 8.40 x 1.5, handing
+# ~4.2 GiB back to the host at roughly a 4-5 % prefill cost -- the same lever
+# at 16,384 costs 8-9 %. The vendor's own notes name 24,576 as the largest
+# arena that allocated with headroom to spare, so this stays inside the range
+# it has tested.
+#
+# The arena is a PER-CALL PREFILL CHUNK, not a prompt ceiling: the
+# 200,192-token conversation is prefilled in arena-sized pieces, so a smaller
+# arena means more pieces and a slower prefill, not a smaller conversation.
+#
+# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
+# the fit check's printed ledger on each boot is the arbiter, not these
+# comments.
+Environment=HALOGEN_MAX_TOK=24576
+# RAM the fit check refuses to count on, watched over by the engine's own
+# n-gram lookup table (47.7 GiB, read through the page cache and never held)
+# and, since 2026-09-16, by frank-small on the same host.
+#
+# RAISED 15 -> 20 with the 2026-09-16 retune. This knob feeds the fit-check
+# budget and is NOT an allocation: raising it cannot hand memory back to the
+# host, because the actual footprint is weights + KV + arena + vision, none of
+# which it touches — it can only take the pool away. What it does is keep the
+# fit check honest at the smaller arena. With the arena at 24,576 and no
+# vision tower, the engine's printed need falls to roughly 25 GiB against a
+# budget of 121.2 - 67.7 - 20 = 33.5, leaving ~8 GiB of margin, so the pool is
+# not clamped.
+#
+# The 2026-09-13 profile for comparison: 103.5 GiB held and "7.1 GiB left for
+# everything else" (310 contiguous 2 MiB blocks) at the doubled 524,288 pool,
+# which is why the pool came down; at 401,408 the same ledger read 9.1 GiB /
+# 1396 blocks. The 09-16 retune gives back the vision tower (1.97) and ~4.2 of
+# arena on top of that for the co-tenant, so the expected figure at this boot
+# is ~15 GiB left. Read the ledger; do not trust this arithmetic.
+#
+# To free host memory, change the POOL or the ARENA, and the reserve then
+# becomes free to rise: HALOGEN_KV_POOL_POSITIONS=262144 hands back 3.6 GiB,
+# HALOGEN_MAX_TOK=16384 hands back another 4.2. Both are one-line edits the
+# fit check confirms on the next boot.
+Environment=HALOGEN_HOST_RESERVE_GIB=20
+# Verbose engine logging: the vendor's first-line evidence for load,
+# allocation and admission questions. Shipping it on means an incident report
+# already has the full startup account instead of needing a restart-into-debug.
+# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
+# over 64 MB, which is steady-state noise here; raise it only for a live
+# OOM hunt.
+Environment=HALOGEN_VERBOSE=1
+# Required by the image: the engine maps the checkpoint and registers it with
+# the GPU in place, which needs host IPC and an unconfined seccomp profile.
+PodmanArgs=--ipc=host --security-opt seccomp=unconfined
+Ulimit=memlock=-1:-1
+# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
+# for a PONG before answering, so 200 means "ready to serve", not "process
+# alive". During the first-start download it stays unhealthy by design.
+HealthCmd=halogen-healthcheck api
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=300s
+Pull=never
+
+[Service]
+# The first start reads ~68 GiB off disk and pins it before serving.
+TimeoutStartSec=900
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/gitea/quadlet/gitea-caddy.container

@@ -1,31 +0,0 @@
-# Quadlet unit: Gitea Caddy reverse proxy (rootless, boot-persistent).
-#
-# Install path on dankclaw: ~/.config/containers/systemd/gitea-caddy.container
-# Unit name: gitea-caddy.service (podman 5.x drops the container- prefix)
-#
-# Serves gitea.franksdank.cloud (public domain, DNS pending). The LAN
-# reaches Gitea directly on port 3000. This file holds no secrets.
-# See docs/Dankclaw/Gitea.md.
-
-[Unit]
-Description=Gitea Caddy reverse proxy container
-Wants=gitea-server.service
-After=gitea-server.service network.target
-
-[Container]
-ContainerName=gitea-caddy
-Network=gitea.network
-Image=docker.io/library/caddy:latest
-PublishPort=80:80
-PublishPort=443:443
-Volume=/opt/gitea/data/caddy/Caddyfile:/etc/caddy/Caddyfile
-Volume=/opt/gitea/data/caddy/site:/srv
-Volume=/opt/gitea/data/caddy/config:/config
-Volume=/opt/gitea/data/caddy/data:/root/.local/share/caddy
-
-[Service]
-TimeoutStartSec=900
-Restart=always
-
-[Install]
-WantedBy=default.target

dankclaw/gitea/quadlet/gitea-postgres.container

@@ -3,8 +3,8 @@
 # Install path on dankclaw: ~/.config/containers/systemd/gitea-postgres.container
 # Unit name: gitea-postgres.service (podman 5.x drops the container- prefix)
 #
-# The DB password lives in /opt/gitea/.env on the host (mode 0600).
+# The DB password lives in /home/dank/Containers/gitea/env on the host (mode 0600).
-# This file holds no secrets. See docs/Dankclaw/Gitea.md.
+# This file holds no secrets. See serverAppConfigs-docs/Dankclaw/Gitea.md.
 # The [Service] EnvironmentFile= line lets systemd expand
 # ${GITEA_DB_PASSWORD} in the generated ExecStart.
 
@@ -16,14 +16,14 @@ After=network.target
 ContainerName=gitea-postgres
 Network=gitea.network
 Image=docker.io/library/postgres:16-alpine
-EnvironmentFile=/opt/gitea/.env
+EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=POSTGRES_USER=gitea
 Environment=POSTGRES_PASSWORD=${GITEA_DB_PASSWORD}
 Environment=POSTGRES_DB=gitea
-Volume=/opt/gitea/data/gitea-postgres:/var/lib/postgresql/data
+Volume=/home/dank/Containers/gitea/postgres:/var/lib/postgresql/data
 
 [Service]
-EnvironmentFile=/opt/gitea/.env
+EnvironmentFile=/home/dank/Containers/gitea/env
 TimeoutStartSec=900
 Restart=always
 

dankclaw/gitea/quadlet/gitea-server.container

@@ -5,8 +5,8 @@
 #
 # The [Service] EnvironmentFile= line lets systemd expand
 # ${GITEA_DB_PASSWORD} in the generated ExecStart.
-# The DB password lives in /opt/gitea/.env on the host (mode 0600).
+# The DB password lives in /home/dank/Containers/gitea/env on the host (mode 0600).
-# This file holds no secrets. See docs/Dankclaw/Gitea.md.
+# This file holds no secrets. See serverAppConfigs-docs/Dankclaw/Gitea.md.
 #
 # Ports (live): 3000 = HTTP UI/API, 2222 = git SSH. The docs table in
 # Dankclaw Stack.md historically said 3001; the live host port is 3000.
@@ -20,7 +20,7 @@ After=gitea-postgres.service network.target
 ContainerName=gitea-server
 Network=gitea.network
 Image=docker.io/gitea/gitea:latest
-EnvironmentFile=/opt/gitea/.env
+EnvironmentFile=/home/dank/Containers/gitea/env
 Environment=DATABASE_TYPE=postgres
 Environment=DATABASE_HOST=gitea-postgres
 Environment=DATABASE_NAME=gitea
@@ -31,12 +31,12 @@ Environment=USER_UID=1000
 Environment=USER_GID=1000
 PublishPort=0.0.0.0:3000:3000
 PublishPort=2222:22
-Volume=/opt/gitea/data/gitea-gitea:/data
+Volume=/home/dank/Containers/gitea/data:/data
 Volume=/etc/timezone:/etc/timezone:ro
 Volume=/etc/localtime:/etc/localtime:ro
 
 [Service]
-EnvironmentFile=/opt/gitea/.env
+EnvironmentFile=/home/dank/Containers/gitea/env
 TimeoutStartSec=900
 Restart=always
 

dankclaw/gitea/quadlet/gitea.network

@@ -4,7 +4,7 @@
 # Unit name: gitea-network.service (podman network name: systemd-gitea)
 #
 # DNS is enabled by default for quadlet networks. This matters: the
-# automatic "podman" default network has dns_enabled=false, so container
+# automatic "podman" default network has dns_enabled=false, so the
-# names (gitea-postgres, gitea-server, gitea-caddy) would not resolve.
+# gitea-postgres and gitea-server container names would not resolve.
 
 [Network]

dankclaw/hermes/config.yaml.example

@@ -0,0 +1,100 @@
+# Hermes configuration backup / non-secret baseline.
+#
+# Source on Dankclaw: /home/dank/Containers/hermes/data/config.yaml
+# This file is a sanitized Git backup, not a complete replacement for the
+# live Hermes state. Do not add dashboard password hashes, phone numbers,
+# OAuth credentials, API keys, sessions, memories, or generated state here.
+# The live directory is bind-mounted into the containers as /data.
+
+model:
+  provider: dankclaw-large
+  base_url: http://192.168.0.143:7770/v1
+  api_key: not-needed
+  default: frank-large
+
+providers:
+  dankclaw-large:
+    api: http://192.168.0.143:7770/v1
+    discover_models: false
+    models:
+      frank-large:
+        context_length: 114688
+  dankserver-small:
+    api: http://192.168.0.235:7770/v1
+    discover_models: false
+    models:
+      frank-medium:
+        context_length: 114688
+
+# Auxiliary routing from the live configuration. Named providers route via
+# their `api:` URL; `custom` entries route by their explicit `base_url`.
+auxiliary:
+  vision: {provider: dankclaw-large, model: frank-large}
+  web_extract: {provider: dankserver-small, model: frank-medium}
+  compression: {provider: dankserver-small, model: frank-medium}
+  skills_hub: {provider: dankserver-small, model: frank-medium}
+  approval: {provider: dankserver-small, model: frank-medium}
+  mcp: {provider: dankserver-small, model: frank-medium}
+  # Title generation goes out over the Responses transport on purpose: Hermes
+  # hardcodes response_format=json_schema on this one call, no local back-end
+  # implements constrained decoding (it is a 400), and this transport is the
+  # only one that drops the field. reasoning.effort=none is required with it —
+  # the call caps output at 64 tokens, which a thinking pass would consume
+  # entirely, leaving an empty title.
+  title_generation:
+    provider: dankserver-small
+    model: frank-medium
+    api_mode: codex_responses
+    extra_body:
+      reasoning:
+        effort: none
+  memory_query_rewrite: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
+  tts_audio_tags: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
+  triage_specifier: {provider: dankclaw-large, model: frank-large}
+  kanban_decomposer: {provider: dankclaw-large, model: frank-large}
+  profile_describer: {provider: dankserver-small, model: frank-medium}
+  curator: {provider: dankserver-small, model: frank-medium}
+
+dashboard:
+  # basic_auth intentionally omitted; its password_hash remains live-only.
+  font: atkinson-hyperlegible
+moa:
+  presets:
+    default:
+      reference_models:
+        - provider: openrouter
+          model: deepseek/deepseek-v4-pro
+          enabled: false
+      degraded_reference_policy: loud
+      fanout: user_turn
+  reference_models:
+    - provider: openrouter
+      model: deepseek/deepseek-v4-pro
+      enabled: false
+  aggregator:
+    provider: openrouter
+    model: anthropic/claude-opus-4.8
+  degraded_reference_policy: loud
+  max_tokens: 4096
+  fanout: user_turn
+  enabled: true
+skills:
+  disabled: []
+onboarding:
+  seen:
+    busy_input_prompt: true
+_config_version: 34
+platforms:
+  signal:
+    # Live config has the Signal adapter disabled; http_url points at the
+    # host signal-cli daemon. The account phone number stays live-only.
+    enabled: false
+    extra:
+      http_url: http://192.168.0.143:8080
+mcp_servers:
+  chrome-devtools:
+    command: /usr/local/bin/npx
+    args:
+      - -y
+      - chrome-devtools-mcp@latest
+      - --browserUrl=http://127.0.0.1:9222

dankclaw/hermes/quadlet/chrome-devtools-protocol.container

@@ -0,0 +1,38 @@
+# Quadlet unit: Chrome DevTools Protocol daemon for the hermes agent (rootless).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/chrome-devtools-protocol.container
+# Unit name: chrome-devtools-protocol.service (container name: chrome-devtools-protocol)
+#
+# Headless Chrome exposing raw CDP over http://127.0.0.1:9222. The hermes
+# agent's chrome-devtools MCP client (mcp_servers.chrome-devtools in
+# /data/config.yaml) spawns npx chrome-devtools-mcp with --browserUrl pointing
+# at this endpoint and drives the browser over it. Both consumers — the Signal
+# gateway and the webui server — run in the host network namespace, so a
+# loopback-only bind is reachable by exactly the processes that may drive the
+# browser and by nobody else on the LAN or WAN.
+#
+# Entrypoint override drops the image's socat front (which would bind
+# 0.0.0.0:9222 in the host namespace and hand browser control to the whole
+# LAN). --no-sandbox is required because rootless podman grants Chrome no user
+# namespace for its sandbox; the content processes therefore run as the uid
+# that already owns this container, capping the blast radius of a hostile page.
+# Image pinned by digest: upgrades are a deliberate unit edit + redeploy, not a
+# tag drift at pull time.
+
+[Unit]
+Description=Chrome DevTools Protocol daemon (headless-shell) for hermes
+
+[Container]
+ContainerName=chrome-devtools-protocol
+Network=host
+Image=docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5b5fd7c0fe99215ff662339c57407ee2e8c0a11af93516b04
+Entrypoint=/headless-shell/headless-shell
+Exec=--no-sandbox --disable-gpu --use-gl=angle --use-angle=swiftshader --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
+ShmSize=1073741824
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/hermes/quadlet/hermes-dashboard.container

@@ -0,0 +1,33 @@
+# Quadlet unit: Hermes web dashboard (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/hermes-dashboard.container
+# Unit name: hermes-dashboard.service (container name: hermes-dashboard)
+#
+# Management UI for the agent (config, API keys, sessions). Published on
+# 9119 on network dankclaw_default; Caddy's hermesadmin.franksdank.cloud vhost
+# reverse-proxies this port directly (no guard chain) because the dashboard
+# enforces its own basic-auth login from /data/config.yaml
+# (basic_auth.username + scrypt password_hash), making it exempt-by-construction.
+# The name has no public DNS record, so the vhost resolves on the LAN only.
+# Shares /data with the gateway: same agent state, two processes.
+
+[Unit]
+Description=Hermes web dashboard container
+
+[Container]
+ContainerName=hermes-dashboard
+Image=localhost/hermes-agent:latest
+Pull=never
+Network=dankclaw_default
+Environment=HERMES_DATA_DIR=/data
+Environment=HERMES_HOME=/data
+Volume=/home/dank/Containers/hermes/data:/data
+PublishPort=9119:9119
+Exec=bash -c 'hermes dashboard --host 0.0.0.0 --port 9119 --no-open --skip-build'
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankclaw/hermes/quadlet/hermes-webui.container

@@ -0,0 +1,65 @@
+# Quadlet unit: Hermex webui (Hermex) container (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/hermes-webui.container
+# Unit name: hermes-webui.service (container name: hermes-webui)
+#
+# Runs the open-source hermes-webui (github.com/nesquena/hermes-webui) in
+# foreground mode: bootstrap.py creates .venv inside the mounted checkout,
+# installs WebUI + agent deps, then execs server.py as a single foreground
+# process. It reads HERMES_HOME=/data, so it drives the SAME agent state
+# (memory, skills, sessions) as the gateway container.
+#
+# Host networking on purpose: the agent's chrome-devtools MCP client spawns
+# npx chrome-devtools-mcp with --browserUrl=http://127.0.0.1:9222, and the CDP
+# daemon binds loopback-only in the host network namespace (see
+# chrome-devtools-protocol.container). Only processes sharing that namespace
+# can drive the browser; a bridge-network client could not reach it at all.
+# The server binds 0.0.0.0:8787 of the host netns, and that is deliberate:
+# this unit is host-networked (the CDP daemon above needs a shared loopback), so
+# Caddy cannot reach it by container name and fronts the vhost
+# hermes.franksdank.cloud with `reverse_proxy host.containers.internal:8787`
+# (netavark's host address, 169.254.1.2) — not the host LAN IP, which Caddy's
+# bridge netns cannot reach for a bare host-netns bind; the measured asymmetry
+# with published ports is recorded in the hermes block of dankclaw/Caddyfile.
+# A loopback bind was tried on 2026-09-16 for a tighter posture and broke that
+# vhost, because a bind is also the only way an off-host process can reach a
+# host-networked container: the LAN address and the tunnel's localhost hop are
+# both served by the wildcard, and a loopback-only bind serves only the latter.
+# Controls are the auth chain in front of the vhost plus the password.
+
+[Unit]
+Description=Hermes webui (Hermex) container
+
+[Container]
+ContainerName=hermes-webui
+Image=localhost/hermes-agent:latest
+Pull=never
+Network=host
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Environment=HERMES_WEBUI_HOST=0.0.0.0
+Environment=HERMES_WEBUI_PORT=8787
+Environment=HERMES_WEBUI_PASSWORD=${HERMES_WEBUI_PASSWORD}
+Environment=HERMES_HOME=/data
+# Headless host: never show the first-run wizard (a profile already
+# exists in /data from the gateway).
+Environment=HERMES_WEBUI_SKIP_ONBOARDING=1
+Volume=/home/dank/Containers/hermes/data:/data
+Volume=/home/dank/Containers/hermes/webui:/webui
+Exec=python3 /webui/bootstrap.py --foreground
+# JSON exec form on purpose: this quadlet generator joins a bare HealthCmd
+# into ONE argv element with \x20 escapes and then shell-executes it — the
+# python one-liner's quoting would be mangled. A JSON array is passed through
+# verbatim. Start period covers first-boot pip installs into /webui/.venv.
+HealthCmd=["python3","-c","import urllib.request;urllib.request.urlopen('http://127.0.0.1:8787/health')"]
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=5
+HealthStartPeriod=600s
+
+[Service]
+Restart=always
+RestartSec=5
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+
+[Install]
+WantedBy=default.target

dankclaw/hermes/quadlet/hermes.container

@@ -0,0 +1,45 @@
+# Quadlet unit: Hermes agent gateway (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/hermes.container
+# Unit name: hermes.service (container name: hermes)
+#
+# The messaging gateway (Signal adapter -> the host signal-cli daemon). The
+# image's default s6 entrypoint would run the interactive TUI, which exits
+# without a TTY; Exec= below replaces it with the headless gateway command,
+# spliced by the v1 quadlet generator as trailing args after the image ref
+# (same splice pattern proven live on authentik-worker). The gateway drops to
+# the unprivileged hermes user (uid 10000) at runtime; /data is world-writable
+# on purpose so that unmapped uid can write agent.log and its state there.
+#
+# Host networking: the adapter dials the signal-cli daemon, a HOST process
+# listening on :8080. The compose era dialed the host LAN IP from a bridge
+# network, where only DNAT'd published ports accept connections to that
+# address — un-forwarded host ports refuse instantly (verified live
+# 2026-08-30). A host-network client dials 127.0.0.1:8080 with no such
+# dependency, and the gateway publishes nothing inbound.
+
+[Unit]
+Description=Hermes agent gateway container
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+Network=host
+Image=localhost/hermes-agent:latest
+Pull=never
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+Environment=HERMES_DATA_DIR=/data
+Environment=HERMES_HOME=/data
+Environment=SIGNAL_HTTP_URL=http://127.0.0.1:8080
+Environment=SIGNAL_ACCOUNT=${SIGNAL_ACCOUNT}
+Environment=SIGNAL_ALLOWED_USERS=${SIGNAL_ALLOWED_USERS}
+Volume=/home/dank/Containers/hermes/data:/data
+Exec=bash -c 'hermes gateway run --force -v'
+
+[Service]
+Restart=always
+RestartSec=5
+EnvironmentFile=/home/dank/serverAppConfigs/dankclaw/.env
+
+[Install]
+WantedBy=default.target

dankclaw/loki/Containerfile

@@ -1,12 +1,21 @@
 # Loki + a static busybox.
 #
-# grafana/loki is distroless: no shell, no wget, so the compose healthcheck
+# grafana/loki is distroless: no shell, no wget, so nothing can run inside the
-# cannot run anything inside the container. This image drops a static
+# container to check it. This image drops a static busybox into it for the
-# busybox into it for the healthcheck command only. All Loki behavior is
+# healthcheck. All Loki behavior is unchanged.
-# unchanged.
+#
+# The healthcheck lives HERE, in exec form, and not in the Quadlet unit: a
+# unit's HealthCmd reaches podman as CMD-SHELL, which needs /bin/sh — the one
+# thing this image does not have. Specifying it here is also what lets the
+# container report its own health instead of looking merely "Up" while wedged.
 #
 # Build on dankclaw:
-#   podman build -t franksdank/loki:latest -f dankclaw/loki/Containerfile .
+#   podman build --format docker -t franksdank/loki:latest \
+#       -f dankclaw/loki/Containerfile .
+#
+# --format docker is required: podman's default OCI image format has no
+# HEALTHCHECK field, so a default-format build warns and silently discards the
+# one below, leaving a container that reports no health at all.
 
 FROM docker.io/library/alpine:latest AS tools
 # busybox-static is statically linked: the target images have no dynamic
@@ -15,4 +24,7 @@ FROM docker.io/library/alpine:latest AS tools
 RUN apk add --no-cache busybox-static
 
 FROM docker.io/grafana/loki:latest
-COPY --from=tools /bin/busybox.static /usr/local/bin/busybox
+COPY --from=tools /bin/busybox.static /usr/local/bin/busybox
+# Exec form (JSON array): no shell is involved.
+HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
+    CMD ["/usr/local/bin/busybox", "wget", "--spider", "-q", "http://localhost:3100/ready"]

dankclaw/marreta/quadlet/marreta-browser.container

@@ -0,0 +1,34 @@
+# Quadlet unit: Marreta browser companion — Lightpanda (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/marreta-browser.container
+# Unit name: marreta-browser.service (container name: marreta_browser)
+#
+# Lightpanda is a headless browser that exposes the Chrome DevTools Protocol
+# over a WebSocket on port 9222. Marreta drives it to render JS-heavy pages
+# the plain curl fetch cannot. It shares gluetun's network namespace so the
+# browser's OWN outbound fetches also exit the WireGuard tunnel — this is
+# required because marreta's PROXY_URL only covers the curl strategy; the
+# CDP session passes no proxy to the browser, so without netns sharing the
+# browser would leak the host's real egress IP.
+#
+# Marreta reaches it on loopback (same netns): ws://127.0.0.1:9222. No
+# published port — nothing outside the netns talks to it directly.
+# See serverAppConfigs-docs/Dankclaw/Marreta.md.
+
+[Unit]
+Description=Marreta Lightpanda browser companion container
+Wants=gluetun.service
+After=gluetun.service network.target
+
+[Container]
+ContainerName=marreta_browser
+Network=gluetun.container
+Image=docker.io/lightpanda/browser:nightly
+
+[Service]
+TimeoutStartSec=300
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/marreta/quadlet/marreta.container

@@ -0,0 +1,66 @@
+# Quadlet unit: Marreta content-cleaning proxy (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/marreta.container
+# Unit name: marreta.service (container name: marreta)
+#
+# Marreta is a Laravel app (serversideup/php:8.4-fpm-nginx) that strips
+# paywalls/tracking from a fetched page. It reuses gluetun's network
+# namespace (Network=gluetun.container) so ALL of its outbound fetches exit
+# the same WireGuard tunnel as SearXNG — the curl strategy AND, because the
+# browser companion shares the same netns, the Lightpanda fetches too
+# (marreta's PROXY_URL does NOT cover the browser: the CDP connection passes
+# no proxy, so netns sharing is the only way to keep the browser's egress on
+# the tunnel).
+#
+# Port: the shared netns already has SearXNG on 8080, so marreta must NOT
+# bind 8080. NGINX_HTTP_PORT=8085 relocates it (the fpm-nginx image does
+# not bake conf.d/*.conf; the site config is envsubst'd from
+# ${NGINX_HTTP_PORT} at container start). Gluetun publishes host 8082 ->
+# 8085 so Caddy can reach it; see dankclaw/searxng/quadlet/gluetun.container.
+# The app reaches its browser companion over loopback (same netns):
+# ws://127.0.0.1:9222.
+#
+# Secrets (APP_KEY, admin creds) live in /home/dank/Containers/marreta/env
+# (mode 0600, dank-owned so the rootless unit can read it); this file holds
+# none. See serverAppConfigs-docs/Dankclaw/Marreta.md.
+
+[Unit]
+Description=Marreta content-cleaning proxy container
+Wants=gluetun.service marreta-browser.service
+After=gluetun.service marreta-browser.service network.target
+
+[Container]
+ContainerName=marreta
+Network=gluetun.container
+Image=ghcr.io/manualdousuario/marreta:latest
+# Relocate off SearXNG's 8080 inside the shared netns.
+Environment=NGINX_HTTP_PORT=8085
+Environment=APP_NAME=Marreta
+Environment=APP_ENV=production
+Environment=APP_DEBUG=false
+Environment=APP_URL=https://marreta.franksdank.cloud
+Environment=APP_LOCALE=en
+Environment="APP_DESCRIPTION=Paywall-free reading"
+Environment=BROWSER_WS_ENDPOINT=ws://127.0.0.1:9222
+Environment=DISABLE_CACHE=false
+# Secrets expanded from the EnvironmentFile below.
+Environment=APP_KEY=${MARRETA_APP_KEY}
+Environment=ADMIN_EMAIL=${MARRETA_ADMIN_EMAIL}
+Environment=ADMIN_PASSWORD=${MARRETA_ADMIN_PASSWORD}
+# Named volume: podman auto-creates it and copyups the image's storage/app
+# ownership (www-data), so the app can write its SQLite DB + cache.
+Volume=marreta_storage:/var/www/html/storage/app
+HealthCmd=["curl","-fsS","-o","/dev/null","http://127.0.0.1:8085/up"]
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=5
+HealthStartPeriod=120s
+
+[Service]
+EnvironmentFile=/home/dank/Containers/marreta/env
+TimeoutStartSec=900
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/alloy.container

@@ -0,0 +1,63 @@
+# Quadlet unit: alloy (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/alloy.container
+# Unit name: alloy.service (container name: alloy)
+#
+# Log shipper. This replaces the compose service of the same name in
+# dankclaw/compose.yaml ("Observability" block).
+#
+# IMAGE. franksdank/alloy is built locally by deploy-remote.sh
+# (build_compose_image) from dankclaw/alloy/Containerfile: upstream
+# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a
+# boot from reaching for that local tag at a registry.
+#
+# NETWORK. It joins dankclaw_default so its config's loki.write endpoint
+# resolves the container name loki:3100.
+#
+# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own
+# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x
+# refuses the ui block), so the compose service had no reachable health
+# endpoint at all: the homepage status card for alloy reported "down"
+# continuously and there was nothing for a deploy to wait on. The command
+# below therefore moves the UI to the container's wildcard address and the
+# port is published on host loopback only, which puts alloy in the same class
+# as the other three: reachable from the host, by Caddy over the private
+# network, and by nothing off-host. Alloy has no authentication, which is
+# exactly why it is not published wider.
+#
+# SOURCES. The host journald socket and the host log/container-storage trees
+# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
+# lives at /opt/alloy/config/config.alloy, installed from
+# shared/alloy/config.alloy on every deploy.
+#
+# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
+# client fails and its log pipeline backs up.
+
+[Unit]
+Description=Alloy log shipper (Dankclaw)
+After=network-online.target loki.service
+Wants=network-online.target loki.service
+
+[Container]
+ContainerName=alloy
+Image=localhost/franksdank/alloy:latest
+Pull=never
+Network=dankclaw_default
+Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
+PublishPort=127.0.0.1:12345:12345
+Volume=/opt/alloy/config:/etc/alloy
+Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
+Volume=/var/log:/var/log:ro
+Volume=/var/lib/containers:/var/lib/containers:ro
+HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/grafana.container

@@ -0,0 +1,79 @@
+# Quadlet unit: grafana (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/grafana.container
+# Unit name: grafana.service (container name: grafana)
+#
+# Dashboards. This replaces the compose service of the same name in
+# dankclaw/compose.yaml ("Observability" block).
+#
+# PUBLISHING. Bound to 127.0.0.1 for the same reason as prometheus: the browser
+# path is https://grafana.franksdank.cloud (Caddy → Anubis → Authentik outpost →
+# grafana:3000 over dankclaw_default) and the host-side homepage status timer
+# probes 127.0.0.1:3002. Publishing on 0.0.0.0 as the compose service did left
+# an off-host path into Grafana that skips the guard chain entirely.
+#
+# AUTH. Grafana is an OAuth2 relying party against Authentik (OAuth2Provider
+# "grafana", confidential client). The guard chain fronts unauthenticated
+# traffic; the login itself happens through the Authentik button inside
+# Grafana. The client secret is NOT written here: it is expanded by systemd
+# from the root .env (EnvironmentFile in [Service], never in [Container], so
+# the container receives exactly this one secret and not the whole file).
+# Without that .env entry the unit will not start — the same dependency every
+# other rendered-secret deploy has.
+#
+# STATE. /opt/grafana/data holds grafana.db. /opt/grafana/provisioning is
+# mounted read-only and was EMPTY as of 2026-09-11, and grafana.db reported
+# zero datasources and zero dashboards: this Grafana is an empty install whose
+# only configured account is the local admin. Nothing here queries Prometheus
+# or Loki yet.
+
+[Unit]
+Description=Grafana dashboards (Dankclaw)
+After=network-online.target prometheus.service
+Wants=network-online.target prometheus.service
+
+[Container]
+ContainerName=grafana
+Image=docker.io/grafana/grafana:latest
+Network=dankclaw_default
+User=0:0
+PublishPort=127.0.0.1:3002:3000
+Environment=GF_SERVER_ROOT_URL=https://grafana.franksdank.cloud
+Environment=GF_AUTH_GENERIC_OAUTH_ENABLED=true
+Environment=GF_AUTH_GENERIC_OAUTH_NAME=Authentik
+Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_ID=grafana-21ca190cde75d60f
+Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
+# Global authorize path: this Authentik build resolves the provider by the
+# client_id query parameter — /application/o/<slug>/authorize/ does not exist.
+Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
+Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
+# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
+Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
+Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
+Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
+Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
+Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
+Volume=/opt/grafana/data:/var/lib/grafana
+Volume=/opt/grafana/provisioning:/etc/grafana/provisioning:ro
+HealthCmd=wget --spider -q http://localhost:3000/api/health
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=60s
+
+[Service]
+# systemd-side only: supplies ${GRAFANA_OAUTH_CLIENT_SECRET} for the expansion
+# above. The container never receives this file.
+#
+# It is the per-service env file the deployer seeds from dankclaw/.env — the
+# fleet convention (caddy, pihole, copyparty, romm and seerr all read one).
+# The repo-root .env that the compose service happened to pick up is a gluetun
+# env file: it carries HOME=/root and the WireGuard keys, and importing it into
+# this service makes podman look for its configuration under /root, which it
+# cannot read, so the unit fails to start. Seed the value into dankclaw/.env.
+EnvironmentFile=/home/dank/Containers/grafana/env
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/loki.container

@@ -0,0 +1,57 @@
+# Quadlet unit: loki (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/loki.container
+# Unit name: loki.service (container name: loki)
+#
+# Log store. This replaces the compose service of the same name in
+# dankclaw/compose.yaml ("Observability" block).
+#
+# IMAGE. franksdank/loki is built locally by deploy-remote.sh
+# (build_compose_image) from dankclaw/loki/Containerfile: upstream
+# grafana/loki plus a static busybox, because loki is distroless and the
+# healthcheck has nothing to run inside it. Pull=never keeps a boot from
+# trying to fetch that local tag from a registry. The localhost/ prefix is
+# required — this host's quadlet path cannot resolve a registry-prefixed ref
+# from local storage at create time.
+#
+# PUBLISHING. Bound to 127.0.0.1: Alloy pushes to loki:3100 over
+# dankclaw_default, Caddy proxies the guarded vhost to loki:3100 the same way,
+# and the homepage status timer probes 127.0.0.1:3100. Loki has
+# auth_enabled: false, so the compose service's 0.0.0.0 binding was an
+# unauthenticated log-ingest and read API on every host interface.
+#
+# CONFIG. /opt/loki/config/loki-config.yaml is installed from
+# shared/loki/loki-config.yaml on every deploy (single-binary, filesystem
+# storage, 30-day retention). Chunks and rules live in /opt/loki/data.
+#
+# Rootless: container root maps to the invoking user (dank), who owns
+# /opt/loki/data; the image's default user (10001) maps to a subuid that
+# cannot write it.
+
+[Unit]
+Description=Loki log store (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=loki
+Image=localhost/franksdank/loki:latest
+Pull=never
+Network=dankclaw_default
+User=0:0
+Exec=-config.file=/etc/loki/loki-config.yaml
+PublishPort=127.0.0.1:3100:3100
+Volume=/opt/loki/config:/etc/loki
+Volume=/opt/loki/data:/loki
+# No HealthCmd here on purpose. A unit's HealthCmd reaches podman as
+# CMD-SHELL, which means `/bin/sh -c <command>` — and this image is
+# distroless, with no shell at all. The healthcheck therefore ships in the
+# image (dankclaw/loki/Containerfile) in exec form, where it also carries its
+# interval, timeout, start period and retry count.
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/node-exporter.container

@@ -0,0 +1,54 @@
+# Quadlet unit: node-exporter (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/node-exporter.container
+# Unit name: node-exporter.service (container name: node-exporter)
+#
+# Host metrics source for Prometheus. This replaces the compose service of the
+# same name in dankclaw/compose.yaml ("Observability" block).
+#
+# HOST NAMESPACE. It must run in the host network namespace to report the
+# host's own interfaces/connections, and --pid=host for the processes and
+# cgroups collectors. "localhost" inside the container is therefore the host's
+# loopback, which is what the homepage status timer probes.
+#
+# LISTEN ADDRESSES. Until 2026-09-11 this bound :9100 on every interface. It
+# now binds exactly two: loopback for the homepage status probe, and the LAN
+# address because Prometheus — which runs in a container on dankclaw_default —
+# scrapes http://192.168.0.143:9100 (shared/prometheus/prometheus.yml,
+# job "node-dankclaw"). node_exporter itself has no authentication, so keeping
+# the wildcard bind off the table matters: this is the second half of the
+# pairing that removes the unauthenticated observability surface, and both
+# consumers are accounted for. Dankserver runs its own copy of this container
+# (scraped by the "node-dankserver" job) — since 2026-09-16 as a Quadlet unit in
+# the same shape (dankserver/observability/quadlet/), because the compose-only
+# version it replaced had been dead for a day and a half before anyone noticed.
+#
+# MOUNTS. The container reads the host's /proc, /sys and / as /host/* read-only
+# and is pointed at them with --path.*; it writes nothing anywhere.
+
+[Unit]
+Description=Prometheus node exporter (Dankclaw host metrics)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=node-exporter
+Image=docker.io/prom/node-exporter:latest
+Network=host
+PodmanArgs=--pid=host
+Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.143:9100
+Volume=/proc:/host/proc:ro
+Volume=/sys:/host/sys:ro
+Volume=/:/rootfs:ro
+HealthCmd=wget --spider -q http://localhost:9100/metrics
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/observability/quadlet/prometheus.container

@@ -0,0 +1,63 @@
+# Quadlet unit: prometheus (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/prometheus.container
+# Unit name: prometheus.service (container name: prometheus)
+#
+# The homelab metrics store. This replaces the compose service of the same
+# name in dankclaw/compose.yaml ("Observability" block): these five containers
+# were started by hand with podman-compose and never came back after a reboot,
+# which the 2026-09-11 reboot for the amd_iommu=off change made obvious.
+#
+# PUBLISHING. The published port is bound to 127.0.0.1 deliberately. Prometheus
+# is reached by Caddy over the dankclaw_default network (the Caddyfile's
+# post-auth handler proxies to prometheus:9090) and by the host-side homepage
+# status timer over 127.0.0.1. The compose service published on 0.0.0.0, which
+# put an UNAUTHENTICATED metrics API — with the lifecycle endpoint enabled —
+# on every host interface: a bypass around the Anubis + Authentik guard on
+# prometheus.franksdank.cloud. No off-host consumer needs it.
+#
+# CONFIG. /opt/prometheus/config/prometheus.yml is installed from
+# shared/prometheus/prometheus.yml by deploy-remote.sh on every deploy, so the
+# repo copy is the source of truth. STATE (30-day retention) lives in
+# /opt/prometheus/data.
+#
+# Rootless: container root maps to the invoking user (dank) in this user
+# namespace and dank owns /opt/prometheus/data. The image's default user
+# (nobody) maps to a subuid that cannot write it.
+#
+# RULES. Five LLM alert rules and one recording rule load from
+# /etc/prometheus/rules/ (installed from shared/prometheus/rules/ by the
+# deployer). There is still **no Alertmanager**, so nothing is delivered: a
+# firing alert is visible in Prometheus and on the Grafana dashboard's ALERTS
+# panel and reaches nobody. Delivery is an operator decision, not an oversight;
+# the fragility is that an undelivered alert looks identical to a quiet one.
+#
+# The config volume is also why rules belong in git: editing them in the UI is
+# lost on the next deploy, since the deployer overwrites the whole directory.
+
+[Unit]
+Description=Prometheus metrics store (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=prometheus
+Image=docker.io/prom/prometheus:latest
+Network=dankclaw_default
+User=0:0
+Exec=--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=30d --web.enable-lifecycle
+PublishPort=127.0.0.1:9090:9090
+Volume=/opt/prometheus/config:/etc/prometheus
+Volume=/opt/prometheus/data:/prometheus
+HealthCmd=wget --spider -q http://localhost:9090/-/healthy
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankclaw/open-webui/quadlet/open-webui.container

@@ -8,7 +8,7 @@
 # its OpenAI connection at the live llama.cpp backend on this host via the
 # host LAN IP (192.168.0.143:7770, model frank-default-large). It publishes 3005
 # (3000 is Gitea's HTTP port). Data persists under
-# /home/dank/Containers/open-webui/data. See docs/Dankclaw/OpenWebUI.md.
+# /home/dank/Containers/open-webui/data. See serverAppConfigs-docs/Dankclaw/OpenWebUI.md.
 #
 # Network history (2026-08-22): this unit used Network=slirp4netns. When the
 # slirp4netns helper process died, the container kept running but lost its tap

dankclaw/pihole/addn-hosts

@@ -1,6 +1,6 @@
 # A records for the flat franksdank.name scheme (dnsmasq addn-hosts format:
 # <IP> <name> [<name> ...]). Served by the Pi-hole quadlet (FTL is a
-# dnsmasq fork): the quadlet mounts this file into /etc/dnsmasq.d/ read-only.
+# dnsmasq fork): the quadlet mounts this file into /etc/pihole/hosts/ read-only.
 #
 # Caddy is consolidated on dankclaw (192.168.0.143): it fronts every service
 # on both hosts, so every name resolves to that one IP. Keep in sync with the
@@ -25,9 +25,8 @@
 192.168.0.143 books.franksdank.cloud
 192.168.0.143 romm.franksdank.cloud
 192.168.0.143 files.franksdank.cloud
-192.168.0.143 jellyfin.franksdank.cloud
+192.168.0.143 jelly.franksdank.cloud
 192.168.0.143 radarr.franksdank.cloud
-192.168.0.143 sonarr.franksdank.cloud
 192.168.0.143 lidarr.franksdank.cloud
 192.168.0.143 readarr.franksdank.cloud
 192.168.0.143 prowlarr.franksdank.cloud
@@ -36,10 +35,13 @@
 # --- AI / search / dev services (dankclaw, local) ---
 192.168.0.143 search.franksdank.cloud
 192.168.0.143 comfyui.franksdank.cloud
-192.168.0.143 chat.franksdank.cloud
+192.168.0.143 ai.franksdank.cloud
 192.168.0.143 hermes.franksdank.cloud
+192.168.0.143 hermesadmin.franksdank.cloud
 192.168.0.143 gitea.franksdank.cloud
 192.168.0.143 grafana.franksdank.cloud
 192.168.0.143 prometheus.franksdank.cloud
 192.168.0.143 loki.franksdank.cloud
 192.168.0.143 pihole.franksdank.cloud
+192.168.0.143 seerr.franksdank.cloud
+192.168.0.143 marreta.franksdank.cloud

dankclaw/pihole/quadlet/pihole.container

@@ -6,7 +6,7 @@
 # The LAN's authoritative DNS for the flat franksdank.cloud / franksdank.casa
 # name scheme. It retired the dnsmasq quadlet (2026-08-24): Pi-hole FTL is a
 # dnsmasq fork, so the addn-hosts record file carries over unchanged and is
-# mounted into /etc/dnsmasq.d/ (the FTL dnsmasq config dir). It serves one A
+# mounted into /etc/pihole/hosts/ (the FTL hostsdir) read-only. It serves one A
 # record per Caddy vhost and forwards everything else to the pinned upstream
 # resolvers (DNS1/DNS2 in the env file). The SURFboard G36 router hands this
 # server out to DHCP clients (DNS statically configured -> 192.168.0.143),
@@ -15,7 +15,7 @@
 # Runs on slirp4netns with port 53 (DNS) and 8083 (web UI) published to the
 # LAN (the same pattern as open-webui). Config + state mount from
 # /home/dank/Containers/pihole. The web UI password is WEBPASSWORD in the env
-# file. See docs/Infrastructure/LAN-DNS.md.
+# file. See serverAppConfigs-docs/Infrastructure/LAN-DNS.md.
 
 [Unit]
 Description=Pi-hole LAN DNS container

dankclaw/pihole/sysctl/99-dnsmasq.conf

@@ -0,0 +1,8 @@
+# Rootless DNS must bind port 53: lower the unprivileged-port threshold so a
+# rootless container may publish 0.0.0.0:53 (80/443 stay allowed; 53 < 80).
+# Single source of truth for net.ipv4.ip_unprivileged_port_start on dankclaw;
+# deploy_pihole installs it to /etc/sysctl.d/ and applies it with
+# `sysctl --system` (the same code path boot uses via systemd-sysctl).
+# A later-sorted drop-in that reset this to 80 took LAN DNS down after the
+# 2026-08-26 reboot — deploy_pihole neutralizes such conflicts.
+net.ipv4.ip_unprivileged_port_start=53

dankclaw/searxng/quadlet/gluetun.container

@@ -6,7 +6,7 @@
 # Creates the WireGuard tunnel to ProtonVPN (US-WA). SearXNG reuses this
 # container's network namespace (see searxng.container), so its outbound search
 # traffic exits here. The WireGuard credentials live in /opt/searxng/.env
-# (mode 0600); this file holds no secrets. See docs/Dankclaw/Gluetun.md.
+# (mode 0600); this file holds no secrets. See serverAppConfigs-docs/Dankclaw/Gluetun.md.
 
 [Unit]
 Description=Gluetun VPN tunnel container
@@ -29,6 +29,7 @@ Environment=WIREGUARD_ENDPOINT_PORT=${WIREGUARD_ENDPOINT_PORT}
 Environment=WIREGUARD_ALLOWED_IPS=${WIREGUARD_ALLOWED_IPS}
 Environment=WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=${WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL}
 PublishPort=8081:8080
+PublishPort=8082:8085
 
 [Service]
 EnvironmentFile=/opt/searxng/.env

dankclaw/searxng/quadlet/searxng-valkey.container

@@ -5,7 +5,7 @@
 #
 # The limiter's rate-limit state store. Joins the searxng network and is
 # reachable by the name "valkey" (settings.yml points there). Data persists in
-# /opt/searxng/valkey. See docs/Dankclaw/Valkey.md.
+# /opt/searxng/valkey. See serverAppConfigs-docs/Dankclaw/Valkey.md.
 
 [Unit]
 Description=SearXNG Valkey cache container

dankclaw/searxng/quadlet/searxng.container

@@ -6,7 +6,7 @@
 # Reuses gluetun's network namespace (Network=gluetun.container) so its
 # outbound traffic exits the WireGuard tunnel. It publishes no port of its own;
 # the port lives on gluetun. It reaches valkey by name on the searxng network.
-# Config mounts from /opt/searxng/config. See docs/Dankclaw/SearXNG.md.
+# Config mounts from /opt/searxng/config. See serverAppConfigs-docs/Dankclaw/SearXNG.md.
 
 [Unit]
 Description=SearXNG search engine container

dankclaw/seerr/quadlet/seerr.container

@@ -0,0 +1,43 @@
+# Quadlet unit: Seerr media request manager (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/seerr.container
+# Unit name: seerr.service (container name: seerr)
+#
+# Seerr is the unified Overseerr/Jellyseerr successor. It fronts the
+# *arr stack now living on FrankBook-Pro (192.168.0.112, resolved via the
+# Pi-hole addn-hosts record frankbook-pro.lan): requests are placed here,
+# routed through the arrs over the LAN, and downloads run inside the Mac's
+# gluetun tunnel. SQLite lives under /home/dank/Containers/seerr/data —
+# no separate database unit (decision D8).
+#
+# Exposure contract: the ONLY published port is loopback (127.0.0.1:5055)
+# for the homepage probe and host-local checks. User traffic must traverse
+# the guard chain: Caddy TLS edge → Anubis :8923 → Authentik outpost →
+# Caddy :8017 post-auth listener → seerr:5055 on dankclaw_default. Never
+# publish 5055 on a LAN or public address.
+#
+# Image pin: ghcr.io/seerr-team/seerr:v3.4.1 (manifest-verified on GHCR
+# 2026-09-01; released 2026-07-30). Upgrade = bump the tag, redeploy.
+#
+# No secrets in this file: OIDC (Authentik) and arr API keys live in
+# Seerr's own settings store inside the data volume, configured at runtime.
+
+[Unit]
+Description=Seerr media request manager container
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=seerr
+Image=ghcr.io/seerr-team/seerr:v3.4.1
+Network=dankclaw_default
+PublishPort=127.0.0.1:5055:5055
+Volume=/home/dank/Containers/seerr/data:/app/config
+
+[Service]
+Restart=always
+RestartSec=5
+TimeoutStartSec=900
+
+[Install]
+WantedBy=default.target

dankclaw/udev/99-comfyui-gpu-access.rules

@@ -0,0 +1,21 @@
+# /etc/udev/rules.d/99-comfyui-gpu-access.rules
+#
+# Why world-writable compute nodes: the comfyui container is spawned by a
+# systemd user unit (comfyui.service, Type=oneshot). User-unit processes are
+# spawned by the user manager with NO supplementary groups, so any spawn path
+# through systemd loses `render`/`video` membership regardless of how the
+# container is launched (keep-groups and named group_add both depend on the
+# spawning process already holding the group — verified 2026-08-31: keep-groups
+# preserved an empty group list under the user manager and /dev/kfd open still
+# failed with EACCES). A mode that grants access to "everyone in this uid
+# domain" is the only spawn-context-independent option short of running the
+# stack rootful.
+#
+# Blast radius: any local process (every container here runs as the same uid
+# domain) can already ptrace the GPU userspace through an existing holder's
+# fd inheritance; on a private single-administrator home server this trades a
+# capability gate that could never be held by the intended user anyway.
+# Note the missing SUBSYSTEM= guard on the kfd line: misc-class devices have no
+# subsystem for udev matching, so a SUBSYSTEM=="misc" guard never matches.
+KERNEL=="kfd", MODE="0666"
+KERNEL=="renderD*", SUBSYSTEM=="drm", MODE="0666"

dankserver/.AGENTTRASH/20260916-vllm-radiance/affinity-proxy.py

@@ -0,0 +1,156 @@
+#!/usr/bin/env python3
+"""HTTP front door for the frank-medium single-GPU replicas.
+
+Why two replicas instead of TP=2: one vLLM process spanning both R9700s needs
+RCCL collectives, and RCCL's kernels require PCIe atomics on every
+participating GPU. The second card sits behind chipset bridges advertising
+`AtomicOpsCap: Routing-`, which no software can enable (see the runbook), so
+no collective can launch on it. Two single-GPU replicas use both cards; this
+service keeps the fleet contract of one endpoint on :7770, streaming included.
+
+Routing decisions live in routing.py — spread single-turn requests, pin
+continuations to the replica whose prefix cache holds their context. This file
+only moves bytes: parse enough of the request to route it, forward it verbatim,
+stream the answer back, and report replica health on /health.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+from contextlib import suppress
+from aiohttp import ClientSession, ClientTimeout, web
+
+import routing
+
+PORT = int(os.environ.get("PORT", "7770"))
+HEALTH_TTL = float(os.environ.get("HEALTH_TTL", "10"))
+# Hop-by-hop headers must not be forwarded (RFC 9110 §7.6.1) — forwarding
+# transfer-encoding or content-length of a rewritten body breaks clients.
+HOP_BY_HOP = {
+    "host", "content-length", "transfer-encoding", "connection",
+    "keep-alive", "proxy-authenticate", "proxy-authorization", "te",
+    "trailers", "upgrade",
+}
+
+UPSTREAMS = routing.UPSTREAMS
+# Replica health, refreshed by the watcher. Module state rather than aiohttp app
+# state: the application is frozen once the site is up.
+_UP: list[bool] = [True] * len(UPSTREAMS)
+
+
+async def health(session: ClientSession) -> web.Response:
+    results = await asyncio.gather(
+        *(session.get(u + "/health", timeout=ClientTimeout(total=5)) for u in UPSTREAMS),
+        return_exceptions=True,
+    )
+    lines, up = [], 0
+    for url, res in zip(UPSTREAMS, results):
+        if isinstance(res, Exception) or res.status != 200:
+            lines.append(f"{url} DOWN")
+        else:
+            up += 1
+            lines.append(f"{url} UP")
+    return web.Response(status=200 if up else 503, text="\n".join(lines) + "\n")
+
+
+async def watch_health() -> None:
+    """Refresh the replica up/down table so routing never targets a dead one.
+
+    One HTTP request per replica per interval, in exchange for never handing a
+    client a 502 after a restart.
+    """
+    global _UP
+    session = ClientSession()
+    try:
+        while True:
+            res = await asyncio.gather(
+                *(session.get(u + "/health", timeout=ClientTimeout(total=5)) for u in UPSTREAMS),
+                return_exceptions=True,
+            )
+            _UP = [not isinstance(r, Exception) and r.status == 200 for r in res]
+            await asyncio.sleep(HEALTH_TTL)
+    finally:
+        await session.close()
+
+
+async def proxy(request: web.Request) -> web.StreamResponse:
+    session: ClientSession = request.app["session"]
+    if request.path == "/health":
+        return await health(session)
+
+    body = await request.read()
+    headers = {k: v for k, v in request.headers.items() if k.lower() not in HOP_BY_HOP}
+
+    # Fail over before anything is written to the client. A replica that
+    # restarts is invisible to the health watcher for up to HEALTH_TTL, and a
+    # deploy restarts both replicas — without this retry a 10 s window of
+    # requests would 502 on a service that has a live card the whole time.
+    # vLLM's OpenAI endpoints are stateless per request, so re-sending a
+    # request that never reached an engine is safe; a request that failed
+    # mid-stream is not retried (that path is below).
+    tried: set[int] = set()
+    upstream = None
+    last_error = "no healthy frank-medium replica"
+    while len(tried) < len(UPSTREAMS):
+        idx = routing.route(body, request.remote, _UP)
+        if idx < 0 or idx in tried:
+            break
+        tried.add(idx)
+        target = UPSTREAMS[idx] + request.path_qs
+        try:
+            upstream = await session.request(
+                request.method, target, headers=headers, data=body or None,
+                timeout=ClientTimeout(total=None, sock_connect=10, sock_read=3600),
+            )
+            break
+        except Exception as exc:
+            # Demote at once: the watcher will confirm, but every request until
+            # then should skip this replica rather than pay the same timeout.
+            # Logged because a silent failover hides a card that is down.
+            _UP[idx] = False
+            last_error = f"{target} unreachable: {exc}"
+            print(f"proxy: failover — {target} unreachable ({exc}); retrying "
+                  f"another replica", flush=True)
+    if upstream is None:
+        # 503 when no replica was even reachable through the routing table,
+        # 502 when replicas were tried and all of them failed.
+        status = 502 if tried else 503
+        return web.Response(status=status, text=f"proxy: {last_error}\n")
+
+    out = web.StreamResponse(status=upstream.status, reason=upstream.reason)
+    for k, v in upstream.headers.items():
+        if k.lower() not in HOP_BY_HOP:
+            out.headers[k] = v
+    await out.prepare(request)
+    try:
+        async for chunk in upstream.content.iter_any():
+            await out.write(chunk)
+    except Exception:
+        # The client went away mid-stream (an aborted generation). Nothing to
+        # report: the engine sees its own cancellation.
+        pass
+    finally:
+        upstream.release()
+    with suppress(Exception):
+        await out.write_eof()
+    return out
+
+
+async def main() -> None:
+    app = web.Application(client_max_size=1024 ** 3)
+    app["session"] = ClientSession()
+    app.router.add_route("*", "/{tail:.*}", proxy)
+    runner = web.AppRunner(app)
+    await runner.setup()
+    await web.TCPSite(runner, "0.0.0.0", PORT).start()
+    print(f"proxy: :{PORT} -> {', '.join(UPSTREAMS)} routing={routing.ROUTING} "
+          f"key={routing.AFFINITY_BYTES}B known_max={routing.KNOWN_MAX}", flush=True)
+    watcher = asyncio.create_task(watch_health())
+    with suppress(asyncio.CancelledError):
+        await asyncio.gather(watcher)
+
+
+if __name__ == "__main__":
+    with suppress(KeyboardInterrupt):
+        asyncio.run(main())

dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-a.container

@@ -0,0 +1,98 @@
+# Quadlet unit: frank-medium replica A (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/frank-medium-a.container
+# Unit name: frank-medium-a.service (container name: frank-medium-a)
+#
+# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
+# tensor-parallel process across both R9700s (see frank-medium.container, kept
+# as the target). That shape cannot run on this host: RCCL's collective kernels
+# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
+# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
+# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
+# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
+# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
+# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
+# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
+# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
+# atomics. Result: every collective that touches GPU 2 dies at launch.
+# Two independent single-GPU replicas use both cards instead; the proxy unit
+# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
+#
+# SHAPE. vLLM binds 0.0.0.0:7771 on the host network directly; no PublishPort.
+# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
+# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
+# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
+# this shape instead of using the 2x7551-32624 pin, which is only valid at
+# TP=2/262144.
+#
+# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=0 (the first R9700). The
+# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
+# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
+# first and HIP indexes into the filtered list, so a single-valued pair
+# (ROCR=1,HIP=1) leaves an empty device set.
+#
+# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
+# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
+[Unit]
+Description=frank-medium replica A — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #1
+After=network-online.target
+
+[Container]
+ContainerName=frank-medium-a
+Image=localhost/radiance-vllm:0.9.3
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+GroupAdd=keep-groups
+# The upstream launcher's exact podman posture (rootless).
+PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
+Environment=HIP_VISIBLE_DEVICES=0
+Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
+Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
+Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
+Environment=SERVED_NAMES=frank-medium
+Environment=PORT=7771
+Environment=TP=1
+Environment=MAXSEQS=4
+Environment=CHUNK=8192
+Environment=MAXLEN=114688
+Environment=GPU_UTIL=0.98
+Environment=KV_MEM=0
+Environment=SPEC_METHOD=dflash
+Environment=SPEC=7
+# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
+Volume=/home/dank/models:/models
+# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
+Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
+# Compile cache. Shared with the other replica and with the TP=2 target: the
+# entries are content-addressed, so shapes simply add subdirectories.
+Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
+# The pinned radiance clone: patches applied at every container start.
+Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
+# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
+# shipped r4d.so predates the GDN overflow fix and NaNs this model.
+Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
+# Entrypoint + healthcheck from this checkout, mounted read-only.
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
+# The entrypoint must be the container command; the image's own entrypoint
+# (which the patched script execs at the end) must not run unpatched.
+PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
+# 200 only once the engine serves; during first-start compilation the
+# container is legitimately unhealthy. HealthCmd is split on spaces into an
+# argv array by the podman 5.x v1 generator, hence the file-based probe.
+HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
+HealthInterval=60s
+HealthTimeout=30s
+HealthRetries=3
+HealthStartPeriod=1800s
+Pull=never
+
+[Service]
+# First start compiles Triton/inductor kernels (several minutes) before the
+# engine loads ~20 GiB of weights.
+TimeoutStartSec=1800
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-b.container

@@ -0,0 +1,98 @@
+# Quadlet unit: frank-medium replica B (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/frank-medium-b.container
+# Unit name: frank-medium-b.service (container name: frank-medium-a)
+#
+# WHY TWO REPLICAS INSTEAD OF TP=2. The intended shape for frank-medium is one
+# tensor-parallel process across both R9700s (see frank-medium.container, kept
+# as the target). That shape cannot run on this host: RCCL's collective kernels
+# carry a hidden hostcall-buffer parameter, and the ROCr runtime refuses to
+# dispatch them on a device without PCIe atomics ("Pcie atomics not enabled,
+# hostcall not supported" -> AQL dispatch failed -> hipErrorIllegalState). The
+# second R9700 sits behind the chipset bridges 05:00.0/06:09.0, which advertise
+# `AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
+# atomics on that path; the amdgpu driver's pci_enable_atomic_ops_to_root()
+# therefore fails for that GPU and KFD marks the CPU<->GPU2 link NO_ATOMICS.
+# The first R9700 sits on CPU root port 00:01.1 (Routing+) and does have
+# atomics. Result: every collective that touches GPU 2 dies at launch.
+# Two independent single-GPU replicas use both cards instead; the proxy unit
+# keeps the single :7770 endpoint. Full evidence: the frank-medium runbook.
+#
+# SHAPE. vLLM binds 0.0.0.0:7772 on the host network directly; no PublishPort.
+# MAXLEN is 114688, not the TP=2 deployment's 262144: one card carries the
+# whole model (~20 GiB of weights + drafter), leaving ~5.2 GiB of KV, and
+# 131072 does not fit (measured: needs 5.58 GiB). KV_MEM=0 lets vLLM profile
+# this shape instead of using the 2x7551-32624 pin, which is only valid at
+# TP=2/262144.
+#
+# DEVICES. /dev/kfd + /dev/dri; HIP_VISIBLE_DEVICES=1 (the second R9700). The
+# 2 GiB Raphael iGPU enumerates as index 2 and is excluded by the pin.
+# Only HIP_VISIBLE_DEVICES is set, never ROCR_VISIBLE_DEVICES: ROCR filters
+# first and HIP indexes into the filtered list, so a single-valued pair
+# (ROCR=1,HIP=1) leaves an empty device set.
+#
+# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
+# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
+[Unit]
+Description=frank-medium replica B — vllm-radiance MXFP4 Qwen3.8-27B on R9700 #2
+After=network-online.target
+
+[Container]
+ContainerName=frank-medium-b
+Image=localhost/radiance-vllm:0.9.3
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+GroupAdd=keep-groups
+# The upstream launcher's exact podman posture (rootless).
+PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
+Environment=HIP_VISIBLE_DEVICES=1
+Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
+Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
+Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
+Environment=SERVED_NAMES=frank-medium
+Environment=PORT=7772
+Environment=TP=1
+Environment=MAXSEQS=4
+Environment=CHUNK=8192
+Environment=MAXLEN=114688
+Environment=GPU_UTIL=0.98
+Environment=KV_MEM=0
+Environment=SPEC_METHOD=dflash
+Environment=SPEC=7
+# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards).
+Volume=/home/dank/models:/models
+# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the entrypoint.
+Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
+# Compile cache. Shared with the other replica and with the TP=2 target: the
+# entries are content-addressed, so shapes simply add subdirectories.
+Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
+# The pinned radiance clone: patches applied at every container start.
+Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
+# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The image's
+# shipped r4d.so predates the GDN overflow fix and NaNs this model.
+Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
+# Entrypoint + healthcheck from this checkout, mounted read-only.
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
+# The entrypoint must be the container command; the image's own entrypoint
+# (which the patched script execs at the end) must not run unpatched.
+PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
+# 200 only once the engine serves; during first-start compilation the
+# container is legitimately unhealthy. HealthCmd is split on spaces into an
+# argv array by the podman 5.x v1 generator, hence the file-based probe.
+HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
+HealthInterval=60s
+HealthTimeout=30s
+HealthRetries=3
+HealthStartPeriod=1800s
+Pull=never
+
+[Service]
+# First start compiles Triton/inductor kernels (several minutes) before the
+# engine loads ~20 GiB of weights.
+TimeoutStartSec=1800
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-healthcheck.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+"""frank-medium container healthcheck: the port's /health must answer 200.
+
+Runs INSIDE the container. The vLLM /health endpoint only answers once the
+engine is up (weights loaded, cudagraphs captured), so a healthy container is
+a serving container, not merely a live process. PORT selects the endpoint so
+the single-GPU replicas (:7771/:7772) and the round-robin proxy (:7770) can
+share this probe. The HealthCmd in the quadlet splits on spaces into an argv
+array (podman 5.x v1 generator), which is why this lives in a file rather
+than an inline python one-liner.
+"""
+import os
+import sys
+import urllib.request
+
+port = os.environ.get("PORT", "7770")
+try:
+    with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=5) as r:
+        sys.exit(0 if r.status == 200 else 1)
+except Exception:
+    sys.exit(1)

dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium-proxy.container

@@ -0,0 +1,61 @@
+# Quadlet unit: frank-medium proxy (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/frank-medium-proxy.container
+# Unit name: frank-medium-proxy.service (container name: frank-medium-proxy)
+#
+# WHAT IT IS. A prefix-affinity reverse proxy in front of the two single-GPU
+# frank-medium replicas (:7771 on R9700 #1, :7772 on R9700 #2). It preserves
+# the fleet contract: one endpoint on host port 7770 answering as the served
+# model name frank-medium, OpenAI /v1, streaming included.
+#
+# WHY IT EXISTS. TP=2 — one vLLM process spanning both cards — is impossible
+# on this host (see frank-medium-a.container and the runbook: RCCL's kernels
+# need PCIe atomics, and the second card's chipset path advertises
+# AtomicOpsCap: Routing-). Two replicas are how both cards get used.
+#
+# ROUTING. Requests are keyed on a hash of their leading 4 KiB of prompt text,
+# so a conversation family — an agent loop re-sending the same system prompt,
+# tool list and file context every turn — keeps landing on the same replica,
+# where --enable-prefix-caching has that prefix warm. Plain round-robin threw
+# half of those cache hits away. A replica that is down is skipped, and
+# ROUTING=roundrobin restores the old behaviour for A/B measurement. /health
+# answers 200 while at least one replica is up and names the ones that are
+# down. The proxy runs on host networking so it can bind 7770 and reach the
+# replicas on loopback.
+#
+# The script is a few dozen lines of aiohttp, mounted read-only out of
+# /home/dank/serverAppConfigs (deploy.sh pulls that checkout before dispatch).
+# It reuses the same image as the replicas: aiohttp is already there and no
+# extra image has to be pulled on this host.
+[Unit]
+Description=frank-medium proxy — prefix-affinity in front of the two single-GPU replicas
+After=network-online.target
+Wants=frank-medium-a.service frank-medium-b.service
+After=frank-medium-a.service frank-medium-b.service
+
+[Container]
+ContainerName=frank-medium-proxy
+Image=localhost/radiance-vllm:0.9.3
+PodmanArgs=--network=host
+Environment=PORT=7770
+Environment=UPSTREAMS=http://127.0.0.1:7771,http://127.0.0.1:7772
+# The proxy is two files: routing.py holds the replica-selection policy, the
+# server holds the HTTP plumbing. Both are mounted because the server imports
+# the policy by name from its own directory (/usr/local/bin is sys.path[0]).
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/routing.py:/usr/local/bin/routing.py:ro
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/proxy/affinity-proxy.py:/usr/local/bin/affinity-proxy.py:ro
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
+PodmanArgs=--entrypoint /usr/local/bin/affinity-proxy.py
+HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
+HealthInterval=60s
+HealthTimeout=30s
+HealthRetries=3
+HealthStartPeriod=60s
+Pull=never
+
+[Service]
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/.AGENTTRASH/20260916-vllm-radiance/frank-medium.container

@@ -0,0 +1,147 @@
+# Quadlet unit: frank-medium (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/frank-medium.container
+# Unit name: frank-medium.service (container name: frank-medium)
+#
+# The dankserver LLM endpoint (:7770, alias frank-medium), since 2026-09-15
+# running the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3
+# image + the ggz14/radiance-vllm-mxfp4 patches) serving
+# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on the 2x Radeon AI PRO
+# R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX llama.cpp
+# deployment, which served the same alias from the Q8_0_ROCMFPX_AGENT GGUF;
+# the MXFP4 stack is tensor-parallel (TP=2, R4D one-shot all-reduce) with the
+# DFlash2-FP8 block-diffusion drafter (SPEC=7) — the measured production
+# configuration of the upstream repo, on our exact hardware signature
+# (2x7551-32624, see the KV pin below).
+#
+# SHAPE. The launcher the upstream repo ships binds the server with
+# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
+# what this unit does too — no PublishPort. The fleet contract is unchanged:
+# :7770, served model name frank-medium (SERVED_NAMES), OpenAI /v1.
+#
+# NOT INSTALLED ON THIS HOST. This is the target shape: one tensor-parallel
+# process across both R9700s. It cannot launch a collective here — RCCL's
+# kernels need PCIe atomics and the second card sits behind chipset bridges
+# advertising `AtomicOpsCap: Routing-` — so the active deployment is the
+# two-replica trio in this directory (frank-medium-a/b/proxy) and
+# deploy-remote.sh deliberately does not install this unit. Keep it for a
+# host whose second card is on an atomic-capable root port; the full evidence
+# is in serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md.
+#
+# DEVICES. /dev/kfd + /dev/dri through the amdgpu/KFD stack; HIP/ROCR device
+# indices 0,1 (the two R9700s). The 2 GiB Raphael iGPU enumerates as index 2
+# and is excluded both by the index pin and by the upstream MIN_GPU_MIB=8192
+# logic this unit replaces. --privileged --ipc=host --security-opt
+# seccomp=unconfined --cap-add SYS_PTRACE is the upstream launcher's exact
+# podman posture.
+#
+# MOUNTS. /models (rw checkpoints), the HF cache (tokenizer files),
+# /cache (Triton/inductor/aiter/vLLM compile cache — first start compiles for
+# several minutes, later starts reuse), /patches (the pinned radiance clone —
+# its patches are applied at EVERY container start by the entrypoint), /r4d
+# (the patched libr4d build; the image's shipped r4d.so NaNs the
+# gated-delta-net and must never be used), and the entrypoint + healthcheck
+# scripts from this checkout (mounted read-only straight out of
+# /home/dank/serverAppConfigs — deploy.sh git-pulls the checkout before
+# dispatching).
+#
+# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
+# path cannot resolve registry-prefixed refs from local storage at create
+# time (same as frank-large on dankclaw). deploy-remote.sh pulls the
+# docker.io ref and re-tags it before the restart. Pull=never keeps boots off
+# the registry.
+#
+# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated,
+# re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new
+# serve-mxfp4.sh container command and bump the pin deliberately.
+#
+# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
+# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
+[Unit]
+Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
+After=network-online.target
+
+[Container]
+ContainerName=frank-medium
+Image=localhost/radiance-vllm:0.9.3
+# vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort.
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+GroupAdd=keep-groups
+# The upstream launcher's exact podman posture (rootless; podman-only flags
+# like --replace are not needed under systemd management).
+PodmanArgs=--privileged --ipc=host --security-opt seccomp=unconfined --cap-add SYS_PTRACE --network=host
+# The two R9700s. The Raphael iGPU (index 2, 2 GiB) stays out: shared system
+# memory is not a compute target.
+Environment=ROCR_VISIBLE_DEVICES=0,1
+Environment=HIP_VISIBLE_DEVICES=0,1
+# The uncensored MXFP4 checkpoint ships its fp8 MTP head already, so no
+# fp8_mtp.py rewrite step exists or is needed (unlike amd/Qwen3.8-27B-Quark-AWQ-MXFP4).
+Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
+Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
+Environment=CHAT_TEMPLATE=/patches/qwen-fixed-v22.3.jinja
+# Fleet identity: the id at /v1/models and in every response body. Every
+# client (Hermes, the harness, services.json) points at frank-medium.
+Environment=SERVED_NAMES=frank-medium
+Environment=PORT=7770
+# Fixed 2-card tensor parallel: the tested configuration of the upstream
+# repo; the R4D one-shot all-reduce kernel serves it natively.
+Environment=TP=2
+# Measured production batch shape (upstream README): MAXSEQS 8 peaks
+# aggregate throughput at 573 t/s; MAXLEN is the model's native context.
+Environment=MAXSEQS=8
+Environment=CHUNK=8192
+Environment=MAXLEN=262144
+Environment=GPU_UTIL=0.98
+# Speculative decoding: the DFlash2 drafter at depth 7 is the configuration
+# every published number (224 t/s single-stream decode, 4749-4810 t/s
+# prefill) was measured with.
+Environment=SPEC_METHOD=dflash
+Environment=SPEC=7
+# KV cache pin, bytes. From the upstream repo's shipped kv-profiles.tsv row
+#   sig=2x7551-32624 maxseqs=8 chunk=8192 maxlen=262144 spec=dflash
+# which is EXACTLY this host (2x R9700 DID 0x7551, 32624 MiB each). Pinned
+# explicitly because this unit does not run the launcher's lookup; valid
+# only at this batch shape (change MAXSEQS/CHUNK/SPEC_METHOD => recalibrate
+# with calibrate-kv.sh or set KV_MEM=0 to let vLLM profile). Worth
+# 943,581 KV tokens vs ~892,799 profiled (~5.7%).
+Environment=KV_MEM=18563072000
+# Checkpoints (rw: nothing downloads at runtime, but vLLM mmaps the shards;
+# keep parity with the launcher's rw mount).
+Volume=/home/dank/models:/models
+# Tokenizer files ride the HF cache; HF_HUB_OFFLINE=1 is set in the
+# entrypoint.
+Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
+# Compile cache. Keyed on model + stack version + every graph-changing knob;
+# never share it across configurations (upstream rule).
+Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
+# The pinned radiance clone: patches applied at every container start.
+Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
+# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The
+# image's shipped r4d.so predates the GDN overflow fix and NaNs this model.
+Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
+# Entrypoint + healthcheck from this checkout, mounted read-only.
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
+Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
+# The entrypoint must be the container command; the image's own entrypoint
+# (which the patched script execs at the end) must not run unpatched.
+PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
+# 200 only once the engine serves; during first-start compilation the
+# container is legitimately unhealthy. HealthCmd is split on spaces into an
+# argv array by the podman 5.x v1 generator, hence the file-based probe.
+HealthCmd=python3 /usr/local/bin/frank-medium-healthcheck.py
+HealthInterval=60s
+HealthTimeout=30s
+HealthRetries=3
+HealthStartPeriod=1800s
+Pull=never
+
+[Service]
+# First start compiles Triton/inductor kernels (several minutes) before the
+# engine loads ~9.5 GiB of weights per card.
+TimeoutStartSec=1800
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/.AGENTTRASH/20260916-vllm-radiance/radiance-entrypoint.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+# frank-medium container entrypoint: vllm-radiance MXFP4 on 2x Radeon AI PRO
+# R9700 (gfx1201), serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq.
+#
+# This is a fork of the patch-and-serve block inside radiance-vllm-mxfp4's
+# serve-mxfp4.sh (clone at /home/dank/GitPackages/radiance-vllm-mxfp4, pinned
+# commit 9735329348ae1c9319f01cbe5f7004ec5f4dba63). The published image alone
+# is NOT sufficient for this model: the shipped libr4d predates the
+# gated-delta-net overflow fix and NaNs the output, and the MXFP4 gates need
+# relaxing, so the repo's patches are applied at every container start from
+# the /patches mount. It runs INSIDE the container (executed as the image
+# command by the frank-medium quadlet); the launcher's host-side GPU/TP/KV
+# detection is not needed here because this fleet is fixed 2x gfx1201 and the
+# KV pin is pinned explicitly in the quadlet.
+#
+# KEEP IN SYNC: if the radiance clone is updated, re-diff this file against
+# the new serve-mxfp4.sh container command and against the DRY_RUN output.
+#
+# Env contract: every knob below is ${VAR:-default}; the quadlet sets the
+# fleet-specific ones (SNAP, PORT, SERVED_NAMES, TP, KV_MEM, ...).
+set -euo pipefail
+
+# ---------------------------------------------------------------- serving shape
+: "${SNAP:=/models/Qwen3.8-27B-Uncensored-MXFP4-awq}"
+: "${DRAFTER:=/models/Qwen3.8-27B-DFlash2-FP8}"
+: "${CHAT_TEMPLATE:=/patches/qwen-fixed-v22.3.jinja}"
+: "${PORT:=7770}"
+: "${SERVED_NAMES:=frank-medium}"
+: "${MAXLEN:=262144}"
+: "${MAXSEQS:=8}"
+# KV_MEM=0 is passed THROUGH as --kv-cache-memory 0, which the image's
+# patched KV sizing reads as "profile the shape" (the launcher's rule).
+# Blanking the variable instead omits the flag and makes vLLM size the cache
+# from --gpu-memory-utilization alone: measured on the single-GPU replicas
+# that overshoots and the allocation dies with torch.OutOfMemoryError right
+# after the weights load (2.16 GiB requested, 116 MiB free). Do not "fix"
+# this into an empty expansion.
+: "${KV_MEM:=18563072000}"   # shipped kv-profiles.tsv row 2x7551-32624 @ 8/8192/262144/dflash
+: "${SPEC_METHOD:=dflash}"
+if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
+: "${MIN_M:=0}"              # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter)
+: "${R4D_ATTN:=1}"
+: "${ASYNC:=0}"
+
+if [ "$R4D_ATTN" = "1" ]; then ATTN=R4D; else ATTN=ROCM_AITER_UNIFIED_ATTN; fi
+if [ "$ASYNC" = 1 ]; then ASYNC_FLAG="--async-scheduling"; UNPAD=false; else
+  ASYNC_FLAG="--no-async-scheduling"; UNPAD=true; fi
+
+# Drafting config. dflash: the DFlash2-FP8 block-diffusion drafter (its
+# attention backend must support full cuda graphs -> TRITON_ATTN). The
+# disable_padded_drafter_batch unpad is the single-stream lever (~+50%).
+if [ "$SPEC_METHOD" = dflash ]; then
+  if [ ! -f "$DRAFTER/config.json" ]; then
+    echo "frank-medium: no dflash drafter at $DRAFTER" >&2
+    exit 1
+  fi
+  DRAFT_SAMPLE=${DRAFT_SAMPLE:-greedy}
+  SPEC_CFG="{\"method\":\"dflash\",\"model\":\"$DRAFTER\",\"num_speculative_tokens\":$SPEC,\"attention_backend\":\"TRITON_ATTN\",\"disable_padded_drafter_batch\":$UNPAD,\"draft_sample_method\":\"$DRAFT_SAMPLE\"}"
+  RADIANCE_DRAFT_RERANK=${RADIANCE_DRAFT_RERANK:-80}
+  RADIANCE_VERIFY_HEAD=${RADIANCE_VERIFY_HEAD:-1}
+else
+  SPEC_CFG="{\"method\":\"mtp\",\"num_speculative_tokens\":$SPEC,\"attention_backend\":\"$ATTN\",\"disable_padded_drafter_batch\":$UNPAD}"
+  RADIANCE_DRAFT_RERANK=${RADIANCE_DRAFT_RERANK:-32}
+  RADIANCE_VERIFY_HEAD=${RADIANCE_VERIFY_HEAD:-0}
+fi
+
+# The AR size gate compares raw bf16 bytes: CHUNK x hidden(5120) x 2. Derived
+# so changing CHUNK cannot silently drop prefill back onto RCCL.
+export RADIANCE_AR_MAX_KB=$(( (CHUNK * 5120 * 2) / 1024 + 4096 ))
+
+# ---------------------------------------------------------------- runtime env
+export HF_HUB_OFFLINE=1
+export VLLM_LOGGING_LEVEL=${VLLM_LOGGING_LEVEL:-INFO}
+export VLLM_NO_USAGE_STATS=${VLLM_NO_USAGE_STATS:-1}
+export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1
+export VLLM_ROCM_USE_AITER_MHA=0 VLLM_ROCM_USE_AITER_MLA=0 VLLM_ROCM_USE_AITER_MOE=0
+export VLLM_ROCM_USE_AITER_LINEAR=0 VLLM_ROCM_USE_AITER_FP8BMM=0
+export VLLM_ROCM_USE_AITER_FP4BMM=0 VLLM_ROCM_USE_AITER_RMSNORM=0
+export NCCL_PROTO=Simple
+export RADIANCE_USE_R4D=${RADIANCE_USE_R4D:-1} RADIANCE_USE_R4D_AR=${RADIANCE_USE_R4D_AR:-1}
+export RADIANCE_DFLASH_SELECTOR_TOPK="${RADIANCE_DFLASH_SELECTOR_TOPK:-}"
+export RADIANCE_R4D_REPORT=1
+export RADIANCE_PRESHUFFLE=${RADIANCE_PRESHUFFLE:-1}
+export RADIANCE_MXFP4_SANITIZE="${RADIANCE_MXFP4_SANITIZE:-0}"
+export RADIANCE_MXFP4=1 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_W4A8_MIN_M="$MIN_M"
+export RADIANCE_FAST_DRAFT=${RADIANCE_FAST_DRAFT:-1}
+export RADIANCE_DRAFT_TAU=${RADIANCE_DRAFT_TAU:-0.20}
+export RADIANCE_DRAFT_RERANK
+export RADIANCE_VERIFY_HEAD
+export RADIANCE_VERIFY_HEAD_MAX_M=${RADIANCE_VERIFY_HEAD_MAX_M:-32}
+export RADIANCE_MXFP4_DEBUG=${RADIANCE_MXFP4_DEBUG:-0}
+export RADIANCE_MXFP4_PUREQUANT=${RADIANCE_MXFP4_PUREQUANT:-0}
+export RADIANCE_MXFP4_SYNC=${RADIANCE_MXFP4_SYNC:-0}
+export RADIANCE_MXFP4_CLONE=${RADIANCE_MXFP4_CLONE:-0}
+export RADIANCE_MXFP4_CHECKX=${RADIANCE_MXFP4_CHECKX:-0}
+export RADIANCE_MXFP4_PADOUT=${RADIANCE_MXFP4_PADOUT:-0}
+export RADIANCE_MXFP4_TN4_MIN_M=${RADIANCE_MXFP4_TN4_MIN_M:-2048}
+# The decode-kernel band must cover MAXSEQS x (SPEC+1) rows: 64 covers 8x8.
+if [ "${MAXSEQS}" -gt 8 ]; then
+  export RADIANCE_MXFP4_DECODE_MAX_M=${RADIANCE_MXFP4_DECODE_MAX_M:-128}
+fi
+export RADIANCE_MXFP4_DECODE_NT=${RADIANCE_MXFP4_DECODE_NT:-1}
+export RADIANCE_MXFP4_A_TILED_MIN_M=${RADIANCE_MXFP4_A_TILED_MIN_M:-513}
+export RADIANCE_MXFP4_WPERM=${RADIANCE_MXFP4_WPERM:-1}
+export RADIANCE_TP_PAD=${RADIANCE_TP_PAD:-0}   # TP=2 here; 3 is the only non-zero value upstream supports
+export RADIANCE_TP_PAD_DRAFTER=${RADIANCE_TP_PAD_DRAFTER:-1}
+export RADIANCE_TP_PAD_STRICT=${RADIANCE_TP_PAD_STRICT:-1}
+export RADIANCE_GDN_MERGE_INPROJ=${RADIANCE_GDN_MERGE_INPROJ:-1}
+export RADIANCE_GDN_NORM_QUANT=${RADIANCE_GDN_NORM_QUANT:-1}
+export RADIANCE_GDN_STRIDED_GATES=${RADIANCE_GDN_STRIDED_GATES:-0}
+export RADIANCE_GDN_EMPTY_OUT=${RADIANCE_GDN_EMPTY_OUT:-0}
+export R4D_ATTN_FP8=${R4D_ATTN_FP8:-3}
+export RADIANCE_DFLASH_CALIB="${RADIANCE_DFLASH_CALIB:-}"
+export RADIANCE_DFLASH_CALIB_TOKENS="${RADIANCE_DFLASH_CALIB_TOKENS:-200000}"
+export RADIANCE_MXFP4_SHADOW="${RADIANCE_MXFP4_SHADOW:-}"
+export RADIANCE_MXFP4_KERNEL_N="${RADIANCE_MXFP4_KERNEL_N:-}"
+export RADIANCE_MXFP4_KERNEL_NK="${RADIANCE_MXFP4_KERNEL_NK:-}"
+export RADIANCE_MXFP4_CHECKALL="${RADIANCE_MXFP4_CHECKALL:-}"
+export RADIANCE_MXFP4_DECODE_KS="${RADIANCE_MXFP4_DECODE_KS:-}"
+export RADIANCE_MXFP4_DECODE_BK="${RADIANCE_MXFP4_DECODE_BK:-}"
+export RADIANCE_MXFP4_PERBLOCK_NK="${RADIANCE_MXFP4_PERBLOCK_NK:-}"
+export RADIANCE_MXFP4_REFLINEAR="${RADIANCE_MXFP4_REFLINEAR:-0}"
+export PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-}"
+export RADIANCE_MXFP4_EPIFAST="${RADIANCE_MXFP4_EPIFAST:-1}"
+export RADIANCE_AR_OVERLAP=${RADIANCE_AR_OVERLAP:-0}
+export RADIANCE_GDN_FUSED_UPDATE=${RADIANCE_GDN_FUSED_UPDATE:-1}
+export RADIANCE_DYNAMIC_WIDTH=${RADIANCE_DYNAMIC_WIDTH:-1}
+export RADIANCE_DYNW_ALPHA=${RADIANCE_DYNW_ALPHA:-0.35}
+export RADIANCE_DYNW_MARGIN=${RADIANCE_DYNW_MARGIN:-2}
+export RADIANCE_DYNW_MIN=${RADIANCE_DYNW_MIN:-2}
+export RADIANCE_DYNW_MIN_BATCH=${RADIANCE_DYNW_MIN_BATCH:-3}
+export RADIANCE_AR_QNB=${RADIANCE_AR_QNB:-96}
+export RADIANCE_AR_QNT=${RADIANCE_AR_QNT:-1024}
+export MXFP4_CUMODE=${MXFP4_CUMODE:-0}
+export RADIANCE_STEP_TRACE=${RADIANCE_STEP_TRACE:-0}
+export RADIANCE_AR_OVERLAP_MIN_M=${RADIANCE_AR_OVERLAP_MIN_M:-2048}
+export RADIANCE_AR_OVERLAP_SLICES=${RADIANCE_AR_OVERLAP_SLICES:-4}
+export RADIANCE_MXFP4_EPIFAST=${RADIANCE_MXFP4_EPIFAST:-1}
+export RADIANCE_MXFP4_R4D_DECODE_MAX_M=${RADIANCE_MXFP4_R4D_DECODE_MAX_M:-0}
+export RADIANCE_TOPK_TRITON_MIN_ROWS=${RADIANCE_TOPK_TRITON_MIN_ROWS:-1}
+export RADIANCE_SKINNY_GEMM=${RADIANCE_SKINNY_GEMM:-1}
+export RADIANCE_MXFP4_HOIST_QUANT=${RADIANCE_MXFP4_HOIST_QUANT:-1}
+export RADIANCE_MXFP4_TRACED_QUANT=${RADIANCE_MXFP4_TRACED_QUANT:-1}
+export RADIANCE_FP8_STREAM=${RADIANCE_FP8_STREAM:-1}
+export RADIANCE_RMS_QUANT_FUSION=${RADIANCE_RMS_QUANT_FUSION:-1}
+export RADIANCE_NVFP4_MXFP4=${RADIANCE_NVFP4_MXFP4:-0}
+export RADIANCE_NVFP4_EXP=${RADIANCE_NVFP4_EXP:-mse}
+export RADIANCE_NVFP4_FP8_LAYERS=${RADIANCE_NVFP4_FP8_LAYERS:-mxfp4}
+export RADIANCE_NVFP4_BF16_LAYERS=${RADIANCE_NVFP4_BF16_LAYERS:-in_proj_ba}
+export RADIANCE_NVFP4_LMHEAD=${RADIANCE_NVFP4_LMHEAD:-bf16}
+export RADIANCE_GDN_PATHS=${RADIANCE_GDN_PATHS:-both}
+export RADIANCE_GDN_NANTRACE=${RADIANCE_GDN_NANTRACE:-0}
+export RADIANCE_MXFP4_CHECK_MAX_M=${RADIANCE_MXFP4_CHECK_MAX_M:-128}
+export RADIANCE_MXFP4_MHIST=${RADIANCE_MXFP4_MHIST:-0}
+export VLLM_CACHE_ROOT=/cache/vllm
+export TORCHINDUCTOR_CACHE_DIR=/cache/inductor
+export TRITON_CACHE_DIR=/cache/triton
+export AITER_ROOT_DIR=/cache/aiter
+export TRITON_CACHE_AUTOTUNING=1
+# The patched libr4d built by setup-mxfp4.sh, bind-mounted at /r4d.
+R4D_SO=${R4D_SO:-/r4d}
+
+# The norm+quant fusion turns on vLLM's passes (one --compilation-config
+# instance; argparse keeps the last, so entries accumulate into one flag).
+CC_ITEMS=""
+if [ "${RADIANCE_NORMQUANT_FUSION:-1}" != "0" ]; then
+  CC_ITEMS="\"pass_config\":{\"fuse_norm_quant\":true,\"fuse_act_quant\":true}"
+fi
+EXTRA=${EXTRA:-}
+if [ -n "$CC_ITEMS" ]; then
+  EXTRA="$EXTRA --compilation-config {$CC_ITEMS}"
+fi
+
+# ---------------------------------------------------------------- patch block
+# Byte-for-byte the launcher's container-start block. The patches come from
+# the /patches mount (the pinned radiance clone). SP must match the image's
+# venv layout; the image tag is pinned in the quadlet.
+SP=/opt/vllm/lib/python3.12/site-packages
+cd /patches
+python3 patch_quark_mxfp4.py
+python3 patch_nvfp4_mxfp4.py
+python3 patch_tp3_pad.py
+python3 patch_ar_maxbytes.py
+python3 patch_topk_triton_rows.py
+python3 patch_dflash_calib.py
+python3 patch_dflash_mxfp4_kv.py
+python3 patch_rmsquant_fusion.py
+python3 patch_verify_head.py
+python3 patch_kv_group_size.py
+python3 patch_topk_composite.py
+python3 patch_gdn_shared_build.py
+python3 patch_dflash_selector_topk.py
+python3 patch_gdn_merge_inproj.py
+python3 patch_dynwidth.py
+python3 patch_async_dynwidth.py
+python3 patch_step_trace.py
+python3 patch_ar_geometry.py
+python3 patch_ar_3rank.py
+python3 patch_gdn_glue.py
+# Non-fatal: fixes content=null on thinking-off requests; not required to serve.
+python3 patch_qwen3_thinkoff.py \
+  || echo "[radiance] WARNING: thinkoff patch did not apply; thinking-off requests will return empty content"
+cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/
+cp radiance_preamble.py /opt/radiance_preamble.py
+cp radiance_nvfp4.py radiance_mxfp4.py radiance_gdn.py radiance_rmsquant.py radiance_drafthead.py \
+   radiance_verifyhead.py radiance_gdnmerge.py radiance_aroverlap.py radiance_topk.py \
+   radiance_arnq.py radiance_tp3pad.py "$SP"/
+hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $([ "${MXFP4_CUMODE:-0}" = 1 ] && echo -mcumode) \
+  $(python3 -m pybind11 --includes) radiance_mxfp4_fp8.hip -o "$SP"/radiance_mxfp4_fp8.so
+if [ -n "${R4D_SO:-}" ] && [ -f /r4d/r4d.so ]; then
+  cp /r4d/r4d.so "$SP"/r4d.so
+  echo "[radiance] using patched r4d.so from $R4D_SO"
+fi
+# Leave /patches before exec: a stale radiance_mxfp4_fp8.so there would shadow
+# the one just compiled into site-packages (cwd precedes it on sys.path).
+cd /
+exec /opt/radiance_entrypoint.sh "$SNAP" --served-model-name $SERVED_NAMES \
+  --host 0.0.0.0 --port "$PORT" \
+  --kv-cache-dtype fp8 --tensor-parallel-size "$TP" \
+  --gpu-memory-utilization "$GPU_UTIL" \
+  ${KV_MEM:+--kv-cache-memory "$KV_MEM"} \
+  --max-model-len "$MAXLEN" --max-num-seqs "$MAXSEQS" --max-num-batched-tokens "$CHUNK" \
+  --attention-backend "$ATTN" \
+  --speculative-config "$SPEC_CFG" \
+  $ASYNC_FLAG $EXTRA \
+  --enable-prefix-caching --mamba-cache-mode align --enable-auto-tool-choice \
+  --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
+  --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}' \
+  --chat-template "$CHAT_TEMPLATE"

dankserver/.AGENTTRASH/20260916-vllm-radiance/routing.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+"""Routing policy for the frank-medium proxy: which replica serves a request.
+
+Kept apart from the HTTP plumbing in affinity-proxy.py so the decisions can be
+tested without aiohttp, a network or a GPU — the first version of these rules
+shipped with a bug that only appeared on a continuation request, and a test
+that loads this module catches that class of mistake directly.
+
+Two cards, two behaviours, and the workload decides which is right:
+
+* A **single-turn** request is one member of a batch (a benchmark run, a burst
+  of parallel subagents). Spreading those over both replicas lets both cards
+  prefill at once; measured on this host, a 16-request burst of 15k-token
+  prompts finishes in 18.6 s spread versus 30.1 s pinned.
+* A **continuation** re-sends a prefix the replica has already prefilled. Pinned
+  to that replica the turn costs a cache lookup; spread, half the turns pay a
+  cold prefill again (measured: 1096 ms versus 2676 ms mean warm-turn TTFT on
+  90k-token contexts).
+
+ROUTING picks the strategy; the default `adaptive` applies both rules above.
+`affinity` and `roundrobin` exist so the choice can be re-measured, and are not
+production settings.
+"""
+
+from __future__ import annotations
+
+import collections
+import hashlib
+import itertools
+import json
+import os
+
+UPSTREAMS = [u.strip() for u in os.environ.get(
+    "UPSTREAMS", "http://127.0.0.1:7771,http://127.0.0.1:7772").split(",") if u.strip()]
+ROUTING = os.environ.get("ROUTING", "adaptive").strip().lower()
+# Leading bytes of prompt text that identify a conversation's shared prefix.
+# 4 KiB covers a system prompt plus the start of the first user turn, which is
+# where agent loops keep the text they resend every turn.
+AFFINITY_BYTES = int(os.environ.get("AFFINITY_BYTES", "4096"))
+# Prefixes remembered for pinning. Bounded so a long-lived proxy cannot grow
+# without limit; 4096 conversations is far beyond what this fleet runs at once.
+KNOWN_MAX = int(os.environ.get("KNOWN_MAX", "4096"))
+
+_counter = itertools.count()
+# digest -> replica index, insertion-ordered for O(1) LRU eviction. A prefix is
+# remembered as soon as any request carries it; only a continuation is pinned
+# there, because only a continuation can profit from the cache entry.
+_known: "collections.OrderedDict[bytes, int]" = collections.OrderedDict()
+
+
+def affinity_key(body: bytes, remote: str | None) -> tuple[bytes, bool]:
+    """Key a request by its shared prefix, and report whether it continues one.
+
+    Chat and completions bodies both carry their prompt text in JSON, so the
+    leading bytes of that text are the prefix; a body we cannot parse (or an
+    endpoint with no prompt, e.g. /v1/models) falls back to the client address,
+    which still beats nothing for a single-client loop.
+    """
+    fallback = (remote or "anonymous").encode()
+    try:
+        payload = json.loads(body)
+    except Exception:
+        return fallback, False
+
+    text = ""
+    continuation = False
+    if isinstance(payload, dict):
+        if isinstance(payload.get("prompt"), str):
+            text = payload["prompt"]
+        elif isinstance(payload.get("messages"), list):
+            continuation = len(payload["messages"]) > 1
+            parts = []
+            for msg in payload["messages"]:
+                if isinstance(msg, dict):
+                    content = msg.get("content")
+                    if isinstance(content, str):
+                        parts.append(content)
+                    elif isinstance(content, list):  # multimodal content blocks
+                        parts.extend(
+                            blk.get("text", "") for blk in content
+                            if isinstance(blk, dict) and isinstance(blk.get("text"), str)
+                        )
+            text = "\n".join(parts)
+    if not text:
+        return fallback, continuation
+    return text[:AFFINITY_BYTES].encode("utf-8", "replace"), continuation
+
+
+def pick(key: bytes, continuation: bool, up: list[bool]) -> int:
+    """Replica for a request, or -1 when none is available.
+
+    Down replicas are never selected — a rollout must not send traffic to an
+    engine that is still compiling its graphs — and a pin whose replica is down
+    is re-established on a live one rather than kept pointing at a corpse.
+    """
+    digest = hashlib.sha1(key).digest()
+
+    if ROUTING == "roundrobin":
+        start = next(_counter) % len(UPSTREAMS)
+    elif ROUTING == "adaptive":
+        entry = _known.get(digest)
+        if continuation and entry is not None and up[entry]:
+            _known.move_to_end(digest)
+            return entry
+        # Anything that is not a continuation spreads by counter — including a
+        # batch of requests that happen to share one prefix. Hashing the prefix
+        # would send every member of such a batch to whichever card the hash
+        # names (measured: 29.8 s for a 16-request burst, against 18.6 s when
+        # spread). The counter is only advanced by these requests, so batch
+        # members alternate across the cards.
+        start = next(_counter) % len(UPSTREAMS)
+    else:  # affinity
+        start = int.from_bytes(digest[:8], "big") % len(UPSTREAMS)
+
+    for offset in range(len(UPSTREAMS)):
+        idx = (start + offset) % len(UPSTREAMS)
+        if up[idx]:
+            if ROUTING == "adaptive":
+                _known[digest] = idx
+                _known.move_to_end(digest)
+                if len(_known) > KNOWN_MAX:
+                    _known.popitem(last=False)
+            return idx
+    return -1
+
+
+def route(body: bytes, remote: str | None, up: list[bool]) -> int:
+    """Convenience wrapper: decode the request body and choose a replica."""
+    return pick(*affinity_key(body, remote), up)

dankserver/.env.example

@@ -20,5 +20,3 @@ COPAY_PASS=changeme
 # Jellyfin
 JELLYFIN_PUBLISH_SERVER_URL=http://192.168.0.235:8096
 
-# *arr Stack
-ARR_TZ=America/New_York

dankserver/calibre-web/quadlet/calibre-web.container

@@ -0,0 +1,43 @@
+# Quadlet unit: Calibre-Web library server (rootless, boot-persistent).
+#
+# Install path on dankserver:
+# ~/.config/containers/systemd/calibre-web.container
+# Unit name: calibre-web.service (podman 5.x drops the container- prefix)
+#
+# The data roots already lived under /home/dank/Containers/calibre-web in the
+# compose era; this unit keeps those exact paths. The image pin is the exact
+# tag of the image the live container runs (image id 986277563e76), so the
+# cutover recreation changes zero bits of application code. No secrets: this
+# unit takes no EnvironmentFile.
+
+[Unit]
+Description=Calibre-Web library server container
+After=network.target
+
+[Container]
+ContainerName=calibre-web
+Image=lscr.io/linuxserver/calibre-web:0.6.27-ls397
+Network=pasta
+Environment=DOCKER_MODS=linuxserver/mods:universal-calibre
+Environment=LANG=en_US.UTF-8
+Environment=LANGUAGE=en_US.UTF-8
+Environment=LSIO_FIRST_PARTY=true
+Environment=OAUTHLIB_RELAX_TOKEN_SCOPE=1
+Environment=PGID=1000
+Environment=PUID=1000
+Environment=QTWEBENGINE_CHROMIUM_FLAGS=--no-sandbox
+Environment=S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
+Environment=S6_STAGE2_HOOK=/docker-mods
+Environment=TERM=xterm
+Environment=TZ=America/Vancouver
+Environment=VIRTUAL_ENV=/lsiopy
+PublishPort=8083:8083
+Volume=/home/dank/Containers/calibre-web/config:/config
+Volume=/home/dank/Containers/calibre-web/books:/books
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/compose.yaml

@@ -74,16 +74,20 @@ services:
   # Copyparty — file sharing and streaming
   # ═══════════════════════════════════════════════
   copyparty:
-    image: ghcr.io/9001/copyparty:latest
+    image: ghcr.io/9001/copyparty-ac:latest
     container_name: copyparty
     restart: unless-stopped
-    environment:
+    # read-only since 2026-08-31: copyparty perms are per-field "letters[,user]"
-      COPAY_PASS: "${COPAY_PASS}"
+    # with no user = every unauthenticated request. Anonymous gets r (browse +
-      COPAY_MODE: rw
+    # download; the guarded files.franksdank.cloud chain reaches copyparty
+    # unauthenticated after authentik). dank gets ra (admin panel). No w/m/d
+    # anywhere: uploads, moves, deletes rejected server-side. Interface: `--help-accounts` in-image.
+    command: ["-a", "dank:${COPAY_PASS}", "-v", "/w::r:ra,dank"]
     ports:
       - "3923:3923"
     volumes:
-      - /opt/copyparty/data:/data
+      - /mnt/entropy:/w:ro
+      - /opt/copyparty/cfg:/cfg
     healthcheck:
       test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3923/')"]
       interval: 30s
@@ -97,7 +101,6 @@ services:
     image: jellyfin/jellyfin:latest
     container_name: jellyfin
     restart: unless-stopped
-    user: "1000:1000"
     ports:
       - "8096:8096/tcp"
       - "8920:8920/tcp"
@@ -109,8 +112,7 @@ services:
     volumes:
       - /opt/jellyfin/config:/config
       - /opt/jellyfin/cache:/cache
-      - /mnt/media/movies:/data/movies:ro
+      - /mnt/entropy:/data:ro
-      - /mnt/media/shows:/data/shows:ro
     devices:
       - /dev/dri:/dev/dri
     group_add:
@@ -144,129 +146,9 @@ services:
       timeout: 10s
       retries: 3
 
-  # ═══════════════════════════════════════════════
-  # *arr Stack — automated media download
-  # ═══════════════════════════════════════════════
-  prowlarr:
-    image: lscr.io/linuxserver/prowlarr:latest
-    container_name: prowlarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/prowlarr/config:/config
-      - /mnt/media/downloads:/downloads:ro
-    ports:
-      - "9696:9696"
-    networks: [appnet]
 
-  qbittorrent:
+  # node-exporter left this file on 2026-09-16 for
-    image: lscr.io/linuxserver/qbittorrent:latest
+  # dankserver/observability/quadlet/node-exporter.container: a compose-only
-    container_name: qbittorrent
+  # container does not survive a reboot here — it died `Exited (143)` and stayed
-    restart: unless-stopped
+  # dead, leaving Prometheus' node-dankserver job scraping nothing — and the
-    environment:
+  # compose form bound the wildcard address. One owner per container name.
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/qbittorrent/config:/config
-      - /mnt/media/downloads:/downloads
-      - /mnt/media/downloads/incomplete:/downloads/incomplete
-    ports:
-      - "8080:8080"
-    networks: [appnet]
-
-  radarr:
-    image: lscr.io/linuxserver/radarr:latest
-    container_name: radarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/radarr/config:/config
-      - /mnt/media/movies:/movies
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "7878:7878"
-    networks: [appnet]
-
-  sonarr:
-    image: lscr.io/linuxserver/sonarr:latest
-    container_name: sonarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/sonarr/config:/config
-      - /mnt/media/tv:/tv
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "8989:8989"
-    networks: [appnet]
-
-  lidarr:
-    image: lscr.io/linuxserver/lidarr:latest
-    container_name: lidarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/lidarr/config:/config
-      - /mnt/media/music:/music
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "8686:8686"
-    networks: [appnet]
-
-  readarr:
-    image: lscr.io/linuxserver/readarr:latest
-    container_name: readarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/readarr/config:/config
-      - /mnt/media/books:/books
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "8787:8787"
-    networks: [appnet]
-
-  # ═══════════════════════════════════════════════
-  # Node Exporter — host metrics
-  # ═══════════════════════════════════════════════
-  node-exporter:
-    image: prom/node-exporter:latest
-    container_name: node-exporter
-    restart: unless-stopped
-    network_mode: host
-    pid: host
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-    command:
-      - --path.procfs=/host/proc
-      - --path.sysfs=/host/sys
-      - --path.rootfs=/rootfs
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:9100/metrics"]
-      interval: 30s
-      timeout: 10s
-      retries: 3

dankserver/copyparty/quadlet/copyparty.container

@@ -0,0 +1,38 @@
+# Quadlet unit: Copyparty file server (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/copyparty.container
+# Unit name: copyparty.service (podman 5.x drops the container- prefix)
+#
+# The image's default entrypoint is /bin/ash /z/cpp.sh -c /z/initcfg; the
+# Command= line below reproduces the compose-era argv exactly: the admin
+# account is passed as -a <user>:<password>. ${COPAY_ACCOUNT} and
+# ${COPAY_PASSWORD} are expanded by systemd at ExecStart time from the
+# host-local env file /home/dank/Containers/copyparty/env (mode 0600).
+# This file holds no secrets. The deployer refuses to start this unit before
+# that env file exists, so a missing env file fails the unit instead of
+# creating an admin account with a garbage password.
+#
+# /w is the entropy volume, mounted read-only: nothing writes into /mnt/entropy
+# anymore; every consumer of the share is a read-only client.
+
+[Unit]
+Description=Copyparty file server container
+After=network.target
+
+[Container]
+ContainerName=copyparty
+Image=ghcr.io/9001/copyparty-ac:latest
+Network=dankserver_appnet
+EnvironmentFile=/home/dank/Containers/copyparty/env
+Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} -v /w::r:ra,dank
+Volume=/mnt/entropy:/w:ro
+Volume=/home/dank/Containers/copyparty/cfg:/cfg
+PublishPort=3923:3923
+
+[Service]
+EnvironmentFile=/home/dank/Containers/copyparty/env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/jellyfin/quadlet/jellyfin.container

@@ -0,0 +1,57 @@
+# Quadlet unit: Jellyfin media server (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/jellyfin.container
+# Unit name: jellyfin.service (podman 5.x drops the container- prefix)
+#
+# Data roots migrated from /opt/jellyfin to /home/dank/Containers/jellyfin at
+# cutover. All of /mnt/entropy is mounted read-only at /data: nothing automated
+# anywhere writes to the entropy volume. Every
+# creation-time environment variable of the compose-era container is kept as
+# an Environment= line (zero drift); none of them are secrets, so this unit
+# needs no EnvironmentFile.
+
+[Unit]
+Description=Jellyfin media server container
+After=network.target
+
+[Container]
+ContainerName=jellyfin
+Image=ghcr.io/jellyfin/jellyfin:latest
+Network=dankserver_appnet
+Environment=DEBIAN_FRONTEND=noninteractive
+Environment=DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
+Environment=JELLYFIN_CACHE_DIR=/cache
+Environment=JELLYFIN_CONFIG_DIR=/config/config
+Environment=JELLYFIN_DATA_DIR=/config
+Environment=JELLYFIN_FFMPEG=/usr/lib/jellyfin-ffmpeg/ffmpeg
+Environment=JELLYFIN_LOG_DIR=/config/log
+Environment=JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
+Environment=JELLYFIN_WEB_DIR=/jellyfin/jellyfin-web
+Environment=LANG=en_US.UTF-8
+Environment=LANGUAGE=en_US:en
+Environment=LC_ALL=en_US.UTF-8
+Environment=LD_PRELOAD=/usr/lib/jellyfin/libjemalloc.so.2
+Environment=MALLOC_TRIM_THRESHOLD_=131072
+Environment=NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
+Environment=NVIDIA_VISIBLE_DEVICES=all
+Environment=XDG_CACHE_HOME=/cache
+PublishPort=8096:8096
+PublishPort=8920:8920
+PublishPort=7359:7359/udp
+PublishPort=1900:1900/udp
+Volume=/home/dank/Containers/jellyfin/config:/config
+Volume=/home/dank/Containers/jellyfin/cache:/cache
+Volume=/mnt/entropy:/data:ro
+AddDevice=/dev/dri
+GroupAdd=keep-groups
+HealthCmd=curl -f http://localhost:8096/health
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/observability/quadlet/node-exporter.container

@@ -0,0 +1,58 @@
+# Quadlet unit: node-exporter (rootless, boot-persistent) — DANKSERVER.
+#
+# Install path on dankserver: ~/.config/containers/systemd/node-exporter.container
+# Unit name: node-exporter.service (container name: node-exporter)
+#
+# WHY THIS FILE EXISTS. Until 2026-09-16 the Dankserver node-exporter ran only
+# from `dankserver/compose.yaml` (the "Observability" block, `restart:
+# unless-stopped`). Rootless Podman does not restart bare containers after the
+# explicit stop that shutdown performs, so it died at a reboot and stayed dead:
+# `podman ps -a` showed it `Exited (143)` and Prometheus' `node-dankserver` job
+# had been scraping nothing since. The compose entry also bound the default
+# wildcard address, which is the unauthenticated-metrics-API-on-every-interface
+# posture that was already removed on Dankclaw. This unit replaces both: one
+# owner per container name (the compose service is deleted), explicit listen
+# addresses, and a generator-provided boot path.
+#
+# HOST NAMESPACE. Networking and PID in the host's namespaces: the collectors
+# report the host's own interfaces, connections, processes and cgroups. The
+# host's /proc, /sys and / are mounted read-only under /host and /rootfs, and
+# node_exporter writes nothing.
+#
+# LISTEN ADDRESSES. Exactly two:
+#   - 127.0.0.1:9100   for host-local probes
+#   - 192.168.0.235:9100 for Prometheus, which scrapes this host from Dankclaw
+#     (shared/prometheus/prometheus.yml, job "node-dankserver"). It cannot use
+#     loopback across hosts, and node_exporter has no authentication, so the
+#     wildcard bind stays off the table. This is the same deliberate LAN
+#     listener posture the observability runbook documents.
+#
+# MIRROR. Keep this file in step with dankclaw/observability/quadlet/
+# node-exporter.container; only the Description and the LAN address differ.
+
+[Unit]
+Description=Prometheus node exporter (Dankserver host metrics)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=node-exporter
+Image=docker.io/prom/node-exporter:latest
+Network=host
+PodmanArgs=--pid=host
+Exec=--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs --web.listen-address=127.0.0.1:9100 --web.listen-address=192.168.0.235:9100
+Volume=/proc:/host/proc:ro
+Volume=/sys:/host/sys:ro
+Volume=/:/rootfs:ro
+HealthCmd=wget --spider -q http://localhost:9100/metrics
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=30s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target

dankserver/plex/quadlet/plex.container

@@ -0,0 +1,22 @@
+[Unit]
+Description=Plex Media Server (quadlet, rootless fleet identity)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+Image=lscr.io/linuxserver/plex:1.43.3.10896-cb3ebc72d-ls322
+ContainerName=plex
+Network=dankserver_appnet:alias=plex
+Environment=PUID=0
+Environment=PGID=0
+Environment=TZ=America/Los_Angeles
+Volume=/var/lib/plexmediaserver:/config
+Volume=/mnt/entropy:/mnt/entropy:ro
+PublishPort=32400:32400
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/romm/quadlet/romm-mariadb.container

@@ -0,0 +1,34 @@
+# Quadlet unit: MariaDB backing store for RomM (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/romm-mariadb.container
+# Unit name: romm-mariadb.service (podman 5.x drops the container- prefix)
+#
+# The alias `mariadb` on dankserver_appnet is what makes DB_HOST=mariadb in
+# the RomM unit resolve. Secrets (MARIADB_ROOT_PASSWORD, MARIADB_PASSWORD)
+# live in the shared host-local env file /home/dank/Containers/romm/env
+# (mode 0600); this file holds no secrets.
+
+[Unit]
+Description=RomM MariaDB database container
+After=network.target
+
+[Container]
+ContainerName=romm-mariadb
+Image=docker.io/library/mariadb:11
+Network=dankserver_appnet:alias=mariadb
+EnvironmentFile=/home/dank/Containers/romm/env
+Environment=MARIADB_DATABASE=romm
+Environment=MARIADB_USER=romm
+Volume=/home/dank/Containers/romm/mariadb:/var/lib/mysql
+HealthCmd=mariadb-admin ping -h localhost
+HealthInterval=10s
+HealthTimeout=5s
+HealthRetries=3
+
+[Service]
+EnvironmentFile=/home/dank/Containers/romm/env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/romm/quadlet/romm-redis.container

@@ -0,0 +1,28 @@
+# Quadlet unit: Redis cache for RomM (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/romm-redis.container
+# Unit name: romm-redis.service (podman 5.x drops the container- prefix)
+#
+# The alias `redis` on dankserver_appnet is what makes REDIS_HOST=redis in
+# the RomM unit resolve. No secrets: this unit takes no EnvironmentFile.
+
+[Unit]
+Description=RomM Redis cache container
+After=network.target
+
+[Container]
+ContainerName=romm-redis
+Image=docker.io/library/redis:7-alpine
+Network=dankserver_appnet:alias=redis
+Volume=/home/dank/Containers/romm/redis:/data
+HealthCmd=redis-cli ping
+HealthInterval=10s
+HealthTimeout=5s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/romm/quadlet/romm.container

@@ -0,0 +1,42 @@
+# Quadlet unit: RomM library manager (rootless, boot-persistent).
+#
+# Install path on dankserver: ~/.config/containers/systemd/romm.container
+# Unit name: romm.service (podman 5.x drops the container- prefix)
+#
+# The whole data root lives at /home/dank/Containers/romm and is mounted at
+# /romm, so config/, library/, resources/ and assets/ sit inside it exactly as
+# the compose-era bind mounts + anonymous volume did. Secrets (DB_PASSWD,
+# ROMM_AUTH_SECRET_KEY, ROMM_PASSWORD, SCREENSCRAPER_DEV_*) live in the
+# host-local env file /home/dank/Containers/romm/env (mode 0600); this file
+# holds no secrets. The network dankserver_appnet must exist before first
+# start (it is the compose-era network; on a fresh host create it first).
+
+[Unit]
+Description=RomM library manager container
+Wants=romm-mariadb.service
+After=romm-mariadb.service romm-redis.service network.target
+
+[Container]
+ContainerName=romm
+Image=ghcr.io/rommapp/romm:latest
+Network=dankserver_appnet
+EnvironmentFile=/home/dank/Containers/romm/env
+Environment=DB_HOST=mariadb
+Environment=DB_NAME=romm
+Environment=DB_USER=romm
+Environment=REDIS_HOST=redis
+Environment=HASHEOUS_API_ENABLED=true
+PublishPort=8085:8080
+Volume=/home/dank/Containers/romm:/romm
+HealthCmd=curl -f http://localhost:8080/health
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+
+[Service]
+EnvironmentFile=/home/dank/Containers/romm/env
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

dankserver/tautulli/quadlet/tautulli.container

@@ -0,0 +1,25 @@
+[Unit]
+Description=Tautulli (quadlet, rootless fleet identity)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+Image=lscr.io/linuxserver/tautulli:v2.18.1-ls242
+ContainerName=tautulli
+Network=dankserver_appnet:alias=tautulli
+Environment=PUID=0
+Environment=PGID=0
+Environment=TZ=America/Los_Angeles
+Volume=/home/dank/Containers/tautulli:/config
+PublishPort=8181:8181
+HealthCmd=curl -fs http://localhost:8181/home
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+
+[Service]
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=default.target

docs/Architecture/Deployment Flow.md

@@ -1,60 +0,0 @@
-# Deployment Flow
-
-Git is the single source of truth. A `deploy.sh` script moves committed
-changes to the target hosts over SSH. One run per target covers both
-workloads on that host: the podman stack and the LLM service.
-
-## Workflow
-
-```mermaid
-graph LR
-    A[Edit in this repo] --> B[Commit]
-    B --> C[Push to origin]
-    C --> D[deploy.sh: pull on target]
-    D --> E[restart stack]
-    E --> F[health check]
-```
-
-1. Edit files locally in this repository.
-2. Commit and push.
-3. Run `./scripts/deploy.sh <dankclaw|dankserver>`.
-4. The script pulls the repo on the target and restarts the podman stack
-   and the LLM service.
-5. It verifies service health after restart (homepage and LLM endpoint).
-
-## Deploy Script
-
-```
-./scripts/deploy.sh dankclaw          # pull + restart on Dankclaw
-./scripts/deploy.sh dankserver        # pull + restart on Dankserver
-./scripts/deploy.sh dankclaw --dry-run
-```
-
-SSH uses the existing key pair (`dank@dankclaw`, `dank@dankserver`). No passwords are stored or documented in this repo. If sudo is needed, use the interactive sudo on the target — never commit a password.
-
-## Targets
-
-| Target | Host | Stack |
-|--------|------|-------|
-| dankclaw | dank@dankclaw (192.168.0.143) | `dankclaw/` |
-| dankserver | dank@dankserver (192.168.0.235) | `dankserver/` |
-
-## Health Verification
-
-The deploy script polls the health endpoint of each service after restart. Service healthchecks are defined in each compose file. A service that fails its healthcheck is reported, not hidden.
-
-## Tests
-
-Run the test suite before pushing:
-
-```
-./tests/run.sh
-```
-
-The suite validates compose YAML, env consistency, Caddyfiles, docs completeness, and script syntax. See [[Testing Policy]].
-
-## See Also
-
-- [[Storage Policy]]
-- [[Testing Policy]]
-- [LLM deployment docs](../LLM/deployment.md) — the LLM half of the same flow

docs/Architecture/Network Architecture.md

@@ -1,63 +0,0 @@
-# Network Architecture
-
-One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) and terminates TLS for every service on the LAN. Pi-hole on Dankclaw answers the service names for LAN clients; Porkbun answers them for the WAN. The G36 router firewall governs external access. SearXNG outbound traffic exits through the Gluetun VPN container.
-
-## Topology
-
-```mermaid
-graph TB
-    subgraph WAN["G36 router 192.168.0.1"]
-        EXT[Internet]
-    end
-    EXT -->|port forwards| C
-    subgraph Dankclaw[192.168.0.143]
-        C[Caddy :80 :443]
-        D[Pi-hole :53]
-        C -->|comfyui:8188| F[ComfyUI]
-        C -->|192.168.0.143:3000| G[Gitea quadlet]
-        C -->|192.168.0.143:8081| SE[SearXNG quadlet]
-        C -->|192.168.0.143:3005| O[OpenWebUI quadlet]
-        SE -->|network_mode service| GL[Gluetun VPN]
-        GL -->|tunnel| EXT
-    end
-    subgraph Dankserver[192.168.0.235]
-        C -->|192.168.0.235:8085| R[RomM]
-        C -->|192.168.0.235:3923| P[Copyparty]
-        C -->|192.168.0.235:8083| W[Calibre-Web]
-        C -->|192.168.0.235:32400| PL[Plex host service]
-    end
-    CLIENT[LAN clients] -->|DHCP DNS| D
-```
-
-Dankserver runs no proxy of its own. Its services expose published LAN ports, and the Dankclaw Caddy routes the names to those ports.
-
-## Hostnames
-
-| Scope | Convention | Resolved by |
-|-------|------------|-------------|
-| LAN | `romm.franksdank.cloud` → 192.168.0.143 | Pi-hole (`addn-hosts`) |
-| WAN | `romm.franksdank.cloud` → 71.227.227.172 | Porkbun A record |
-
-Users type a name and nothing else. Caddy maps the name to the upstream. The split between the two resolvers removes the need for router NAT loopback. See [[LAN-DNS]] for the flow and the router hand-off.
-
-## TLS
-
-Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through the DNS-01 challenge via the Porkbun module. No inbound port 80/443 is required. Details in [[Caddy]].
-
-## Gluetun Path (SearXNG only)
-
-SearXNG attaches to the Gluetun network namespace (`network_mode: service:gluetun`). Only SearXNG shares the tunnel. Everything else on Dankclaw uses the normal LAN route. This matches KB Pattern 2 in [[Gluetun]].
-
-## Access Policy
-
-- The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state.
-- Caddy adds HTTP basic auth on the homepage and on sensitive services.
-- LAN subnet: `192.168.0.0/24`.
-
-## See Also
-
-- [[LAN-DNS]] — name resolution, Pi-hole, router hand-off
-- [[Caddy]] — vhost map, TLS, basic auth
-- [[Dankserver Stack]], [[Dankclaw Stack]]
-- [[Gluetun]]
-- Knowledge base: [[Gluetun]], [[SearXNG]]

docs/Architecture/Storage Policy.md

@@ -1,40 +0,0 @@
-# Storage Policy
-
-All persistent data uses **bind mounts**: a host directory mapped directly into the container. No named volumes. This matches the preference of this repo and the existing Gitea stack on Dankclaw.
-
-## Why Bind Mounts
-
-- The exact host location of every byte is visible and controllable.
-- Backups are a simple `tar` of known host directories.
-- The data survives container recreation and `podman-compose down`.
-- Ownership is explicit: everything runs as UID 1000 (`dank`) so host tools can read it.
-
-## Conventions
-
-| Stack | Host root |
-|-------|-----------|
-| Dankserver | `/opt/<service>/...` |
-| Dankclaw | `/opt/<service>/...` (Gitea keeps `/opt/gitea/data/...`) |
-
-## Ownership
-
-Containers run as `PUID=1000` / `PGID=1000`. Host directories must be owned by `dank`:
-
-```bash
-sudo chown -R 1000:1000 /opt/<service>
-```
-
-## Secrets
-
-Secrets never live in bind-mounted configs that get committed. They are:
-
-1. Set in `.env` files on the host (git-ignored, `.env.example` committed as template), or
-2. Injected as environment variables from the deploy script.
-
-The live Gitea database password exists only in `.env` on Dankclaw. A test enforces that it never appears in this repo.
-
-## See Also
-
-- [[Deployment Flow]]
-- [[Gitea]] — the existing bind-mount stack being migrated
-- [[Backblaze Integration]] — future backup target for these mounts

docs/Architecture/Testing Policy.md

@@ -1,23 +0,0 @@
-# Testing Policy
-
-This repository tests its configuration at defined seams. The tests run with `bats`; run them with `./tests/run.sh` before every push.
-
-## Seams Under Test
-
-| Seam | File | What it asserts |
-|------|------|-----------------|
-| Compose validity | `tests/compose_validity.bats` | YAML parses, services have images, no named volumes |
-| Env consistency | `tests/env_consistency.bats` | every `${VAR}` is defined in `.env.example`; no real secrets; the live gitea password never appears |
-| Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present |
-| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page |
-| Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available |
-| Deployment flow | `tests/deployment_flow.bats` | deploy.sh exists, targets known, SSH reachable (skips offline) |
-
-## Red-Green
-
-Tests are written first. A failing test means the seam is not satisfied; the config change that makes it pass is the only acceptable fix. No `--no-verify`, no deleting tests without approval.
-
-## See Also
-
-- [[Deployment Flow]]
-- [[index]]

docs/Dankclaw/Dankclaw Stack.md

@@ -1,61 +0,0 @@
-# Dankclaw Stack
-
-Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
-hosts the LLM inference endpoint (`frank-default-large`, `:7770`, CIRU
-ROCmFPX llama.cpp), the agent tools, the privacy search, and the
-observability stack. The llama.cpp endpoint (`frank-default-small`,
-`:7770`) runs on Dankserver (192.168.0.235).
-
-```mermaid
-graph LR
-    U[User] --> C[Caddy :80/:443]
-    C --> O[OpenWebUI]
-    C --> S[SearXNG]
-    C --> G[Gitea]
-    C --> F[ComfyUI]
-    C --> D[Grafana]
-    S --> V[Gluetun VPN]
-    O --> L[llama.cpp dankserver :7770]
-    O --> H[Hermes]
-    O --> E[CIRU llama.cpp :7770]
-```
-
-## Services
-
-| Service | Hostname | Port | Docs |
-|---------|----------|------|------|
-| SearXNG | search.dankclaw.local | 8081 | [[SearXNG]] |
-| ComfyUI | comfyui.dankclaw.local | 8188 | [[ComfyUI]] |
-| OpenWebUI | chat.dankclaw.local | 3005 | [[OpenWebUI]] |
-| Hermes | hermes.dankclaw.local | 8090 | [[Hermes]] |
-| Gitea | gitea.dankclaw.local | 3000 | [[Gitea]] |
-| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
-| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
-| Homepage | dankclaw.local | 80/443 | — |
-| frank-default-large (host service, Dankclaw) | — | 7770 | [[deployment]] |
-| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
-
-The last two rows are bare-metal systemd services, not containers. This
-repo owns their scripts, units, and knowledge base (see
-[docs/LLM/index.md](../LLM/index.md)). Current mapping (2026-08-22):
-`frank-default-large` runs the CIRU ROCmFPX llama.cpp on Dankclaw;
-`frank-default-small` runs the upstream llama.cpp on Dankserver.
-
-## Boot Persistence
-
-The Gitea trio, the SearXNG trio (gluetun, searxng, valkey), and open-webui run as quadlet units, so they start at boot. User linger is on. ComfyUI runs as a systemd unit around a bare process. The remaining services in the table are defined in `compose.yaml` and are not yet boot-persistent.
-
-Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw` and open-webui with `scripts/deploy-open-webui-quadlet.sh dankclaw`. The Gitea units install to `~/.config/containers/systemd/` (see [[Gitea]]).
-
-## Routing Rules
-
-- SearXNG outbound traffic exits through the [[Gluetun]] VPN container.
-- Every other service uses the normal LAN route.
-- The homepage lists all services with status.
-
-## See Also
-
-- [[index]] — repository index
-- [[Network Architecture]] — routing and proxy
-- [[Observability]] — Grafana, Prometheus, logs
-- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host

docs/Dankclaw/Gitea.md

@@ -1,97 +0,0 @@
-# Gitea
-
-Gitea is a self-hosted Git service with issues, pull requests, and a web UI. On Dankclaw it runs as three Quadlet-managed containers in the rootless podman session of user `dank`.
-
-**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`) — **Public domain:** `gitea.franksdank.cloud` (DNS pending) — **Docs:** https://docs.gitea.com
-
-## Containers
-
-| Container | Unit | Image | Purpose |
-|-----------|------|-------|---------|
-| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` | Database |
-| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` | Web UI, API, git (HTTP :3000, SSH :2222) |
-| `gitea-caddy` | `gitea-caddy.service` | `docker.io/library/caddy:latest` | Reverse proxy for the public domain |
-
-The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name. The automatic `podman` default network has DNS disabled. Do not run the Gitea containers on it.
-
-## Hosted repositories
-
-| Repository | Purpose |
-|------------|---------|
-| `dank/serverAppConfigs` | this repo — Podman stacks + LLM fleet |
-| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) |
-
-The LLM fleet knowledge base: [docs/LLM/index.md](../LLM/index.md)
-
-## Management
-
-Run all commands as `dank` on Dankclaw.
-
-| Action | Command |
-|--------|---------|
-| Start all | `systemctl --user start gitea-postgres gitea-server gitea-caddy` |
-| Stop all | `systemctl --user stop gitea-caddy gitea-server gitea-postgres` |
-| Restart Gitea only | `systemctl --user restart gitea-server` |
-| Status | `systemctl --user status gitea-postgres gitea-server gitea-caddy` |
-| Container logs | `podman logs -f gitea-server` |
-| Unit logs | `journalctl --user -u gitea-server -f` |
-
-The units start in dependency order:
-
-```mermaid
-graph LR
-    N[gitea-network.service] --> P[gitea-postgres.service]
-    P --> S[gitea-server.service]
-    S --> C[gitea-caddy.service]
-```
-
-## Boot persistence
-
-User-level linger is enabled for `dank` (`loginctl enable-linger dank`). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target`. The podman generator applies it at boot; no symlink is needed.
-
-The containers use the quadlet fresh-start model. A stop removes the container (`podman run --rm --replace`). The next start recreates it. The bind-mounted data survives.
-
-## Secrets
-
-The database password lives in `/opt/gitea/.env` on the host (mode 0600). The quadlet files reference it with `EnvironmentFile=`. The password never enters this repository. Placeholders live in [`dankclaw/gitea/.env.example`](../../dankclaw/gitea/.env.example).
-
-Do not change the password in place. Change the PostgreSQL role first, then update `/opt/gitea/.env` and the `[database] PASSWD` line in the app.ini (see Gotchas), then restart the stack.
-
-## Configuration files
-
-| File | Location | Repo copy |
-|------|----------|-----------|
-| Quadlet units | `~/.config/containers/systemd/gitea-{postgres,server,caddy}.container`, `gitea.network` | [`dankclaw/gitea/quadlet/`](../../dankclaw/gitea/quadlet/) |
-| Caddyfile | `/opt/gitea/data/caddy/Caddyfile` | host-only (not in repo) |
-| Secrets | `/opt/gitea/.env` (0600) | [`dankclaw/gitea/.env.example`](../../dankclaw/gitea/.env.example) |
-
-After editing a repo copy, copy it to the target, then run `systemctl --user daemon-reload` and restart the affected units.
-
-## Data volumes
-
-| Container path | Host path | Contents |
-|----------------|-----------|----------|
-| `/data` | `/opt/gitea/data/gitea-gitea` | Git repositories, app.ini, attachments |
-| `/var/lib/postgresql/data` | `/opt/gitea/data/gitea-postgres` | PostgreSQL data |
-| `/etc/caddy/Caddyfile`, `/root/.local/share/caddy` | `/opt/gitea/data/caddy/...` | Caddy config and TLS state |
-
-Back up `/opt/gitea/data/` (tar) or `pg_dump` the database. See the knowledge base page [[Gitea]].
-
-## Gotchas
-
-- **app.ini wins over env vars.** Gitea writes `/data/gitea/conf/app.ini` (host: `/opt/gitea/data/gitea-gitea/gitea/conf/app.ini`) on first start. The `[database] HOST` and `PASSWD` lines persist and override the `DATABASE_HOST`/`DATABASE_PASSWD` env vars. When you rename the database container, edit the `HOST` line.
-- **The files in the data directory are owned by host UID 100999**, not `dank` (rootless ID mapping: container uid 1000 → host 100999). Edit them inside the container with `podman exec --user 1000 gitea-server ...`.
-- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
-- **Port 3000 vs 3001.** The live host port is 3000. The full-stack [`dankclaw/compose.yaml`](../../dankclaw/compose.yaml) maps 3001. Align the two before deploying the full stack.
-- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it. Its containers collide with the quadlet units on ports 3000, 2222, 80, and 443.
-- **Rootless cannot bind ports below 1024 by default.** `gitea-caddy` publishes 80 and 443. Without a host sysctl, `rootlessport` fails with `permission denied` and the unit crash-loops under `Restart=always`. The fix is a one-time sysctl: `net.ipv4.ip_unprivileged_port_start=80`, persisted in `/etc/sysctl.d/99-rootless-ports.conf`. Applied 2026-08-18 after a 13,000-restart crash loop.
-
-## Incident: 2026-08-16 — down after unattended reboot
-
-`unattended-upgrades` rebooted Dankclaw at 15:01 on 2026-08-16. The Gitea containers ran through podman-compose with `restart: unless-stopped` and no systemd units. Rootless podman does not auto-start containers at boot without units. Gitea stayed down until a manual start at 16:02. The fix: migrate the stack to quadlet units with linger (this page). After the fix, verify on the next reboot.
-
-## See Also
-
-- [[Dankclaw Stack]]
-- [[Postgres]]
-- Knowledge base: [[Gitea]] (tools)

docs/Dankclaw/Hermes-Dashboard.md

@@ -1,37 +0,0 @@
-# Hermes-Dashboard
-
-The Hermes web dashboard is a browser UI for managing the agent. It shows config, API keys, and sessions. It is a management tool. It is not a chat window. Chat happens over Signal or the CLI. See [[Hermes]].
-
-**Host:** Dankclaw — **Port:** `9119` — **URL:** `http://192.168.0.143:9119`
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
-| Command | `hermes dashboard --host 0.0.0.0 --port 9119 --no-open --skip-build` |
-| Port | `9119` (published to the LAN) |
-| Auth | HTTP basic auth (username + scrypt password hash) |
-| Data | shares `/home/dank/hermes-data` with the gateway |
-
-## Why a separate container
-
-The gateway container runs `hermes gateway run` (the Signal platform). That entrypoint does not start the dashboard. The dashboard is a separate process. So it runs in its own compose service (`hermes-dashboard`). Both services mount the same data volume. Both restart at boot (`restart: unless-stopped`).
-
-## Authentication
-
-The dashboard refuses to bind to a non-loopback address without an auth provider. There is no unauthenticated public bind. This repo uses basic auth:
-
-- `dashboard.basic_auth.username` and `dashboard.basic_auth.password_hash` live in the git-ignored `/home/dank/hermes-data/config.yaml`.
-- The stored value is a one-way scrypt hash. The plaintext password is never written to a file or the repo.
-- The hash is generated with `plugins.dashboard_auth.basic.hash_password`.
-
-## Access
-
-Open `http://192.168.0.143:9119` in a browser on the LAN. Log in with the configured username and password.
-
-## Gotchas
-
-- The dashboard and the gateway share one data volume. A config change made in the dashboard takes effect when the gateway restarts.
-- If the shared password is reused elsewhere, treat it as exposed. Rotate it and regenerate the hash.
-- `--skip-build` uses the prebuilt web assets in the image. Do not remove it unless the assets are rebuilt.

docs/Dankclaw/Hermes.md

@@ -1,40 +0,0 @@
-# Hermes
-
-Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
-
-**Host:** Dankclaw — **Hostname:** `hermes.dankclaw.local` — **Docs:** https://hermes-agent.nousresearch.com/docs
-
-## Facts (from research, primary source)
-
-| Item | Value |
-|------|-------|
-| Image | built from `NousResearch/hermes-agent` (official Dockerfile in repo) |
-| License | MIT |
-| Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
-| Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
-| Memory | FTS5 session search, Honcho user modeling |
-| Skills | self-created, self-improving during use |
-
-## Configuration
-
-- Point Hermes at the local model endpoint (`http://localhost:7770/v1`) or an external provider through its provider config.
-- The gateway process exposes one port for chat integration; the exact port is set in the Hermes config (`8090` convention in this repo).
-- The data volume persists skills, memory, and session history.
-
-## Volumes
-
-| Container path | Host path (convention) |
-|----------------|------------------------|
-| `/data` | `/opt/hermes/data` |
-
-## Gotchas
-
-- Hermes is under active development; pin the image tag for reproducibility.
-- Model choice drives quality; the local `frank-default-large` (Dankclaw, 7770) is the intended backend.
-- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
-
-## See Also
-
-- [[Dankclaw Stack]]
-- [[OpenWebUI]] — the chat UI that pairs with Hermes
-- Knowledge base: [[SearXNG]] (Hermes as the search agent)

docs/Dankserver/ARR-Stack.md

@@ -1,369 +0,0 @@
-# *arr Stack
-
-The *arr suite is a collection of automated media management tools for Usenet and BitTorrent. For Dankserver — which runs Copyparty, Calibre-Web, RomM, and Jellyfin — the *arr stack provides automatic downloading and library management that Copyparty alone does not cover: quality profile enforcement, series/movie discovery, metadata enrichment, and automated file organization.
-
-The core chain is: **Prowlarr** (indexer API aggregator) → **Sonarr/Radarr/Lidarr/Readarr** (media-specific managers) → **qBittorrent** (download client) → **Jellyfin** (media server scan → auto-refresh).
-
-## Service Overview
-
-| Service | Role | Image | Port | Min RAM |
-|---------|------|-------|------|---------|
-| Prowlarr | Indexer management | `lscr.io/linuxserver/prowlarr:latest` | 9696 | 256 MB |
-| Radarr | Movie management | `lscr.io/linuxserver/radarr:latest` | 7878 | 512 MB |
-| Sonarr | TV show management | `lscr.io/linuxserver/sonarr:latest` | 8989 | 512 MB |
-| Lidarr | Music management | `lscr.io/linuxserver/lidarr:latest` | 8686 | 256 MB |
-| Readarr | Ebook/audiobook management | `lscr.io/linuxserver/readarr:latest` | 8787 | 256 MB |
-| qBittorrent | Torrent download client | `lscr.io/linuxserver/qbittorrent:latest` | 8080 | 512 MB |
-| Jackett | Indexer proxy (optional) | `lscr.io/linuxserver/jackett:latest` | 9117 | 256 MB |
-| Whisparr | NSFW movie management (optional) | `lscr.io/linuxserver/whisparr:latest` | 6969 | 256 MB |
-
-**Minimum total RAM:** ~3 GB for full stack (headroom recommended).
-
-## Architecture
-
-```mermaid
-graph LR
-    I[Indexers<br/>Torrents, Usenet] --> P[Prowlarr<br/>Indexer Manager]
-    P --> S[Sonarr<br/>TV]
-    P --> R[Radarr<br/>Movies]
-    P --> L[Lidarr<br/>Music]
-    P --> Rd[Readarr<br/>Books]
-    S --> Q[qBittorrent]
-    R --> Q
-    L --> Q
-    Rd --> Q
-    Q --> M1[/tv/]
-    Q --> M2[/movies/]
-    Q --> M3[/music/]
-    Q --> M4[/books/]
-    M1 --> J[Jellyfin]
-    M2 --> J
-    M3 --> J
-    M4 --> CW[Calibre-Web]
-```
-
-## Docker Compose
-
-```yaml
-networks:
-  appnet:
-    driver: bridge
-
-services:
-  prowlarr:
-    image: lscr.io/linuxserver/prowlarr:latest
-    container_name: prowlarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/prowlarr/config:/config
-      - /mnt/media/downloads:/downloads:ro
-    ports:
-      - "9696:9696"
-    networks: [appnet]
-
-  radarr:
-    image: lscr.io/linuxserver/radarr:latest
-    container_name: radarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/radarr/config:/config
-      - /mnt/media/movies:/movies
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "7878:7878"
-    networks: [appnet]
-
-  sonarr:
-    image: lscr.io/linuxserver/sonarr:latest
-    container_name: sonarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/sonarr/config:/config
-      - /mnt/media/tv:/tv
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "8989:8989"
-    networks: [appnet]
-
-  lidarr:
-    image: lscr.io/linuxserver/lidarr:latest
-    container_name: lidarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/lidarr/config:/config
-      - /mnt/media/music:/music
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "8686:8686"
-    networks: [appnet]
-
-  readarr:
-    image: lscr.io/linuxserver/readarr:latest
-    container_name: readarr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/readarr/config:/config
-      - /mnt/media/books:/books
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "8787:8787"
-    networks: [appnet]
-
-  qbittorrent:
-    image: lscr.io/linuxserver/qbittorrent:latest
-    container_name: qbittorrent
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/qbittorrent/config:/config
-      - /mnt/media/downloads:/downloads
-      - /mnt/media/downloads/incomplete:/downloads/incomplete
-    ports:
-      - "8080:8080"
-    networks: [appnet]
-
-  # Optional: Jackett for niche indexers
-  jackett:
-    image: lscr.io/linuxserver/jackett:latest
-    container_name: jackett
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/jackett/config:/config
-      - /opt/jackett/torrences:/torrences
-    ports:
-      - "9117:9117"
-    networks: [appnet]
-
-  # Optional: Whisparr for adult content
-  whisparr:
-    image: lscr.io/linuxserver/whisparr:latest
-    container_name: whisparr
-    restart: unless-stopped
-    environment:
-      - PUID=1000
-      - PGID=1000
-      - TZ=America/New_York
-      - UMASK_SET=022
-    volumes:
-      - /opt/whisparr/config:/config
-      - /mnt/media/whisparr-movies:/whisparr-movies
-      - /mnt/media/downloads:/downloads
-    ports:
-      - "6969:6969"
-    networks: [appnet]
-```
-
-## Caddy Configuration
-
-```caddy
-radarr.dankserver.local {
-    reverse_proxy localhost:7878
-}
-
-sonarr.dankserver.local {
-    reverse_proxy localhost:8989
-}
-
-lidarr.dankserver.local {
-    reverse_proxy localhost:8686
-}
-
-readarr.dankserver.local {
-    reverse_proxy localhost:8787
-}
-
-prowlarr.dankserver.local {
-    reverse_proxy localhost:9696
-}
-
-qbittorrent.dankserver.local {
-    reverse_proxy localhost:8080
-}
-
-# Optional: Jackett
-jackett.dankserver.local {
-    reverse_proxy localhost:9117
-}
-
-# Optional: Whisparr
-whisparr.dankserver.local {
-    reverse_proxy localhost:6969
-}
-```
-
-## Configuration Steps
-
-### Phase 1: Download Client
-
-1. Deploy qBittorrent first (or Transmission).
-2. In the qBittorrent Web UI, set **No auth mode** for LAN (Settings → Web UI).
-3. Create download categories: `movies`, `tv`, `music`, `books`.
-4. Change the default Web UI password.
-
-### Phase 2: Indexer Management
-
-1. Deploy Prowlarr.
-2. Add indexers (3–5 recommended for testing). Use the **Test** button on each.
-3. Add qBittorrent as a download client in Prowlarr.
-4. Verify search from the Prowlarr web UI.
-
-### Phase 3: Arr Apps (one at a time)
-
-1. Deploy Sonarr first (TV shows are the most common use case).
-2. Add Prowlarr as the indexer source (Settings → Indexers → Add Prowlarr).
-3. Add qBittorrent as the download client with categories.
-4. Add a test series. Wait for the episode to download. Verify in `/tv/`.
-5. Repeat for Radarr, Lidarr, Readarr.
-
-### Phase 4: Media Server Integration
-
-1. Configure Jellyfin libraries pointing to `/movies`, `/tv`, `/music`, `/books`.
-2. Set **On Download** notifications in each arr app → Jellyfin webhook URL.
-3. Test end-to-end: Add a movie in Radarr → download → verify in Jellyfin.
-
-## Quality Profiles
-
-Each arr app uses a hierarchical quality system:
-
-1. **Quality Profile** — Defines tiers (e.g., 2160p → 1080p BluRay → 720p).
-2. **Cutoff** — The quality target. Downloads at or above cutoff are accepted.
-3. **Custom Formats** — Additional filters (e.g., "Has HDR", "Is Director's Cut").
-4. **Search on Grab** — When content becomes available, all indexers are queried.
-
-### Radarr (Movies)
-
-```
-Profile: Movies 4K
-  Tiers:
-    1. 2160p BluRay REMUX
-    2. 2160p BluRay
-    3. 2160p WEB-DL
-    4. 1080p BluRay
-    5. 1080p WEB-DL
-    6. 720p
-  Cutoff: 1080p BluRay
-```
-
-### Sonarr (TV)
-
-```
-Profile: TV - Standard
-  Tiers: 2160p → 1080p → 720p
-  Min Size: 500 MB
-  Max Size: 50 GB
-  Cutoff: 1080p
-  Language Profile: English audio preferred
-```
-
-### Lidarr (Music)
-
-```
-Profile: Music - Standard
-  Tiers: Lossless (FLAC) → 320 kbps → 192 kbps
-  Cutoff: 320 kbps (or Lossless)
-```
-
-## Metadata Sources
-
-| App | Primary Source | Secondary |
-|-----|---------------|-----------|
-| Radarr | TMDB | TheMovieDB API |
-| Sonarr | TMDB | TVDB |
-| Lidarr | MusicBrainz | — |
-| Readarr | Goodreads, StoryGraph | — |
-| Whisparr | XH | TMDB |
-
-## *arr Stack vs Copyparty
-
-| Feature | Copyparty | *arr Stack |
-|---------|-----------|------------|
-| Auto-download from torrents | Basic (RSS + scripts) | Native, integrated |
-| Quality profiles | No | Multi-tier, custom formats |
-| Series/movie tracking | No | Watch lists, air dates |
-| Manual library management | Yes | Yes, metadata-driven |
-| Multi-user watchlists | No | Yes |
-| Complexity | Low | Medium |
-| Resource usage | Minimal | Moderate |
-
-Use Copyparty for curated file collections. Use *arr for automatic downloads. Use both together for maximum functionality — point *arr download directories at Copyparty's watched folders.
-
-## Disk Layout
-
-```
-/mnt/media/
-├── movies/         → Radarr root
-├── tv/             → Sonarr root
-├── music/          → Lidarr root
-├── books/          → Readarr root
-├── whisparr-movies/ → Whisparr root
-├── downloads/      → qBittorrent download dir
-│   ├── movies/
-│   ├── tv/
-│   ├── music/
-│   └── books/
-└── incomplete/     → qBittorrent incomplete downloads
-```
-
-## Gotchas
-
-- **Timezone mismatch:** All containers should use the same TZ. Sonarr's scheduler is timezone-sensitive.
-- **File permissions:** Set `umask=022` in qBittorrent. Ensure arr apps can read/move downloaded files.
-- **Network conflicts:** *arr apps use many ports. Check for conflicts with existing Dankserver services.
-- **Prowlarr API key rotation:** If rotated, update it in every arr app.
-- **qBittorrent Web UI password:** Change default on first run.
-- **Readarr stability:** Readarr is still in development. Do not rely on it for mission-critical collections.
-- **Whisparr separate Jellyfin library:** Keep adult content separate from regular media.
-- **RSS feed in qBittorrent:** Can auto-add torrents matching a pattern. Useful for podcast-like series.
-
-## Best Practices
-
-1. Use tags to route content to specific download clients or categories.
-2. Enable the Blocklist in arr apps to prevent re-downloading failed torrents.
-3. Set download limits in qBittorrent to avoid bandwidth saturation.
-4. Back up arr configs regularly (they are small SQLite DBs + JSON).
-5. Monitor Prowlarr indexer health — broken indexers hide new content.
-6. Start with default quality profiles. Add custom formats gradually.
-7. Back up arr configs: `./radarr/config`, `./sonarr/config`, etc.
-
-## See Also
-
-- [[Dankserver Stack]]
-- [[Jellyfin]] — media server that consumes *arr downloads
-- [[Copyparty]] — alternative file viewer
-- [[Calibre-Web]] — manual ebook management (Readarr auto-downloads into the same library)
-- [Sonarr Wiki](https://wiki.servarr.com/sonarr)
-- [Radarr Wiki](https://wiki.servarr.com/radarr)

docs/Dankserver/Copyparty.md

@@ -1,106 +0,0 @@
-# Copyparty
-
-Copyparty is a lightweight file server and media streamer that runs as a single Python binary. It handles large payloads, media streaming, resumable uploads, directory browsing, and WebDAV from one process.
-
-**Host:** Dankserver — **Hostname:** `files.dankserver.local` — **Docs:** https://github.com/9001/copyparty
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Latest version | v1.20.21 (2026-08) |
-| Image | `ghcr.io/9001/copyparty:latest` (official Dockerfile in repo) |
-| Port | 3923 (host) → 3923 (container) |
-| Volume | bind mount the shared root directory |
-| Auth | Per-directory `.cpy` config files, HTTP basic auth, `dirkeys`, `filekeys` |
-| Streaming | HTTP range requests, HLS, WebDAV, WOPI (Office editing) |
-| Docker editions | `dj` (document + media), `iv` (image + video), `v` (video only) |
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Description |
-|----------|-------------|
-| `COPAY_PASS` | Master password for the web UI |
-| `COPAY_MODE` | Set to `rw` for read-write shares |
-
-### Configuration File Options
-
-Copyparty reads its configuration from `initcfg` (or `config` in the `/data` directory). Key options:
-
-- `use-bwrap: n` — Disable bwrap sandboxing for ffmpeg (default in Docker). Set to `y` to enable.
-- `shadowing //NULL` — Full unmap of a volume from the filesystem.
-- Config env-var expansion: Use `${VAR}` syntax in `[global]` section for environment variable substitution.
-
-### Docker Editions
-
-Different editions bundle different libraries:
-
-| Edition | Bundled Libraries | Use Case |
-|---------|-------------------|----------|
-| `dj` | Document + image + video libraries | Full-featured server |
-| `iv` | Image + video libraries | Image and video focused |
-| `v` | Video only | Transcoding only, minimal footprint |
-
-The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum compatibility.
-
-## Volumes
-
-| Container path | Host path (convention) |
-|----------------|------------------------|
-| `/data` | `/opt/copyparty/data` |
-
-## Docker Compose
-
-```yaml
-services:
-  copyparty:
-    image: ghcr.io/9001/copyparty:latest
-    container_name: copyparty
-    restart: unless-stopped
-    environment:
-      COPAY_PASS: "${COPAY_PASS}"
-      COPAY_MODE: rw
-    ports:
-      - "3923:3923"
-    volumes:
-      - /opt/copyparty/data:/data
-    healthcheck:
-      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3923/')"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-```
-
-## Healthcheck
-
-A simple HTTP check on port 3923 works. The root path returns HTTP 200 with the directory listing.
-
-## WOPI Integration
-
-Copyparty v1.20.9+ supports WOPI (Web Application Open Platform Interface) for editing Office documents in the web UI via Collabora Online. Enable with the `--wopi-accs` flag to limit WOPI access to specific paths.
-
-## Security Advisories
-
-| Version | CVE | Issue |
-|---------|-----|-------|
-| v1.20.19 | GHSA-phv8-wgjp-g4p9 | FTP upload outside volume |
-| v1.20.17 | GHSA-x5pq-m9p8-f4vx | dirkey/filekey confusion |
-| v1.20.9 | CVE-2026-27948 / GHSA-62cr-6wp5-q43h | XSS via file upload |
-
-Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`.
-
-## Gotchas
-
-- **Large transfers:** Disable TLS inside the app and let Caddy terminate TLS. This avoids double encryption overhead.
-- **Dedicated share directory:** Do not give Copyparty the whole disk. Mount only the intended share path.
-- **Upload tokens scoped per folder:** Prevents accidental global write access.
-- **CSP nonces (v1.20.17+):** Security hardening for JavaScript execution. May break third-party JS plugins (`--js-browser` / `--html-head`).
-- **Custom thumbnail extractors (v1.20.19+):** New `thumbex` framework. Requires configuration in `initcfg`.
-
-## See Also
-
-- [[Dankserver Stack]]
-- [[Caddy]] — TLS termination and authentication in front
-- [[ARR-Stack]] — Copyparty as a viewer alongside automated download management

docs/Dankserver/Dankserver Stack.md

@@ -1,55 +0,0 @@
-# Dankserver Stack
-
-Dankserver (192.168.0.235) runs the media and library stack. Every service is reachable by hostname through the Caddy reverse proxy on the host.
-
-```mermaid
-graph LR
-    U[User] --> C[Caddy :80/:443]
-    C --> R[RomM]
-    C --> P[Copyparty]
-    C --> W[Calibre-Web]
-    C --> J[Jellyfin]
-    C --> S[Sonarr]
-    C --> D[Radarr]
-    C --> L[Lidarr]
-    C --> A[Readarr]
-    C --> O[Prowlarr]
-    W --> L2[Calibre library]
-```
-
-## Services
-
-| Service | Hostname | Port | Docs |
-|---------|----------|------|------|
-| RomM | romm.dankserver.local | 8085 | [[RomM]] |
-| Copyparty | files.dankserver.local | 3923 | [[Copyparty]] |
-| Calibre-Web | books.dankserver.local | 8083 | [[Calibre-Web]] |
-| Jellyfin | jellyfin.dankserver.local | 8096 | [[Jellyfin]] |
-| Sonarr | sonarr.dankserver.local | 8989 | [[ARR-Stack]] |
-| Radarr | radarr.dankserver.local | 7878 | [[ARR-Stack]] |
-| Lidarr | lidarr.dankserver.local | 8686 | [[ARR-Stack]] |
-| Readarr | readarr.dankserver.local | 8787 | [[ARR-Stack]] |
-| Prowlarr | prowlarr.dankserver.local | 9696 | [[ARR-Stack]] |
-| qBittorrent | qbittorrent.dankserver.local | 8080 | [[ARR-Stack]] |
-
-## Optional Services
-
-These services are included in the compose file but may be disabled.
-
-| Service | Hostname | Port | Docs |
-|---------|----------|------|------|
-| Jackett | jackett.dankserver.local | 9117 | [[ARR-Stack]] |
-| Whisparr | whisparr.dankserver.local | 6969 | [[ARR-Stack]] |
-
-## Removed Services
-
-The following services ran on Dankserver and are now retired from this stack:
-
-- OpenWebUI — moved to Dankclaw (see [[OpenWebUI]])
-- llama-swap — retired; no replacement
-- SearXNG — moved to Dankclaw (see [[SearXNG]])
-
-## See Also
-
-- [[index]] — repository index
-- [[Network Architecture]] — routing and proxy

docs/Dankserver/Jellyfin.md

@@ -1,173 +0,0 @@
-# Jellyfin
-
-Jellyfin is a free, open-source media server and suite of applications for organizing, managing, and streaming personal media collections (movies, TV shows, music, photos) to networked devices. It is a community fork of Emby, written in C# (.NET), and uses FFmpeg for transcoding.
-
-**Host:** Dankserver — **Hostname:** `jellyfin.dankserver.local` — **Docs:** https://jellyfin.org
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Latest version | Check https://jellyfin.org/downloads |
-| Image | `jellyfin/jellyfin:latest` (Docker Hub) or `ghcr.io/jellyfin/jellyfin` |
-| Base | Debian-based |
-| Primary port | 8096 (HTTP, web UI + API) |
-| HTTPS port | 8920 (optional) |
-| Client discovery | 7359/udp |
-| DLNA | 1900/udp (optional) |
-| GPU passthrough | `/dev/dri` (Intel QSV / AMD VA-API) or NVIDIA NVENC |
-| License | MIT |
-
-## Architecture
-
-```mermaid
-graph LR
-    U[User] --> C[Caddy :80/:443]
-    C --> J[Jellyfin :8096]
-    J --> Lib[/media/movies]
-    J --> Lib2[/media/shows]
-    J --> Lib3[/media/music]
-    J --> Conf[Jellyfin config]
-    J --> Cache[Jellyfin cache]
-```
-
-## Docker Compose
-
-```yaml
-services:
-  jellyfin:
-    image: jellyfin/jellyfin:latest
-    container_name: jellyfin
-    restart: unless-stopped
-    user: "1000:1000"
-    ports:
-      - "8096:8096/tcp"
-      - "8920:8920/tcp"
-      - "7359:7359/udp"
-      - "1900:1900/udp"
-    environment:
-      - JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
-      - DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
-    volumes:
-      - /opt/jellyfin/config:/config
-      - /opt/jellyfin/cache:/cache
-      - /mnt/media/movies:/data/movies:ro
-      - /mnt/media/shows:/data/shows:ro
-    devices:
-      - /dev/dri:/dev/dri
-    group_add:
-      - "992"    # Host render group GID — verify with `getent group render`
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-```
-
-## Caddy Configuration
-
-```caddy
-jellyfin.dankserver.local {
-    reverse_proxy 127.0.0.1:8096
-
-    transport http {
-        versions h2c 1.1
-    }
-}
-```
-
-Set **Base URL** to `/` (or the path you configure) in Jellyfin Dashboard → Networking. Add `127.0.0.1` to **Known Proxies**.
-
-## Storage
-
-| Mount | Purpose | Size Estimate |
-|-------|---------|---------------|
-| `/opt/jellyfin/config` | SQLite database, settings, cached images | 10–100 GB |
-| `/opt/jellyfin/cache` | Transcoding segments, extracted posters | Self-cleaning per session |
-
-The database must be on local storage (SSD/NVMe preferred). Media libraries can be on NFS/mergerfs. If media is on NFS, set `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`.
-
-## Authentication
-
-- **Built-in auth (default):** Per-user libraries, parental controls, max bitrate limits. Manage via Dashboard → Users.
-- **Caddy basic auth:** Single login layer. Does not integrate with per-user Jellyfin permissions.
-- **LDAP/OIDC:** Available via plugins (LDAP-Auth, Jellyfin SSO). Integrates with Authentik, Keycloak, etc.
-
-For Dankserver, built-in auth is recommended. Use Caddy basic auth on top for external access.
-
-## Metadata Scraping
-
-| Provider | Content | Type |
-|----------|---------|------|
-| TheMovieDb (TMDb) | Movies, TV shows, posters, fan art, overviews | Default |
-| OMDb API | Movies only, English metadata | Default |
-| TheTVDB | TV shows, episodes | Plugin |
-| fanart.tv | Extra artwork, banners, thumbnails | Plugin |
-| AniDB | Anime metadata | Plugin |
-| MusicBrainz | Music metadata | Plugin |
-| Open Subtitles | Subtitle downloads | Plugin |
-| Intro Skipper | TV show intro detection | Plugin |
-
-## Clients
-
-| Platform | App | Quality |
-|----------|-----|---------|
-| Web | Jellyfin Web | Excellent |
-| Android | Jellyfin (Google Play / F-Droid) | Excellent |
-| Android TV / Fire TV | Jellyfin | Good |
-| iOS / iPadOS | Swiftfin (third-party) | Good |
-| Desktop | Jellyfin Media Player (mpv-based) | Excellent |
-| Roku | Jellyfin (Channel Store) | Good |
-| Kodi | Jellyfin addon | Excellent |
-| Samsung/LG Smart TVs | TV browser | Fair |
-
-## Jellyfin vs Plex
-
-| Feature | Jellyfin | Plex |
-|---------|----------|------|
-| Cost | Free, no tiers | Free core; Plex Pass ($5/mo or $120 lifetime) |
-| License | MIT | Proprietary |
-| Account required | No | Yes |
-| Hardware transcoding | Free | Plex Pass required |
-| Mobile sync | Free | Plex Pass |
-| Live TV / DVR | Free | Plex Pass |
-| Client apps | Good, improving | Excellent, widest device support |
-
-Run Jellyfin alongside Plex for an incremental migration. Point Jellyfin at the same media folders as Plex. Watch history and metadata will not transfer cleanly.
-
-## Resource Requirements
-
-- **Idle / Direct Play:** ~200–500 MB RAM, <5% CPU. Negligible.
-- **1080p transcode (CPU):** 40–60% CPU on a modern multi-core CPU.
-- **1080p transcode (with QSV):** 3–5% CPU.
-- **4K HDR transcode (with QSV):** 5–15% CPU.
-- **Minimum host RAM:** 4 GB. 8 GB comfortable for transcoding.
-
-## Deployment Checklist
-
-1. Verify GPU on host: `ls -l /dev/dri/`, `lspci | grep -i vga`, `getent group render`.
-2. Install media drivers: `sudo apt install intel-media-va-driver-non-free vainfo intel-gpu-tools`.
-3. Create directories: `/opt/jellyfin/config`, `/opt/jellyfin/cache`.
-4. Write compose file, verify device and GID.
-5. Deploy: `podman compose up -d`.
-6. Test GPU passthrough: `podman exec jellyfin /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128`.
-7. Configure Caddy reverse proxy. Set Known Proxies in Jellyfin.
-8. Add media libraries in Dashboard.
-9. Install plugins: Open Subtitles, fanart.tv, Intro Skipper.
-10. Test playback on Android/Chrome.
-
-## Gotchas
-
-- **GPU passthrough silently failing:** `/dev/dri` must be passed and the correct render node used. Verify with `vainfo` inside the container.
-- **Wrong render group GID:** Common GID is 992, but varies per host. Run `getent group render` to find it.
-- **Subtitle burn-in triggers full transcode:** PGS (image-based) subtitles force full video transcoding. SRT/ASS (text-based) play fine.
-- **Database on NFS:** SQLite on network storage causes slow startup and corruption risk. Keep `/config` on local storage.
-- **Client bitrate limit too low:** Many clients default to 20 Mbps. Set Max Streaming Bitrate to Original or Maximum.
-- **Naming convention mismatches:** Non-standard folder/file names cause missing metadata. Use `Movie Title (Year)` for movies and `Show Name/Season 01/Show Name S01E01.mkv` for TV.
-
-## See Also
-
-- [[Dankserver Stack]]
-- [[ARR-Stack]] — automated download into Jellyfin libraries
-- [[Copyparty]] — alternative file viewer for existing collections
-- Plex — existing media server on this host

docs/Dankserver/lidarr.md

@@ -1,53 +0,0 @@
-# Lidarr
-
-Lidarr manages music albums in the *arr stack on Dankserver. It automatically searches, downloads, and organizes albums from indexers into the Jellyfin music library.
-
-**Host:** Dankserver — **Hostname:** `lidarr.dankserver.local` — **Docs:** https://wiki.servarr.com/lidarr
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `lscr.io/linuxserver/lidarr:latest` |
-| Port | 8686 (host) → 8686 (container) |
-| Min RAM | 256 MB |
-| Metadata | MusicBrainz |
-
-## Docker Compose
-
-```yaml
-lidarr:
-  image: lscr.io/linuxserver/lidarr:latest
-  container_name: lidarr
-  restart: unless-stopped
-  environment:
-    - PUID=1000
-    - PGID=1000
-    - TZ=America/New_York
-    - UMASK_SET=022
-  volumes:
-    - /opt/lidarr/config:/config
-    - /mnt/media/music:/music
-    - /mnt/media/downloads:/downloads
-  ports:
-    - "8686:8686"
-  networks: [appnet]
-```
-
-## Configuration
-
-1. Add Prowlarr as the indexer source (prefer torrent-based indexers for FLAC).
-2. Add qBittorrent as the download client with the `music` category.
-3. Set quality profile (Lossless → High Bitrate → Standard).
-4. Set root folder to `/music`.
-5. Add artists and verify downloads.
-
-## Gotchas
-
-- Album variants: Lidarr supports multiple releases of the same album (vinyl vs CD). Use "Preferred" and "Cutoff" quality profiles.
-- Music videos can be optionally downloaded separately from albums.
-
-## See Also
-
-- [[ARR-Stack]] — full *arr stack documentation
-- [[Jellyfin]] — media server that consumes Lidarr downloads

docs/Dankserver/mariadb.md

@@ -1,47 +0,0 @@
-# MariaDB (RomM Backend)
-
-MariaDB 11 provides the database backend for RomM on Dankserver. RomM 5.2.0 requires MariaDB (or PostgreSQL) for its data store.
-
-**Host:** Dankserver — **Docs:** https://mariadb.com/kb/en/docker-mariadb-information/
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `docker.io/mariadb:11` |
-| Port | 3306 (internal, within `appnet`) |
-| Database name | `romm` |
-| Data volume | `/opt/romm/mariadb` |
-
-## Docker Compose
-
-```yaml
-mariadb:
-  image: docker.io/mariadb:11
-  container_name: romm-mariadb
-  restart: unless-stopped
-  environment:
-    MARIADB_ROOT_PASSWORD: "${MARIADB_ROOT_PASSWORD}"
-    MARIADB_DATABASE: romm
-    MARIADB_USER: "${ROMM_DB_USER}"
-    MARIADB_PASSWORD: "${ROMM_DB_PASSWD}"
-  volumes:
-    - /opt/romm/mariadb:/var/lib/mysql
-  healthcheck:
-    test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
-    interval: 10s
-    timeout: 5s
-    retries: 3
-```
-
-## Backup
-
-Back up the volume directory periodically:
-
-```bash
-sudo rsync -a /opt/romm/mariadb /mnt/backup/romm-mariadb/
-```
-
-## See Also
-
-- [[RomM]] — service that uses this database

docs/Dankserver/prowlarr.md

@@ -1,55 +0,0 @@
-# Prowlarr
-
-Prowlarr is the indexer management layer for the *arr stack. It aggregates indexers (torrent and Usenet) and exposes a unified API that Sonarr, Radarr, Lidarr, and Readarr consume.
-
-**Host:** Dankserver — **Hostname:** `prowlarr.dankserver.local` — **Docs:** https://wiki.servarr.com/prowlarr
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `lscr.io/linuxserver/prowlarr:latest` |
-| Port | 9696 (host) → 9696 (container) |
-| Min RAM | 256 MB |
-| Auth | API key (Settings → General) |
-
-## Configuration
-
-1. Add indexers: Settings → Indexers → Add.
-2. Test each indexer with the **Test** button.
-3. Assign tags (e.g., `movies`, `tv`, `music`) to indexers for routing.
-4. Add download client: Settings → Download Clients → Add qBittorrent.
-
-## Docker Compose
-
-```yaml
-prowlarr:
-  image: lscr.io/linuxserver/prowlarr:latest
-  container_name: prowlarr
-  restart: unless-stopped
-  environment:
-    - PUID=1000
-    - PGID=1000
-    - TZ=America/New_York
-    - UMASK_SET=022
-  volumes:
-    - /opt/prowlarr/config:/config
-    - /mnt/media/downloads:/downloads:ro
-  ports:
-    - "9696:9696"
-  networks: [appnet]
-```
-
-## Gotchas
-
-- Prowlarr database grows with indexer data. Set up periodic vacuum/backup.
-- Indexer sync has a 15–60 minute delay. Search results may lag.
-- Rotate API key: update it in every arr app.
-
-## See Also
-
-- [[ARR-Stack]] — full *arr stack documentation
-- [[radarr]]
-- [[sonarr]]
-- [[lidarr]]
-- [[readarr]]

docs/Dankserver/qbittorrent.md

@@ -1,52 +0,0 @@
-# qBittorrent
-
-qBittorrent is the torrent download client for the *arr stack on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr, and Readarr via category tagging.
-
-**Host:** Dankserver — **Hostname:** `qbittorrent.dankserver.local` — **Docs:** https://www.qbittorrent.org
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `lscr.io/linuxserver/qbittorrent:latest` |
-| Port | 8080 (host) → 8080 (container) |
-| Min RAM | 512 MB |
-| Auth | Web UI username/password; LAN mode available |
-
-## Configuration
-
-1. Change the default Web UI password (Settings → Web UI).
-2. Enable "No auth mode for LAN" for arr app connectivity (Settings → Web UI → uncheck "HTTP authentication required for clients").
-3. Create download categories: `movies`, `tv`, `music`, `books`.
-4. Set max active downloads: Settings → Downloads (recommended: 5 global, 3 per category).
-
-## Docker Compose
-
-```yaml
-qbittorrent:
-  image: lscr.io/linuxserver/qbittorrent:latest
-  container_name: qbittorrent
-  restart: unless-stopped
-  environment:
-    - PUID=1000
-    - PGID=1000
-    - TZ=America/New_York
-    - UMASK_SET=022
-  volumes:
-    - /opt/qbittorrent/config:/config
-    - /mnt/media/downloads:/downloads
-    - /mnt/media/downloads/incomplete:/downloads/incomplete
-  ports:
-    - "8080:8080"
-  networks: [appnet]
-```
-
-## Gotchas
-
-- qBittorrent Web UI password changes on first run (check `config/qBittorrent.conf`).
-- Ensure umask is set to `022` so downloaded files are readable by arr apps.
-- Monitor bandwidth limits to avoid saturating the network.
-
-## See Also
-
-- [[ARR-Stack]] — full *arr stack documentation

docs/Dankserver/radarr.md

@@ -1,54 +0,0 @@
-# Radarr
-
-Radarr manages movies in the *arr stack on Dankserver. It automatically searches, downloads, and organizes movies from indexers into the Jellyfin media library.
-
-**Host:** Dankserver — **Hostname:** `radarr.dankserver.local` — **Docs:** https://wiki.servarr.com/radarr
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `lscr.io/linuxserver/radarr:latest` (v6 stable) |
-| Port | 7878 (host) → 7878 (container) |
-| Min RAM | 512 MB |
-| Metadata | TMDB / TheMovieDB API |
-
-## Docker Compose
-
-```yaml
-radarr:
-  image: lscr.io/linuxserver/radarr:latest
-  container_name: radarr
-  restart: unless-stopped
-  environment:
-    - PUID=1000
-    - PGID=1000
-    - TZ=America/New_York
-    - UMASK_SET=022
-  volumes:
-    - /opt/radarr/config:/config
-    - /mnt/media/movies:/movies
-    - /mnt/media/downloads:/downloads
-  ports:
-    - "7878:7878"
-  networks: [appnet]
-```
-
-## Configuration
-
-1. Add Prowlarr as the indexer source.
-2. Add qBittorrent as the download client with the `movies` category.
-3. Set quality profile and cutoff quality.
-4. Set root folder to `/movies`.
-5. Add movies and verify downloads.
-
-## Gotchas
-
-- Radarr v6 is the current stable (v6.3.x as of 2026-08). `:latest` points to the current release.
-- Movie renaming produces `Title (Year)` format. Jellyfin scanner handles this.
-- File permissions: qBittorrent downloads as PUID; arr apps must read/move files.
-
-## See Also
-
-- [[ARR-Stack]] — full *arr stack documentation
-- [[Jellyfin]] — media server that consumes Radarr downloads

docs/Dankserver/readarr.md

@@ -1,57 +0,0 @@
-# Readarr
-
-> **RETIRED (May 2026).** The Servarr Team retired Readarr: its metadata became unusable and the community migration to Open Library stalled. There is no upstream development. Alternatives: the Bookshelf community fork, or the Calibre content server. See [Readarr (Retired)](https://wiki.servarr.com/readarr).
-
-Readarr manages ebooks and audiobooks in the *arr stack on Dankserver. It automatically searches, downloads, and organizes books into the Calibre library.
-
-**Host:** Dankserver — **Hostname:** `readarr.dankserver.local` — **Docs:** https://wiki.servarr.com/readarr
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `lscr.io/linuxserver/readarr:latest` (v0.2.x development) |
-| Port | 8787 (host) → 8787 (container) |
-| Min RAM | 256 MB |
-| Metadata | Goodreads / StoryGraph |
-
-## Docker Compose
-
-```yaml
-readarr:
-  image: lscr.io/linuxserver/readarr:latest
-  container_name: readarr
-  restart: unless-stopped
-  environment:
-    - PUID=1000
-    - PGID=1000
-    - TZ=America/New_York
-    - UMASK_SET=022
-  volumes:
-    - /opt/readarr/config:/config
-    - /mnt/media/books:/books
-    - /mnt/media/downloads:/downloads
-  ports:
-    - "8787:8787"
-  networks: [appnet]
-```
-
-## Configuration
-
-1. Add Prowlarr as the indexer source (torrent indexers with book content).
-2. Add qBittorrent as the download client with the `books` category.
-3. Set author profile (format preference: epub, mobi, etc.).
-4. Set quality profile for audiobooks (320 kbps MP3, 128 kbps, lossless).
-5. Set root folder to `/books`.
-6. Add authors and verify downloads.
-
-## Gotchas
-
-- Readarr is in development (~v0.2.x). Some features may be incomplete.
-- Calibre integration is folder-based, not API-based. Readarr drops files into `/books` which Calibre-Web serves.
-- Not all indexers have all formats (epub, mobi, pdf). Goodreads/StoryGraph metadata helps match editions.
-
-## See Also
-
-- [[ARR-Stack]] — full *arr stack documentation
-- [[Calibre-Web]] — ebook web frontend that serves Readarr downloads

docs/Dankserver/redis.md

@@ -1,40 +0,0 @@
-# Redis (RomM Cache / Queue)
-
-Redis 7 provides the caching and task queue backend for RomM on Dankserver. RomM 5.2.0 requires Redis (or Valkey) for its task queue.
-
-**Host:** Dankserver — **Docs:** https://redis.io/docs/
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `docker.io/redis:7-alpine` |
-| Port | 6379 (internal, within `appnet`) |
-| Data volume | `/opt/romm/redis` |
-| Auth | None (local-only, no password needed) |
-
-## Docker Compose
-
-```yaml
-redis:
-  image: docker.io/redis:7-alpine
-  container_name: romm-redis
-  restart: unless-stopped
-  volumes:
-    - /opt/romm/redis:/data
-  healthcheck:
-    test: ["CMD", "redis-cli", "ping"]
-    interval: 10s
-    timeout: 5s
-    retries: 3
-```
-
-## Backup
-
-```bash
-sudo rsync -a /opt/romm/redis /mnt/backup/romm-redis/
-```
-
-## See Also
-
-- [[RomM]] — service that uses this cache

docs/Dankserver/sonarr.md

@@ -1,55 +0,0 @@
-# Sonarr
-
-Sonarr manages TV shows in the *arr stack on Dankserver. It automatically searches, downloads, and organizes episodes from indexers into the Jellyfin media library.
-
-**Host:** Dankserver — **Hostname:** `sonarr.dankserver.local` — **Docs:** https://wiki.servarr.com/sonarr
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `lscr.io/linuxserver/sonarr:latest` (v4 stable) |
-| Port | 8989 (host) → 8989 (container) |
-| Min RAM | 512 MB |
-| Metadata | TMDB / TVDB |
-
-## Docker Compose
-
-```yaml
-sonarr:
-  image: lscr.io/linuxserver/sonarr:latest
-  container_name: sonarr
-  restart: unless-stopped
-  environment:
-    - PUID=1000
-    - PGID=1000
-    - TZ=America/New_York
-    - UMASK_SET=022
-  volumes:
-    - /opt/sonarr/config:/config
-    - /mnt/media/tv:/tv
-    - /mnt/media/downloads:/downloads
-  ports:
-    - "8989:8989"
-  networks: [appnet]
-```
-
-## Configuration
-
-1. Add Prowlarr as the indexer source.
-2. Add qBittorrent as the download client with the `tv` category.
-3. Set quality profile, cutoff, and minimum/maximum episode sizes.
-4. Set root folder to `/tv`.
-5. Add series and verify downloads.
-
-## Gotchas
-
-- Sonarr v4 is the current stable. The v3 branch uses the `:v3` tag.
-- Anime has different quality profiles and naming conventions than Standard series.
-- Missing episodes: set "Unmonitor Missing Episodes" to auto-unmonitor old episodes.
-- Timezone-sensitive scheduler: ensure all containers use the same TZ.
-
-## See Also
-
-- [[ARR-Stack]] — full *arr stack documentation
-- [[Jellyfin]] — media server that consumes Sonarr downloads

docs/Infrastructure/Alloy.md

@@ -1,47 +0,0 @@
-# Alloy
-
-Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). One Alloy instance runs per host and forwards container and journal logs to Loki, and optionally metrics to Prometheus.
-
-**Hosts:** Dankclaw + Dankserver (one per host) — **Docs:** https://grafana.com/docs/alloy/latest
-
-## Facts (from research)
-
-| Item | Value |
-|------|-------|
-| Image | `franksdank/alloy:latest` (local build with a static busybox healthcheck) |
-| Port | none published (push-only) |
-| Config | River syntax, `config.alloy` |
-| Successor to | Promtail (EOL 2026-03-02) |
-| Inputs | journald socket, container log files |
-| Output | Loki push endpoint |
-
-## Why Alloy and not Promtail
-
-Promtail reached end-of-life on 2026-03-02. Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
-
-## Configuration
-
-River config at `shared/alloy/config.alloy`:
-
-- Read journald logs from the host's `/run/systemd/journal/socket`.
-- Read container logs from `/var/lib/containers/...` (podman) or the docker path.
-- Forward to `http://loki:3100/loki/api/v1/push` with `host` and `job` labels.
-
-## Volumes
-
-| Container path | Host path (convention) |
-|----------------|------------------------|
-| `/etc/alloy/config.alloy` | `shared/alloy/config.alloy` (bind mount) |
-| `/run/systemd/journal/socket` | host journald socket (read-only) |
-| `/var/log` | host logs (read-only) |
-
-## Gotchas
-
-- The journald socket mount is required for journal scraping.
-- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
-- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
-
-## See Also
-
-- [[Loki]] — the log backend
-- [[Observability]] — the full stack

docs/Infrastructure/Authentik.md

@@ -1,152 +0,0 @@
-# Authentik — the homelab's SSO / OIDC provider
-
-Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
-homelab. It fronts the Caddy vhosts that do not carry their own
-application-level authentication. Apps that have their own login (Grafana,
-Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped in
-forward_auth — they handle their own session.
-
-- **Version:** `2026.8.0` (pinned in `dankclaw/compose.yaml`)
-- **Host:** dankclaw (`192.168.0.143`)
-- **Name:** `auth.franksdank.cloud`
-- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
-  `authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)
-
-## Why Authentik
-
-The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
-`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
-for all configuration, a native Caddy `forward_auth` outpost, and a footprint
-that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10,
-so the stack is 3 services, not 4.
-
-## Architecture
-
-```mermaid
-graph LR
-    subgraph Browser
-        U[User]
-    end
-    subgraph Caddy [Caddy - auth gate]
-        FA[forward_auth]
-    end
-    subgraph AK [Authentik]
-        SRV[server :9000]
-        WKR[worker]
-        PG[(postgres)]
-        OUT[outpost /auth/caddy]
-    end
-    subgraph App [Target app]
-        A1[SearXNG / Hermes / Prometheus<br/>RomM / Copyparty / homepage]
-    end
-
-    U -->|1. GET /| FA
-    FA -->|2. GET /outpost.../auth/caddy| OUT
-    OUT -->|3. no session: 302 login| U
-    U -->|4. login at auth.franksdank.cloud| SRV
-    SRV --> PG
-    WKR --> PG
-    SRV -->|5. 2xx: session valid| FA
-    FA -->|6. proxy with headers| A1
-```
-
-1. The browser hits a protected vhost (e.g. `search.franksdank.cloud`).
-2. Caddy `forward_auth` clones the request to the Authentik outpost
-   (`/outpost.goauthentik.io/auth/caddy`).
-3. With no valid session, the outpost returns a `302` to the Authentik login
-   page (`auth.franksdank.cloud`).
-4. The user logs in at Authentik (email + password, TOTP/WebAuthn MFA, or an
-   upstream provider).
-5. On the next request the outpost returns `2xx`; Caddy copies the
-   `X-Authentik-*` headers and proxies to the app.
-
-## The three compose services
-
-| Service | Role | Image |
-|---|---|---|
-| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
-| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.0` |
-| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.0` |
-
-- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
-  moved to PostgreSQL in 2025.10 (Redis removed).
-- **Server** and **worker** run the same image; the `command` selects the role
-  (`server` / `worker`).
-- The worker runs as `0:0` (rootless: container root maps to the host user that
-  owns the data dir).
-- Data persists under `/home/dank/authentik/` (bind mounts, repo policy — no
-  named volumes).
-
-## Caddy integration
-
-The Caddyfile wraps the no-auth vhosts in a `route` block. The pattern (one
-example; the others are identical):
-
-```caddyfile
-search.franksdank.cloud {
-	route {
-		# 1. The outpost path is proxied to Authentik (login UI + session).
-		reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
-		# 2. forward_auth checks the session before serving the app.
-		forward_auth authentik-server:9000 {
-			uri /outpost.goauthentik.io/auth/caddy
-			copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
-			trusted_proxies private_ranges
-		}
-		# 3. The app.
-		reverse_proxy 192.168.0.143:8081
-	}
-}
-```
-
-Vhosts protected by forward_auth: the homepage (`franksdank.cloud`,
-`www.franksdank.cloud`), SearXNG, Hermes, Prometheus, RomM, Copyparty. The
-`auth.franksdank.cloud` vhost proxies to the server directly (no forward_auth —
-it *is* the login).
-
-## Configuration (secrets)
-
-Two secrets live in the host `.env` (git-ignored; placeholders in
-`.env.example`):
-
-- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
-- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
-
-## First-time setup
-
-1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
-   health-gated), reloads Caddy.
-2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
-   creates the superuser (email + password) and generates the DB schema.
-3. In the admin interface: **Directory → Applications** → add a **Proxy**
-   application (or an **OAuth2/OpenID Connect** provider) for each vhost you
-   protect. The Caddy outpost reads the application's policy.
-4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
-   Connect** provider, create the app with the correct redirect URI, and paste
-   the `client_id` / `client_secret` into the app's SSO settings.
-
-## Operations
-
-- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
-  probes this).
-- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
-- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
-  truth). The `/home/dank/authentik/` dir holds media + static files.
-- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run
-  `./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup.
-
-## Failure modes
-
-| Symptom | Cause | Fix |
-|---|---|---|
-| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
-| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
-| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
-| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
-
-## Cross-references
-
-- [Caddy](Caddy.md) — the reverse proxy that carries the forward_auth directives
-- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
-- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
-- Research: `.docaudit/research/auth.md` — the full provider comparison

docs/Infrastructure/Grafana.md

@@ -1,46 +0,0 @@
-# Grafana
-
-Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, with basicauth).
-
-**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs
-
-## Facts
-
-| Item | Value |
-|------|-------|
-| Image | `docker.io/grafana/grafana:latest` |
-| Port | 3002 (host) → 3000 (container) |
-| Data volume | dashboards, users, datasource config |
-| Datasource | Prometheus (on the same host) |
-| Provisioning | dashboards in `/etc/grafana/provisioning/dashboards` |
-
-## Configuration
-
-```yaml
-environment:
-  GF_SECURITY_ADMIN_USER: "${GRAFANA_ADMIN_USER}"
-  GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
-  GF_SERVER_ROOT_URL: "https://grafana.franksdank.cloud"
-volumes:
-  - /opt/grafana/data:/var/lib/grafana
-  - /opt/grafana/provisioning:/etc/grafana/provisioning:ro
-```
-
-## Datasource
-
-Point Grafana at the Prometheus service by name (`http://prometheus:9090`) on the shared podman network.
-
-## Healthcheck
-
-Grafana responds on `/api/health` with 200 and `"database":"ok"`.
-
-## Gotchas
-
-- Change the default admin credentials at first boot (env or UI).
-- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
-
-## See Also
-
-- [[Dankclaw Stack]]
-- [[Prometheus]] — the datasource
-- [[Observability]] — the whole stack