serverAppConfigs/serverAppConfigs-docs/log.md

Bundle Change Log

2026-09-28

• scripts/frank-primary.env, runbook — prefix caching DEPLOYED on frank-primary (Frank's go-ahead; commit 41e9189): E8-qualified, zero VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200, completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is in dank's ROOTLESS podman store (sudo podman inspects an empty store and reports nothing); probe via the shim's docker or the unit MainPID.
• /scripts/frank-primary.env, /serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md, /serverAppConfigs-docs/index.md — frank-primary prefill/decode bench matrix (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT under page-cache churn (measured 740 s) — launch.sh gained R9V_PLE_READY_TIMEOUT forwarding (fork 2de0357, local-only) and prod now sets 1800.
• Incident — frank-primary down 00:06 → ~11:00 (11h): the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the R9V_PLE_READY_TIMEOUT=1800 fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own podman wait + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.

2026-09-27

• /scripts/frank-primary.env, /scripts/frank-primary.service, /scripts/serve-frank-primary.sh, /scripts/deploy-remote.sh, /tests/frank_primary_unit.bats, /serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md (new), AGENTS.md + LLM index — frank-primary cut over to the Q8_0 payload (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in R9V_CHAT_TEMPLATE_FILE/R9V_CHAT_TEMPLATE_KWARGS passthrough (fork commit 30d4a32, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at /home/dank/r9v-state/q8xl/setup.json + the quantized-inject dev-overlay adapter (fork 363d606) + R9V_FULL_MUTABLE_CACHE=0; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's speculative_config is a single dict, no chaining, so llama.cpp's draft-mtp,ngram-mod pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
• Dankserver frank-primary slot (experiment window) — the radiance-vllm-qwen4exp fork qualified the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: runtimes/dev-overrides/qwen4_exp.py dequantizes quantized block_inject shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and R9V_FULL_MUTABLE_CACHE=0 passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (mapping_bytes=54400261120, mapping_rss_bytes == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt; record at fork docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md.
• Dankserver frank-primary restore — uncensored UD-IQ4_XS back on 0.0.0.0:7770 as frank-primary-r9v from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); frank-primary.service (llama.cpp) runtime-masked for the window (/run/systemd/system/frank-primary.service → /dev/null — the /etc relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs restart=no, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env; scratch in dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/.

2026-09-25

• /serverAppConfigs-docs/Dankserver/Jellyfin.md — Jellyfin LDAP 401 resolved: the auth works via the modern /Users/AuthenticateByName endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy /Users/Auth/Password route, which Jellyfin 12.1 removed (the LegacyEmbyRouteRewriteMiddleware is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
• /shared/homepage/services.json, /shared/homepage/docs.html — Suggested uses added to the three LLM model descriptions (status cards + the /docs model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-small → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to /srv/homepage; bats 19/19 + non-Mac-mini e2e green (d3c8197).
• /shared/homepage/{services.json,app.js,style.css,docs.html} — Model cards reworked to a structured three-line shape: each LLM card now carries a details object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (app.js descHTML() falls back to the plain description for non-LLM cards; new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-small = fast utility model. The /docs model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
• Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik 2026.8.0→2026.8.3 (server/worker/outpost), tautulli v2.18.1-ls242→ls245, calibre-web 0.6.27-ls397→ls402, plex 1.43.3.10896→1.43.4.10903; seerr already at v3.4.1. Local-build rebuilds (forced --no-cache, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh :latest, 23986f31→e1e87aec). anubis already at latest stable v1.27.0 (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
• /dankserver/{tautulli,calibre-web,plex}/quadlet/*.container, /dankclaw/authentik/quadlet/*.container — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead dankserver/compose.yaml plex entry synced (0b1ea2e).

2026-09-22

• /serverAppConfigs-docs/** — OKF v0.1 frontmatter migration: type/description/tags/timestamp added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); timestamp added to LLM/frank-medium-batch-ubatch.md and LLM/frank-small-benchmark.md; unquoted YAML scalars (colons, *arr) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved LLM/Pipeline Parallelism/index.md; log.md restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
• /serverAppConfigs-docs/** — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in Shared Memory Architecture Dankclaw.md, deployment.md + Architecture/Deployment Flow.md rewritten to the Quadlet/deploy-plan.py truth, Calibre.md 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), Copyparty.md v1.20.24, RomM.md 5.3.0 breaking change; *arr docs moved to a new FrankBook-Pro/ section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B).
• /serverAppConfigs-docs/Research/ — Three new research briefs (OKF): Local TTS and Voice Cloning.md, AI Job Search and Resume Tools.md, Most Popular Self-Hosted AI Apps.md (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report popular-self-hosted-ai-apps.html; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + /api/v1/images/generations, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.

2026-09-21

• /dankserver/frank-medium/quadlet/frank-medium.container — frank-medium cut back to the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq); KV_DTYPE became a quadlet tunable, set auto (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
• /dankserver/calibre-server/quadlet/calibre-server.container — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, --auth-mode=basic pinned, internal port 8090.
• /serverAppConfigs-docs/Dankserver/Calibre.md — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
• /scripts/serve-frank-large.sh, /scripts/frank-large.env, /scripts/frank-large.service, /scripts/download-frank-large-unsloth.sh — frank-large (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB), 200k ctx, MTP depth 2 via explicit -md; the split parts ARE the payload — concatenating them trips invalid split file name (part 1's header advertises file_count=3); draft acceptance verified live 0.83–0.92.
• /serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md — New runbook; frank-large-ciru-orca.md marked RETIRED (rollback target); AGENTS.md + LLM index updated.
• /shared/prometheus/prometheus.yml — node-dankclaw target fixed to host.containers.internal:9100 (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); Infrastructure/Prometheus.md targets table + tests/observability_quadlet.bats contract updated.

2026-09-20

• /dankserver/frank-medium/quadlet/frank-medium.container — Checkpoint swapped MXFP4 → FP8 (orcarouter/Qwen3.8-27B-Uncensored-FP8, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via scripts/hf-manifest.py + scripts/hf-fetch.sh + scripts/swap-frank-medium.sh; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
• /dankserver/frank-medium/quadlet/frank-medium.container — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to scripts/.AGENTTRASH/20260920-llama-layersplit/ (never systemctl disable); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
• /scripts/build-llama-rdna4.sh — Now takes GGML_HIP_RCCL; the RCCL build (build-rdna4-rocm714-rccl) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — invalid device function).
• /shared/prometheus/rules/llm-fleet.yml, /shared/prometheus/prometheus.yml, /shared/grafana/provisioning/ — Monitoring spans the vllm: + llamacpp: namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to host.containers.internal; the prefix-cache reuse rule divides by cached + prompt (llama.cpp's prompt_tokens_total EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
• /scripts/dankserver-route-primary.sh — eno1 pinned primary (metric 100), eno2 retained failover (metric 200); applied with nmcli device reapply (no connection reapply in NM 1.54).
• Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge AtomicOpsCap: Routing+, both endpoints ReqEn+ — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
• /serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); frank-medium-llamacpp-dflash2.md carries the SUPERSEDED banner; Dual-R9700-Cooperation.md states the measured verdict; Dankserver System Specs.md gained a Motherboard section; frank-medium-model-swap.md is the new swap runbook; AGENTS.md topology/sections/lessons flipped.

2026-09-19

• /scripts/frank-large.service, /scripts/serve-frank-large.sh — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
• /serverAppConfigs-docs/LLM/frank-large-ciru-orca.md — New runbook.
• /scripts/frank-medium.env, /scripts/serve-frank-medium.sh — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
• /scripts/frank-small.env, /scripts/serve-frank-small.sh — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
• Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
• AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.

2026-09-18

• /scripts/frank-medium.env — CACHE_RAM 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; tests/frank_medium_unit.bats seam added; --ctx-checkpoints 0 confirmed a deliberate deviation (build default is 32) (1acca43).
• Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned llama-server from third_party/llamacpp/build-merge (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.

2026-09-17

• /scripts/frank-small.env, /scripts/serve-frank-small.sh — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M + separate MTP assistant sidecar (--spec-type draft-mtp,ngram-mod); q8_0 KV (build-cuda rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON, else ~12× prefill collapse); --fit chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every reasoning_effort level.
• /scripts/frank-small.env — CTX_SIZE=1048576 / PARALLEL=4 (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
• /serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md — New ACTIVE runbook; frank-small-qwen38-gsq-rco.md retired as rollback target.
• /shared/homepage/ — The three LLM cards moved to the user page with live descriptions; new static /docs user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
• /scripts/deploy-remote.sh — The pending pass now prunes stale pending state entries for retired services into ~/.AGENTTRASH and keeps going (a stale frank-default-sightsound entry made every --pending run exit 2) (02a7876).
• Fleet audit — Every vhost classified against the guard policy (no new exposure); systemd-hermes logged as a boot-persistence gap (running container, no quadlet); stray stopped test-vllm container on dankserver held for approval.

2026-09-16

• /scripts/bench-batch-ubatch.sh — New harness: one llama-bench invocation per cell (a test that cannot allocate exit(1)s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (LimitMEMLOCK=infinity is load-bearing) (87845fc, 77a07ff).
• /scripts/frank-medium.env — UBATCH_SIZE 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and TENSOR_SPLIT 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); tests/frank_medium_unit.bats balance seam re-fitted as a shape assertion.
• /serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
• /dankclaw/frank-large/quadlet/frank-large.container — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
• /scripts/frank-small.env, /scripts/frank-small.service, /scripts/serve-frank-small.sh — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (draft-mtp), BF16 projector, 131072 ctx, f16 KV, port 7771.
• /scripts/deploy-remote.sh — deploy_llm_large flipped back to halogen with a wait_port_free 7770 gate (the first cutover died on Address already in use two seconds after the old server exited); deploy_llm_small added.
• /serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md, /serverAppConfigs-docs/LLM/halogen-flash-server.md — New / re-activated runbooks; frank-large-ik-uncensored.md marked retired.
• /serverAppConfigs-docs/LLM/x4-pair-experiment.md — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by -ot itself (has_tensor_overrides() → pipeline_parallel → n_copies == 1); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (-dev A,B is a SWEEP, not a list — correct -dev A/B -ts x/y gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
• Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every podman command failed, all 28 quadlets looped, podman system migrate panics in 5.7.0; a reboot repaired it. The reboot exposed hermes.franksdank.cloud 502: the Caddyfile dialed 192.168.0.143:8787 (a host bind, not a published port) — fixed to host.containers.internal:8787, and the seam that had accepted any numeric upstream was strengthened.
• Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; halogen-flash-server.md gained a "Refreshing the weights" note.

2026-09-15

• /scripts/frank-medium.env, /scripts/serve-frank-medium.sh, /scripts/frank-medium.service — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (retire_root_llm_small() archived the CUDA-era unit).
• /serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md — New runbook; freetoken-frank-small.md and kingjones-q8agent-rocmfpx.md marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
• Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the ld.so.conf hijack (staged in ~/.AGENTTRASH/20260915-rocm633-cleanup/).
• /scripts/frank-large.service, /scripts/serve-frank-large.sh — frank-large cut to ik_llama.cpp (halogen quadlet retired to ~/.AGENTTRASH/20260915-ik-swap/); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
• /serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md — New runbook with the full evidence chain.

2026-09-14

• /dankclaw/compose.yaml, /dankclaw/.env.example — New hermes-webui service (Hermex): bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data, loopback-only publish 127.0.0.1:8787, password-gated via HERMES_WEBUI_PASSWORD.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories.
• /tests/hermes_webui.bats — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
• /serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md — New page; remote access = Termius local forward over the existing router :22 port forward.

2026-09-12

• /dankclaw/marreta/quadlet/ — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; NGINX_HTTP_PORT=8085 (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
• /tests/marreta_quadlet.bats — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
• /serverAppConfigs-docs/Dankclaw/Marreta.md — New page (incl. the base64:-prefixed APP_KEY gotcha).
• Live — authentik ProxyProvider marreta (pk 13) + Application link: the outpost's config API filters application__isnull=False, so a provider with no linked Application is never loaded (the login-wall root cause); a null invalidation_flow crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec 16-marreta.spec.ts.

2026-09-04

• /frankbook-pro/ — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, /opt config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed 10.88.0.1:8080 → 127.0.0.1:8080.
• /tests/frankbook_quadlet.bats — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
• /dankserver/ — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned /opt/<svc>/config dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and userns_mode must NEVER be added (podman-compose runs services in pods; --userns + --pod is rejected).
• /dankclaw/Caddyfile, /dankclaw/pihole/addn-hosts — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
• /tests/e2e/ — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).

2026-09-03

• /serverAppConfigs-docs/ — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, Prowlarr.md/prowlarr.md deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.

2026-09-01

• /frankbook-pro/gluetun/, /frankbook-pro/qbittorrent/ — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (WebUI\Password_PBKDF2, not the legacy WebUI\Password key qB ≥ 4.2 ignores); DefaultSavePath → /Volumes/COMPLETE/Finished Torrents; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
• /serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
• /dankclaw/authentik/quadlet/authentik-outpost-ldap.container — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via ak shell ORM; live bind tests proved two rules: the token intent must be verification, and the provider needs the search_full_directory object permission.
• /serverAppConfigs-docs/Infrastructure/Authentik.md, /serverAppConfigs-docs/Dankclaw/Jellyfin.md — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (LdapSearchAttributes is comma-separated; username attribute cn).
• /scripts/frank-small.env, /scripts/serve-frank-small.sh — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in _dequant_fp8_weight); reverted to Huihui FP8 within the hour; freetoken-frank-small.md quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
• /dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/ — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move /opt/<svc> → /home/dank/Containers/<svc>); v1 generator quirks fixed (Command=→Exec=, Device=→AddDevice=, HealthCmd= space-splitting, the HOME=/root EnvironmentFile trap).
• /dankclaw/ — Data roots of gitea, authentik, and the hermes trios moved to /home/dank/Containers/<svc>/ while running (rename is inode-bound — zero downtime); deploy_hermes ordering fixed (generated units reload AFTER the unit files are copied).
• /dankclaw/seerr/quadlet/seerr.container — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider seerr-proxy; /tests/seerr_quadlet.bats new seam; /serverAppConfigs-docs/Dankclaw/Seerr.md new page.
• /scripts/deploy-remote.sh — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + HostHeaderValidation=false + the four categories; full 7-service deploy run.
• /frankbook-pro/qbittorrent/quadlet/qbittorrent.container — qB pinned 4.6.7 (5.x's 204 login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
• /serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md, /serverAppConfigs-docs/Architecture/Storage Policy.md, /serverAppConfigs-docs/Architecture/Deployment Flow.md — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.

2026-08-30

• /scripts/frank-npu.env, /scripts/serve-frank-npu.sh, /scripts/frank-npu.service — New frank-npu on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (/dev/accel/accel0 err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with iommu=pt + reboot; vision + audio towers proven end-to-end; health probe is /v1/version (FLM has no /health route) (c3ed3a6, cf0c541).
• /serverAppConfigs-docs/LLM/frank-npu.md — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
• /dankclaw/frank-large/quadlet/frank-large.container — frank-large: swapped to kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the pinned kingjones30/ROCmFPX fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
• /scripts/frank-small.env — frank-small (FreeToken): model swapped to batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (native block-fp8); ~40 tok/s verified.
• /serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md — Reactivated as the large runbook; agentionai-fast-imatrix.md marked RETIRED; freetoken-frank-small.md model sections updated; LLM index + AGENTS fleet tables refreshed.
• /serverAppConfigs-docs/Dankserver/prowlarr.md, /serverAppConfigs-docs/Dankserver/qbittorrent.md — Rewritten: the arr↔Prowlarr feed contract (/{indexerId}/api, the global key — the hand-rolled /torznab/ paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (WebUI\Password_PBKDF2, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).

2026-08-29

• /dankclaw/Caddyfile — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
• /dankclaw/anubis/policy.json — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile).
• /serverAppConfigs-docs/Infrastructure/Authentik.md, /serverAppConfigs-docs/Infrastructure/Anubis.md, /serverAppConfigs-docs/Architecture/Network Architecture.md — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original Host vs each provider's external_host (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null authorization_flow/invalidation_flow starve the outpost's config fetch; the public zone gained the auth A record via the Porkbun API (ee1e038).

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
• /dankclaw/gitea/quadlet/gitea-caddy.container — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
• Incident — Post-reboot: Caddy (a bare compose container — restart: unless-stopped does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: net.ipv4.ip_unprivileged_port_start reset to 80 by a later-sorted sysctl drop-in AND a stale line in /etc/sysctl.conf, which systemd-sysctl applies AFTER all of sysctl.d/.
• /AGENTS.md — New MANDATORY "Boot Persistence & Self-Healing" section.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (joins dankclaw_default, Pull=never, env from /home/dank/Containers/caddy/env); removed from compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — Repo source of truth for ip_unprivileged_port_start=53; deploy_pihole stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies sysctl --system (the boot code path).
• /tests/caddy_quadlet.bats (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
• /serverAppConfigs-docs/Infrastructure/Caddy.md, /serverAppConfigs-docs/Infrastructure/Pi-hole.md, /serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md, /serverAppConfigs-docs/Dankclaw/Gitea.md, /serverAppConfigs-docs/Architecture/Deployment Flow.md — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.