serverAppConfigs/serverAppConfigs-docs/log.md

Bundle Change Log

2026-09-29

• frank-large tuning investigation: a measured NEGATIVE result, nothing changed. Frank asked for iterative decode/prefill improvement at a 32k working point without touching the model or quantization. No win exists at gufo's served settings, so no config change was shipped. frank-large was stopped for the whole investigation (~4h) per the benchmarking policy and restored healthy in 9s afterwards.
  • Measured on the repo's own prose and shell sources at 32,762 prompt tokens, salted per run so a warm prefix cache could not flatter a later run, 3–5 repetitions per configuration.
  • --prefill-chunk is dead: 512 → 8192 moves prefill 410.7–412.4 t/s. A 16x change in chunk size, 0.4% out.
  • --draft-tokens is flat: 5 → 10 gives 13.79–15.86 t/s decode, all inside the run-to-run noise band. The 5-rep confirmation round reversed the 3-rep leaders — dt9 + chunk4096 looked best and came last (12.86), while the deployed config came first (14.71).
  • --draft-policy fixed vs adaptive: no effect. --min-draft-tokens: rejected by the engine ("DFlash2 requires ... 1"). MTP: unsupported for Q8 targets (Unsupported Qwen MTP tensor type for token_embd.weight: Q8_0).
  • --context 262144 → 40960 was tested as a diagnostic (the deployed value preallocates KV ~8x larger than a 32k conversation needs): no gain (13.93 vs 14.71 t/s), and it would break models.yml's declared contextWindow: 262144. Not shipped.
  • What actually moves the number is speculation itself: 6.65 t/s off → ~15 t/s on, 5.3x — already enabled. Prefill is insensitive to everything tried, consistent with gufo's own EXPERIMENTS.md where dozens of kernel variants are recorded rejected for "no release throughput improvement".
  • Two traps are now in AGENTS.md's benchmarking policy, because they cost most of the wall time and both produced false results: (1) gufo bench cannot rank speculative settings and inverted the ranking — its corpus reports acceptance=0.933 +/- 0.000 with an identical trace sha256 across draft widths (the same highly-predictable text every run, 93% of proposals accepted). It reported 29–37 t/s where real traffic gives ~15, and made dt16 look 6% faster than dt7 when real text made it 14% slower. (2) A live A/B needs >=128 generated tokens; at 8, inter_token is first-token noise and the first pass of the sweep produced a nonsense ordering that the 128-token pass reversed.
  • Also worth recording: the Server-Timing header is ttft;dur=402.485 — the unit is in the name half, so splitting on = keys the dict ttft;dur and every lookup silently returns nothing. Four configurations reported "no Server-Timing" while the header was present on all of them.
• dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s, mechanism measured. The ad hoc qwen-fp8 container (tcclaviger RDNA4 fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full flag-level A/B battery at the 32k working point; every number is from the server's usage counters (SSE chunks are invalid under spec decode). New runbook: dankserver-qwen38-fp8-expert-offload-tuning.
  • Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1, TUNABLE off — 26.80/25.15/26.81 tok/s at 166/1087/4064 ctx, acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
  • Ruled out by measurement: gather geometry 16x32 (−9%) and 32x64 (parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral); prefetch depth 2 (neutral — the tier is 16 slots/layer against a ~2,400-slab working set at 73% misses); 2-way concurrency (summed 25.49 < single 26.8 — under TP both streams' misses gather on each rank, the links do not add capacity, and the shared arena thrashes).
  • TunableOp rejected: it silently skips under expert offload unless CLAV_TUNABLEOP_WITH_OFFLOAD=1; once forced on it corrupts numerics — repetition loops, the server's repetition-halt firing at ~140 of 256 tokens, "79.8 tok/s" measured on halted degenerate completions, and a fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no persistent cache in the container lifecycle.
  • Infeasible: UTIL=0.99 (free-memory wall: 31.29 of 31.86 GiB free at worker init); --max-num-seqs 2 (compile constraint violation at CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not prepared" — PLE prep is wired only for the MTP spec path).
  • The wall, measured: the link does Gen5 x8 = 31.5 GB/s/direction; the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91% of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s = 42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait (peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the fork's gather read path — fork-patch territory, not flags.
  • Telemetry fork bugs: the torch-profiler worker stop hangs on ROCm (>10 min, two attempts, while the AsyncLLM wrapper logs success in 1 s), and the default torch_profiler_with_stack=true grew a 6.5-min profile window's trace buffer ~40 GB → the kernel OOM-killed the engine core (anon-rss 95.2 GB, oom-kill in journalctl -k). The per-layer expert-cache hit rates (R4D_LRU_TELEMETRY=1) are reachable ONLY through that stop path, so the telemetry JSON is unreachable without a fork patch.
  • 256k ctx boots (the last original-goal dimension): a 262,144-token window needs 4.0 GiB of KV (hybrid attention + mamba pages ≈ 16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused (4.0 GiB needed > 2.98 available; the engine's own max-len estimate at that pool: 184,224). At KV 4.5 GiB it serves 36.65/34.21/34.77 (arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just under 35). KV-exact-4.0 arm in flight for the last +3.5% slots. Correction recorded: the planner sizes the arena AFTER the KV override (the earlier "before" reading was wrong) — the arena is what remains.

2026-09-28

• Dankclaw model cleanup (disk was at 93%, 137 G free). Frank asked to clean up old models. 510 G staged to ~/.AGENTTRASH/20260928-retired-models/ — moved, NOT deleted, with a MANIFEST.md giving a restore command per item. Nothing is freed until the purge is approved; that is the point of the trash policy, and the number to know before expecting the disk to move.
  • Staged: CIRU Orca 128 G, halogen 123 G, unsloth Flash-Next IQ4_XS 91 G, mradermacher IQ4_XS 91 G, unsloth 27B 29.3 G, Kairic Edge 27 G, ISTA-DASLab GSQ-RCO 13 G, vLLM Gemma4 w4a16 9.6 G, vLLM Gemma4 drafter 838 M. Every one maps to a runtime already retired in the runbook index.
  • The 29.3 G rollback target went with them, by Frank's decision. The payload-only rollback is now a re-fetch first; both the runbook and AGENTS.md say so. The DFlash2 drafter was NOT reclaimed — only the .gguf was taken out of that directory, because the live frank-large drafter lives in it. The manifest warns against restoring the directory wholesale.
  • Kept deliberately: checkpoints/ 23 G. It is unreferenced by the repo, but ComfyUI is live and its workflows were not searched — not confirmed dead, so not taken. logs/x4e-run.log also stays: it is a sparse file, 3.3 G apparent but 4 K real, so removing it frees nothing.
  • A live reference check came first, and it earned its keep. serve-frank-small.sh's fallback MODEL_DIR still pointed at the retired GSQ-RCO payload frank-small served on 2026-09-16/17. The env file has always overridden it, so the drift was invisible — but a missing or unreadable env file would have turned a config problem into a "missing asset" naming a model that no longer exists. Repointed at Gemma4-12B-QAT/, which is what the env file already sets. Every other hit was a comment, a retired fetch script's destination, or retire_halogen_large_quadlet (which moves a quadlet, not a model).
  • Not taken, by Frank's decision — scope was models only: podman images (279 G reclaimable, 95%; 170 images, 23 active) and the uv+pip caches (73 G). Together that is 352 G of regenerable garbage larger than the models reclaimed, and it is the obvious next move.
  • Pre-existing: 118 G of trash from 2026-09-21 was already staged (20260921-frank-large-assembled 88 G, -redownload 30 G) and never purged. Total now staged on this host: 627 G.
  • Verified after the moves: frank-large and frank-small both active, both /health 200, and all three live assets (Swift target, DFlash2 drafter, Gemma4 target) still present.
• frank-large payload swapped to Swift-1.5 (uncensored) (production change, Dankclaw). Frank asked whether the UD Q8 of ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF would load on gufo, then said to do it. Port 7770, alias frank-large, one 262,144 session, the DFlash2 drafter and the client contract are unchanged; the 27B dense weights are an abliterated (uncensored) fine-tune of the same base, under the non-OSI SWIFT OPEN LICENSE 1.0.
  • Compatibility was established from the GGUF headers, not from the filename. Both files were range-fetched and parsed: same general.architecture (qwen35), same 866 tensors with identical names in identical order, zero per-tensor ggml_type or dim mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at blk.64, same geometry. The two scary metadata differences are both no-ops, confirmed against gufo's source: general.file_type (15 vs 7) is "deliberately not consulted" by gguf_reader.cpp — the file gufo already served records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra qwen35.attention.recurrent_layers key is never read by gufo anywhere in src/; layer kinds come from full_attention_interval, which both files carry as 4 and which agrees with it.
  • Fetched and verified: 31,457,990,272 bytes, sha256 9076f99a… matching the Hub LFS oid, via the new /scripts/download-frank-large-swift.sh. It sends no auth header — the repo is public and ungated, and a token on the curl command line is world-readable via /proc/<pid>/cmdline. The anonymous transfer ran at ~1.8 MiB/s; resuming with the host's token through a mode-600 curl -K config took it to ~7 MiB/s. The download survived an ssh drop by running setsid nohup, not under the session.
  • Load test, service stopped first (the GPU policy): 9.2 s from load_started to load_completed, 66.5 GiB of the 116.7 GiB iGPU at context capacity 262,144, no loader warnings. Contract battery green: context_length 262144, content + reasoning_content, finish_reason: tool_calls, SSE, and a two-turn tool loop with prior reasoning preserved. Spot-checks: "17*23" → 391; asked for a lockpicking scene it wrote one without hedging, confirming the abliteration.
  • THE TEMPLATE LAW IS GONE, by decision. Frank asked to "use our chat template"; gufo cannot take it. This is not a missing flag — src/models/qwen/chat_template.cpp is 587 lines of hand-written C++ that appends strings, with no Jinja engine, no --chat-template-file, no env override and no template body in the binary. Qwen-Sharp v22.4.0 is 453 lines of Jinja2 with namespace(), macros and raise_exception, so supplying it is a port, not a config line. Frank chose to ship on gufo's stock renderer and accept the loss: the terse system-prompt injection, in-band <|think_*|> control, the json|xml tool_call_format switch, tool-argument truncation and add_vision_id prefixes are all gone. gufo's renderer does keep the effort instruction strings and preserve_thinking. Worth watching: an abliterated model now runs without the prompt that tells it to stop waffling — the cheap fix is to send the terse text as an ordinary system message from models.yml, not to touch a template.
  • The payload is now two files in two directories. The target moved repo; the DFlash2 drafter did not move, so DRAFTER_DIR is its own tunable. That broke the deployer's stray-run sweep, which keyed on the target filename and would have silently stopped matching — a stray run would then have held 7770 past the port gate. It now keys on the drafter, which every frank-large command line carries via --dflash-model and which did not move.
  • /tests/frank_large_unit.bats — 20 assertions. The mutation pass found four real gaps on the first run, all now fixed: two fetch assertions were satisfied by a header comment rather than the URL assignment, so repointing a repo left them green; and nothing bound the env's MODEL_FILE/DRAFTER_FILE to the filenames the fetch scripts actually download, so a re-upload or an env edit would break the endpoint with a "missing asset" no test predicted. There is now an explicit cross-file invariant. A second harness bug worth recording: a bats --filter that matches no test reports zero not ok, which looks exactly like a mutation that survived — the harness now asserts the 1..N count before believing a result. 16/16 mutations caught.
  • Docs: runbook gained a Payload section (the header evidence), The template law section and a two-way Rollback; AGENTS.md fleet table, frank-large section and rollback.
• frank-large runtime swapped to gufo (production change, Dankclaw). Frank asked for github.com/gufo-org/gufo — a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151) — in place of the stock llama.cpp prebuilt. Port 7770, alias frank-large, one 262,144-token session and the client contract were all unchanged; only the engine was new, and the payload was the unsloth Qwen3.8-27B UD-Q8_K_XL at that point — it was swapped for the abliterated Swift-1.5 later the same day, see the entry above.
  • The runtime is built from source on the host. gufo ships no prebuilt Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host has no Nix. Pinned at v0.1.1 / b0f8673, installed to /home/dank/GitPackages/gufo-v0.1.1/ by the new /scripts/build-gufo.sh, which is reproducible and fails closed on a moved tag. Unlike the unsloth prebuilt it replaces, gufo links the host ROCm 7.2.4 through its RPATH, so the HIP dev packages must stay.
  • Two host workarounds, both encoded in the build script. Ubuntu's librocwmma-dev ships a truncated rocWMMA header set into /usr/include that shadows AMD's complete one; the build died on 'internal/accessors.hpp' file not found, and a -I cannot fix it because in HIP mode the host compilation pass searches the system directories first. The package has no reverse dependencies; it is removed. Second, Ubuntu 26.04 ships /usr/include/c++/16 and the ROCm clang prefers the newest tree, whose <atomic_wait.h> needs a std::__detail::__notify_impl symbol version the installed libstdc++.so.6 (GLIBCXX_3.4.35) does not export — the link failed until --gcc-install-dir was pinned to GCC 15.
  • gufo diagnose reports [PASS] on gfx1151 (ROCm 7.2.3, amdgpu loaded, 20 CUs). It self-caps compute queues at 2 resident for serve llm after reading /sys/class/kfd/kfd/proc/*/queues/*/type, which matters because ComfyUI holds queues on the same device.
  • Shape: one session, --context 262144, DFlash2 with the adaptive block policy capped at --draft-tokens 7 (llama.cpp's fixed depth 4 is gone), greedy, --think auto --preserve-thinking on, text-only.
  • Client-contract deltas, all in the runbook: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token-prompt failure mode cannot recur); reasoning_effort now accepts the full OpenAI set, so models.yml's reasoningEffortMap fold is no longer required — kept anyway so this commit does not also move every high role from xhigh to high effort; and there is no KV-quantization or prompt-cache-RAM flag.
  • Measured at cutover: 13 s from load_started to listening (target 8 s, drafter 2 s, session 3 s) against a 20-minute LLM_WAIT budget; RSS 30.6 GiB; a live short-prompt request returned Server-Timing: ttft;dur=549.8, inter_token;dur=24.2 (~41 tok/s). Verified live: /health 200, /ready 200, /v1/models → frank-large at context_length 262144, a chat completion carrying reasoning_content, a tool_calls response with finish_reason: tool_calls, and SSE streaming.
  • The trade, stated plainly. gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode +7 % at zero depth — but −14 % at 65,536 tokens of depth and −24 % at 131,072. On the 8-bit target the crossover is ~32k and frank-large runs one 262,144-token conversation, so long agent sessions sit on the losing side of it. The Q4_K_XL target shows no such regression; revisiting the quantization is a separate decision.
  • /tests/frank_large_unit.bats — Rewritten around behaviour: the seam runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. A caveat found while writing it: bats 1.13 ignores a failing A && B in the middle of a test (bash exempts the non-final element of an AND-list from errexit), so every such assertion was split into standalone [ ] lines. Mutations confirmed the seam bites: dead runtime gate, DRAFT_TOKENS=0, a wrong context/sessions/port, an alias swap, a moved drafter, a removed missing-asset message and a neutered commit pin each fail it.
  • /scripts/deploy-remote.sh — deploy_llm_large gates on the runtime binary as well as the payload and returns before it stops the serving unit, so a host whose gufo was never built reports the failure instead of taking the endpoint down on the way there.
  • Docs: new /serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md; frank-large-unsloth-qwen38-27b-q8xl.md marked RETIRED (still the rollback target — the prebuilt and the payload are both intact on disk); AGENTS.md fleet table, frank-large section, bench policy and runbook index; both vault indexes; Dankclaw System Specs.md; README.md.
  • e2e gate: 32/33. The one failure is 192.168.0.112 (the FrankBook-Pro Mac mini) at 100 % packet loss — offline, and the same failure recorded in 564a781. Not related to this change; no frank-large test failed.
• Frontier-MoE feasibility study on Dankserver (research + build, no production change). Frank asked which of DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box, and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run them fast. Findings, all evidenced:
  • V4.1 does not fit. 475.27 GiB checkpoint (measured from the HF API), of which 189.13 GiB is Engram in two 94.56 GiB shards. A token selects 240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a ~23 tok/s ceiling — below the 35 tok/s goal before any attention or dense work. The 1000 t/s prefill target is ~20x away: the backend refuses layer-major prefill for V4.1, so the only legal prefill is token-by-token.
  • Affinity (Codeberg, StillDeadcode) is a DeepSeek-V4-Flash engine claiming 110 tok/s on two R9700s; it builds clean here (hipcc 23, gfx1201, zero errors). The 110 works because the 101 GiB container leaves the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the quantiser, container, slab, host pool and placement engine (expert tensor naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's new indexer, the DSpark draft geometry and Engram — and Engram fails silently, because V4.1 has no num_hash_layers key so Affinity's model.h:36 default of 3 survives. Weeks of work, same capacity wall.
  • Lucebox is the only engine that runs V4.1 on gfx1201, on the unmerged feat/ds41-engine branch. Its own DS41.md measures 0.8-1.9 tok/s on an R9700 (best tuned three-tier case 9.1) with 61 GB of RAM; Dankserver's 246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is downloading for a three-arm bench.
• /serverAppConfigs-docs/LLM/Dankserver System Specs.md — RAM corrected 60 GiB -> 246 GiB (stale since the 2026-09-20 board swap; the Lucebox runbook had it right) and storage refreshed; /mnt/entropy is at 100% (244 G) and must not be planned for payload.
• /serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md — new runbook; indexed in LLM/index.md.
• LANDMINE, not changed here: /home/dank/models was cleared by Frank mid-session to free space. frank-primary is still answering :7770 from mmapped-but-unlinked inodes (/models is empty inside the running container) — the endpoint cannot be restarted until the 175 GiB payload is re-fetched. A reboot or a container restart takes frank-primary down.
• Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio (f187165 PLE 1800, 41e9189 prefix caching, ba2100e docs) was found already MERGED to main + pushed AND already live on frank-primary: the running frank-primary-r9v container (booted 12:13 from the ba2100e checkout) carries --enable-prefix-caching in argv and VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 in env, /health 200 — but deploy-state/frank-primary was still stamped f187165. A deploy.sh run would have re-selected frank-primary and reloaded the 175 GiB payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself runs on. Stamped the marker f187165 → ba2100e (status deployed, atomic mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone failure 11-admin is the FrankBook-Pro Mac mini (192.168.0.112:8989) offline (100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a regression.
• scripts/frank-primary.env, runbook — prefix caching DEPLOYED on frank-primary (Frank's go-ahead; commit 41e9189): E8-qualified, zero VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200, completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is in dank's ROOTLESS podman store (sudo podman inspects an empty store and reports nothing); probe via the shim's docker or the unit MainPID.
• /scripts/frank-primary.env, /serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md, /serverAppConfigs-docs/index.md — frank-primary prefill/decode bench matrix (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s VLLM_PLE_OFFLOAD_READY_TIMEOUT under page-cache churn (measured 740 s) — launch.sh gained R9V_PLE_READY_TIMEOUT forwarding (fork 2de0357, local-only) and prod now sets 1800.
• Incident — frank-primary down 00:06 → ~11:00 (11h): the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the R9V_PLE_READY_TIMEOUT=1800 fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own podman wait + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.

2026-09-27

• /scripts/frank-primary.env, /scripts/frank-primary.service, /scripts/serve-frank-primary.sh, /scripts/deploy-remote.sh, /tests/frank_primary_unit.bats, /serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md (new), AGENTS.md + LLM index — frank-primary cut over to the Q8_0 payload (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in R9V_CHAT_TEMPLATE_FILE/R9V_CHAT_TEMPLATE_KWARGS passthrough (fork commit 30d4a32, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at /home/dank/r9v-state/q8xl/setup.json + the quantized-inject dev-overlay adapter (fork 363d606) + R9V_FULL_MUTABLE_CACHE=0; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's speculative_config is a single dict, no chaining, so llama.cpp's draft-mtp,ngram-mod pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
• Dankserver frank-primary slot (experiment window) — the radiance-vllm-qwen4exp fork qualified the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: runtimes/dev-overrides/qwen4_exp.py dequantizes quantized block_inject shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and R9V_FULL_MUTABLE_CACHE=0 passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (mapping_bytes=54400261120, mapping_rss_bytes == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt; record at fork docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md.
• Dankserver frank-primary restore — uncensored UD-IQ4_XS back on 0.0.0.0:7770 as frank-primary-r9v from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); frank-primary.service (llama.cpp) runtime-masked for the window (/run/systemd/system/frank-primary.service → /dev/null — the /etc relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs restart=no, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env; scratch in dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/.

2026-09-25

• /serverAppConfigs-docs/Dankserver/Jellyfin.md — Jellyfin LDAP 401 resolved: the auth works via the modern /Users/AuthenticateByName endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy /Users/Auth/Password route, which Jellyfin 12.1 removed (the LegacyEmbyRouteRewriteMiddleware is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
• /shared/homepage/services.json, /shared/homepage/docs.html — Suggested uses added to the three LLM model descriptions (status cards + the /docs model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-small → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to /srv/homepage; bats 19/19 + non-Mac-mini e2e green (d3c8197).
• /shared/homepage/{services.json,app.js,style.css,docs.html} — Model cards reworked to a structured three-line shape: each LLM card now carries a details object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (app.js descHTML() falls back to the plain description for non-LLM cards; new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-small = fast utility model. The /docs model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
• Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik 2026.8.0→2026.8.3 (server/worker/outpost), tautulli v2.18.1-ls242→ls245, calibre-web 0.6.27-ls397→ls402, plex 1.43.3.10896→1.43.4.10903; seerr already at v3.4.1. Local-build rebuilds (forced --no-cache, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh :latest, 23986f31→e1e87aec). anubis already at latest stable v1.27.0 (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
• /dankserver/{tautulli,calibre-web,plex}/quadlet/*.container, /dankclaw/authentik/quadlet/*.container — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead dankserver/compose.yaml plex entry synced (0b1ea2e).

2026-09-22

• /serverAppConfigs-docs/** — OKF v0.1 frontmatter migration: type/description/tags/timestamp added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); timestamp added to LLM/frank-medium-batch-ubatch.md and LLM/frank-small-benchmark.md; unquoted YAML scalars (colons, *arr) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved LLM/Pipeline Parallelism/index.md; log.md restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
• /serverAppConfigs-docs/** — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in Shared Memory Architecture Dankclaw.md, deployment.md + Architecture/Deployment Flow.md rewritten to the Quadlet/deploy-plan.py truth, Calibre.md 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), Copyparty.md v1.20.24, RomM.md 5.3.0 breaking change; *arr docs moved to a new FrankBook-Pro/ section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B).
• /serverAppConfigs-docs/Research/ — Three new research briefs (OKF): Local TTS and Voice Cloning.md, AI Job Search and Resume Tools.md, Most Popular Self-Hosted AI Apps.md (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report popular-self-hosted-ai-apps.html; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + /api/v1/images/generations, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.

2026-09-21

• /dankserver/frank-medium/quadlet/frank-medium.container — frank-medium cut back to the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq); KV_DTYPE became a quadlet tunable, set auto (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
• /dankserver/calibre-server/quadlet/calibre-server.container — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, --auth-mode=basic pinned, internal port 8090.
• /serverAppConfigs-docs/Dankserver/Calibre.md — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
• /scripts/serve-frank-large.sh, /scripts/frank-large.env, /scripts/frank-large.service, /scripts/download-frank-large-unsloth.sh — frank-large (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB), 200k ctx, MTP depth 2 via explicit -md; the split parts ARE the payload — concatenating them trips invalid split file name (part 1's header advertises file_count=3); draft acceptance verified live 0.83–0.92.
• /serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md — New runbook; frank-large-ciru-orca.md marked RETIRED (rollback target); AGENTS.md + LLM index updated.
• /shared/prometheus/prometheus.yml — node-dankclaw target fixed to host.containers.internal:9100 (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); Infrastructure/Prometheus.md targets table + tests/observability_quadlet.bats contract updated.

2026-09-20

• /dankserver/frank-medium/quadlet/frank-medium.container — Checkpoint swapped MXFP4 → FP8 (orcarouter/Qwen3.8-27B-Uncensored-FP8, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via scripts/hf-manifest.py + scripts/hf-fetch.sh + scripts/swap-frank-medium.sh; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
• /dankserver/frank-medium/quadlet/frank-medium.container — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to scripts/.AGENTTRASH/20260920-llama-layersplit/ (never systemctl disable); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
• /scripts/build-llama-rdna4.sh — Now takes GGML_HIP_RCCL; the RCCL build (build-rdna4-rocm714-rccl) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — invalid device function).
• /shared/prometheus/rules/llm-fleet.yml, /shared/prometheus/prometheus.yml, /shared/grafana/provisioning/ — Monitoring spans the vllm: + llamacpp: namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to host.containers.internal; the prefix-cache reuse rule divides by cached + prompt (llama.cpp's prompt_tokens_total EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
• /scripts/dankserver-route-primary.sh — eno1 pinned primary (metric 100), eno2 retained failover (metric 200); applied with nmcli device reapply (no connection reapply in NM 1.54).
• Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge AtomicOpsCap: Routing+, both endpoints ReqEn+ — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
• /serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); frank-medium-llamacpp-dflash2.md carries the SUPERSEDED banner; Dual-R9700-Cooperation.md states the measured verdict; Dankserver System Specs.md gained a Motherboard section; frank-medium-model-swap.md is the new swap runbook; AGENTS.md topology/sections/lessons flipped.

2026-09-19

• /scripts/frank-large.service, /scripts/serve-frank-large.sh — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
• /serverAppConfigs-docs/LLM/frank-large-ciru-orca.md — New runbook.
• /scripts/frank-medium.env, /scripts/serve-frank-medium.sh — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
• /scripts/frank-small.env, /scripts/serve-frank-small.sh — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
• Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
• AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.

2026-09-18

• /scripts/frank-medium.env — CACHE_RAM 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; tests/frank_medium_unit.bats seam added; --ctx-checkpoints 0 confirmed a deliberate deviation (build default is 32) (1acca43).
• Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned llama-server from third_party/llamacpp/build-merge (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.

2026-09-17

• /scripts/frank-small.env, /scripts/serve-frank-small.sh — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M + separate MTP assistant sidecar (--spec-type draft-mtp,ngram-mod); q8_0 KV (build-cuda rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON, else ~12× prefill collapse); --fit chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every reasoning_effort level.
• /scripts/frank-small.env — CTX_SIZE=1048576 / PARALLEL=4 (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
• /serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md — New ACTIVE runbook; frank-small-qwen38-gsq-rco.md retired as rollback target.
• /shared/homepage/ — The three LLM cards moved to the user page with live descriptions; new static /docs user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
• /scripts/deploy-remote.sh — The pending pass now prunes stale pending state entries for retired services into ~/.AGENTTRASH and keeps going (a stale frank-default-sightsound entry made every --pending run exit 2) (02a7876).
• Fleet audit — Every vhost classified against the guard policy (no new exposure); systemd-hermes logged as a boot-persistence gap (running container, no quadlet); stray stopped test-vllm container on dankserver held for approval.

2026-09-16

• /scripts/bench-batch-ubatch.sh — New harness: one llama-bench invocation per cell (a test that cannot allocate exit(1)s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (LimitMEMLOCK=infinity is load-bearing) (87845fc, 77a07ff).
• /scripts/frank-medium.env — UBATCH_SIZE 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and TENSOR_SPLIT 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); tests/frank_medium_unit.bats balance seam re-fitted as a shape assertion.
• /serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
• /dankclaw/frank-large/quadlet/frank-large.container — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
• /scripts/frank-small.env, /scripts/frank-small.service, /scripts/serve-frank-small.sh — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (draft-mtp), BF16 projector, 131072 ctx, f16 KV, port 7771.
• /scripts/deploy-remote.sh — deploy_llm_large flipped back to halogen with a wait_port_free 7770 gate (the first cutover died on Address already in use two seconds after the old server exited); deploy_llm_small added.
• /serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md, /serverAppConfigs-docs/LLM/halogen-flash-server.md — New / re-activated runbooks; frank-large-ik-uncensored.md marked retired.
• /serverAppConfigs-docs/LLM/x4-pair-experiment.md — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by -ot itself (has_tensor_overrides() → pipeline_parallel → n_copies == 1); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (-dev A,B is a SWEEP, not a list — correct -dev A/B -ts x/y gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
• Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every podman command failed, all 28 quadlets looped, podman system migrate panics in 5.7.0; a reboot repaired it. The reboot exposed hermes.franksdank.cloud 502: the Caddyfile dialed 192.168.0.143:8787 (a host bind, not a published port) — fixed to host.containers.internal:8787, and the seam that had accepted any numeric upstream was strengthened.
• Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; halogen-flash-server.md gained a "Refreshing the weights" note.

2026-09-15

• /scripts/frank-medium.env, /scripts/serve-frank-medium.sh, /scripts/frank-medium.service — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (retire_root_llm_small() archived the CUDA-era unit).
• /serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md — New runbook; freetoken-frank-small.md and kingjones-q8agent-rocmfpx.md marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
• Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the ld.so.conf hijack (staged in ~/.AGENTTRASH/20260915-rocm633-cleanup/).
• /scripts/frank-large.service, /scripts/serve-frank-large.sh — frank-large cut to ik_llama.cpp (halogen quadlet retired to ~/.AGENTTRASH/20260915-ik-swap/); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
• /serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md — New runbook with the full evidence chain.

2026-09-14

• /dankclaw/compose.yaml, /dankclaw/.env.example — New hermes-webui service (Hermex): bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data, loopback-only publish 127.0.0.1:8787, password-gated via HERMES_WEBUI_PASSWORD.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories.
• /tests/hermes_webui.bats — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
• /serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md — New page; remote access = Termius local forward over the existing router :22 port forward.

2026-09-12

• /dankclaw/marreta/quadlet/ — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; NGINX_HTTP_PORT=8085 (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
• /tests/marreta_quadlet.bats — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
• /serverAppConfigs-docs/Dankclaw/Marreta.md — New page (incl. the base64:-prefixed APP_KEY gotcha).
• Live — authentik ProxyProvider marreta (pk 13) + Application link: the outpost's config API filters application__isnull=False, so a provider with no linked Application is never loaded (the login-wall root cause); a null invalidation_flow crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec 16-marreta.spec.ts.

2026-09-04

• /frankbook-pro/ — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, /opt config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed 10.88.0.1:8080 → 127.0.0.1:8080.
• /tests/frankbook_quadlet.bats — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
• /dankserver/ — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned /opt/<svc>/config dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and userns_mode must NEVER be added (podman-compose runs services in pods; --userns + --pod is rejected).
• /dankclaw/Caddyfile, /dankclaw/pihole/addn-hosts — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
• /tests/e2e/ — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).

2026-09-03

• /serverAppConfigs-docs/ — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, Prowlarr.md/prowlarr.md deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.

2026-09-01

• /frankbook-pro/gluetun/, /frankbook-pro/qbittorrent/ — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (WebUI\Password_PBKDF2, not the legacy WebUI\Password key qB ≥ 4.2 ignores); DefaultSavePath → /Volumes/COMPLETE/Finished Torrents; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
• /serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
• /dankclaw/authentik/quadlet/authentik-outpost-ldap.container — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via ak shell ORM; live bind tests proved two rules: the token intent must be verification, and the provider needs the search_full_directory object permission.
• /serverAppConfigs-docs/Infrastructure/Authentik.md, /serverAppConfigs-docs/Dankclaw/Jellyfin.md — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (LdapSearchAttributes is comma-separated; username attribute cn).
• /scripts/frank-small.env, /scripts/serve-frank-small.sh — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in _dequant_fp8_weight); reverted to Huihui FP8 within the hour; freetoken-frank-small.md quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
• /dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/ — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move /opt/<svc> → /home/dank/Containers/<svc>); v1 generator quirks fixed (Command=→Exec=, Device=→AddDevice=, HealthCmd= space-splitting, the HOME=/root EnvironmentFile trap).
• /dankclaw/ — Data roots of gitea, authentik, and the hermes trios moved to /home/dank/Containers/<svc>/ while running (rename is inode-bound — zero downtime); deploy_hermes ordering fixed (generated units reload AFTER the unit files are copied).
• /dankclaw/seerr/quadlet/seerr.container — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider seerr-proxy; /tests/seerr_quadlet.bats new seam; /serverAppConfigs-docs/Dankclaw/Seerr.md new page.
• /scripts/deploy-remote.sh — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + HostHeaderValidation=false + the four categories; full 7-service deploy run.
• /frankbook-pro/qbittorrent/quadlet/qbittorrent.container — qB pinned 4.6.7 (5.x's 204 login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
• /serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md, /serverAppConfigs-docs/Architecture/Storage Policy.md, /serverAppConfigs-docs/Architecture/Deployment Flow.md — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.

2026-08-30

• /scripts/frank-npu.env, /scripts/serve-frank-npu.sh, /scripts/frank-npu.service — New frank-npu on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (/dev/accel/accel0 err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with iommu=pt + reboot; vision + audio towers proven end-to-end; health probe is /v1/version (FLM has no /health route) (c3ed3a6, cf0c541).
• /serverAppConfigs-docs/LLM/frank-npu.md — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
• /dankclaw/frank-large/quadlet/frank-large.container — frank-large: swapped to kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the pinned kingjones30/ROCmFPX fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
• /scripts/frank-small.env — frank-small (FreeToken): model swapped to batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (native block-fp8); ~40 tok/s verified.
• /serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md — Reactivated as the large runbook; agentionai-fast-imatrix.md marked RETIRED; freetoken-frank-small.md model sections updated; LLM index + AGENTS fleet tables refreshed.
• /serverAppConfigs-docs/Dankserver/prowlarr.md, /serverAppConfigs-docs/Dankserver/qbittorrent.md — Rewritten: the arr↔Prowlarr feed contract (/{indexerId}/api, the global key — the hand-rolled /torznab/ paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (WebUI\Password_PBKDF2, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).

2026-08-29

• /dankclaw/Caddyfile — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
• /dankclaw/anubis/policy.json — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile).
• /serverAppConfigs-docs/Infrastructure/Authentik.md, /serverAppConfigs-docs/Infrastructure/Anubis.md, /serverAppConfigs-docs/Architecture/Network Architecture.md — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original Host vs each provider's external_host (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null authorization_flow/invalidation_flow starve the outpost's config fetch; the public zone gained the auth A record via the Porkbun API (ee1e038).

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
• /dankclaw/gitea/quadlet/gitea-caddy.container — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
• Incident — Post-reboot: Caddy (a bare compose container — restart: unless-stopped does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: net.ipv4.ip_unprivileged_port_start reset to 80 by a later-sorted sysctl drop-in AND a stale line in /etc/sysctl.conf, which systemd-sysctl applies AFTER all of sysctl.d/.
• /AGENTS.md — New MANDATORY "Boot Persistence & Self-Healing" section.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (joins dankclaw_default, Pull=never, env from /home/dank/Containers/caddy/env); removed from compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — Repo source of truth for ip_unprivileged_port_start=53; deploy_pihole stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies sysctl --system (the boot code path).
• /tests/caddy_quadlet.bats (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
• /serverAppConfigs-docs/Infrastructure/Caddy.md, /serverAppConfigs-docs/Infrastructure/Pi-hole.md, /serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md, /serverAppConfigs-docs/Dankclaw/Gitea.md, /serverAppConfigs-docs/Architecture/Deployment Flow.md — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.