serverAppConfigs/serverAppConfigs-docs/log.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Bundle Change Log	Chronological record of meaningful changes to the serverAppConfigs knowledge base.	changelog documentation maintenance	2026-09-14

Bundle Change Log

2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone

• Hardware. dankserver now runs an ASUS ProArt B850-Creator WiFi NEO (was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board bifurcates the CPU GPP ports x8/x8 at Gen5, so both cards sit on CPU root ports: card 0 00:01.1 → 01:00.0 → 02:00.0 → 03:00.0, card 1 00:01.3 → 05:00.0 → 06:00.0 → 07:00.0. Every bridge reads AtomicOpsCap: Routing+ and both endpoints read AtomicOpsCtl: ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition Dual-R9700-Cooperation's test plan named before any model would be touched. Bandwidth per card ~31.5 GB/s (kernel: 252.056 Gb/s … limited by 32.0 GT/s PCIe x8): card 0 unchanged from Gen4 x16, card 1 up ~16x from Gen4 x1.
• Verdict, stated honestly. vLLM TP=2 / PP=2 and llama.cpp --split-mode row are unblocked, not proven — no collective has been dispatched since the swap, GGML_HIP_RCCL is still OFF in scripts/build-llama-rdna4.sh, and the 272 t/s "tensor" number on record measured a butterfly fallback, not an allreduce. librccl.so.1 + hipcc already exist in the bundled TheRock toolchain, so an RCCL-backed rebuild needs no new host packages. The probe still owes a stop-and-mask window.
• Side effects of the swap, recorded. GPU BDFs moved (0b:00.0 → 07:00.0), iGPU 10:00.0 → 11:00.0, WiFi interface wlp8s0 → wlp14s0, kernel now 7.0.0-31-generic, SMT back on (16 logical CPUs; llama.cpp still --threads 8), and the board's second 5GbE port is cabled as eno2 = 192.168.0.247 with the lower-metric default route while eno1 keeps the fleet's static 192.168.0.235. Two addresses, two default routes — needs a decision.
• Docs cut over (no service change; frank-medium stayed up and healthy throughout): Dual-R9700-Cooperation short answer rewritten plus a re-measured 2026-09-20 topology table; Dankserver System Specs gained a Motherboard section and corrected GPU/iGPU/network/kernel rows; frank-medium-llamacpp-dflash2 hardware row and rationale now split past from present; AGENTS.md topology table, the frank-medium section, and the PCIe lesson bullets no longer assert a wall that no longer exists. The dated work-log sections that describe the old board stay as history.

2026-09-19 — fleet cutover: frank-large CIRU Orca, frank-medium orcarouter Q8_0, frank-small Gemma4 revert

• frank-large (Dankclaw :7770): halogen → CIRU v4.4.1 Orca. The private CIRU v4.4.1 runtime (gfx1151 HIP build + QSA window hotfix, 381/381 ninja targets) now serves jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b (79.4 GiB main + 4.1 GiB MTP + 52.4 GiB PLE + f16 mmproj, all sha256-verified) as a ROOT systemd unit — one 262,144-token conversation (1 slot, the MTP constraint), MTP depth 4 (acceptance 0.656), 4096 MiB PLE cache, text only. The launcher's runtime-root gate is fed from the SDK venv by the serve script. Halogen is the rollback target. (b6ce3af, f68bda5; runbook frank-large-ciru-orca).
• frank-medium (Dankserver :7770): ajgazin Swift UD-Q8_K_XL → orcarouter Q8_0. Model swap to orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the target's own MTP head stays in-file, one SPEC_TYPE away); split re-balanced 35:30 → 32:33 (operator call: the most even split 65 layers allow); pool 393216 → 327680 (2 slots of 163840, inside the trained 262144 window — no rope extrapolation); f16 KV + 16 GiB host-RAM prompt cache. (d374ab9).
• frank-small (Dankclaw :7771): vLLM → llama.cpp Gemma4-12B Q4_K_M. The same-day vLLM detour (google/gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin) was reverted back to mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M with the MTP assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262144. The vLLM quadlet is retired. (374041c; runbook frank-small-gemma4-12b).
• 3090 RGB off. The EVGA FTW3 Ultra V2's iCX3 controller (i2c-1, addr 0x2D) was set to MODE_OFF and SAVED to EEPROM — persists across reboots with no Linux-side daemon; works headless.
• Docs wrap-up (764cf41, 88331e2): AGENTS.md (frank-large + frank-medium sections, topology rows), new frank-large-ciru-orca runbook, frank-medium runbook (header + deployed-flags table), both indexes, halogen runbook retirement banner. 331 bats + 32 e2e green.

2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30

• New harness, shipped before use (scripts/bench-batch-ubatch.sh, 87845fc, fixed in 77a07ff): one llama-bench invocation per cell — a test that cannot allocate calls exit(1), so one infeasible ubatch would take the whole sweep with it — a VRAM sampler per cell, and the service stopped for the run and started again from a trap. It must run from a transient unit: the deploy's LimitMEMLOCK=infinity is load-bearing for ROCm staging buffers and an SSH shell on these hosts only gets 8 MiB.
• The sweep (23 of 28 cells, a 64k prefill measured at 64k depth, 3 reps): prefill peaks at ubatch 512. 256 and 384 lose ~11 % (SMs not saturated), 2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 / 1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048 measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap between 512 and the deployed 1536 real.
• The logical --batch-size is irrelevant: 2048 through 8192 land within 2 % of each other at a fixed ubatch and use identical VRAM, because the compute buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at 64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform ~32 % of prefill.
• Deployed change (a11fae0): UBATCH_SIZE 1536 → 512, which is both faster and 3.71 GiB cheaper, and TENSOR_SPLIT 36,29 → 35,30, because the ubatch buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB for the 1024-unit move, measured at 262144 over two slots) and shrinking them handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: 6.44 / 6.90 GiB free (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a 0.46 GiB gap — one DeltaNet layer, which is the granularity floor here.
• The balance seam in tests/frank_medium_unit.bats was re-fitted at the new deployed point with both ubatch coefficients measured at this shape; every neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 → 3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal TENSOR_SPLIT=36,29 assertion became a shape assertion — two shares summing to the model's 65 blocks with ROCm1 taking the smaller — so re-measuring the balance no longer breaks a test about parallelism.
• Measurement trap worth keeping: re-loading immediately after a kill fails with unable to allocate ROCm0 buffer. amdgpu does not hand ~26 GiB back at once, and a 180 s settle is not enough; it reads exactly like a configuration that does not fit. The measurement helper grew a wait-for-empty gate.
• Docs: frank-medium-batch-ubatch (new page), the runbook's deployed flags and split history, and AGENTS.md's batch lesson. Dual-R9700-Cooperation gained the live AtomicOpsCap / AtomicOpsCtl chain — ReqEn+ on card 0 against ReqEn− on card 1, because the kernel refuses the requester bit when a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a vendor claim rather than a fact.

2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired

• The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so frank-small is retired and frank-medium took the :7770 endpoint.
• Runtime: charlie12345/ROCmFPX llama.cpp c49ebdb built for gfx1201 with the fork's self-contained ROCm 7.14 toolchain (TheRock nightly, RPATH-bundled libs) — the host's broken mixed ROCm (AMD-repo 6.3.3 hijacking libhsa via /etc/ld.so.conf.d/10-rocm-opencl.conf) is fully bypassed. Model: kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (ftype 115, 26.28 GiB) + Q4_0 MTP draft head; mmproj unused (vision stays on frank-large).
• Parallelism decision, measured not argued: llama-bench with the correct multi-device syntax (-dev ROCm0/ROCm1 — comma form silently benches each card separately): layer 1991/18.8 t/s (pp4096/tg128) vs tensor 271.8/26.3 vs single card 1160.9/19.4. Tensor's decode win costs a 7x prefill collapse (no RCCL → "meta-backend butterfly" allreduce); served with MTP, layer also wins decode (35.7–46.2 tok/s vs 28.7). Operator picked layer ("pipeline parallelism"); the data agrees on both axes. Draft head pinned to ROCm1 (card holding the last target layers); under tensor mode an unpinned draft aborts the KV build (GGML_ASSERT(buffer) in ggml-backend.cpp).
• Verified live before and after the systemd cutover: health ~18 s, MTP acceptance 60–69 %, cold 949-token prefill in 2036 ms, and prefix caching works with the draft head (turn 2 reprocessed 22 tokens, cache_n 945) — the model card's mtp-prompt-cache-fix.patch defect is already fixed in c49ebdb (common_prompt_checkpoint::data_spec).
• Deploy: deploy.sh dankserver --service frank-medium (new retire_root_llm_small() archived the CUDA-era unit to ~/.AGENTTRASH/20260915-frank-small-retire/host-units/); e2e gate 30/30 including 11-admin (the blankpad→192.168.0.112 path issue has cleared).
• Host cleanup (the "Both" half of the plan): purged 167 stale AMD-repo ~24.04 ROCm 6.3.3 packages, removed the radeon apt pin and the ld.so.conf hijack (staged in ~/.AGENTTRASH/20260915-rocm633-cleanup/); libhsa-runtime64 now resolves only to Ubuntu's ROCm 7.1 and rocm-smi 7.1.1 (Ubuntu package) works. frank-medium unaffected (self-contained).
• Client cutover: blankpad models.yml/config.yml renamed the model id to frank-medium (ctx 131072, maxTokens 32768); Hermes live config on Dankclaw repointed dankserver-small routes to frank-medium.
• New: /serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md runbook; updated AGENTS.md, README, docs indexes, Dankserver System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages; freetoken-frank-small and kingjones-q8agent marked retired/superseded.

2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw

• Added manualdousuario/marreta (Laravel / serversideup/php:8.4-fpm-nginx) as a rootless Quadlet pair: marreta + marreta-browser (Lightpanda), both joining gluetun's netns so all fetch egress exits the same ProtonVPN tunnel as SearXNG.
• Port-collision fix: the shared netns already has SearXNG on 8080, so marreta is relocated to NGINX_HTTP_PORT=8085 (verified against the serversideup source: the image does not bake conf.d/*.conf; the entrypoint envsubsts ${NGINX_HTTP_PORT} at start). Gluetun publishes 8082:8085 on marreta's behalf; Caddy dials 192.168.0.143:8082.
• Egress rationale: netns sharing, not PROXY_URL — the browser's CDP fetch passes no proxy, so only netns keeps the browser off the home IP.
• Guard chain: marreta.franksdank.cloud → Anubis :8923 → Authentik → post-auth :8017 @marreta → 192.168.0.143:8082. Access model is any authenticated user (no group gate). Requires a one-time ProxyProvider (external_host https://marreta.franksdank.cloud/, internal_host http://caddy:8017) on the embedded outpost.
• Secrets seeded to /home/dank/Containers/marreta/env (0600, dank-owned) from MARRETA_APP_KEY / MARRETA_ADMIN_EMAIL / MARRETA_ADMIN_PASSWORD.
• New test seam tests/marreta_quadlet.bats (17 contracts). deploy-plan.py: marreta joins the gluetun netns family. Doc: Marreta.
• Deployed live 2026-09-12 (rev 91e9410) after standing up passwordless SSH from blankpad to the fleet. Ran deploy.sh dankclaw --service gluetun --service marreta --service caddy (gluetun first: the new 8082:8085 publish needs a tunnel recreate, which also cycles SearXNG). Authentik ProxyProvider marreta registered (pk 13). Verified: unauth → Anubis challenge; post-auth :8017 → 200; container healthy; netns sharing proven; SearXNG unaffected.
• Gotcha fixed in-flight: the first seed used a bare-base64 APP_KEY, which 500s every Laravel app route (Unsupported cipher or incorrect key length). Re-keyed with the required base64: prefix before any encrypted data persisted. See Marreta § Secrets.
• Login-wall fixed 2026-09-12 (post-deploy). The homepage Marreta link parked on marreta.franksdank.cloud/if/flow/default-authentication-flow/ instead of the app. Root cause chain, in the order found:
  1. marreta-proxy had invalidation_flow = null, which crashed the Rust embedded outpost's serde (invalid type: null, expected a formatted UUID string) and took down every provider on the outpost. Set to default-provider-invalidation-flow.
  2. authentication_flow / authorization_flow were null; set to default-authentication-flow / default-provider-authorization-implicit-consent.
  3. The real blocker: the outpost's config API filters ProxyProvider.objects.filter(application__isnull=False). marreta-proxy had no linked Application, so the outpost never loaded it and the request fell through to raw flow serving on the marreta host — where the IdP session cookie (scoped to auth.franksdank.cloud) is invisible. Created Application(name="Marreta", slug="marreta", provider=marreta-proxy).
• After systemctl --user restart authentik-server authentik-outpost, the outpost loaded marreta-proxy and the OAuth dance completes: / → /outpost.goauthentik.io/start → auth…/o/authorize → callback → Marreta. Regression test: tests/e2e/specs/16-marreta.spec.ts. Providers remain live-only (no repo blueprint) — a provenance gap to close.

2026-09-04 — *arr fleet consolidated on FrankBook-Pro; all egress VPN-gated

• Operator decision: *FrankBook-Pro is the sole arr deployment. The Dankserver compose copy (booted earlier the same day) was removed; /opt config roots trashed on the host; compose/services pruned.
• All six fleet containers (qB + five arrs) now share gluetun's network namespace — all egress exits the ProtonVPN tunnel by construction (AllowedIPs=0.0.0.0/0). The arrs publish no ports of their own; their UI ports (9696/8989/7878/8686/8787) moved onto the gluetun unit. Proven live: residential 71.227.227.172 vs netns-wide 51.179.177.14.
• New test seam tests/frankbook_quadlet.bats: every arr+qB unit must join gluetun's netns and publish no port; gluetun must publish the whole fleet's port set. deploy-plan.py: gluetun changes carry the whole netns family.
• arr→qB download clients re-pointed 10.88.0.1:8080 (old bridge-gateway hack, dead inside the tunnel) → 127.0.0.1:8080; verified by real qB login from inside the netns. Caddy vhosts + homepage probes repoint to 192.168.0.112; homepage all green.
• Defects discovered: Prowlarr's four Application rows are GONE (empty table; sync broken until rebuilt with loopback baseUrls) and readarr's qB proxy auth defect needs a retest post-move. EZTV/1337x CF-block parking may be stale now that egress is VPN — retest before enabling.
• E2E specs (11-admin/12-crawl) de-hardcoded: expected open-link counts now derive from the live services.json registry. AGENTS.md: deployments are complete only when unit tests AND Playwright E2E both pass.

2026-09-04 — dankserver *arr stack reactivated; rootless identity model documented

• ARR-Stack: the Dankserver compose deployment (committed 2026-08-29, never booted — root-owned /opt/<svc>/config dirs crashed every app at logger init before any port bound) is live as of 2026-09-04 behind six own-auth TLS vhosts (<svc>.franksdank.cloud), all verified 200, homepage probes green. Documented the verified rootless identity model — container root (PUID=0) maps to host uid 1000 (dank) by default on podman 5.7.0 (/proc/<pid>/uid_map = 0 1000 1, 1 100000 65536), so data roots are 1000:1000 and userns_mode must never be added to compose services (podman-compose 1.5.0 runs services in pods; --userns + --pod is rejected). FrankBook-Pro differences noted: no gluetun egress gate on the Dankserver qB, empty (fresh) app configs, Sonarr in the homepage user section and the rest in admin.

2026-09-03 — full fact-check sweep of the docs tree (docs-only, deploy no-op)

• Scope: every page under /serverAppConfigs-docs/ re-verified against live hosts (read-only SSH), quadlet unit files, and the Caddyfile; surgical edits only — no pages created, renamed, or deleted. Per the deploy planner, a docs-only change is a deploy no-op.
• Dankserver core rewritten to live quadlet facts: the *arr stack + qBittorrent
  • gluetun run on FrankBook-Pro (units installed to /home/core/.config/containers/systemd/ via the frankbook-pro deploy handler; PUID=0/PGID=0, data roots /Volumes/COMPLETE/<svc>-data:/config; qB pinned 4.6.7 sharing gluetun's netns with 8080 + 6881 published on the gluetun unit; readarr pinned amd64-1.0.980; no vhosts — direct IP:port, exempt by construction from the guard chain). Jellyfin, Plex, RomM (8085→8080 guarded), Calibre-Web (0.6.27-ls397 + universal-calibre mod; no standalone calibre-server), mariadb/redis/Tautulli/Entropy verified against live units.
• Caddy guard chain documented to the live mechanism end-to-end: Anubis :8923 → internal :8016 → post-auth :8017 routing; guarded vhosts (search, grafana, prometheus, loki, seerr) route post-auth to container-name upstreams; exemptions named by construction reason (auth/IdP itself, gitea smart-HTTP, ai→:3005 own auth, hermes/hermesadmin internal-only via DNS absence, pihole basicauth edge → :8083).
• Gitea.md: data roots + secrets corrected to the live unit paths (/home/dank/Containers/gitea/{data,postgres,env}); UID gotcha corrected — the units carry no User=/UserNS override, so container uid 1000 is literally host uid 1000 (no subuid offset for these containers).
• Dedupe: Prowlarr.md/prowlarr.md collapsed into one canonical lowercase page via rename (git history retains both versions; index link unchanged).
• LLM docs: archive pages (DFlash DSpark Speculative Decoding, ling-3.0-flash-trial) reworded from current-tense to dated historical framing with pointers to the active runbook kingjones-strix-rocmfp4; LLM index.md + deployment.md fleet tables and health-gate text updated to the 2026-08-30 fleet state (STRIX on the pinned ROCmFPX fork; Huihui abliterated FP8 on FreeToken; sightsound = FLM v1.0.1 on XDNA2, port 7771, health /v1/version — FLM has no /health route).
• Suite green after the sweep: ./tests/run.sh → 233 pass / 0 fail, including the docs-completeness seam that requires every page to stay index-linked.

2026-09-01 (late) — FrankBook-Pro stack completed: tunnel live, qB wired into the arrs

• WireGuard credentials provisioned (second Proton config, server PA-12, endpoint 51.179.177.2:51820) → /home/core/Containers/gluetun/env (0600, VM-local); deploy.sh frankbook-pro --service gluetun --service qbittorrent brought both up; egress verified (Proton PA exit, ~19 MB/s on a live torrent). The .conf stays in ~/Downloads; the private key never enters the repo.
• gvproxy binds wildcard *:6881/udp (Go sets SO_REUSEADDR on TCP but not UDP) → the socat 6881 tcp+udp LaunchAgents collided at every boot and were retired; swarm inbound (if ever needed) is the Proton NAT-PMP path. macOS application firewall still filters gvproxy inbound from non-local sources (stealth mode) — cross-host 8080 needs socketfilterfw allow.
• qB wiring: first-seed bug (hash under legacy WebUI\Password key, ignored by qB ≥ 4.2) fixed live (WebUI\Password_PBKDF2) and in scripts/deploy-remote.sh; DefaultSavePath repointed to /Volumes/COMPLETE/Finished Torrents — qB 5.2 API adds with a category do not inherit the category save path (no auto-TMM); download clients created in all four arrs (sonarr/radarr/lidarr via API, tests pass; readarr by direct readarr.db insert — its EOL validator rejects qB 5's 204 login as auth failure); Prowlarr Applications sync sonarr/radarr/ lidarr (readarr omitted: prowlarr's indexer push 500s on EOL schema). Legacy Prowlarr-* indexer rows (hostname prowlarr:9696, unresolvable on the Mac default network) deleted. E2E: Sintel fetched + downloaded through the tunnel at ~19 MB/s, completed into Finished Torrents.
• Docs: FrankBook-Pro rewritten to the active-tunnel reality.

2026-09-01 - WG endpoint identified as ProtonVPN Seattle; Mac gate is a second dashboard config

• RDAP: 149.102.254.77 = CDNEXT-CG-SEA (Datacamp Limited, Proton's Seattle hosting partner); Dankclaw's tunnel v6 2a07:b944::2:2 sits in CH-PROTONVPN-20160719; observed exit IP 149.22.88.19 = CDNEXT-SEA-CG. The "custom WG server" is a ProtonVPN Seattle WireGuard server - there is no peer-registration step on any box of ours (full 65535-port scan: no SSH anywhere, just silent TLS listeners - 443, 4443-4468, 5995, 7770, 8443 - Proton's protocol-stealth port map, no admin surface).
• Mac gate corrected: provision a second Proton WG config (dashboard, named e.g. frankbook) - one config = one keypair; two hosts on Dankclaw's config bounce the server-side endpoint and kill SearXNG. Values paste straight into the gluetun custom env. Earlier prepared keypair on dankclaw (~/frankbook-wg/) retired to ~/.AGENTTRASH/ (Proton generates the keypair per config; no BYO pubkey).
• FrankBook-Pro provisioning section rewritten.

2026-09-01 — Jellyfin authentication moved to Authentik LDAP

• /dankclaw/authentik/quadlet/authentik-outpost-ldap.container (4ad85fa, digest-pinned in f4ef2b8) — new authentik-outpost-ldap.service on Dankclaw serving LDAP :389/:636 for Jellyfin. Image ghcr.io/goauthentik/ldap pinned by amd64 single-platform digest: podman 5.7.0 refuses the tag because it resolves to a nested manifest list ("unexpectedly received a manifest list instead of a manifest for a single image") — resolve via podman manifest inspect | jq and pin.
• Authentik objects provisioned via ak shell ORM (live, idempotent get_or_create): LDAP provider Jellyfin LDAP, service account svc-jellyfin-ldap + bind token, groups jellyfin_users / jellyfin_admins, application jellyfin.
• /scripts/deploy-remote.sh (691da47) — authentik-outpost-ldap added to service_is_allowed (the deploy runner has its own allowlist beyond deploy-plan.py's).
• Two authentik-side rules proven by live bind tests (ldapsearch through the published port):
  1. Bind token intent must be verification — app_password tokens are rejected by the authorization flow's password stage with Invalid password.
  2. Search requires the search_full_directory object permission on the provider (granted via RBAC role svc-jellyfin-ldap-search); without it the outpost returns only the bind user, so memberOf filters match zero entries.
• Jellyfin side (host-only, no repo change): LDAP-Auth 23.0.0.0 installed from the GitHub releases zip (the repo.jellyfin.org stable manifest path 404s), config XML at config/plugins/configurations/LDAP-Auth.xml, username attribute cn (authentik 2026.x maps uid to an opaque hash), and — the big one — LdapSearchAttributes is comma-separated (Split(',') in the plugin source; semicolons silently produce "Found no users matching X"). End-to-end verified with a throwaway jellyfin_users member (AuthenticateByName 200, non-admin policy), then removed from both authentik and the Jellyfin DB directly (its admin-token cleanup 401'd — the admin password in the session archive was OCR-corrupted; verified offline against the stored PBKDF2-SHA512 hash that the real bootstrap value still authenticates, unchanged).
• Docs: Authentik gained the LDAP-outpost section (objects table + rules), Jellyfin gained the LDAP-Auth plugin section. Docs-only commit → deploy no-op.

2026-09-01 — frank-small: two compressed-tensors FP8 swaps fail to load; Huihui restored

• /scripts/frank-small.env, /scripts/serve-frank-small.sh, /scripts/frank-small.service (af6cd72 → 34ab6b4 → reverted in 4d5bea4) — attempted model swaps on FreeToken v0.1.2, both live-verified as loader failures and reverted the same day to the Huihui FP8 checkpoint:
  • ornith-ai/Ornith-1.5-35B-A3B-FP8 (compressed-tensors per-channel): ValueError: Missing MoE expert source layers: {'gate_up': ...} — the checkpoint's expert tensors do not follow the mlp.experts.{N}.gate_proj/up_proj/down_proj naming the MoE provider scans for.
  • lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8 (compressed-tensors block-fp8 128×128, standard naming): experts resolve but _dequant_fp8_weight multiplies weight × weight_scale directly — block scales do not broadcast → RuntimeError: The size of tensor a (2048) must match ....
• Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no outage); service was back on Huihui within the hour and re-verified with a real completion (reasoning parses; 17*23 → 391, natural stop).
• /serverAppConfigs-docs/LLM/freetoken-frank-small.md — quant matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both compressed-tensors shapes documented with their exact errors.
• Deploy health-check trap (second occurrence): deploy.sh reported "healthy after approximately 6s" for BOTH failed loads — the health gate passes while the doomed worker is still starting. A model swap is only proven by the journal (Uvicorn startup) + a real completion.
• Staged for trash (awaiting purge approval): /home/dank/models/ornith-1.5-35b-a3b-fp8 (37 GB) and /home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8 (35 GB).

2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated

• Incident context: Dankserver hard-crashed and rebooted 2026-09-01; every bare compose-era media container (RomM trio, Jellyfin, Calibre-Web, Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule exists for. Recovery doubled as the Phase 1 cutover.
• Phase 1 (dankserver): six quadlet units shipped (dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/) + first-cutover deploy handlers (retire bare containers → rescue RomM's anonymous /romm volume → merge-move /opt/<svc> into /home/dank/Containers/<svc> → ordered start with health gates). All six verified: units active, is-enabled=generated (wants in the generator dir), ports 8085→8080, 8096, 8083, 3923 answering, jellyfin holds /dev/dri render nodes, entropy binds provably ro except copyparty's /w. MariaDB continuity proven via the real dank admin row (created 2026-08-31), not a fresh bootstrap. Stale compose-era container-*.service generate-systemd units retired to ~/.AGENTTRASH/20260901-stale-generate-systemd.
• Deployer findings fixed (each caught live, each with a new seam): v1 generator rejects Command= (→ Exec=), Device= (→ AddDevice=, whole-unit rejection is silent), and splits spaces in HealthCmd= (→ omitted for copyparty); migrate_data_root must merge into pre-seeded destinations and rename mapped-subuid roots (100998) with sudo mv (numeric-preserving, keep mode skips the chown — chowning mapped data breaks the in-container users); quadlet unit dir must be created on a host that never had quadlet; missing PublishPort = app answers inside the netns only (copyparty 3923); seeded env files must carry compose-provided vars only — the image-baked HOME=/root in a [Service] EnvironmentFile= kills the podman client itself.
• Phase 2 (dankclaw): data roots of gitea, authentik, hermes trios moved to /home/dank/Containers/<svc>/ while running (rename is inode-bound — zero downtime), units repointed, redeployed through deploy.sh --service; auth chain verified end-to-end (outpost ping → 302 to flow). deploy_hermes ordering bug fixed: generated units must be reloaded after unit files are copied, not before (stale ExecStart crash-looped the trio on the old path).
• Docs: RomM, Jellyfin, Calibre-Web, Copyparty updated to the quadlet reality; Storage Policy now documents the Containers layout
  • the two ownership regimes; Deployment Flow service-manager table corrected (dankserver media is no longer "bare containers, inventory only").

2026-08-30 (night II) — frank-npu deployed; NPU SVA/IOMMU root cause

• Deployed frank-npu on Dankclaw: FLM v1.0.1 (/usr/bin/flm, dpkg fastflowlm) serving the whole local registry (36 tags) on port 52626; primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy crash-looped every 10 s: Open /dev/accel/accel0 failed (err=-19).
• Root cause (live-verified): the XDNA2 driver's SVA bind fails with ret -19 whenever the AMD IOMMU is off; the kernel cmdline carried iommu=off (a perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The same root cause explains the amdgpu ... no vmid pasid mapping supported kernel line — a consequence, not a GPU fault.
• Fix: iommu=pt in GRUB_CMDLINE_LINUX_DEFAULT + reboot (passthrough keeps the iGPU default domain identity-mapped — zero DMA-translation overhead for frank-large — while enabling IOMMU hardware + PASID, which SVA needs; PASID domains are paged regardless of the default-domain type). Post-boot: 30 IOMMU groups, zero SVA-bind errors since boot, unit enabled, NRestarts=0 since boot.
• Live-verified behavior: whole-registry serving with per-request routing by exact FLM tag (unknown tags silently serve llama3.2:1b); vision tower proved end-to-end (generated red PNG as image_url data-URI → correct color answer, 279 prompt tokens consumed); audio tower proven with a control pair (440 Hz tone via input_audio = 37 prompt tokens vs 15 for the identical text-only control). Health probe is /v1/version — FLM has no /health route.
• /scripts/frank-npu.env, /scripts/serve-frank-npu.sh, /scripts/frank-npu.service + deployer allowlist entries (c3ed3a6, cf0c541) — service files and deploy wiring; unit is boot-persistent (WantedBy=multi-user.target, enabled, Restart=on-failure, RestartSec=10, LimitMEMLOCK=infinity — FLM requires unlimited memlock).
• Docs: new runbook frank-npu (linked from both indexes), AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard requirement. Client wiring: pi models.json registers the endpoint under the Dankclaw provider with the exact FLM registry tag as model id.

2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken

• Dankclaw / frank-large: swapped to kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's pinned kingjones30/ROCmFPX runtime (built at /home/dank/GitPackages/kingjones-ROCmFPX/build, LLAMA_BUILD_WEBUI=OFF — npm/HF-bucket asset provisioning aborts the link at 98% otherwise). On-host 113.47 GiB file verified byte-identical to the HF LFS oid (no re-download). Kept our args (401408×2, ub 512, fa on, fit off); fa-on works UNPATCHED on this model — the QSA F32 assert was agentionai-specific. No spec decode (card ships none; MTP heads unloadable). Validated live: ~225 s load, health 200, reasoning completions, real 111k-token prefill at 213–232 t/s. Deployed via git flow; manual-process race during validation caught and fixed (unit was crash-looping on a bound port while its health check passed — check MainPID, not just /health).
• Dankserver / frank-small: swapped the model within FreeToken v0.1.2 to batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (native block-fp8, same loader path and pool plan; 35 GB downloaded via hf, official checkpoint retained for rollback). Learned: FreeToken's chat API 422s without an explicit model field. ~40 tok/s verified post-deploy. Both .service Descriptions refreshed.
• Docs: kingjones-strix-rocmfp4 reactivated as the large runbook (runtime build + swap procedure + validation), agentionai-fast-imatrix marked RETIRED (rollback record), freetoken-frank-small model sections updated, LLM index + AGENTS fleet tables refreshed.

2026-08-30 (late) — download chain proven end to end; qB credentials persisted

• Cracked the arr↔Prowlarr feed contract: route is /{indexerId}/api (Prowlarr NewznabController; absolute /api/v1/indexer/{id}/newznab), auth = the Prowlarr global key. The previously hand-rolled /torznab/{id}/feed paths were Forms-auth catch-alls returning 302 → /login — which the arrs XML-parsed as "unexpected token '/'". A source-verified librarian answer settled it in one shot.
• Re-pointed all four arr indexer rows (Radarr→YTS id7, Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API; every PUT passed the app's live connection validator (202); RSS enabled.
• TV + music chains proven by live search: Sonarr returned 76 current releases via Prowlarr-Nyaa; Lidarr initially logged 0 active indexers (new rows default all-off) — enabling the search flags yielded 18 releases for YOASOBI – THE BOOK.
• First real grab: MoviesSearch → 3 YTS releases → The Dark Knight (2008) 2160p (7.5 GiB) → qBittorrent 100 % → imported (hasFile: true, 8,078,103,833 B at /movies/The Dark Knight (2008), hardlinked from the shared /mnt/entropy filesystem), queue empty.
• Prowlarr Application auto-sync found broken by version skew (Unable to cast object of type 'NzbDrone.Core.Thing…' against Radarr 6.3) — the manual rows are the integration; documented.
• qBittorrent permanent credentials installed: the WebUI temp password rotated on every boot (it had silently broken the arr clients once this session). qb5 removed the API password endpoint and the lscr image has no env support — wrote WebUI\Password_PBKDF2 (source-verified base64(salt):base64(PBKDF2-HMAC-SHA512/100k) form) into the conf with the container stopped; login survives podman restart (204). All arr download clients updated to it; value kept host-local in /home/dank/.qbpass (0600), never in git.
• Readarr parked with reason: abandoned upstream; its qB client cannot authenticate against qB 5 while three sibling apps pass with identical credentials. Indexer row healthy; books pipeline awaits a successor.
• Docs: rewrote Dankserver/prowlarr.md (live wiring + six verified gotchas incl. the joiner bug and the 302-masquerading-as-auth failure) and Dankserver/qbittorrent.md (persistent-credential recipe, Referer CSRF note).

2026-08-29 (late) — guarded-host sign-in fixed end to end

• Diagnosed both user symptoms: Invalid next URL = the flow being entered with an absolute cross-host next= (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had no auth A record (added via Porkbun API, auth A 71.227.227.172 TTL 300).
• Disproved meta-header routing live (full meta-header set still 404s; matching Host alone answers). The 2026.8 embedded outpost routes solely on the original Host vs each proxy provider's external_host.
• Provisioned proxy providers + applications for search, hermes, prometheus, romm, files (pks 2–6) via ORM, attached to the embedded outpost; internal_host = http://caddy:8017 everywhere.
• Wedge found + fixed live: providers created with null authorization_flow/invalidation_flow starve the outpost's config fetch (Failed to fetch outpost from API, retrying in 3 seconds forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
• /dankclaw/Caddyfile (29f9a86) — :8016 is now a uniform whole-request reverse_proxy authentik-server:9000 preserving Host; deleted the per-vhost redirect/flow design and stale comments.
• /dankclaw/anubis/policy.json (2a72efa) — ALLOW ^/outpost\.goauthentik\.io/ so login/transport paths are reachable before the challenge dance; app paths remain challenged.
• Gotcha: the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). --service anubis deploy restart is mandatory after policy edits.
• /tests/authentik.bats — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host next= anywhere in the Caddyfile). Suite green (0 not-ok).
• /docs/Infrastructure/Authentik.md, /docs/Infrastructure/Anubis.md, /docs/Architecture/Network Architecture.md (ee1e038) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
• Verified by headless JS client: https://search.franksdank.cloud/ → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 state JWT. Credential submission is the user's step.

2026-09-14

• Creation: Hermes-WebUI — new hermes-webui compose service on Dankclaw (the Hermex server): runs hermes-webui's bootstrap.py --foreground in the agent image, shares /home/dank/hermes-data with the gateway (identical rootless Podman uid mapping), publishes 127.0.0.1:8787 only, password-gated via HERMES_WEBUI_PASSWORD in the host .env.
• /dankclaw/compose.yaml, /dankclaw/.env.example — service definition and password placeholder; container-internal bind is 0.0.0.0 (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/deploy-remote.sh — hermes-webui registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
• /tests/hermes_webui.bats — new seam: foreground entrypoint, loopback publish + auth, shared volume, /health healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
• Remote access pattern documented: iPhone Termius local forward 127.0.0.1:8787 -> 127.0.0.1:8787 over the existing router port forward to Dankclaw :22; Hermex points at http://localhost:8787. No Tailscale, no Cloudflare.

2026-08-26

• /dankclaw/Caddyfile — Both Plex vhosts now proxy to PMS over a TLS upstream (insecure_skip_verify); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
• /docs/Infrastructure/Caddy.md — Documented the Plex TLS-upstream requirement and the /web/ vs / distinction.
• /tests/caddyfile.bats — Added a test pinning the TLS upstream transport on both Plex vhosts.
• /docs/Dankclaw/Gitea.md — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
• /docs/Dankclaw/Dankclaw Stack.md — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
• /docs/Architecture/Deployment Flow.md — Replaced the split-proxy safety description with the canonical Caddy ownership model.
• /dankclaw/gitea/quadlet/gitea-caddy.container — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
• /scripts/deploy-plan.py — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
• /scripts/deploy.sh and /scripts/deploy-remote.sh — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
• /tests/caddyfile.bats, /tests/deployment_scoping.bats, /tests/deployment_flow.bats, /tests/frank_large_config.bats — Added migration coverage and corrected the existing ubatch harness assertion.
• /docs/index.md — Bundle index remains the root navigation entry for the updated pages.
• Boot-persistence audit (post-reboot, 2026-08-26 evening): after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (restart: unless-stopped does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): net.ipv4.ip_unprivileged_port_start was reset to 80 by two sources — a later-sorted sysctl drop-in (99-rootless-ports.conf) and a stale line in /etc/sysctl.conf, which systemd-sysctl applies after all of sysctl.d/ (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the /etc/sysctl.conf line was neutralized too).
• /AGENTS.md — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (WantedBy=default.target + enabled + linger, Restart=always, repo-provenance .container), verified before a service is declared deployed.
• /dankclaw/caddy/quadlet/caddy.container, /dankclaw/caddy/env.example — Caddy is now a boot-persistent quadlet (caddy.service): joins dankclaw_default so container-name upstreams keep resolving, Pull=never (local-only image), Porkbun credentials via /home/dank/Containers/caddy/env (deploy writes it from .env), bare-IP-catch-all health check. The caddy service was removed from /dankclaw/compose.yaml.
• /dankclaw/pihole/sysctl/99-dnsmasq.conf — repo source of truth for ip_unprivileged_port_start=53; deploy_pihole installs it to /etc/sysctl.d/, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale /etc/sysctl.conf line (applied after all of sysctl.d), and applies sysctl --system (the boot code path).
• /scripts/deploy-remote.sh — deploy_caddy installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the default.target.wants symlink from the unit's [Install] section at every user-manager start — is-enabled reports generated for all healthy quadlets, and an explicit systemctl enable fails on generated units (learned live during the deploy). deploy_pihole enforces the sysctl prerequisites.
• /tests/caddy_quadlet.bats (new seam), /tests/caddyfile.bats, /tests/deployment_flow.bats, /tests/pihole.bats — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
• /docs/Infrastructure/Caddy.md, /docs/Infrastructure/Pi-hole.md, /docs/Dankclaw/Dankclaw Stack.md — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.

2026-09-01

• dankserver SSH recovered after the reboot (the "Not allowed at this time" port-22 responder vanished with it); the media stack had been restarted bare ~35 min post-boot, then the quadlet cutover completed for all six services (romm trio 13:45, jellyfin 13:49, calibre-web 13:50, copyparty 13:53 PDT) — units generated from the repo, data roots migrated to /home/dank/Containers/<svc>, /opt/{romm,jellyfin,calibre-web,copyparty} retired, all four endpoints answering, legacy container-* units gone. Dankserver deploy-state for the six services records deployed revisions through 1a257c0.
• /scripts/deploy-remote.sh — frankbook handler fix: the credential-gate SKIP state now propagates Mac-side before restart+verify (a parked qbittorrent previously failed the batch and aborted the five arrs); qB first-boot seed writes a PBKDF2 @ByteArray WebUI password (plaintext is ignored by qB ≥ 4.2), sets HostHeaderValidation=false + ServerDomains=* for the socat forwarder path, and pre-creates the four categories with save_path=/Volumes/COMPLETE/Finished Torrents.
• frankbook-pro — full 7-service deploy run: gluetun + qbittorrent SKIP cleanly (WireGuard credentials pending — the Mac needs its own peer; reusing Dankclaw's key would flip its tunnel), prowlarr/sonarr/radarr/lidarr/readarr deployed, healthy, and reachable from dankserver over the LAN via the socat LaunchAgents (192.168.0.112).
• jellyfin GPU posture on dankserver: AddDevice=/dev/dri + GroupAdd=keep-groups in the unit (parity with pre-migration reality — no /dev/nvidia* in the container, no working VA-API driver in the image; the old bare container had the same. CPU/direct-play serves the household).
• /docs/Infrastructure/FrankBook-Pro.md (new), /docs/index.md — documented the Mac stack: VM placement, WireGuard gate, deploy flow, socat LaunchAgents, and the qBittorrent PBKDF2 seed.
• /dankclaw/seerr/quadlet/seerr.container — New Seerr (seerr-team/seerr v3.4.1) request-manager unit on Dankclaw: loopback-only publish 127.0.0.1:5055, SQLite under /home/dank/Containers/seerr/data, on dankclaw_default for the post-auth route.
• /dankclaw/Caddyfile, /dankclaw/pihole/addn-hosts — seerr.franksdank.cloud vhost (Anubis guard + :8017 @seerr post-auth handler) and LAN DNS record, deployed in lockstep with the Authentik outpost provider seerr-proxy (external_host https://seerr.franksdank.cloud/, internal_host http://caddy:8017) and the Porkbun A record 192.168.0.143.
• /scripts/deploy-remote.sh — deploy_seerr handler (install_quadlet + pre-create bind dir + restart).
• /tests/seerr_quadlet.bats — new seam: boot persistence, self-healing, pinned tag, loopback-only publish, data-root contract, appnet join, no env/envfile secrets.
• /serverAppConfigs-docs/Dankclaw/Seerr.md, /serverAppConfigs-docs/index.md — service doc + index entry.
• Lessons: rootless podman uid mapping is continuous (container N → host 100000+N-1; uid 1000 → 100999 — an off-by-one chown EACCES-crash-looped first boot); new quadlet bind-source dirs must pre-exist on target (statfs 125 otherwise); new vhost TLS issues at Caddy reload (~15 s dns-01), not at first request.
• Authentik outpost claim cache fixed (root cause of the persistent /admin 403 despite homepage-admins membership): the outpost serves per-browser-session claims from authentik_providers_proxy_proxysession until row expiry — akadmin's live row predated the group add and carried ["authentik Admins"] only, and the 403 never generates an authentik event. Deleted his 4 rows; the next visit re-authorizes silently against the untouched authentik login session. Silent re-auth proven live: deleted qa-e2e-admin's 13 rows, ran the e2e admin spec against the stale storage state — 2 passed in 2.9 s, no login prompt. Mechanism + diagnosis SQL documented in /serverAppConfigs-docs/Infrastructure/Authentik.md (new "Outpost claim cache" section + failure-mode row); the doc's data-path references updated to the 2026-09-01 Containers layout.
• WireGuard credentials provisioned (Panama PA-12 conf → gluetun env inside the VM): tunnel egress verified (exit IP Panama, not residential), qB WebUI published per plan, and Prowlarr → qBittorrent wired — all five download clients (prowlarr, sonarr, radarr, lidarr, readarr) test 200 against live qB through the podman gateway 10.88.0.1:8080; Prowlarr's client created via its /downloadclient/schema template (hand-minimal body 400s on create-time test).
• /frankbook-pro/qbittorrent/quadlet/qbittorrent.container — pinned lscr.io/linuxserver/qbittorrent:4.6.7: latest had resolved to qB 5.x, whose 204 empty-body login success Readarr's retired client rejects as "Authentication failure" at runtime (not just in API creates — the prior session's "runtime cookie auth unaffected" assumption was wrong); 4.6.7 answers the legacy 200 + "Ok.". Gotcha: after 5.x ran, its conf rewrite left a PBKDF2 line 4.6.7 rejects — every login invalid credentials; rewriting the same line fresh (same password, 100k iterations) fixed it.
• /docs/Infrastructure/FrankBook-Pro.md — wiring, pin rationale, PBKDF2 version-flip gotcha, and E2E result (test torrent through the Panama tunnel at ~17 MB/s, INCOMPLETE → COMPLETE into Finished Torrents, deleted with data).
• /scripts/deploy.sh + /scripts/deploy-plan.py + /tests/dankserver_quadlet.bats — committed earlier today (b206f47): dankserver quadlet standardization units/handlers/routing.

2026-09-04 — Sonarr moved to the admin side; internal-only by construction

• Operator directive (fixing an earlier seerr/sonarr mix-up): Sonarr is an admin-side service and must be locally reachable only. The homepage registry flipped its card from the user category to admin and repointed the link at http://192.168.0.112:8989 (LAN-direct). Users keep requesting TV through Seerr, which stays on the user side behind the guard chain.
• Exposure closure: the unguarded public vhost sonarr.franksdank.cloud (direct reverse_proxy 192.168.0.112:8989, no auth) and its post-auth @sonarr handler were retired from the Caddyfile; the Pi-hole addn-hosts A record for the name was dropped with it. No public Porkbun record ever existed (NXDOMAIN at the authoritative zone), so after this change there is no WAN path to Sonarr at all — manual DNS could previously reach the vhost, which served the zero-auth UI to anyone who resolved the name manually at the public edge; the vhost (added in e0bade6, 2026-09-04) is retired.

2026-09-15/16 — frank-large: ik_llama.cpp cutover + the decode campaign

• Deployed (0c1e10a, via clean-worktree stash cycle around the peer's uncommitted files): halogen quadlet retired to ~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet, frank-large now a root systemd unit (scripts/frank-large.service → /etc/systemd/system/) running the committed serve script. Service verified live: health 200, reasoning completions, vision, slot persistence.
• Overnight performance campaign against the halogen bar (Frank: ~30 t/s decode). Result: decode stuck at 3.8-4.3 t/s across every safe configuration; prefill ~192 t/s = the CPU AVX512-VNNI ceiling. Full evidence chain in frank-large-ik-uncensored: (1) ~100 CPU↔GPU backend splits per token with the 3090 at 5% SM utilization — the decode wall is GPU-section launch/sync overhead, not compute or memory placement; (2) T1 ≈ T16 decode (3.97 vs 3.8-4.0) — the wall is a serial critical path; (3) ik's CUDA-graph capture aborts on qwen4exp's per-layer ggml_repeat_4d hc-broadcast ops; (4) mainline llama.cpp 37b53fd runs 1.0 t/s on this model (CPU MoE kernels 4x slower than ik's iqk) — ruled out; (5) the GGUF has no MTP head (mradermacher stripped it) — no speculative decoding; (6) 84 GiB of experts cannot fit 24 GiB VRAM, so halogen's all-resident 30 t/s is physically out of reach for this model on this host. Re-evaluate when mainline's hc-ops + graph handling mature (ggml-org PR #28901). /- Measurement integrity traps hit live: the systemd unit's Restart=always respawned pkill'd servers mid-experiment (phantom instances answering health — manual experiments must sudo systemctl stop frank-large.service first; plain systemctl --user lines are stale halogen-era docs); a concurrent -j16 build starved a decode measurement into a false timeout; warm-cache cat benchmarks invalidated a concurrent perf window.
• -ncmoe 42 (+6 GPU expert layers) measured 4.54 t/s decode at 22.0/24.6 GiB VRAM — documented as a tuning lever, not shipped (OOM margin at 262k ctx is operator's call). -ncmoe 48 -thp at 4.6 t/s holds a 110 GiB anonymous host buffer — co-tenant killer, never ship.
• Trash staged for approval: dankclaw ~/.AGENTTRASH/20260916-ik-partfiles/ (111 GiB, the 3 GGUF download parts, superseded by the concatenated verified GGUF), ~/.AGENTTRASH/20260915-ik-swap/ (halogen quadlet), repo tests/.AGENTTRASH/frank_large_quadlet.bats (retired seam), scripts/.AGENTTRASH/frank-small* (earlier retirement).
• Docs: serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md (new runbook), LLM + root indexes relinked, AGENTS.md topology row + Active Services section + bench-policy unit type. Committed 795b0b3; tests 300 ok with the single pre-existing not-ok-73 (comfyui selector bug, unrelated).

2026-09-16 — Dankclaw re-split: halogen returns as frank-large, llama.cpp takes the 3090 as frank-small

• /dankclaw/frank-large/quadlet/frank-large.container — halogen-flash-server restored as the active runtime (the 09-15 ik swap never removed the file) and retuned to the shape Frank asked for: HALOGEN_CTX 262144 → 200192, HALOGEN_KV_POOL_POSITIONS 401408 → 400384 (exactly 2 × 200192), HALOGEN_KV_SLOTS 3 → 2, HALOGEN_MAX_TOK 32768 → 24576 and HALOGEN_HOST_RESERVE_GIB 15 → 20 (arena −4.2 GiB, for the co-tenant), HALOGEN_VISION_TOWER unset (the tower moved to frank-small). Live ledger: budget 121 − 67.7 − 20 = 33.5 GiB against a need of 27.2, so no clamp; 68.0 GiB weights + 11.0 KV + 16.5 working = 95.4 GiB held, 14.7 GiB left for everything else (475 contiguous 2 MiB blocks). The co-tenant stall NOTE is back, and the boot logged 4,350 compaction stalls (285 failed) — tight and fragmented, not out of memory.
• /scripts/frank-small.env, frank-small.service, serve-frank-small.sh (new) — mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S: the -mtp build (the head is blk.64.nextn.* in the SAME file, so --spec-type draft-mtp runs the draft context on the main model's weights — no second copy, no -md) plus the BF16 projector, 131072 ctx / 1 slot, f16 KV, --jinja and --metrics pinned, port 7771, root systemd unit. 22,362 MiB of the card's 24,576 in use. Verified: /health, chat with reasoning_content, a tool_calls round trip, slot timing draft acceptance 0.66–0.94, and an image answered from the projector.
• /scripts/deploy-remote.sh — deploy_llm_large flipped back to halogen (retire the ik ROOT unit into ~/.AGENTTRASH/20260916-halogen-restore/host-units, pull + re-tag the image, wait_port_free 7770, install the Quadlet, start) and deploy_llm_small added (root unit, model-file gate, health on 7771). The port gate exists because the FIRST cutover attempt failed exactly there: pasta reported Address already in use about a second after the ik server exited, the unit died with code 126, and only Restart=on-failure recovered it. A redeploy now stops this service's own quadlet before waiting, or the gate would deadlock against itself.
• /scripts/deploy-plan.py, /scripts/deploy.sh, /scripts/test-changed.sh — frank-small added to SERVICE_ORDER and the Dankclaw inventory (the stale dankclaw:frank-npu entry dropped with it); the changed-test mapping for dankclaw/frank-large/* pointed at tests/frank_large_quadlet.bats, retired on 09-15, so the wrapper had been failing on that path ever since — now tests/frank_large_unit.bats.
• Retired to scripts/.AGENTTRASH/: the ik trio (frank-large.env, frank-large.service, serve-frank-large.sh), chat-template.jinja (only the ik runtime read it), and four benchmark harnesses bound to retired runtimes.
• /tests/frank_large_unit.bats rewritten for the Quadlet (the pool == 2 × ctx invariant, no vision tower, the port gate) and /tests/frank_small_unit.bats added (root-unit shape, MTP-from-the-file contract, KV budget vs the card, port separation). One assertion failed on its first run because the negative grep matched a COMMENT in the serve script — the test now strips comments before grepping the effective flags.
• shared/homepage/services.json, shared/prometheus/prometheus.yml, rules/llm-fleet.yml, grafana/.../llm-fleet.json — a frank-small card at 127.0.0.1:7771 and a frank-small scrape job at 192.168.0.143:7771; the frank-large job removed, because the halogen engine has no /metrics at all (its routes are /health, /cache and /v1/*), so a target there would have been red by construction; the dashboard's endpoint picker had been sourced from a vLLM-only metric, so a llama.cpp job could never be selected. Live: 11 targets, 7 up, 4 down (comfyui, gitea, services — all structural), and all three LLM homepage cards up.
• Harness (~/.omp/agent/models.yml + config.yml, dev box) — third provider llamacpp (dankclaw-3090) for :7771; frank-large ctx 262144 → 200192 and input: [text]; roles smol, tiny, task, vision → frank-small (the image path and the small/fast roles), slow/plan/designer/advisor → frank-large. Two failures found by driving it, both fixed: (1) frank-small's template accepts only xhigh|medium|low for reasoning_effort and 500s on anything else, and subagents run at high — every subagent on that endpoint died until compat.reasoningEffortMap folded the levels onto that set; (2) the ik runtime that had been serving frank-large rejected every request carrying tools (500 tools param requires --jinja flag), which is what killed a monitoring subagent earlier in the day. Both endpoints now answer tool calls, and a subagent completes a tool-using turn.
• Docs: frank-small-qwen38-gsq-rco.md (new runbook), halogen-flash-server.md re-activated with the 09-16 ledger, frank-large-ik-uncensored.md marked retired, LLM index, deployment flow, Observability and AGENTS.md topology/sections/lessons. Three AGENTS.md defects found on the way: a truncated sentence mid-file ("There is **no audio"), a harness path that does not exist (/home/dank/.omp/agent/models.yml — the harness runs on the dev box), and the benchmarking stop/mask recipe still describing frank-large as a root unit.
• Deploy: ./scripts/deploy.sh dankclaw --service frank-large --service frank-small — per-service health gates, then the e2e gate PASSED (31 passed, 1 flaky-retried). Suite: 323 seams ok, 0 failures.

2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do

• Built mainline llama.cpp 37b53fd with both the CUDA and Vulkan backends (build-dual; glslc and libvulkan were already installed), to test the design the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the iGPU, PLE in LPDDR, one hop per boundary.
• Device map found: CUDA0 = the 3090, Vulkan0 = the same 3090, and Vulkan1 = the iGPU (RADV STRIX_HALO, uma: 1, 114 GiB GTT). The iGPU being Vulkan1 and not Vulkan0 is exactly the detail that silently measures the wrong device.
• Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored): experts 75.0 GiB, per_layer_token_embd 32.8 GiB, everything else 3.2 GiB; 12 of 48 layers are full attention ([3,7,11,…,47]), so only they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read per token at 200k context.
• Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan, PLE on CPU) 32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 → 231.8 into a 64k cache, with the iGPU at 99–100 % busy; iGPU trunk + CPU experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design aborted in the scheduler — pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE) — while both per-layer cross-backend forms failed to load. Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only fallback side that works.
• The comparison that matters: at fleet-realistic depth the iGPU-only run matches halogen's field ~25 tok/s with the 3090 idle — but it commits 116 of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not deployable beside the fleet's ~28 co-tenants. Throughput parity, memory bankruptcy.
• Also recorded: GGML_CUDA_NO_PINNED is mainline's pinning guard (ik called it GGML_CUDA_NO_PINNED_WEIGHTS); systemctl mask is refused for a relinked root unit (frank-small's unit is a symlink into the repo); and llama-bench's tg instances run with n_prompt = 0, so -d is required to put KV into the measurement at all. Both services were stopped and restored (both healthy; halogen's ledger read 400384 positions, ~27.2 GiB against a 33.5 GiB budget with 15.2 GiB left).
• New page: x4-pair-experiment.

2026-09-16 (later) — the x4 pair, round two: the first conclusion was a parsing artifact

• Asked whether pure Vulkan changes the answer (both devices can run Vulkan — Vulkan0 = RTX 3090 (NV_coopmat2), Vulkan1 = RADV STRIX_HALO (KHR_coopmat)). It does not: the per-tensor design still aborts, byte-identically.
• The real gate is -ot, not the driver mix. has_tensor_overrides() is a required term of pipeline_parallel in llama-context.cpp:433 — the ONLY place in the tree that reads it — and that flag is what sets n_copies in ggml_backend_sched_new (ggml-backend.cpp:25). With n_copies == 1 a pre-allocated weight on a non-primary device has no backend, so the graph aborts at ggml-backend.cpp:941 with the leaf's own op name ((NONE)). -ot exps=CPU works only because the CPU backend is last in the list.
• Then the correction. llama-bench --help says -dev <dev0/dev1/...>: / splits a device LIST, , splits SWEEP values. So -dev CUDA0,Vulkan1 was two single-device runs — the earlier "CUDA + Vulkan cannot serve one model", the "load failures", and the OOMs were artifacts (the OOM was 111 GiB on the 24 GiB card). The bench table's dev column had been printing the truth all along.
• With the correct syntax two-device splits work, in both flavours (64k depth, -d 65536; placement proven by nvidia-smi sampled during the run):
  • pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 — 12.8 GiB on the 3090, 27.55 tok/s (pp512@64k 224.3)
  • cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 — 15.0 GiB on the 3090, 27.23 tok/s with pp512@64k 268.5
  • against the iGPU alone: 25.77 tok/s / 231.8 — so +7 % decode, +16 % prefill
• And the split still cannot reach the design's premise: whole layers move their experts with them (1.62 GiB per layer), the 12 attention layers are spread every 4th across the depth, and the 32.8 GiB PLE must stay on the main device — so KV cannot be consolidated on the 3090.
• Incident, same evening: the rootless podman pause process died — trigger NOT established (no OOM kill in journalctl -k for that window, so "the 116 GiB runs squeezed it out" is a guess, not a finding). Every podman command then failed (invalid internal status … could not find any running process), every container start failed on aardvark-dns binding 10.89.x.1:53 (the rootless netns holding the bridges was gone), and all 28 quadlets looped. podman system migrate — podman's own named remedy — panics in 5.7.0 (teardownNetworkBackend). Repair was a reboot: all 28 containers returned on their own (linger + WantedBy=default.target present), e2e gate green after.
• A second, pre-existing defect surfaced by that reboot: hermes.franksdank.cloud returned 502. hermes-webui is Network=host, so 8787 is a bare host-netns bind and netavark DNATs only published ports — from inside the Caddy container 192.168.0.143:8787 and 10.89.1.1:8787 are refused while host.containers.internal:8787 (169.254.1.2) answers 200. Fixed in the Caddyfile, the stale claim in its comment replaced with the measurement, and the seam that guarded it strengthened (it accepted any numeric upstream, so the LAN IP had been passing it; reverting now fails it — teeth verified).
• Both endpoints restored and healthy: frank-large 400384 positions, ~27.2 GiB against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page: x4-pair-experiment (rewritten).

2026-09-16 (night) — model cache cleared on instruction, then re-fetched, and the update was real

• Operator instruction: purge the model cache outright so no run can serve a stale payload. Done with one-time rm authorisation, and the two live payloads were cleared and immediately re-fetched — because deleting them without a re-fetch means neither endpoint can start.
• Cleared: the Uncensored i1-Q4_K_M payload (112G), the Q4_K_S part-fragments (105G), the retired kingjones STRIX and agentionai ROCmFP4 payloads (114G + 863M), an unreferenced Qwen3.5 vision dir (8.8G), and all of ~/.AGENTTRASH including the ik part-files (119G). Disk went 298 GB → 895 GB free; 814 GB after re-fetching what the services need.
• The re-fetch produced genuine updates. halogen's sidecars had been stale — overlay.hgn 2,477,677,120 → 2,572,466,560 B, overlay-speed.hgn 2,383,306,048 → 2,478,095,488 B — while w4b.hgn came back the same size (124,068,083,904). So "clear and re-pull" is an update mechanism here, and the sidecars are versioned in place: a stale copy is invisible from the file name. frank-small's 27B + mmproj re-fetched to hashes matching the Hub exactly and are unchanged.
• Trap: podman refuses a bind mount whose source directory is missing (Error: statfs /home/dank/models/halogen: no such file or directory), so frank-large looped on Restart=on-failure until mkdir -p recreated the empty directory — then the engine fetched the repo itself (HALOGEN_DOWNLOAD), which is the documented refresh path.
• Q4_K_S vs Q4_K_M, measured on the same host and build: +2.1 / +1.6 / +2.6 % (iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 % and +2.0 % for the two layer-split configs, ~0 on both pp512@64k. Host RAM bottomed at 10.0 GiB available in A* against ~5 GiB for Q4_K_M. A memory lever with a small speed dividend — nothing structural moves.
• IQ4_XS pulled in parallel with the bench, as asked: 97,473,155,360 bytes, sha256 b4787265…d0574f9d, matching the Hub's LFS digest. Kept as the fresh payload for the next comparison; Q4_K_S was cleared once the bench was done.
• Held back deliberately: models/checkpoints (23 GB of Flux 2 Klein — no unit mounts it, but its upstream is licence-gated, so it is not mine to destroy) plus the small tool dirs under models/ and models/logs, which is the evidence for this page.
• Verified after the refresh: /v1/models correct on 7770/7771, real completions returning pong on both, halogen's ledger 68.0 GiB weights locked / 11.0 GiB KV pool, 2 slots over one 400384-position pool, and 307 contiguous 2 MiB blocks left (240 before the refresh).
• Runbook: halogen-flash-server gains a "Refreshing the weights" note; the quant comparison is appended to x4-pair-experiment.

2026-09-17 — frank-small swapped to Gemma4-12B QAT (Q4_K_M)

• Cutover: frank-small (Dankclaw RTX 3090, :7771) replaced ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M — 7.38 GiB main + 175 MiB BF16 mmproj (with audio encoder) + 254 MiB mtp-gemma-4-12B-it.gguf assistant sidecar, all sha256-verified.
• Spec decode: the MTP head is now a separate gemma4-assistant file loaded via --model-draft (shares the target's last layers' KV, llama-model.cpp:2662), chained behind draftless ngram-mod: --spec-type draft-mtp,ngram-mod. Measured acceptance 0.38–0.46, mean draft len ~2.2.
• KV: q8_0 K and V; build-cuda rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON (required, else ~12x prefill collapse). CTX_SIZE unset → --fit on chose the full 262,144-token window; idle card usage 11,996 / 24,576 MiB (was 22,362).
• Template behavior: Gemma4 accepts every reasoning_effort level (all seven probed: 200) — the Qwen3.8 gate is gone; harness map kept as a safety net.
• Verified live: health 200, chat + reasoning_content, tool_calls, vision (live JPEG), deploy e2e gate passed (31 passed, 1 flaky crawl).
• Harness: models.yml frank-small contextWindow: 262144.
• Note: the boot log's Gemma4Assistant requires ctx_other to be set error is --fit's own probe — normal, self-recovers seconds later.
• Runbook: frank-small-gemma4-12b (active); frank-small-qwen38-gsq-rco retired 2026-09-17 as rollback target.

2026-09-17 (later) — frank-small: four full windows, max-length throughput sweep

• Slot ceiling measured: 1 full-ctx slot idles 11,996 MiB, 4 idle 22,346 MiB -> 3,450 MiB/slot, ~8,546 fixed; a 5th projects past the 24,576 MiB card. Deployed CTX_SIZE=1048576 / PARALLEL=4 (each slot the FULL 262,144 window; explicit CTX bypasses --fit on purpose — fit would divide the pool at parallel>1). The bats seam now pins the division and the budget inequality.
• Audio verified: JFK WAV via input_audio -> verbatim transcript.
• Batch sweep at max length (salted API probes, pp ~254k / tg @ ~253k ctx): ubatch 512 -> pp 912–924, tg 90–204; ubatch 1024 -> pp 962–965, tg 226–302 (deployed); ubatch 2048 -> OOM in the MTP draft's pp compute buffer (~3.3 GiB vs 2.2 GiB headroom).
• Negative results kept: draft n-max 4 + poll 100 decoded WORSE at 250k (230–257) — longer verification rounds re-read more long-ctx attention than the accepted token earns. No CUDA-graph flags exist in this build, so launch-overhead tuning is off the table.
• Concurrency caveat: a 254k prefill beside a decode drops it ~240 -> ~100–135 t/s; one engine, no per-slot QoS.
• Housekeeping during the sweep: the earlier live service was discovered to be an orphaned manual boot holding 7771 past the unit stop (kill -9'd); the sweep now refuses to measure a port it does not own. Peer frank-medium WIP was stash-held around the deploy and self-committed as 23f0978/74dda30 — nothing lost.
• Live after deploy: n_slots = 4, n_ctx_slot = 262144, idle 23,300 MiB, e2e gate 32 passed.

2026-09-17 (late) — homepage: live model info + user docs; deploy-runner lockout fixed

• Reported "www unreachable internally" — server side healthy end to end. LAN DNS → 192.168.0.143, TCP 443 open, Anubis challenge served, @home 200 from inside Caddy, apex/www redirects correct, no 5xx in the journal. Fault pattern matches the known client-side split-DNS issue (client off Pi-hole → WAN IP, no 443 forward). frank-large's 5 s health timeout during the audit was the documented halogen PING stall — 200/1 s minutes later, self-recovered; no restart performed.
• Fleet sweep: dankclaw 30 containers up (all with enabled quadlet units except systemd-hermes, a running container with no quadlet — logged as a boot-persistence gap); dankserver media stack + frank-medium all up, units enabled; FrankBook-Pro *arr/qB ports all answering (302/200). Undeclared stopped container test-vllm on dankserver — stray leftover, held for approval, not removed.
• Security audit: every vhost classified against the guard policy. Guarded (Anubis→Authentik): www, search, grafana, prometheus, loki, seerr, marreta, romm, files. Documented exemptions verified behaviorally: books anon /all → 302 login (gated), ai lands on sign-in, WAN 443 still unforwarded from LAN peers, and all admin-only names (prowlarr/lidarr/readarr/qb/pihole/hermes/hermesadmin) NXDOMAIN on public resolvers. No new exposure found; nothing to secure further.
• Homepage shipped (32751c0): the three LLM cards moved to the user page with live descriptions (halogen w4b iGPU / 35:30 layer-split MTP 27B / Gemma4-12B QAT vision path) and docs: /docs#models; new static user docs page at /docs (access chain, model guide, media how-tos, troubleshooting), routed in the post-auth listener, shipped by deploy_homepage, covered by new pages-seam tests.
• Runner lockout fixed (02a7876+): a stale pending state entry for the retired frank-default-sightsound endpoint made every --pending run exit 2 at the dispatcher before the e2e gate. The pending pass now prunes entries the target no longer allows into ~/.AGENTTRASH and keeps going; behavioral seam test added, superseded scoping test migrated.
• Deploy verified live: sightsound pruned (state 0, trashed 20260917), /docs 200 on the post-auth listener, all three model cards up, bats 329 green, e2e gate 32 passed.

2026-09-18 (morning) — frank-medium found stopped after a bench round

• Symptom: connection refused on 192.168.0.235:7770. Not network — the unit itself was inactive (enabled, unmasked, NRestarts=0).
• Root cause (from sudo journal): the AGENTS.md bench protocol was executed from PWD=/home/dank — stop + mask at 02:47:00, unmask
  • start at 04:21:27, then a final stop at 04:25:59 that was never followed by the protocol's step 3 (unmask/start + health verify). A bench round ended and left the endpoint down; no crash involved.
• Restore: GPU confirmed idle first (both R9700s ~60 MiB, no llama-bench/llama-server procs), then systemctl start frank-medium. Health 200 on first probe; /proc/MainPID/cmdline confirms the deployed geometry (--tensor-split 35,30, ctx 393216, 2 slots, batch 6144x512, draft-mtp); a real completion returns verbatim. Fingerprint b11009-fb27a525d = the pinned runtime.
• Lesson: the restore half of the bench protocol is not optional. If you bench from an agent session, the session owes the fleet a health 200 before it ends. Consider making the bench harness self-restoring.

2026-09-18 (midday) — frank-medium down: orphaned merge-build server held both R9700s

• Symptom: connection refused on 192.168.0.235:7770. Unit was inactive, enabled, unmasked (/etc/systemd/system/frank-medium.service symlinked to the repo unit, not /dev/null), NRestarts=0 — a clean stop, not a crash.
• Not the morning's case: the GPUs were not idle. rocm-smi read 15.5 GiB on GPU0 and 17.1 GiB on GPU1. ps named the holder: PID 2519965, llama-server from third_party/llamacpp/build-merge (gitignored scratch), orphaned at PPID 1, bound 127.0.0.1:17770, -c 16384 --split-mode layer --tensor-split 1,1 -ngl 99 -t 16.
• Timeline (sudo journal + /tmp/layersplit.log): start 08:14:02 → stop 09:19:44 → merge build spawned 09:19:47 → log reached model loaded / listening at 09:20 → nothing after. The merge-build load validation passed; the cleanup-and-restore half never ran.
• Why prod could not come back: the 35:30 split needs ~30.8 GiB on ROCm0 and ~28.9 on ROCm1. The stray owned roughly half of each card, so systemctl start would have died on allocation, not on config.
• Proof it was abandoned, not parked: /proc/<pid>/stat utime+stime identical across a 10 s sample (frozen CPU), zero established connections to :17770, loopback-only bind, 3h05m since last log write.
• Restore: preserved the test log to /tmp/layersplit.log.preserved-20260918, kill 2519965 (exited in <1 s, both cards → 60 MiB baseline), then systemctl start frank-medium. Health 503 → 200 in ~20 s (payload warm in page cache). /proc/MainPID/cmdline confirms deployed geometry (--tensor-split 35,30, ctx 393216, 2 slots, --flash-attn on, --spec-type draft-mtp); steady state ROCm0 32.9 GiB / ROCm1 30.7 GiB. Real completion returns verbatim with reasoning_content and spec-decode engaged: draft_n 39 / accepted 35 (89.7 %), tg 40.6 t/s. Fingerprint b11009-fb27a525d = the pinned runtime.
• Lesson: "GPUs idle" is a check, not an assumption. A stopped prod unit plus busy cards means something else took them — find the holder with ps/ss and prove it abandoned (frozen CPU ticks, no conns) before reclaiming. Scratch servers on loopback are invisible to the endpoint monitor, so they strand the fleet instead of failing loudly.

2026-09-18 (afternoon) — frank-medium: the host-RAM prompt cache doubles to 16 GiB

• The old cap was binding, not theoretical. --cache-ram caps the total bytes of saved prompt states in HOST RAM (server_prompt_cache), and alloc() refuses any single state larger than the whole cap with one WRN. The journal held 137 such refusals in the two days from 2026-09-16 — prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping — i.e. every conversation past ~170k tokens was uncacheable, and its next turn re-prefilled the whole thing.
• The cap is a token budget in disguise. A state is the serialized KV of a whole sequence plus its constant-size DeltaNet state plus the MTP draft context's own KV: 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) = 64 KiB per token, plus 4096 B for the draft context. So 8192 MiB held ~120k tokens and 16384 MiB holds ~240k — and a FULL 196608-token slot state (~12.8 GiB), which the old cap refused outright, now fits with ~3 GiB spare.
• Deployed change (1acca43): CACHE_RAM 8192 → 16384 in scripts/frank-medium.env, with the law and the journal evidence in the file. Host RAM, allocated on demand and never pinned, so no VRAM number moves; bounded rather than -1 because dankserver shares its 62 GiB with the media-stack co-tenants.
• New seam (tests/frank_medium_unit.bats): the cap must clear one full slot's state and may not claim more than half the host's RAM. It fails on the old value (8192 < 13056 MiB) and passes on the new one — checked both ways.
• Adjacent correction: the same block claimed 0 was the build's default for --ctx-checkpoints; this binary's --help says 32, so the 0 is a deliberate deviation. Fixed in the env comment and in the runbook's flag table.