docs: distill the managed skill library into the vault (waves 1-7)

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.

serverAppConfigs-docs/Infrastructure/Anubis.md

@@ -131,6 +131,55 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
    distinguish challenge from deny at this release; grep the body for
    `Access Denied` / `Making sure you` instead.
 
+## Guard-chain layer diagnosis (a user reports a guarded page "broken")
+
+Use when a guarded page (e.g. the `www.franksdank.cloud` homepage) reads as
+"broken / links gone / not loading" and the fault must be localized to
+server-side or client-side/session. Chain topology:
+`Browser → Caddy TLS edge → Anubis :8923 → Caddy :8016 → Authentik outpost
+→ Caddy :8017 (@home/@data handlers) → app/static`.
+
+Layer-by-layer checks (run from the host via ssh):
+
+1. **Caddy unit + guard containers**: `systemctl --user is-active
+   caddy.service`; `podman ps --format '{{.Names}}\t{{.Status}}' | grep -iE
+   'anubis|authentik|caddy'` — expect `Up … (healthy)`.
+2. **Anubis challenge page validity**: `curl -sk
+   https://www.franksdank.cloud/ --resolve www.franksdank.cloud:443:192.168.0.143`
+   — a healthy challenge contains `"anubis_challenge"` JSON with
+   `difficulty` and the main JS.
+3. **Outpost errors**: `podman logs --tail 40 authentik-outpost 2>&1 | grep
+   -iE 'error|exception|traceback|500|provider'` — routine dramatiq task
+   logs are normal.
+4. **Post-auth handlers from INSIDE the caddy container** (8016/8017 live in
+   its netns — `curl http://127.0.0.1:8017` from the HOST returns `000`
+   connection refused, which is EXPECTED, not a fault):
+   `podman exec caddy sh -c 'wget -q -S -T5 --header "Host:
+   www.franksdank.cloud" -O- http://127.0.0.1:8017/'` and the same for
+   `/services.json`, `/status.json`. Expect 200 + body. (BusyBox wget in
+   this container has NO `-H` flag — use `--header "Host: …"`.)
+5. **Files on disk**: `ls -la /srv/homepage/` and validate `status.json`
+   parses (`python3 -c "import json;json.load(open('/srv/homepage/status.json'))"`).
+
+Tool/behavior notes: the anubis container is distroless — `podman exec anubis
+sh` fails (`sh: executable not found`); inspect via its logs or the served
+challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
+podman IP) for ALL clients — rootless-podman published-port SNAT;
+clients are indistinguishable by IP and clearance is cookie-based. Do NOT
+mistake that for the bug. Homepage render: `app.js` does
+`Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
+failure the catch sets `#summary` to `status unavailable: <err>` and the
+grid stays EMPTY — which a user reads as "links gone / not loading." A
+30-service status.json with a 9-service services.json renders fine
+(stateByName lookup).
+
+**Decision:** if every layer is healthy (challenge valid, outpost clean,
+handlers 200, files render), the fault is client-side/session. STOP
+thrashing the live system — repeated failed browser challenges share the
+masked 10.89.1.22 identity. Get the authenticated view: ask for the exact
+symptom (challenge spinning? login loop? empty grid? "status unavailable"
+text?) or the authentik login to drive the page in a real browser.
+
 ## Verify live
 
 ```

serverAppConfigs-docs/Infrastructure/Authentik.md

@@ -342,6 +342,168 @@ The silent re-auth path was proven end-to-end: after deleting
 `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
 state and passed with no login prompt (2 tests, 2.9 s).
 
+## Guard-chain flip recipe (adding a guarded vhost)
+
+Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
+client → TLS edge → Anubis :8923 → outpost decision → internal
+`http://:8017` listener → Host-matched handler → real upstream.
+
+**Order (never reorder):**
+
+1. **Provider first.** Create the outpost ProxyProvider before touching
+   routes, or authed traffic hits the outpost error page instead of the app.
+   Fleet convention for `external_host` is the exact browser-facing origin
+   (the inventory table carries both bare origins and trailing slashes;
+   routing matches on hostname, verified 2026-09-02).
+   `internal_host = http://caddy:8017`, all three flows set, and an
+   Application attached — the two provisioning rules above.
+2. **Route flip in ONE commit/deploy**: the vhost becomes
+   `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and
+   inside the `http://:8017` listener add `@<svc> host <vhost>` +
+   `handle @<svc> { reverse_proxy <upstream> }`. Anubis guard set, Caddy TLS
+   edge, `:8017` routing table, and provider list change together.
+3. Deploy via the repo deploy flow (commit first; deploy pushes main itself).
+
+**Verification (all three):**
+
+- Unauthenticated probe from a client: `curl -s https://<vhost>/` → HTTP 200
+  challenge page; grep the body for `challenge|not a bot|anubis` (~8
+  markers on a live challenge). A 502 here instead of the challenge = chain
+  mis-wired.
+- Post-auth routing proof: grep the LIVE config inside the container —
+  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
+  after the deploy prints its reload line; confirm the reload actually ran,
+  don't trust exit 0.
+- Outpost reachability: `podman exec authentik-server getent hosts caddy`
+  resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
+
+**Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
+000 (connection refused) — the post-auth listener lives in the caddy
+container's netns; probe inside it or grep the config. Non-browser clients
+cannot solve the JS challenge — before flipping, audit service-to-service
+traffic: ingestion/push paths must target internal addresses (`loki:3100`),
+never a public hostname of a guarded service. Exempt-by-construction
+categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP
+endpoints, services with their own complete auth — document the exemption
+in the vhost comment.
+
+## Driving the ORM (`ak shell`)
+
+All authentik object work happens through the server container's ORM shell —
+no repo change, no deploy (these providers are NOT repo-managed; capture new
+provider+application pairs as a blueprint or setup script to avoid drift).
+
+```bash
+ssh dankclaw 'podman exec -i authentik-server ak shell' <<'EOF' | sed -n '/229 objects/,$p'
+from authentik.core.models import User, Group
+...
+EOF
+```
+
+- The heredoc IS the program: `ak shell` reads all of stdin and execs it
+  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
+  one).
+- Bootstrap prints JSON log lines, then
+  `### authentik shell (2026.8.3)` and `229 objects imported automatically…`
+  — pipe through `sed -n '/229 objects/,$p'` to drop the noise.
+- 229 models are pre-imported. Import paths (each wrong guess costs a failed
+  exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
+  `PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
+  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
+  `LDAPProvider` → `authentik.providers.ldap.models`;
+  `authentik.core.models.application` does NOT exist as an import path; there
+  is no `authentik.policies.group` module in this build.
+- For live DB truth prefer the postgres container
+  (`podman exec authentik-postgres psql -U authentik -d authentik -At -c
+  "select * from …"`) — **query with `select *` and stop guessing column
+  names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
+  FieldError — `type` moved to subclass tables; the outpost `enabled` column
+  may not exist under that name; provider pk is `oauth2provider_ptr_id`).
+
+## User provisioning playbook (user-level account)
+
+There is no `User.create_user` classmethod, and `is_superuser` is NOT a
+constructor kwarg (it is a property derived from group membership) — passing
+it raises `TypeError: User() got unexpected keyword arguments`:
+
+```python
+from authentik.core.models import User, Group
+u = User(username="ridley", email="ridley@franksdank.cloud", name="Ridley", is_active=True)
+u.set_password("...")        # password arrives on stdin only, never argv
+u.save()
+for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
+    u.groups.add(Group.objects.get(name=g))
+```
+
+`path` defaults to `users` (regular). The access map (verified live
+2026-10-04): there are NO per-app policy bindings on the proxy providers
+(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
+access is decided entirely by GROUP MEMBERSHIP consumed downstream —
+
+- `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
+  regex gate) and the homepage admin panel; regular users get only
+  `category: user` panels from `services.json`.
+- `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
+  "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
+  "Open WebUI roles" property mapping → `roles` claim. Verify live with
+  `u.app_entitlements(app)`.
+- `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
+  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
+- `Non-Admin Users` = the plain-user marker group (precedents: alex,
+  qa-e2e).
+- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
+  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
+- Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
+  Plex, Seerr, Marreta) need no per-user group: any authenticated user
+  passes the outpost.
+
+Verify login behaviorally — `authenticate()` is the exact PasswordStage code
+path:
+
+```python
+from django.contrib.auth import authenticate
+assert authenticate(request=None, username="ridley", password="...")   # -> User
+assert authenticate(request=None, username="ridley", password="nope") is None
+```
+
+Look flows up by slug, not name:
+`Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
+"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
+Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
+enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
+
+More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
+can be null (guard with `getattr`); `LDAPProvider` has
+`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
+`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
+`OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
+(use `type(p).__name__`); `User` has no `.status`.
+
+Never rotate `AUTHENTIK_SECRET_KEY` to fix claim issues — it invalidates
+every outpost session fleet-wide; purge the stale `proxysession` rows
+instead (section above).
+
+## Probing the outpost from inside the netns
+
+Rootless podman gives each container its own netns; **host `ss -ltn` cannot
+see listeners inside other containers' namespaces** — a missing listener on
+the host is an observation artifact, not a fault. Distroless images have no
+shell (`podman exec authentik-server sh` fails), so probe by joining the
+namespace:
+
+```
+podman run --rm --network container:<container> --entrypoint /bin/busybox \
+  docker.io/library/busybox:latest \
+  wget -q -T 4 -t 1 -O /dev/null -S "<url>" 2>&1 | head -8
+```
+
+- Quote URLs over ssh: the remote shell is zsh and bare `?` in query strings
+  triggers globbing (`no matches found`).
+- Boot transience: immediately after a restart the same probe can return 503
+  (configure task pending); poll until the answer changes before concluding.
+- `podman inspect --format "{{.Config.NetMode}}"` errors on this podman
+  version — the field is `.HostConfig.NetworkMode`.
+
 ## Failure modes
 
 | Symptom | Cause | Fix |

serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md

@@ -0,0 +1,158 @@
+---
+type: reference
+title: Homepage Status Probes
+description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix).
+tags: [homepage, status, probes, e2e, homepage-status]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Homepage Status Probes
+
+The homepage is a static site —
+`shared/homepage/{index.html,admin.html,app.js,style.css,services.json,docs.html}`
+— served from `/srv/homepage` on **dankclaw** only (`deploy_homepage` in
+`scripts/deploy-remote.sh` refuses other targets). A checker
+(`scripts/homepage-status.py`, run by the `homepage-status.timer` unit) probes
+each card's target and writes `/srv/homepage/status.json` (mode **0644** so
+Caddy can serve it — losing the mode 403s the page).
+
+## The checker's vantage and its three false-down shapes
+
+The checker probes TCP/HTTP **from the host netns**. It only has three states
+— `wip` | `up` | `down`; there is no "unknown", so an unprobeable service can
+never render honestly and must be dropped or re-pointed, not "fixed".
+
+1. **TLS-only app probed with `method:"http"`.** Plex Media Server serves TLS
+   only on `32400`; plain-HTTP GET → TLS alert → curl exit 52, code `000` →
+   card says down while PMS answers `200` over HTTPS. Fix:
+   `"method":"tcp"` (connect is the honest liveness signal; the public URL
+   stays HTTP because Caddy re-keys).
+2. **Retired port.** The Hermes card probed `127.0.0.1:8090` after the
+   compose→quadlet move moved the dashboard to `9119` and the webui to
+   `8787`; nothing listened on 8090, so a healthy stack read `down`.
+3. **Unpublished port / wrong netns.** `gitea-postgres` (`db:5432`, net
+   `systemd-gitea`), `searxng-valkey` (`valkey:6379`, net `systemd-searxng`)
+   and `alloy` (OTLP 12345) publish **no** ports. Container DNS aliases are
+   netns-scoped, so from the host these are unreachable *by construction* —
+   no probe value makes them honest. Drop those cards.
+
+A host-vantage probe can only reach a listener **published to the host**
+(rootlessport/pasta); unpublished ports are invisible from the host netns.
+See [[Container Host Reachability]] for the netns rules.
+
+## Triage procedure
+
+1. Read the card's probe target in `shared/homepage/services.json`; note
+   method (`tcp` vs `http`).
+2. Does that port have a **host-visible listener**?
+   `podman ps -a --format '{{.Names}}|{{.Status}}'`, then per container
+   `podman inspect <c> --format '{{json .NetworkSettings.Networks}}
+   {{.HostConfig.NetworkMode}}'` and `podman port <c>` (empty output =
+   published nothing → structurally unprobeable from the host).
+3. Probe from the **correct vantage**: from a bridge-net container the host
+   is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on
+   `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`),
+   which belongs to the default `podman` net (`10.88.0.0/16`). Probe with
+   bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so
+   they expand **inside** `podman exec bash -c`.
+4. Classify: real outage (container exited — read `State.ExitCode`: 137 =
+   SIGKILL, 143 = SIGTERM/manual stop; `OOMKilled=false` rules out the OOM
+   killer) vs false down (running + unreachable target).
+5. Fix: re-point or re-method for HTTP/TLS mismatches; **drop** cards whose
+   only exposure is an internal netns; never invent a probe target.
+6. Verify by re-running the checker path, not by eyeballing: after deploy,
+   re-read `/srv/homepage/status.json` and diff the `down` list
+   before/after.
+
+Removing a card's `probe` key without a code change crashes the checker
+(`svc["probe"]` KeyError) — dropping the whole entry is the only zero-code
+fix. The `description` field is consumed only by `app.js` `descHTML()`, so
+adding a `details` object (role/spec/inputs/use) is schema-safe:
+`tests/homepage_config.bats` checks only name/category/section/probe.
+
+## Local render check (before every deploy)
+
+The page fetches `services.json` + `status.json` at load; `file://` is
+CORS-blocked, so serve over HTTP:
+
+```bash
+cd shared/homepage
+python3 -c "import json; s=json.load(open('services.json')); \
+  json.dump({'services':[{'name':x['name'],'state':'up'} for x in s]}, open('status.json','w'), indent=2)"
+python3 -m http.server 8899 --bind 127.0.0.1
+```
+
+Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s**
+(panels render after the async fetch; checking immediately returns 0 panels),
+extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/
+`.desc-inputs`/`.desc-use` from the card's `details` object, other cards a
+plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for
+visual proof. Expected local noise: `deployed.js` 404s (baked in at deploy
+time) — harmless.
+
+The scratch `status.json` is untracked and NOT git-ignored — move it to
+`shared/homepage/.AGENTTRASH/` after the check (repo policy: never `rm`).
+
+CSS palette lives in `style.css` `:root` (`--fg` bright green, `--dim` dim,
+`--head` pale, `--link` cyan, `--wip` yellow, `--down` red). Add new tones as
+`--*` vars and include any new `.desc-*` class in the `html[data-bare="1"]`
+reset so `?bare=1` strips it to black.
+
+## Deploy
+
+```
+./scripts/deploy.sh dankclaw --service homepage --skip-e2e
+```
+
+Copies the assets to `/srv/homepage`, regenerates `deployed.js` (git commit
+datetime), restarts the `homepage-status` timer. Verify live:
+`ssh dank@192.168.0.143 'grep … /srv/homepage/…'`.
+
+**`--skip-e2e` is justified for homepage-only changes when the Mac mini is
+asleep**: the e2e admin crawl (`tests/e2e/specs/11-admin.spec.ts:24`) walks
+Prowlarr at `192.168.0.112:8989` on FrankBook-Pro; an asleep mini (ARP STALE,
+no remote wake) times out that test — a pre-existing environmental failure,
+not a regression. All non-Mac-mini e2e tests pass.
+
+## e2e raw-IP *arr failures — probe from the browser's box, not a peer
+
+When `11-admin` fails with `ERR_CONNECTION_TIMED_OUT`, the instinct is "the
+Mac is asleep". Sometimes true; not always. Playwright runs on the dev box,
+so a healthy probe from dankclaw proves nothing about the **browser's** path:
+
+```bash
+# 1. From the box that RUNS the browser:
+curl -s -o /dev/null -w '%{http_code}\n' --max-time 7 http://192.168.0.112:8989/
+# 2. From a LAN peer:
+ssh dankclaw 'for p in 8989 7878 9696; do curl -s -o /dev/null -w "$p %{http_code}\n" \
+  --max-time 6 http://192.168.0.112:$p/; done'
+# 3. Liveness (ICMP, not TCP):
+ping -c 2 -W 3 192.168.0.112
+```
+
+| peer probe | dev-box probe | ping | meaning |
+|---|---|---|---|
+| 200 | 200 | ok | transient — re-run the spec |
+| timeout | timeout | fails | Mac asleep; wake, ports return in seconds |
+| **200** | **rc=28** | **ok** | **path problem, not sleep** — ICMP allowed, TCP blocked. Waking cannot help |
+
+Third row: confirm the Mac is innocent through the dankclaw jump
+(`pgrep -fl socat`; `netstat -an | grep -c 8989`; loopback
+`curl http://127.0.0.1:8989/` → 200; `/usr/libexec/ApplicationFirewall/socketfilterfw
+--getglobalstate` → disabled). Forwarders up + loopback 200 ⇒ the block is
+between the dev box and the Mac.
+
+Don't hold a deploy hostage to a known-environmental red gate: verify the
+deployed service **directly** (its own ledger/health/real completion), state
+the gate was skipped or partially red and why (`--skip-e2e` prints the
+warning, so the skip is recorded), and report the residual gap plainly
+("29/30 e2e pass; 11-admin fails on Sonarr's raw IP from the dev box; peer
+probe returns 200; deploy verified by other means"). Do not weaken or skip
+the spec — it guards a real SSO-plus-RBAC contract.
+
+## Related
+
+- [[Container Host Reachability]] — the netns rules behind the vantage
+- [[Observability Provisioning]] — the monitoring stack reading the same fleet
+- [[Deploy State Triage]] — service-scoped deploys and the e2e gate
+- [[Testing Policy]] — the seam/e2e contract this page's checks live in

serverAppConfigs-docs/Infrastructure/LAN-DNS.md

@@ -158,6 +158,119 @@ The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the
 | HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
 | Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
 
+## Per-client fix: resolver drop-in + browser DoH bypass
+
+Symptom: a LAN machine gets `connection timeout` (curl rc=28) on
+`https://*.franksdank.cloud` while the server is healthy. Root cause: the
+public DNS hands out the WAN IP `71.227.227.172`, which has **no 443
+port-forward** (filtered). Any resolver that is NOT Pi-hole produces the WAN
+answer → SYN to a filtered port → timeout. There are TWO resolver layers to
+check, in order:
+
+**Layer 1 — systemd-resolved (fixes curl/shell).** A route-only drop-in
+sends `*.franksdank.cloud` ONLY to Pi-hole and never to the default-route
+servers:
+
+```bash
+dig +short www.franksdank.cloud @192.168.0.143   # expect 192.168.0.143
+dig +short www.franksdank.cloud @1.1.1.1         # expect 71.227.227.172 (the trap)
+getent hosts www.franksdank.cloud                # what the OS actually resolves
+curl -sk --resolve www.franksdank.cloud:443:192.168.0.143 \
+     https://www.franksdank.cloud/ -o /dev/null -w '%{http_code}\n'   # 200 = server fine
+
+sudo mkdir -p /etc/systemd/resolved.conf.d
+sudo tee /etc/systemd/resolved.conf.d/franksdank-lan.conf >/dev/null <<'EOF'
+[Resolve]
+DNS=192.168.0.143
+Domains=~franksdank.cloud ~franksdank.casa
+EOF
+sudo systemctl restart systemd-resolved && resolvectl flush-caches
+```
+
+The `~` (route-only) prefix is load-bearing: it routes only those domains to
+Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`;
+`resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`.
+
+**Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If
+Layer 1 is verified but the BROWSER still times out, the browser is doing
+its own DNS over HTTPS and never asks systemd-resolved — it queries the TRR
+provider (Cloudflare et al.) directly → WAN answer → timeout. Prove it:
+`curl -sk -m 6 https://71.227.227.172/` → rc=28 (the WAN answer is a dead
+end). Diagnosis details:
+
+- Firefox profile may be at the XDG path `~/.config/mozilla/firefox/`, NOT
+  `~/.mozilla/firefox/` — find it via the running process if the default
+  path is empty.
+- `grep network.trr. <profile>/prefs.js` — **no entries = shipped default,
+  and since Firefox 113 the shipped default is DoH ON.**
+
+Fix (surgical, keeps DoH for everything else): Firefox → Settings → Privacy
+& Security → **DNS over HTTPS** → "enabled for these domains" → **Add…** →
+`franksdank.cloud` (and `franksdank.casa`). The exception list = domains
+that do NOT use DoH; they fall back to the system resolver, which has the
+route-only drop-in. Or set DoH to **Off** entirely (coarser). Chrome:
+Settings → Privacy and security → Security → "Use secure DNS" → off, or
+"Use the same settings as your system". Do NOT hand-edit `prefs.js` while
+the browser is running — it holds prefs in memory and rewrites the file on
+exit, clobbering the edit.
+
+This is a per-client fix: every LAN client that must reach the homelab by
+name needs Pi-hole as its DNS (or the drop-in) AND a browser that does not
+bypass it with DoH. The server side needs no change. Verified on blankpad
+2026-09-21: the drop-in was present since 09-12, curl returned 200, and
+Firefox 155 with default (ON) DoH was the remaining hole.
+
+## Porkbun zone ops (franksdank.cloud)
+
+Operate the public DNS zone via the Porkbun API v3, run **on Dankclaw** so
+the API key/secret sourced from `/home/dank/serverAppConfigs/.env`
+(`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) never leaves the host. Pipe
+scripts to the target with `ssh -o BatchMode=yes dankclaw 'bash -s' <
+/tmp/script.sh`.
+
+Endpoints (all POST, JSON body):
+
+- List: `https://api.porkbun.com/api/json/v3/dns/retrieve/franksdank.cloud`
+  — auth via headers `X-API-Key` + `X-Secret-API-Key` (Bearer is wrong
+  here). Response shape is **top-level `.records[]`**, NOT
+  `.content.records[]` as some docs claim — parse with
+  `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`.
+- Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey).
+- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`,
+  `type`, `content`, `ttl`, `prio`).
+
+Quirks that bite (verified live 2026-08-31):
+
+1. **`name` is a RELATIVE label on edit.** Passing `ai.franksdank.cloud`
+   yields the record `ai.franksdank.cloud.franksdank.cloud`. Pass the bare
+   label (`ai`) and the API appends the zone itself. Verify after every
+   rename with a retrieve before declaring done.
+2. **Rate limit answers as an HTML 404-style page, not JSON.** A batch of
+   rapid mutations makes some calls return `<!DOCTYPE html>...` where jq
+   then errors (`Cannot iterate over null` / `Invalid numeric literal`). Do
+   not assume the call succeeded — sleep ~8 s and retry the failed ids;
+   verify by re-listing the zone at the end.
+3. Deletes/renames are per-record-id: always retrieve first, map name→id,
+   then mutate by id.
+
+Rename procedure (vhost rename = DNS + Caddy together):
+
+1. `retrieve` → collect ids for the renamed names.
+2. `edit` each id with the new **relative** label, preserving
+   type/content/ttl.
+3. Re-retrieve and confirm `<new>.franksdank.cloud A <ip>`.
+4. In-repo lockstep before deploy: rename the Caddyfile vhost block AND the
+   Pi-hole `dankclaw/pihole/addn-hosts` line in the same commit — a test
+   fails the suite if any Caddyfile vhost lacks an addn-hosts A record on
+   192.168.0.143.
+5. Deploy via `scripts/deploy.sh dankclaw` (plans pihole+caddy from changed
+   paths). Capture the suite's true exit with
+   `out=$(./tests/run.sh 2>&1); rc=$?` — piping to `tail` masks the exit
+   code behind `tail`'s own status and has shipped untested commits here.
+6. Verify live: `dig +short <name> @192.168.0.143` then
+   `curl -o /dev/null -w %{http_code} https://<new-name>` (expect 200/302;
+   first hit triggers dns-01 issuance, which works without inbound port 80).
+
 ## See Also
 
 - [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)

serverAppConfigs-docs/Infrastructure/Observability Provisioning.md

@@ -0,0 +1,95 @@
+---
+type: reference
+title: Observability Provisioning
+description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification.
+tags: [observability, prometheus, grafana, provisioning, deployment]
+timestamp: 2026-10-05T00:00:00-07:00
+---
+
+# Observability Provisioning
+
+The monitoring stack is **provisioned as code**: dashboards, datasources,
+scrape jobs and alert rules live in the repo and ride the normal deploy flow.
+No hand-edits inside containers — they are wiped on recreate and never
+committed.
+
+## The config triad
+
+| Repo path | What it holds |
+|---|---|
+| `shared/prometheus/prometheus.yml` | scrape jobs (global + per-host) |
+| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
+| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
+
+Targets and datasources reference **container names** (`loki:3100`,
+`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
+by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
+are host-netns services, not containers. Prometheus 3 refuses HTML responses
+as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
+exporter (see [[Stale Signal Audit]]).
+
+## The deploy path
+
+`scripts/deploy-remote.sh` carries two copier functions:
+
+- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
+
+`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
+`shared/grafana/**` to the `prometheus`/`grafana` services, so
+`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
+touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
+exist, the copier functions exist, and the provisioned paths match the units'
+mounts.
+
+## Entry points
+
+**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
+vantage first: container name if the exporter is a container on the same
+netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
+if it is an HTTP probe, not a metrics endpoint.
+
+**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
+Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
+Rules reload with the prometheus service.
+
+**Add a datasource** — drop a YAML file into
+`shared/grafana/provisioning/datasources/`. Point it at the container name of
+the backend (`http://prometheus:9090`), not loopback.
+
+**Add a dashboard** — drop the JSON into
+`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
+unified storage, not (only) the SQLite table — grep the API, not the DB, when
+verifying (see the verify section of [[Fleet Audit and Posture]]).
+
+## Verify through the APIs, not the filesystem
+
+```bash
+# prometheus: rules and targets actually loaded
+curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
+curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
+  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
+# grafana: datasource + dashboard provisioned
+curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
+```
+
+Health answers from a stale process and provisioned resources can hide in
+unified storage — a file existing on disk is not proof the running server
+loaded it. See [[Stale Signal Audit]] for the never-green-target decision
+table (refused = outage, 404/HTML = config defect).
+
+## Current delivered state (2026-09-17)
+
+Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
+loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
+unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
+via `host.containers.internal`) → signal-cli. Some targets read down for
+structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
+the blackbox exporter. See [[Observability]].
+
+## Related
+
+- [[Observability]] — stack topology and the alert delivery chain
+- [[Stale Signal Audit]] — classifying down targets before "fixing" them
+- [[Quadlet Lifecycle]] — the units that host these services
+- [[Deploy State Triage]] — how the planner selects these services