docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
2026-10-18 13:15:43 -07:00
parent 466fd477d8
commit e7f0d93c86
67 changed files with 2619 additions and 1152 deletions
@@ -8,7 +8,7 @@ timestamp: 2026-09-17

 # Alertmanager

-Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel and reached nobody.
+Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel (`shared/grafana/provisioning/dashboards/llm-fleet.json`, the "Firing fleet alerts" panel) and reached nobody.

 **Host:** Dankclaw — **Port:** 9093 (loopback) — **Docs:** https://prometheus.io/docs/alerting/latest/alertmanager/

@@ -17,13 +17,14 @@ Alertmanager receives firing rules from Prometheus, groups and dedupes them, and
 | Item | Value |
 |------|-------|
 | Image | `docker.io/prom/alertmanager:latest` |
-| Published | `127.0.0.1:9093` → 9093, loopback only (see [[Observability]] → Exposure) |
+| Published | `127.0.0.1:9093` → 9093, loopback only — never a wildcard (`alertmanager.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9093` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
 | Unit | `dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service` |
+| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alertmanager.container:49-54`); installed at `~/.config/containers/systemd/alertmanager.container` and reporting `generated` + `active` (`systemctl --user is-enabled|is-active alertmanager` 2026-10-18) |
 | Config | `/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy |
 | Data | `/opt/alertmanager/data` (silences, notification log) |
-| Health | `wget --spider -q http://localhost:9093/-/healthy` |
-| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` |
-| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h |
+| Health | `wget --spider -q http://localhost:9093/-/healthy` (`alertmanager.container:43`; the deployer waits on the same path on the published loopback, `scripts/deploy-remote.sh:532`) |
+| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` (`shared/prometheus/alertmanager.yml:27-30`) |
+| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h (`shared/prometheus/alertmanager.yml:21-25`) |

 ## Why the webhook address is `host.containers.internal`

@@ -36,7 +37,9 @@ The Signal relay is a host-side process, and the address it must bind is the one
 - POST `/alertmanager` — the Alertmanager webhook handler; formats the group and sends it.
 - GET `/health` — liveness for the deployer's health wait.
 - Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h `repeat_interval` during a relay hiccup.
- Sends through `/usr/local/bin/signal-cli -a +12062476336 -o json send +12062476336 -m <text>` — Frank's existing account (state dir `318311`); the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
+- Sends through `/usr/local/bin/signal-cli -a +12062476336 -o json send +12062476336 -m <text>` (`scripts/signal-relay.py:88-104`) — Frank's existing account (state dir `318311`, `ls ~/.local/share/signal-cli/data` 2026-10-18: `318311`, `318311.d`, `accounts.json`); the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
+
+Delivery is measured, not assumed. The relay's journal carries successful sends for every rule in the set — `journalctl --user -u signal-relay` (2026-10-18) ends `signal-relay: sent to +12062476336: [CRITICAL] LlmEndpointDown (1 firing)`, `… [WARNING] LlmSpecDecodeOff (1 firing)`, `… [INFO] LlmPrefixCacheCollapse (1 firing)`, with matching `(1 resolved)` lines and `suppressed duplicate` where the 15-minute window collapsed a repeat. Each of those lines is printed only after `signal-cli` exits 0 (`scripts/signal-relay.py:124-126`), and `ss -ltnp` (2026-10-18) shows `python3` bound to `192.168.0.143:18093`, with `curl -s http://192.168.0.143:18093/health` → `ok`. What the phone itself shows cannot be checked from this host. *(unverified 2026-10-18: handset receipt)*

 ## Deployment

@@ -51,8 +54,8 @@ The Signal relay is a host-side process, and the address it must bind is the one
 - The webhook URL and the relay's listen port are a **cross-file contract**: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. `tests/observability_quadlet.bats` pins both halves.
 - The container address is `alertmanager:9093`, never the published port: Prometheus is in a container and the port is loopback-only.
 - signal-cli account state is on the host, not in the container — a wiped `~/.local/share/signal-cli` silences delivery while every status page stays green.
- signal-cli 0.14.7's `-a` flag parses an all-digit value as a **phone number** and never falls back to the `accounts.json` name: the account's numeric name (`318311`) dies with `Invalid account (phone number)`, while `-a +12062476336` works. The message goes via `-m`, not a positional — a positional message is parsed as a second recipient (`Invalid phone number '…'`).
- A failed send is **not** marked in the dedupe map, so the next Alertmanager repeat retries it — but the relay returns 200 to the webhook regardless (a relay bug must not 500 Alertmanager), so a dead signal-cli shows up only in the relay's journal, not in Alertmanager's.
+- signal-cli 0.14.7 (`signal-cli --version` on Dankclaw, 2026-10-18) parses an all-digit `-a` value as a **phone number** and never falls back to the `accounts.json` name: the account's numeric name (`318311`) dies with `Invalid account (phone number)`, while `-a +12062476336` works (`scripts/signal-relay.py:36-41`). The message goes via `-m`, not a positional — a positional message is parsed as a second recipient (`Invalid phone number '…'`).
+- A failed send is **not** marked in the dedupe map (`scripts/signal-relay.py:124-129`), so the next Alertmanager repeat retries it — but a valid webhook always gets a 200 back, even when the handler raises (`scripts/signal-relay.py:155-161`: a relay bug must not 500 Alertmanager; only an unparseable payload gets a 400), so a dead signal-cli shows up only in the relay's journal, not in Alertmanager's.

 ## See Also

@@ -8,7 +8,7 @@ timestamp: 2026-09-11

 # Alloy

-Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). It forwards container and journal logs to Loki.
+Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below.

 **Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest

@@ -19,22 +19,24 @@ Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-
 | Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) |
 | Published | `127.0.0.1:12345` → 12345 (its own UI/health port) |
 | Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` |
-| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/` |
+| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) |
 | Inputs | host journald socket, podman container logs, `/var/log` |
-| Output | `http://loki:3100/loki/api/v1/push` |
-| Successor to | Promtail (EOL 2026-03-02) |
+| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) |
+| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) |
+| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) |
+| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" |

 ## Why Alloy and not Promtail

-Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
+Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`).

 ## Configuration

 `shared/alloy/config.alloy` is the source of truth:

- Reads journald logs from the host's `/run/systemd/journal/socket`.
- Reads container logs from `/var/lib/containers/storage/...` (podman).
- Forwards to Loki with `job` labels.
+- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`).
+- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`).
+- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).

 ## Volumes

@@ -47,7 +49,17 @@ Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs

 ## The UI port is published on purpose

-Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured"). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` and publishes `127.0.0.1:12345` — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider.
+Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
+
+## Delivery state (2026-10-18): healthy, shipping nothing
+
+The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
+
+- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
+- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
+- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
+
+No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.

 ## Gotchas

@@ -35,37 +35,67 @@ graph LR
 |------|-------|
 | Runtime | Quadlet unit `anubis.service` (`dankclaw/anubis/quadlet/anubis.container`) — boot-persistent, self-healing |
 | Image | `ghcr.io/techarohq/anubis:v1.27.0`, run under a `localhost/` alias tag (see gotcha 1) |
-| Ports | `:8923` internal (no published port); metrics on `:9090` reachable only inside `dankclaw_default` |
+| Ports | `:8923` internal (no published port — `podman ps` shows no port mapping, 2026-10-18); metrics on `:9090` reachable only inside `dankclaw_default` (`dankclaw/anubis/quadlet/anubis.container:3,58-61`) |
 | Config surface | flags via flagenv — `TARGET=http://caddy:8016` sets `--target`, `POLICY_FNAME=/policy/policy.json` sets `--policy-fname`, `COOKIE_DOMAIN=franksdank.cloud` sets `--cookie-domain` |
-| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container; parsed at process start only (no reload signal exists — every policy change needs a unit restart) |
+| Policy | `dankclaw/anubis/policy.json` mounted read-only into the container (`anubis.container:52`); parsed at process start only — the deployer restarts the unit after every policy change (`scripts/deploy-remote.sh:1091`) and no reload is wired here *(unverified 2026-10-18: whether upstream Anubis answers any reload signal at all; upstream docs are themselves behind an Anubis challenge, so this could not be read)* |
 | Store | memory JWTs signed by a **persistent** ED25519 key (`ED25519_PRIVATE_KEY_HEX`, host file `/home/dank/Containers/anubis/env`, mode 0600, gitea-style `${VAR}` expansion) — restarts keep every solved challenge valid; with no key set Anubis mints an ephemeral one per boot |

-Policy content: an `ALLOW` rule for Anubis's own endpoints (the
-`/.within.website/x/cmd/anubis/*` API and static namespace — the challenge
-JS calls `make-challenge`/`pass-challenge` under that prefix, and challenging
-those endpoints makes the challenge challenge itself), an `ALLOW` rule for
-the Authentik transport prefix `^/outpost\.goauthentik\.io/` (login, start,
-callback and the outpost's static assets must be reachable before any
-session or challenge dance completes — authentik's own session gate protects
-them), then DENY for AI-scraper user-agent regexes (GPTBot, Claude-*,
-Perplexity*, CCBot, Bytespider, Amazonbot, meta-externalagent, and friends),
-then CHALLENGE for everything else (`path_regex: "."` catch-all). DNSBL off.
-Valid action verbs
-at this tag are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT` is not one of them
-(first deploy of that rule crash-looped the unit on exactly that typo;
-verified against the image binary's string table). The policy schema at this
-tag is JSON-only — no `(data)/` imports and no CEL; first match wins (JSON
-still loads at v1.27.0 — scratch-verified 2026-09-02 before the pin bump).
+Policy content, in the file's own order — first match wins, so the DENY
+really does come first (`dankclaw/anubis/policy.json`, 2026-10-18): a **DENY**
+for AI-scraper user-agent regexes (GPTBot, ChatGPT-User, OAI-SearchBot,
+Claude-\*, anthropic-ai, Perplexity\*, CCBot, Bytespider, Amazonbot,
+meta-externalagent, Google-Extended, DuckAssistBot, cohere-ai, YouBot,
+KimiBot, Applebot-Extended; lines 2-7), then an **ALLOW** for Anubis's own
+endpoint namespace `^/\.within\.website` (lines 8-12 — the challenge JS calls
+`make-challenge`/`pass-challenge` under that prefix, and challenging those
+endpoints makes the challenge challenge itself), then an **ALLOW** for the
+Authentik transport prefix `^/outpost\.goauthentik\.io/` (lines 13-17 — login,
+start, callback and the outpost's static assets must be reachable before any
+session or challenge dance completes; authentik's own session gate protects
+them), then **CHALLENGE** for everything else (`path_regex: "."` catch-all,
+lines 18-22). DNSBL off (`dnsbl: false`, line 23). Live on 2026-10-18: a
+`Mozilla/5.0` request to `https://www.franksdank.cloud/` returns 200 carrying
+`anubis_challenge` and the `cachebuster=v1.27.0` marker, while
+`-A GPTBot` returns the deny body — see gotcha 9 for why that is also a 200.
+
+Valid action verbs in this file are `ALLOW`, `CHALLENGE`, `DENY` — `ACCEPT`
+is not one of them (first deploy of that rule crash-looped the unit on
+exactly that typo; verified against the image binary's string table). The
+local policy uses only the JSON rule form. Upstream documents the policy file
+as YAML with JSON accepted because it is valid YAML "as of version v1.17.0 or
+later" (https://anubis.techaro.lol/docs/admin/policies/), and import
+statements exist in the 1.x line (https://github.com/TecharoHQ/anubis/releases
+lists "allow wildcard matches in import statements" on the road to v1.27.0) —
+so the older local statement "JSON-only, no `(data)/` imports, no CEL at this
+tag" is a v1.13-era claim kept here as history and is **not** a property of
+v1.27.0; whether CEL expressions are accepted at this exact tag was not
+verifiable *(unverified 2026-10-18)*. What is verified is that this JSON
+policy loads and is enforced (container `healthy`; 136 challenges issued in the
+retained log since the 01:25 boot, scraper deny confirmed by probe).
 Upgrading the pin = pull + re-tag the `localhost/` alias, restart; verify
 the flag surface with `podman run --rm ... --help` and the policy against
-the new tag first (done for v1.27.0).
+the new tag first (done for v1.27.0). Upstream is ahead of the pin: Anubis's
+own documentation site served its challenge page with
+`cacheBuster=v1.28.2-0.20261012194510-…` when fetched on 2026-10-18, so
+v1.28.x exists.

 ## Exempt by construction

-These never route through Anubis (no policy bypass rule needed): plex
-(plain-HTTP legacy), auth.franksdank.cloud (the IdP login itself must stay
-ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
-`/info/refs` and `git-*-pack` paths pass unchallenged).
+These never route through Anubis (no policy bypass rule needed): `auth.`
+(the IdP login itself must stay ungated, `dankclaw/Caddyfile:90-99`), `gitea`
+(git smart-HTTP clients cannot solve JS challenges — `/info/refs` and
+`git-*-pack` paths pass unchallenged, `dankclaw/Caddyfile:110-120`), `ai.`
+OpenWebUI (own SSO + API-key auth, `dankclaw/Caddyfile:145-151`), `pihole.`
+(a deliberate basicauth edge instead, `dankclaw/Caddyfile:162-171`), `hermes.`
+and `hermesadmin.` (app-enforced passwords, no public DNS for the first,
+`dankclaw/Caddyfile:179-201`), `plex.` including the plain-HTTP legacy name
+(`dankclaw/Caddyfile:203-238`), the media vhosts `books.`/`jelly.`
+(`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts
+(`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so
+it is exempt by absence — and therefore unauthenticated on the LAN at
+`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`,
+`grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.`
+(`dankclaw/Caddyfile:283-423`).

 ## Gotchas (all verified live; 1–6 on 2026-08-28, 7 on 2026-08-30, 8–9 on 2026-09-02)

@@ -77,8 +107,12 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
   path, a container created from a registry-prefixed image ref fails with
   `image not known` even when the image is present in storage; the identical
   argv works from a shell or a transient unit. The unit therefore runs the
-   image under a `localhost/techarohq/anubis:v1.13.0` alias tag — deploy
-   pulls the canonical ref, then `podman tag`s the alias (idempotent).
+   image under a `localhost/techarohq/anubis:v1.27.0` alias tag — deploy
+   pulls the canonical ref, then `podman tag`s the alias (idempotent;
+   `scripts/deploy-remote.sh:1078-1084`, and `podman inspect anubis` reports
+   `localhost/techarohq/anubis:v1.27.0` on 2026-10-18). The pin used to be
+   v1.13.0; that name is retired, and the unit sets `Pull=never`
+   (`anubis.container:54-56`) so a boot cannot stall on ghcr rate limits.
 3. **Health probe exec form.** The image is a distroless ko-build (no shell;
   entrypoint `/ko-app/anubis`). This quadlet generator joins `HealthCmd`
   into one `\x20`-escaped argv element and podman then runs it shell-form —
@@ -89,9 +123,13 @@ ungated), and gitea (git smart-HTTP clients cannot solve JS challenges —
   guarded vhost return 502 (the chain is reverse-proxy-THROUGH Anubis, not
   forward_auth — Caddy's public listeners proxy to `:8923`). A true rollback
   is a Caddyfile revert + redeploy of the caddy service.
-5. **Deploy handler pull guard.** `deploy_anubis` guards the pull with
-   `podman image exists`; a cold anonymous ghcr token request can 403 on the
-   first fetch — retry once, the auth cache makes the second attempt local.
+5. **Deploy handler pull guard.** `deploy_anubis`
+   (`scripts/deploy-remote.sh:1074-1092`) guards the pull with
+   `podman image exists ghcr.io/techarohq/anubis:v1.27.0`; a cold anonymous
+   ghcr token request can 403 on the first fetch — retry once, the auth cache
+   makes the second attempt local. The handler always ends with
+   `systemctl --user restart anubis.service` + a health wait, which is what
+   makes a policy change land (gotcha 7).
 6. **JS-capable clients are the canary.** Non-JS clients (curl, bots,
   scrapers) never solve challenges — they just receive the challenge page,
   so every curl probe looks green while the verification step is broken.
@@ -163,15 +201,20 @@ Layer-by-layer checks (run from the host via ssh):

 Tool/behavior notes: the anubis container is distroless — `podman exec anubis
 sh` fails (`sh: executable not found`); inspect via its logs or the served
-challenge page. Anubis logs `x-real-ip: 10.89.1.22` (the Caddy container's
-podman IP) for ALL clients — rootless-podman published-port SNAT;
-clients are indistinguishable by IP and clearance is cookie-based. Do NOT
-mistake that for the bug. Homepage render: `app.js` does
+challenge page. Anubis logs the Caddy container's podman address as
+`x-real-ip` for ALL clients — rootless-podman published-port SNAT; clients
+are indistinguishable by IP and clearance is cookie-based. Do NOT mistake
+that for the bug. The address is whatever netavark gave Caddy on
+`dankclaw_default` — all 138 logged `x-real-ip` values in the container's
+retained log read `10.89.1.8` (2026-10-18), and it is not a pinned address,
+so never fingerprint "the Caddy hop" on a literal `10.89.1.22`-style value.
+Homepage render: `app.js` does
 `Promise.all([fetch services.json, fetch status.json])`; on ANY fetch
 failure the catch sets `#summary` to `status unavailable: <err>` and the
 grid stays EMPTY — which a user reads as "links gone / not loading." A
-30-service status.json with a 9-service services.json renders fine
-(stateByName lookup).
+status.json that carries states for more services than services.json lists
+renders fine (stateByName lookup); today both files carry 35 entries
+(`shared/homepage/services.json`, `/srv/homepage/status.json`, 2026-10-18).

 **Decision:** if every layer is healthy (challenge valid, outpost clean,
 handlers 200, files render), the fault is client-side/session. STOP
@@ -194,3 +237,10 @@ curl -s -A "GPTBot" https://www.franksdank.cloud/ | grep -q "Access Denied" && e
 # outpost transport exempt (ALLOW rule)
 curl -s https://search.franksdank.cloud/outpost.goauthentik.io/login | grep -q Making\ sure && echo BROKEN-still-challenged || echo outpost-transport-exempt
 ```
+
+Ran on 2026-10-18 from Dankclaw with `--resolve …:443:192.168.0.143`: the apex
+308s to `https://www.franksdank.cloud/`; `www.` returns 200 with
+`anubis_challenge` + `cachebuster=v1.27.0`; `-A GPTBot` returns 200 whose body
+contains `Access Denied`; `search./outpost.goauthentik.io/login` returns 302
+(exempt, not challenged). `systemctl --user is-active anubis.service` →
+`active`, `is-enabled` → `generated`, container health `healthy`.
@@ -11,27 +11,43 @@ timestamp: 2026-09-16

 Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
 homelab. It protects the Caddy vhosts that do not carry their own
-application-level authentication with its embedded proxy outpost. Apps that
-have their own login (Grafana, Gitea, OpenWebUI, Jellyfin, the *arr stack,
-Pi-hole) are not wrapped — they handle their own session.
+application-level authentication with its proxy outpost. Not wrapped: Gitea,
+OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) —
+they handle their own session. Grafana is the exception that is both: it is a
+guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2
+relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`,
+`dankclaw/Caddyfile:374-377`; live provider table below).

- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`)
+- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`;
+  the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`,
+  2026-10-18)
 - **Host:** dankclaw (`192.168.0.143`)
 - **Name:** `auth.franksdank.cloud`
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
-  `authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)
- **LDAP outpost:** a 4th, quadlet-deployed unit (`authentik-outpost-ldap`)
-  serves LDAP for Jellyfin — see the section below.
+- **Stack: five quadlet units, no compose.** `authentik-postgres`,
+  `authentik-server`, `authentik-worker` (`deploy_authentik`,
+  `scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost`
+  (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`,
+  `deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and
+  `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`,
+  `scripts/deploy-remote.sh:1020-1032`). All five report `generated` +
+  `active`, `Restart=always`, `WantedBy=default.target`, with linger on
+  (`systemctl --user`, 2026-10-18). The "3 compose services" wording is the
+  pre-quadlet era, retired.
+- **LDAP outpost:** serves LDAP for Jellyfin — see the section below.

 ## Why Authentik

-The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
-`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
-for all configuration, an outpost that terminates auth for reverse-proxied
-apps, and a footprint that fits on a 3-user homelab (~1–1.5 GB idle). Redis
-was removed in 2025.10, so the stack is 3 services, not 4. (The integration
-mechanism changed in 2026.x — see Architecture; the research file predates
-the pin.)
+The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was
+recorded in `.docaudit/research/auth.md`, which no longer exists in the repo or
+in the host checkout (`ls`, 2026-10-18) *(unverified 2026-10-18: the
+comparison itself)*. In short: full OIDC with single-logout, a web GUI for all
+configuration, an outpost that terminates auth for reverse-proxied apps, and a
+footprint that fits on a 3-user homelab — measured idle today
+`podman stats --no-stream`: postgres 233.8 MB + server 739.1 MB + worker
+292.4 MB + outpost 294.3 MB + ldap outpost 46.6 MB ≈ **1.6 GB** across five
+containers (the older "~1–1.5 GB" figure was the three-service stack). Redis
+was removed in 2025.10; the integration mechanism changed again in 2026.x (see
+Architecture).

 ## Architecture

@@ -46,6 +62,25 @@ graph LR
    S --> PG[(postgres)]
 ```

+Who answers was re-measured 2026-10-18 from inside the `caddy` container with a
+tenant-matching `Host: search.franksdank.cloud`:
+`http://authentik-server:9000/outpost.goauthentik.io/start` → **302 into the
+authentication flow**, while `http://authentik-outpost:9000/…` answers **404**
+for `/outpost.goauthentik.io/*` *and* for `/if/flow/default-authentication-flow/`.
+So the **server process** still terminates the proxy-provider decision, exactly
+as Caddy's `:8016 → authentik-server:9000` hop assumes
+(`dankclaw/Caddyfile:93-98`). The `authentik-outpost` unit is installed,
+enabled (`generated`) and active, and it is the DB's `authentik Embedded
+Outpost` (`7e39655b-…`, nine providers attached) holding the websocket channel
+to core — but it serves no HTTP surface on the paths the chain needs. The unit
+header's stronger claim that the URL space "is served by the Rust worker
+binary, not the server"
+(`dankclaw/authentik/quadlet/authentik-outpost.container:6-14`) is **not** what
+this probe shows; do not point a new listener at `authentik-outpost:9000`
+without re-measuring. Note also that a probe without a tenant `Host` (e.g.
+`curl http://127.0.0.1:9000/…`) 404s on paths that work — that is the
+Host-routing rule below, not an outage.
+
 Two mechanisms were disproved live (2026-08-29/30) and one is the actual
 contract:

@@ -97,7 +132,7 @@ http://:8016 {
 }
 ```

-The embedded outpost inside the server process decides by `Host`:
+The outpost decides by `Host`:
 no session → the provider's `authentication_flow` is served in place;
 session valid → the request is forwarded to the provider's `internal_host`
 (`http://caddy:8017`), where per-vhost `handle` blocks reach the apps.
@@ -107,10 +142,15 @@ routing table, and the Authentik proxy-provider list must be the same set of
 hosts. A guarded host with no provider 404s at login time; a provider with no
 DNS record breaks off-LAN clients.

-Live provider inventory (verified 2026-09-02), all attached to outpost
-`7e39655b-…` ("authentik Embedded Outpost"), all with
-`internal_host = http://caddy:8017`, all three flows set, and each linked to
-an Application (see the provisioning rules below):
+Live provider inventory (read from the `authentik-postgres` container with
+read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost
+`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9
+members in `authentik_outposts_outpost_providers`), all with
+`internal_host = http://caddy:8017`, all carrying the **same three flow
+UUIDs** (`default-authentication-flow`,
+`default-provider-authorization-implicit-consent`,
+`default-provider-invalidation-flow`), and each backed by an Application
+(the `slug` column below):

 | Provider pk | name | external_host | app slug |
 |---|---|---|---|
@@ -122,13 +162,21 @@ an Application (see the provisioning rules below):
 | 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki |
 | 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard |
 | 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard |
+| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta |

-Provider 3 (`hermes-proxy`) still exists but is dormant — its vhost was
-decommissioned from Caddy on 2026-08-31. Provider 9 (`grafana`) and 12
-(`Seerr OIDC`) are OAuth2 relying-party providers, not proxy providers; the
-outpost does not deserialize them. Trailing slashes on `external_host` (the
-runbook-recipe style) and bare origins both intercept correctly — the outpost
-matches on hostname.
+That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so
+the lockstep rule holds today. Two corrections against the 2026-09-02 table:
+**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and
+**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3`
+returns zero rows, so it is deleted, not "dormant"; its vhost survived but is
+exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties,
+not proxy providers, so the proxy outpost does not deserialize them: pk 9
+`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC`
+(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule
+below rejects), and pk 14 `Open WebUI`
+(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on
+`external_host` (the runbook-recipe style) and bare origins both intercept
+correctly — the outpost matches on hostname.

 Two provisioning rules, each verified live by an outage:

@@ -188,47 +236,65 @@ implementation diverges from upstream in ways that bit us; all verified live:
  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
  build until the evaluator is patched or upgraded. Clients consuming this
  provider must key identity off `sub`.
- Scope mappings for this provider exist (`grafana-email-scope`,
-  `grafana-profile-scope`, Python-syntax expressions) and will start emitting
-  claims the moment the evaluator bug is fixed — no config change needed then.
+- Scope mappings for this provider exist — `grafana-email-scope` and
+  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
+  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
+  the evaluator bug is fixed; no config change is needed then.

 ## LDAP outpost (Jellyfin auth, live 2026-09-01)

-A fourth unit, `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
-→ `authentik-outpost-ldap.service`, serves LDAP for Jellyfin's LDAP-Auth
-plugin. The image is pinned by its **amd64 single-platform digest**
-(`ghcr.io/goauthentik/ldap@sha256:fa33bc8f522113830e1662f4c034352f9eb08e0a7d3da2ddadad9a4b33b4cbf4`):
-podman 5.7.0 refuses the
-`ghcr.io/goauthentik/ldap:2026.8.0` tag because the tag resolves to a
-*nested* manifest list (index → per-platform manifest lists) —
-`unexpectedly received a manifest list instead of a manifest for a single
-image`. Resolve once with `podman manifest inspect <image> | jq -r
-'.manifests[] | select(.platform.architecture=="amd64") | .digest'` and pin
-that digest.
+The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`
+→ `authentik-outpost-ldap.service`, one of the five units in this stack —
+serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`,
+`Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64
+single-platform digest**
+`ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9`
+(`dankclaw/authentik/quadlet/authentik-outpost-ldap.container:26`, matching the
+running image in `podman ps`, 2026-10-18). The digest in this page before 2026-10-18
+(`…fa33bc8f…`) is superseded. The reason for pinning a digest at all: podman
+5.7.0 refuses the plain tag because it resolves to a *nested* manifest list
+(index → per-platform manifest lists) — `unexpectedly received a manifest list
+instead of a manifest for a single image` — first hit on `:2026.8.0`, still the
+rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect
+<image> | jq -r '.manifests[] | select(.platform.architecture=="amd64") |
+.digest'` and pin that digest.

- Ports **389** (ldap) and **636** (ldaps) publish on the LAN; Jellyfin on
-  Dankserver binds `192.168.0.143:389`. The outpost token travels in
-  `AUTHENTIK_LDAP_OUTPOST_TOKEN` from the host `.env` (git-ignored).
+- Ports **389** (ldap) and **636** (ldaps) publish on the LAN
+  (`PublishPort=389:3389`, `636:6636`,
+  `dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp`
+  shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach
+  them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds
+  (2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN`
+  from the host `dankclaw/.env` (git-ignored).

 ### Authentik-side objects (provisioned via `ak shell` ORM)

+Read from the live database (read-only `SELECT`s against the
+`authentik-postgres` container, 2026-10-18):
+
 | Object | Value |
 |---|---|
-| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud` |
-| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` |
-| Service account | `svc-jellyfin-ldap` — passwordless/unusable; authenticates by token |
-| Bind token | intent **`verification`**, non-expiring — its key is the LDAP bind password |
-| RBAC role | `svc-jellyfin-ldap-search` carrying object-level permission `authentik_providers_ldap.search_full_directory` on the provider |
-| Groups | `jellyfin_users`, `jellyfin_admins` |
-| Application | `jellyfin` (slug `jellyfin`) |
+| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` |
+| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* |
+| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* |
+| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — the page previously recorded `intent=verification`; the live value is `app_password` |
+| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) |
+| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) |
+| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (the page previously said slug `jellyfin`, which does not exist) |

-Hard-won rules, each verified live:
+Hard-won rules (the first is now contested by the live database):

- **Token intent is load-bearing.** A bind token with `intent=app_password`
-  is rejected by the flow's password stage with `Invalid password`;
-  `intent=verification` passes. The outpost's direct binder executes the
-  provider's authorization flow, and that password stage only accepts
-  verification-intent tokens as static passwords.
+- **Token intent: contested, do not trust either value blindly.** This page
+  long asserted that a bind token with `intent=app_password` is rejected by the
+  flow's password stage with `Invalid password` and only `intent=verification`
+  passes. The live token for `svc-jellyfin-ldap` is `app_password`
+  (`authentik_core_token`, 2026-10-18), so either the rule is wrong or the
+  current Jellyfin bind path is not that token. Nothing observable settles it
+  today: the LDAP outpost's retained log holds only its 5-minute
+  `initialized direct binder` / `Update providers` config cycles and no bind
+  activity *(unverified 2026-10-18: no bind was performed — reproducing it
+  requires the bind secret)*. Re-verify with a real bind before changing the
+  token's intent either way.
 - **Search needs `search_full_directory`.** Without the RBAC grant the
  outpost returns only the bind user itself — a `(memberOf=…)` filter then
  matches zero entries and the plugin logs `Found no users matching X`.
@@ -236,19 +302,38 @@ Hard-won rules, each verified live:
  and retest.
 - `ak shell` import path for the provider model in 2026.8:
  `from authentik.providers.ldap.models import LDAPProvider`.
+  *(unverified 2026-10-18: importing it outside a configured Django shell fails
+  with `ImproperlyConfigured`, and no `ak shell` was driven against the live
+  auth service for this check)*

 ## Configuration (secrets)

-Two secrets live in the host `.env` (git-ignored; placeholders in
-`.env.example`):
+**Four** variables live in the host env file the units read,
+`/home/dank/serverAppConfigs/dankclaw/.env` (git-ignored; the quadlets expand
+`${…}` from it via `EnvironmentFile`):

 - `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
 - `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
+- `AUTHENTIK_OUTPOST_TOKEN` — proxy outpost token
+  (`dankclaw/authentik/quadlet/authentik-outpost.container:34`).
+- `AUTHENTIK_LDAP_OUTPOST_TOKEN` — LDAP outpost token
+  (`…/authentik-outpost-ldap.container:29`).
+
+Only the first two have placeholders in `dankclaw/.env.example`; the two outpost
+tokens have no example entry (there is no repository-root `.env.example` —
+`find`, 2026-10-18). Rotating `AUTHENTIK_SECRET_KEY` fleet-wide invalidates
+every outpost session (see the ORM playbook).

 ## First-time setup

-1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
-   health-gated), reloads Caddy.
+1. `./scripts/deploy.sh dankclaw --service authentik-server` (any of
+   `authentik-postgres|authentik-server|authentik-worker` runs
+   `deploy_authentik`, `scripts/deploy-remote.sh:980-1004`) — installs the three
+   core quadlets, starts postgres health-gated (`wait_container_healthy
+   authentik-postgres 120`), then the server (300 s health wait) and the worker.
+   The outposts are separate handlers: `--service authentik-outpost` and
+   `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`,
+   dispatch at `:1579-1588`).
 2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
   creates the superuser (email + password) and generates the DB schema.
 3. For every guarded vhost, in the admin interface: **Directory →
@@ -257,20 +342,36 @@ Two secrets live in the host `.env` (git-ignored; placeholders in
   (`https://<host>`), `internal_host = http://caddy:8017`, and **all three**
   flows set (see the provisioning pitfall above). Attach the provider to the
   embedded outpost.
-4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
-   Connect** provider, create the app with the correct redirect URI, and paste
-   the `client_id` / `client_secret` into the app's SSO settings.
+4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create
+   the app with the correct redirect URI, and paste the `client_id` /
+   `client_secret` into the app's SSO settings. The relying parties that exist
+   today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14);
+   **Gitea has no provider row at all** (live provider table, 2026-10-18) — it
+   authenticates itself and stays off the chain by design
+   (`dankclaw/Caddyfile:110-120`).

 ## Operations

- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
-  probes this).
+- **Health:** there is **no** HTTP health path on this build —
+  `/-/health/`, `/-/live/` and `/api/-/health/` all answer **404**, on
+  `127.0.0.1:9000` and through `https://auth.franksdank.cloud` alike
+  (`curl`, 2026-10-18). The container's own probe is Django-side:
+  `HealthCmd=python /manage.py check --database default`
+  (`dankclaw/authentik/quadlet/authentik-server.container:36`), and
+  `podman inspect authentik-server` reports `healthy`. A usable liveness URL is
+  a flow page: `https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`
+  → 200 (2026-10-18). The earlier "the server healthcheck probes `/-/health/`"
+  line was wrong in both halves.
 - **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
 - **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
-  truth). The `/home/dank/Containers/authentik/data` dir holds media + static files.
- **Upgrades:** bump the image tag in the three units under
-  `dankclaw/authentik/quadlet/`, run `./scripts/deploy.sh dankclaw`. Authentik
-  migrates the schema on startup.
+  truth). The `/home/dank/Containers/authentik/data` dir holds media + static
+  files (mounted by all three app units).
+- **Upgrades:** bump the tag in **all three app units** under
+  `dankclaw/authentik/quadlet/` and re-deploy; `authentik-outpost` carries the
+  same tag (it runs the same image, `Exec=worker`) and `authentik-outpost-ldap`
+  is **digest-pinned**, so its pin must be re-resolved for the new version (see
+  the LDAP section) — bumping only the tag would leave the LDAP outpost behind.
+  Authentik migrates the schema on startup.

 ## Self-serve user-settings flow (password change)

@@ -374,8 +475,11 @@ client → TLS edge → Anubis :8923 → outpost decision → internal
  `podman exec caddy sh -c "grep -n -A3 <handler> /etc/caddy/Caddyfile"` —
  after the deploy prints its reload line; confirm the reload actually ran,
  don't trust exit 0.
- Outpost reachability: `podman exec authentik-server getent hosts caddy`
-  resolves the compose DNS alias (10.89.1.x) that `internal_host` dials.
+- Outpost reachability: `podman exec authentik-server getent hosts caddy` →
+  `10.89.1.8 caddy.dns.podman` (2026-10-18) — that is Caddy's address on the
+  `dankclaw_default` podman network, the alias `internal_host` dials. It is a
+  netavark allocation, not a pinned address and not a compose artifact (the
+  compose stack is gone); never hard-code it.

 **Pitfalls:** probing `http://localhost:8017/` from the host returns HTTP
 000 (connection refused) — the post-auth listener lives in the caddy
@@ -403,22 +507,36 @@ EOF
 - The heredoc IS the program: `ak shell` reads all of stdin and execs it
  (heredoc-through-ssh quoting breaks; the raw-stdin pattern is the proven
  one).
- Bootstrap prints JSON log lines, then
-  `### authentik shell (2026.8.3)` and `229 objects imported automatically…`
-  — pipe through `sed -n '/229 objects/,$p'` to drop the noise.
- 229 models are pre-imported. Import paths (each wrong guess costs a failed
-  exec): `User`, `Group`, `Application`, `ApplicationEntitlement`,
-  `PropertyMapping` → `authentik.core.models`; `ProxyProvider` →
+- Bootstrap prints JSON log lines, then a shell banner and an
+  "`NNN objects imported automatically…`" line — pipe through
+  `sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18:
+  the exact banner text and the 229-object count — no `ak shell` was driven
+  against the live auth service for this audit)*.
+- Import paths (each wrong guess costs a failed exec): `User`, `Group`,
+  `Application`, `ApplicationEntitlement`, `PropertyMapping` →
+  `authentik.core.models`; `ProxyProvider` →
  `authentik.providers.proxy.models` (NOT `authentik.outposts.models`);
  `LDAPProvider` → `authentik.providers.ldap.models`;
  `authentik.core.models.application` does NOT exist as an import path; there
  is no `authentik.policies.group` module in this build.
- For live DB truth prefer the postgres container
-  (`podman exec authentik-postgres psql -U authentik -d authentik -At -c
-  "select * from …"`) — **query with `select *` and stop guessing column
-  names**: 2026.x renamed columns (`Provider.objects.filter(type=…)` raises
-  FieldError — `type` moved to subclass tables; the outpost `enabled` column
-  may not exist under that name; provider pk is `oauth2provider_ptr_id`).
+  *(unverified 2026-10-18: these module paths were not re-executed — importing
+  outside a configured Django shell fails with `ImproperlyConfigured`)*
+- For live DB truth prefer the postgres container, and it works without any
+  credential — the local socket is trusted, so a read-only audit is just
+  `podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -`
+  with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify
+  before guessing: 2026.x renamed both tables and columns, and the models are
+  MTI all the way down — `authentik_core_provider` holds `id`, `name` and the
+  three `*_flow_id` columns; the proxy table is
+  `authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column
+  really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2
+  table in 2026.8), the LDAP table is
+  `authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`;
+  `authentik_core_application` attaches to a provider through `provider_id`
+  (its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost`
+  is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError`
+  is the same story from the ORM side: `type` lives in the subclass tables.
+  Never select secret columns (`client_secret`, `key`, `password`) into a log.

 ## User provisioning playbook (user-level account)

@@ -435,33 +553,42 @@ for g in ("Non-Admin Users", "jellyfin_users", "openwebui_users"):
    u.groups.add(Group.objects.get(name=g))
 ```

-`path` defaults to `users` (regular). The access map (verified live
-2026-10-04): there are NO per-app policy bindings on the proxy providers
-(all use the implicit-consent "Authorize Application" flow), so guarded-vhost
-access is decided entirely by GROUP MEMBERSHIP consumed downstream —
+`path` defaults to `users` (regular). The access map (re-checked against the
+live DB 2026-10-18): **every one of the nine proxy providers carries the same
+authorization flow UUID**, `default-provider-authorization-implicit-consent`
+(name "Authorize Application"), i.e. there are no per-app policy bindings to
+delta, so guarded-vhost access is decided entirely by GROUP MEMBERSHIP consumed
+downstream —

 - `homepage-admins` → Caddy RBAC for homepage `/admin` (`X-Authentik-Groups`
  regex gate) and the homepage admin panel; regular users get only
  `category: user` panels from `services.json`.
 - `openwebui_users` / `openwebui_admins` → ApplicationEntitlements
-  "Open WebUI Users"/"Open WebUI Admins" (bound by group), consumed by the
-  "Open WebUI roles" property mapping → `roles` claim. Verify live with
+  **"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the
+  `Open WebUI` application (`authentik_core_applicationentitlement`,
+  2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles`
+  claim *(unverified 2026-10-18: the group→entitlement binding and the mapping
+  expression were not read; the mapping itself is not visible under that name
+  in `authentik_core_propertymapping`)*. Verify live with
  `u.app_entitlements(app)`.
 - `jellyfin_users` / `jellyfin_admins` → Jellyfin via the LDAP outpost
  (filter `(memberOf=cn=jellyfin_users,ou=groups,dc=franksdank,dc=cloud)`).
 - `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
  every human account).
 - **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
-  debo, freya, mitch, eric — mitch (pk 22) and eric (pk 23) created
-  2026-10-15, kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
-  and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
-  `<user>@franksdank.cloud` + display name +
-  `is_active=True` + the password set via `set_password`, membership in
-  `Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was
-  recreated from scratch on 2026-10-08 into this shape (the old row had an
-  empty email and no `openwebui_users`; audit first — the row carried no MFA
-  device, token, entitlement, or proxy session, and events survive a user
-  delete because `Event.user` is a denormalized JSON field, not a FK).
+  debo, freya, mitch, eric). The pks in this page are confirmed against
+  `authentik_core_user` on 2026-10-18: ridley 13, kassie 14, alex 15,
+  artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 —
+  and every one carries the `<user>@franksdank.cloud` address the template
+  prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked
+  *(unverified 2026-10-18)*. Template: `username` +
+  `<user>@franksdank.cloud` + display name + `is_active=True` + the password
+  set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` +
+  `openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this
+  shape (the old row had an empty email and no `openwebui_users`; audit first —
+  the row carried no MFA device, token, entitlement, or proxy session, and
+  events survive a user delete because `Event.user` is a denormalized JSON
+  field, not a FK).
 - EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`,
  `openwebui_admins`, `authentik Admins`, `authentik Read-only`.
 - Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty,
@@ -479,14 +606,24 @@ assert authenticate(request=None, username="ridley", password="nope") is None

 Look flows up by slug, not name:
 `Flow.objects.get(slug="default-authentication-flow")` (its `.name` is
-"Welcome to authentik!" — `.get(name=…)` fails). Stage order:
-Identification(10) → Password(20) → AuthenticatorValidate(30; no-op with no
-enrolled device) → UserLogin(100). With no MFA enrolled, nothing else blocks.
+"Welcome to authentik!" — `.get(name=…)` fails; both confirmed from
+`authentik_flows_flow`, 2026-10-18). Stage order, read from
+`authentik_flows_flowstagebinding` the same day:
+`default-authentication-identification`(10) →
+`default-authentication-password`(20) →
+`default-authentication-mfa-validation`(30; no-op with no enrolled device) →
+`default-authentication-login`(100). With no MFA enrolled, nothing else blocks.

 More field gotchas (each cost a failed exec): `ApplicationEntitlement.application`
 can be null (guard with `getattr`); `LDAPProvider` has
-`search_mode`/`bind_mode`/`base_dn`, lacks `search_base`/`of_groups`/
-`organization`; `OAuth2Provider` has no `.scopes`; `Outpost.config` is an
+`search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/
+`organization` — the backing table
+(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`,
+`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`,
+`uid_start_number`, `search_mode`, `bind_mode`, `mfa_support`
+(`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes`
+(scopes are linked through `ProviderPropertyMapping`, which is why
+`filter(provider=…)` on a scope mapping is empty); `Outpost.config` is an
 `OutpostConfig` object, not a dict; `Policy` has no uniform `.policy_cls`
 (use `type(p).__name__`); `User` has no `.status`.

@@ -536,3 +673,5 @@ podman run --rm --network container:<container> --entrypoint /bin/busybox \
 - [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin
 - [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
 - Research: `.docaudit/research/auth.md` — the full provider comparison
+  *(the file is gone from the repo and from the host checkout as of
+  2026-10-18)*
@@ -9,7 +9,7 @@ timestamp: 2026-08-29

 # Caddy

-Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
+Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and gates the guarded vhosts through the two-tier chain — Anubis bot-challenge first, then Authentik — rather than through HTTP basic auth (`dankclaw/Caddyfile:14-24`). Basic auth survives on exactly one vhost today: the Pi-hole admin UI (`dankclaw/Caddyfile:166-171`).

 **Docs:** https://caddyserver.com/docs

@@ -20,9 +20,9 @@ Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent q
 | Runtime | Quadlet unit `caddy.service` (`dankclaw/caddy/quadlet/caddy.container`) — boot-persistent, self-healing |
 | Image | `franksdank/caddy-porkbun:latest` (local build, `dankclaw/caddy/Containerfile`) |
 | Config | `/home/dank/Containers/caddy/Caddyfile` (bind mount, rendered by deploy) |
-| Data | `/home/dank/Containers/caddy/data` (certs) and `.../config` (ACME state) |
+| Data | `/home/dank/Containers/caddy/data` — Caddy's whole store: `data/caddy/certificates/` (leaf certs) and `data/caddy/acme/` (the ACME account). `.../config` holds only `config/caddy/autosave.json`; it is not where ACME state lives (host `ls`, 2026-10-18) |
 | Ports | 80, 443 on 192.168.0.143 |
-| TLS | Let's Encrypt, DNS-01 via the Porkbun module (one cert per vhost) |
+| TLS | Let's Encrypt, DNS-01 via the Porkbun module — one certificate per name, not a wildcard: 31 per-name directories under `/home/dank/Containers/caddy/data/caddy/certificates/acme-v02.api.letsencrypt.org-directory/` (host `ls`, 2026-10-18) |
 | Runs on | Dankclaw only — Dankserver has no Caddy |

 The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before start. See [[Deployment Flow]].
@@ -32,13 +32,13 @@ The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeh
 Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").

 - **Install path on Dankclaw:** `~/.config/containers/systemd/caddy.container` (unit `caddy.service`)
- **Network:** `dankclaw_default` (the compose bridge — the Caddyfile's container-name upstreams must keep resolving; if no compose service has created the network yet at boot, quadlet creates it first and compose reuses it by name later)
+- **Network:** `dankclaw_default` — the long-lived bridge (`10.89.1.0/24`, gateway `10.89.1.1`, `podman network inspect dankclaw_default`, 2026-10-18) that the old compose stack created; the Caddyfile's container-name upstreams must keep resolving on it. `dankclaw/compose.yaml` no longer exists in the repo or on the host (`ls`, 2026-10-18), so from now on the quadlet units that name this network are the only thing that creates it at boot.
 - **Published ports:** 80/tcp, 443/tcp
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy writes it from the checkout's `.env`; secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all (302) inside the container
+- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy creates it from the checkout's `dankclaw/.env` if it is missing and then fails if either variable is absent (`scripts/deploy-remote.sh:405-412`); secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
+- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all inside the container, which answers **308** (not 302) — `curl -sI http://127.0.0.1:80/` → `308`, `Location: https://www.franksdank.cloud/` (host, 2026-10-18; `dankclaw/Caddyfile:60-62`)
 - **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
 - **Enable state:** `systemctl --user is-enabled caddy` reports `generated` — that is the healthy state for a quadlet. The podman generator creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start (linger covers boot); an explicit `systemctl enable` fails on generated units, so the deploy does not run it
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs and starts the unit
+- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` → `deploy_caddy()` (`scripts/deploy-remote.sh:383-429`) renders the Caddyfile, rebuilds the image when the Containerfile hash changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), installs the quadlet, restarts the unit, and issues `podman exec caddy caddy reload` afterwards

 ## The porkbun image

@@ -57,7 +57,7 @@ acme_dns porkbun {
 }
 ```

-The deploy writes both variables from the host `.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `.env.example` (and `dankclaw/caddy/env.example`) hold placeholders. The Porkbun account holds only `franksdank.cloud`.
+The deploy writes both variables from `dankclaw/.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `dankclaw/.env.example` and `dankclaw/caddy/env.example` hold placeholders (there is no repository-root `env.example` — `find`, 2026-10-18). The account holds `franksdank.cloud` and not `franksdank.casa` — that asymmetry is why `plex.franksdank.casa` stays plain HTTP (`dankclaw/Caddyfile:204-207`); whether it holds any other domain is not checkable from this repo *(unverified 2026-10-18)*.

 ## Vhost map

@@ -65,28 +65,32 @@ Every vhost uses the flat `franksdank.cloud` convention. The full list lives in

 | Upstream | Reach |
 |----------|-------|
-| Dankclaw services (compose network) | by container name, e.g. `comfyui:8188` |
-| Dankclaw quadlet services | by published LAN port, e.g. `192.168.0.143:3000` (Gitea) |
-| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM) |
+| Dankclaw services on `dankclaw_default` | by container name: `authentik-server:9000`, `anubis:8923`, `grafana:3000`, `prometheus:9090`, `loki:3100`, `seerr:5055` (`dankclaw/Caddyfile:94,75,376,381,366,371`) |
+| Dankclaw quadlet services on isolated networks | by published LAN port: `192.168.0.143:3000` (Gitea), `:3005` (OpenWebUI), `:8081` (SearXNG via Gluetun), `:8082` (Marreta), `:8083` (Pi-hole), `:9119` (Hermes dashboard) |
+| Host-network containers on Dankclaw | `host.containers.internal:8787` (Hermes WebUI) — a host-netns bind is not a published port, so netavark's host address is the only upstream that reaches it (`dankclaw/Caddyfile:179-188`) |
+| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM), `:32400` (Plex), `:3923` (Copyparty) |
+| FrankBook-Pro services | by published LAN port on `192.168.0.112` (the *arr stack and qBittorrent; those ports publish on that host's Gluetun netns owner) |

-The bare-IP catch-all `:80` 302-redirects to `https://dankclaw.franksdank.cloud/`. The legacy name `dankserver.franksdank.cloud` 302-redirects there too. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.
+ComfyUI is **not** in this table: it has no vhost. It publishes `0.0.0.0:8188` and is reached at `comfyui.franksdank.cloud:8188` through the Pi-hole A record, with no Caddy and no authentication on the LAN (`dankclaw/comfyui/quadlet/comfyui.container:24-30`).
+
+The bare-IP catch-all `:80` 308-redirects to `https://www.franksdank.cloud/` (`dankclaw/Caddyfile:60-62`, status measured 2026-10-18), and so do the two legacy names `dankclaw.franksdank.cloud` and `dankserver.franksdank.cloud` (`dankclaw/Caddyfile:81-88`) — `dankclaw.` is a redirect target of the past, not of `:80`. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). The version line holds: the container on Dankserver is `lscr.io/linuxserver/plex:1.43.4.10903-e5521bd8c-ls325` (`podman ps`, 2026-10-18). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.

 ## Basic auth

 The Caddyfile in git carries a **bare** placeholder; deploy.sh substitutes the **full** hash from the host `.env` (`CADDY_BASICAUTH_HASH`):

 ```
-chat.franksdank.cloud {
+pihole.franksdank.cloud {
    basicauth {
-        dank REPLACE_WITH_HASH
+        dankuser REPLACE_WITH_HASH
    }
-    reverse_proxy 192.168.0.143:3005
+    reverse_proxy 192.168.0.143:8083
 }
 ```

-After rendering, the line reads `dank $2a$14$xxxx…` (one complete hash). Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password). `tests/deployment_flow.bats` pins the render contract.
+After rendering, the line reads `dankuser <hash>` (one complete hash) — `tests/deployment_flow.bats:96-99` pins the `dankuser` name and the single-hash form, and `tests/caddyfile.bats:75` pins the bare `dankuser REPLACE_WITH_HASH` template. Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password; `tests/caddyfile.bats:67-72` forbids the prefix). `tests/deployment_flow.bats` pins the render contract.

-The homepage (and SearXNG, Hermes, Prometheus, RomM, Copyparty) use `forward_auth` (Authentik SSO) rather than basicauth; see [[Homepage]] and [[Authentik]]. The basicauth hash above is still rendered for the vhosts that keep it (Grafana, Pi-hole).
+Everything else authenticates, not basicauth. The guarded set — `www.` (homepage), `search.`, `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.` — proxies **all** traffic to `anubis:8923`, which verifies the client and forwards it back to Caddy's internal listener `http://:8016`; Authentik's proxy-provider mode serves the login inline (this build exposes **no** `forward_auth` endpoint) and forwards authenticated traffic to `http://:8017`, where the request is routed straight to its app (`dankclaw/Caddyfile:14-24,283-423`). The exempt-by-construction set never enters that chain: `auth.` (the IdP itself), `ai.` and `gitea.` (own logins), `hermes.` and `hermesadmin.` (app-enforced passwords), `pihole.` (the basicauth edge above), and the media/*arr vhosts. Grafana left basicauth on 2026-09-01 and is an OAuth2 relying party against Authentik (`dankclaw/Caddyfile:122-132`). See [[Homepage]] and [[Authentik]].

 ## Operations

@@ -102,21 +106,24 @@ The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile ch
 podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
 ```

-Certificates and ACME state live in the caddy data volume. Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01.
+Certificates and ACME state live in the caddy data volume (`data/caddy/`). Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01. The store also still holds leaf certificates for names the Caddyfile no longer defines — `sonarr.`, `chat.`, `comfyui.`, `ntfy.`, `jellyfin.` (`.franksdank.cloud`) each have a directory there (host `ls`, 2026-10-18) — retired vhosts leave certificates behind, which is harmless but is the record that those names once existed.

 ## Gotchas

- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit — do not re-add it to `compose.yaml`.
+- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit, and the repo has no `dankclaw/compose.yaml` left to re-add it to (`ls`, 2026-10-18).
 - `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
 - The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
- A new vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]].
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]].
+- A public vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]]. The deliberate exception is an **internal-only** name: `hermes.franksdank.cloud` and `hermesadmin.franksdank.cloud` are in `addn-hosts` and the Caddyfile but have no public record, which is what keeps them LAN-only — DNS-01 issuance needs no public A record (`dankclaw/Caddyfile:173-179`).
+- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]]. *(unverified 2026-10-18: hairpin behaviour is only observable from a WAN-side client; nothing on the LAN or in this repo can prove it)*
 - Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
 - `transport http {versions h2c 1.1}` on a vhost makes Caddy send an h2c
  upgrade preface; .NET/Kestrel backends (Jellyfin) reject it and Caddy turns
  the failure into a 502 while the same port answers 200 over plain HTTP/1.1.
-  Match the transport to what the backend speaks: Jellyfin = default HTTP/1.1,
-  Plex PMS = TLS (PMS drops plain HTTP off-loopback, see the test suite).
+  Match the transport to what the backend speaks: Jellyfin = default HTTP/1.1
+  (`dankclaw/Caddyfile:248-251` carries no transport block), Plex PMS = TLS
+  (`dankclaw/Caddyfile:221-227`; PMS drops plain HTTP off-loopback — the
+  `tls_insecure_skip_verify` hop is measured in `tests/caddyfile.bats`, and no
+  vhost in the current Caddyfile sets `versions h2c`).

 ## See Also

@@ -18,11 +18,11 @@ Grafana visualizes the metrics that Prometheus collects from every service. It r
 | Item | Value |
 |------|-------|
 | Image | `docker.io/grafana/grafana:latest` |
-| Published | `127.0.0.1:3002` → 3000, loopback only (see [[Observability]] → Exposure) |
-| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service` |
-| Data | `/opt/grafana/data` — `grafana.db`: accounts, sessions, anything created in the UI |
-| Provisioning mount | `/opt/grafana/provisioning` (read-only) — **empty as of 2026-09-11** |
-| Datasource | **none configured** — see the state note in [[Observability]] |
+| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
+| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
+| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
+| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
+| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |

 ## Configuration

@@ -36,25 +36,33 @@ EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
 Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
 ```

-`seed_grafana_env` in `scripts/deploy-remote.sh` seeds that file from
-`dankclaw/.env` with a single-variable filter and fails the deploy when the
-value is missing. The container receives the one expanded value, never the file.
+`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

 ## Datasource

-None is configured. Grafana cannot query anything yet; pointing it at
-`http://prometheus:9090` and `http://loki:3100` (by container name, on the
-`dankclaw_default` network) is outstanding work — see the state note in
-[[Observability]].
+The Prometheus datasource is declared in git, not clicked into `grafana.db`:
+`shared/grafana/provisioning/datasources/prometheus.yml` points at
+`http://prometheus:9090` — the container name on `dankclaw_default`, the same
+address Caddy proxies to, never the loopback-published port — and carries
+`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
+scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
+alongside it by `dashboards/provider.yml` (provider `fleet`,
+`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
+
+A Loki datasource is still **not** provisioned — the tree contains exactly one
+datasource file — so nothing in Grafana queries logs today, which matters less
+than it sounds because Loki holds no streams (see [[Loki]] → Contents).
+Verification is through the API, not the filesystem:
+`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.

 ## Healthcheck

-Grafana responds on `/api/health` with 200 and `"database":"ok"`.
+Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).

 ## Gotchas

 - Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
+- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
 - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
  no provider token (`token=None`), so no identity header ever reaches Grafana;
  RP mode is topology-independent. The full verified contract (provider pk 9,
@@ -62,10 +70,13 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`.
  Basic client auth, form-encoded token body, and the fork's userinfo-claims
  bug that leaves identity at the `sub` hash only) is in
  [[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana:
-  users auto-upsert as Viewer with `login = <sub hex>`; no email/groups/role
-  claims reach Grafana until the evaluator bug is fixed — do not write role
-  mappings that depend on those claims. The local admin account stays the only
-  Admin.
+  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
+  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
+  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
+  reach Grafana until the evaluator bug is fixed — do not write role mappings
+  that depend on those claims. The local admin account stays the only Admin.
+  (The Viewer-default outcome and the `<sub hex>` login are not readable from
+  the host without the database or an admin password. *(unverified 2026-10-18)*)
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
  file is a gluetun env file and it carries `HOME=/root`; systemd imports it
  for the service, podman then looks for its configuration under `/root`,
@@ -9,7 +9,7 @@ timestamp: 2026-09-11

 # Loki

-Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs. It replaces the need for a full ELK stack.
+Loki is the log aggregation backend of the observability stack. It is configured to store and query container and system logs, replacing the need for a full ELK stack — and as measured on 2026-10-18 the store is **empty**: the service is healthy and queryable, and it holds zero streams. See "Contents" below.

 **Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest

@@ -18,43 +18,59 @@ Loki is the log aggregation backend of the observability stack. It stores and qu
 | Item | Value |
 |------|-------|
 | Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) |
-| Published | `127.0.0.1:3100` → 3100, loopback only |
-| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service` |
-| Mode | single-binary (all components in one process) |
-| Storage | filesystem, `/opt/loki/data` |
-| Auth | `auth_enabled: false` (single-tenant homelab) |
-| Schema | v13 / TSDB |
-| Retention | 30 days, enforced by the compactor |
+| Published | `127.0.0.1:3100` → 3100, loopback only (`dankclaw/observability/quadlet/loki.container:43`; `ss -ltnp` 2026-10-18 shows no other bind) |
+| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`loki.container:52-57`), installed at `~/.config/containers/systemd/loki.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active loki` 2026-10-18) |
+| Mode | single-binary (all components in one process — the `Exec` line passes no `-target`, `loki.container:42`) |
+| Storage | filesystem, `/opt/loki/data` → `/loki` (`loki.container:45`; `loki-config.yaml:10-16`) |
+| Auth | `auth_enabled: false` (single-tenant homelab, `loki-config.yaml:4`) |
+| Schema | v13 / TSDB (`loki-config.yaml:22-30`) |
+| Retention | 30 days, enforced by the compactor (`shared/loki/loki-config.yaml:33` `retention_period: 720h`, `:37-44` `retention_enabled: true`) |
+| Contents (2026-10-18) | **zero streams** — see "Contents" |

 ## Configuration

-`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy. Key settings:
+`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy (`scripts/deploy-remote.sh:276-279`). Key settings:

- `schema_config` v13 with `store: tsdb` and a filesystem object store.
- `compactor` with `retention_enabled: true` so logs do not grow without bound.
- `replication_factor: 1` for a single instance.
+- `schema_config` v13 with `store: tsdb` and a filesystem object store (`loki-config.yaml:22-30`).
+- `compactor` with `retention_enabled: true` so logs do not grow without bound (`loki-config.yaml:37-44`).
+- `replication_factor: 1` for a single instance (`loki-config.yaml:16`).

 ## Volumes

 | Container path | Host path |
 |----------------|-----------|
-| `/loki` | `/opt/loki/data` |
-| `/etc/loki` | `/opt/loki/config` |
+| `/loki` | `/opt/loki/data` (`loki.container:45`) |
+| `/etc/loki` | `/opt/loki/config` (`loki.container:44`) |

 ## Healthcheck

-Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile` as an exec-form `HEALTHCHECK` instead (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it).
+Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile:29-30` as an exec-form `HEALTHCHECK` instead, probing `http://localhost:3100/ready` (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it: `dankclaw/loki/Containerfile:13-17`, `scripts/deploy-remote.sh:558-562`). The deployer waits on the same endpoint through the published loopback, `curl http://127.0.0.1:3100/ready` (`deploy-remote.sh`, the `loki)` case of `deploy_observability`), and `podman ps` reports the container `healthy` (2026-10-18).

 ## Log Shipping

-The [[Alloy]] agent on Dankclaw ships container and journal logs to `http://loki:3100/loki/api/v1/push` — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge).
+The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge). **Nothing arrives today:** Alloy's own counters read zero lines read and zero files tailed, with both file globs refused at start (`journalctl --user -u alloy` 2026-10-18); see [[Alloy]] → "Delivery state".
+
+## Contents (2026-10-18): healthy, empty
+
+The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
+
+- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
+- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
+  `chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
+  `find /opt/loki/data -type f` counted **8 files**, all WAL segments,
+  cluster-seed and delete-request bookkeeping, **no chunk objects**
+  (2026-10-18; the byte total drifts as the WAL rolls, so it is not a
+  useful invariant — the file census is).
+- Prometheus-side health is green: the `loki` job scrapes `loki:3100/metrics` and reports up (`shared/prometheus/prometheus.yml:57-61`), and the homepage card probes `127.0.0.1:3100` (`shared/homepage/services.json:207`).
+
+So "30-day retention" is a policy that has never had anything to delete, and no page should describe log history as available. The cause is upstream of Loki — the shipper ships nothing — and no fix is claimed here; the check that matters is a label query, not `/ready`.

 ## Gotchas

 - Filesystem storage grows; keep compactor retention on.
 - Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
 - The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
- Nothing queries Loki yet: Grafana has no datasource configured. See the state note in [[Observability]].
+- Nothing queries Loki: Grafana's only provisioned datasource is Prometheus (`shared/grafana/provisioning/datasources/prometheus.yml:14-21`, uid `prometheus`), and there is no Loki datasource file in that tree. Adding one is a YAML file, not a UI click — see [[Grafana]] → Datasource.

 ## See Also

@@ -10,27 +10,29 @@ timestamp: 2026-09-11

 Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host, each in its own container.

-**Hosts:** Dankclaw (unit) + Dankserver (compose service) — **Docs:** https://prometheus.io/docs/guides/node-exporter
+**Hosts:** Dankclaw (unit) + Dankserver (unit) — **Docs:** https://prometheus.io/docs/guides/node-exporter
+
+Both hosts run the same shape as a Quadlet unit: `dankclaw/observability/quadlet/node-exporter.container` and `dankserver/observability/quadlet/node-exporter.container` (the compose-only Dankserver copy died for a day and a half before anyone noticed, so it became a unit on 2026-09-16). Both report `generated` + `active` on the host and bind their own two addresses (`systemctl --user is-enabled|is-active node-exporter`, `ss -ltnp` on dankserver, 2026-10-18); `tests/observability_quadlet.bats:419` pins the Dankserver side to a unit, not compose.

 ## Facts

 | Item | Value |
 |------|-------|
 | Image | `prom/node-exporter:latest` |
-| Listen addresses | `127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw) |
-| Network | `Network=host` + `--pid=host` |
+| Listen addresses | `127.0.0.1:9100` and `192.168.0.143:9100` (Dankclaw, `ss -ltnp` 2026-10-18; Dankserver binds `127.0.0.1:9100` + `192.168.0.235:9100`) |
+| Network | `Network=host` + `--pid=host` (`dankclaw/observability/quadlet/node-exporter.container:37-38`) |
 | Unit | `dankclaw/observability/quadlet/node-exporter.container` → `node-exporter.service` |
-| Paths | `--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` |
+| Paths | `--path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/rootfs` (`node-exporter.container:39`) |
 | Auth | none — it is an unauthenticated metrics endpoint by design |

 ## Why it is the one LAN listener

-Every other observability service publishes on loopback. This one cannot:
+Every other observability service publishes a port on loopback. This one runs in the host network namespace instead, and netavark DNATs only *published* ports — so a bridge container cannot reach a host-netns bind at the host's LAN address, and the exporter has to be reachable at an address the consumers can actually dial:

- **Prometheus** runs in a container on the `dankclaw_default` network and cannot reach the host's loopback address, so it needs the host's LAN IP; `shared/prometheus/prometheus.yml` scrapes `192.168.0.143:9100` under the `node-dankclaw` job.
- **The homepage status timer** runs on the host and probes `127.0.0.1:9100`.
+- **Prometheus** runs in a container on `dankclaw_default` and scrapes `host.containers.internal:9100` (`shared/prometheus/prometheus.yml:37-39`) — the gateway address netavark delivers a bridge container's host-bound dial to. It does **not** scrape `192.168.0.143:9100`; that address is refused by construction from this container, the same rule that sends `frank-dragoon` and `frank-zealot` to the gateway. `node-dankserver` is a different machine, so its LAN address is used (`prometheus.yml:41-43`).
+- **The homepage status timer** runs on the host and probes `127.0.0.1:9100` (`shared/homepage/services.json:220`).

-The unit therefore binds exactly those two addresses with two `--web.listen-address` flags — never the wildcard. It has no authentication, so it is never vhosted and never published beyond the LAN.
+The unit therefore binds exactly those two addresses with two `--web.listen-address` flags (`node-exporter.container:39`) — never the wildcard: `ss -ltnp` (2026-10-18) shows `127.0.0.1:9100` and `192.168.0.143:9100` and nothing else. The LAN bind stays in the unit because LAN machines scrape it directly and the seam test pins it, not because this Prometheus needs it. It has no authentication, so it is never vhosted and never published beyond the LAN.

 ## Mounts

@@ -46,14 +48,14 @@ The mount paths and the `--path.*` flags are a pair: changing one without the ot

 | Host | Target |
 |------|--------|
-| Dankclaw | `192.168.0.143:9100` |
+| Dankclaw | `host.containers.internal:9100` (same-host bridge container → the gateway, `prometheus.yml:39`; live target `http://host.containers.internal:9100/metrics` is up, `curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18) |
 | Dankserver | `192.168.0.235:9100` |

-The scrape jobs live in `shared/prometheus/prometheus.yml`; see [[Prometheus]]. The `tests/observability_quadlet.bats` seam pins the Dankclaw address to this unit's listen address so the two cannot drift apart.
+The scrape jobs live in `shared/prometheus/prometheus.yml`; see [[Prometheus]]. The `tests/observability_quadlet.bats` seam pins both halves: the unit's two `--web.listen-address` values (`:103-104`) and the `node-dankclaw` target string `"host.containers.internal:9100"` (`:108-131`), so the exporter and its only Prometheus consumer cannot drift apart.

 ## Gotchas

- Host networking is required (or a published port); the deploy's health wait probes `127.0.0.1:9100/metrics`.
+- Host networking is required (or a published port); the deploy's health wait probes `127.0.0.1:9100/metrics` (`scripts/deploy-remote.sh:536`) and the unit's `HealthCmd` the same address from inside the host namespace (`node-exporter.container:43`).
 - The `/proc`, `/sys` and root mounts must be read-only.
 - Each host's instance reflects only its own host; do not try to share one.
 - Collectors are the image defaults — no per-collector tuning and no textfile collector for custom metrics.
@@ -21,37 +21,61 @@ committed.
 | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
 | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |

-Targets and datasources reference **container names** (`loki:3100`,
-`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
-by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
-are host-netns services, not containers. Prometheus 3 refuses HTML responses
-as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
-exporter (see [[Stale Signal Audit]]).
+Targets and datasources reference **container names** (`grafana:3000`,
+`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
+loopback-published ports; `alloy` is **not scraped at all** (no job in the
+config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
+`shared/homepage/services.json:233`). The LLM endpoints split by topology: the
+two same-host ones (`frank-dragoon` `:7770`, `frank-zealot` `:7771`) are
+scraped as `host.containers.internal:<port>` because a bridge container cannot
+reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
+the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
+(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
+HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
+[[Stale Signal Audit]]).

 ## The deploy path

-`scripts/deploy-remote.sh` carries two copier functions:
+`scripts/deploy-remote.sh` carries one copier per config surface, all called
+from `deploy_observability`:

- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
+- `copy_prometheus_config` → `/opt/prometheus/config/prometheus.yml` **and** `config/rules/llm-fleet.yml` (`:219-227`)
+- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
+- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
+- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
+- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
+- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)

-`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
-`shared/grafana/**` to the `prometheus`/`grafana` services, so
-`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
-touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
-exist, the copier functions exist, and the provisioned paths match the units'
-mounts.
+`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
+`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
+`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
+`shared/grafana/**` to `grafana` (`:261-264`); changes under
+`dankclaw/observability/` select the touched unit and its dependency consumer
+(`:321-328`), and a change to either relay file redeploys `alertmanager`
+(`:399-402`). So `deploy.sh dankclaw --service grafana` (etc.) restarts only
+what the change touched. Seam tests in `tests/observability_quadlet.bats` pin
+the unit files exist and are boot-persistent/self-healing (`:36-62`), that no
+unit publishes on a wildcard (`:63-81`), that the copiers install each unit
+(`:250`), that grafana provisioning is declared in git and installed
+(`:345`), and that the provisioned paths match the units' mounts.

 ## Entry points

 **Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
-vantage first: container name if the exporter is a container on the same
-netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
-if it is an HTTP probe, not a metrics endpoint.
+vantage first: container name if the exporter is a container on
+`dankclaw_default`; `host.containers.internal:<port>` if it is a process in the
+host's own network namespace **on this same machine** (a bridge container
+cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only
+if it is a process on a **different** machine; a blackbox module if it is an
+HTTP probe, not a metrics endpoint.

 **Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
-Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
-Rules reload with the prometheus service.
+Current delivered set: **4 alert rules + 2 recording rules** —
+`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
+`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
+`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
+confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
+reports 4 alerting + 2 recording. Rules reload with the prometheus service.

 **Add a datasource** — drop a YAML file into
 `shared/grafana/provisioning/datasources/`. Point it at the container name of
@@ -60,7 +84,9 @@ the backend (`http://prometheus:9090`), not loopback.
 **Add a dashboard** — drop the JSON into
 `shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
 unified storage, not (only) the SQLite table — grep the API, not the DB, when
-verifying (see the verify section of [[Fleet Audit and Posture]]).
+verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
+2026-10-18: the storage-backend claim needs an admin password to confirm
+against `grafana.db` and the API side by side)*

 ## Verify through the APIs, not the filesystem

@@ -73,34 +99,80 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
 curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
 ```

+The two Prometheus queries were run on 2026-10-18 and are what every number on
+this page comes from: 4 alerting + 2 recording rules, and `up` for
+`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
+`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
+listed in [[Prometheus]]. The Grafana query needs the local admin password and
+was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
+proof of installation only — see the next paragraph.
+
 Health answers from a stale process and provisioned resources can hide in
 unified storage — a file existing on disk is not proof the running server
 loaded it. See [[Stale Signal Audit]] for the never-green-target decision
 table (refused = outage, 404/HTML = config defect).

-## Current delivered state (2026-09-17)
+## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)

 Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
 loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
 unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
-via `host.containers.internal`) → signal-cli. Some targets read down for
+via `host.containers.internal`) → signal-cli, addressed by number
+`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
+2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
+`curl -s http://192.168.0.143:18093/health` → `ok`, and
+`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
+lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
+`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
 structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
-the blackbox exporter. See [[Observability]].
+the blackbox exporter. See [[Prometheus]] → Targets.

-**A llama.cpp endpoint is invisible unless it opts in.** `up{job="frank-zealot"}`
-has been 0 for the entire retained history of the metric — the oldest sample,
-2026-10-11, is already 0 — while the service itself answered `/health` 200 the
-whole time. Cause: llama.cpp answers **501 Not Implemented** on `/metrics` unless
-the server is started with `--metrics`, so the scrape was refused at the protocol
-layer and `LlmEndpointDown` (critical) sat active across four payload generations.
-Fixed 2026-10-18 by the `METRICS=1` knob in `scripts/frank-zealot.env`. Two laws
-fall out of it: a target can read down beside a healthy service, so read
-`lastError` from `/api/v1/targets` before believing an outage; and Alertmanager
-runs here without persistence, so every restart re-stamps `startsAt` on a
-long-standing alert — a "fresh" alert after a reboot is not evidence of a fresh
-fault. Remaining known downs are the structural ones above (HTML-scraped probes,
-Gitea metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
-`frank-dragoon` has always been green.
+**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
+`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
+the oldest sample, 2026-10-11, is already 0 — while the service itself answered
+`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented**
+on `/metrics` unless the server is started with `--metrics`, so the scrape was
+refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing:
+the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
+-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
+on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
+`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target
+reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
+How many payload generations it spanned is not readable from any host
+artifact. *(unverified 2026-10-18)*
+
+Two laws fall out of it. A target can read down beside a healthy service, so
+read `lastError` from `/api/v1/targets` before believing an outage. And rule
+state lives in Prometheus' process, not on disk — a restart re-stamps
+`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
+evidence of a fresh fault. (Alertmanager by contrast **does** persist:
+`--storage.path=/alertmanager` on a host volume keeps silences and the
+notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
+`/opt/alertmanager/data`.)
+
+Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
+metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
+`frank-dragoon` has always been green. The active-alert count is a
+moment-in-time value and should never be documented as a constant:
+`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
+`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
+minutes later, and was empty again by 11:56Z; the relay logged the matching
+firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
+
+**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
+production `frank-zealot` root unit was stopped **on purpose** for a benchmark
+window on the 3090 — the repo Benchmarking Policy says stop the production
+service before benching — and `:7771` answered from a manually booted candidate
+arm (`systemctl is-active frank-zealot` → `failed` after
+`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
+`TimeoutStopSec`, the known shutdown shape for these engines). So
+`up{job="frank-zealot"} == 1` that day measured the trial, not the production
+shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408
+PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract.
+The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
+reinstalls and starts it, restoring boot persistence and self-healing. Never
+quote a candidate arm's flags or VRAM as fleet state — the process and its shape
+changed between two censuses ten minutes apart.

 ## Related

@@ -5,3 +5,112 @@ description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Al
 tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
 timestamp: 2026-09-11
 ---
+
+# Observability on Dankclaw
+
+## Summary
+
+Six containers and one host-side process: Prometheus (metrics), Grafana
+(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
+metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
+carrying every firing group to Signal. All of it is Quadlet-managed,
+boot-persistent and loopback-only, deployed by
+`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
+end to end; the log half is configured, healthy, and carries no data.
+
+This body was rewritten on 2026-10-18 from live evidence: the previous version
+was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
+compliance pass"), while sibling pages kept pointing at a "state note" on this
+page that no longer existed. Every statement below carries the command or file
+that proves it today.
+
+## The stack
+
+| Service | Container | Published | Unit | Page |
+|---|---|---|---|---|
+| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
+| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
+| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
+| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
+| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
+| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
+| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |
+
+Every unit lives in `dankclaw/observability/quadlet/`, carries
+`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
+`~/.config/containers/systemd/<name>.container`, and reports `generated` +
+`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
+linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
+the same day, and `homepage-status.timer` — the root unit that probes all of
+these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
+the probe registry `shared/homepage/services.json` to `/srv/homepage`).
+Dankserver runs one member of this family, node-exporter, as its own unit
+(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
+`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
+
+## Exposure
+
+The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
+shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`,
+`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
+inside, and several carry lifecycle or query APIs. Browser access goes through
+Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
+`anubis:8923`, and the post-auth handlers route by container name on
+`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`),
+`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`),
+`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
+Alertmanager has no vhost at all. node-exporter is the exception that must live
+in the host network namespace, and it binds exactly two addresses rather than
+the wildcard — see [[Node-Exporter]].
+
+The relay's bind is the one address a bridge container can reach on the host:
+`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
+and the podman bridge subnet `10.89.0.0/16` before the DROP
+(`scripts/deploy-remote.sh:255-262`).
+
+## Deployment
+
+`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`,
+which installs the unit, copies that service's config
+(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`,
+`copy_alloy_config`, `copy_grafana_provisioning`, plus
+`install_signal_relay` with the alertmanager deploy), then waits on the loopback
+health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`,
+`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`,
+`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
+`grafana` after `prometheus`, `alloy` after `loki`
+(`grafana.container:32-33`, `alloy.container:38-39`). See
+[[Observability Provisioning]] for what is provisioned as code and how to
+verify it through the APIs.
+
+## Current state (2026-10-18)
+
+* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
+  eight up targets; the four downs are structural (the `comfyui` and `services`
+  HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
+  4 alert rules + 2 recording rules (`/api/v1/rules`, matching
+  `shared/prometheus/rules/llm-fleet.yml`).
+* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
+  relay, and `journalctl --user -u signal-relay` records successful
+  `sent to +12062476336:` lines for every rule in the set.
+* **Dashboards: working.** One provisioned Prometheus datasource (uid
+  `prometheus`) and the `llm-fleet` dashboard, both installed from
+  `shared/grafana/provisioning/`.
+* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
+  file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
+  is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
+  (8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
+  no label names.
+  See [[Alloy]] → Delivery state and [[Loki]] → Contents.
+* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
+  structural downs have no real uptime signal), no Alertmanager vhost.
+
+## Citations
+
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
+- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
+- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
+- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
+- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
+- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
+- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`
@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists."
+description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the provisioned Grafana datasource. Down targets today are structural, not outages."
 tags:
  - infrastructure
  - prometheus
@@ -9,7 +9,7 @@ timestamp: 2026-09-21

 # Prometheus

-Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds Grafana once a datasource exists. node-exporter supplies host metrics from both machines.
+Prometheus collects metrics from the services listed below by scraping their `/metrics` endpoints, and it is the datasource behind the provisioned `llm-fleet` dashboard in [[Grafana]] (datasource uid `prometheus`, `shared/grafana/provisioning/datasources/prometheus.yml:14-21`). node-exporter supplies host metrics from both machines.

 **Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis →
 Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
@@ -20,12 +20,12 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | Item | Value |
 |------|-------|
 | Image | `docker.io/prom/prometheus:latest` |
-| Published | `127.0.0.1:9090` → 9090, loopback only (see [[Observability]] → Exposure) |
-| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service` |
-| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy |
-| Data | `/opt/prometheus/data` (TSDB) |
-| Retention | `--storage.tsdb.retention.time=30d` |
-| Alerting | 5 alert rules + 1 recording rule (`shared/prometheus/rules/llm-fleet.yml`), evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` → Signal relay → signal-cli (see [[Alertmanager]]) |
+| Published | `127.0.0.1:9090` → 9090, loopback only — never a wildcard (`dankclaw/observability/quadlet/prometheus.container:50`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9090` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
+| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`prometheus.container:59-64`), installed at `~/.config/containers/systemd/prometheus.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active prometheus` 2026-10-18) |
+| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
+| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
+| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
+| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |

 ## Targets

@@ -37,16 +37,25 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
 | `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
 | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
-| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
-| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
+| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works |
+| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) |

-**Three jobs read down, and they are not outages** (audited 2026-09-16):
+The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18).

-| Job | Target | Why it never reports up |
-|-----|--------|-------------------------|
-| `comfyui` | `comfyui:8188` | scrapes `/`, which is HTML — "unsupported Content-Type" |
-| `services` | `192.168.0.143:8081`, `:3005` | same: up/down intent, implemented as a scrape |
-| `gitea` | `192.168.0.143:3000/metrics` | Gitea's exporter is off until `[metrics] ENABLED = true` |
+**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
+
+| Job | Target | `health` | `lastError`, verbatim |
+|-----|--------|----------|------------------------|
+| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
+| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
+| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
+| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
+
+Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
+
+**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service.
+
+Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.

 A scrape cannot express "is this HTTP service answering?" — that is what a
 probe is for, and the remedy is the **blackbox exporter** in probe mode
@@ -61,23 +70,23 @@ URLs.

 | Container path | Host path |
 |----------------|-----------|
-| `/etc/prometheus` | `/opt/prometheus/config` |
-| `/prometheus` | `/opt/prometheus/data` |
+| `/etc/prometheus` | `/opt/prometheus/config` (`prometheus.container:51`) |
+| `/prometheus` | `/opt/prometheus/data` (`prometheus.container:52`) |

 ## Healthcheck

-Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on.
+Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on (`prometheus.container:53`, `scripts/deploy-remote.sh` — the `prometheus)` case of `deploy_observability` waits on `http://127.0.0.1:9090/-/healthy`).

 ## Gotchas

- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `192.168.0.143:9100`, which is exactly the LAN address `node-exporter.container` binds (`Network=host`, `--web.listen-address` twice). Change one without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats` seam pins the pairing.
+- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `host.containers.internal:9100` (`prometheus.yml:37-39`), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; `node-exporter.container:39` binds `127.0.0.1:9100` + `192.168.0.143:9100`. Change one side without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats:108-131` seam pins the pairing in both directions.
 - The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently.
 - Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Set a retention policy or the data volume grows without bound (30 days here).
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`), which is why this API must never be published beyond loopback.
+- Set a retention policy or the data volume grows without bound (30 days here — `--storage.tsdb.retention.time=30d`, `prometheus.container:49`).
+- The lifecycle endpoint is enabled (`--web.enable-lifecycle`, `prometheus.container:49`), which is why this API must never be published beyond loopback.

 ## See Also

- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16)
+- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16, commit `6a19143`, installed on the host by `scripts/deploy-remote.sh:286-300`)
 - [[Node-Exporter]] — the host metrics source
 - [[Observability]] — the whole stack, including what is still unconfigured