feat(observability): per-container metrics via cadvisor + Containers dashboard

Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.
Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56 ) from feat/e2e-audit-coverage into main
2026-10-20 22:51:42 -07:00 · 2026-10-20 21:21:44 -07:00 · 2026-10-20 21:20:53 -07:00 · 2026-10-20 21:16:23 -07:00 · 2026-10-20 21:10:06 -07:00 · 2026-10-20 21:04:02 -07:00
20 changed files with 1013 additions and 53 deletions
@@ -0,0 +1,56 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKCLAW.
+#
+# Install path on dankclaw: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# WHY THIS EXISTS. The app stack is ~34 rootless Podman containers and until
+# now nothing reported per-container CPU/memory — Grafana could only see the
+# host (node_exporter) and the LLM engines. cadvisor reads the host cgroup
+# tree and exports container_cpu_* / container_memory_* per container.
+#
+# THE LOAD-BEARING FLAG IS --cgroupns=host. Podman starts every container in a
+# PRIVATE cgroup namespace, so an exporter container sees only its OWN cgroup
+# subtree: cadvisor exported a single root series until this flag was added,
+# then it walked the whole tree (996 per-container series measured 2026-10-20).
+# The quadlet unit name is carried in the cgroup path — app.slice/<name>.service/
+# libpod-payload-<hex> — so Prometheus relabels that path into a `container`
+# label (shared/prometheus/prometheus.yml, cadvisor-dankclaw job) and no
+# libpod/podman socket is needed to name containers.
+#
+# BINDING. cadvisor has no per-interface listen flag (unlike node_exporter's
+# --web.listen-address), so it binds 0.0.0.0 INSIDE its own network namespace.
+# The bridge PublishPort is what keeps the unauthenticated endpoint off the
+# host's other interfaces: it publishes exactly loopback (host-local probes and
+# the deployer's health gate) and this host's LAN address (Prometheus, a bridge
+# container, reaches a host-published port at its LAN address — the same netavark
+# rule node-exporter relies on, only here the port really is published).
+#
+# It is rootless on purpose: dank owns the app containers' cgroups, so a
+# rootless cadvisor reads every one of them; root would buy nothing here.
+
+[Unit]
+Description=cAdvisor container metrics (Dankclaw)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.143:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
 # client_id query parameter — /application/o/<slug>/authorize/ does not exist.
 Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
 Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
-# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
-Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
-Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
+# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
+# ONLY when the token carries no `email` claim (authentik has no /emails path and
+# 404s it, failing the login) — so the claim must actually be there: the fix is
+# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
+# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
+# <...>/userinfo/?api=1/emails, a 403.
+Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
+# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
+# not a known scope here; requesting it makes the authorize view crash.
+# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
+# truncates SCOPES to its first token.
+Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
+Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
+Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
+Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
+# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
+# reaches this callback — the Anubis -> Authentik guard fronts every request.
+Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
 Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
 Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
 Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
@@ -0,0 +1,39 @@
+# Quadlet unit: cadvisor (rootless, boot-persistent) — DANKSERVER.
+#
+# Install path on dankserver: ~/.config/containers/systemd/cadvisor.container
+# Unit name: cadvisor.service (container name: cadvisor)
+#
+# The Dankserver twin of dankclaw/observability/quadlet/cadvisor.container.
+# Same rootless shape, same --cgroupns=host requirement (see that file for the
+# full story), only the LAN address differs: this host publishes on
+# 192.168.0.235:8080 so Prometheus on Dankclaw can scrape it over the real
+# network (cross-host, so the host.containers.internal trick does not apply).
+# It reports the media/library stack (romm, plex, jellyfin, copyparty, calibre,
+# tautulli) by their quadlet unit names via the cgroup path.
+
+[Unit]
+Description=cAdvisor container metrics (Dankserver)
+After=network-online.target
+Wants=network-online.target
+
+[Container]
+ContainerName=cadvisor
+Image=gcr.io/cadvisor/cadvisor:v0.49.1
+PodmanArgs=--pid=host --cgroupns=host
+PublishPort=127.0.0.1:8080:8080
+PublishPort=192.168.0.235:8080:8080
+Volume=/sys:/sys:ro
+Volume=/:/rootfs:ro
+Exec=--housekeeping_interval=30s
+HealthCmd=wget --spider -q http://localhost:8080/healthz
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=40s
+
+[Service]
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
+#
+# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
+# Two DB-side facts (the provider was created by hand via `ak shell`, never
+# repo-managed) silently broke that login, and BOTH had previously been
+# mis-recorded here as a single "the property-mapping evaluator is broken, so
+# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
+#
+#   1. provider.authorization_flow was NULL. The authorize view builds a flow
+#      planner from it and crashes with
+#      `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
+#      "Server Error — Powered by authentik", before any claim is touched.
+#      Fix: point it at default-provider-authorization-implicit-consent, the
+#      flow every other provider in this instance already uses.
+#   2. The `grafana-email-scope` ScopeMapping's expression was the bare string
+#      `user.email`. authentik merges a scope mapping's result into the token as
+#      a MAPPING: a returned dict adds its keys as claims; a returned string is
+#      keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
+#      claim is ever emitted. Grafana, finding no email, falls back to
+#      GET <api_url>/emails, authentik 404s that path, and the login dies with
+#      "Login failed: InternalError". Fix: return
+#      {"email": user.email, "email_verified": True} — the shape the stock
+#      OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
+#      mapping has always worked, which is the proof the evaluator is fine.
+#
+# This is authentik DB state. It is NOT in any deployable file: a blueprint
+# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
+# facts and the symptom returns verbatim. Re-run this after any such rebuild.
+# Every step is idempotent.
+#
+# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
+# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
+# dankclaw/observability/quadlet/grafana.container and deploys normally.
+#
+# Usage: authentik-grafana-sso-fix.sh          (override host: AUTHENTIK_HOST=…)
+set -euo pipefail
+HOST="${AUTHENTIK_HOST:-dankclaw}"
+
+ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
+from authentik.flows.models import Flow
+from authentik.providers.oauth2.models import OAuth2Provider
+
+p = OAuth2Provider.objects.filter(name="grafana").first()
+if not p:
+    raise SystemExit("no OAuth2Provider named 'grafana'")
+
+# 1. The authorization flow every other provider uses.
+want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
+if not want:
+    raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
+if p.authorization_flow_id != want.pk:
+    p.authorization_flow = want
+    p.save()
+    print(f"set grafana.authorization_flow -> {want.slug}")
+else:
+    print(f"grafana.authorization_flow already {want.slug}")
+
+# 2. The email scope mapping must RETURN A DICT, not a bare string.
+m = p.property_mappings.filter(name="grafana-email-scope").first()
+if not m:
+    raise SystemExit("grafana-email-scope mapping is missing")
+fixed = 'return {"email": user.email, "email_verified": True}'
+if m.expression.strip() != fixed:
+    m.expression = fixed
+    m.save()
+    print("repaired grafana-email-scope -> returns {email, email_verified}")
+else:
+    print("grafana-email-scope expression already correct")
+
+# 3. The id_token must carry the scope claims; Grafana reads identity from it.
+if not p.include_claims_in_id_token:
+    p.include_claims_in_id_token = True
+    p.save()
+    print("enabled include_claims_in_id_token")
+else:
+    print("include_claims_in_id_token already on")
+PYEOF
+
+echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
+echo "the round trip must land on / and /api/user must answer 200 with an email."
@@ -39,6 +39,7 @@ SERVICE_ORDER = (
    "loki",
    "alloy",
    "node-exporter",
+    "cadvisor",
    "alertmanager",
    "authentik-postgres",
    "authentik-server",
@@ -92,6 +93,7 @@ DANKCLAW_SERVICES = {
    "loki",
    "alloy",
    "node-exporter",
+    "cadvisor",
    "alertmanager",
    "authentik-postgres",
    "authentik-server",
@@ -124,6 +126,7 @@ DANKSERVER_SERVICES = {
    "readarr",
    "frank-archon",
    "node-exporter",
+    "cadvisor",
 }

 # FrankBook-Pro runs the *arr stack + qBittorrent inside a podman machine on
@@ -374,7 +377,7 @@ def services_for_path(path: str, target: str) -> set[str]:
            # consumer (prometheus before grafana, loki before alloy).
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "alertmanager"}:
+                if name in {"prometheus", "grafana", "loki", "alloy", "node-exporter", "cadvisor", "alertmanager"}:
                    _add(services, name)
            return services
        if len(parts) >= 2 and parts[1] == "gitea":
@@ -417,8 +420,8 @@ def services_for_path(path: str, target: str) -> set[str]:
            # node-dankserver job had nothing to scrape.
            if len(parts) >= 4 and parts[2] == "quadlet":
                name = parts[3].removesuffix(".container")
-                if name == "node-exporter":
-                    _add(services, "node-exporter")
+                if name in {"node-exporter", "cadvisor"}:
+                    _add(services, name)
            return services
        # frank-archon's quadlet-era directory left the tree on 2026-09-26
        # (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
@@ -86,7 +86,7 @@ service_is_allowed() {
        dankclaw:open-webui|dankclaw:gitea-postgres|dankclaw:gitea-server|\
        dankclaw:comfyui|dankclaw:hermes|\
        dankclaw:hermes-dashboard|dankclaw:hermes-webui|dankclaw:prometheus|dankclaw:grafana|\
-        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:alertmanager|\
+        dankclaw:loki|dankclaw:alloy|dankclaw:node-exporter|dankclaw:cadvisor|dankclaw:alertmanager|\
        dankclaw:authentik-postgres|dankclaw:authentik-server|\
        dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
        dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
@@ -97,7 +97,7 @@ service_is_allowed() {
        dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
        dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
        dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
-        dankserver:frank-archon|dankserver:node-exporter|\
+        dankserver:frank-archon|dankserver:node-exporter|dankserver:cadvisor|\
        frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
        frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
        frankbook-pro:readarr)
@@ -520,6 +520,8 @@ deploy_observability() {
            ;;
        node-exporter)
            ;;
+        cadvisor)
+            ;;
        *)
            echo "unknown observability service: $service" >&2
            return 2
@@ -533,6 +535,7 @@ deploy_observability() {
        grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
        loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
        node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
+        cadvisor) wait_for_health cadvisor http://127.0.0.1:8080/healthz 90 ;;
    esac
 }

@@ -2000,7 +2003,7 @@ deploy_service() {
        plex) deploy_plex ;;
        tautulli) deploy_tautulli ;;
        alloy) deploy_alloy ;;
-        prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
+        prometheus|grafana|loki|node-exporter|cadvisor|alertmanager) deploy_observability "$service" ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
    esac
@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
   appears. Create the Application with `provider_id` set (the UI does this
   automatically; raw API/ORM drives must do it explicitly).

-## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
+## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)

 The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
 verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -100,23 +100,32 @@ implementation diverges from upstream in ways that bit us; all verified live:
 - **Provider resolution is by `client_id` query param** on the global
  `/application/o/authorize/` route; per-slug authorize paths do not exist in
  this build. Token/userinfo live at `/application/o/token/` and
-  `/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
-  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
-  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
-  becomes the *expression source* instead of the context; every call raises a
-  swallowed `PropertyMappingExpressionException` and the claim is skipped.
-  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
-  provider FK (linkage lives in the `ProviderPropertyMapping` through model)
-  so `filter(provider=...)` always yields an empty queryset and tokens asked
-  to carry scopes get zero. Net effect: userinfo/introspection answers with
-  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
-  build until the evaluator is patched or upgraded. Clients consuming this
-  provider must key identity off `sub`.
- Scope mappings for this provider exist — `grafana-email-scope` and
-  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
-  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
-  the evaluator bug is fixed; no config change is needed then.
+  `/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
+- **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
+  object has no attribute 'slug'`.** A provider created via raw `ak shell`
+  `create()` leaves `authorization_flow` unset; the authorize view then crashes
+  building the flow planner (`flows/planner.py`) and returns "Server Error —
+  Powered by authentik" before any claim is built. This was previously
+  mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
+  `authorization_flow = default-provider-authorization-implicit-consent` (the
+  flow every other provider in this instance already uses).
+- **A scope mapping that returns a non-dict emits no claim.** authentik merges a
+  granted scope mapping's evaluated result into the token **as a mapping**: a
+  dict contributes its keys as claims; a bare string is keyed under the
+  mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
+  bare string `user.email`, so no `email` claim landed — the evaluator was
+  never broken (the dict-valued `grafana-profile-scope` mapping has always
+  delivered, and the stock OpenID mappings return dicts). Fix:
+  `return {"email": user.email, "email_verified": True}`. With no email, Grafana
+  v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
+  InternalError".
+- **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
+  no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
+- **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
+  clients. Both IdP fixes are idempotent in
+  `scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
+  the provider or a blueprint import, which revert them). Grafana's half plus the
+  verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].

 ## LDAP provider objects (provisioned via `ak shell` ORM)

@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
 tags:
  - infrastructure
  - grafana
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---

 # Grafana
@@ -64,19 +64,28 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
 - Change the default admin credentials at first boot (env or UI).
 - Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
 - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
-  no provider token (`token=None`), so no identity header ever reaches Grafana;
-  RP mode is topology-independent. The full verified contract (provider pk 9,
-  client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
-  Basic client auth, form-encoded token body, and the fork's userinfo-claims
-  bug that leaves identity at the `sub` hash only) is in
-  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
-  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
-  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
-  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
-  reach Grafana until the evaluator bug is fixed — do not write role mappings
-  that depend on those claims. The local admin account stays the only Admin.
-  (The Viewer-default outcome and the `<sub hex>` login are not readable from
-  the host without the database or an admin password. *(unverified 2026-10-18)*)
+  no provider token (`token=None`), so no identity header reaches Grafana; the
+  login is Grafana's own "Sign in with Authentik" button against the `grafana`
+  confidential-client provider. **Verified end-to-end 2026-10-20** after four
+  independent breakages had made it fail. The two Authentik-side causes (a NULL
+  `authorization_flow` → authorize 500; an email scope mapping that returned a
+  bare string so no `email` claim was emitted) and the full wire contract live in
+  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
+  re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
+  an ak-shell re-creation of the provider or a blueprint import, both of which
+  revert it silently). The Grafana half is the unit's own env (`grafana.container`):
+  `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
+  authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
+  attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
+  (safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
+  when the token carries no `email` claim — so it is the email claim itself (an
+  IdP fix, above) that makes the login land, not any Grafana toggle.
+- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
+  `ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
+  `email`. The local admin account stays the only Admin. The landing that the
+  edge-only 13-sso spec never checked is now asserted by
+  `tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
+  an email).
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
  file is a gluetun env file and it carries `HOME=/root`; systemd imports it
  for the service, podman then looks for its configuration under `/root`
@@ -10,9 +10,9 @@ timestamp: 2026-10-20

 ## Summary

-Six containers and one host-side process: Prometheus (metrics), Grafana
+Seven containers and one host-side process: Prometheus (metrics), Grafana
 (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
-metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
+metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
 carrying every firing group to Signal. All of it is Quadlet-managed,
 boot-persistent and loopback-only, deployed by
 `scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
@@ -34,6 +34,7 @@ that proves it today.
 | Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
 | Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
 | node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
+| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
 | Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |

 Every unit lives in `dankclaw/observability/quadlet/`, carries
@@ -44,9 +45,10 @@ linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
 the same day, and `homepage-status.timer` — the root unit that probes all of
 these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
 the probe registry `shared/homepage/services.json` to `/srv/homepage`).
-Dankserver runs one member of this family, node-exporter, as its own unit
-(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
-`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).
+Dankserver runs two members of this family, node-exporter and cadvisor, as
+its own units (`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
+`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
+cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`).

 ## Exposure

@@ -98,10 +100,23 @@ verify it through the APIs.
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
 * **Dashboards: working.** Provisioned datasources — Prometheus (uid
-  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
+  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
+  `containers`, `observability` and `logs-overview` dashboards, all
  installed from `shared/grafana/provisioning/`. The Loki datasource is how
  you read logs: Loki has no web UI of its own, so Grafana Explore is the
  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
+  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
+  per-container CPU/memory here, and on this rootless-Podman host it works on one
+  condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
+  namespace, so without that flag cadvisor sees only its own cgroup root and
+  exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
+  names nothing itself — the libpod socket resolved only 2 of 34 rootless
+  containers — but the quadlet unit name is carried in the cgroup path
+  (`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
+  `container` label. Network and filesystem are NOT exported for these containers
+  (no netns/classid attribution under the host-PID layout), so the dashboard
+  graphs only CPU, memory and OOM and does not fake the rest.
 * **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
  a rootless Alloy cannot read the `0640` journal files, and the container-log
  globs point at a root-storage layout that does not exist here. The fix is built
@@ -117,7 +132,7 @@ verify it through the APIs.

 ## Citations

- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
+- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
 - `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
 - `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
 - `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
@@ -0,0 +1,137 @@
+{
+  "uid": "containers",
+  "title": "Containers — Dankclaw & Dankserver",
+  "description": "Per-container CPU and memory for the quadlet app stacks, from cadvisor (a rootless quadlet on each host). cadvisor keys each series by the cgroup path, and Prometheus relabels the quadlet unit name (app.slice/<name>.service) into a `container` label. Panels use only the metric families cadvisor actually exports for these rootless containers: CPU usage and memory working-set/limit/OOM. Network and filesystem are NOT exported for them (no netns/classid attribution under the rootless+host-PID layout), so no panels pretend otherwise. See serverAppConfigs-docs/Infrastructure/Observability.md.",
+  "tags": ["fleet", "cadvisor", "containers"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "host",
+        "label": "Host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\"}, host)", "refId": "host_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      },
+      {
+        "name": "container",
+        "label": "Container",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\"}, container)", "refId": "container_var" },
+        "refresh": 2,
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1,
+      "type": "row",
+      "title": "CPU",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "panels": []
+    },
+    {
+      "id": 2,
+      "type": "timeseries",
+      "title": "CPU usage (cores)",
+      "description": "CPU-seconds/s per container = cores busy. A steady high line is a busy service; a spike is a burst (a render, an index, a backup). Only containers are shown, not the host or its daemons.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "type": "bargauge",
+      "title": "Top containers by CPU (cores)",
+      "description": "Instant ranking over the selected window. Tells you which container to look at when the host is loaded.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 1 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": null, "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "yellow", "value": 1 }, { "color": "red", "value": 4 } ] } }, "overrides": [] },
+      "options": { "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "displayMode": "gradient", "orientation": "horizontal", "showUnfilled": true },
+      "targets": [
+        {
+          "expr": "topk(12, sum by (host, container) (rate(container_cpu_usage_seconds_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m])))",
+          "legendFormat": "{{host}} / {{container}}",
+          "format": "table",
+          "instant": true,
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 4,
+      "type": "row",
+      "title": "Memory",
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 10 },
+      "panels": []
+    },
+    {
+      "id": 5,
+      "type": "timeseries",
+      "title": "Memory working set",
+      "description": "container_memory_working_set_bytes per container — the number the OOM killer acts on. Cache-inclusive and reclaimable; a flat high line is resident data, a sawtooth is churn.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes", "min": 0, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 6,
+      "type": "timeseries",
+      "title": "Memory vs limit (%)",
+      "description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "id": 7,
+      "type": "timeseries",
+      "title": "OOM kills",
+      "description": "container_oom_events_total rate. Any non-zero line means the kernel killed a process inside that container — the honest signal that its memory limit is too tight for its real working set.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 20, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "sum by (host, container) (rate(container_oom_events_total{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}[5m]))",
+          "legendFormat": "{{host}} / {{container}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ]
+}
@@ -0,0 +1,144 @@
+{
+  "uid": "host-system",
+  "title": "Hosts \u2014 Dankclaw & Dankserver",
+  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
+  "tags": ["host", "system"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Uptime",
+      "description": "Time since boot. A drop means a reboot.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Memory used",
+      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
+      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "Load 5m / core",
+      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Open file handles",
+      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "timeseries", "title": "CPU busy %",
+      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 6, "type": "timeseries", "title": "Load average",
+      "description": "load1/5/15 per host.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
+        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
+        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
+      ]
+    },
+    {
+      "id": 7, "type": "timeseries", "title": "Memory breakdown",
+      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
+        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
+      ]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Filesystem free %",
+      "description": "Per mountpoint. The media/library stacks fill disks.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
+      "targets": [
+        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Disk throughput",
+      "description": "Read/write bytes/s per device.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
+      ]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Disk busy %",
+      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
+      "targets": [
+        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Network throughput",
+      "description": "Receive (up) / transmit (down) bytes/s per interface.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
+      ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Network errors / drops",
+      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
+        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
+      ]
+    }
+  ]
+}
@@ -0,0 +1,75 @@
+{
+  "uid": "logs-overview",
+  "title": "Logs \u2014 Loki (systemd journal)",
+  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
+  "tags": ["logs", "loki"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-1h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "container",
+        "label": "container",
+        "type": "query",
+        "datasource": { "type": "loki", "uid": "loki" },
+        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
+      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
+      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "timeseries", "title": "Total log rate",
+      "description": "All journal lines per second.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "timeseries", "title": "Error-level line rate",
+      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "timeseries", "title": "Errors by container",
+      "description": "Which unit is throwing. Rank by the top offenders.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
+      "description": "Absolute count over the window, not a rate.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
+    },
+    {
+      "id": 6, "type": "logs", "title": "Journal (filtered)",
+      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
+      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
+      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
+    }
+  ]
+}
@@ -0,0 +1,96 @@
+{
+  "uid": "observability",
+  "title": "Observability \u2014 self-monitoring",
+  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
+  "tags": ["observability", "meta"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Targets up",
+      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
+      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Targets down",
+      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
+      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "TSDB head series",
+      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
+      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Discovered targets",
+      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "stat", "title": "Prometheus mem",
+      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 6, "type": "stat", "title": "Grafana up",
+      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 7, "type": "table", "title": "Target health",
+      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
+      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
+      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
+      "description": "Total samples ingested across all jobs.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Loki request rate",
+      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
+      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
+    }
+  ]
+}
@@ -131,3 +131,63 @@ scrape_configs:
      - targets: ["192.168.0.235:7770"]
        labels:
          service: frank-archon
+
+  # Per-container metrics from cadvisor (a rootless quadlet on each host).
+  # cadvisor keys every series by the cgroup path; for a quadlet container that
+  # path is …/app.slice/<name>.service[/libpod-payload-<hex>], so the unit NAME
+  # lives in it. The relabel below pulls that name out into a `container` label
+  # and keeps only the per-unit aggregate — cadvisor's raw host root,
+  # system.slice daemons and machine.slice scopes carry no such name and are
+  # dropped, as is the per-CPU-core split (only cpu="total" survives), so the
+  # scrape stays small. cadvisor cannot bind per-interface, so it is reached at
+  # the host's PUBLISHED LAN port: unlike node-exporter's host-net bind (which
+  # is why node-dankclaw scrapes via host.containers.internal), a published port
+  # IS reachable from this bridge container at the host LAN address.
+  - job_name: "cadvisor-dankclaw"
+    static_configs:
+      - targets: ["192.168.0.143:8080"]
+        labels:
+          host: dankclaw
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      # Keep only the unit-level cgroup; drop its payload/control children so a
+      # container is not counted twice.
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      # Drop everything that is not a named unit (no `container` was extracted).
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      # cpu="total" for cpu series; metrics with no cpu label (memory, network,
+      # filesystem) are kept.
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
+
+  - job_name: "cadvisor-dankserver"
+    static_configs:
+      - targets: ["192.168.0.235:8080"]
+        labels:
+          host: dankserver
+    metric_relabel_configs:
+      - source_labels: [id]
+        regex: '.*/app\.slice/([^.]+)\.service(?:/.*)?$'
+        target_label: container
+        replacement: '$1'
+      - source_labels: [id]
+        regex: '.*/\.service/.*'
+        action: drop
+      - source_labels: [container]
+        regex: '.+'
+        action: keep
+      - source_labels: [cpu]
+        regex: '^$|total'
+        action: keep
+      - regex: 'id|name|image|container_label.*|machine_id|boot_id'
+        action: labeldrop
@@ -29,8 +29,14 @@ export default defineConfig({
      // green suite with a test that does not exist. 17- joined on 2026-09-16
      // (the hermes vhost, whose upstream is a host address rather than a
      // container name, so it needs its own pass condition). 18- joined on
-      // 2026-09-24 (OpenWebUI SSO through authentik).
-      testMatch: /(10|12|13|16|17|18)-.*\.spec\.ts/,
+      // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
+      // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
+      // follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
+      // 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
+      // spec until now: the 17-hermes "uncovered vhost broke silently" class).
+      // 22- joins Prometheus's app-is-really-there proof (its own API), closing
+      // the "not a login wall but maybe a 404" hole 20- closed for Grafana.
+      testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
      use: { storageState: ".auth/qa-e2e.json" },
      dependencies: ["setup"],
    },
@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
 // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
 // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
 // added here or the guard-chain audit is incomplete.
-const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"];
+const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];

 // Markers that prove app content leaked past the guard to an anonymous user.
 const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
  "seerr.franksdank.cloud": ["Request Movie"],
  "grafana.franksdank.cloud": ["Dashboards"],
  "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
+  "notebook.franksdank.cloud": ["Open Notebook"],
 };

 test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {
@@ -0,0 +1,56 @@
+import { expect, test } from "@playwright/test";
+
+// Grafana (grafana.franksdank.cloud) is an OAuth2 RELYING PARTY, not an
+// outpost-fronted proxy: the guard chain (13-sso) only proves an authenticated
+// session reaches Grafana's login page. The login itself happens through
+// Grafana's own "Sign in with Authentik" button, which the edge test never
+// follows — that is exactly how the RP login could be dead (500 on authorize,
+// or an email-claim gap bouncing back to /login) while 13- still passed green.
+// This spec drives the real RP round-trip and asserts a live Grafana session.
+
+const HOST = "grafana.franksdank.cloud";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("SSO: Grafana relying-party login lands a real session", async ({ page, request }) => {
+  test.setTimeout(180_000);
+
+  // Warm the guard: the vhost runs its own Anubis proof-of-work on first visit.
+  await page.goto(`https://${HOST}/`);
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes(CHALLENGE),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  // The "Sign in with Authentik" button targets this RP start URL exactly.
+  // Going straight there is the same code path without depending on the SPA's
+  // button markup.
+  await page.goto(`https://${HOST}/login/generic_oauth`);
+
+  // The chain: grafana -> auth (implicit consent, the session is already
+  // there) -> back to grafana with the code -> grafana provisions the user and
+  // redirects to "/". A dead provider/claims config bounces back to /login.
+  await expect
+    .poll(
+      async () => {
+        const u = new URL(page.url());
+        return u.hostname === HOST && u.pathname !== "/login";
+      },
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  const body = await page.locator("body").innerText();
+  expect(body).not.toContain("Login failed");
+  expect(body.toLowerCase()).not.toContain("powered by authentik"); // IdP flow shell
+  expect(body).not.toContain(CHALLENGE);
+
+  // A genuine logged-in session exists: /api/user answers 200 with an email,
+  // not 401. This is the assertion the edge-only spec could never make.
+  const resp = await page.request.get(`https://${HOST}/api/user`);
+  expect(resp.status()).toBe(200);
+  const user = await resp.json();
+  expect(typeof user.email).toBe("string");
+  expect(user.email.length).toBeGreaterThan(0);
+});
@@ -0,0 +1,52 @@
+import { expect, test } from "@playwright/test";
+
+// open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
+// vhost whose outpost (notebook-proxy) fronts an app with its OWN password
+// login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
+// (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
+// login page is the expected landing. Until now this host appeared in neither
+// the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
+// have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
+// 14- now covers the unauthenticated gate; this spec owns authenticated
+// reachability with a POSITIVE app marker, not just "not a login wall".
+
+const VHOST = "https://notebook.franksdank.cloud/";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
+  test.setTimeout(180_000);
+  await page.goto(VHOST);
+
+  // The Anubis proof-of-work clears in a real browser and the outpost hands off
+  // to the app; the user must never be parked on an authentik flow.
+  await expect
+    .poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("notebook.franksdank.cloud");
+  expect(
+    new URL(page.url()).pathname.startsWith("/if/"),
+    "must not be parked on an authentik flow",
+  ).toBe(false);
+  expect(
+    await page.locator('input[name="uidField"]').count(),
+    "must not show the authentik login form",
+  ).toBe(0);
+
+  // Positive identity: the page really is open-notebook's own login, not a
+  // Caddy error or the IdP shell. The app answering its own login is the pass.
+  await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
+    timeout: 30_000,
+  });
+  await expect(page.locator("body"), "open-notebook login marker").toContainText(
+    /enter your password/i,
+    { timeout: 30_000 },
+  );
+  const body = await page.locator("body").innerText();
+  expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
+  expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
+
+  const resp = await page.request.get(VHOST);
+  expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
+});
@@ -0,0 +1,33 @@
+import { expect, test } from "@playwright/test";
+
+// Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
+// as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
+// under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
+// This is the app-is-really-there proof 20- provides for Grafana: with the
+// standard session the outpost hands off silently to the Prometheus UI, and the
+// app's own API answers success. A guard-route or upstream regression that lets
+// any non-Prometheus answer through fails here.
+
+test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
+  test.setTimeout(180_000);
+  await page.goto("https://prometheus.franksdank.cloud/");
+
+  // Challenge clears, outpost hands off silently to the app (no credential prompt).
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("prometheus.franksdank.cloud");
+  expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
+  await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
+
+  // Load-bearing assertion: the Prometheus API itself answers, authenticated,
+  // through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
+  const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
+  expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
+  expect((await api.json()).status, "Prometheus API must report success").toBe("success");
+});
@@ -21,7 +21,7 @@ load helpers

 UNIT_DIR="$REPO_ROOT/dankclaw/observability/quadlet"
 REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
-UNITS=(prometheus grafana loki alloy node-exporter alertmanager)
+UNITS=(prometheus grafana loki alloy node-exporter cadvisor alertmanager)

 # container_section <unit> — the [Container] key/value block only
 container_section() {
@@ -33,7 +33,7 @@ service_section() {
    awk '/^\[Service\]/{f=1;next} /^\[/{f=0} f' "$1"
 }

-@test "all six observability units exist under dankclaw/observability/quadlet/" {
+@test "all seven observability units exist under dankclaw/observability/quadlet/" {
    local svc
    for svc in "${UNITS[@]}"; do
        [ -f "$UNIT_DIR/$svc.container" ] || { echo "missing $svc.container"; return 1; }
@@ -277,6 +277,7 @@ loki http://127.0.0.1:3100/ready
 alloy http://127.0.0.1:12345/-/healthy
 node-exporter http://127.0.0.1:9100/metrics
 alertmanager http://127.0.0.1:9093/-/healthy
+cadvisor http://127.0.0.1:8080/healthz
 PAIRS
 }

@@ -568,3 +569,72 @@ assert any('ALERTS' in e for e in exprs), exprs
    [ "$status" -eq 0 ]
    [ "$output" = "alertmanager" ]
 }
+
+@test "cadvisor exports container metrics only because of --cgroupns=host, and binds exactly its two consumers" {
+    # Podman starts every container in a private cgroup namespace, so a cadvisor
+    # without --cgroupns=host sees only its own cgroup root and exports a single
+    # series (measured 2026-10-20: 1 series before the flag, 996 after). That
+    # flag and the host /sys mount are the entire mechanism — pin them, plus the
+    # publish set: loopback (host probe + health gate) and this host's LAN
+    # address (Prometheus reaches a host-published port there). Never a wildcard.
+    local unit lan
+    for pair in "dankclaw/observability/quadlet:192.168.0.143" "dankserver/observability/quadlet:192.168.0.235"; do
+        unit="$REPO_ROOT/${pair%%:*}/cadvisor.container"
+        lan="${pair##*:}"
+        [ -f "$unit" ] || { echo "missing $unit"; return 1; }
+        grep -q -- '--cgroupns=host' "$unit" || { echo "$unit lacks --cgroupns=host"; return 1; }
+        grep -q -- '--pid=host' "$unit" || { echo "$unit lacks --pid=host"; return 1; }
+        grep -q '^Volume=/sys:/sys:ro$' "$unit" || { echo "$unit lacks the host /sys mount"; return 1; }
+        grep -q '^WantedBy=default.target$' "$unit" || { echo "$unit is not boot-persistent"; return 1; }
+        grep -q '^Restart=always$' "$unit" || { echo "$unit is not self-healing"; return 1; }
+        grep -q '^HealthCmd=' "$unit" || { echo "$unit has no health check"; return 1; }
+        grep -q "^PublishPort=127.0.0.1:8080:8080\$" "$unit" || { echo "$unit must publish loopback :8080"; return 1; }
+        grep -q "^PublishPort=$lan:8080:8080\$" "$unit" || { echo "$unit must publish $lan:8080"; return 1; }
+        ! grep -qE '^PublishPort=(0\.0\.0\.0|:)' "$unit" || { echo "$unit publishes on a wildcard"; return 1; }
+    done
+}
+
+@test "Prometheus scrapes cadvisor where it is published and relabels the cgroup path to a container name" {
+    # Cross-file contract: cadvisor publishes the LAN port; a cadvisor scrape
+    # job pointing anywhere else reads nothing (unlike node-exporter's host-net
+    # bind, a published port IS reachable from this bridge container at the LAN
+    # address). And the whole dashboard depends on the id -> container relabel:
+    # cadvisor names nothing itself, the quadlet unit name is only in the cgroup
+    # path. If either half changes alone, the Containers dashboard is blank.
+    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
+    grep -q 'job_name: "cadvisor-dankclaw"' "$cfg" || { echo "no cadvisor-dankclaw job"; return 1; }
+    grep -q 'job_name: "cadvisor-dankserver"' "$cfg" || { echo "no cadvisor-dankserver job"; return 1; }
+    grep -q '"192.168.0.143:8080"' "$cfg" || { echo "cadvisor-dankclaw does not scrape the published LAN port"; return 1; }
+    grep -q '"192.168.0.235:8080"' "$cfg" || { echo "cadvisor-dankserver does not scrape the published LAN port"; return 1; }
+    # The relabel that turns …/app.slice/<name>.service… into a `container`.
+    grep -q 'app\\.slice/(\[\^\.\]+)\\.service' "$cfg" || { echo "no cgroup-path -> container relabel"; return 1; }
+    grep -q 'target_label: container' "$cfg" || { echo "relabel does not set the container label"; return 1; }
+}
+
+@test "the Containers dashboard is provisioned from git" {
+    local dash="$REPO_ROOT/shared/grafana/provisioning/dashboards/containers.json"
+    [ -f "$dash" ]
+    python3 -c "
+import json
+d = json.load(open('$dash'))
+assert d['uid'] == 'containers', d['uid']
+names = [t['expr'] for p in d['panels'] for t in p.get('targets', [])]
+# Only families cadvisor actually exports for rootless quadlet containers are
+# graphed; network/fs are absent for them, so a network/fs panel would be a
+# permanent empty lie.
+assert any('container_cpu_usage_seconds_total' in e for e in names), names
+assert any('container_memory_working_set_bytes' in e for e in names), names
+assert not any('container_network' in e or 'container_fs_' in e for e in names), names
+"
+}
+
+@test "the planner selects cadvisor for its unit on each target" {
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankclaw \
+        --paths dankclaw/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+    run python3 "$REPO_ROOT/scripts/deploy-plan.py" --target dankserver \
+        --paths dankserver/observability/quadlet/cadvisor.container
+    [ "$status" -eq 0 ]
+    [ "$output" = "cadvisor" ]
+}
Author	SHA1	Message	Date
Frank	31908c623f	feat(observability): per-container metrics via cadvisor + Containers dashboard Per-container CPU/memory was invisible for the quadlet app stack — Grafana could only see the host (node_exporter) and the LLM engines. This adds a rootless cadvisor quadlet on each host and a `containers` dashboard. cadvisor works on this rootless-Podman / cgroup-v2 host on one condition: --cgroupns=host. Podman starts every container in a private cgroup namespace, so without it cadvisor sees only its own cgroup root and exports a single series (measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the rootless app containers (it resolved 2 of 34), so naming comes from elsewhere: the quadlet unit name is carried in the cgroup path (app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into a `container` label. No podman socket needed. cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge container publishing exactly 127.0.0.1:8080 and the host LAN address — never a wildcard. Prometheus (a bridge container) reaches a host-published port at the LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 / 192.168.0.235:8080. Only the families cadvisor actually exports for these containers are graphed — CPU usage, memory working-set/limit, OOM — not network or filesystem, which are absent under the host-PID layout. Wired through the deployer (allowlist, deploy_observability case + /healthz gate, dispatcher), the planner (both targets), and observability_quadlet.bats.	2026-10-20 22:51:42 -07:00
dank	961713e92d	Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56 ) from feat/e2e-audit-coverage into main	2026-10-20 21:21:44 -07:00
dank	01db29e754	Merge pull request 'feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring)' (#55 ) from feat/grafana-starter-dashboards into main	2026-10-20 21:20:53 -07:00
Frank	ab06e4c221	test(e2e): cover the two guard-surface gaps found in the full-suite audit Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based); two real coverage gaps surfaced: - notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken upstream would stay invisible - the exact 17-hermes class. Added to the 14- guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a new 21- that proves authenticated reachability to open-notebook's own login through the guard chain, with a positive app marker. - prometheus.franksdank.cloud was only ever asserted '<500 and not a login wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really- there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo answers {status:success}. Both specs wired into the standard project (playwright testMatch), which the e2e_scaffold bats gate enforces. Verified live against the fleet; full bats suite green.	2026-10-20 21:16:23 -07:00
Frank	1006e96312	feat(grafana): starter dashboards for host/system, Loki logs, and observability self-monitoring llm-fleet.json already owns LLM capacity+spec-decode. These three cover the domains that had NO dashboard: - host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both Dankclaw and Dankserver (templated by job node-dankclaw\|node-dankserver) - logs-overview: Loki volume + per-container error rate + a sample log panel; container is parsed from the JSON line at query time (journal streams have no container_name stream label) - observability: the stack's own health - Prometheus target up/down + scrape interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem. Panel queries verified live against Prometheus/Loki; provisioned dashboards confirmed via /api/search. Alertmanager is running but NOT scraped by Prometheus - noted in the dashboard, no empty panels built on it.	2026-10-20 21:10:06 -07:00
dank	7d85fed1c9	Merge pull request 'Fix Grafana Authentik SSO (500 + missing email claim) + RP landing test' (#54 ) from fix/grafana-authentik-sso into main	2026-10-20 21:04:02 -07:00
Frank	d8196ce499	docs: Grafana reads claims from userinfo, not USE_ID_TOKEN The login lands once authentik emits the email claim (the IdP fix); Grafana needs no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in the literal-secret seam. Correct the two SSO notes accordingly.	2026-10-20 20:41:57 -07:00
Frank	6f442a14a3	fix(observability): repair Grafana Authentik OAuth2 relying-party login The Authentik SSO button returned "Server Error" (500): the grafana provider had a NULL authorization_flow and the authorize view crashed building a None flow planner. After that, the login failed with "InternalError" — no email claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s. Unit env: request authentik real scopes (openid email profile, not the GitLab user:email that crashes the authorize view); standard userinfo API_URL with no ?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare string instead of {"email":..}). Verified end to end: RP login lands on / and /api/user answers 200 with the real email, user provisioned as Viewer.	2026-10-20 20:41:36 -07:00
Frank	afe72781bc	docs: correct Grafana Authentik SSO facts The vault claimed the fork property-mapping evaluator was broken and userinfo returned only sub, so email could never reach Grafana (outcome unverified). That was a misdiagnosis: the real causes were the NULL authorization_flow (the NoneType.slug crash) and the email scope mapping returning a bare string rather than a dict. Record the four verified causes and fixes, point to the repair script and the new landing spec, and drop the unverified caveat.	2026-10-20 20:40:15 -07:00
Frank	200039d589	test(e2e): cover Grafana relying-party SSO landing (spec 20) 13-sso proves an authenticated session reaches Grafana login page but never follows the OAuth button, so the relying-party login could be dead (500, or an email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives /login/generic_oauth through the real Anubis+authentik round-trip and asserts a live session (/api/user 200 with a non-empty email). Wire 20- into the standard project (qa-e2e session).	2026-10-20 20:40:15 -07:00
Frank	91d9864bb0	fix(observability): repair Grafana Authentik OAuth2 relying-party login The Authentik SSO button returned "Server Error" (500) because the grafana provider authorization_flow was NULL and the authorize view crashed building a None flow planner. After that, the login failed with "InternalError": no email claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s. Unit env: request authentik real scopes (openid email profile, not the GitLab user:email that crashes the authorize view); read claims from the id_token (USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into a /emails 404); pin email/sub/preferred_username attribute paths; allow first-login provisioning since the Anubis->Authentik guard fronts the callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side causes (NULL authorization_flow; the grafana-email-scope mapping that returned a bare string instead of {"email":..}). Verified end to end: RP login lands on / and /api/user answers 200 with the real email, user provisioned as Viewer.	2026-10-20 20:40:15 -07:00
dank	980491668c	Merge pull request 'Alloy ships logs: root host-network service so it can read the journal' (#53 ) from feat/alloy-root-journal into main	2026-10-20 20:05:00 -07:00
dank	2fe68e69af	Merge pull request 'Observability: vllm rule group for frank-archon + Loki datasource; correct stale log-pipeline facts' (#52 ) from fix/observability-vllm-loki into main	2026-10-20 19:39:38 -07:00