Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56 ) from feat/e2e-audit-coverage into main

Merge pull request 'feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring)' (#55 ) from feat/grafana-starter-dashboards into main
test(e2e): cover the two guard-surface gaps found in the full-suite audit
2026-10-20 21:21:44 -07:00 · 2026-10-20 21:20:53 -07:00 · 2026-10-20 21:16:23 -07:00 · 2026-10-20 21:10:06 -07:00 · 2026-10-20 21:04:02 -07:00 · 2026-10-20 20:41:57 -07:00
28 changed files with 1089 additions and 249 deletions
@@ -166,9 +166,11 @@ bare compose container never returned; 2026-09-11: five observability
 services had no units). The observability services now have units under
 `dankclaw/observability/quadlet/` and their compose definitions were removed —
 one owner per container name. The stack is **configured and delivering as of
-2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
-provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
-llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
+2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet`
+dashboard are provisioned from `shared/grafana/provisioning/`, and
+`shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet`
+(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert
+ recording rules, and firing rules are
 delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
 alertmanager.container`, loopback :9093) → host-side Signal relay
 (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
@@ -1,55 +1,51 @@
-# Quadlet unit: alloy (rootless, boot-persistent).
+# Quadlet unit: alloy (ROOT system service, boot-persistent).
 #
-# Install path on dankclaw: ~/.config/containers/systemd/alloy.container
-# Unit name: alloy.service (container name: alloy)
+# Install path on dankclaw: /etc/containers/systemd/alloy.container
+# Unit name: alloy.service (container name: alloy) — SYSTEM manager, root.
 #
-# Log shipper. This replaces the compose service of the same name in
-# dankclaw/compose.yaml ("Observability" block).
+# WHY ROOT (not rootless, unlike its observability siblings). Alloy reads the
+# host journal. The journal files are root:systemd-journal 0640 (other::---);
+# a rootless container carries no supplementary groups and cannot open them —
+# a rootless read of system.journal returns DENIED (see the Alloy page in the
+# vault). Running as root, one loki.source.journal reads system.journal AND
+# user-1000.journal, so it carries both host logs and the rootless Quadlet
+# containers' logs. Validated 2026-10-20: a root probe read 31,825 journal
+# lines and pushed 26,966 into Loki in 20 s, zero drops.
 #
-# IMAGE. franksdank/alloy is built locally by deploy-remote.sh
-# (build_compose_image) from dankclaw/alloy/Containerfile: upstream
-# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a
-# boot from reaching for that local tag at a registry.
+# NETWORK=host. Loki publishes 127.0.0.1:3100 (loopback only) on the host, and
+# no bridge container — rootless or root — can reach the host's loopback, so
+# this unit runs on the host network and pushes to 127.0.0.1:3100. The side
+# effect: the UI must bind loopback too, so Exec sets the listen-addr to
+# 127.0.0.1 (the old rootless unit used 0.0.0.0 only because container loopback
+# was not host loopback; under Network=host they are the same address, so the
+# unauthenticated UI stays loopback-only — nothing off-host reaches it).
 #
-# NETWORK. It joins dankclaw_default so its config's loki.write endpoint
-# resolves the container name loki:3100.
+# THE /etc/machine-id MOUNT IS LOAD-BEARING. sd_journal locates the journal
+# directory by the local machine id; without it the container opens nothing and
+# the journal source reads exactly 0 lines — silently, no error, health green.
+# This mount is the entire reason the pipeline carries data. Never remove it.
 #
-# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own
-# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x
-# refuses the ui block), so the compose service had no reachable health
-# endpoint at all: the homepage status card for alloy reported "down"
-# continuously and there was nothing for a deploy to wait on. The command
-# below therefore moves the UI to the container's wildcard address and the
-# port is published on host loopback only, which puts alloy in the same class
-# as the other three: reachable from the host, by Caddy over the private
-# network, and by nothing off-host. Alloy has no authentication, which is
-# exactly why it is not published wider.
-#
-# SOURCES. The host journald socket and the host log/container-storage trees
-# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
-# lives at /opt/alloy/config/config.alloy, installed from
-# shared/alloy/config.alloy on every deploy.
-#
-# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
-# client fails and its log pipeline backs up.
+# IMAGE. Built rootless by build_compose_image alloy, then mirrored into the
+# root image store by the deployer (a root build runs in the build container's
+# own resolver and fails on DNS here). Pull=never: boot must not reach for the
+# local tag at a registry.

 [Unit]
 Description=Alloy log shipper (Dankclaw)
-After=network-online.target loki.service
-Wants=network-online.target loki.service
+After=network-online.target

 [Container]
 ContainerName=alloy
 Image=localhost/franksdank/alloy:latest
 Pull=never
-Network=dankclaw_default
-Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy
-PublishPort=127.0.0.1:12345:12345
-Volume=/opt/alloy/config:/etc/alloy
+Network=host
+Exec=run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy
+Volume=/opt/alloy/config:/etc/alloy:ro
+Volume=/etc/machine-id:/etc/machine-id:ro
 Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
-Volume=/var/log:/var/log:ro
-Volume=/var/lib/containers:/var/lib/containers:ro
-HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy
+Volume=/var/log/journal:/var/log/journal:ro
+Volume=/run/log/journal:/run/log/journal:ro
+HealthCmd=/usr/local/bin/busybox wget --spider -q http://127.0.0.1:12345/-/healthy
 HealthInterval=30s
 HealthTimeout=10s
 HealthRetries=3
@@ -60,4 +56,4 @@ Restart=always
 RestartSec=10

 [Install]
-WantedBy=default.target
+WantedBy=multi-user.target
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
 # client_id query parameter — /application/o/<slug>/authorize/ does not exist.
 Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
 Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
-# ?api=1 = introspection-style userinfo: returns custom claims (groups) too.
-Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1
-Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false
+# Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
+# ONLY when the token carries no `email` claim (authentik has no /emails path and
+# 404s it, failing the login) — so the claim must actually be there: the fix is
+# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
+# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
+# <...>/userinfo/?api=1/emails, a 403.
+Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
+# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
+# not a known scope here; requesting it makes the authorize view crash.
+# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
+# truncates SCOPES to its first token.
+Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
+Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
+Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
+Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
+# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
+# reaches this callback — the Anubis -> Authentik guard fronts every request.
+Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
 Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
 Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
 Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
+#
+# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
+# Two DB-side facts (the provider was created by hand via `ak shell`, never
+# repo-managed) silently broke that login, and BOTH had previously been
+# mis-recorded here as a single "the property-mapping evaluator is broken, so
+# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
+#
+#   1. provider.authorization_flow was NULL. The authorize view builds a flow
+#      planner from it and crashes with
+#      `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
+#      "Server Error — Powered by authentik", before any claim is touched.
+#      Fix: point it at default-provider-authorization-implicit-consent, the
+#      flow every other provider in this instance already uses.
+#   2. The `grafana-email-scope` ScopeMapping's expression was the bare string
+#      `user.email`. authentik merges a scope mapping's result into the token as
+#      a MAPPING: a returned dict adds its keys as claims; a returned string is
+#      keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
+#      claim is ever emitted. Grafana, finding no email, falls back to
+#      GET <api_url>/emails, authentik 404s that path, and the login dies with
+#      "Login failed: InternalError". Fix: return
+#      {"email": user.email, "email_verified": True} — the shape the stock
+#      OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
+#      mapping has always worked, which is the proof the evaluator is fine.
+#
+# This is authentik DB state. It is NOT in any deployable file: a blueprint
+# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
+# facts and the symptom returns verbatim. Re-run this after any such rebuild.
+# Every step is idempotent.
+#
+# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
+# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
+# dankclaw/observability/quadlet/grafana.container and deploys normally.
+#
+# Usage: authentik-grafana-sso-fix.sh          (override host: AUTHENTIK_HOST=…)
+set -euo pipefail
+HOST="${AUTHENTIK_HOST:-dankclaw}"
+
+ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
+from authentik.flows.models import Flow
+from authentik.providers.oauth2.models import OAuth2Provider
+
+p = OAuth2Provider.objects.filter(name="grafana").first()
+if not p:
+    raise SystemExit("no OAuth2Provider named 'grafana'")
+
+# 1. The authorization flow every other provider uses.
+want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
+if not want:
+    raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
+if p.authorization_flow_id != want.pk:
+    p.authorization_flow = want
+    p.save()
+    print(f"set grafana.authorization_flow -> {want.slug}")
+else:
+    print(f"grafana.authorization_flow already {want.slug}")
+
+# 2. The email scope mapping must RETURN A DICT, not a bare string.
+m = p.property_mappings.filter(name="grafana-email-scope").first()
+if not m:
+    raise SystemExit("grafana-email-scope mapping is missing")
+fixed = 'return {"email": user.email, "email_verified": True}'
+if m.expression.strip() != fixed:
+    m.expression = fixed
+    m.save()
+    print("repaired grafana-email-scope -> returns {email, email_verified}")
+else:
+    print("grafana-email-scope expression already correct")
+
+# 3. The id_token must carry the scope claims; Grafana reads identity from it.
+if not p.include_claims_in_id_token:
+    p.include_claims_in_id_token = True
+    p.save()
+    print("enabled include_claims_in_id_token")
+else:
+    print("include_claims_in_id_token already on")
+PYEOF
+
+echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
+echo "the round trip must land on / and /api/user must answer 200 with an email."
@@ -487,13 +487,15 @@ seed_grafana_env() {
        || { echo "grafana env missing GRAFANA_OAUTH_CLIENT_SECRET; set it in dankclaw/.env" >&2; return 1; }
 }

-# The five observability containers left compose on 2026-09-11 (they were
-# started by hand with podman-compose and never came back after a reboot).
-# They are ordinary rootless Quadlet units now: same images, same mounts, same
-# arguments, same container names, so nothing that addresses them by name —
-# Caddy, Prometheus' own scrape config, Alloy's loki.write endpoint — changed.
-# Only the published ports did: the three HTTP UIs bind 127.0.0.1, because the
-# compose services published on 0.0.0.0 and none of them authenticates.
+# The observability containers left compose on 2026-09-11 (they were started by
+# hand with podman-compose and never came back after a reboot). Five are ordinary
+# rootless Quadlet units: same images, same mounts, same arguments, same
+# container names, so nothing that addresses them by name — Caddy, Prometheus'
+# own scrape config — changed. Only the published ports did: the three HTTP UIs
+# bind 127.0.0.1, because the compose services published on 0.0.0.0 and none of
+# them authenticates. Alloy is the exception: a ROOT system service on
+# Network=host, because reading the host journal needs privileges a rootless
+# container cannot get. It is handled by deploy_alloy, not the loop below.
 deploy_observability() {
    local service="$1"
    # Observability units live under <target>/observability/quadlet: Dankclaw
@@ -512,10 +514,6 @@ deploy_observability() {
            copy_loki_config
            build_compose_image loki
            ;;
-        alloy)
-            copy_alloy_config
-            build_compose_image alloy
-            ;;
        grafana)
            copy_grafana_provisioning
            seed_grafana_env || return 1
@@ -534,11 +532,71 @@ deploy_observability() {
        alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;;
        grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
        loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
-        alloy) wait_for_health alloy http://127.0.0.1:12345/-/healthy 60 ;;
        node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
    esac
 }

+# Mirror an image the deployer built rootless into the ROOT image store. A root
+# quadlet reads root storage; a root build (build_compose_image is rootless) is
+# not possible on this host — the build container's resolver fails DNS — so the
+# only way to get a root-readable copy is save-and-load. Compares image ids so a
+# rebuilt alloy actually re-lands in root storage instead of leaving a stale tag.
+mirror_image_to_root() {
+    local ref="$1" rid
+    rid="$(podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" || {
+        echo "ERROR: rootless image $ref is not built; cannot mirror to root" >&2
+        return 1
+    }
+    if [[ "$(run_privileged podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" == "$rid" ]]; then
+        return 0
+    fi
+    local tar; tar="$(mktemp /var/tmp/alloy-mirror.XXXXXX.tar)"
+    podman save -o "$tar" "$ref"
+    run_privileged podman load -i "$tar"
+    rm -f "$tar"
+}
+
+# Install a quadlet into the SYSTEM generator dir (root manager) instead of the
+# user dir. Boot persistence is the unit's own [Install] section (multi-user
+# target); the v1 system generator creates the .wants on daemon-reload, so —
+# like the user units — no `systemctl enable` (generated units cannot be enabled
+# here).
+install_system_quadlet() {
+    local source_dir="$1" unit="$2"
+    run_privileged mkdir -p /etc/containers/systemd
+    run_privileged cp "$source_dir/$unit.container" /etc/containers/systemd/
+    run_privileged systemctl daemon-reload
+}
+
+# Alloy as a ROOT system service on Network=host. See the unit file: a rootless
+# container cannot read the 0640 journal files, and it cannot reach Loki's
+# 127.0.0.1:3100 publish from a bridge netns. Root + host network fixes both;
+# the price is that this one container runs as root and on the host network.
+deploy_alloy() {
+    copy_alloy_config
+    build_compose_image alloy                      # rootless build (resolver works)
+    mirror_image_to_root localhost/franksdank/alloy:latest
+
+    # Retire the rootless unit BEFORE the system one starts: both managers
+    # generate alloy.service, and two containers named "alloy" both binding
+    # 127.0.0.1:12345 cannot coexist. Move the user quadlet out of the generator
+    # dir (it regenerates from there) and stop the user unit.
+    if [[ -f "$quadlet_unit_dir/alloy.container" ]] ||
+       systemctl --user is-active --quiet alloy.service 2>/dev/null; then
+        systemctl --user stop alloy.service 2>/dev/null || true
+        if [[ -f "$quadlet_unit_dir/alloy.container" ]]; then
+            local trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-alloy-root-cutover"
+            mkdir -p "$trash"
+            mv -- "$quadlet_unit_dir/alloy.container" "$trash/alloy.container"
+            echo "retired rootless alloy.container -> $trash"
+        fi
+        systemctl --user daemon-reload
+    fi
+
+    install_system_quadlet dankclaw/observability/quadlet alloy
+    run_privileged systemctl restart alloy.service
+    wait_for_health alloy http://127.0.0.1:12345/-/healthy 60
+}
 build_compose_image() {
    local service="$1" image cfhash oldhash containerfile
    # Every file the image COPIES IN, beside the Containerfile. The skip below
@@ -1941,7 +1999,8 @@ deploy_service() {
        copyparty) deploy_copyparty ;;
        plex) deploy_plex ;;
        tautulli) deploy_tautulli ;;
-        prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;;
+        alloy) deploy_alloy ;;
+        prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
        comfyui) deploy_comfyui ;;
        *) deploy_compose_service "$service" ;;
    esac
@@ -7,7 +7,7 @@
 # whichever TTS engine wins the engine decision (dankclaw/voiceservice/app.py
 # is a single stdlib-only file in a locally built image). Consumers — omp's
 # TTS device, OpenWebUI, open-notebook — address a VOICE BY NAME
-# (voice=narrator1, or the harness' voice_id=narrator1) and never see a path:
+# (voice=wilwheaton, or the harness' voice_id=wilwheaton) and never see a path:
 # a name is real exactly when /voices/<name>/reference.wav exists, and
 # anything else is a 400 that lists the names that are.
 #
@@ -65,7 +65,7 @@ TTS_UPSTREAM_MODEL=voxcpm2
 # name still 400s with the list that exists — this covers the absent case only —
 # and the journal marks such a line `(default)` so an unrequested voice never
 # reads as a requested one. Any pack directory in /home/dank/voices is legal.
-DEFAULT_VOICE=narrator1
+DEFAULT_VOICE=wilwheaton

 # CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not
 # configured": the endpoint answers a 500 that says so instead of running
@@ -21,7 +21,7 @@ never learn a path.
 **Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`
 both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
 the same change. Probes on the deployed endpoint: `/health` answers
-`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`
+`{"status":"ok","backend":"omni-http","voices":["dr.kern","garrickhagon","jeffhays","vaati","wilwheaton"]}`
 and two different voices rendered the same sentence into **different** bytes
 (24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
 pass-through of one default. The operator harness routes its `speech` role here;
@@ -42,8 +42,8 @@ render needs the app installed, not this endpoint.

 ## Naming a voice

-A voice **is** a directory name. `voice=narrator1` resolves to
-`/voices/narrator1/reference.wav`, and that file — not a manifest, not a config
+A voice **is** a directory name. `voice=wilwheaton` resolves to
+`/voices/wilwheaton/reference.wav`, and that file — not a manifest, not a config
 entry — is what makes the name real. omp's TTS device may send `voice_id`; the
 shim accepts either. A call that sends **neither** — the harness's normal shape,
 because no harness setting carries a voice for this transport — speaks
@@ -59,10 +59,11 @@ loudly instead of letting it half-work.

 Names are user-visible: an unknown voice comes back as
 `400 unknown voice 'x'; available voices: …`, and that list is the honest
-census. The podcast speakers are `narrator1`, `narrator2`, `narrator3` —
-[[Open Notebook]] is configured with exactly those three, so renaming them
-breaks episode rendering, not just speech. A personal voice gets the person's
-name, lowercase.
+census. As of 2026-10-20 the live packs are `dr.kern`, `garrickhagon`,
+`jeffhays`, `vaati`, `wilwheaton`. [[Open Notebook]]'s `dank-duo` profile speaks
+`wilwheaton` (Frank) and `vaati` (Comrade), and `DEFAULT_VOICE` is `wilwheaton`.
+Renaming a pack that a speaker profile names breaks episode rendering, not just
+speech. A personal voice gets the person's name, lowercase.

 ## Voice packs

@@ -221,5 +222,5 @@ harness bundle and measured the day of the flip:
 | Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
 | First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
 | The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
-| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
+| A consumer rejects `wilwheaton` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
 ---
@@ -45,6 +45,8 @@ open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook))

 The app's own login stays on behind the chain. It is not redundant: publishing the API on loopback does **not** hide it from this host's other containers, which can dial `open-notebook:5055` by name on `dankclaw_default`. The chain decides *who from outside* gets in; `OPEN_NOTEBOOK_PASSWORD` decides what anything that already reached the port may do.

+**Single tenant — there are no user accounts, so this is an admin-side tool.** Upstream ships one global password and nothing else: `PasswordAuthMiddleware` compares `Bearer <OPEN_NOTEBOOK_PASSWORD>`, and the SurrealDB schema has no user/account/owner table (verified in the running image, 2026-10-20). One shared password therefore means **one shared workspace** — the guard chain can gate the door but cannot partition the data, and Authentik identity is never consumed. Multi-user is *approved but unbuilt* upstream (issue **#712**, "design phase, targeted post-v2"); OIDC/SSO was declined for now (**#607** closed). Real per-person isolation means **one deployment per person**, not a flag to flip. For that reason the homepage lists open-notebook under **admin**, not as a user-facing app.
+
 ## What the image actually runs

 One container, three processes under supervisord (`supervisord.conf` upstream): `uvicorn api.main:app --port 5055` (`:12`), the job worker (`:25`), and the Next.js frontend (`:39`). Two consequences the config depends on:
@@ -68,7 +70,7 @@ Credentials live in the database, encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`,
 | Model row `voxcpm2` | type `text_to_speech` on the frank-tts credential. The name is decorative: frank-tts ignores the request's `model` field and synthesizes whatever `voice` names ([[Frank TTS]]) |
 | Model row `frank-archon` | type `language`; `PUT /api/models/defaults` points chat, transformation and large-context at it |
 | Model row `qwen3-embedding-0.6b` | type `embedding` on the NPU-lane credential; `default_embedding_model` points at it, so every note save embeds through the XDNA2 lane ([[frank-dragoon]]) |
-| Speaker profile `dank-duo` | two speakers, Frank → `voice_id: narrator1`, Comrade → `narrator2`, `voice_model` = the TTS row |
+| Speaker profile `dank-duo` | two speakers, Frank → `voice_id: wilwheaton`, Comrade → `vaati`, `voice_model` = the TTS row |
 | Episode profile `dank-daily` | `num_segments: 4`, `speaker_config` = `dank-duo`, outline and transcript LLM = the archon row, briefing that ends "Never invent numbers." |

 Order that works over REST: credentials → model rows (`POST /api/models` with `credential: <credential id>`) → `PUT /api/models/defaults` → speaker profile (`voice_model: <model id>`) → episode profile (`speaker_config` / `outline_llm` / `transcript_llm`). `POST /api/models/{id}/test` proves each row: the language row answered with a real completion from frank-archon; the TTS row reached frank-tts and came back `unknown voice 'alloy'; available voices: narrator1, narrator2, narrator3` — the app's own test path hard-codes an OpenAI voice, so that message *is* the proof the leg works.
@@ -106,13 +108,13 @@ A **speaker profile** carries one to four speakers (`docs/3-USER-GUIDE/creating-

 | Speaker | Voice ID to type |
 |---------|------------------|
-| Speaker 1 | `narrator1` |
-| Speaker 2 | `narrator2` |
-| Speaker 3 | `narrator3` |
+| Speaker 1 | `wilwheaton` |
+| Speaker 2 | `vaati` |
+| Speaker 3 | `dr.kern` |

 Those are the pack names frank-tts resolves from `/home/dank/voices` (`scripts/frank-tts.env` header: a voice is real exactly when `/voices/<name>/reference.wav` exists). Two traps, both from upstream's shipped defaults:

- The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `narrator1`/`narrator2` pair (or a third/fourth) rather than editing the seeded ones in place.
+- The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `wilwheaton`/`vaati` pair (or a third/fourth) rather than editing the seeded ones in place.
 - Two speakers with the same voice are hard to follow (`creating-podcasts.md:89`). Distinct packs per speaker is the point of the 1–4 shape.

 ## Two buttons that fail against frank-tts, and why that is cosmetic
@@ -290,9 +290,12 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.

 ## Text-to-speech (frank-tts)

-TTS is the fleet's one OpenAI-compatible speech endpoint —
-`http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url`
-in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That
+TTS is the fleet's one OpenAI-compatible speech endpoint. The config DB
+`audio.tts.*` is set to engine `openai`, `audio.tts.openai.api_base_url`
+`http://192.168.0.143:7772/v1`, `audio.tts.voice` `wilwheaton` (re-keyed
+2026-10-20 from the retired `narrator1`; 7772 is a published port so
+`host.containers.internal` also reaches it, but the LAN IP is the canonical
+on-host form). That
 port is the rootless quadlet `frank-tts` (this repo:
 `dankclaw/voiceservice/quadlet/frank-tts.container`
 `scripts/frank-tts.env`), and its shim resolves `voice=<name>` against
@@ -306,8 +309,8 @@ ships an in-page Kokoro implementation (chunk carrying a frozen voice table
 `onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype*
 setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
 `browser-kokoro` is chosen the page synthesizes locally and rejects any
-voice outside its fixed pack — so `narrator1` fails with
-`Voice: "narrator1" not found. Should be one of: af_heart, …` **without a
+voice outside its fixed pack — so `wilwheaton` fails with
+`Voice: "wilwheaton" not found. Should be one of: af_heart, …` **without a
 single request reaching the server**. Personal audio settings are
 browser: no `user.settings` row and no container env carries an audio
 override, yet the toast still appears ⇒ the choice lives in that browser's
@@ -3,70 +3,111 @@ type: reference
 description: "Alloy is Grafana's log and telemetry shipper."
 tags:
  - infrastructure
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---

 # Alloy

-Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below.
+Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life
+as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It
+forwards the host journal — including the rootless Quadlet containers' logs —
+to Loki. Unlike its observability siblings it runs as a **root system service on
+the host network**, because a rootless container cannot read the journal (see
+*Why root* below).

-**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest
+**Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345
+(loopback) — **Docs:** https://grafana.com/docs/alloy/latest

 ## Facts

 | Item | Value |
 |------|-------|
-| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) |
-| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) |
-| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` |
-| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) |
-| Inputs | host journald socket, podman container logs, `/var/log` |
-| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) |
-| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) |
-| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) |
-| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" |
+| Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) |
+| Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) |
+| Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy |
+| Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish |
+| UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) |
+| Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) |
+| Output | `http://127.0.0.1:3100/loki/api/v1/push` |
+| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) |
+| Successor to | Promtail (EOL 2026-03-02) |
+| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs |

-## Why Alloy and not Promtail
+## Why root (the load-bearing decision)

-Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`).
+A rootless Alloy delivered **zero** lines, silently and with a green health
+check. Two independent causes, both proven on the host 2026-10-20:
+
+1. **The journal is not readable by a rootless container.** The files are
+   `root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a
+   rootless container has no supplementary groups, so opening them returns
+   `DENIED`. `loki.source.journal` then sees an empty view and the counter just
+   sits at 0 — no error logged.
+2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless
+   Quadlet containers log via the journald driver into `user-1000.journal`; the
+   old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`)
+   point at a root-storage layout that does not exist here.
+
+**Root fixes both at once.** As root the container opens every journal file
+under the machine-id directory — `system.journal` *and* `user-1000.journal` —
+so a single `loki.source.journal` carries host services and the rootless
+containers' logs together. Validated with a throwaway root probe before writing
+the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero
+drops**, and Loki's label set went from empty to populated.
+
+### The two mounts that make it work
+
+- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal
+  directory by the *local* machine id. Without it the container opens nothing and
+  the journal source reads exactly 0 lines — silently, health green. This single
+  mount is the whole difference between a working and an empty pipeline; do not
+  remove it. (A root probe without it reads 0; with it, 31,825.)
+- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge
+  container — rootless or root — can reach the host's loopback, so Alloy must
+  share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds
+  `127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before.

 ## Configuration

-`shared/alloy/config.alloy` is the source of truth:
-
- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`).
- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`).
- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
+`shared/alloy/config.alloy` is the source of truth — one `loki.source.journal`
+(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write`
+endpoint. There is deliberately **no** container-file source: container stdout is
+in the journal.

 ## Volumes

 | Container path | Host path |
 |----------------|-----------|
-| `/etc/alloy` | `/opt/alloy/config` |
-| `/run/systemd/journal/socket` | host journald socket (read-only) |
-| `/var/log` | host logs (read-only) |
-| `/var/lib/containers` | podman storage (read-only) |
+| `/etc/alloy` | `/opt/alloy/config` (ro) |
+| `/etc/machine-id` | host machine id (ro) — **required** |
+| `/run/systemd/journal/socket` | host journald socket (ro) |
+| `/var/log/journal` | persistent journal files (ro) |
+| `/run/log/journal` | runtime journal files (ro) |

-## The UI port is published on purpose
+## The UI stays on loopback

-Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`).
-
-## Delivery state (2026-10-18): healthy, shipping nothing
-
-The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18:
-
- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18).
- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
-
-No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
+Alloy 1.x refuses to configure the `ui` service from a config block, so the UI
+address is set on the command line instead. Under the old rootless unit that
+meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under
+`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's
+loopback, reachable from the host and from nothing off-host. Alloy has no
+authentication, which is exactly why it is never published wider. The homepage
+card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`.

 ## Gotchas

- The journald socket mount is required for journal scraping.
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up.
+- **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes.
+  This is the failure mode that hid the broken pipeline; check
+  `loki_source_journal_target_lines_total` and the Loki label query, never
+  `/healthy`.
+- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a
+  rootless user unit — a system unit cannot order against a user unit. Alloy
+  retries its push until Loki answers, so ordering is unnecessary.
+- **The image must be in the ROOT store.** The deployer builds rootless and
+  mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a
+  rootless tag fails at create.
+- One Alloy per host; do not run a single instance that tries to reach both
+  hosts' logs.

 ## See Also

@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
   appears. Create the Application with `provider_id` set (the UI does this
   automatically; raw API/ORM drives must do it explicitly).

-## OAuth2 relying-party quirks (this fork, verified live 2026-08-31)
+## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)

 The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
 verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -100,23 +100,32 @@ implementation diverges from upstream in ways that bit us; all verified live:
 - **Provider resolution is by `client_id` query param** on the global
  `/application/o/authorize/` route; per-slug authorize paths do not exist in
  this build. Token/userinfo live at `/application/o/token/` and
-  `/application/o/userinfo/?api=1`.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.**
-  `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to
-  `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg
-  becomes the *expression source* instead of the context; every call raises a
-  swallowed `PropertyMappingExpressionException` and the claim is skipped.
-  Scope-grant filtering has the same class of bug: `ScopeMapping` has no
-  provider FK (linkage lives in the `ProviderPropertyMapping` through model)
-  so `filter(provider=...)` always yields an empty queryset and tokens asked
-  to carry scopes get zero. Net effect: userinfo/introspection answers with
-  `{sub}` only — email/groups/entitlements claims cannot be delivered by this
-  build until the evaluator is patched or upgraded. Clients consuming this
-  provider must key identity off `sub`.
- Scope mappings for this provider exist — `grafana-email-scope` and
-  `grafana-profile-scope` are both rows in `authentik_core_propertymapping`
-  (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment
-  the evaluator bug is fixed; no config change is needed then.
+  `/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
+- **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
+  object has no attribute 'slug'`.** A provider created via raw `ak shell`
+  `create()` leaves `authorization_flow` unset; the authorize view then crashes
+  building the flow planner (`flows/planner.py`) and returns "Server Error —
+  Powered by authentik" before any claim is built. This was previously
+  mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
+  `authorization_flow = default-provider-authorization-implicit-consent` (the
+  flow every other provider in this instance already uses).
+- **A scope mapping that returns a non-dict emits no claim.** authentik merges a
+  granted scope mapping's evaluated result into the token **as a mapping**: a
+  dict contributes its keys as claims; a bare string is keyed under the
+  mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
+  bare string `user.email`, so no `email` claim landed — the evaluator was
+  never broken (the dict-valued `grafana-profile-scope` mapping has always
+  delivered, and the stock OpenID mappings return dicts). Fix:
+  `return {"email": user.email, "email_verified": True}`. With no email, Grafana
+  v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
+  InternalError".
+- **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
+  no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
+- **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
+  clients. Both IdP fixes are idempotent in
+  `scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
+  the provider or a blueprint import, which revert them). Grafana's half plus the
+  verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].

 ## LDAP provider objects (provisioned via `ak shell` ORM)

@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
 tags:
  - infrastructure
  - grafana
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---

 # Grafana
@@ -64,19 +64,28 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
 - Change the default admin credentials at first boot (env or UI).
 - Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
 - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
-  no provider token (`token=None`), so no identity header ever reaches Grafana;
-  RP mode is topology-independent. The full verified contract (provider pk 9,
-  client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths,
-  Basic client auth, form-encoded token body, and the fork's userinfo-claims
-  bug that leaves identity at the `sub` hash only) is in
-  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana:
-  The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`
-  `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
-  users land as Viewer with `login = <sub hex>`; no email/groups/role claims
-  reach Grafana until the evaluator bug is fixed — do not write role mappings
-  that depend on those claims. The local admin account stays the only Admin.
-  (The Viewer-default outcome and the `<sub hex>` login are not readable from
-  the host without the database or an admin password. *(unverified 2026-10-18)*)
+  no provider token (`token=None`), so no identity header reaches Grafana; the
+  login is Grafana's own "Sign in with Authentik" button against the `grafana`
+  confidential-client provider. **Verified end-to-end 2026-10-20** after four
+  independent breakages had made it fail. The two Authentik-side causes (a NULL
+  `authorization_flow` → authorize 500; an email scope mapping that returned a
+  bare string so no `email` claim was emitted) and the full wire contract live in
+  [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
+  re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
+  an ak-shell re-creation of the provider or a blueprint import, both of which
+  revert it silently). The Grafana half is the unit's own env (`grafana.container`):
+  `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
+  authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
+  attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
+  (safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
+  when the token carries no `email` claim — so it is the email claim itself (an
+  IdP fix, above) that makes the login land, not any Grafana toggle.
+- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
+  `ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
+  `email`. The local admin account stays the only Admin. The landing that the
+  edge-only 13-sso spec never checked is now asserted by
+  `tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
+  an email).
 - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
  file is a gluetun env file and it carries `HOME=/root`; systemd imports it
  for the service, podman then looks for its configuration under `/root`
@@ -3,7 +3,7 @@ type: reference
 title: "Observability on Dankclaw"
 description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
 tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
-timestamp: 2026-09-11
+timestamp: 2026-10-20
 ---

 # Observability on Dankclaw
@@ -83,27 +83,37 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
 [[Observability Provisioning]] for what is provisioned as code and how to
 verify it through the APIs.

-## Current state (2026-10-18)
+## Current state (2026-10-20)

 * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
-  eight up targets; the four downs are structural (the `comfyui` and `services`
-  HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set:
-  4 alert rules + 2 recording rules (`/api/v1/rules`, matching
-  `shared/prometheus/rules/llm-fleet.yml`).
+  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
+  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
+  and both node exporters. The four downs are structural (the `comfyui` and
+  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
+  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
+  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
+  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
+  time-to-first-token), each keyed only to the namespace its runtime exports.
 * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
-* **Dashboards: working.** One provisioned Prometheus datasource (uid
-  `prometheus`) and the `llm-fleet` dashboard, both installed from
-  `shared/grafana/provisioning/`.
-* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both
-  file globs refused) and Loki therefore holds zero streams — `/opt/loki/data`
-  is a few hundred KB of WAL/compactor bookkeeping with **no chunk files**
-  (8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns
-  no label names.
-  See [[Alloy]] → Delivery state and [[Loki]] → Contents.
-* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the
-  structural downs have no real uptime signal), no Alertmanager vhost.
+* **Dashboards: working.** Provisioned datasources — Prometheus (uid
+  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
+  installed from `shared/grafana/provisioning/`. The Loki datasource is how
+  you read logs: Loki has no web UI of its own, so Grafana Explore is the
+  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
+* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
+  a rootless Alloy cannot read the `0640` journal files, and the container-log
+  globs point at a root-storage layout that does not exist here. The fix is built
+  and tested in-repo (2026-10-20): Alloy is now a **root system service on
+  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
+  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
+  lines and pushed 26,966 with zero drops. The **live** instance is still the old
+  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
+  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
+* **Not configured:** no blackbox exporter (the structural downs have no real
+  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
+  not yet deployed (see *Logs* above).

 ## Citations

@@ -1,22 +1,34 @@
 // Alloy configuration — ships host and container logs to Loki.
 // One instance per host. Deployed to /opt/alloy/config/config.alloy.
+//
+// Runs as a ROOT system service on Network=host (see
+// dankclaw/observability/quadlet/alloy.container), so it reaches Loki on the
+// host's loopback and reads every journal — host services and rootless-quadlet
+// containers alike.

 logging {
  level = "info"
 }

-// The alloy UI service listens on 127.0.0.1:12345 by default; the
-// in-container healthcheck probes it there. alloy 1.x refuses to
-// configure the ui service from a config block, so there is no block.
+// The alloy UI listens on 127.0.0.1:12345; the in-container healthcheck probes
+// it there. alloy 1.x refuses to configure the ui service from a config block,
+// so there is no ui block.

-// Ship logs to Loki (container name on the podman network).
+// Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
+// host network, so it pushes by loopback, never by container name — a root
+// container is not on the rootless dankclaw_default network.
 loki.write "default" {
  endpoint {
-    url = "http://loki:3100/loki/api/v1/push"
+    url = "http://127.0.0.1:3100/loki/api/v1/push"
  }
 }

-// Systemd journal from the host socket.
+// The systemd journal. As root with /etc/machine-id + the journal dirs mounted,
+// this one source reads system.journal AND user-*.journal, so it carries both
+// host services and the rootless Quadlet containers' logs. There is deliberately
+// no container-file source: podman container stdout lands in the journal, and
+// the old /var/lib/containers/.../ctr.log and /var/log/containers globs pointed
+// at a root-storage layout that does not exist on this host.
 loki.source.journal "journal" {
  format_as_json = true
  labels = {
@@ -24,12 +36,3 @@ loki.source.journal "journal" {
  }
  forward_to = [loki.write.default.receiver]
 }
-
-// Container logs (podman storage path; adjust for docker).
-loki.source.file "containers" {
-  targets = [
-    {__path__ = "/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log", job = "podman"},
-    {__path__ = "/var/log/containers/*.log", job = "containers"},
-  ]
-  forward_to = [loki.write.default.receiver]
-}
@@ -0,0 +1,144 @@
+{
+  "uid": "host-system",
+  "title": "Hosts \u2014 Dankclaw & Dankserver",
+  "description": "node_exporter for both servers. Starter set \u2014 throw-at-the-wall breadth; refine the windows and drop the noise. The picker selects the host; CPU/load/memory/filesystem/network are per-node.",
+  "tags": ["host", "system"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "node",
+        "label": "host",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prometheus" },
+        "query": { "query": "label_values(up{job=~\"node-.*\"}, job)", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Uptime",
+      "description": "Time since boot. A drop means a reboot.",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "time() - node_boot_time_seconds{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Memory used",
+      "description": "1 - MemAvailable/MemTotal. The honest \"in use\" figure (excludes reclaimable cache).",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.8 }, { "color": "red", "value": 0.92 }] } }, "overrides": [] },
+      "targets": [{ "expr": "1 - node_memory_MemAvailable_bytes{job=~\"$node\"} / node_memory_MemTotal_bytes{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "Load 5m / core",
+      "description": "Per-core load5. >1 sustained means runnable tasks are queuing behind cores.",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Open file handles",
+      "description": "FDs allocated vs the kernel maximum. Rising toward the max precedes EMFILE across the host.",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 0.9 }] } }, "overrides": [] },
+      "targets": [{ "expr": "node_filefd_allocated{job=~\"$node\"} / node_filefd_maximum{job=~\"$node\"}", "legendFormat": "{{job}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "timeseries", "title": "CPU busy %",
+      "description": "CPU seconds/s in each mode = cores busy per mode; the stack sums to total cores in use (16/host here). A user or system spike points at what is loading the box; steal > 0 means the hypervisor is throttling.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "min": 0, "custom": { "fillOpacity": 10, "lineWidth": 1, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,mode) (rate(node_cpu_seconds_total{job=~\"$node\",mode!=\"idle\",mode!=\"iowait\"}[5m]))", "legendFormat": "{{job}} {{mode}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 6, "type": "timeseries", "title": "Load average",
+      "description": "load1/5/15 per host.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_load1{job=~\"$node\"}", "legendFormat": "{{job}} load1", "refId": "A" },
+        { "expr": "node_load5{job=~\"$node\"}", "legendFormat": "{{job}} load5", "refId": "B" },
+        { "expr": "node_load15{job=~\"$node\"}", "legendFormat": "{{job}} load15", "refId": "C" }
+      ]
+    },
+    {
+      "id": 7, "type": "timeseries", "title": "Memory breakdown",
+      "description": "Total, free, buffers+cache. Reclaimable cache is not pressure; MemAvailable is the real headroom.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [
+        { "expr": "node_memory_MemTotal_bytes{job=~\"$node\"} - node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} used", "refId": "A" },
+        { "expr": "node_memory_MemAvailable_bytes{job=~\"$node\"}", "legendFormat": "{{job}} available", "refId": "B" }
+      ]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Filesystem free %",
+      "description": "Per mountpoint. The media/library stacks fill disks.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 12 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 0.15 }] } }, "overrides": [] },
+      "targets": [
+        { "expr": "node_filesystem_avail_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"} / node_filesystem_size_bytes{job=~\"$node\",fstype!~\"tmpfs|overlay|squashfs\"}", "legendFormat": "{{job}} {{mountpoint}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Disk throughput",
+      "description": "Read/write bytes/s per device.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_disk_read_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} R", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_disk_written_bytes_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} W", "refId": "B" }
+      ]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Disk busy %",
+      "description": "Fraction of time the device had I/O in flight. Near 1 = the device is the bottleneck.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 20 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
+      "targets": [
+        { "expr": "rate(node_disk_io_time_seconds_total{job=~\"$node\"}[5m])", "legendFormat": "{{job}} {{device}}", "refId": "A" }
+      ]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Network throughput",
+      "description": "Receive (up) / transmit (down) bytes/s per interface.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "Bps" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} RX", "refId": "A" },
+        { "expr": "-sum by (job,device) (rate(node_network_transmit_bytes_total{job=~\"$node\",device!~\"lo|veth.*|br.*|docker.*\"}[5m]))", "legendFormat": "{{job}} {{device}} TX", "refId": "B" }
+      ]
+    },
+    {
+      "id": 12, "type": "timeseries", "title": "Network errors / drops",
+      "description": "Interface errors and drops \u2014 the signature of a saturated link or a bad NIC/cable.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 28 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
+      "targets": [
+        { "expr": "sum by (job,device) (rate(node_network_receive_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} RX err", "refId": "A" },
+        { "expr": "sum by (job,device) (rate(node_network_transmit_errs_total{job=~\"$node\"}[5m]))", "legendFormat": "{{job}} {{device}} TX err", "refId": "B" }
+      ]
+    }
+  ]
+}
@@ -0,0 +1,75 @@
+{
+  "uid": "logs-overview",
+  "title": "Logs \u2014 Loki (systemd journal)",
+  "description": "Everything in Loki arrives as the journal (job=systemd-journal); the container is a JSON field parsed at query time, not a stream label. Starter breadth \u2014 the volume/error panels group by the parsed container_name. Loki keeps ~7 days; wide ranges are cheap counts, not the raw log panel.",
+  "tags": ["logs", "loki"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-1h", "to": "now" },
+  "templating": {
+    "list": [
+      {
+        "name": "container",
+        "label": "container",
+        "type": "query",
+        "datasource": { "type": "loki", "uid": "loki" },
+        "query": { "query": "{job=\"systemd-journal\"} | json", "label": "container_name", "refId": "A" },
+        "includeAll": true,
+        "multi": true,
+        "refresh": 2,
+        "current": { "text": "All", "value": "$__all" }
+      }
+    ]
+  },
+  "panels": [
+    {
+      "id": 1, "type": "timeseries", "title": "Log volume by container (lines/s)",
+      "description": "Parse the journal JSON, count lines per container. Spikes = a chatty or crashing unit.",
+      "gridPos": { "h": 9, "w": 24, "x": 0, "y": 0 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps", "custom": { "fillOpacity": 20, "stacking": { "mode": "normal" } } }, "overrides": [] },
+      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "timeseries", "title": "Total log rate",
+      "description": "All journal lines per second.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 9 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} [$__auto]))", "legendFormat": "all lines/s", "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "timeseries", "title": "Error-level line rate",
+      "description": "Lines matching error/panic/fatal/fail. A blunt text match \u2014 the journal carries no structured level for all units.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 9 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps", "color": { "mode": "fixed", "fixedColor": "red" } }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal|fail)\" [$__auto]))", "legendFormat": "error lines/s", "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "timeseries", "title": "Errors by container",
+      "description": "Which unit is throwing. Rank by the top offenders.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 17 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (container_name) (count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" | json [$__auto]))", "legendFormat": "{{container_name}}", "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "stat", "title": "Error lines (last 15m)",
+      "description": "Absolute count over the window, not a rate.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 17 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 50 }, { "color": "red", "value": 500 }] } }, "overrides": [] },
+      "targets": [{ "expr": "sum(count_over_time({job=\"systemd-journal\"} |~ \"(?i)(error|panic|fatal)\" [15m]))", "instant": true, "legendFormat": "errors/15m", "refId": "A" }]
+    },
+    {
+      "id": 6, "type": "logs", "title": "Journal (filtered)",
+      "description": "Raw lines for the selected containers. Set the container picker; keep the window tight.",
+      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 25 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "options": { "showTime": true, "wrapLogMessage": true, "sortOrder": "Descending", "enableLogDetails": true },
+      "targets": [{ "expr": "{job=\"systemd-journal\"} | json | container_name=~\"$container\"", "refId": "A" }]
+    }
+  ]
+}
@@ -0,0 +1,96 @@
+{
+  "uid": "observability",
+  "title": "Observability \u2014 self-monitoring",
+  "description": "The health of the stack that watches the fleet: Prometheus scrape/TSDB, Loki ingest, Grafana. If this is red the dashboards lie. Note: Alertmanager is running but NOT scraped by Prometheus (no job) \u2014 its metrics are absent by design gap, not fault.",
+  "tags": ["observability", "meta"],
+  "timezone": "browser",
+  "schemaVersion": 39,
+  "refresh": "30s",
+  "time": { "from": "now-6h", "to": "now" },
+  "panels": [
+    {
+      "id": 1, "type": "stat", "title": "Targets up",
+      "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] } }, "overrides": [] },
+      "targets": [{ "expr": "count(up == 1)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 2, "type": "stat", "title": "Targets down",
+      "description": "Known structural downs: comfyui (no metrics endpoint), gitea (metrics off), and the homepage 'services' probes. Real regressions ride up from here too \u2014 check the table before dismissing.",
+      "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "count(up == 0)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 3, "type": "stat", "title": "TSDB head series",
+      "description": "Active series in memory. A jump = a new exporter or a label-cardinality leak.",
+      "gridPos": { "h": 4, "w": 5, "x": 8, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "prometheus_tsdb_head_series", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 4, "type": "stat", "title": "Discovered targets",
+      "gridPos": { "h": 4, "w": 5, "x": 13, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "sum(prometheus_sd_discovered_targets)", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 5, "type": "stat", "title": "Prometheus mem",
+      "gridPos": { "h": 4, "w": 3, "x": 18, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "bytes" }, "overrides": [] },
+      "targets": [{ "expr": "process_resident_memory_bytes{job=\"prometheus\"}", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 6, "type": "stat", "title": "Grafana up",
+      "gridPos": { "h": 4, "w": 3, "x": 21, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "mappings": [{ "type": "value", "options": { "1": { "text": "UP", "color": "green" }, "0": { "text": "DOWN", "color": "red" } } }], "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } }, "overrides": [] },
+      "targets": [{ "expr": "up{job=\"grafana\"}", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 7, "type": "table", "title": "Target health",
+      "description": "Every scrape target and its current up. Sort by Value to surface regressions.",
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "options": { "sortBy": [{ "displayName": "Value", "sortOrder": "Asc" }] },
+      "targets": [{ "expr": "up", "format": "table", "instant": true, "refId": "A" }]
+    },
+    {
+      "id": 8, "type": "timeseries", "title": "Scrape interval drift (p99 actual)",
+      "description": "Actual seconds between scrapes (summary quantile). Above the 15s scrape_interval = Prometheus is falling behind its own cadence (slow endpoints or overload).",
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 4 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [{ "expr": "prometheus_target_interval_length_seconds{quantile=\"0.99\"}", "legendFormat": "{{instance}}", "refId": "A" }]
+    },
+    {
+      "id": 9, "type": "timeseries", "title": "Scrape samples/s",
+      "description": "Total samples ingested across all jobs.",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
+      "targets": [{ "expr": "sum(rate(scrape_samples_scraped[5m]))", "legendFormat": "samples/s", "refId": "A" }]
+    },
+    {
+      "id": 10, "type": "timeseries", "title": "Loki request rate",
+      "description": "Loki served requests/s (push, query). A drop to zero means ingest or query has stopped.",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 13 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (route) (rate(loki_request_duration_seconds_count[5m]))", "legendFormat": "{{route}}", "refId": "A" }]
+    },
+    {
+      "id": 11, "type": "timeseries", "title": "Exporter process CPU",
+      "description": "CPU by monitored process \u2014 spot a runaway exporter or the Prometheus process itself.",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 21 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "fieldConfig": { "defaults": { "unit": "percentunit" }, "overrides": [] },
+      "targets": [{ "expr": "sum by (job) (rate(process_cpu_seconds_total[5m]))", "legendFormat": "{{job}}", "refId": "A" }]
+    }
+  ]
+}
@@ -0,0 +1,17 @@
+# Grafana datasource for Loki, declared in git rather than clicked into
+# grafana.db — same reason as the Prometheus datasource: a UI-configured
+# datasource is invisible to review, lost with the data volume, and unknown
+# to the repository. Loki OSS ships no web UI of its own, so Explore through
+# this datasource is the only way a human reads the log store.
+#
+# URL is the container name on dankclaw_default (the same address Caddy
+# proxies to), not the published loopback port.
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    uid: loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    editable: false
@@ -30,9 +30,9 @@
  },
  {
    "name": "open-notebook",
-    "category": "user",
+    "category": "admin",
    "section": "AI",
-    "description": "Podcast generator — notes in, spoken episode out: script written by frank-zealot, voices rendered by frank-tts. Guarded twice: Authentik at the edge, then its own login.",
+    "description": "Podcast generator — notes in, spoken episode out: script written by frank-archon, voices rendered by frank-tts. Guarded twice: Authentik at the edge, then its own login. Single shared password, no per-user accounts — admin-only.",
    "url": "http://notebook.franksdank.cloud",
    "docs": "",
    "probe": "127.0.0.1:8502",
@@ -1,10 +1,15 @@
 # Alerting and recording rules for the LLM fleet.
 #
-# One metric namespace: llamacpp:* — frank-zealot and frank-dragoon, both
-# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16
-# and 2026-09-20). frank-archon LEFT the family 2026-10-05: FreeToken (the
-# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was
-# deleted rather than left never-green.
+# Two metric namespaces, because two runtime families are scraped:
+#
+#   * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which
+#     re-exports the llama.cpp metric names). Verified from /metrics on
+#     2026-09-16 / 2026-09-20.
+#   * vllm:* — frank-archon, the radiance engine on Dankserver since
+#     2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so
+#     it is a separate scrape job in prometheus.yml and the rules below are a
+#     separate group. (The old claim that its scrape job was deleted is false:
+#     it re-joined the day it left the FreeToken engine and is `up` today.)
 # (The netavark published-port quirk that forces the Dankclaw endpoints
 # onto the gateway is same-host only.)
 #
@@ -22,7 +27,8 @@
 #     cached by cached + prompt.
 #   * Speculative decoding: drafts and accepted tokens are both counters,
 #     so LlmSpecDecodeOff watches the pair.
-#   * No latency histograms exist, so there is no p95 alert.
+#   * No latency histograms in llama.cpp, so no p95 alert there; radiance's
+#     vllm:* namespace DOES export one, so the vllm-fleet group alerts on it.
 #
 # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
 # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
@@ -112,6 +118,84 @@ groups:
          description: >-
            Prometheus cannot reach the LLM endpoint. frank-zealot and
            frank-dragoon are root units on Dankclaw: `systemctl is-active
-            frank-zealot` / `frank-dragoon`. Both scraped endpoints export
-            the llamacpp:* family. (frank-archon left the scrape set
-            2026-10-05 — FreeToken exports no /metrics.)
+            frank-zealot` / `frank-dragoon` (both export llamacpp:* —
+            dragoon via halogen's llama.cpp-compatible names). frank-archon
+            is a radiance endpoint on Dankserver (`systemctl show
+            frank-archon` there) exporting the vllm:* family; the
+            `frank-.*` selector covers all three regardless of namespace.
+
+  # frank-archon (radiance) exports the vLLM-compatible namespace, NOT
+  # llamacpp:*. Every rule here is keyed to a series the exporter actually
+  # emits (verified from /metrics 2026-10-20: num_requests_waiting,
+  # gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total,
+  # time_to_first_token_seconds). The llamacpp rules above evaluate nothing for
+  # this job and these evaluate nothing for llama.cpp/halogen — two clean
+  # namespaces, each keyed only to what its runtime exports.
+  - name: vllm-fleet
+    interval: 30s
+    rules:
+      # p95 time-to-first-token, shared with the dashboard. When the endpoint
+      # is idle every bucket rate is zero, histogram_quantile yields NaN, and
+      # neither this nor the alert below can fire on an idle engine.
+      - record: job:vllm_ttft_p95:rate5m
+        expr: |
+          histogram_quantile(0.95, sum by (job, le)
+              (rate(vllm:time_to_first_token_seconds_bucket[5m])))
+
+      # Radiance exports a real queue-depth gauge (llama.cpp only shows the
+      # wall as requests_deferred). --max-num-seqs is small, so a sustained
+      # wait is either real load or this endpoint carrying the fleet alone.
+      - alert: VllmRequestsQueueing
+        expr: vllm:num_requests_waiting > 0
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: requests waiting for 10 minutes"
+          description: >-
+            The radiance scheduler has requests queued. Either the fleet is
+            genuinely busy or this endpoint is carrying it alone.
+
+      # Radiance reports prefix reuse directly as a hit-rate gauge — no
+      # counter division needed. The token-rate floor keeps an idle engine
+      # (hit rate 0 with no traffic) from firing.
+      - alert: VllmPrefixCacheCollapse
+        expr: >-
+          vllm:gpu_prefix_cache_hit_rate < 0.20
+            and rate(vllm:prompt_tokens_total[5m]) > 0.05
+        for: 15m
+        labels:
+          severity: info
+        annotations:
+          summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes"
+          description: >-
+            Nearly every request prefilling from scratch. With several large
+            contexts in flight this is the capacity wall, not a routing fault.
+
+      # Preemption is the KV-thrash signature radiance reports that llama.cpp
+      # cannot: the engine evicts a running request to admit another. Any
+      # sustained rate means the KV cache cannot hold the concurrency in flight.
+      - alert: VllmPreempting
+        expr: rate(vllm:num_preemptions_total[5m]) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: engine preempting requests"
+          description: >-
+            The engine is evicting running requests to admit new ones — the KV
+            cache is undersized for the concurrency in flight.
+
+      # p95 first-token latency, kept HIGH on purpose: a large-prompt prefill
+      # legitimately spends many seconds before token 1, so this fires only on
+      # a wedged scheduler or a queue backed up behind one request.
+      - alert: VllmTtftSlow
+        expr: job:vllm_ttft_p95:rate5m > 120
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s"
+          description: >-
+            First-token latency is pathological, not an honest big prefill.
+            Likely a wedged scheduler or one request stalling the queue.
@@ -240,14 +240,12 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
 }

@test "alloy's UI address comes from the unit, not the config file" {
-    # alloy 1.x refuses to configure the ui service from a config block
-    # ("service ui does not support being configured"), so the unit moves the
-    # UI off the container's loopback with a command-line flag. Without that
-    # flag the port cannot be published and the service has no reachable
-    # health endpoint at all — which is why the homepage status card for
-    # alloy read "down" while it ran under compose.
+    # alloy 1.x refuses to configure the ui service from a config block, so the
+    # unit sets the UI address on the command line. As a Network=host root unit
+    # that address is the host loopback directly (no PublishPort) — reachable
+    # from the host, from nothing off-host.
    ! grep -qE '^(ui|server) \{' "$REPO_ROOT/shared/alloy/config.alloy"
    local unit="$REPO_ROOT/dankclaw/observability/quadlet/alloy.container"
-    grep -q -- '--server.http.listen-addr=0.0.0.0:12345' "$unit"
-    grep -q '^PublishPort=127.0.0.1:12345:12345$' "$unit"
+    grep -q -- '--server.http.listen-addr=127.0.0.1:12345' "$unit"
+    ! grep -q '^PublishPort=' "$unit"
 }
@@ -29,8 +29,14 @@ export default defineConfig({
      // green suite with a test that does not exist. 17- joined on 2026-09-16
      // (the hermes vhost, whose upstream is a host address rather than a
      // container name, so it needs its own pass condition). 18- joined on
-      // 2026-09-24 (OpenWebUI SSO through authentik).
-      testMatch: /(10|12|13|16|17|18)-.*\.spec\.ts/,
+      // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
+      // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
+      // follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
+      // 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
+      // spec until now: the 17-hermes "uncovered vhost broke silently" class).
+      // 22- joins Prometheus's app-is-really-there proof (its own API), closing
+      // the "not a login wall but maybe a 404" hole 20- closed for Grafana.
+      testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
      use: { storageState: ".auth/qa-e2e.json" },
      dependencies: ["setup"],
    },
@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
 // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
 // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
 // added here or the guard-chain audit is incomplete.
-const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"];
+const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];

 // Markers that prove app content leaked past the guard to an anonymous user.
 const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
  "seerr.franksdank.cloud": ["Request Movie"],
  "grafana.franksdank.cloud": ["Dashboards"],
  "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
+  "notebook.franksdank.cloud": ["Open Notebook"],
 };

 test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {
@@ -0,0 +1,56 @@
+import { expect, test } from "@playwright/test";
+
+// Grafana (grafana.franksdank.cloud) is an OAuth2 RELYING PARTY, not an
+// outpost-fronted proxy: the guard chain (13-sso) only proves an authenticated
+// session reaches Grafana's login page. The login itself happens through
+// Grafana's own "Sign in with Authentik" button, which the edge test never
+// follows — that is exactly how the RP login could be dead (500 on authorize,
+// or an email-claim gap bouncing back to /login) while 13- still passed green.
+// This spec drives the real RP round-trip and asserts a live Grafana session.
+
+const HOST = "grafana.franksdank.cloud";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("SSO: Grafana relying-party login lands a real session", async ({ page, request }) => {
+  test.setTimeout(180_000);
+
+  // Warm the guard: the vhost runs its own Anubis proof-of-work on first visit.
+  await page.goto(`https://${HOST}/`);
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes(CHALLENGE),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  // The "Sign in with Authentik" button targets this RP start URL exactly.
+  // Going straight there is the same code path without depending on the SPA's
+  // button markup.
+  await page.goto(`https://${HOST}/login/generic_oauth`);
+
+  // The chain: grafana -> auth (implicit consent, the session is already
+  // there) -> back to grafana with the code -> grafana provisions the user and
+  // redirects to "/". A dead provider/claims config bounces back to /login.
+  await expect
+    .poll(
+      async () => {
+        const u = new URL(page.url());
+        return u.hostname === HOST && u.pathname !== "/login";
+      },
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+
+  const body = await page.locator("body").innerText();
+  expect(body).not.toContain("Login failed");
+  expect(body.toLowerCase()).not.toContain("powered by authentik"); // IdP flow shell
+  expect(body).not.toContain(CHALLENGE);
+
+  // A genuine logged-in session exists: /api/user answers 200 with an email,
+  // not 401. This is the assertion the edge-only spec could never make.
+  const resp = await page.request.get(`https://${HOST}/api/user`);
+  expect(resp.status()).toBe(200);
+  const user = await resp.json();
+  expect(typeof user.email).toBe("string");
+  expect(user.email.length).toBeGreaterThan(0);
+});
@@ -0,0 +1,52 @@
+import { expect, test } from "@playwright/test";
+
+// open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
+// vhost whose outpost (notebook-proxy) fronts an app with its OWN password
+// login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
+// (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
+// login page is the expected landing. Until now this host appeared in neither
+// the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
+// have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
+// 14- now covers the unauthenticated gate; this spec owns authenticated
+// reachability with a POSITIVE app marker, not just "not a login wall".
+
+const VHOST = "https://notebook.franksdank.cloud/";
+const CHALLENGE = "Making sure you're not a bot";
+
+test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
+  test.setTimeout(180_000);
+  await page.goto(VHOST);
+
+  // The Anubis proof-of-work clears in a real browser and the outpost hands off
+  // to the app; the user must never be parked on an authentik flow.
+  await expect
+    .poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("notebook.franksdank.cloud");
+  expect(
+    new URL(page.url()).pathname.startsWith("/if/"),
+    "must not be parked on an authentik flow",
+  ).toBe(false);
+  expect(
+    await page.locator('input[name="uidField"]').count(),
+    "must not show the authentik login form",
+  ).toBe(0);
+
+  // Positive identity: the page really is open-notebook's own login, not a
+  // Caddy error or the IdP shell. The app answering its own login is the pass.
+  await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
+    timeout: 30_000,
+  });
+  await expect(page.locator("body"), "open-notebook login marker").toContainText(
+    /enter your password/i,
+    { timeout: 30_000 },
+  );
+  const body = await page.locator("body").innerText();
+  expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
+  expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
+
+  const resp = await page.request.get(VHOST);
+  expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
+});
@@ -0,0 +1,33 @@
+import { expect, test } from "@playwright/test";
+
+// Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
+// as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
+// under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
+// This is the app-is-really-there proof 20- provides for Grafana: with the
+// standard session the outpost hands off silently to the Prometheus UI, and the
+// app's own API answers success. A guard-route or upstream regression that lets
+// any non-Prometheus answer through fails here.
+
+test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
+  test.setTimeout(180_000);
+  await page.goto("https://prometheus.franksdank.cloud/");
+
+  // Challenge clears, outpost hands off silently to the app (no credential prompt).
+  await expect
+    .poll(
+      async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
+      { timeout: 90_000 },
+    )
+    .toBe(true);
+  await expect
+    .poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
+    .toBe("prometheus.franksdank.cloud");
+  expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
+  await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
+
+  // Load-bearing assertion: the Prometheus API itself answers, authenticated,
+  // through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
+  const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
+  expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
+  expect((await api.json()).status, "Prometheus API must report success").toBe("success");
+});
@@ -41,13 +41,17 @@ service_section() {
 }

@test "every observability unit is boot-persistent" {
-    # Generated units cannot be enabled on this host's podman 5.x v1 quadlet
-    # generator; [Install] + linger is the entire boot-persistence story.
+    # Generated units cannot be enabled on this host's podman quadlet generator;
+    # [Install] + linger is the boot-persistence story for the rootless five.
+    # Alloy is a ROOT system unit, so its [Install] targets multi-user.target.
    local svc
    for svc in "${UNITS[@]}"; do
+        [[ "$svc" == alloy ]] && continue
        grep -q '^WantedBy=default.target$' "$UNIT_DIR/$svc.container" \
            || { echo "$svc is not boot-persistent"; return 1; }
    done
+    grep -q '^WantedBy=multi-user.target$' "$UNIT_DIR/alloy.container" \
+        || { echo "alloy (root unit) is not boot-persistent"; return 1; }
 }

@test "every observability unit is self-healing" {
@@ -140,15 +144,20 @@ service_section() {
    grep -q 'reverse_proxy prometheus:9090' "$REPO_ROOT/dankclaw/Caddyfile"
    grep -q 'reverse_proxy loki:3100' "$REPO_ROOT/dankclaw/Caddyfile"
    local svc
-    for svc in prometheus grafana loki alloy; do
+    for svc in prometheus grafana loki; do
        grep -q "^ContainerName=$svc$" "$UNIT_DIR/$svc.container" \
            || { echo "$svc unit renames its container"; return 1; }
        grep -q '^Network=dankclaw_default$' "$UNIT_DIR/$svc.container" \
            || { echo "$svc is not on the network Caddy and its peers use"; return 1; }
    done
+    # Alloy is not on that network: it is a root container on Network=host, the
+    # only way to reach Loki's 127.0.0.1:3100 publish from outside a bridge.
+    grep -q '^ContainerName=alloy$' "$UNIT_DIR/alloy.container"
+    grep -q '^Network=host$' "$UNIT_DIR/alloy.container" \
+        || { echo "alloy must be Network=host to reach Loki on the host loopback"; return 1; }
    grep -q '"grafana:3000"' "$REPO_ROOT/shared/prometheus/prometheus.yml"
    grep -q '"loki:3100"' "$REPO_ROOT/shared/prometheus/prometheus.yml"
-    grep -q 'url = "http://loki:3100/loki/api/v1/push"' "$REPO_ROOT/shared/alloy/config.alloy"
+    grep -q 'url = "http://127.0.0.1:3100/loki/api/v1/push"' "$REPO_ROOT/shared/alloy/config.alloy"
 }

@test "every observability unit has a health check" {
@@ -239,12 +248,14 @@ service_section() {
 }

@test "startup order follows the data dependencies" {
-    # Grafana is useless without Prometheus and Alloy drops logs without Loki;
-    # both pairs are expressed as systemd ordering, not launch luck.
+    # Grafana is useless without Prometheus (both rootless, same manager). Alloy
+    # used to order on loki.service; as a ROOT unit it lives on the SYSTEM
+    # manager and cannot order against a rootless user unit, so it depends only
+    # on the network and retries its push until Loki answers.
    grep -q 'After=network-online.target prometheus.service' "$UNIT_DIR/grafana.container"
    grep -q 'Wants=network-online.target prometheus.service' "$UNIT_DIR/grafana.container"
-    grep -q 'After=network-online.target loki.service' "$UNIT_DIR/alloy.container"
-    grep -q 'Wants=network-online.target loki.service' "$UNIT_DIR/alloy.container"
+    grep -q '^After=network-online.target$' "$UNIT_DIR/alloy.container"
+    ! grep -q 'loki.service' "$UNIT_DIR/alloy.container"
 }

@test "the deployer installs, restarts and health-checks each unit" {
@@ -281,6 +292,24 @@ PAIRS
    grep -q 'build_compose_image alloy' "$REMOTE_DEPLOY"
 }

+@test "alloy is a root host-network unit that reads every journal" {
+    # The whole point of the root cutover: a rootless container cannot open the
+    # 0640 journal files. Pin the properties that make the pipeline carry data.
+    local u="$UNIT_DIR/alloy.container"
+    grep -q '^Network=host$' "$u"
+    grep -q '^Volume=/etc/machine-id:/etc/machine-id:ro$' "$u" \
+        || { echo "alloy missing the load-bearing machine-id mount"; return 1; }
+    grep -q '^Volume=/var/log/journal:/var/log/journal:ro$' "$u"
+    grep -q '^Volume=/run/log/journal:/run/log/journal:ro$' "$u"
+    grep -q -- '--server.http.listen-addr=127.0.0.1:12345' "$u"
+    # Deployer: system install + root image mirror, not the user quadlet loop.
+    grep -q '^deploy_alloy() {' "$REMOTE_DEPLOY"
+    grep -q 'install_system_quadlet dankclaw/observability/quadlet alloy' "$REMOTE_DEPLOY"
+    grep -q '/etc/containers/systemd/' "$REMOTE_DEPLOY"
+    grep -q 'mirror_image_to_root localhost/franksdank/alloy:latest' "$REMOTE_DEPLOY"
+    grep -q 'alloy) deploy_alloy ;;' "$REMOTE_DEPLOY"
+}
+
@test "no dankclaw compose file competes with the quadlet units" {
    # Two owners for one container name is how a compose run resurrects a
    # stale duplicate next to the quadlet-managed one. This file was retired on
@@ -313,33 +342,33 @@ PAIRS
    done
 }

-@test "prometheus scrapes every LLM endpoint that exports metrics" {
-    # The LLM endpoints are the services whose behaviour this fleet argues about
-    # most (KV pressure, prefix-cache reuse, queue depth). Both Dankclaw
-    # endpoints export the llamacpp:* family. frank-archon LEFT the set
-    # 2026-10-05: the FreeToken PR587 ROCm engine has NO /metrics endpoint —
-    # scraping it was a structural never-green (404), so the job was deleted
-    # rather than left vacuous (the Lucebox DS4.1 precedent, 2026-09-26).
-    # The two Dankclaw endpoints are HOST processes reached through
-    # host.containers.internal: this Prometheus is a bridge-networked
-    # container, and on this netavark host a container can only reach the
-    # host's LAN IP at a PUBLISHED port — a host-process bind like
-    # llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it
-    # answers at 169.254.1.2:7770 (why frank-zealot stayed red by construction
-    # for days until the 2026-09-20 /targets audit).
+@test "prometheus scrapes every LLM endpoint, in its real metric namespace" {
+    # Every LLM endpoint that exports /metrics must have a scrape job. As of
+    # 2026-10-05 there are TWO runtime namespaces scraped:
+    #   * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen,
+    #     re-exporting the llama.cpp names), both root host processes on
+    #     Dankclaw reached via host.containers.internal — this Prometheus is a
+    #     bridge container and a host-process bind is refused at the LAN IP,
+    #     which is why frank-zealot read red by construction for days.
+    #   * vllm:* — frank-archon (radiance) on Dankserver, a different machine,
+    #     so plain LAN routing reaches 192.168.0.235:7770. It RE-JOINED the
+    #     scrape set with radiance; the old "job deleted, exports no /metrics"
+    #     note belonged to the retired FreeToken engine and is false today, so
+    #     this seam now asserts archon IS scraped (a bare `! grep` here was
+    #     set -e-exempt and never enforced the opposite).
    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
    grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-zealot not scraped"; return 1; }
    grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-dragoon not scraped"; return 1; }
-    ! grep -q '"192.168.0.235:7770"' "$cfg"
-    grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \
-        && { echo "frank-archon folded into a Dankclaw gateway target"; return 1; }
-    for job in frank-zealot frank-dragoon; do
+    grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-archon not scraped"; return 1; }
+    for job in frank-zealot frank-dragoon frank-archon; do
        grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
    done
-    ! grep -q 'job_name: "frank-archon"' "$cfg"
-    # None of them is reached through a guarded vhost: Prometheus cannot solve
-    # an Anubis challenge, so a franksdank.cloud target is red by construction.
-    ! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
+    # No LLM endpoint is reached through a guarded vhost: Prometheus cannot
+    # solve an Anubis challenge, so a franksdank.cloud target is red by
+    # construction.
+    grep -qE '^[[:space:]]*-[[:space:]]*"?https?://[^"]*franksdank' "$cfg" \
+        && { echo "a scrape target points at a guarded vhost"; return 1; }
+    true
 }

@test "grafana provisioning is declared in git and installed by the deployer" {
@@ -378,28 +407,44 @@ names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rule
 for want in ('job:llm_prefix_cache_reuse:rate5m',
             'job:llm_spec_accept_length:rate5m',
             'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
-             'LlmSpecDecodeOff', 'LlmEndpointDown'):
+             'LlmSpecDecodeOff', 'LlmEndpointDown',
+             'job:vllm_ttft_p95:rate5m', 'VllmRequestsQueueing',
+             'VllmPrefixCacheCollapse', 'VllmPreempting', 'VllmTtftSlow'):
    assert want in names, want
 # Every rule must be keyed to a series that actually exists. A rule naming a
 # series nobody exports evaluates nothing forever and errors never — the
-# failure this seam exists for. Since the 2026-09-26 cutover the only scraped
-# LLM runtime is llama.cpp (frank-zealot, frank-dragoon); the vllm: family left
-# with frank-medium's radiance runtime and must not survive in any expr.
+# failure this seam exists for. Two runtimes are scraped in two namespaces:
+# llama.cpp/halogen (llamacpp:*) for frank-zealot and frank-dragoon, and
+# radiance (vllm:*) for frank-archon. Both families were read straight off the
+# endpoints' /metrics; anything else is a guess.
 exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
-# llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20
-# (frank-zealot, frank-dragoon). Anything else is a guess.
-live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
-        'llamacpp:requests_deferred', 'llamacpp:requests_processing',
-        'llamacpp:spec_decode_num_drafts_total',
-        'llamacpp:spec_decode_num_accepted_tokens_total',
-        'llamacpp:spec_decode_num_draft_tokens_total',
-        'llamacpp:tokens_predicted_total',
-        'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
-        'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode'}
+live = {
+    # llama.cpp / halogen, measured from /metrics 2026-09-16 and 2026-09-20.
+    'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
+    'llamacpp:requests_deferred', 'llamacpp:requests_processing',
+    'llamacpp:spec_decode_num_drafts_total',
+    'llamacpp:spec_decode_num_accepted_tokens_total',
+    'llamacpp:spec_decode_num_draft_tokens_total',
+    'llamacpp:tokens_predicted_total',
+    'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
+    'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode',
+    # radiance (frank-archon), measured from /metrics 2026-10-20.
+    'vllm:num_requests_waiting', 'vllm:gpu_prefix_cache_hit_rate',
+    'vllm:num_preemptions_total', 'vllm:prompt_tokens_total',
+    'vllm:time_to_first_token_seconds_bucket',
+}
 used = {m for e in exprs
        for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
 assert used <= live, used - live
-assert not any(':vllm' in e for e in exprs), 'vllm: series in rules but no vllm endpoint is scraped'
+# The two namespaces must not bleed across groups: a llamacpp: selector
+# evaluates nothing for the radiance job and vice versa, so each group may
+# reference only its own family.
+per = {g['name']: {m.split(':')[0] for r in g['rules']
+                   for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', r.get('expr', ''))}
+       for g in d['groups']}
+for name, fams in per.items():
+    allowed = {'llamacpp'} if name == 'llm-fleet' else {'vllm'}
+    assert fams <= allowed, (name, fams)
 print('rules ok:', len(names))
 "
    # A rule file that nothing loads evaluates nothing, and Prometheus reports
@@ -290,7 +290,9 @@ vhost = sys.argv[2]
 cards = [s for s in data if s["name"] == "open-notebook"]
 assert len(cards) == 1, f"expected exactly one open-notebook card, got {len(cards)}"
 card = cards[0]
-assert card["category"] == "user" and card["section"] == "AI", card
+# category moved to "admin": single shared password, no per-user accounts, so it
+# is positioned as an admin-side tool, not a user-facing app (see the vault page).
+assert card["category"] == "admin" and card["section"] == "AI", card
 assert card["probe"] == "127.0.0.1:8502", card["probe"]
 assert card["method"] == "http"
 # Deployed and guarded: the card is live and its link is the vhost, so the
Author	SHA1	Message	Date
dank	961713e92d	Merge pull request 'test(e2e): cover notebook + prometheus guard-surface gaps (full-suite audit)' (#56 ) from feat/e2e-audit-coverage into main	2026-10-20 21:21:44 -07:00
dank	01db29e754	Merge pull request 'feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring)' (#55 ) from feat/grafana-starter-dashboards into main	2026-10-20 21:20:53 -07:00
Frank	ab06e4c221	test(e2e): cover the two guard-surface gaps found in the full-suite audit Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based); two real coverage gaps surfaced: - notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken upstream would stay invisible - the exact 17-hermes class. Added to the 14- guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a new 21- that proves authenticated reachability to open-notebook's own login through the guard chain, with a positive app marker. - prometheus.franksdank.cloud was only ever asserted '<500 and not a login wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really- there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo answers {status:success}. Both specs wired into the standard project (playwright testMatch), which the e2e_scaffold bats gate enforces. Verified live against the fleet; full bats suite green.	2026-10-20 21:16:23 -07:00
Frank	1006e96312	feat(grafana): starter dashboards for host/system, Loki logs, and observability self-monitoring llm-fleet.json already owns LLM capacity+spec-decode. These three cover the domains that had NO dashboard: - host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both Dankclaw and Dankserver (templated by job node-dankclaw\|node-dankserver) - logs-overview: Loki volume + per-container error rate + a sample log panel; container is parsed from the JSON line at query time (journal streams have no container_name stream label) - observability: the stack's own health - Prometheus target up/down + scrape interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem. Panel queries verified live against Prometheus/Loki; provisioned dashboards confirmed via /api/search. Alertmanager is running but NOT scraped by Prometheus - noted in the dashboard, no empty panels built on it.	2026-10-20 21:10:06 -07:00
dank	7d85fed1c9	Merge pull request 'Fix Grafana Authentik SSO (500 + missing email claim) + RP landing test' (#54 ) from fix/grafana-authentik-sso into main	2026-10-20 21:04:02 -07:00
Frank	d8196ce499	docs: Grafana reads claims from userinfo, not USE_ID_TOKEN The login lands once authentik emits the email claim (the IdP fix); Grafana needs no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in the literal-secret seam. Correct the two SSO notes accordingly.	2026-10-20 20:41:57 -07:00
Frank	6f442a14a3	fix(observability): repair Grafana Authentik OAuth2 relying-party login The Authentik SSO button returned "Server Error" (500): the grafana provider had a NULL authorization_flow and the authorize view crashed building a None flow planner. After that, the login failed with "InternalError" — no email claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s. Unit env: request authentik real scopes (openid email profile, not the GitLab user:email that crashes the authorize view); standard userinfo API_URL with no ?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare string instead of {"email":..}). Verified end to end: RP login lands on / and /api/user answers 200 with the real email, user provisioned as Viewer.	2026-10-20 20:41:36 -07:00
Frank	afe72781bc	docs: correct Grafana Authentik SSO facts The vault claimed the fork property-mapping evaluator was broken and userinfo returned only sub, so email could never reach Grafana (outcome unverified). That was a misdiagnosis: the real causes were the NULL authorization_flow (the NoneType.slug crash) and the email scope mapping returning a bare string rather than a dict. Record the four verified causes and fixes, point to the repair script and the new landing spec, and drop the unverified caveat.	2026-10-20 20:40:15 -07:00
Frank	200039d589	test(e2e): cover Grafana relying-party SSO landing (spec 20) 13-sso proves an authenticated session reaches Grafana login page but never follows the OAuth button, so the relying-party login could be dead (500, or an email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives /login/generic_oauth through the real Anubis+authentik round-trip and asserts a live session (/api/user 200 with a non-empty email). Wire 20- into the standard project (qa-e2e session).	2026-10-20 20:40:15 -07:00
Frank	91d9864bb0	fix(observability): repair Grafana Authentik OAuth2 relying-party login The Authentik SSO button returned "Server Error" (500) because the grafana provider authorization_flow was NULL and the authorize view crashed building a None flow planner. After that, the login failed with "InternalError": no email claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s. Unit env: request authentik real scopes (openid email profile, not the GitLab user:email that crashes the authorize view); read claims from the id_token (USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into a /emails 404); pin email/sub/preferred_username attribute paths; allow first-login provisioning since the Anubis->Authentik guard fronts the callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side causes (NULL authorization_flow; the grafana-email-scope mapping that returned a bare string instead of {"email":..}). Verified end to end: RP login lands on / and /api/user answers 200 with the real email, user provisioned as Viewer.	2026-10-20 20:40:15 -07:00
dank	980491668c	Merge pull request 'Alloy ships logs: root host-network service so it can read the journal' (#53 ) from feat/alloy-root-journal into main	2026-10-20 20:05:00 -07:00
Frank	e494617599	docs(alloy): document the root journal-read cutover Alloy page rewritten to the shipped root design (why root, the two mounts that make it work, deploy-pending status); Observability current state marks the log pipeline fixed-in-code, awaiting deploy.	2026-10-20 19:50:15 -07:00
Frank	9809ed9c74	test(observability): pin the root alloy shape The seams encoded alloy-as-rootless (default.target, dankclaw_default, loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update them to the root/system/host shape and add a test pinning the load-bearing properties: Network=host, the /etc/machine-id mount, the journal mounts, loopback UI, and the system-install + root-mirror deploy path.	2026-10-20 19:50:15 -07:00
Frank	634fef242b	feat(alloy): run it as a root host-network service so it reads the journal A rootless Alloy shipped zero lines: the journal files are root:systemd-journal 0640 (other::---) and a rootless container carries no supplementary groups, so sd_journal_open sees an empty view silently; the rootless quadlet container logs also live in user-1000.journal, not the ctr.log paths the old config tailed. As root, one loki.source.journal reads system.journal + user-*.journal and carries both. Network=host because Loki publishes 127.0.0.1:3100 and no bridge container can reach the host loopback; the UI binds loopback directly (no PublishPort). The /etc/machine-id mount is load-bearing — sd_journal finds the journal dir by the local machine id; without it the source reads 0 lines, health green. deploy_alloy routes around the rootless loop: build rootless, mirror the image into the root store (a root build fails on DNS in the build container), retire the old rootless alloy unit, install to /etc/containers/systemd. Validated with a throwaway root probe: 31,825 lines read, 26,966 pushed to Loki in 20s, zero drops.	2026-10-20 19:50:15 -07:00
dank	2fe68e69af	Merge pull request 'Observability: vllm rule group for frank-archon + Loki datasource; correct stale log-pipeline facts' (#52 ) from fix/observability-vllm-loki into main	2026-10-20 19:39:38 -07:00
Frank	70dd7fbe6d	docs(observability): correct current state and log-pipeline root cause AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a Loki datasource is provisioned beside Prometheus. Observability.md: current-state census refreshed (three LLM targets up, two rule groups, Loki datasource); logs still carry no data. Alloy.md: replace the 'wrong glob' theory with the proven cause - a rootless container cannot read the 0640 systemd-journal files (a rootless busybox read returns DENIED), and rootless quadlet logs land in the user journal the system-only source never reads. Closing it is a permission/architecture decision, not a config glob.	2026-10-20 19:28:20 -07:00
Frank	2626332828	feat(grafana): provision the Loki datasource Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100 so logs are readable through Grafana Explore. Loki ships no web UI of its own and the guarded vhost serves only the API; this is the human surface.	2026-10-20 19:28:20 -07:00
Frank	eb4d9910cb	feat(observability): add vllm-fleet rule group for frank-archon radiance exports vllm:* metrics (verified on the live endpoint), not llamacpp:: nothing in llm-fleet could ever see frank-archon. Add a vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache collapse, preemption, p95 time-to-first-token) keyed to vllm:. Split the observability seam so llm-fleet rules bind llamacpp:* and vllm-fleet rules bind vllm:*, and neither may evaluate the other's namespace. Drops the retired no-metrics-for-archon doctrine the seam still asserted and the vacuous namespace grep.	2026-10-20 19:28:20 -07:00
dank	5cc2b3c25a	Merge pull request 'docs(openwebui): TTS reflects live frank-tts wiring (wilwheaton, LAN IP)' (#51 ) from docs/openwebui-tts-voice into main	2026-10-20 18:20:04 -07:00
Frank	18a29c40d2	docs(openwebui): TTS config reflects the live frank-tts wiring The Text-to-speech section recorded audio.tts base_url host.containers.internal and voice narrator1. Re-keyed the live config DB 2026-10-20: voice wilwheaton (narrator1 retired), api_base_url the canonical host LAN IP 192.168.0.143:7772. 7772 is published so host.containers.internal also reaches it, but the LAN IP matches this page's own on-host law. Browser-kokoro example voice updated to the current pack; the dated 2026-10-18 verification is left as history.	2026-10-20 18:18:01 -07:00
dank	73a7ec3fc1	Merge pull request 'chore(frank-tts): repoint narrator slots to live voice packs' (#50 ) from chore/narrator-voice-repoint into main	2026-10-20 18:12:52 -07:00
Frank	130e9d739f	chore(frank-tts): repoint narrator slots to the live voice packs Frank replaced the narrator1/2/3 packs with cloned character voices (wilwheaton, vaati, dr.kern, garrickhagon, jeffhays). The three narrator slots are now carried by real packs: narrator1=wilwheaton, narrator2=vaati, narrator3=dr.kern. - frank-tts.env DEFAULT_VOICE narrator1 -> wilwheaton (a voice-less request 400'd on the dead name; wilwheaton is narrator1's successor). - Docs: correct the /health census and the podcast-speaker references in Frank TTS.md and Open Notebook.md to the live packs. Dated 2026-10-18/19 measurement logs are left as history. Open Notebook's live dank-duo speaker profile was repointed over REST (Frank->wilwheaton, Comrade->vaati) - app-side, not in this commit. Bats fixtures keep narrator1 as a self-contained sample pack; untouched.	2026-10-20 18:11:40 -07:00
dank	854890a155	Merge pull request 'chore(open-notebook): admin-side positioning + single-tenant note' (#49 ) from chore/open-notebook-admin-tenant into main	2026-10-20 17:44:45 -07:00
Frank	d9f3d1be1e	chore(open-notebook): position as admin-side; document single-tenant open-notebook has no user accounts upstream (PasswordAuthMiddleware = one shared password; SurrealDB has no user/account/owner table — verified in the running image). One shared password = one shared workspace; the guard chain gates access but cannot partition data, and Authentik identity is never consumed. Multi-user is approved-but-unbuilt upstream (#712, post-v2); OIDC declined (#607). So move the homepage card from the user grid to the admin tier and add the single-tenant boundary to the vault page. Fix the card's stale 'frank-zealot' -> frank-archon. Update the seam's incidental category pin to the new intentional value. Password rotated on the host (not repo).	2026-10-20 17:38:22 -07:00
dank	afa3a4521c	Merge pull request 'docs(authentik): record 7-day login-session TTL + setter script' (#48 ) from chore/authentik-session-ttl into main	2026-10-20 16:57:29 -07:00