Compare commits

...
Author SHA1 Message Date
Frank ab06e4c221 test(e2e): cover the two guard-surface gaps found in the full-suite audit
Audit of all 12 e2e specs for the 'green-but-vacuous' class that broke
Grafana's SSO. The suite is otherwise sound (incident-driven, leakage-based);
two real coverage gaps surfaced:

- notebook.franksdank.cloud (open-notebook, deployed 2026-10-19) was a guarded
  vhost in NEITHER the 13-/14- GUARDED lists NOR any dedicated spec. A broken
  upstream would stay invisible - the exact 17-hermes class. Added to the 14-
  guard audit (unauth -> authentik gate, 'Open Notebook' leak marker) plus a
  new 21- that proves authenticated reachability to open-notebook's own login
  through the guard chain, with a positive app marker.
- prometheus.franksdank.cloud was only ever asserted '<500 and not a login
  wall' (13-/11-), which a 404 also passes. New 22- adds the app-is-really-
  there proof 20- gives Grafana: SSO handoff + its own /api/v1/status/buildinfo
  answers {status:success}.

Both specs wired into the standard project (playwright testMatch), which the
e2e_scaffold bats gate enforces. Verified live against the fleet; full bats
suite green.
2026-10-20 21:16:23 -07:00
dank 7d85fed1c9 Merge pull request 'Fix Grafana Authentik SSO (500 + missing email claim) + RP landing test' (#54) from fix/grafana-authentik-sso into main 2026-10-20 21:04:02 -07:00
Frank d8196ce499 docs: Grafana reads claims from userinfo, not USE_ID_TOKEN
The login lands once authentik emits the email claim (the IdP fix); Grafana needs
no USE_ID_TOKEN toggle, and dropping it also avoids a *_TOKEN false-positive in
the literal-secret seam. Correct the two SSO notes accordingly.
2026-10-20 20:41:57 -07:00
Frank 6f442a14a3 fix(observability): repair Grafana Authentik OAuth2 relying-party login
The Authentik SSO button returned "Server Error" (500): the grafana provider had
a NULL authorization_flow and the authorize view crashed building a None flow
planner. After that, the login failed with "InternalError" — no email claim
reached Grafana, so it fell back to GET <api_url>/emails, which authentik 404s.

Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); standard userinfo API_URL with no
?api=1 (Grafana v13 turns ?api=1 into a /emails 403); email/sub/preferred_username
attribute paths; ALLOW_SIGN_UP=true since the Anubis->Authentik guard fronts the
callback. scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two
IdP-side causes (NULL authorization_flow; grafana-email-scope returning a bare
string instead of {"email":..}). Verified end to end: RP login lands on / and
/api/user answers 200 with the real email, user provisioned as Viewer.
2026-10-20 20:41:36 -07:00
Frank afe72781bc docs: correct Grafana Authentik SSO facts
The vault claimed the fork property-mapping evaluator was broken and userinfo
returned only sub, so email could never reach Grafana (outcome unverified). That
was a misdiagnosis: the real causes were the NULL authorization_flow (the
NoneType.slug crash) and the email scope mapping returning a bare string rather
than a dict. Record the four verified causes and fixes, point to the repair
script and the new landing spec, and drop the unverified caveat.
2026-10-20 20:40:15 -07:00
Frank 200039d589 test(e2e): cover Grafana relying-party SSO landing (spec 20)
13-sso proves an authenticated session reaches Grafana login page but never
follows the OAuth button, so the relying-party login could be dead (500, or an
email-claim gap bouncing to /login) while the suite stayed green. Spec 20 drives
/login/generic_oauth through the real Anubis+authentik round-trip and asserts a
live session (/api/user 200 with a non-empty email). Wire 20- into the standard
project (qa-e2e session).
2026-10-20 20:40:15 -07:00
Frank 91d9864bb0 fix(observability): repair Grafana Authentik OAuth2 relying-party login
The Authentik SSO button returned "Server Error" (500) because the grafana
provider authorization_flow was NULL and the authorize view crashed building a
None flow planner. After that, the login failed with "InternalError": no email
claim reached Grafana, so it fell back to GET <api_url>/emails, which authentik
404s.

Unit env: request authentik real scopes (openid email profile, not the GitLab
user:email that crashes the authorize view); read claims from the id_token
(USE_ID_TOKEN=true) and drop ?api=1 from the API URL (Grafana v13 turns it into
a /emails 404); pin email/sub/preferred_username attribute paths; allow
first-login provisioning since the Anubis->Authentik guard fronts the callback.

scripts/authentik-grafana-sso-fix.sh idempotently re-applies the two IdP-side
causes (NULL authorization_flow; the grafana-email-scope mapping that returned
a bare string instead of {"email":..}). Verified end to end: RP login lands on /
and /api/user answers 200 with the real email, user provisioned as Viewer.
2026-10-20 20:40:15 -07:00
dank 980491668c Merge pull request 'Alloy ships logs: root host-network service so it can read the journal' (#53) from feat/alloy-root-journal into main 2026-10-20 20:05:00 -07:00
Frank e494617599 docs(alloy): document the root journal-read cutover
Alloy page rewritten to the shipped root design (why root, the two
mounts that make it work, deploy-pending status); Observability current
state marks the log pipeline fixed-in-code, awaiting deploy.
2026-10-20 19:50:15 -07:00
Frank 9809ed9c74 test(observability): pin the root alloy shape
The seams encoded alloy-as-rootless (default.target, dankclaw_default,
loki:3100 by name, 0.0.0.0+PublishPort UI, After=loki.service). Update
them to the root/system/host shape and add a test pinning the
load-bearing properties: Network=host, the /etc/machine-id mount, the
journal mounts, loopback UI, and the system-install + root-mirror
deploy path.
2026-10-20 19:50:15 -07:00
Frank 634fef242b feat(alloy): run it as a root host-network service so it reads the journal
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.

As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.

deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
2026-10-20 19:50:15 -07:00
dank 2fe68e69af Merge pull request 'Observability: vllm rule group for frank-archon + Loki datasource; correct stale log-pipeline facts' (#52) from fix/observability-vllm-loki into main 2026-10-20 19:39:38 -07:00
Frank 70dd7fbe6d docs(observability): correct current state and log-pipeline root cause
AGENTS.md: the rule set is now two groups (llm-fleet + vllm-fleet) and a
Loki datasource is provisioned beside Prometheus.
Observability.md: current-state census refreshed (three LLM targets up,
two rule groups, Loki datasource); logs still carry no data.
Alloy.md: replace the 'wrong glob' theory with the proven cause - a
rootless container cannot read the 0640 systemd-journal files (a rootless
busybox read returns DENIED), and rootless quadlet logs land in the user
journal the system-only source never reads. Closing it is a
permission/architecture decision, not a config glob.
2026-10-20 19:28:20 -07:00
Frank 2626332828 feat(grafana): provision the Loki datasource
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
2026-10-20 19:28:20 -07:00
Frank eb4d9910cb feat(observability): add vllm-fleet rule group for frank-archon
radiance exports vllm:* metrics (verified on the live endpoint), not
llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a
vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache
collapse, preemption, p95 time-to-first-token) keyed to vllm:*.

Split the observability seam so llm-fleet rules bind llamacpp:* and
vllm-fleet rules bind vllm:*, and neither may evaluate the other's
namespace. Drops the retired no-metrics-for-archon doctrine the seam
still asserted and the vacuous namespace grep.
2026-10-20 19:28:20 -07:00
dank 5cc2b3c25a Merge pull request 'docs(openwebui): TTS reflects live frank-tts wiring (wilwheaton, LAN IP)' (#51) from docs/openwebui-tts-voice into main 2026-10-20 18:20:04 -07:00
Frank 18a29c40d2 docs(openwebui): TTS config reflects the live frank-tts wiring
The Text-to-speech section recorded audio.tts base_url host.containers.internal
and voice narrator1. Re-keyed the live config DB 2026-10-20: voice wilwheaton
(narrator1 retired), api_base_url the canonical host LAN IP 192.168.0.143:7772.
7772 is published so host.containers.internal also reaches it, but the LAN IP
matches this page's own on-host law. Browser-kokoro example voice updated to the
current pack; the dated 2026-10-18 verification is left as history.
2026-10-20 18:18:01 -07:00
dank 73a7ec3fc1 Merge pull request 'chore(frank-tts): repoint narrator slots to live voice packs' (#50) from chore/narrator-voice-repoint into main 2026-10-20 18:12:52 -07:00
Frank 130e9d739f chore(frank-tts): repoint narrator slots to the live voice packs
Frank replaced the narrator1/2/3 packs with cloned character voices
(wilwheaton, vaati, dr.kern, garrickhagon, jeffhays). The three narrator
slots are now carried by real packs: narrator1=wilwheaton, narrator2=vaati,
narrator3=dr.kern.

- frank-tts.env DEFAULT_VOICE narrator1 -> wilwheaton (a voice-less request
  400'd on the dead name; wilwheaton is narrator1's successor).
- Docs: correct the /health census and the podcast-speaker references in
  Frank TTS.md and Open Notebook.md to the live packs. Dated 2026-10-18/19
  measurement logs are left as history.

Open Notebook's live dank-duo speaker profile was repointed over REST
(Frank->wilwheaton, Comrade->vaati) - app-side, not in this commit.
Bats fixtures keep narrator1 as a self-contained sample pack; untouched.
2026-10-20 18:11:40 -07:00
dank 854890a155 Merge pull request 'chore(open-notebook): admin-side positioning + single-tenant note' (#49) from chore/open-notebook-admin-tenant into main 2026-10-20 17:44:45 -07:00
Frank d9f3d1be1e chore(open-notebook): position as admin-side; document single-tenant
open-notebook has no user accounts upstream (PasswordAuthMiddleware = one
shared password; SurrealDB has no user/account/owner table — verified in the
running image). One shared password = one shared workspace; the guard chain
gates access but cannot partition data, and Authentik identity is never
consumed. Multi-user is approved-but-unbuilt upstream (#712, post-v2); OIDC
declined (#607). So move the homepage card from the user grid to the admin
tier and add the single-tenant boundary to the vault page. Fix the card's
stale 'frank-zealot' -> frank-archon. Update the seam's incidental category
pin to the new intentional value. Password rotated on the host (not repo).
2026-10-20 17:38:22 -07:00
dank afa3a4521c Merge pull request 'docs(authentik): record 7-day login-session TTL + setter script' (#48) from chore/authentik-session-ttl into main 2026-10-20 16:57:29 -07:00
Frank 9cc8ca1c88 docs(authentik): record the 7-day login-session TTL and add a setter script
The login-session lifetime (raised to days=7 on 2026-10-20) is owned by the
default-authentication-login stage's session_duration, not a global setting and
not SESSION_COOKIE_AGE. Authentik exposes no SESSION_SAVE_EVERY_REQUEST, so the
window is absolute. session_duration is a STRING idiom; a timedelta stores a form
timedelta_from_string rejects, crashing login. Add scripts/authentik-set-login-session-duration.sh
that validates the form before writing (DB state, not repo-managed, re-run after
a blueprint re-import). Trim the page back under the OKF 500-line cap. tests/run.sh green.
2026-10-20 16:55:44 -07:00
dank 9c05c40aa0 Merge pull request 'docs(agent): scaffold agent-skills config in the vault' (#47) from chore/agent-skills-vault-setup into main 2026-10-20 16:11:47 -07:00
27 changed files with 821 additions and 243 deletions

No files matched your search

+5 -3
View File
@@ -166,9 +166,11 @@ bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed — `dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured and delivering as of one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are 2026-09-17**: Grafana datasources (Prometheus and Loki) and the `llm-fleet`
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/ dashboard are provisioned from `shared/grafana/provisioning/`, and
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are `shared/prometheus/rules/llm-fleet.yml` loads two rule groups — `llm-fleet`
(llama.cpp/halogen, `llamacpp:*`) and `vllm-fleet` (radiance, `vllm:*`) — alert
+ recording rules, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/ delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN (`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
+35 -39
View File
@@ -1,55 +1,51 @@
# Quadlet unit: alloy (rootless, boot-persistent). # Quadlet unit: alloy (ROOT system service, boot-persistent).
# #
# Install path on dankclaw: ~/.config/containers/systemd/alloy.container # Install path on dankclaw: /etc/containers/systemd/alloy.container
# Unit name: alloy.service (container name: alloy) # Unit name: alloy.service (container name: alloy) — SYSTEM manager, root.
# #
# Log shipper. This replaces the compose service of the same name in # WHY ROOT (not rootless, unlike its observability siblings). Alloy reads the
# dankclaw/compose.yaml ("Observability" block). # host journal. The journal files are root:systemd-journal 0640 (other::---);
# a rootless container carries no supplementary groups and cannot open them —
# a rootless read of system.journal returns DENIED (see the Alloy page in the
# vault). Running as root, one loki.source.journal reads system.journal AND
# user-1000.journal, so it carries both host logs and the rootless Quadlet
# containers' logs. Validated 2026-10-20: a root probe read 31,825 journal
# lines and pushed 26,966 into Loki in 20 s, zero drops.
# #
# IMAGE. franksdank/alloy is built locally by deploy-remote.sh # NETWORK=host. Loki publishes 127.0.0.1:3100 (loopback only) on the host, and
# (build_compose_image) from dankclaw/alloy/Containerfile: upstream # no bridge container — rootless or root — can reach the host's loopback, so
# grafana/alloy plus a static busybox for the healthcheck. Pull=never keeps a # this unit runs on the host network and pushes to 127.0.0.1:3100. The side
# boot from reaching for that local tag at a registry. # effect: the UI must bind loopback too, so Exec sets the listen-addr to
# 127.0.0.1 (the old rootless unit used 0.0.0.0 only because container loopback
# was not host loopback; under Network=host they are the same address, so the
# unauthenticated UI stays loopback-only — nothing off-host reaches it).
# #
# NETWORK. It joins dankclaw_default so its config's loki.write endpoint # THE /etc/machine-id MOUNT IS LOAD-BEARING. sd_journal locates the journal
# resolves the container name loki:3100. # directory by the local machine id; without it the container opens nothing and
# the journal source reads exactly 0 lines — silently, no error, health green.
# This mount is the entire reason the pipeline carries data. Never remove it.
# #
# THE UI PORT EXISTS ON PURPOSE. Alloy's UI listens on the container's own # IMAGE. Built rootless by build_compose_image alloy, then mirrored into the
# 127.0.0.1 by default and cannot be moved from the config file (alloy 1.x # root image store by the deployer (a root build runs in the build container's
# refuses the ui block), so the compose service had no reachable health # own resolver and fails on DNS here). Pull=never: boot must not reach for the
# endpoint at all: the homepage status card for alloy reported "down" # local tag at a registry.
# continuously and there was nothing for a deploy to wait on. The command
# below therefore moves the UI to the container's wildcard address and the
# port is published on host loopback only, which puts alloy in the same class
# as the other three: reachable from the host, by Caddy over the private
# network, and by nothing off-host. Alloy has no authentication, which is
# exactly why it is not published wider.
#
# SOURCES. The host journald socket and the host log/container-storage trees
# are bind-mounted read-only; Alloy tails them and pushes to Loki. The config
# lives at /opt/alloy/config/config.alloy, installed from
# shared/alloy/config.alloy on every deploy.
#
# ORDERING. Wants/After loki.service: without a reachable Loki, Alloy's write
# client fails and its log pipeline backs up.
[Unit] [Unit]
Description=Alloy log shipper (Dankclaw) Description=Alloy log shipper (Dankclaw)
After=network-online.target loki.service After=network-online.target
Wants=network-online.target loki.service
[Container] [Container]
ContainerName=alloy ContainerName=alloy
Image=localhost/franksdank/alloy:latest Image=localhost/franksdank/alloy:latest
Pull=never Pull=never
Network=dankclaw_default Network=host
Exec=run --server.http.listen-addr=0.0.0.0:12345 /etc/alloy/config.alloy Exec=run --server.http.listen-addr=127.0.0.1:12345 /etc/alloy/config.alloy
PublishPort=127.0.0.1:12345:12345 Volume=/opt/alloy/config:/etc/alloy:ro
Volume=/opt/alloy/config:/etc/alloy Volume=/etc/machine-id:/etc/machine-id:ro
Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro Volume=/run/systemd/journal/socket:/run/systemd/journal/socket:ro
Volume=/var/log:/var/log:ro Volume=/var/log/journal:/var/log/journal:ro
Volume=/var/lib/containers:/var/lib/containers:ro Volume=/run/log/journal:/run/log/journal:ro
HealthCmd=/usr/local/bin/busybox wget --spider -q http://localhost:12345/-/healthy HealthCmd=/usr/local/bin/busybox wget --spider -q http://127.0.0.1:12345/-/healthy
HealthInterval=30s HealthInterval=30s
HealthTimeout=10s HealthTimeout=10s
HealthRetries=3 HealthRetries=3
@@ -60,4 +56,4 @@ Restart=always
RestartSec=10 RestartSec=10
[Install] [Install]
WantedBy=default.target WantedBy=multi-user.target
@@ -47,9 +47,24 @@ Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
# client_id query parameter — /application/o/<slug>/authorize/ does not exist. # client_id query parameter — /application/o/<slug>/authorize/ does not exist.
Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/ Environment=GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://auth.franksdank.cloud/application/o/authorize/
Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/ Environment=GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://auth.franksdank.cloud/application/o/token/
# ?api=1 = introspection-style userinfo: returns custom claims (groups) too. # Standard OIDC userinfo. Grafana v13 fetches <api_url>/emails for the address
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/?api=1 # ONLY when the token carries no `email` claim (authentik has no /emails path and
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=false # 404s it, failing the login) — so the claim must actually be there: the fix is
# the grafana-email-scope mapping (see scripts/authentik-grafana-sso-fix.sh),
# not a Grafana flag. Do NOT append ?api=1 to this URL: Grafana then GETs
# <...>/userinfo/?api=1/emails, a 403.
Environment=GF_AUTH_GENERIC_OAUTH_API_URL=https://auth.franksdank.cloud/application/o/userinfo/
# authentik's scopes are openid/email/profile. `user:email` (a GitLab-ism) is
# not a known scope here; requesting it makes the authorize view crash.
# Quoted: quadlet word-splits Environment=, so an unquoted spaced list
# truncates SCOPES to its first token.
Environment="GF_AUTH_GENERIC_OAUTH_SCOPES=openid email profile"
Environment=GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
Environment=GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=sub
Environment=GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=preferred_username
# First SSO login auto-provisions the user. Safe: unauthenticated traffic never
# reaches this callback — the Anubis -> Authentik guard fronts every request.
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP=true
Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true Environment=GF_AUTH_GENERIC_OAUTH_AUTO_UPSERT=true
Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true Environment=GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_ORG=true
Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1 Environment=GF_AUTH_GENERIC_OAUTH_ORG_ID=1
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env bash
# Repair the Grafana OAuth2 relying-party SSO on Dankclaw's authentik.
#
# grafana.franksdank.cloud logs in through authentik as an OAuth2 relying party.
# Two DB-side facts (the provider was created by hand via `ak shell`, never
# repo-managed) silently broke that login, and BOTH had previously been
# mis-recorded here as a single "the property-mapping evaluator is broken, so
# userinfo returns only `sub`" myth. They are neither, and the evaluator is fine:
#
# 1. provider.authorization_flow was NULL. The authorize view builds a flow
# planner from it and crashes with
# `AttributeError: 'NoneType' object has no attribute 'slug'` — an HTTP 500
# "Server Error — Powered by authentik", before any claim is touched.
# Fix: point it at default-provider-authorization-implicit-consent, the
# flow every other provider in this instance already uses.
# 2. The `grafana-email-scope` ScopeMapping's expression was the bare string
# `user.email`. authentik merges a scope mapping's result into the token as
# a MAPPING: a returned dict adds its keys as claims; a returned string is
# keyed under the mapping's NAME ("grafana-email-scope"), so no `email`
# claim is ever emitted. Grafana, finding no email, falls back to
# GET <api_url>/emails, authentik 404s that path, and the login dies with
# "Login failed: InternalError". Fix: return
# {"email": user.email, "email_verified": True} — the shape the stock
# OpenID 'email' mapping uses. The dict-valued `grafana-profile-scope`
# mapping has always worked, which is the proof the evaluator is fine.
#
# This is authentik DB state. It is NOT in any deployable file: a blueprint
# re-import, or an ak-shell re-creation of the grafana provider, reverts BOTH
# facts and the symptom returns verbatim. Re-run this after any such rebuild.
# Every step is idempotent.
#
# Grafana's own half — SCOPES="openid email profile", a standard userinfo API_URL
# with no `?api=1`, and ALLOW_SIGN_UP=true — lives in
# dankclaw/observability/quadlet/grafana.container and deploys normally.
#
# Usage: authentik-grafana-sso-fix.sh (override host: AUTHENTIK_HOST=…)
set -euo pipefail
HOST="${AUTHENTIK_HOST:-dankclaw}"
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<'PYEOF'
from authentik.flows.models import Flow
from authentik.providers.oauth2.models import OAuth2Provider
p = OAuth2Provider.objects.filter(name="grafana").first()
if not p:
raise SystemExit("no OAuth2Provider named 'grafana'")
# 1. The authorization flow every other provider uses.
want = Flow.objects.filter(slug="default-provider-authorization-implicit-consent").first()
if not want:
raise SystemExit("default-provider-authorization-implicit-consent flow is missing")
if p.authorization_flow_id != want.pk:
p.authorization_flow = want
p.save()
print(f"set grafana.authorization_flow -> {want.slug}")
else:
print(f"grafana.authorization_flow already {want.slug}")
# 2. The email scope mapping must RETURN A DICT, not a bare string.
m = p.property_mappings.filter(name="grafana-email-scope").first()
if not m:
raise SystemExit("grafana-email-scope mapping is missing")
fixed = 'return {"email": user.email, "email_verified": True}'
if m.expression.strip() != fixed:
m.expression = fixed
m.save()
print("repaired grafana-email-scope -> returns {email, email_verified}")
else:
print("grafana-email-scope expression already correct")
# 3. The id_token must carry the scope claims; Grafana reads identity from it.
if not p.include_claims_in_id_token:
p.include_claims_in_id_token = True
p.save()
print("enabled include_claims_in_id_token")
else:
print("include_claims_in_id_token already on")
PYEOF
echo "Done. Verify: sign in at https://grafana.franksdank.cloud via Authentik;"
echo "the round trip must land on / and /api/user must answer 200 with an email."
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Set the authentik login-session TTL on Dankclaw.
#
# The login-session lifetime is governed by the `default-authentication-login`
# stage's `session_duration`, NOT a global setting and NOT SESSION_COOKIE_AGE
# (that is the pre-login session, sessions.unauthenticated_age). Authentik does
# not expose SESSION_SAVE_EVERY_REQUEST at all, so the session is an absolute
# window from login, not a sliding one.
#
# The field is a STRING in authentik's own idiom ("days=7", "seconds=30",
# "hours=12") — not a Python timedelta. Writing str(timedelta(days=7)) stores
# "7 days, 0:00:00", which authentik's timedelta_from_string parser REJECTS
# (ValueError), and login then crashes computing the session expiry. Always pass
# the "key=value" form; this script validates it before writing.
#
# Existing sessions keep their old expiry; the new TTL applies from the next
# login. This is DB state — it is NOT repo-managed and a blueprint re-import or
# rebuild-from-scratch reverts it. Run this again after such a rebuild.
#
# Usage: authentik-set-login-session-duration.sh ["days=7"] (default days=7)
set -euo pipefail
DUR="${1:-days=7}"
STAGE="default-authentication-login"
HOST="${AUTHENTIK_HOST:-dankclaw}"
# Refuse anything that is not the safe key=value duration idiom.
if ! printf '%s' "$DUR" | grep -Eq '^[A-Za-z]+=[0-9]+([ ]+[A-Za-z]+=[0-9]+)*$'; then
echo "invalid duration '$DUR' — use authentik form, e.g. 'days=7' or 'seconds=30'" >&2
exit 2
fi
ssh "$HOST" 'podman exec -i authentik-server ak shell' <<PYEOF
from django.apps import apps
from authentik.lib.utils.time import timedelta_from_string
dur = "$DUR"
timedelta_from_string(dur) # validate the form before touching the row
ULS = apps.get_model("authentik_stages_user_login", "UserLoginStage")
s = ULS.objects.get(name="$STAGE")
s.session_duration = dur
s.save(update_fields=["session_duration"])
back = ULS.objects.get(name="$STAGE").session_duration
assert back == dur, "persisted value mismatch: %r" % back
print("session_duration set to", repr(back), "->", timedelta_from_string(back))
PYEOF
echo "Login-session TTL for '$STAGE' is now $DUR (applies on next login)."
+72 -13
View File
@@ -487,13 +487,15 @@ seed_grafana_env() {
|| { echo "grafana env missing GRAFANA_OAUTH_CLIENT_SECRET; set it in dankclaw/.env" >&2; return 1; } || { echo "grafana env missing GRAFANA_OAUTH_CLIENT_SECRET; set it in dankclaw/.env" >&2; return 1; }
} }
# The five observability containers left compose on 2026-09-11 (they were # The observability containers left compose on 2026-09-11 (they were started by
# started by hand with podman-compose and never came back after a reboot). # hand with podman-compose and never came back after a reboot). Five are ordinary
# They are ordinary rootless Quadlet units now: same images, same mounts, same # rootless Quadlet units: same images, same mounts, same arguments, same
# arguments, same container names, so nothing that addresses them by name — # container names, so nothing that addresses them by name — Caddy, Prometheus'
# Caddy, Prometheus' own scrape config, Alloy's loki.write endpoint — changed. # own scrape config — changed. Only the published ports did: the three HTTP UIs
# Only the published ports did: the three HTTP UIs bind 127.0.0.1, because the # bind 127.0.0.1, because the compose services published on 0.0.0.0 and none of
# compose services published on 0.0.0.0 and none of them authenticates. # them authenticates. Alloy is the exception: a ROOT system service on
# Network=host, because reading the host journal needs privileges a rootless
# container cannot get. It is handled by deploy_alloy, not the loop below.
deploy_observability() { deploy_observability() {
local service="$1" local service="$1"
# Observability units live under <target>/observability/quadlet: Dankclaw # Observability units live under <target>/observability/quadlet: Dankclaw
@@ -512,10 +514,6 @@ deploy_observability() {
copy_loki_config copy_loki_config
build_compose_image loki build_compose_image loki
;; ;;
alloy)
copy_alloy_config
build_compose_image alloy
;;
grafana) grafana)
copy_grafana_provisioning copy_grafana_provisioning
seed_grafana_env || return 1 seed_grafana_env || return 1
@@ -534,11 +532,71 @@ deploy_observability() {
alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;; alertmanager) wait_for_health alertmanager http://127.0.0.1:9093/-/healthy 60 ;;
grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;; grafana) wait_for_health grafana http://127.0.0.1:3002/api/health 90 ;;
loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;; loki) wait_for_health loki http://127.0.0.1:3100/ready 60 ;;
alloy) wait_for_health alloy http://127.0.0.1:12345/-/healthy 60 ;;
node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;; node-exporter) wait_for_health node-exporter http://127.0.0.1:9100/metrics 60 ;;
esac esac
} }
# Mirror an image the deployer built rootless into the ROOT image store. A root
# quadlet reads root storage; a root build (build_compose_image is rootless) is
# not possible on this host — the build container's resolver fails DNS — so the
# only way to get a root-readable copy is save-and-load. Compares image ids so a
# rebuilt alloy actually re-lands in root storage instead of leaving a stale tag.
mirror_image_to_root() {
local ref="$1" rid
rid="$(podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" || {
echo "ERROR: rootless image $ref is not built; cannot mirror to root" >&2
return 1
}
if [[ "$(run_privileged podman image inspect "$ref" --format '{{.Id}}' 2>/dev/null)" == "$rid" ]]; then
return 0
fi
local tar; tar="$(mktemp /var/tmp/alloy-mirror.XXXXXX.tar)"
podman save -o "$tar" "$ref"
run_privileged podman load -i "$tar"
rm -f "$tar"
}
# Install a quadlet into the SYSTEM generator dir (root manager) instead of the
# user dir. Boot persistence is the unit's own [Install] section (multi-user
# target); the v1 system generator creates the .wants on daemon-reload, so —
# like the user units — no `systemctl enable` (generated units cannot be enabled
# here).
install_system_quadlet() {
local source_dir="$1" unit="$2"
run_privileged mkdir -p /etc/containers/systemd
run_privileged cp "$source_dir/$unit.container" /etc/containers/systemd/
run_privileged systemctl daemon-reload
}
# Alloy as a ROOT system service on Network=host. See the unit file: a rootless
# container cannot read the 0640 journal files, and it cannot reach Loki's
# 127.0.0.1:3100 publish from a bridge netns. Root + host network fixes both;
# the price is that this one container runs as root and on the host network.
deploy_alloy() {
copy_alloy_config
build_compose_image alloy # rootless build (resolver works)
mirror_image_to_root localhost/franksdank/alloy:latest
# Retire the rootless unit BEFORE the system one starts: both managers
# generate alloy.service, and two containers named "alloy" both binding
# 127.0.0.1:12345 cannot coexist. Move the user quadlet out of the generator
# dir (it regenerates from there) and stop the user unit.
if [[ -f "$quadlet_unit_dir/alloy.container" ]] ||
systemctl --user is-active --quiet alloy.service 2>/dev/null; then
systemctl --user stop alloy.service 2>/dev/null || true
if [[ -f "$quadlet_unit_dir/alloy.container" ]]; then
local trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-alloy-root-cutover"
mkdir -p "$trash"
mv -- "$quadlet_unit_dir/alloy.container" "$trash/alloy.container"
echo "retired rootless alloy.container -> $trash"
fi
systemctl --user daemon-reload
fi
install_system_quadlet dankclaw/observability/quadlet alloy
run_privileged systemctl restart alloy.service
wait_for_health alloy http://127.0.0.1:12345/-/healthy 60
}
build_compose_image() { build_compose_image() {
local service="$1" image cfhash oldhash containerfile local service="$1" image cfhash oldhash containerfile
# Every file the image COPIES IN, beside the Containerfile. The skip below # Every file the image COPIES IN, beside the Containerfile. The skip below
@@ -1941,7 +1999,8 @@ deploy_service() {
copyparty) deploy_copyparty ;; copyparty) deploy_copyparty ;;
plex) deploy_plex ;; plex) deploy_plex ;;
tautulli) deploy_tautulli ;; tautulli) deploy_tautulli ;;
prometheus|grafana|loki|alloy|node-exporter|alertmanager) deploy_observability "$service" ;; alloy) deploy_alloy ;;
prometheus|grafana|loki|node-exporter|alertmanager) deploy_observability "$service" ;;
comfyui) deploy_comfyui ;; comfyui) deploy_comfyui ;;
*) deploy_compose_service "$service" ;; *) deploy_compose_service "$service" ;;
esac esac
+2 -2
View File
@@ -7,7 +7,7 @@
# whichever TTS engine wins the engine decision (dankclaw/voiceservice/app.py # whichever TTS engine wins the engine decision (dankclaw/voiceservice/app.py
# is a single stdlib-only file in a locally built image). Consumers — omp's # is a single stdlib-only file in a locally built image). Consumers — omp's
# TTS device, OpenWebUI, open-notebook — address a VOICE BY NAME # TTS device, OpenWebUI, open-notebook — address a VOICE BY NAME
# (voice=narrator1, or the harness' voice_id=narrator1) and never see a path: # (voice=wilwheaton, or the harness' voice_id=wilwheaton) and never see a path:
# a name is real exactly when /voices/<name>/reference.wav exists, and # a name is real exactly when /voices/<name>/reference.wav exists, and
# anything else is a 400 that lists the names that are. # anything else is a 400 that lists the names that are.
# #
@@ -65,7 +65,7 @@ TTS_UPSTREAM_MODEL=voxcpm2
# name still 400s with the list that exists — this covers the absent case only — # name still 400s with the list that exists — this covers the absent case only —
# and the journal marks such a line `(default)` so an unrequested voice never # and the journal marks such a line `(default)` so an unrequested voice never
# reads as a requested one. Any pack directory in /home/dank/voices is legal. # reads as a requested one. Any pack directory in /home/dank/voices is legal.
DEFAULT_VOICE=narrator1 DEFAULT_VOICE=wilwheaton
# CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not # CLI-backend knobs, inert while TTS_BACKEND=omni-http. Empty means "not
# configured": the endpoint answers a 500 that says so instead of running # configured": the endpoint answers a 500 that says so instead of running
+9 -8
View File
@@ -21,7 +21,7 @@ never learn a path.
**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts` **Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`
both units `active`, no drop-ins, and the homepage card's `wip` gate came off in both units `active`, no drop-ins, and the homepage card's `wip` gate came off in
the same change. Probes on the deployed endpoint: `/health` answers the same change. Probes on the deployed endpoint: `/health` answers
`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}` `{"status":"ok","backend":"omni-http","voices":["dr.kern","garrickhagon","jeffhays","vaati","wilwheaton"]}`
and two different voices rendered the same sentence into **different** bytes and two different voices rendered the same sentence into **different** bytes
(24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a (24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a
pass-through of one default. The operator harness routes its `speech` role here; pass-through of one default. The operator harness routes its `speech` role here;
@@ -42,8 +42,8 @@ render needs the app installed, not this endpoint.
## Naming a voice ## Naming a voice
A voice **is** a directory name. `voice=narrator1` resolves to A voice **is** a directory name. `voice=wilwheaton` resolves to
`/voices/narrator1/reference.wav`, and that file — not a manifest, not a config `/voices/wilwheaton/reference.wav`, and that file — not a manifest, not a config
entry — is what makes the name real. omp's TTS device may send `voice_id`; the entry — is what makes the name real. omp's TTS device may send `voice_id`; the
shim accepts either. A call that sends **neither** — the harness's normal shape, shim accepts either. A call that sends **neither** — the harness's normal shape,
because no harness setting carries a voice for this transport — speaks because no harness setting carries a voice for this transport — speaks
@@ -59,10 +59,11 @@ loudly instead of letting it half-work.
Names are user-visible: an unknown voice comes back as Names are user-visible: an unknown voice comes back as
`400 unknown voice 'x'; available voices: …`, and that list is the honest `400 unknown voice 'x'; available voices: …`, and that list is the honest
census. The podcast speakers are `narrator1`, `narrator2`, `narrator3` — census. As of 2026-10-20 the live packs are `dr.kern`, `garrickhagon`,
[[Open Notebook]] is configured with exactly those three, so renaming them `jeffhays`, `vaati`, `wilwheaton`. [[Open Notebook]]'s `dank-duo` profile speaks
breaks episode rendering, not just speech. A personal voice gets the person's `wilwheaton` (Frank) and `vaati` (Comrade), and `DEFAULT_VOICE` is `wilwheaton`.
name, lowercase. Renaming a pack that a speaker profile names breaks episode rendering, not just
speech. A personal voice gets the person's name, lowercase.
## Voice packs ## Voice packs
@@ -221,5 +222,5 @@ harness bundle and measured the day of the flip:
| Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` | | Endpoint answers, synthesis fails | the engine: `podman healthcheck run frank-tts-engine`, then `systemctl --user status frank-tts-engine.service` |
| First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) | | First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) |
| The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` | | The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` |
| A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] | | A consumer rejects `wilwheaton` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] |
--- ---
@@ -45,6 +45,8 @@ open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook))
The app's own login stays on behind the chain. It is not redundant: publishing the API on loopback does **not** hide it from this host's other containers, which can dial `open-notebook:5055` by name on `dankclaw_default`. The chain decides *who from outside* gets in; `OPEN_NOTEBOOK_PASSWORD` decides what anything that already reached the port may do. The app's own login stays on behind the chain. It is not redundant: publishing the API on loopback does **not** hide it from this host's other containers, which can dial `open-notebook:5055` by name on `dankclaw_default`. The chain decides *who from outside* gets in; `OPEN_NOTEBOOK_PASSWORD` decides what anything that already reached the port may do.
**Single tenant — there are no user accounts, so this is an admin-side tool.** Upstream ships one global password and nothing else: `PasswordAuthMiddleware` compares `Bearer <OPEN_NOTEBOOK_PASSWORD>`, and the SurrealDB schema has no user/account/owner table (verified in the running image, 2026-10-20). One shared password therefore means **one shared workspace** — the guard chain can gate the door but cannot partition the data, and Authentik identity is never consumed. Multi-user is *approved but unbuilt* upstream (issue **#712**, "design phase, targeted post-v2"); OIDC/SSO was declined for now (**#607** closed). Real per-person isolation means **one deployment per person**, not a flag to flip. For that reason the homepage lists open-notebook under **admin**, not as a user-facing app.
## What the image actually runs ## What the image actually runs
One container, three processes under supervisord (`supervisord.conf` upstream): `uvicorn api.main:app --port 5055` (`:12`), the job worker (`:25`), and the Next.js frontend (`:39`). Two consequences the config depends on: One container, three processes under supervisord (`supervisord.conf` upstream): `uvicorn api.main:app --port 5055` (`:12`), the job worker (`:25`), and the Next.js frontend (`:39`). Two consequences the config depends on:
@@ -68,7 +70,7 @@ Credentials live in the database, encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`,
| Model row `voxcpm2` | type `text_to_speech` on the frank-tts credential. The name is decorative: frank-tts ignores the request's `model` field and synthesizes whatever `voice` names ([[Frank TTS]]) | | Model row `voxcpm2` | type `text_to_speech` on the frank-tts credential. The name is decorative: frank-tts ignores the request's `model` field and synthesizes whatever `voice` names ([[Frank TTS]]) |
| Model row `frank-archon` | type `language`; `PUT /api/models/defaults` points chat, transformation and large-context at it | | Model row `frank-archon` | type `language`; `PUT /api/models/defaults` points chat, transformation and large-context at it |
| Model row `qwen3-embedding-0.6b` | type `embedding` on the NPU-lane credential; `default_embedding_model` points at it, so every note save embeds through the XDNA2 lane ([[frank-dragoon]]) | | Model row `qwen3-embedding-0.6b` | type `embedding` on the NPU-lane credential; `default_embedding_model` points at it, so every note save embeds through the XDNA2 lane ([[frank-dragoon]]) |
| Speaker profile `dank-duo` | two speakers, Frank → `voice_id: narrator1`, Comrade → `narrator2`, `voice_model` = the TTS row | | Speaker profile `dank-duo` | two speakers, Frank → `voice_id: wilwheaton`, Comrade → `vaati`, `voice_model` = the TTS row |
| Episode profile `dank-daily` | `num_segments: 4`, `speaker_config` = `dank-duo`, outline and transcript LLM = the archon row, briefing that ends "Never invent numbers." | | Episode profile `dank-daily` | `num_segments: 4`, `speaker_config` = `dank-duo`, outline and transcript LLM = the archon row, briefing that ends "Never invent numbers." |
Order that works over REST: credentials → model rows (`POST /api/models` with `credential: <credential id>`) → `PUT /api/models/defaults` → speaker profile (`voice_model: <model id>`) → episode profile (`speaker_config` / `outline_llm` / `transcript_llm`). `POST /api/models/{id}/test` proves each row: the language row answered with a real completion from frank-archon; the TTS row reached frank-tts and came back `unknown voice 'alloy'; available voices: narrator1, narrator2, narrator3` — the app's own test path hard-codes an OpenAI voice, so that message *is* the proof the leg works. Order that works over REST: credentials → model rows (`POST /api/models` with `credential: <credential id>`) → `PUT /api/models/defaults` → speaker profile (`voice_model: <model id>`) → episode profile (`speaker_config` / `outline_llm` / `transcript_llm`). `POST /api/models/{id}/test` proves each row: the language row answered with a real completion from frank-archon; the TTS row reached frank-tts and came back `unknown voice 'alloy'; available voices: narrator1, narrator2, narrator3` — the app's own test path hard-codes an OpenAI voice, so that message *is* the proof the leg works.
@@ -106,13 +108,13 @@ A **speaker profile** carries one to four speakers (`docs/3-USER-GUIDE/creating-
| Speaker | Voice ID to type | | Speaker | Voice ID to type |
|---------|------------------| |---------|------------------|
| Speaker 1 | `narrator1` | | Speaker 1 | `wilwheaton` |
| Speaker 2 | `narrator2` | | Speaker 2 | `vaati` |
| Speaker 3 | `narrator3` | | Speaker 3 | `dr.kern` |
Those are the pack names frank-tts resolves from `/home/dank/voices` (`scripts/frank-tts.env` header: a voice is real exactly when `/voices/<name>/reference.wav` exists). Two traps, both from upstream's shipped defaults: Those are the pack names frank-tts resolves from `/home/dank/voices` (`scripts/frank-tts.env` header: a voice is real exactly when `/voices/<name>/reference.wav` exists). Two traps, both from upstream's shipped defaults:
- The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `narrator1`/`narrator2` pair (or a third/fourth) rather than editing the seeded ones in place. - The three seeded speaker profiles carry **OpenAI voice names** (`nova`, `alloy`, …) and **no voice model at all** — they show a "Setup required" badge, and generation fails with a voice error until they are edited (`creating-podcasts.md:13`, `:33`; `local-tts.md:89`). Add a `wilwheaton`/`vaati` pair (or a third/fourth) rather than editing the seeded ones in place.
- Two speakers with the same voice are hard to follow (`creating-podcasts.md:89`). Distinct packs per speaker is the point of the 1–4 shape. - Two speakers with the same voice are hard to follow (`creating-podcasts.md:89`). Distinct packs per speaker is the point of the 1–4 shape.
## Two buttons that fail against frank-tts, and why that is cosmetic ## Two buttons that fail against frank-tts, and why that is cosmetic
+8 -5
View File
@@ -290,9 +290,12 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
## Text-to-speech (frank-tts) ## Text-to-speech (frank-tts)
TTS is the fleet's one OpenAI-compatible speech endpoint — TTS is the fleet's one OpenAI-compatible speech endpoint. The config DB
`http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url` `audio.tts.*` is set to engine `openai`, `audio.tts.openai.api_base_url`
in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That `http://192.168.0.143:7772/v1`, `audio.tts.voice` `wilwheaton` (re-keyed
2026-10-20 from the retired `narrator1`; 7772 is a published port so
`host.containers.internal` also reaches it, but the LAN IP is the canonical
on-host form). That
port is the rootless quadlet `frank-tts` (this repo: port is the rootless quadlet `frank-tts` (this repo:
`dankclaw/voiceservice/quadlet/frank-tts.container` `dankclaw/voiceservice/quadlet/frank-tts.container`
`scripts/frank-tts.env`), and its shim resolves `voice=<name>` against `scripts/frank-tts.env`), and its shim resolves `voice=<name>` against
@@ -306,8 +309,8 @@ ships an in-page Kokoro implementation (chunk carrying a frozen voice table
`onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype* `onnx-community/Kokoro-82M-v1.0-ONNX`, plus a personal *Kokoro.js dtype*
setting). Its engine select takes `""` (Default) or `browser-kokoro`; when setting). Its engine select takes `""` (Default) or `browser-kokoro`; when
`browser-kokoro` is chosen the page synthesizes locally and rejects any `browser-kokoro` is chosen the page synthesizes locally and rejects any
voice outside its fixed pack — so `narrator1` fails with voice outside its fixed pack — so `wilwheaton` fails with
`Voice: "narrator1" not found. Should be one of: af_heart, …` **without a `Voice: "wilwheaton" not found. Should be one of: af_heart, …` **without a
single request reaching the server**. Personal audio settings are single request reaching the server**. Personal audio settings are
browser: no `user.settings` row and no container env carries an audio browser: no `user.settings` row and no container env carries an audio
override, yet the toast still appears ⇒ the choice lives in that browser's override, yet the toast still appears ⇒ the choice lives in that browser's
+80 -39
View File
@@ -3,70 +3,111 @@ type: reference
description: "Alloy is Grafana's log and telemetry shipper." description: "Alloy is Grafana's log and telemetry shipper."
tags: tags:
- infrastructure - infrastructure
timestamp: 2026-09-11 timestamp: 2026-10-20
--- ---
# Alloy # Alloy
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It is wired to forward container and journal logs to Loki, and every unit in the path is healthy — but as of 2026-10-18 it is delivering **zero** log lines. See "Delivery state" below. Alloy is Grafana's log and telemetry shipper. It replaces Promtail (end of life
as of 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/). It
forwards the host journal — including the rootless Quadlet containers' logs —
to Loki. Unlike its observability siblings it runs as a **root system service on
the host network**, because a rootless container cannot read the journal (see
*Why root* below).
**Host:** Dankclaw (one instance per host) — **Port:** 12345 (loopback) — **Docs:** https://grafana.com/docs/alloy/latest **Host:** Dankclaw (one instance per host) — **UI/health:** 127.0.0.1:12345
(loopback) — **Docs:** https://grafana.com/docs/alloy/latest
## Facts ## Facts
| Item | Value | | Item | Value |
|------|-------| |------|-------|
| Image | `franksdank/alloy:latest` (local build: upstream + a static busybox for the healthcheck) | | Image | `localhost/franksdank/alloy:latest` — built rootless, then mirrored into the root image store by the deployer (a root build fails on DNS in this host's build container) |
| Published | `127.0.0.1:12345` → 12345 (its own UI/health port) | | Unit | `dankclaw/observability/quadlet/alloy.container` → system `alloy.service`, installed at `/etc/containers/systemd/` (root manager) |
| Unit | `dankclaw/observability/quadlet/alloy.container` → `alloy.service` | | Config | `shared/alloy/config.alloy` → `/opt/alloy/config/config.alloy` on every deploy |
| Config | `shared/alloy/config.alloy`, installed to `/opt/alloy/config/config.alloy` on every deploy (`scripts/deploy-remote.sh:281-284`) | | Network | `host` — required so it can reach Loki's `127.0.0.1:3100` publish |
| Inputs | host journald socket, podman container logs, `/var/log` | | UI | `--server.http.listen-addr=127.0.0.1:12345` (loopback; nothing off-host reaches it) |
| Output | `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) | | Inputs | host journal: `system.journal` + `user-*.journal` (root reads both) |
| Successor to | Promtail (EOL 2026-03-02, https://grafana.com/docs/loki/latest/send-data/promtail/) | | Output | `http://127.0.0.1:3100/loki/api/v1/push` |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alloy.container:58-63`); installed at `~/.config/containers/systemd/alloy.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active alloy` 2026-10-18) | | Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=multi-user.target`; boot persistence is the `[Install]` section (generated system units cannot be `enable`d here) |
| Delivery | **Configured and healthy, currently delivering zero data** — see "Delivery state" | | Successor to | Promtail (EOL 2026-03-02) |
| Deploy status | root cutover built and tested in-repo (2026-10-20); the **live** instance is still the old rootless unit until `./scripts/deploy.sh dankclaw --service alloy` runs |
## Why Alloy and not Promtail ## Why root (the load-bearing decision)
Promtail reached end-of-life on 2026-03-02 and Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one: no Promtail config or unit has ever existed in this repository (`git log -S promtail` over the whole history returns no commit that added one; the compose-era service of the same name was already Alloy — `dankclaw/observability/quadlet/alloy.container:6-7`). A rootless Alloy delivered **zero** lines, silently and with a green health
check. Two independent causes, both proven on the host 2026-10-20:
1. **The journal is not readable by a rootless container.** The files are
`root:systemd-journal 0640` (`group:adm`, `other::---`). Even bind-mounted, a
rootless container has no supplementary groups, so opening them returns
`DENIED`. `loki.source.journal` then sees an empty view and the counter just
sits at 0 — no error logged.
2. **Container logs live in the *user* journal, not `ctr.log`.** The rootless
Quadlet containers log via the journald driver into `user-1000.journal`; the
old file globs (`/var/lib/containers/.../ctr.log`, `/var/log/containers/*.log`)
point at a root-storage layout that does not exist here.
**Root fixes both at once.** As root the container opens every journal file
under the machine-id directory — `system.journal` *and* `user-1000.journal` —
so a single `loki.source.journal` carries host services and the rootless
containers' logs together. Validated with a throwaway root probe before writing
the unit: **31,825 journal lines read, 26,966 pushed to Loki in 20 s, zero
drops**, and Loki's label set went from empty to populated.
### The two mounts that make it work
- **`/etc/machine-id` (ro) — load-bearing.** `sd_journal` locates the journal
directory by the *local* machine id. Without it the container opens nothing and
the journal source reads exactly 0 lines — silently, health green. This single
mount is the whole difference between a working and an empty pipeline; do not
remove it. (A root probe without it reads 0; with it, 31,825.)
- **`Network=host`.** Loki publishes `127.0.0.1:3100` (loopback only). No bridge
container — rootless or root — can reach the host's loopback, so Alloy must
share the host network to push to `127.0.0.1:3100`. Consequence: the UI binds
`127.0.0.1` directly (no `PublishPort`), so it stays loopback-only as before.
## Configuration ## Configuration
`shared/alloy/config.alloy` is the source of truth: `shared/alloy/config.alloy` is the source of truth — one `loki.source.journal`
(labelled `job="systemd-journal"`) forwarding to the loopback `loki.write`
- Reads journald logs from the host's `/run/systemd/journal/socket`, labelled `job="systemd-journal"` (`shared/alloy/config.alloy:20-26`). endpoint. There is deliberately **no** container-file source: container stdout is
- Reads container logs from two globs: `/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log` (`job="podman"`) and `/var/log/containers/*.log` (`job="containers"`) (`config.alloy:29-35`). in the journal.
- Forwards everything to `http://loki:3100/loki/api/v1/push` by container name on `dankclaw_default` (`config.alloy:13-17`).
## Volumes ## Volumes
| Container path | Host path | | Container path | Host path |
|----------------|-----------| |----------------|-----------|
| `/etc/alloy` | `/opt/alloy/config` | | `/etc/alloy` | `/opt/alloy/config` (ro) |
| `/run/systemd/journal/socket` | host journald socket (read-only) | | `/etc/machine-id` | host machine id (ro) — **required** |
| `/var/log` | host logs (read-only) | | `/run/systemd/journal/socket` | host journald socket (ro) |
| `/var/lib/containers` | podman storage (read-only) | | `/var/log/journal` | persistent journal files (ro) |
| `/run/log/journal` | runtime journal files (ro) |
## The UI port is published on purpose ## The UI stays on loopback
Alloy's UI listens on the container's own `127.0.0.1:12345` by default, and Alloy 1.x refuses to configure the `ui` service from a config block ("service ui does not support being configured", `shared/alloy/config.alloy:8-10`). Under compose that meant **no reachable health endpoint at all**: the homepage status card for alloy read `down` continuously and a deploy had nothing to wait on. The unit therefore passes `--server.http.listen-addr=0.0.0.0:12345` (`dankclaw/observability/quadlet/alloy.container:46`) and publishes `127.0.0.1:12345` (`alloy.container:47`, confirmed by `ss -ltnp` 2026-10-18) — same posture as the other three: reachable from the host and from Caddy's network, and from nothing off-host. Alloy has no authentication, which is exactly why it is not published wider. The homepage card probes that same loopback port (`shared/homepage/services.json:233`). Alloy 1.x refuses to configure the `ui` service from a config block, so the UI
address is set on the command line instead. Under the old rootless unit that
## Delivery state (2026-10-18): healthy, shipping nothing meant `0.0.0.0` inside the container plus a published `127.0.0.1:12345`; under
`Network=host` the listen-addr is `127.0.0.1:12345` directly — the host's
The unit is `active`, the container reports `healthy`, the log says `Alloy is ready`, and the pipeline carries no data at all. Measured on 2026-10-18: loopback, reachable from the host and from nothing off-host. Alloy has no
authentication, which is exactly why it is never published wider. The homepage
- `curl -s http://127.0.0.1:12345/metrics` → `loki_source_journal_target_lines_total{component_id="loki.source.journal.journal"} 0` and `loki_source_file_files_active_total{component_id="loki.source.file.containers"} 0`. The journal source logs no error; the counter simply never moves. The journal files it would have to read through the `/var/log` bind are `root:systemd-journal 0640` with an ACL (`ls -l /var/log/journal/64a287d7…/` 2026-10-18). card and the deploy health gate both dial `http://127.0.0.1:12345/-/healthy`.
- Both file globs were refused at start (`journalctl --user -u alloy`, 2026-10-18T08:25:27Z), verbatim: `failed to create source, skipping … component_id=loki.source.file.containers error="failed to tail file, stat failed: stat /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log: permission denied"` and the same message for `/var/log/containers/*.log: no such file or directory`.
- Downstream that means Loki holds zero streams — `GET /loki/api/v1/labels` returns `{"status":"success"}` with no `data` — see [[Loki]].
No fix is claimed here: the unit, the config and the push endpoint are all correct as written, and the gap is source-side access (and a Kubernetes-style glob that does not exist on this host). Anyone closing it should re-check the two counters above, not the health status — health is exactly what hides this failure mode.
## Gotchas ## Gotchas
- The journald socket mount is required for journal scraping. - **No `/etc/machine-id` mount ⇒ silent zero.** The health check still passes.
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host. This is the failure mode that hid the broken pipeline; check
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs. `loki_source_journal_target_lines_total` and the Loki label query, never
- The unit `Wants=`/`After=` `loki.service`: without a reachable Loki the write client fails and the pipeline backs up. `/healthy`.
- **`After=loki.service` is gone.** Alloy is on the system manager; Loki is a
rootless user unit — a system unit cannot order against a user unit. Alloy
retries its push until Loki answers, so ordering is unnecessary.
- **The image must be in the ROOT store.** The deployer builds rootless and
mirrors (`mirror_image_to_root`); a `Pull=never` root quadlet with only a
rootless tag fails at create.
- One Alloy per host; do not run a single instance that tries to reach both
hosts' logs.
## See Also ## See Also
@@ -81,7 +81,7 @@ Two provisioning rules, each verified live by an outage:
appears. Create the Application with `provider_id` set (the UI does this appears. Create the Application with `provider_id` set (the UI does this
automatically; raw API/ORM drives must do it explicitly). automatically; raw API/ORM drives must do it explicitly).
## OAuth2 relying-party quirks (this fork, verified live 2026-08-31) ## OAuth2 relying-party quirks (this fork; corrected live 2026-10-20)
The Grafana RP provider (`grafana`, pk 9, confidential client) was created and The Grafana RP provider (`grafana`, pk 9, confidential client) was created and
verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider
@@ -100,23 +100,32 @@ implementation diverges from upstream in ways that bit us; all verified live:
- **Provider resolution is by `client_id` query param** on the global - **Provider resolution is by `client_id` query param** on the global
`/application/o/authorize/` route; per-slug authorize paths do not exist in `/application/o/authorize/` route; per-slug authorize paths do not exist in
this build. Token/userinfo live at `/application/o/token/` and this build. Token/userinfo live at `/application/o/token/` and
`/application/o/userinfo/?api=1`. `/application/o/userinfo/` — do **not** append `?api=1` to a Grafana API URL.
- **Property-mapping evaluator is broken → userinfo returns `sub` only.** - **NULL `authorization_flow` → authorize 500, `AttributeError: 'NoneType'
`PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to object has no attribute 'slug'`.** A provider created via raw `ak shell`
`BaseEvaluator.evaluate(expression_source)`, so the first context kwarg `create()` leaves `authorization_flow` unset; the authorize view then crashes
becomes the *expression source* instead of the context; every call raises a building the flow planner (`flows/planner.py`) and returns "Server Error —
swallowed `PropertyMappingExpressionException` and the claim is skipped. Powered by authentik" before any claim is built. This was previously
Scope-grant filtering has the same class of bug: `ScopeMapping` has no mis-attributed to a broken evaluator; the crash is the missing flow. Fix: set
provider FK (linkage lives in the `ProviderPropertyMapping` through model) `authorization_flow = default-provider-authorization-implicit-consent` (the
so `filter(provider=...)` always yields an empty queryset and tokens asked flow every other provider in this instance already uses).
to carry scopes get zero. Net effect: userinfo/introspection answers with - **A scope mapping that returns a non-dict emits no claim.** authentik merges a
`{sub}` only — email/groups/entitlements claims cannot be delivered by this granted scope mapping's evaluated result into the token **as a mapping**: a
build until the evaluator is patched or upgraded. Clients consuming this dict contributes its keys as claims; a bare string is keyed under the
provider must key identity off `sub`. mapping's NAME, so nothing usable appears. `grafana-email-scope` returned the
- Scope mappings for this provider exist — `grafana-email-scope` and bare string `user.email`, so no `email` claim landed — the evaluator was
`grafana-profile-scope` are both rows in `authentik_core_propertymapping` never broken (the dict-valued `grafana-profile-scope` mapping has always
(read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment delivered, and the stock OpenID mappings return dicts). Fix:
the evaluator bug is fixed; no config change is needed then. `return {"email": user.email, "email_verified": True}`. With no email, Grafana
v13 falls back to GET `<api_url>/emails`; authentik 404s it → "Login failed:
InternalError".
- **Grafana reads identity from the standard userinfo endpoint** (`API_URL` with
no `?api=1`); no `USE_ID_TOKEN` is needed once the `email` claim exists.
- **Net (2026-10-20):** the evaluator is fine; email/profile claims reach RP
clients. Both IdP fixes are idempotent in
`scripts/authentik-grafana-sso-fix.sh` (re-run after an ak-shell re-creation of
the provider or a blueprint import, which revert them). Grafana's half plus the
verified outcome (Viewer, `login = <sub hex>`, email present) are in [[Grafana]].
## LDAP provider objects (provisioned via `ak shell` ORM) ## LDAP provider objects (provisioned via `ak shell` ORM)
@@ -267,6 +267,12 @@ The silent re-auth path was proven end-to-end: after deleting
`qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage `qa-e2e-admin`'s rows, the e2e admin spec ran against the stale storage
state and passed with no login prompt (2 tests, 2.9 s). state and passed with no login prompt (2 tests, 2.9 s).
## Login-session lifetime (how often you re-authenticate)
How long a login stays valid is the **`default-authentication-login` stage's `session_duration`** (`UserLoginStage`) — not a global setting, and not `SESSION_COOKIE_AGE` (that is the *pre-login* session). Raised to **`days=7`** on 2026-10-20 (was stage value `0` = the 24 h framework default). It is an **absolute** window, never sliding: authentik implements `SESSION_SAVE_EVERY_REQUEST` nowhere, so activity does not extend it and no supported knob makes it one.
**`session_duration` is a STRING** in authentik's idiom (`"days=7"`, `"seconds=30"`, `"hours=12"`). A Python `timedelta` stores `"7 days, 0:00:00"`, which `authentik.lib.utils.time.timedelta_from_string` rejects with `ValueError` and login then crashes computing the expiry — so set it with **`scripts/authentik-set-login-session-duration.sh`** (validates the form first). Any non-zero value applies to *every* login (the "remember me" box only adds `remember_me_offset` on top) and a positive `set_expiry` also overrides `SESSION_EXPIRE_AT_BROWSER_CLOSE`, so closing the browser no longer logs you out; **existing sessions keep their old expiry** — delete the user's `authentik_core_session`/`…proxysession` rows and re-login to force it now. This is **DB state, not repo-managed**: a blueprint re-import or rebuild reverts the stage to `0`, so re-run the script after one. The other two login stages (`default-source-*`) drive external-source/enrollment logins and stay at the framework default.
## Guard-chain flip recipe (adding a guarded vhost) ## Guard-chain flip recipe (adding a guarded vhost)
Move a public vhost from `reverse_proxy <app>:<port>` onto the chain: Move a public vhost from `reverse_proxy <app>:<port>` onto the chain:
+23 -14
View File
@@ -4,7 +4,7 @@ description: "Grafana visualizes the metrics that Prometheus collects from every
tags: tags:
- infrastructure - infrastructure
- grafana - grafana
timestamp: 2026-09-11 timestamp: 2026-10-20
--- ---
# Grafana # Grafana
@@ -64,19 +64,28 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-
- Change the default admin credentials at first boot (env or UI). - Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up. - Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries - **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
no provider token (`token=None`), so no identity header ever reaches Grafana; no provider token (`token=None`), so no identity header reaches Grafana; the
RP mode is topology-independent. The full verified contract (provider pk 9, login is Grafana's own "Sign in with Authentik" button against the `grafana`
client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths, confidential-client provider. **Verified end-to-end 2026-10-20** after four
Basic client auth, form-encoded token body, and the fork's userinfo-claims independent breakages had made it fail. The two Authentik-side causes (a NULL
bug that leaves identity at the `sub` hash only) is in `authorization_flow` → authorize 500; an email scope mapping that returned a
[[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana: bare string so no `email` claim was emitted) and the full wire contract live in
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true` [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks"; the IdP half is
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so re-applied by `scripts/authentik-grafana-sso-fix.sh` (idempotent — re-run after
users land as Viewer with `login = <sub hex>`; no email/groups/role claims an ak-shell re-creation of the provider or a blueprint import, both of which
reach Grafana until the evaluator bug is fixed — do not write role mappings revert it silently). The Grafana half is the unit's own env (`grafana.container`):
that depend on those claims. The local admin account stays the only Admin. `SCOPES="openid email profile"` (a GitLab-style `user:email` scope crashes the
(The Viewer-default outcome and the `<sub hex>` login are not readable from authorize view), the standard userinfo `API_URL` with **no `?api=1`**, the
the host without the database or an admin password. *(unverified 2026-10-18)*) attribute paths `email`/`sub`/`preferred_username`, and `ALLOW_SIGN_UP=true`
(safe: the guard fronts the callback). Grafana v13 only GETs `<api_url>/emails`
when the token carries no `email` claim — so it is the email claim itself (an
IdP fix, above) that makes the login land, not any Grafana toggle.
- **What a new login becomes:** a Viewer in org 1 (`ALLOW_ASSIGN_ORG=true`,
`ORG_ID=1`, no role-attribute mapping), with `login = <sub hex>` but the real
`email`. The local admin account stays the only Admin. The landing that the
edge-only 13-sso spec never checked is now asserted by
`tests/e2e/specs/20-grafana-sso.spec.ts` (RP round-trip → `/api/user` 200 with
an email).
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
file is a gluetun env file and it carries `HOME=/root`; systemd imports it file is a gluetun env file and it carries `HOME=/root`; systemd imports it
for the service, podman then looks for its configuration under `/root` for the service, podman then looks for its configuration under `/root`
@@ -3,7 +3,7 @@ type: reference
title: "Observability on Dankclaw" title: "Observability on Dankclaw"
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured." description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-09-11 timestamp: 2026-10-20
--- ---
# Observability on Dankclaw # Observability on Dankclaw
@@ -83,27 +83,37 @@ health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
[[Observability Provisioning]] for what is provisioned as code and how to [[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs. verify it through the APIs.
## Current state (2026-10-18) ## Current state (2026-10-20)
* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports * **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
eight up targets; the four downs are structural (the `comfyui` and `services` eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
4 alert rules + 2 recording rules (`/api/v1/rules`, matching and both node exporters. The four downs are structural (the `comfyui` and
`shared/prometheus/rules/llm-fleet.yml`). `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
(2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
time-to-first-token), each keyed only to the namespace its runtime exports.
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the * **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
relay, and `journalctl --user -u signal-relay` records successful relay, and `journalctl --user -u signal-relay` records successful
`sent to +12062476336:` lines for every rule in the set. `sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** One provisioned Prometheus datasource (uid * **Dashboards: working.** Provisioned datasources — Prometheus (uid
`prometheus`) and the `llm-fleet` dashboard, both installed from `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
`shared/grafana/provisioning/`. installed from `shared/grafana/provisioning/`. The Loki datasource is how
* **Logs: configured, empty.** Alloy ships zero lines (journal counter 0, both you read logs: Loki has no web UI of its own, so Grafana Explore is the
file globs refused) and Loki therefore holds zero streams — `/opt/loki/data` human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
is a few hundred KB of WAL/compactor bookkeeping with **no chunk files** * **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
(8 files total by `find`, 2026-10-18) and `GET /loki/api/v1/labels` returns a rootless Alloy cannot read the `0640` journal files, and the container-log
no label names. globs point at a root-storage layout that does not exist here. The fix is built
See [[Alloy]] → Delivery state and [[Loki]] → Contents. and tested in-repo (2026-10-20): Alloy is now a **root system service on
* **Not configured:** no Loki datasource in Grafana, no blackbox exporter (the `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
structural downs have no real uptime signal), no Alertmanager vhost. Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
lines and pushed 26,966 with zero drops. The **live** instance is still the old
rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
[[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real
uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
not yet deployed (see *Logs* above).
## Citations ## Citations
+18 -15
View File
@@ -1,22 +1,34 @@
// Alloy configuration — ships host and container logs to Loki. // Alloy configuration — ships host and container logs to Loki.
// One instance per host. Deployed to /opt/alloy/config/config.alloy. // One instance per host. Deployed to /opt/alloy/config/config.alloy.
//
// Runs as a ROOT system service on Network=host (see
// dankclaw/observability/quadlet/alloy.container), so it reaches Loki on the
// host's loopback and reads every journal — host services and rootless-quadlet
// containers alike.
logging { logging {
level = "info" level = "info"
} }
// The alloy UI service listens on 127.0.0.1:12345 by default; the // The alloy UI listens on 127.0.0.1:12345; the in-container healthcheck probes
// in-container healthcheck probes it there. alloy 1.x refuses to // it there. alloy 1.x refuses to configure the ui service from a config block,
// configure the ui service from a config block, so there is no block. // so there is no ui block.
// Ship logs to Loki (container name on the podman network). // Loki is published on the host's loopback (127.0.0.1:3100). Alloy shares the
// host network, so it pushes by loopback, never by container name — a root
// container is not on the rootless dankclaw_default network.
loki.write "default" { loki.write "default" {
endpoint { endpoint {
url = "http://loki:3100/loki/api/v1/push" url = "http://127.0.0.1:3100/loki/api/v1/push"
} }
} }
// Systemd journal from the host socket. // The systemd journal. As root with /etc/machine-id + the journal dirs mounted,
// this one source reads system.journal AND user-*.journal, so it carries both
// host services and the rootless Quadlet containers' logs. There is deliberately
// no container-file source: podman container stdout lands in the journal, and
// the old /var/lib/containers/.../ctr.log and /var/log/containers globs pointed
// at a root-storage layout that does not exist on this host.
loki.source.journal "journal" { loki.source.journal "journal" {
format_as_json = true format_as_json = true
labels = { labels = {
@@ -24,12 +36,3 @@ loki.source.journal "journal" {
} }
forward_to = [loki.write.default.receiver] forward_to = [loki.write.default.receiver]
} }
// Container logs (podman storage path; adjust for docker).
loki.source.file "containers" {
targets = [
{__path__ = "/var/lib/containers/storage/overlay-containers/*/userdata/ctr.log", job = "podman"},
{__path__ = "/var/log/containers/*.log", job = "containers"},
]
forward_to = [loki.write.default.receiver]
}
@@ -0,0 +1,17 @@
# Grafana datasource for Loki, declared in git rather than clicked into
# grafana.db — same reason as the Prometheus datasource: a UI-configured
# datasource is invisible to review, lost with the data volume, and unknown
# to the repository. Loki OSS ships no web UI of its own, so Explore through
# this datasource is the only way a human reads the log store.
#
# URL is the container name on dankclaw_default (the same address Caddy
# proxies to), not the published loopback port.
apiVersion: 1
datasources:
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
editable: false
+2 -2
View File
@@ -30,9 +30,9 @@
}, },
{ {
"name": "open-notebook", "name": "open-notebook",
"category": "user", "category": "admin",
"section": "AI", "section": "AI",
"description": "Podcast generator — notes in, spoken episode out: script written by frank-zealot, voices rendered by frank-tts. Guarded twice: Authentik at the edge, then its own login.", "description": "Podcast generator — notes in, spoken episode out: script written by frank-archon, voices rendered by frank-tts. Guarded twice: Authentik at the edge, then its own login. Single shared password, no per-user accounts — admin-only.",
"url": "http://notebook.franksdank.cloud", "url": "http://notebook.franksdank.cloud",
"docs": "", "docs": "",
"probe": "127.0.0.1:8502", "probe": "127.0.0.1:8502",
+93 -9
View File
@@ -1,10 +1,15 @@
# Alerting and recording rules for the LLM fleet. # Alerting and recording rules for the LLM fleet.
# #
# One metric namespace: llamacpp:* — frank-zealot and frank-dragoon, both # Two metric namespaces, because two runtime families are scraped:
# llama.cpp with /metrics (the family verified from /metrics on 2026-09-16 #
# and 2026-09-20). frank-archon LEFT the family 2026-10-05: FreeToken (the # * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen, which
# PR587 ROCm engine) has no /metrics endpoint, so its scrape job was # re-exports the llama.cpp metric names). Verified from /metrics on
# deleted rather than left never-green. # 2026-09-16 / 2026-09-20.
# * vllm:* — frank-archon, the radiance engine on Dankserver since
# 2026-10-05. It exports vLLM-compatible names, a DIFFERENT namespace, so
# it is a separate scrape job in prometheus.yml and the rules below are a
# separate group. (The old claim that its scrape job was deleted is false:
# it re-joined the day it left the FreeToken engine and is `up` today.)
# (The netavark published-port quirk that forces the Dankclaw endpoints # (The netavark published-port quirk that forces the Dankclaw endpoints
# onto the gateway is same-host only.) # onto the gateway is same-host only.)
# #
@@ -22,7 +27,8 @@
# cached by cached + prompt. # cached by cached + prompt.
# * Speculative decoding: drafts and accepted tokens are both counters, # * Speculative decoding: drafts and accepted tokens are both counters,
# so LlmSpecDecodeOff watches the pair. # so LlmSpecDecodeOff watches the pair.
# * No latency histograms exist, so there is no p95 alert. # * No latency histograms in llama.cpp, so no p95 alert there; radiance's
# vllm:* namespace DOES export one, so the vllm-fleet group alerts on it.
# #
# IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
# Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/ # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
@@ -112,6 +118,84 @@ groups:
description: >- description: >-
Prometheus cannot reach the LLM endpoint. frank-zealot and Prometheus cannot reach the LLM endpoint. frank-zealot and
frank-dragoon are root units on Dankclaw: `systemctl is-active frank-dragoon are root units on Dankclaw: `systemctl is-active
frank-zealot` / `frank-dragoon`. Both scraped endpoints export frank-zealot` / `frank-dragoon` (both export llamacpp:* —
the llamacpp:* family. (frank-archon left the scrape set dragoon via halogen's llama.cpp-compatible names). frank-archon
2026-10-05 — FreeToken exports no /metrics.) is a radiance endpoint on Dankserver (`systemctl show
frank-archon` there) exporting the vllm:* family; the
`frank-.*` selector covers all three regardless of namespace.
# frank-archon (radiance) exports the vLLM-compatible namespace, NOT
# llamacpp:*. Every rule here is keyed to a series the exporter actually
# emits (verified from /metrics 2026-10-20: num_requests_waiting,
# gpu_prefix_cache_hit_rate, num_preemptions_total, prompt_tokens_total,
# time_to_first_token_seconds). The llamacpp rules above evaluate nothing for
# this job and these evaluate nothing for llama.cpp/halogen — two clean
# namespaces, each keyed only to what its runtime exports.
- name: vllm-fleet
interval: 30s
rules:
# p95 time-to-first-token, shared with the dashboard. When the endpoint
# is idle every bucket rate is zero, histogram_quantile yields NaN, and
# neither this nor the alert below can fire on an idle engine.
- record: job:vllm_ttft_p95:rate5m
expr: |
histogram_quantile(0.95, sum by (job, le)
(rate(vllm:time_to_first_token_seconds_bucket[5m])))
# Radiance exports a real queue-depth gauge (llama.cpp only shows the
# wall as requests_deferred). --max-num-seqs is small, so a sustained
# wait is either real load or this endpoint carrying the fleet alone.
- alert: VllmRequestsQueueing
expr: vllm:num_requests_waiting > 0
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: requests waiting for 10 minutes"
description: >-
The radiance scheduler has requests queued. Either the fleet is
genuinely busy or this endpoint is carrying it alone.
# Radiance reports prefix reuse directly as a hit-rate gauge — no
# counter division needed. The token-rate floor keeps an idle engine
# (hit rate 0 with no traffic) from firing.
- alert: VllmPrefixCacheCollapse
expr: >-
vllm:gpu_prefix_cache_hit_rate < 0.20
and rate(vllm:prompt_tokens_total[5m]) > 0.05
for: 15m
labels:
severity: info
annotations:
summary: "{{ $labels.job }}: prefix hit rate below 20 % for 15 minutes"
description: >-
Nearly every request prefilling from scratch. With several large
contexts in flight this is the capacity wall, not a routing fault.
# Preemption is the KV-thrash signature radiance reports that llama.cpp
# cannot: the engine evicts a running request to admit another. Any
# sustained rate means the KV cache cannot hold the concurrency in flight.
- alert: VllmPreempting
expr: rate(vllm:num_preemptions_total[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: engine preempting requests"
description: >-
The engine is evicting running requests to admit new ones — the KV
cache is undersized for the concurrency in flight.
# p95 first-token latency, kept HIGH on purpose: a large-prompt prefill
# legitimately spends many seconds before token 1, so this fires only on
# a wedged scheduler or a queue backed up behind one request.
- alert: VllmTtftSlow
expr: job:vllm_ttft_p95:rate5m > 120
for: 10m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: p95 time-to-first-token over 120 s"
description: >-
First-token latency is pathological, not an honest big prefill.
Likely a wedged scheduler or one request stalling the queue.
+6 -8
View File
@@ -240,14 +240,12 @@ CADDYFILE="$REPO_ROOT/dankclaw/Caddyfile"
} }
@test "alloy's UI address comes from the unit, not the config file" { @test "alloy's UI address comes from the unit, not the config file" {
# alloy 1.x refuses to configure the ui service from a config block # alloy 1.x refuses to configure the ui service from a config block, so the
# ("service ui does not support being configured"), so the unit moves the # unit sets the UI address on the command line. As a Network=host root unit
# UI off the container's loopback with a command-line flag. Without that # that address is the host loopback directly (no PublishPort) — reachable
# flag the port cannot be published and the service has no reachable # from the host, from nothing off-host.
# health endpoint at all — which is why the homepage status card for
# alloy read "down" while it ran under compose.
! grep -qE '^(ui|server) \{' "$REPO_ROOT/shared/alloy/config.alloy" ! grep -qE '^(ui|server) \{' "$REPO_ROOT/shared/alloy/config.alloy"
local unit="$REPO_ROOT/dankclaw/observability/quadlet/alloy.container" local unit="$REPO_ROOT/dankclaw/observability/quadlet/alloy.container"
grep -q -- '--server.http.listen-addr=0.0.0.0:12345' "$unit" grep -q -- '--server.http.listen-addr=127.0.0.1:12345' "$unit"
grep -q '^PublishPort=127.0.0.1:12345:12345$' "$unit" ! grep -q '^PublishPort=' "$unit"
} }
+8 -2
View File
@@ -29,8 +29,14 @@ export default defineConfig({
// green suite with a test that does not exist. 17- joined on 2026-09-16 // green suite with a test that does not exist. 17- joined on 2026-09-16
// (the hermes vhost, whose upstream is a host address rather than a // (the hermes vhost, whose upstream is a host address rather than a
// container name, so it needs its own pass condition). 18- joined on // container name, so it needs its own pass condition). 18- joined on
// 2026-09-24 (OpenWebUI SSO through authentik). // 2026-09-24 (OpenWebUI SSO through authentik). 20- joined on 2026-10-21
testMatch: /(10|12|13|16|17|18)-.*\.spec\.ts/, // (Grafana's OAuth2 relying-party login, which the edge-only 13- never
// follows). 21- joined 2026-10-21 (the open-notebook vhost, deployed
// 2026-10-19 - the newest guarded app, in neither GUARDED list nor any
// spec until now: the 17-hermes "uncovered vhost broke silently" class).
// 22- joins Prometheus's app-is-really-there proof (its own API), closing
// the "not a login wall but maybe a 404" hole 20- closed for Grafana.
testMatch: /(10|12|13|16|17|18|20|21|22)-.*\.spec\.ts/,
use: { storageState: ".auth/qa-e2e.json" }, use: { storageState: ".auth/qa-e2e.json" },
dependencies: ["setup"], dependencies: ["setup"],
}, },
+2 -1
View File
@@ -15,7 +15,7 @@ const BASE = "https://www.franksdank.cloud";
// Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile). // Every vhost that Caddy routes through anubis:8923 (see dankclaw/Caddyfile).
// Keep this list in lockstep with the Caddyfile: a new guarded vhost must be // Keep this list in lockstep with the Caddyfile: a new guarded vhost must be
// added here or the guard-chain audit is incomplete. // added here or the guard-chain audit is incomplete.
const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki"]; const GUARDED = ["www", "search", "romm", "files", "seerr", "grafana", "prometheus", "loki", "notebook"];
// Markers that prove app content leaked past the guard to an anonymous user. // Markers that prove app content leaked past the guard to an anonymous user.
const LEAK_MARKERS: Record<string, string[]> = { const LEAK_MARKERS: Record<string, string[]> = {
@@ -25,6 +25,7 @@ const LEAK_MARKERS: Record<string, string[]> = {
"seerr.franksdank.cloud": ["Request Movie"], "seerr.franksdank.cloud": ["Request Movie"],
"grafana.franksdank.cloud": ["Dashboards"], "grafana.franksdank.cloud": ["Dashboards"],
"prometheus.franksdank.cloud": ["Expression Browser", "Graph"], "prometheus.franksdank.cloud": ["Expression Browser", "Graph"],
"notebook.franksdank.cloud": ["Open Notebook"],
}; };
test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => { test("homepage root (/) is guarded for unauthenticated users", async ({ page }) => {
+56
View File
@@ -0,0 +1,56 @@
import { expect, test } from "@playwright/test";
// Grafana (grafana.franksdank.cloud) is an OAuth2 RELYING PARTY, not an
// outpost-fronted proxy: the guard chain (13-sso) only proves an authenticated
// session reaches Grafana's login page. The login itself happens through
// Grafana's own "Sign in with Authentik" button, which the edge test never
// follows — that is exactly how the RP login could be dead (500 on authorize,
// or an email-claim gap bouncing back to /login) while 13- still passed green.
// This spec drives the real RP round-trip and asserts a live Grafana session.
const HOST = "grafana.franksdank.cloud";
const CHALLENGE = "Making sure you're not a bot";
test("SSO: Grafana relying-party login lands a real session", async ({ page, request }) => {
test.setTimeout(180_000);
// Warm the guard: the vhost runs its own Anubis proof-of-work on first visit.
await page.goto(`https://${HOST}/`);
await expect
.poll(
async () => !(await page.locator("body").innerText()).includes(CHALLENGE),
{ timeout: 90_000 },
)
.toBe(true);
// The "Sign in with Authentik" button targets this RP start URL exactly.
// Going straight there is the same code path without depending on the SPA's
// button markup.
await page.goto(`https://${HOST}/login/generic_oauth`);
// The chain: grafana -> auth (implicit consent, the session is already
// there) -> back to grafana with the code -> grafana provisions the user and
// redirects to "/". A dead provider/claims config bounces back to /login.
await expect
.poll(
async () => {
const u = new URL(page.url());
return u.hostname === HOST && u.pathname !== "/login";
},
{ timeout: 90_000 },
)
.toBe(true);
const body = await page.locator("body").innerText();
expect(body).not.toContain("Login failed");
expect(body.toLowerCase()).not.toContain("powered by authentik"); // IdP flow shell
expect(body).not.toContain(CHALLENGE);
// A genuine logged-in session exists: /api/user answers 200 with an email,
// not 401. This is the assertion the edge-only spec could never make.
const resp = await page.request.get(`https://${HOST}/api/user`);
expect(resp.status()).toBe(200);
const user = await resp.json();
expect(typeof user.email).toBe("string");
expect(user.email.length).toBeGreaterThan(0);
});
+52
View File
@@ -0,0 +1,52 @@
import { expect, test } from "@playwright/test";
// open-notebook (notebook.franksdank.cloud), deployed 2026-10-19, is a guarded
// vhost whose outpost (notebook-proxy) fronts an app with its OWN password
// login - the same shape as hermes (17-) and OpenWebUI (18-): the guard chain
// (Anubis PoW -> authentik outpost) must hand off to the app, and the app's own
// login page is the expected landing. Until now this host appeared in neither
// the 13-/14- GUARDED lists nor any dedicated spec, so a broken upstream would
// have stayed invisible - the exact 17-hermes "uncovered vhost" failure class.
// 14- now covers the unauthenticated gate; this spec owns authenticated
// reachability with a POSITIVE app marker, not just "not a login wall".
const VHOST = "https://notebook.franksdank.cloud/";
const CHALLENGE = "Making sure you're not a bot";
test("notebook vhost reaches the open-notebook app through the guard chain", async ({ page }) => {
test.setTimeout(180_000);
await page.goto(VHOST);
// The Anubis proof-of-work clears in a real browser and the outpost hands off
// to the app; the user must never be parked on an authentik flow.
await expect
.poll(async () => !(await page.locator("body").innerText()).includes(CHALLENGE), { timeout: 90_000 })
.toBe(true);
await expect
.poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
.toBe("notebook.franksdank.cloud");
expect(
new URL(page.url()).pathname.startsWith("/if/"),
"must not be parked on an authentik flow",
).toBe(false);
expect(
await page.locator('input[name="uidField"]').count(),
"must not show the authentik login form",
).toBe(0);
// Positive identity: the page really is open-notebook's own login, not a
// Caddy error or the IdP shell. The app answering its own login is the pass.
await expect(page, "destination title must be Open Notebook").toHaveTitle(/open notebook/i, {
timeout: 30_000,
});
await expect(page.locator("body"), "open-notebook login marker").toContainText(
/enter your password/i,
{ timeout: 30_000 },
);
const body = await page.locator("body").innerText();
expect(body, "must not leak the IdP flow shell").not.toContain("Powered by authentik");
expect(body, "must not be a Caddy upstream error").not.toContain("Bad Gateway");
const resp = await page.request.get(VHOST);
expect(resp.status(), "open-notebook must answer, not error").toBeLessThan(500);
});
+33
View File
@@ -0,0 +1,33 @@
import { expect, test } from "@playwright/test";
// Prometheus (prometheus.franksdank.cloud) sits behind the same embedded outpost
// as Grafana. 13-sso and 11-admin only ever assert it is "not a login wall and
// under 500" - which a Prometheus serving a 404 or a blank page would ALSO pass.
// This is the app-is-really-there proof 20- provides for Grafana: with the
// standard session the outpost hands off silently to the Prometheus UI, and the
// app's own API answers success. A guard-route or upstream regression that lets
// any non-Prometheus answer through fails here.
test("SSO: Prometheus app answers its own API through authentik", async ({ page, request }) => {
test.setTimeout(180_000);
await page.goto("https://prometheus.franksdank.cloud/");
// Challenge clears, outpost hands off silently to the app (no credential prompt).
await expect
.poll(
async () => !(await page.locator("body").innerText()).includes("Making sure you're not a bot"),
{ timeout: 90_000 },
)
.toBe(true);
await expect
.poll(() => new URL(page.url()).hostname, { timeout: 60_000 })
.toBe("prometheus.franksdank.cloud");
expect(await page.locator('input[name="uidField"]').count(), "no authentik form").toBe(0);
await expect(page, "must land on the Prometheus UI").toHaveTitle(/prometheus/i, { timeout: 30_000 });
// Load-bearing assertion: the Prometheus API itself answers, authenticated,
// through the outpost. An IdP bounce, a 404, or a raw upstream error fails here.
const api = await request.get("https://prometheus.franksdank.cloud/api/v1/status/buildinfo");
expect(api.status(), "Prometheus API must answer 200 through the guard").toBe(200);
expect((await api.json()).status, "Prometheus API must report success").toBe("success");
});
+84 -39
View File
@@ -41,13 +41,17 @@ service_section() {
} }
@test "every observability unit is boot-persistent" { @test "every observability unit is boot-persistent" {
# Generated units cannot be enabled on this host's podman 5.x v1 quadlet # Generated units cannot be enabled on this host's podman quadlet generator;
# generator; [Install] + linger is the entire boot-persistence story. # [Install] + linger is the boot-persistence story for the rootless five.
# Alloy is a ROOT system unit, so its [Install] targets multi-user.target.
local svc local svc
for svc in "${UNITS[@]}"; do for svc in "${UNITS[@]}"; do
[[ "$svc" == alloy ]] && continue
grep -q '^WantedBy=default.target$' "$UNIT_DIR/$svc.container" \ grep -q '^WantedBy=default.target$' "$UNIT_DIR/$svc.container" \
|| { echo "$svc is not boot-persistent"; return 1; } || { echo "$svc is not boot-persistent"; return 1; }
done done
grep -q '^WantedBy=multi-user.target$' "$UNIT_DIR/alloy.container" \
|| { echo "alloy (root unit) is not boot-persistent"; return 1; }
} }
@test "every observability unit is self-healing" { @test "every observability unit is self-healing" {
@@ -140,15 +144,20 @@ service_section() {
grep -q 'reverse_proxy prometheus:9090' "$REPO_ROOT/dankclaw/Caddyfile" grep -q 'reverse_proxy prometheus:9090' "$REPO_ROOT/dankclaw/Caddyfile"
grep -q 'reverse_proxy loki:3100' "$REPO_ROOT/dankclaw/Caddyfile" grep -q 'reverse_proxy loki:3100' "$REPO_ROOT/dankclaw/Caddyfile"
local svc local svc
for svc in prometheus grafana loki alloy; do for svc in prometheus grafana loki; do
grep -q "^ContainerName=$svc$" "$UNIT_DIR/$svc.container" \ grep -q "^ContainerName=$svc$" "$UNIT_DIR/$svc.container" \
|| { echo "$svc unit renames its container"; return 1; } || { echo "$svc unit renames its container"; return 1; }
grep -q '^Network=dankclaw_default$' "$UNIT_DIR/$svc.container" \ grep -q '^Network=dankclaw_default$' "$UNIT_DIR/$svc.container" \
|| { echo "$svc is not on the network Caddy and its peers use"; return 1; } || { echo "$svc is not on the network Caddy and its peers use"; return 1; }
done done
# Alloy is not on that network: it is a root container on Network=host, the
# only way to reach Loki's 127.0.0.1:3100 publish from outside a bridge.
grep -q '^ContainerName=alloy$' "$UNIT_DIR/alloy.container"
grep -q '^Network=host$' "$UNIT_DIR/alloy.container" \
|| { echo "alloy must be Network=host to reach Loki on the host loopback"; return 1; }
grep -q '"grafana:3000"' "$REPO_ROOT/shared/prometheus/prometheus.yml" grep -q '"grafana:3000"' "$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q '"loki:3100"' "$REPO_ROOT/shared/prometheus/prometheus.yml" grep -q '"loki:3100"' "$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q 'url = "http://loki:3100/loki/api/v1/push"' "$REPO_ROOT/shared/alloy/config.alloy" grep -q 'url = "http://127.0.0.1:3100/loki/api/v1/push"' "$REPO_ROOT/shared/alloy/config.alloy"
} }
@test "every observability unit has a health check" { @test "every observability unit has a health check" {
@@ -239,12 +248,14 @@ service_section() {
} }
@test "startup order follows the data dependencies" { @test "startup order follows the data dependencies" {
# Grafana is useless without Prometheus and Alloy drops logs without Loki; # Grafana is useless without Prometheus (both rootless, same manager). Alloy
# both pairs are expressed as systemd ordering, not launch luck. # used to order on loki.service; as a ROOT unit it lives on the SYSTEM
# manager and cannot order against a rootless user unit, so it depends only
# on the network and retries its push until Loki answers.
grep -q 'After=network-online.target prometheus.service' "$UNIT_DIR/grafana.container" grep -q 'After=network-online.target prometheus.service' "$UNIT_DIR/grafana.container"
grep -q 'Wants=network-online.target prometheus.service' "$UNIT_DIR/grafana.container" grep -q 'Wants=network-online.target prometheus.service' "$UNIT_DIR/grafana.container"
grep -q 'After=network-online.target loki.service' "$UNIT_DIR/alloy.container" grep -q '^After=network-online.target$' "$UNIT_DIR/alloy.container"
grep -q 'Wants=network-online.target loki.service' "$UNIT_DIR/alloy.container" ! grep -q 'loki.service' "$UNIT_DIR/alloy.container"
} }
@test "the deployer installs, restarts and health-checks each unit" { @test "the deployer installs, restarts and health-checks each unit" {
@@ -281,6 +292,24 @@ PAIRS
grep -q 'build_compose_image alloy' "$REMOTE_DEPLOY" grep -q 'build_compose_image alloy' "$REMOTE_DEPLOY"
} }
@test "alloy is a root host-network unit that reads every journal" {
# The whole point of the root cutover: a rootless container cannot open the
# 0640 journal files. Pin the properties that make the pipeline carry data.
local u="$UNIT_DIR/alloy.container"
grep -q '^Network=host$' "$u"
grep -q '^Volume=/etc/machine-id:/etc/machine-id:ro$' "$u" \
|| { echo "alloy missing the load-bearing machine-id mount"; return 1; }
grep -q '^Volume=/var/log/journal:/var/log/journal:ro$' "$u"
grep -q '^Volume=/run/log/journal:/run/log/journal:ro$' "$u"
grep -q -- '--server.http.listen-addr=127.0.0.1:12345' "$u"
# Deployer: system install + root image mirror, not the user quadlet loop.
grep -q '^deploy_alloy() {' "$REMOTE_DEPLOY"
grep -q 'install_system_quadlet dankclaw/observability/quadlet alloy' "$REMOTE_DEPLOY"
grep -q '/etc/containers/systemd/' "$REMOTE_DEPLOY"
grep -q 'mirror_image_to_root localhost/franksdank/alloy:latest' "$REMOTE_DEPLOY"
grep -q 'alloy) deploy_alloy ;;' "$REMOTE_DEPLOY"
}
@test "no dankclaw compose file competes with the quadlet units" { @test "no dankclaw compose file competes with the quadlet units" {
# Two owners for one container name is how a compose run resurrects a # Two owners for one container name is how a compose run resurrects a
# stale duplicate next to the quadlet-managed one. This file was retired on # stale duplicate next to the quadlet-managed one. This file was retired on
@@ -313,33 +342,33 @@ PAIRS
done done
} }
@test "prometheus scrapes every LLM endpoint that exports metrics" { @test "prometheus scrapes every LLM endpoint, in its real metric namespace" {
# The LLM endpoints are the services whose behaviour this fleet argues about # Every LLM endpoint that exports /metrics must have a scrape job. As of
# most (KV pressure, prefix-cache reuse, queue depth). Both Dankclaw # 2026-10-05 there are TWO runtime namespaces scraped:
# endpoints export the llamacpp:* family. frank-archon LEFT the set # * llamacpp:* — frank-zealot (llama.cpp) and frank-dragoon (halogen,
# 2026-10-05: the FreeToken PR587 ROCm engine has NO /metrics endpoint — # re-exporting the llama.cpp names), both root host processes on
# scraping it was a structural never-green (404), so the job was deleted # Dankclaw reached via host.containers.internal — this Prometheus is a
# rather than left vacuous (the Lucebox DS4.1 precedent, 2026-09-26). # bridge container and a host-process bind is refused at the LAN IP,
# The two Dankclaw endpoints are HOST processes reached through # which is why frank-zealot read red by construction for days.
# host.containers.internal: this Prometheus is a bridge-networked # * vllm:* — frank-archon (radiance) on Dankserver, a different machine,
# container, and on this netavark host a container can only reach the # so plain LAN routing reaches 192.168.0.235:7770. It RE-JOINED the
# host's LAN IP at a PUBLISHED port — a host-process bind like # scrape set with radiance; the old "job deleted, exports no /metrics"
# llama-server's 0.0.0.0:7770 is refused at 192.168.0.143:7770 while it # note belonged to the retired FreeToken engine and is false today, so
# answers at 169.254.1.2:7770 (why frank-zealot stayed red by construction # this seam now asserts archon IS scraped (a bare `! grep` here was
# for days until the 2026-09-20 /targets audit). # set -e-exempt and never enforced the opposite).
local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml" local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-zealot not scraped"; return 1; } grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-zealot not scraped"; return 1; }
grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-dragoon not scraped"; return 1; } grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-dragoon not scraped"; return 1; }
! grep -q '"192.168.0.235:7770"' "$cfg" grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-archon not scraped"; return 1; }
grep -q '"host.containers.internal:7770,192.168.0.235:7770"' "$cfg" \ for job in frank-zealot frank-dragoon frank-archon; do
&& { echo "frank-archon folded into a Dankclaw gateway target"; return 1; }
for job in frank-zealot frank-dragoon; do
grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; } grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
done done
! grep -q 'job_name: "frank-archon"' "$cfg" # No LLM endpoint is reached through a guarded vhost: Prometheus cannot
# None of them is reached through a guarded vhost: Prometheus cannot solve # solve an Anubis challenge, so a franksdank.cloud target is red by
# an Anubis challenge, so a franksdank.cloud target is red by construction. # construction.
! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg" grep -qE '^[[:space:]]*-[[:space:]]*"?https?://[^"]*franksdank' "$cfg" \
&& { echo "a scrape target points at a guarded vhost"; return 1; }
true
} }
@test "grafana provisioning is declared in git and installed by the deployer" { @test "grafana provisioning is declared in git and installed by the deployer" {
@@ -378,28 +407,44 @@ names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rule
for want in ('job:llm_prefix_cache_reuse:rate5m', for want in ('job:llm_prefix_cache_reuse:rate5m',
'job:llm_spec_accept_length:rate5m', 'job:llm_spec_accept_length:rate5m',
'LlmRequestsQueueing', 'LlmPrefixCacheCollapse', 'LlmRequestsQueueing', 'LlmPrefixCacheCollapse',
'LlmSpecDecodeOff', 'LlmEndpointDown'): 'LlmSpecDecodeOff', 'LlmEndpointDown',
'job:vllm_ttft_p95:rate5m', 'VllmRequestsQueueing',
'VllmPrefixCacheCollapse', 'VllmPreempting', 'VllmTtftSlow'):
assert want in names, want assert want in names, want
# Every rule must be keyed to a series that actually exists. A rule naming a # Every rule must be keyed to a series that actually exists. A rule naming a
# series nobody exports evaluates nothing forever and errors never — the # series nobody exports evaluates nothing forever and errors never — the
# failure this seam exists for. Since the 2026-09-26 cutover the only scraped # failure this seam exists for. Two runtimes are scraped in two namespaces:
# LLM runtime is llama.cpp (frank-zealot, frank-dragoon); the vllm: family left # llama.cpp/halogen (llamacpp:*) for frank-zealot and frank-dragoon, and
# with frank-medium's radiance runtime and must not survive in any expr. # radiance (vllm:*) for frank-archon. Both families were read straight off the
# endpoints' /metrics; anything else is a guess.
exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']] exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
# llama.cpp families measured from /metrics on 2026-09-16 and 2026-09-20 live = {
# (frank-zealot, frank-dragoon). Anything else is a guess. # llama.cpp / halogen, measured from /metrics 2026-09-16 and 2026-09-20.
live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total', 'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
'llamacpp:requests_deferred', 'llamacpp:requests_processing', 'llamacpp:requests_deferred', 'llamacpp:requests_processing',
'llamacpp:spec_decode_num_drafts_total', 'llamacpp:spec_decode_num_drafts_total',
'llamacpp:spec_decode_num_accepted_tokens_total', 'llamacpp:spec_decode_num_accepted_tokens_total',
'llamacpp:spec_decode_num_draft_tokens_total', 'llamacpp:spec_decode_num_draft_tokens_total',
'llamacpp:tokens_predicted_total', 'llamacpp:tokens_predicted_total',
'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds', 'llamacpp:predicted_tokens_seconds', 'llamacpp:prompt_tokens_seconds',
'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode'} 'llamacpp:n_decode_total', 'llamacpp:n_busy_slots_per_decode',
# radiance (frank-archon), measured from /metrics 2026-10-20.
'vllm:num_requests_waiting', 'vllm:gpu_prefix_cache_hit_rate',
'vllm:num_preemptions_total', 'vllm:prompt_tokens_total',
'vllm:time_to_first_token_seconds_bucket',
}
used = {m for e in exprs used = {m for e in exprs
for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)} for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
assert used <= live, used - live assert used <= live, used - live
assert not any(':vllm' in e for e in exprs), 'vllm: series in rules but no vllm endpoint is scraped' # The two namespaces must not bleed across groups: a llamacpp: selector
# evaluates nothing for the radiance job and vice versa, so each group may
# reference only its own family.
per = {g['name']: {m.split(':')[0] for r in g['rules']
for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', r.get('expr', ''))}
for g in d['groups']}
for name, fams in per.items():
allowed = {'llamacpp'} if name == 'llm-fleet' else {'vllm'}
assert fams <= allowed, (name, fams)
print('rules ok:', len(names)) print('rules ok:', len(names))
" "
# A rule file that nothing loads evaluates nothing, and Prometheus reports # A rule file that nothing loads evaluates nothing, and Prometheus reports
+3 -1
View File
@@ -290,7 +290,9 @@ vhost = sys.argv[2]
cards = [s for s in data if s["name"] == "open-notebook"] cards = [s for s in data if s["name"] == "open-notebook"]
assert len(cards) == 1, f"expected exactly one open-notebook card, got {len(cards)}" assert len(cards) == 1, f"expected exactly one open-notebook card, got {len(cards)}"
card = cards[0] card = cards[0]
assert card["category"] == "user" and card["section"] == "AI", card # category moved to "admin": single shared password, no per-user accounts, so it
# is positioned as an admin-side tool, not a user-facing app (see the vault page).
assert card["category"] == "admin" and card["section"] == "AI", card
assert card["probe"] == "127.0.0.1:8502", card["probe"] assert card["probe"] == "127.0.0.1:8502", card["probe"]
assert card["method"] == "http" assert card["method"] == "http"
# Deployed and guarded: the card is live and its link is the vhost, so the # Deployed and guarded: the card is live and its link is the vhost, so the