dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): fleet-wide fact re-verification pass (67 pages)

Browse Source 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank committed 2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
67 files changed
+2619 -1152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Infrastructure/Grafana.md
+29 -18
View File
@@ -18,11 +18,11 @@ Grafana visualizes the metrics that Prometheus collects from every service. It r
| Item | Value |
|------|-------|
| Image | `docker.io/grafana/grafana:latest` |
| Published | `127.0.0.1:3002` → 3000, loopback only (see [[Observability]] → Exposure) |
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service` |
| Data | `/opt/grafana/data` — `grafana.db`: accounts, sessions, anything created in the UI |
| Provisioning mount | `/opt/grafana/provisioning` (read-only) — **empty as of 2026-09-11** |
| Datasource | **none configured** — see the state note in [[Observability]] |
| Published | `127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind) |
| Unit | `dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active grafana` 2026-10-18) |
| Data | `/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI |
| Provisioning mount | `/opt/grafana/provisioning` (read-only, `grafana.container:57`) — **populated as code**: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/*` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount **was** empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree |
| Datasource | one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see [[Loki]] |
## Configuration
@@ -36,25 +36,33 @@ EnvironmentFile=/home/dank/Containers/grafana/env # systemd-side only
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}
```
`seed_grafana_env` in `scripts/deploy-remote.sh` seeds that file from
`dankclaw/.env` with a single-variable filter and fails the deploy when the
value is missing. The container receives the one expanded value, never the file.
`seed_grafana_env` in `scripts/deploy-remote.sh:477-486` seeds that file from `dankclaw/.env` with a single-variable filter (`awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …'`, mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.
## Datasource
None is configured. Grafana cannot query anything yet; pointing it at
`http://prometheus:9090` and `http://loki:3100` (by container name, on the
`dankclaw_default` network) is outstanding work — see the state note in
[[Observability]].
The Prometheus datasource is declared in git, not clicked into `grafana.db`:
`shared/grafana/provisioning/datasources/prometheus.yml` points at
`http://prometheus:9090` — the container name on `dankclaw_default`, the same
address Caddy proxies to, never the loopback-published port — and carries
`timeInterval: 15s` so Grafana's rate windows cannot be shorter than the
scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned
alongside it by `dashboards/provider.yml` (provider `fleet`,
`updateIntervalSeconds: 30`, `allowUiUpdates: true`).
A Loki datasource is still **not** provisioned — the tree contains exactly one
datasource file — so nothing in Grafana queries logs today, which matters less
than it sounds because Loki holds no streams (see [[Loki]] → Contents).
Verification is through the API, not the filesystem:
`curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources`.
## Healthcheck
Grafana responds on `/api/health` with 200 and `"database":"ok"`.
Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026-10-18 on `http://127.0.0.1:3002/api/health`, which also reports version `13.2.2` (`podman ps` reports the container `healthy` on the unit's `wget --spider -q http://localhost:3000/api/health`, `grafana.container:58`).
## Gotchas
- Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
- Dashboards provisioned as code (dashboard JSON plus a YAML provider, `shared/grafana/provisioning/dashboards/`) survive container recreation; UI-created ones live in the data volume — back it up.
- **Auth = OAuth2 relying-party, not auth_proxy.** The embedded outpost carries
no provider token (`token=None`), so no identity header ever reaches Grafana;
RP mode is topology-independent. The full verified contract (provider pk 9,
@@ -62,10 +70,13 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`.
Basic client auth, form-encoded token body, and the fork's userinfo-claims
bug that leaves identity at the `sub` hash only) is in
[[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana:
users auto-upsert as Viewer with `login = <sub hex>`; no email/groups/role
claims reach Grafana until the evaluator bug is fixed — do not write role
mappings that depend on those claims. The local admin account stays the only
Admin.
The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`,
`ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so
users land as Viewer with `login = <sub hex>`; no email/groups/role claims
reach Grafana until the evaluator bug is fixed — do not write role mappings
that depend on those claims. The local admin account stays the only Admin.
(The Viewer-default outcome and the `<sub hex>` login are not readable from
the host without the database or an admin password. *(unverified 2026-10-18)*)
- **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That
file is a gluetun env file and it carries `HOME=/root`; systemd imports it
for the service, podman then looks for its configuration under `/root`,