Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

5.9 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Grafana

Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at grafana.franksdank.cloud (via Caddy, behind the Anubis challenge + embedded-outpost gate; login happens inside Grafana via the Authentik OAuth2 button).

Host: Dankclaw — Hostname: grafana.franksdank.cloud — Port: 3002 (internal) — Docs: https://grafana.com/docs

Facts

Item	Value
Image	`docker.io/grafana/grafana:latest`
Published	`127.0.0.1:3002` → 3000, loopback only — never a wildcard (`dankclaw/observability/quadlet/grafana.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:3002` and no other bind)
Unit	`dankclaw/observability/quadlet/grafana.container` → `grafana.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`grafana.container:75-79`), installed at `~/.config/containers/systemd/grafana.container`, reporting `generated` + `active` (`systemctl --user is-enabled
Data	`/opt/grafana/data` — `grafana.db` (present on the host: `ls /opt/grafana/data/grafana.db`, 2026-10-18): accounts, sessions, anything created in the UI
Provisioning mount	`/opt/grafana/provisioning` (read-only, `grafana.container:57`) — populated as code: `datasources/prometheus.yml`, `dashboards/provider.yml`, `dashboards/llm-fleet.json` are on the host (`ls -la /opt/grafana/provisioning/` 2026-10-18) and are installed on every deploy by `copy_grafana_provisioning` (`scripts/deploy-remote.sh:286-300`). History: the mount was* empty as of 2026-09-11, before provisioning was written to git on 2026-09-16 (commit `6a19143`); `tests/observability_quadlet.bats:345` pins the tree
Datasource	one, provisioned: Prometheus, uid `prometheus`, `http://prometheus:9090`, default, not editable, `timeInterval: 15s` (`shared/grafana/provisioning/datasources/prometheus.yml:14-26`). No Loki datasource exists — see Loki

Configuration

Everything comes from the Quadlet unit. The Authentik OAuth2 client secret is the only credential, and systemd expands it from a per-service env file:

[Service]
EnvironmentFile=/home/dank/Containers/grafana/env      # systemd-side only
[Container]
Environment=GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=${GRAFANA_OAUTH_CLIENT_SECRET}

seed_grafana_env in scripts/deploy-remote.sh:477-486 seeds that file from dankclaw/.env with a single-variable filter (awk -F= '/^GRAFANA_OAUTH_CLIENT_SECRET=/ …', mode 600) and fails the deploy when the value is missing. The container receives the one expanded value, never the file.

Datasource

The Prometheus datasource is declared in git, not clicked into grafana.db: shared/grafana/provisioning/datasources/prometheus.yml points at http://prometheus:9090 — the container name on dankclaw_default, the same address Caddy proxies to, never the loopback-published port — and carries timeInterval: 15s so Grafana's rate windows cannot be shorter than the scrape interval (prometheus.yml:5). The llm-fleet dashboard is provisioned alongside it by dashboards/provider.yml (provider fleet, updateIntervalSeconds: 30, allowUiUpdates: true).

A Loki datasource is still not provisioned — the tree contains exactly one datasource file — so nothing in Grafana queries logs today, which matters less than it sounds because Loki holds no streams (see Loki → Contents). Verification is through the API, not the filesystem: curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources.

Healthcheck

Grafana responds on /api/health with 200 and "database":"ok"; measured 2026-10-18 on http://127.0.0.1:3002/api/health, which also reports version 13.2.2 (podman ps reports the container healthy on the unit's wget --spider -q http://localhost:3000/api/health, grafana.container:58).

Gotchas

Change the default admin credentials at first boot (env or UI).
Dashboards provisioned as code (dashboard JSON plus a YAML provider, shared/grafana/provisioning/dashboards/) survive container recreation; UI-created ones live in the data volume — back it up.
Auth = OAuth2 relying-party, not auth_proxy. The embedded outpost carries no provider token (token=None), so no identity header ever reaches Grafana; RP mode is topology-independent. The full verified contract (provider pk 9, client_id grafana-21ca190cde75d60f, global authorize/token/userinfo paths, Basic client auth, form-encoded token body, and the fork's userinfo-claims bug that leaves identity at the sub hash only) is in Authentik Outpost Reference → "OAuth2 relying-party quirks". Consequences for Grafana: The unit sets ALLOW_SIGN_UP=false, AUTO_UPSERT=true, ALLOW_ASSIGN_ORG=true, ORG_ID=1 and no role-attribute mapping (grafana.container:52-55), so users land as Viewer with login = <sub hex>; no email/groups/role claims reach Grafana until the evaluator bug is fixed — do not write role mappings that depend on those claims. The local admin account stays the only Admin. (The Viewer-default outcome and the <sub hex> login are not readable from the host without the database or an admin password. (unverified 2026-10-18))
Never point this unit's EnvironmentFile at the repo-root .env. That file is a gluetun env file and it carries HOME=/root; systemd imports it for the service, podman then looks for its configuration under /root, cannot read it, and the unit dies at start with stat /root/.config: permission denied. The unit reads /home/dank/Containers/grafana/env, which the deployer seeds with exactly one variable.

5.9 KiB Raw Blame History