Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
45 lines
4.7 KiB
Markdown
45 lines
4.7 KiB
Markdown
---
|
|
name: grafana-provisioned-dashboard-verify
|
|
description: "Author and verify repo-provisioned Grafana dashboards on Dankclaw live: staging to the host provider path, the Grafana-13 unified-storage false-negative on grafana.db, the authenticated /api/search check, per-panel PromQL/LogQL validation before shipping, and the Loki $__auto shell-expansion trap in test harnesses."
|
|
---
|
|
|
|
Verify or ship Grafana dashboards that are provisioned from the repo (`shared/grafana/provisioning/dashboards/*.json`, auto-loaded by the `fleet` file provider). Use when adding/staging a dashboard, or when asked to confirm provisioned dashboards actually load.
|
|
|
|
## Paths (Dankclaw)
|
|
- Repo source: `shared/grafana/provisioning/dashboards/` (git is the owner).
|
|
- Host provider dir: `/opt/grafana/provisioning/dashboards` (root-owned), bind-mounted `:ro` into the container at `/etc/grafana/provisioning/dashboards` (from `dankclaw/observability/quadlet/grafana.container`: `Volume=/opt/grafana/provisioning:/etc/grafana/provisioning:ro`).
|
|
- Provider reload interval = 30s, so a staged file appears within ~35s.
|
|
|
|
## Stage on the host (canonical install is `deploy.sh dankclaw --service grafana` after merge)
|
|
```
|
|
scp -q <files>.json dankclaw:/tmp/
|
|
ssh dankclaw 'for f in <names>; do sudo install -m644 -o root -g root /tmp/$f.json /opt/grafana/provisioning/dashboards/$f.json; done'
|
|
```
|
|
|
|
## Verification — what DOES and DOESN'T work
|
|
|
|
**DO NOT trust the sqlite `dashboard` table.** Grafana 13 uses **unified storage**: file-provisioned dashboards do NOT live in the legacy `dashboard` table. Reading `grafana.db` (`sudo python3 - <<PY ... select ... from dashboard`) returns **total dashboards: 0** even when everything is provisioned correctly — a false negative. There is also no `sqlite3` binary in the grafana image or on the host. Grafana logs **no** line on a successful provider load (only errors), so a clean `podman logs grafana` proves only "no error", not "loaded".
|
|
|
|
**The authoritative check is `/api/search` through an authenticated browser session.** No admin password is configured and anonymous is off — do NOT reset the Grafana admin to do a read check. Drive the RP SSO login with the Playwright `qa-e2e` storage state, then GET the API:
|
|
```js
|
|
const ctx = await b.newContext({ storageState: '.auth/qa-e2e.json', userAgent: "<chrome UA>" });
|
|
const p = await ctx.newPage();
|
|
await p.goto('https://grafana.franksdank.cloud/', { waitUntil: 'domcontentloaded' });
|
|
for (let i=0;i<30;i++){ const t=await p.locator('body').innerText(); if(!t.includes('not a bot')) break; await p.waitForTimeout(3000);} // clear Anubis PoW
|
|
await p.goto('https://grafana.franksdank.cloud/login/generic_oauth', { waitUntil: 'domcontentloaded' });
|
|
await p.waitForTimeout(6000);
|
|
const r = await ctx.request.get('https://grafana.franksdank.cloud/api/search?type=dash-db&limit=50'); // 200 + [{uid,title}...] = provisioned
|
|
```
|
|
Run it from the checkout that has `tests/e2e/node_modules` + `.auth` + `.env.local` (a fresh worktree lacks them; symlink `node_modules` and copy `.auth` + `.env.local`). Throwaway `_*.cjs`, then move to `.AGENTTRASH/`.
|
|
|
|
## Validate every panel query BEFORE shipping (catch empty / mis-scaled panels)
|
|
Query Prometheus (`http://127.0.0.1:9090/api/v1/query`) and Loki (`http://127.0.0.1:3100/loki/api/v1/query`) on Dankclaw loopback and confirm non-empty. Real traps found here:
|
|
- **Unit mislabel**: `sum by(job,mode)(rate(node_cpu_seconds_total{mode!="idle"}[5m]))` returns **cores per mode** (up to #cores) — unit `short`, NOT `percentunit`. For a per-host busy fraction use `1 - avg by(job)(rate(node_cpu_seconds_total{mode="idle"}[5m]))`.
|
|
- **Missing metrics**: `scrape_duration_seconds_bucket` and `prometheus_*`-prefixed self metrics may be absent — confirm from `/api/v1/label/__name__/values?match[]=<metric>` before building a panel; `prometheus_target_interval_length_seconds` (a summary with a `quantile` label) exists.
|
|
- **Alertmanager** runs on :9093 but is **NOT a Prometheus scrape target** → its metrics are unavailable; don't build panels on it (they read empty).
|
|
- **Loki streams**: journal arrives as `{job="systemd-journal"}`; `container_name` is a **parsed JSON field, not a stream label** → group at query time: `sum by (container_name)(count_over_time({job="systemd-journal"} | json [$__auto]))`.
|
|
- **Shell trap in test harnesses**: `$__auto` inside a double-quoted ssh/curl string is eaten by the local shell → empty query → Loki returns empty (looks like a broken panel, is a harness bug). For live validation substitute a concrete `[5m]` window, and drive Loki queries with an on-host Python `urllib` client to avoid quoting hell entirely.
|
|
|
|
## Datasource uids
|
|
`prometheus` (uid `prometheus`), `loki` (uid `loki`); reference as `"datasource": {"type":"prometheus","uid":"prometheus"}` in panel JSON.
|