command-center/agent/managed-skills/grafana-provisioned-dashboard-verify/SKILL.md at 88d486b37bc4b46de23a99f0fd9816772221f786

0 Fork 0

Files

T

Frank 8af40c402b chore(agent): commit the machine tier and the doctrine text the checkout was already running

Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and
`merge --ff-only` refuses while tracked paths are dirty. Discard is not available here
(workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed.

- `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already
  byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new
  prose — the commit exists so the trunk merge can proceed.
- `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat
  owns (AGENTS.md § Live-sync tiers).

`ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the
only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the
inert all-zero AWS example id — and they appear inside the two skill pages that teach
that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged
gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal
dies with this landing. No `--no-verify` was used: the gate ran and its verdict is
recorded here.

Refs #61

2026-10-21 01:44:51 -07:00

4.7 KiB

Raw Blame History

name, description

name	description
grafana-provisioned-dashboard-verify	Author and verify repo-provisioned Grafana dashboards on Dankclaw live: staging to the host provider path, the Grafana-13 unified-storage false-negative on grafana.db, the authenticated /api/search check, per-panel PromQL/LogQL validation before shipping, and the Loki $__auto shell-expansion trap in test harnesses.

Verify or ship Grafana dashboards that are provisioned from the repo (shared/grafana/provisioning/dashboards/*.json, auto-loaded by the fleet file provider). Use when adding/staging a dashboard, or when asked to confirm provisioned dashboards actually load.

Paths (Dankclaw)

Repo source: shared/grafana/provisioning/dashboards/ (git is the owner).
Host provider dir: /opt/grafana/provisioning/dashboards (root-owned), bind-mounted :ro into the container at /etc/grafana/provisioning/dashboards (from dankclaw/observability/quadlet/grafana.container: Volume=/opt/grafana/provisioning:/etc/grafana/provisioning:ro).
Provider reload interval = 30s, so a staged file appears within ~35s.

Stage on the host (canonical install is `deploy.sh dankclaw --service grafana` after merge)

scp -q <files>.json dankclaw:/tmp/
ssh dankclaw 'for f in <names>; do sudo install -m644 -o root -g root /tmp/$f.json /opt/grafana/provisioning/dashboards/$f.json; done'

Verification — what DOES and DOESN'T work

DO NOT trust the sqlite dashboard table. Grafana 13 uses unified storage: file-provisioned dashboards do NOT live in the legacy dashboard table. Reading grafana.db (sudo python3 - <<PY ... select ... from dashboard) returns total dashboards: 0 even when everything is provisioned correctly — a false negative. There is also no sqlite3 binary in the grafana image or on the host. Grafana logs no line on a successful provider load (only errors), so a clean podman logs grafana proves only "no error", not "loaded".

The authoritative check is /api/search through an authenticated browser session. No admin password is configured and anonymous is off — do NOT reset the Grafana admin to do a read check. Drive the RP SSO login with the Playwright qa-e2e storage state, then GET the API:

const ctx = await b.newContext({ storageState: '.auth/qa-e2e.json', userAgent: "<chrome UA>" });
const p = await ctx.newPage();
await p.goto('https://grafana.franksdank.cloud/', { waitUntil: 'domcontentloaded' });
for (let i=0;i<30;i++){ const t=await p.locator('body').innerText(); if(!t.includes('not a bot')) break; await p.waitForTimeout(3000);}   // clear Anubis PoW
await p.goto('https://grafana.franksdank.cloud/login/generic_oauth', { waitUntil: 'domcontentloaded' });
await p.waitForTimeout(6000);
const r = await ctx.request.get('https://grafana.franksdank.cloud/api/search?type=dash-db&limit=50');  // 200 + [{uid,title}...] = provisioned

Run it from the checkout that has tests/e2e/node_modules + .auth + .env.local (a fresh worktree lacks them; symlink node_modules and copy .auth + .env.local). Throwaway _*.cjs, then move to .AGENTTRASH/.

Validate every panel query BEFORE shipping (catch empty / mis-scaled panels)

Query Prometheus (http://127.0.0.1:9090/api/v1/query) and Loki (http://127.0.0.1:3100/loki/api/v1/query) on Dankclaw loopback and confirm non-empty. Real traps found here:

Unit mislabel: sum by(job,mode)(rate(node_cpu_seconds_total{mode!="idle"}[5m])) returns cores per mode (up to #cores) — unit short, NOT percentunit. For a per-host busy fraction use 1 - avg by(job)(rate(node_cpu_seconds_total{mode="idle"}[5m])).
Missing metrics: scrape_duration_seconds_bucket and prometheus_*-prefixed self metrics may be absent — confirm from /api/v1/label/__name__/values?match[]=<metric> before building a panel; prometheus_target_interval_length_seconds (a summary with a quantile label) exists.
Alertmanager runs on :9093 but is NOT a Prometheus scrape target → its metrics are unavailable; don't build panels on it (they read empty).
Loki streams: journal arrives as {job="systemd-journal"}; container_name is a parsed JSON field, not a stream label → group at query time: sum by (container_name)(count_over_time({job="systemd-journal"} | json [$__auto])).
Shell trap in test harnesses: $__auto inside a double-quoted ssh/curl string is eaten by the local shell → empty query → Loki returns empty (looks like a broken panel, is a harness bug). For live validation substitute a concrete [5m] window, and drive Loki queries with an on-host Python urllib client to avoid quoting hell entirely.

Datasource uids

prometheus (uid prometheus), loki (uid loki); reference as "datasource": {"type":"prometheus","uid":"prometheus"} in panel JSON.

4.7 KiB Raw Blame History