command-center/agent/managed-skills/grafana-provisioned-dashboard-verify/SKILL.md

name, description

name	description
grafana-provisioned-dashboard-verify	Author and verify repo-provisioned dashboards on Dankclaw live: Grafana 13 has no unauthenticated API and no local admin, so prove through the e2e SSO browser path, and close the copier gap that hides never-provisioned boards.

When to use

You changed shared/grafana/provisioning/dashboards/*.json (or panel PromQL) and must prove the LIVE boards actually receive data — or a panel/whole board reads empty and you must classify: dead expression vs never-provisioned vs legitimately-empty.

The two gaps this skill exists for

1. The copier gap. A repo dashboard JSON is NOT a live dashboard until the deployer carries it. Until 2026-10-21 copy_grafana_provisioning in scripts/deploy-remote.sh named llm-fleet.json alone: the containers board was merged, seam-tested, and documented as "working" yet had NEVER been provisioned; the other boards existed on disk (/opt/grafana/provisioning/dashboards/) only via one-time hand-copies, and one had silently gone stale. Fixed: copier globs dashboards/*.json; GRAFANA_PROV_ROOT is the test seam; a bats case executes the copier for real into a sandbox.
2. The API-proof illusion. The vault used to say "verify through the APIs". On Grafana 13.2.2 that is dead: /api/health, /api/frontend, /api/org, /login all 404 unauthenticated, and there is NO local admin password (root .env sets no GF_SECURITY_ADMIN_*; the only account is the Authentik RP login). Do not hunt for credentials; use the browser path below.

Procedure

1. Mechanical expression census (over SSH — Prometheus binds loopback)

Extract every panel expr from the repo JSONs with python, substitute template vars ($host,$container,$job → .*, $__auto/$interval → 5m), and run each as an instant query against http://127.0.0.1:9090/api/v1/query via ssh dankclaw. curl to 127.0.0.1:9090 from the dev box is blocked; keep queries on-host. Watch zsh globbing on ?/[ — quote the remote command. Empty result ≠ defect automatically:

• ALERTS panel empty = healthy (ALERTS carries only firing/pending series).
• Zero by design: no cgroup limits exist on the fleet → a memory-vs-limit panel is dead by premise (rewrite as % of host total, don't delete).
• Real join defects: cadvisor host label vs node-exporter job label sets are incompatible — fix with group_left label_replace(node_memory_MemTotal_bytes{job=~"node-.*"}, "host", "$1", "job", "node-(.*)"), and count by (job) (expr) / on(job) count by (job) (...) needs the on(job) matcher when label sets differ. Parens around on(host) after group_left are illegal; group_left label_replace(...) without them parses fine.

2. Prove the provisioned files (filesystem = installation proof only)

ssh dankclaw 'for f in /opt/grafana/provisioning/dashboards/*.json; do \
  cmp -s "$f" "/home/dank/serverAppConfigs/shared/grafana/provisioning/dashboards/$(basename "$f")" \
  && echo "$(basename $f) same" || echo "$(basename $f) DIFFERS"; done'

Missing file or DIFFERS ⇒ the copier (or an unmerged branch) is the bug, not the JSON.

3. Browser proof (the real surface)

Throwaway .mjs placed inside tests/e2e/ (@playwright/test resolves from node_modules there; a script in /tmp fails with ERR_MODULE_NOT_FOUND). Node on this box is Zig-rewritten v26: readFileSync(p,"text") throws ERR_INVALID_ARG_VALUE — use readFileSync(p).toString(). Sequence, exactly as specs/20-grafana-sso.spec.ts drives it:

1. newContext({storageState: ".auth/qa-e2e-admin.json", userAgent: REAL_UA}) — state is bound to the homepage origin; on the grafana vhost it is only the Authentik session cookie, so you MUST:
2. goto https://grafana.franksdank.cloud/ and poll until body lacks "Making sure you're not a bot" (Anubis PoW warm-up).
3. goto .../login/generic_oauth and poll until url host is the vhost and path ≠ /login (RP round-trip). Landing on /login = auth flow broken, not the dashboard.
4. goto /d/<uid>?orgId=1&from=now-1h&to=now, wait ~10 s, then read panels: titles via .panel-title,[class*='panel-title']; a panel is NO-DATA iff its container text contains "No data", RENDERED iff a descendant matches svg:not(:empty),canvas,img. Panels lazy-render; scroll (mouse.wheel) before judging. Classify: PANEL-NOT-FOUND for a whole board ⇒ board not provisioned (step 2). NO-DATA with a proven-live expression ⇒ variable/label mismatch — re-run the expression with template vars expanded. Throwaway scripts → ~/.AGENTTRASH/<date>-topic/ after use; never rm.

Sandboxed deployment-function tests (bats)

To behaviorally test a deploy-remote.sh copier: extract with sed -n '/^func()/,/^}/p', eval it, stub run_privileged() { "$@"; }, point the function's *_ROOT seam env var at a sandbox, execute, assert files. Traps:

• VAR=x func prefix assignments do NOT bind for shell functions — export VAR=... first.
• mktemp -d + trap 'rm -rf' RETURN races bats' trap machinery: the sandbox vanishes mid-function and bats blames the trap line. Use $BATS_TEST_TMPDIR (auto-cleaned).
• Prove red on the OLD body by patching a mktemp -d copy of tests/ shared/ scripts/ and running bats there — no checkout mutation.
• Full suite must run from the repo root (./tests/run.sh), not from a subdirectory.

Land flow (repo law)

Branch off fresh origin/main; merge needs Frank's explicit current OK; Gitea merge endpoint wants JSON {"Do":"merge"} (form-encoded Do=merge 422s with [Do]: Required); then deploy.sh dankclaw --service grafana and repeat steps 2–3 as the post-deploy gate.