48 lines
5.6 KiB
Markdown
48 lines
5.6 KiB
Markdown
---
|
||
name: grafana-provisioned-dashboard-verify
|
||
description: "Author and verify repo-provisioned dashboards on Dankclaw live: Grafana 13 has no unauthenticated API and no local admin, so prove through the e2e SSO browser path, and close the copier gap that hides never-provisioned boards."
|
||
---
|
||
|
||
## When to use
|
||
You changed `shared/grafana/provisioning/dashboards/*.json` (or panel PromQL) and must prove the LIVE boards actually receive data — or a panel/whole board reads empty and you must classify: dead expression vs never-provisioned vs legitimately-empty.
|
||
|
||
## The two gaps this skill exists for
|
||
1. **The copier gap.** A repo dashboard JSON is NOT a live dashboard until the deployer carries it. Until 2026-10-21 `copy_grafana_provisioning` in `scripts/deploy-remote.sh` named `llm-fleet.json` alone: the `containers` board was merged, seam-tested, and documented as "working" yet had NEVER been provisioned; the other boards existed on disk (`/opt/grafana/provisioning/dashboards/`) only via one-time hand-copies, and one had silently gone stale. Fixed: copier globs `dashboards/*.json`; `GRAFANA_PROV_ROOT` is the test seam; a bats case executes the copier for real into a sandbox.
|
||
2. **The API-proof illusion.** The vault used to say "verify through the APIs". On Grafana 13.2.2 that is dead: `/api/health`, `/api/frontend`, `/api/org`, `/login` all 404 **unauthenticated**, and there is NO local admin password (root `.env` sets no `GF_SECURITY_ADMIN_*`; the only account is the Authentik RP login). Do not hunt for credentials; use the browser path below.
|
||
|
||
## Procedure
|
||
### 1. Mechanical expression census (over SSH — Prometheus binds loopback)
|
||
Extract every panel `expr` from the repo JSONs with python, substitute template vars (`$host`,`$container`,`$job` → `.*`, `$__auto`/`$interval` → `5m`), and run each as an instant query against `http://127.0.0.1:9090/api/v1/query` via `ssh dankclaw`. `curl` to `127.0.0.1:9090` from the dev box is blocked; keep queries on-host. Watch zsh globbing on `?`/`[` — quote the remote command.
|
||
Empty result ≠ defect automatically:
|
||
- `ALERTS` panel empty = **healthy** (ALERTS carries only firing/pending series).
|
||
- Zero by design: no cgroup limits exist on the fleet → a memory-vs-limit panel is dead by premise (rewrite as % of host total, don't delete).
|
||
- Real join defects: cadvisor `host` label vs node-exporter `job` label sets are incompatible — fix with `group_left label_replace(node_memory_MemTotal_bytes{job=~"node-.*"}, "host", "$1", "job", "node-(.*)")`, and `count by (job) (expr) / on(job) count by (job) (...)` needs the `on(job)` matcher when label sets differ. Parens around `on(host)` after `group_left` are **illegal**; `group_left label_replace(...)` without them parses fine.
|
||
|
||
### 2. Prove the provisioned files (filesystem = installation proof only)
|
||
```
|
||
ssh dankclaw 'for f in /opt/grafana/provisioning/dashboards/*.json; do \
|
||
cmp -s "$f" "/home/dank/serverAppConfigs/shared/grafana/provisioning/dashboards/$(basename "$f")" \
|
||
&& echo "$(basename $f) same" || echo "$(basename $f) DIFFERS"; done'
|
||
```
|
||
Missing file or DIFFERS ⇒ the copier (or an unmerged branch) is the bug, not the JSON.
|
||
|
||
### 3. Browser proof (the real surface)
|
||
Throwaway `.mjs` placed **inside `tests/e2e/`** (`@playwright/test` resolves from `node_modules` there; a script in `/tmp` fails with `ERR_MODULE_NOT_FOUND`). Node on this box is Zig-rewritten v26: `readFileSync(p,"text")` throws `ERR_INVALID_ARG_VALUE` — use `readFileSync(p).toString()`.
|
||
Sequence, exactly as `specs/20-grafana-sso.spec.ts` drives it:
|
||
1. `newContext({storageState: ".auth/qa-e2e-admin.json", userAgent: REAL_UA})` — state is bound to the homepage origin; on the grafana vhost it is only the Authentik session cookie, so you MUST:
|
||
2. `goto https://grafana.franksdank.cloud/` and poll until body lacks `"Making sure you're not a bot"` (Anubis PoW warm-up).
|
||
3. `goto .../login/generic_oauth` and poll until `url` host is the vhost and path ≠ `/login` (RP round-trip). Landing on `/login` = auth flow broken, not the dashboard.
|
||
4. `goto /d/<uid>?orgId=1&from=now-1h&to=now`, wait ~10 s, then read panels: titles via `.panel-title,[class*='panel-title']`; a panel is `NO-DATA` iff its container text contains `"No data"`, `RENDERED` iff a descendant matches `svg:not(:empty),canvas,img`. Panels lazy-render; scroll (`mouse.wheel`) before judging.
|
||
Classify: `PANEL-NOT-FOUND` for a whole board ⇒ board not provisioned (step 2). `NO-DATA` with a proven-live expression ⇒ variable/label mismatch — re-run the expression with template vars expanded.
|
||
Throwaway scripts → `~/.AGENTTRASH/<date>-topic/` after use; never `rm`.
|
||
|
||
## Sandboxed deployment-function tests (bats)
|
||
To behaviorally test a deploy-remote.sh copier: extract with `sed -n '/^func()/,/^}/p'`, `eval` it, stub `run_privileged() { "$@"; }`, point the function's `*_ROOT` seam env var at a sandbox, execute, assert files. Traps:
|
||
- `VAR=x func` prefix assignments do NOT bind for shell **functions** — `export VAR=...` first.
|
||
- `mktemp -d` + `trap 'rm -rf' RETURN` races bats' trap machinery: the sandbox vanishes mid-function and bats blames the `trap` line. Use `$BATS_TEST_TMPDIR` (auto-cleaned).
|
||
- Prove red on the OLD body by patching a `mktemp -d` copy of `tests/ shared/ scripts/` and running bats there — no checkout mutation.
|
||
- Full suite must run from the repo root (`./tests/run.sh`), not from a subdirectory.
|
||
|
||
## Land flow (repo law)
|
||
Branch off fresh `origin/main`; merge needs Frank's explicit current OK; Gitea merge endpoint wants **JSON** `{"Do":"merge"}` (form-encoded `Do=merge` 422s with `[Do]: Required`); then `deploy.sh dankclaw --service grafana` and repeat steps 2–3 as the post-deploy gate.
|