From 6a12ffaef401899e29d638e9dcaf0bc5110961fd Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 21 Oct 2026 10:47:56 -0700 Subject: [PATCH] live: heartbeat sync 2026-10-21T10:47:56-07:00 (Refs #11) --- .../SKILL.md | 69 ++++++++++--------- .../state.json | 2 +- .../state.json | 2 +- .../state.json | 2 +- .../state.json | 2 +- .../state.json | 2 +- 6 files changed, 41 insertions(+), 38 deletions(-) diff --git a/agent/managed-skills/grafana-provisioned-dashboard-verify/SKILL.md b/agent/managed-skills/grafana-provisioned-dashboard-verify/SKILL.md index 6821041..1c60e6f 100644 --- a/agent/managed-skills/grafana-provisioned-dashboard-verify/SKILL.md +++ b/agent/managed-skills/grafana-provisioned-dashboard-verify/SKILL.md @@ -1,44 +1,47 @@ --- name: grafana-provisioned-dashboard-verify -description: "Author and verify repo-provisioned Grafana dashboards on Dankclaw live: staging to the host provider path, the Grafana-13 unified-storage false-negative on grafana.db, the authenticated /api/search check, per-panel PromQL/LogQL validation before shipping, and the Loki $__auto shell-expansion trap in test harnesses." +description: "Author and verify repo-provisioned dashboards on Dankclaw live: Grafana 13 has no unauthenticated API and no local admin, so prove through the e2e SSO browser path, and close the copier gap that hides never-provisioned boards." --- -Verify or ship Grafana dashboards that are provisioned from the repo (`shared/grafana/provisioning/dashboards/*.json`, auto-loaded by the `fleet` file provider). Use when adding/staging a dashboard, or when asked to confirm provisioned dashboards actually load. +## When to use +You changed `shared/grafana/provisioning/dashboards/*.json` (or panel PromQL) and must prove the LIVE boards actually receive data — or a panel/whole board reads empty and you must classify: dead expression vs never-provisioned vs legitimately-empty. -## Paths (Dankclaw) -- Repo source: `shared/grafana/provisioning/dashboards/` (git is the owner). -- Host provider dir: `/opt/grafana/provisioning/dashboards` (root-owned), bind-mounted `:ro` into the container at `/etc/grafana/provisioning/dashboards` (from `dankclaw/observability/quadlet/grafana.container`: `Volume=/opt/grafana/provisioning:/etc/grafana/provisioning:ro`). -- Provider reload interval = 30s, so a staged file appears within ~35s. +## The two gaps this skill exists for +1. **The copier gap.** A repo dashboard JSON is NOT a live dashboard until the deployer carries it. Until 2026-10-21 `copy_grafana_provisioning` in `scripts/deploy-remote.sh` named `llm-fleet.json` alone: the `containers` board was merged, seam-tested, and documented as "working" yet had NEVER been provisioned; the other boards existed on disk (`/opt/grafana/provisioning/dashboards/`) only via one-time hand-copies, and one had silently gone stale. Fixed: copier globs `dashboards/*.json`; `GRAFANA_PROV_ROOT` is the test seam; a bats case executes the copier for real into a sandbox. +2. **The API-proof illusion.** The vault used to say "verify through the APIs". On Grafana 13.2.2 that is dead: `/api/health`, `/api/frontend`, `/api/org`, `/login` all 404 **unauthenticated**, and there is NO local admin password (root `.env` sets no `GF_SECURITY_ADMIN_*`; the only account is the Authentik RP login). Do not hunt for credentials; use the browser path below. -## Stage on the host (canonical install is `deploy.sh dankclaw --service grafana` after merge) +## Procedure +### 1. Mechanical expression census (over SSH — Prometheus binds loopback) +Extract every panel `expr` from the repo JSONs with python, substitute template vars (`$host`,`$container`,`$job` → `.*`, `$__auto`/`$interval` → `5m`), and run each as an instant query against `http://127.0.0.1:9090/api/v1/query` via `ssh dankclaw`. `curl` to `127.0.0.1:9090` from the dev box is blocked; keep queries on-host. Watch zsh globbing on `?`/`[` — quote the remote command. +Empty result ≠ defect automatically: +- `ALERTS` panel empty = **healthy** (ALERTS carries only firing/pending series). +- Zero by design: no cgroup limits exist on the fleet → a memory-vs-limit panel is dead by premise (rewrite as % of host total, don't delete). +- Real join defects: cadvisor `host` label vs node-exporter `job` label sets are incompatible — fix with `group_left label_replace(node_memory_MemTotal_bytes{job=~"node-.*"}, "host", "$1", "job", "node-(.*)")`, and `count by (job) (expr) / on(job) count by (job) (...)` needs the `on(job)` matcher when label sets differ. Parens around `on(host)` after `group_left` are **illegal**; `group_left label_replace(...)` without them parses fine. + +### 2. Prove the provisioned files (filesystem = installation proof only) ``` -scp -q .json dankclaw:/tmp/ -ssh dankclaw 'for f in ; do sudo install -m644 -o root -g root /tmp/$f.json /opt/grafana/provisioning/dashboards/$f.json; done' +ssh dankclaw 'for f in /opt/grafana/provisioning/dashboards/*.json; do \ + cmp -s "$f" "/home/dank/serverAppConfigs/shared/grafana/provisioning/dashboards/$(basename "$f")" \ + && echo "$(basename $f) same" || echo "$(basename $f) DIFFERS"; done' ``` +Missing file or DIFFERS ⇒ the copier (or an unmerged branch) is the bug, not the JSON. -## Verification — what DOES and DOESN'T work +### 3. Browser proof (the real surface) +Throwaway `.mjs` placed **inside `tests/e2e/`** (`@playwright/test` resolves from `node_modules` there; a script in `/tmp` fails with `ERR_MODULE_NOT_FOUND`). Node on this box is Zig-rewritten v26: `readFileSync(p,"text")` throws `ERR_INVALID_ARG_VALUE` — use `readFileSync(p).toString()`. +Sequence, exactly as `specs/20-grafana-sso.spec.ts` drives it: +1. `newContext({storageState: ".auth/qa-e2e-admin.json", userAgent: REAL_UA})` — state is bound to the homepage origin; on the grafana vhost it is only the Authentik session cookie, so you MUST: +2. `goto https://grafana.franksdank.cloud/` and poll until body lacks `"Making sure you're not a bot"` (Anubis PoW warm-up). +3. `goto .../login/generic_oauth` and poll until `url` host is the vhost and path ≠ `/login` (RP round-trip). Landing on `/login` = auth flow broken, not the dashboard. +4. `goto /d/?orgId=1&from=now-1h&to=now`, wait ~10 s, then read panels: titles via `.panel-title,[class*='panel-title']`; a panel is `NO-DATA` iff its container text contains `"No data"`, `RENDERED` iff a descendant matches `svg:not(:empty),canvas,img`. Panels lazy-render; scroll (`mouse.wheel`) before judging. +Classify: `PANEL-NOT-FOUND` for a whole board ⇒ board not provisioned (step 2). `NO-DATA` with a proven-live expression ⇒ variable/label mismatch — re-run the expression with template vars expanded. +Throwaway scripts → `~/.AGENTTRASH/-topic/` after use; never `rm`. -**DO NOT trust the sqlite `dashboard` table.** Grafana 13 uses **unified storage**: file-provisioned dashboards do NOT live in the legacy `dashboard` table. Reading `grafana.db` (`sudo python3 - <" }); -const p = await ctx.newPage(); -await p.goto('https://grafana.franksdank.cloud/', { waitUntil: 'domcontentloaded' }); -for (let i=0;i<30;i++){ const t=await p.locator('body').innerText(); if(!t.includes('not a bot')) break; await p.waitForTimeout(3000);} // clear Anubis PoW -await p.goto('https://grafana.franksdank.cloud/login/generic_oauth', { waitUntil: 'domcontentloaded' }); -await p.waitForTimeout(6000); -const r = await ctx.request.get('https://grafana.franksdank.cloud/api/search?type=dash-db&limit=50'); // 200 + [{uid,title}...] = provisioned -``` -Run it from the checkout that has `tests/e2e/node_modules` + `.auth` + `.env.local` (a fresh worktree lacks them; symlink `node_modules` and copy `.auth` + `.env.local`). Throwaway `_*.cjs`, then move to `.AGENTTRASH/`. - -## Validate every panel query BEFORE shipping (catch empty / mis-scaled panels) -Query Prometheus (`http://127.0.0.1:9090/api/v1/query`) and Loki (`http://127.0.0.1:3100/loki/api/v1/query`) on Dankclaw loopback and confirm non-empty. Real traps found here: -- **Unit mislabel**: `sum by(job,mode)(rate(node_cpu_seconds_total{mode!="idle"}[5m]))` returns **cores per mode** (up to #cores) — unit `short`, NOT `percentunit`. For a per-host busy fraction use `1 - avg by(job)(rate(node_cpu_seconds_total{mode="idle"}[5m]))`. -- **Missing metrics**: `scrape_duration_seconds_bucket` and `prometheus_*`-prefixed self metrics may be absent — confirm from `/api/v1/label/__name__/values?match[]=` before building a panel; `prometheus_target_interval_length_seconds` (a summary with a `quantile` label) exists. -- **Alertmanager** runs on :9093 but is **NOT a Prometheus scrape target** → its metrics are unavailable; don't build panels on it (they read empty). -- **Loki streams**: journal arrives as `{job="systemd-journal"}`; `container_name` is a **parsed JSON field, not a stream label** → group at query time: `sum by (container_name)(count_over_time({job="systemd-journal"} | json [$__auto]))`. -- **Shell trap in test harnesses**: `$__auto` inside a double-quoted ssh/curl string is eaten by the local shell → empty query → Loki returns empty (looks like a broken panel, is a harness bug). For live validation substitute a concrete `[5m]` window, and drive Loki queries with an on-host Python `urllib` client to avoid quoting hell entirely. - -## Datasource uids -`prometheus` (uid `prometheus`), `loki` (uid `loki`); reference as `"datasource": {"type":"prometheus","uid":"prometheus"}` in panel JSON. +## Land flow (repo law) +Branch off fresh `origin/main`; merge needs Frank's explicit current OK; Gitea merge endpoint wants **JSON** `{"Do":"merge"}` (form-encoded `Do=merge` 422s with `[Do]: Required`); then `deploy.sh dankclaw --service grafana` and repeat steps 2–3 as the post-deploy gate. diff --git a/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json b/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json index 2d85cc6..16854fa 100644 --- a/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json +++ b/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792601206018, + "lastConsolidatedAt": 1792604806049, "lastResult": { "at": 1792573818108, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json b/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json index b22d25e..1fb1548 100644 --- a/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json +++ b/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792600941734, + "lastConsolidatedAt": 1792604841754, "lastResult": { "at": 1792574016725, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json b/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json index 4ed4472..8f68a1d 100644 --- a/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json +++ b/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792601138964, + "lastConsolidatedAt": 1792604738996, "lastResult": { "at": 1792556461435, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json b/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json index e02c79c..7fcf959 100644 --- a/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json +++ b/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792601190359, + "lastConsolidatedAt": 1792604850405, "lastResult": { "at": 1792566095222, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json b/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json index ad8406f..0a1670a 100644 --- a/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json +++ b/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1792601083860 + "lastConsolidatedAt": 1792604683894 }