feat(grafana): starter dashboards (host/system, Loki logs, observability self-monitoring) #55

Merged
dank merged 1 commits from feat/grafana-starter-dashboards into main 2026-10-20 21:20:53 -07:00
Owner

What

Three starter Grafana dashboards, provisioned from the repo, auto-loaded by the existing fleet file provider. llm-fleet.json already owns LLM capacity; these cover the domains that had no dashboard.

Dashboard Covers
host-system node_exporter CPU/Load/Mem/Disk/Net/Uptime/Filesystem/FDs for Dankclaw + Dankserver (templated by job)
logs-overview Loki volume, per-container error rate, sample-log panel (container parsed from the JSON line - journal streams have no container_name stream label)
observability Stack health: Prometheus target up/down + scrape-interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem

Evidence

  • Provisioned: GET /api/search?type=dash-db returns all four dashboards (200).
  • Every panel query verified live against Prometheus/Loki (non-empty; Loki | json grouping = 9696 events/5m).
  • CPU panel unit corrected to cores; scrape-p99 panel swapped to prometheus_target_interval_length_seconds (scrape_duration_seconds_bucket absent in this build).

Audit finding (deliberate)

Alertmanager runs on :9093 but is NOT a Prometheus scrape target - no panels built on it (would read empty). Remedy is the scrape-target/blackbox fix already in Observability.

Deploy

Next deploy.sh dankclaw --service grafana installs these; already staged on the host for live review.


cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: wt-20261020-125416-6d1fc1f

## What Three starter Grafana dashboards, provisioned from the repo, auto-loaded by the existing fleet file provider. `llm-fleet.json` already owns LLM capacity; these cover the domains that had no dashboard. |Dashboard|Covers| |---|---| |host-system|node_exporter CPU/Load/Mem/Disk/Net/Uptime/Filesystem/FDs for Dankclaw + Dankserver (templated by job)| |logs-overview|Loki volume, per-container error rate, sample-log panel (container parsed from the JSON line - journal streams have no container_name stream label)| |observability|Stack health: Prometheus target up/down + scrape-interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem| ## Evidence - Provisioned: GET /api/search?type=dash-db returns all four dashboards (200). - Every panel query verified live against Prometheus/Loki (non-empty; Loki | json grouping = 9696 events/5m). - CPU panel unit corrected to cores; scrape-p99 panel swapped to prometheus_target_interval_length_seconds (scrape_duration_seconds_bucket absent in this build). ## Audit finding (deliberate) Alertmanager runs on :9093 but is NOT a Prometheus scrape target - no panels built on it (would read empty). Remedy is the scrape-target/blackbox fix already in [[Observability]]. ## Deploy Next `deploy.sh dankclaw --service grafana` installs these; already staged on the host for live review. --- cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: wt-20261020-125416-6d1fc1f
dank added 1 commit 2026-10-20 21:10:47 -07:00
llm-fleet.json already owns LLM capacity+spec-decode. These three cover the
domains that had NO dashboard:
- host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both
  Dankclaw and Dankserver (templated by job node-dankclaw|node-dankserver)
- logs-overview: Loki volume + per-container error rate + a sample log panel;
  container is parsed from the JSON line at query time (journal streams have
  no container_name stream label)
- observability: the stack's own health - Prometheus target up/down + scrape
  interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem.

Panel queries verified live against Prometheus/Loki; provisioned dashboards
confirmed via /api/search. Alertmanager is running but NOT scraped by
Prometheus - noted in the dashboard, no empty panels built on it.
dank merged commit 01db29e754 into main 2026-10-20 21:20:53 -07:00
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dank/serverAppConfigs#55