0 Fork 0

Files

T

Frank 135bb2194d agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills

- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh
- models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes
- niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0
- memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills
- managed-skills: 40+ new fleet runbook skills
- agent/skills: import built-in skill set

2026-09-17 12:27:08 -07:00

4.4 KiB

Raw Blame History

name, description

name	description
grafana-provisioning-as-code	Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative.

Grafana provisioning as code (Dankclaw fleet)

Use when adding or changing a Grafana datasource or dashboard on this fleet, or when a dashboard "does not appear" after a deploy.

Shape of the thing

Grafana on Dankclaw is an empty install configured entirely from git. The quadlet mounts /opt/grafana/provisioning read-only into the container, so a provisioned artefact exists in exactly three places:

shared/grafana/provisioning/datasources/<name>.yml    # repo: source of truth
shared/grafana/provisioning/dashboards/provider.yml   # dashboard loader config
shared/grafana/provisioning/dashboards/<name>.json    # dashboards
        |  deploy-remote.sh: copy_grafana_provisioning()
        v
/opt/grafana/provisioning/...                          # host: installed state
        |  read-only mount
        v
/etc/grafana/provisioning/...                          # generated: inside container

Nothing is ever clicked into grafana.db. A UI-configured datasource would be invisible to review, unknown to anyone reading the repo, and lost with the data volume.

Procedure

Write the files under shared/grafana/provisioning/.
- Datasource: apiVersion: 1, a fixed uid (dashboards reference it by uid, so a random one breaks them), access: proxy, and the URL by container name (http://prometheus:9090) — the published port is loopback-only and unreachable from inside the network.
- Diagram: provider.yml with allowUiUpdates: true (three-person homelab; failing an operator who wants a different rate window is the wrong trade) and a JSON dashboard under a stable uid.
Add the file(s) to copy_grafana_provisioning() in scripts/deploy-remote.sh (it mkdir -ps the two directories and cps each file — follow copy_prometheus_config for style).
Planner: shared/grafana/** → grafana in scripts/deploy-plan.py; verify with python3 scripts/deploy-plan.py --target dankclaw --paths <file>.
Add/update a seam in tests/observability_quadlet.bats: the JSON must parse, the datasource URL must be the container name, the unit must mount /opt/grafana/provisioning, and the deployer must call the copier. Then ./tests/run.sh.
Apply: ./scripts/deploy.sh dankclaw --service grafana (pulls, installs the provisioning tree, restarts Grafana, then runs the e2e gate).

Verify — and the trap that makes the obvious check lie

A provisioned dashboard does not appear in the legacy dashboard table of grafana.db on the current Grafana build; dashboards live in unified storage behind the dashboard.grafana.app API group. The datasource does land in data_source, and that asymmetry is easy to misread as "datasource worked, dashboard failed". A sqlite3/python query against dashboard is a false negative, not evidence.

Check in this order instead:

# 1. provisioning actually ran (these loggers name themselves)
podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"

# 2. the dashboard exists (uid is what you set in the JSON)
curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"

# 3. the datasource really reaches Prometheus (proves the whole chain)
curl -s -u admin:admin \
  "http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"

Grafana publishes on loopback :3002 only; the off-host path is the guarded vhost. Its admin login is still the default admin:admin — which is why every assertion above needs credentials, and which is worth changing.

Before writing any panel or config comment

Grep the metric names out of the live endpoint; naming a series that does not exist has already shipped in this repo once (vllm:prefix_cache_hit_rate — the real pair is vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give the hit rate). A comment naming an unqueryable metric, or a panel that silently renders "No data", is worse than no panel:

curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm

4.4 KiB Raw Blame History