Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
93 lines
4.4 KiB
Markdown
93 lines
4.4 KiB
Markdown
---
|
|
name: grafana-provisioning-as-code
|
|
description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative."
|
|
---
|
|
|
|
# Grafana provisioning as code (Dankclaw fleet)
|
|
|
|
Use when adding or changing a Grafana datasource or dashboard on this fleet, or
|
|
when a dashboard "does not appear" after a deploy.
|
|
|
|
## Shape of the thing
|
|
|
|
Grafana on Dankclaw is an **empty install configured entirely from git**. The
|
|
quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a
|
|
provisioned artefact exists in exactly three places:
|
|
|
|
```
|
|
shared/grafana/provisioning/datasources/<name>.yml # repo: source of truth
|
|
shared/grafana/provisioning/dashboards/provider.yml # dashboard loader config
|
|
shared/grafana/provisioning/dashboards/<name>.json # dashboards
|
|
| deploy-remote.sh: copy_grafana_provisioning()
|
|
v
|
|
/opt/grafana/provisioning/... # host: installed state
|
|
| read-only mount
|
|
v
|
|
/etc/grafana/provisioning/... # generated: inside container
|
|
```
|
|
|
|
Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be
|
|
invisible to review, unknown to anyone reading the repo, and lost with the data
|
|
volume.
|
|
|
|
## Procedure
|
|
|
|
1. Write the files under `shared/grafana/provisioning/`.
|
|
- Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by
|
|
uid, so a random one breaks them), `access: proxy`, and the URL by
|
|
**container name** (`http://prometheus:9090`) — the published port is
|
|
loopback-only and unreachable from inside the network.
|
|
- Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab;
|
|
failing an operator who wants a different rate window is the wrong trade)
|
|
and a JSON dashboard under a stable `uid`.
|
|
2. Add the file(s) to `copy_grafana_provisioning()` in
|
|
`scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each
|
|
file — follow `copy_prometheus_config` for style).
|
|
3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify
|
|
with `python3 scripts/deploy-plan.py --target dankclaw --paths <file>`.
|
|
4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse,
|
|
the datasource URL must be the container name, the unit must mount
|
|
`/opt/grafana/provisioning`, and the deployer must call the copier. Then
|
|
`./tests/run.sh`.
|
|
5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the
|
|
provisioning tree, restarts Grafana, then runs the e2e gate).
|
|
|
|
## Verify — and the trap that makes the obvious check lie
|
|
|
|
A **provisioned dashboard does not appear in the legacy `dashboard` table** of
|
|
`grafana.db` on the current Grafana build; dashboards live in unified storage
|
|
behind the `dashboard.grafana.app` API group. The datasource *does* land in
|
|
`data_source`, and that asymmetry is easy to misread as "datasource worked,
|
|
dashboard failed". A `sqlite3`/python query against `dashboard` is a **false
|
|
negative**, not evidence.
|
|
|
|
Check in this order instead:
|
|
|
|
```bash
|
|
# 1. provisioning actually ran (these loggers name themselves)
|
|
podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"
|
|
|
|
# 2. the dashboard exists (uid is what you set in the JSON)
|
|
curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"
|
|
|
|
# 3. the datasource really reaches Prometheus (proves the whole chain)
|
|
curl -s -u admin:admin \
|
|
"http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"
|
|
```
|
|
|
|
Grafana publishes on loopback `:3002` only; the off-host path is the guarded
|
|
vhost. Its admin login is still the default `admin:admin` — which is why every
|
|
assertion above needs credentials, and which is worth changing.
|
|
|
|
## Before writing any panel or config comment
|
|
|
|
Grep the metric names out of the live endpoint; naming a series that does not
|
|
exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the
|
|
real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`,
|
|
whose *rates* give the hit rate). A comment naming an unqueryable metric, or a
|
|
panel that silently renders "No data", is worse than no panel:
|
|
|
|
```bash
|
|
curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm
|
|
```
|