agent: repoint roles to frank-medium/dankclaw-3090; sync niri tweaks, memories, managed skills
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
This commit is contained in:
Frank committed
1 parent
83c9c92b3f
commit
135bb2194d
103 files changed
+5333
-170
No files matched your search
@@ -0,0 +1,92 @@
|
||||
---
|
||||
name: grafana-provisioning-as-code
|
||||
description: "Provision a Grafana datasource or dashboard as code on the Dankclaw fleet (shared/grafana/provisioning, deployer copier, planner, seams) and verify it through the Grafana API — including the unified-storage trap that makes the grafana.db check a false negative."
|
||||
---
|
||||
|
||||
# Grafana provisioning as code (Dankclaw fleet)
|
||||
|
||||
Use when adding or changing a Grafana datasource or dashboard on this fleet, or
|
||||
when a dashboard "does not appear" after a deploy.
|
||||
|
||||
## Shape of the thing
|
||||
|
||||
Grafana on Dankclaw is an **empty install configured entirely from git**. The
|
||||
quadlet mounts `/opt/grafana/provisioning` read-only into the container, so a
|
||||
provisioned artefact exists in exactly three places:
|
||||
|
||||
```
|
||||
shared/grafana/provisioning/datasources/<name>.yml # repo: source of truth
|
||||
shared/grafana/provisioning/dashboards/provider.yml # dashboard loader config
|
||||
shared/grafana/provisioning/dashboards/<name>.json # dashboards
|
||||
| deploy-remote.sh: copy_grafana_provisioning()
|
||||
v
|
||||
/opt/grafana/provisioning/... # host: installed state
|
||||
| read-only mount
|
||||
v
|
||||
/etc/grafana/provisioning/... # generated: inside container
|
||||
```
|
||||
|
||||
Nothing is ever clicked into `grafana.db`. A UI-configured datasource would be
|
||||
invisible to review, unknown to anyone reading the repo, and lost with the data
|
||||
volume.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Write the files under `shared/grafana/provisioning/`.
|
||||
- Datasource: `apiVersion: 1`, a **fixed `uid`** (dashboards reference it by
|
||||
uid, so a random one breaks them), `access: proxy`, and the URL by
|
||||
**container name** (`http://prometheus:9090`) — the published port is
|
||||
loopback-only and unreachable from inside the network.
|
||||
- Diagram: `provider.yml` with `allowUiUpdates: true` (three-person homelab;
|
||||
failing an operator who wants a different rate window is the wrong trade)
|
||||
and a JSON dashboard under a stable `uid`.
|
||||
2. Add the file(s) to `copy_grafana_provisioning()` in
|
||||
`scripts/deploy-remote.sh` (it `mkdir -p`s the two directories and `cp`s each
|
||||
file — follow `copy_prometheus_config` for style).
|
||||
3. Planner: `shared/grafana/**` → `grafana` in `scripts/deploy-plan.py`; verify
|
||||
with `python3 scripts/deploy-plan.py --target dankclaw --paths <file>`.
|
||||
4. Add/update a seam in `tests/observability_quadlet.bats`: the JSON must parse,
|
||||
the datasource URL must be the container name, the unit must mount
|
||||
`/opt/grafana/provisioning`, and the deployer must call the copier. Then
|
||||
`./tests/run.sh`.
|
||||
5. Apply: `./scripts/deploy.sh dankclaw --service grafana` (pulls, installs the
|
||||
provisioning tree, restarts Grafana, then runs the e2e gate).
|
||||
|
||||
## Verify — and the trap that makes the obvious check lie
|
||||
|
||||
A **provisioned dashboard does not appear in the legacy `dashboard` table** of
|
||||
`grafana.db` on the current Grafana build; dashboards live in unified storage
|
||||
behind the `dashboard.grafana.app` API group. The datasource *does* land in
|
||||
`data_source`, and that asymmetry is easy to misread as "datasource worked,
|
||||
dashboard failed". A `sqlite3`/python query against `dashboard` is a **false
|
||||
negative**, not evidence.
|
||||
|
||||
Check in this order instead:
|
||||
|
||||
```bash
|
||||
# 1. provisioning actually ran (these loggers name themselves)
|
||||
podman logs --since 10m grafana | grep -iE "inserting datasource|provision dashboards"
|
||||
|
||||
# 2. the dashboard exists (uid is what you set in the JSON)
|
||||
curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"
|
||||
|
||||
# 3. the datasource really reaches Prometheus (proves the whole chain)
|
||||
curl -s -u admin:admin \
|
||||
"http://127.0.0.1:3002/api/datasources/proxy/uid/<uid>/api/v1/query?query=up"
|
||||
```
|
||||
|
||||
Grafana publishes on loopback `:3002` only; the off-host path is the guarded
|
||||
vhost. Its admin login is still the default `admin:admin` — which is why every
|
||||
assertion above needs credentials, and which is worth changing.
|
||||
|
||||
## Before writing any panel or config comment
|
||||
|
||||
Grep the metric names out of the live endpoint; naming a series that does not
|
||||
exist has already shipped in this repo once (`vllm:prefix_cache_hit_rate` — the
|
||||
real pair is `vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total`,
|
||||
whose *rates* give the hit rate). A comment naming an unqueryable metric, or a
|
||||
panel that silently renders "No data", is worse than no panel:
|
||||
|
||||
```bash
|
||||
curl -s http://192.168.0.235:7770/metrics | awk '{print $1}' | sed 's/{.*//' | sort -u | grep vllm
|
||||
```
|
||||
Reference in new issue
Block a user