Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
95 lines
4.9 KiB
Markdown
95 lines
4.9 KiB
Markdown
---
|
|
name: grafana-prometheus-provisioning-as-code
|
|
description: "Add or change Grafana datasources/dashboards and Prometheus scrape jobs/alert rules in this homelab repo as code — including the planner+copier+seam triad, the silent failure modes, and the API-based verification that avoids two false negatives."
|
|
---
|
|
|
|
# Observability as code (Grafana + Prometheus, Dankclaw fleet)
|
|
|
|
Use when adding a scrape target, an alert rule, a datasource, or a dashboard to
|
|
this homelab. Everything lives in git; Grafana and Prometheus only ever read
|
|
what the deployer installs.
|
|
|
|
## The triad — every change touches three places
|
|
|
|
1. **Config in the repo**
|
|
- `shared/prometheus/prometheus.yml` — `scrape_configs`, and `rule_files:`
|
|
pointing at `/etc/prometheus/rules/*.yml`
|
|
- `shared/prometheus/rules/<name>.yml` — alert/recording rules
|
|
- `shared/grafana/provisioning/datasources/prometheus.yml` — datasource
|
|
- `shared/grafana/provisioning/dashboards/{provider.yml,<name>.json}`
|
|
2. **The copier in `scripts/deploy-remote.sh`** — `copy_prometheus_config`
|
|
installs config **and** rules into `/opt/prometheus/config/` (the container
|
|
mounts that to `/etc/prometheus`); `copy_grafana_provisioning` installs into
|
|
`/opt/grafana/provisioning/` (the grafana unit already mounts it read-only).
|
|
Paths are absolute and `run_privileged`.
|
|
3. **The planner in `scripts/deploy-plan.py`** — a `shared/<group>/**` change
|
|
must select the service that reads it (`shared/prometheus` → prometheus,
|
|
`shared/grafana` → grafana). Missing mapping = the file deploys nowhere and
|
|
nothing reports an error.
|
|
|
|
Add a seam in `tests/observability_quadlet.bats` for each change: the file
|
|
exists, the config references it, the deployer copies it, and — where two files
|
|
must agree (target vs published port) — that they do.
|
|
|
|
## Reach targets by container name, never the loopback publish
|
|
|
|
Datasources use `http://prometheus:9090` and scrape jobs for co-located
|
|
services use `prometheus:9090` / `loki:3100`. Published ports are
|
|
`127.0.0.1`-only by design (see the exposure rules), so a browser-facing URL or
|
|
a published port in a datasource is wrong from inside the network. LLM
|
|
endpoints are the exception: they are on other hosts, so use their LAN address.
|
|
|
|
## Three failure modes that produce no error message
|
|
|
|
- **A rule file nothing loads evaluates nothing.** Prometheus reports no
|
|
problem for a `rule_files` entry that matches nothing, or for a rules file
|
|
nobody references. Ship both together and assert the reference in a seam.
|
|
- **A provisioned dashboard is not in the legacy `dashboard` table** on the
|
|
current Grafana. Reading `grafana.db` for it returns empty and looks like a
|
|
failed provision. It lives in unified storage behind
|
|
`dashboard.grafana.app`; query the API instead.
|
|
- **Lifetime averages are not per-request evidence.** e.g. an engine's
|
|
"Prefix cache hit rate: 29 %" is every prompt it ever saw, including
|
|
benchmarks. Measure the behaviour you care about with a targeted probe.
|
|
|
|
## Verify through the channel the consumer uses
|
|
|
|
```bash
|
|
# Prometheus: config and rules parse/load
|
|
ssh dankclaw 'podman exec prometheus promtool check rules /etc/prometheus/rules/<name>.yml'
|
|
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/rules?type=alert" \
|
|
| python3 -c "import json,sys; print([r[\"name\"] for g in json.load(sys.stdin)[\"data\"][\"groups\"] for r in g[\"rules\"]])"'
|
|
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | ...' # every target health=up
|
|
ssh dankclaw 'curl -s "http://127.0.0.1:9090/api/v1/alerts" | ...' # 0 firing when healthy
|
|
|
|
# Grafana: provisioned objects and a live query (admin/admin on this install)
|
|
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/search?type=dash-db"'
|
|
ssh dankclaw 'curl -s -u admin:admin "http://127.0.0.1:3002/api/dashboards/uid/<uid>"'
|
|
ssh dankclaw 'curl -s -u admin:admin \
|
|
"http://127.0.0.1:3002/api/datasources/proxy/uid/prometheus/api/v1/query?query=count(up==1)"'
|
|
```
|
|
|
|
A target that is `up` proves reachability, not data: also run one query for a
|
|
series you expect and confirm it returns a sample.
|
|
|
|
## Grafana facts for this install
|
|
|
|
- No UI-configured state: `editable: false` on the datasource, fixed
|
|
`uid: prometheus` so dashboards reference it without a lookup.
|
|
- Dashboards re-read from disk every 30 s (`updateIntervalSeconds`), so a JSON
|
|
edit appears without a restart — but the file must still be *installed*, which
|
|
means a deploy.
|
|
- There is **no Alertmanager and no contact point**: rules evaluate, alerts show
|
|
on `/alerts`, in the `ALERTS` metric and on a dashboard panel, and reach
|
|
nobody. Say that plainly in docs rather than implying observability.
|
|
|
|
## Naming metrics
|
|
|
|
Never write a series name from memory into a config, a rule, or a dashboard —
|
|
one shipped comment here named a metric that does not exist. Dump the live
|
|
names first and paste them:
|
|
|
|
```bash
|
|
ssh dankserver 'curl -s http://127.0.0.1:7770/metrics | grep -E "^vllm:" | awk "{print \$1}" | sed "s/{.*//" | sort -u'
|
|
```
|