dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: observability state after the Grafana provisioning

Browse Source 
The page said Grafana has zero datasources and zero dashboards; that is now
false, so it says what exists (Prometheus datasource, LLM fleet dashboard), how
it was verified (the API — a provisioned dashboard does not show up in the
legacy dashboard table on this Grafana version, which is a trap worth recording)
and what is still missing (nothing alerted on, Loki unread, correlation manual).
The section heading drops "UNCONFIGURED" because it no longer is, entirely.
This commit is contained in:
Frank
2026-09-16 06:42:24 -07:00
parent 6a191430ca
commit c272407e6e
1 changed files with 4 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/Infrastructure/Observability.md
+4 -4
View File
@@ -31,17 +31,17 @@ graph LR
| [[Alloy]] | log shipping | `franksdank/alloy` (local build) | `127.0.0.1:12345` |
| [[Node-Exporter]] | host metrics | `prom/node-exporter` | `192.168.0.143:9100`, host netns |
## Current state: installed, healthy, and UNCONFIGURED
## Current state: installed, healthy, and mostly unconfigured
Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
- **Grafana has zero datasources and zero dashboards.** `grafana.db` reports 0 rows in `data_source` and 0 in `dashboard`; `/opt/grafana/provisioning/` is empty, and the only account is the local `admin`. Nothing in the UI can query Prometheus or Loki yet. Datasources, dashboards, and provisioning-as-code are all outstanding work.
- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
- **Prometheus has no alerting at all.** `/opt/prometheus/config/` contains `prometheus.yml` and nothing else — no rule files, no recording rules — and there is no Alertmanager anywhere in the stack. It scrapes and retains; it never notifies anyone.
- **Loki is configured but has no reader.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; nothing queries it except manual Explore once Grafana is wired up.
- **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
- **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
- **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
The honest summary: metrics and logs are being collected and retained, and nothing is being visualized, correlated or alerted on. Any claim of "observability" beyond raw collection is aspirational until Grafana has datasources and there is at least one alerting path.
The honest summary: metrics and logs are collected and retained, one dashboard now visualizes the LLM fleet, and **nothing is alerted on** — there is no rules file and no Alertmanager anywhere, so every graph in this stack is read after the fact by a human, and correlating metrics with logs is still manual.
## Deployment