fix(grafana): dashboard census fixes — every panel now receives data

Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).

- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
  non-matching pairs and the right side collapses to {job} while the left
  carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
  live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
  container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
  so the '> 0' filter can never match. Rewritten to share-of-host-total via
  label_replace join of the node exporter's MemTotal by host (the parens
  form is a PromQL parse error: nothing may follow group_left); 62 series
  live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
  (ALERTS carries only firing/pending series); description now says so,
  per the Stale Signal rule that 0-series-on-ALERTS is not a defect.

logs-overview and observability: all targets already returning data.
This commit is contained in:
Frank
2026-10-21 02:45:57 -07:00
parent a61308d72e
commit c96d0228a4
3 changed files with 5 additions and 5 deletions
@@ -104,14 +104,14 @@
{
"id": 6,
"type": "timeseries",
"title": "Memory vs limit (%)",
"description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
"title": "Memory vs host total (%)",
"description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
"targets": [
{
"expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
"expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
"legendFormat": "{{host}} / {{container}}",
"refId": "A"
}
@@ -54,7 +54,7 @@
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
"targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
"targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
},
{
"id": 4, "type": "stat", "title": "Open file handles",
@@ -194,7 +194,7 @@
"id": 6,
"type": "timeseries",
"title": "Firing fleet alerts",
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
"gridPos": {
"h": 8,
"w": 12,