fix(grafana): dashboard census fixes — every panel now receives data

Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).

- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
  non-matching pairs and the right side collapses to {job} while the left
  carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
  live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
  container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
  so the '> 0' filter can never match. Rewritten to share-of-host-total via
  label_replace join of the node exporter's MemTotal by host (the parens
  form is a PromQL parse error: nothing may follow group_left); 62 series
  live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
  (ALERTS carries only firing/pending series); description now says so,
  per the Stale Signal rule that 0-series-on-ALERTS is not a defect.

logs-overview and observability: all targets already returning data.
This commit is contained in:
Frank committed 2026-10-21 02:45:57 -07:00
1 parent a61308d72e
commit c96d0228a4
3 files changed
+5 -5

No files matched your search

@@ -104,14 +104,14 @@
{
"id": 6,
"type": "timeseries",
"title": "Memory vs limit (%)",
"description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
"title": "Memory vs host total (%)",
"description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
"targets": [
{
"expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
"expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
"legendFormat": "{{host}} / {{container}}",
"refId": "A"
}