fix(grafana): dashboard census fixes — every panel now receives data
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).
- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
non-matching pairs and the right side collapses to {job} while the left
carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
so the '> 0' filter can never match. Rewritten to share-of-host-total via
label_replace join of the node exporter's MemTotal by host (the parens
form is a PromQL parse error: nothing may follow group_left); 62 series
live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
(ALERTS carries only firing/pending series); description now says so,
per the Stale Signal rule that 0-series-on-ALERTS is not a defect.
logs-overview and observability: all targets already returning data.
This commit is contained in:
@@ -104,14 +104,14 @@
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Memory vs limit (%)",
|
||||
"description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
|
||||
"title": "Memory vs host total (%)",
|
||||
"description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
|
||||
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
|
||||
"expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
|
||||
"legendFormat": "{{host}} / {{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
|
||||
@@ -54,7 +54,7 @@
|
||||
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
"targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "stat", "title": "Open file handles",
|
||||
|
||||
@@ -194,7 +194,7 @@
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Firing fleet alerts",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
|
||||
Reference in New Issue
Block a user