command-center/agent/managed-skills/observability-stack-live-census/SKILL.md

name, description

name	description
observability-stack-live-census	Census the Dankclaw observability stack (Prometheus/Grafana/Loki/Alloy/Alertmanager/signal-relay) live and audit per-endpoint alert coverage; reconcile against the owner page and rule-file comments, which go stale.

Observability stack — live census + alert-coverage audit

Use when asked "status of the observability stack" on Dankclaw, or before claiming a service is/isn't monitored. Everything runs rootless on Dankclaw: ssh dankclaw bash -s <<'EOF' … EOF. Verify live; the owner page (serverAppConfigs-docs/Infrastructure/Observability.md) and shared/prometheus/rules/llm-fleet.yml comments go stale.

Loopback ports (do NOT probe the wrong one)

• Prometheus 127.0.0.1:9090 · Alertmanager 127.0.0.1:9093 · Loki 127.0.0.1:3100 · Alloy 127.0.0.1:12345 · signal-relay 192.168.0.143:18093
• Grafana host port is 127.0.0.1:3002 (→ container :3000). Host :3000 is a rootlessport for some other rootless container and answers 200-on-/ but 404-on-/api/health. Probing :3000 and seeing 404 is a false alarm — always use 3002 for Grafana. GF_SERVER_ROOT_URL=https://grafana.franksdank.cloud.

Units + containers

for u in prometheus grafana loki alloy alertmanager node-exporter blackbox signal-relay; do systemctl --user is-active $u.service; systemctl --user is-enabled $u.service; done

• Quadlet units report generated (healthy), not enabled; signal-relay is a real host unit (enabled).
• blackbox = not-found/inactive → the known gap (no blackbox exporter).
• podman ps for Up N days (healthy).

Prometheus — targets / rules / alerts

• Targets: GET :9090/api/v1/targets → .data.activeTargets[] | .health, .labels.job, .lastError. Expect ~8 up / 4 down. The 4 downs are structural, not outages: comfyui + two services (HTML content-type, no real /metrics) and gitea (404, metrics disabled). Root remedy = deploy a blackbox exporter (not yet present).
• Rules: GET :9090/api/v1/rules → count .data.groups[].rules. The llm-fleet group is 4 alert + 2 recording (6 total). AGENTS.md saying "5 alert + 1 recording" is wrong — trust the rule file.
• Firing: GET :9090/api/v1/alerts; Alertmanager: GET :9093/api/v2/alerts and /api/v2/silences.

Alert-coverage per runtime — the sharp step (catches real blind spots)

The llm-fleet perf alerts (LlmRequestsQueueing/LlmPrefixCacheCollapse/LlmSpecDecodeOff) key on the llamacpp:* family. Which endpoints actually emit it:

GET :9090/api/v1/query?query=count by (job)({__name__=~"llamacpp:.+"})
GET :9090/api/v1/query?query=up{job=~"frank-.*"}
GET :9090/api/v1/query?query=count by (__name__)({job="frank-archon"})   # what archon really exports

Ground truth (2026-10-20):

• frank-zealot (llama.cpp) → llamacpp:* (17 series) → perf alerts cover it.
• frank-dragoon (halogen) → still emits llamacpp:* (10 series; halogen re-uses the names) → perf alerts cover it.
• frank-archon (radiance) → exports the vllm:* family (vllm:num_requests_waiting, vllm:gpu_prefix_cache_hit_rate, vllm:num_preemptions_total, vllm:time_to_first_token_seconds_bucket = a real latency histogram), not llamacpp:* → NO perf alert fires on archon; only LlmEndpointDown (up{job=~"frank-.*"}) catches it. Up/down yes, perf/queueing/prefix/p95 = collected but unalerted.
• The rule-file header comment claims "frank-archon LEFT the scrape set 2026-10-05, FreeToken exports no /metrics" — stale/false: archon (radiance) is up and exports vllm:*. LlmEndpointDown's description repeats the same stale claim. Never assert coverage from a comment; query the families.

Log pipeline emptiness test

Loki is "configured, healthy, empty": GET :3100/loki/api/v1/labels → {"status":"success"} with no data array = 0 label names = 0 streams. Confirm: find /opt/loki/data -name '*.tsm' -o -name 'chunks-*' | wc -l → 0 chunks (≈8 WAL/bookkeeping files only). Alloy ships 0 lines (journal counter 0, both file globs refused per the page — a config bug, not capacity). No Loki datasource in Grafana. Report the log half as non-functional, not merely "healthy."

Reporting posture

• Metrics half works (up/down for all 3 endpoints + node-exporter + self-scrape); log half dead-on-arrival; blackbox + archon-perf-alerts + Loki-datasource + Alertmanager-vhost are the open gaps.
• Grafana provisioning is file-based — check podman exec grafana ls /etc/grafana/provisioning/{datasources,dashboards} (expect prometheus.yml, llm-fleet.json) rather than fighting the API for creds.
• Alertmanager has no persistence: a reboot re-stamps startsAt on long-firing alerts.
• Flag stale facts (AGENTS.md rule split, archon-rejoin comments) but let the user decide whether to fix docs.