---
type: reference
title: "Observability on Dankclaw"
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, gpu-exporter, quadlet, dankclaw]
timestamp: 2026-10-20
---

# Observability on Dankclaw

## Summary

Seven containers and two host-side processes: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics), cadvisor (per-container metrics) and Alertmanager (alert delivery),
plus two host `python3` processes — `gpu-exporter` (per-GPU metrics) and `signal-relay` (Signal delivery). The containers are all Quadlet-managed,
boot-persistent and loopback-only, deployed by
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
compliance pass"), while sibling pages kept pointing at a "state note" on this
page that no longer existed. Every statement below carries the command or file
that proves it today.

## The stack

| Service | Container | Published | Unit | Page |
|---|---|---|---|---|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
| cadvisor | `gcr.io/cadvisor/cadvisor:v0.49.1` | bridge publish `127.0.0.1:8080` + `192.168.0.143:8080` | `cadvisor.container` | — |
| GPU exporter | host `python3`, not a container | `127.0.0.1:9401` + host LAN `:9401` | `scripts/gpu-exporter.service` (user unit) | — |
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |

Every unit lives in `dankclaw/observability/quadlet/`, carries
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
the same day, and `homepage-status.timer` — the root unit that probes all of
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
Dankserver runs the same family: node-exporter and cadvisor as its own quadlets
(`dankserver/observability/quadlet/{node-exporter,cadvisor}.container`,
`generated` + `active`; node-exporter binds `127.0.0.1:9100` + `192.168.0.235:9100`,
cadvisor publishes `127.0.0.1:8080` + `192.168.0.235:8080`), plus the host-process
`gpu-exporter` (a user unit installed the same way as `signal-relay`).

## Exposure

The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
inside, and several carry lifecycle or query APIs. Browser access goes through
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
`anubis:8923`, and the post-auth handlers route by container name on
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see [[Node-Exporter]].

**Same-host reachability from a bridge container is measured, not inferred**
(2026-10-21, from inside the Prometheus container): `host.containers.internal`
(the podman gateway) reaches host services whether they bind host-net or are
published ports — node `:9100`, cadvisor `:8080`, gpu `:9401` all answer there
— while the host's own LAN IP is refused (`192.168.0.143:8080`, `:9401`), so
every same-host Prometheus target uses the gateway and only cross-host targets
(dankserver) use plain LAN. Whether LAN works at all is network-specific: the
relay at `192.168.0.143:18093` IS reached from alertmanager's network, with
iptables INPUT admitting loopback, the host itself and the podman bridge subnet
`10.89.0.0/16` before the DROP (`scripts/deploy-remote.sh:255-262`). Never
point a scrape target at a host LAN IP from its own host on the strength of
one working example; dial the gateway and probe.

## Deployment

`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
which installs the unit, copies that service's config
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
`copy_alloy_config`, `copy_grafana_provisioning`, plus
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
`grafana` after `prometheus`, `alloy` after `loki`
(`grafana.container:32-33`, `alloy.container:38-39`). See
[[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs.

## Current state (2026-10-20)

* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
  and both node exporters. The four downs are structural (the `comfyui` and
  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
  time-to-first-token), each keyed only to the namespace its runtime exports.
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet`, `host-system`,
  `containers`, `observability` and `logs-overview` dashboards, all
  installed from `shared/grafana/provisioning/`. The Loki datasource is how
  you read logs: Loki has no web UI of its own, so Grafana Explore is the
  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
* **Per-container metrics: the `containers` dashboard (added 2026-10-20; live
  after `deploy.sh <host> --service cadvisor`).** cadvisor is the only way to see
  per-container CPU/memory here, and on this rootless-Podman host it works on one
  condition: `--cgroupns=host`. Podman starts every container in a PRIVATE cgroup
  namespace, so without that flag cadvisor sees only its own cgroup root and
  exports a single series (measured 2026-10-20: 1 series off, 996 on). cadvisor
  names nothing itself — the libpod socket resolved only 2 of 34 rootless
  containers — but the quadlet unit name is carried in the cgroup path
  (`app.slice/<name>.service/…`), so `prometheus.yml` relabels that path into a
  `container` label. Network and filesystem are NOT exported for these containers
  (no netns/classid attribution under the host-PID layout), so the dashboard
  graphs only CPU, memory and OOM and does not fake the rest.
* **Per-GPU metrics: the `host-system` GPU panels (added 2026-10-20; live after
  `deploy.sh <host> --service gpu-exporter`).** `scripts/gpu-exporter.py` is a
  host-process user unit on BOTH hosts — not a quadlet, because it reads the host's
  own DRM sysfs and shells to `nvidia-smi`, neither of which a container can do
  without GPU passthrough. It reads each AMD card's `gpu_busy_percent` /
  `mem_info_vram_*` / `hwmon temp1_input` and one `nvidia-smi --query-gpu` CSV, and
  emits unified `gpu_*` families (utilization %, VRAM used/total bytes, temperature
  °C, power W) labelled `gpu` / `vendor` / `model`, with `host` added by the scrape
  job. Two laws: **absent means absent, never a zero** — the R9700s expose no sysfs
  `power1_average`, so they carry no `gpu_power_watts` line at all (only NVIDIA and
  the Dankclaw 8060S iGPU report power); and it binds loopback + the host LAN, like
  node-exporter, never the wildcard. The dashboard graphs util/VRAM/VRAM-%/temp/power.
* **Logs: fixed in code, awaiting deploy.** The pipeline shipped nothing because
  a rootless Alloy cannot read the `0640` journal files, and the container-log
  globs point at a root-storage layout that does not exist here. The fix is built
  and tested in-repo (2026-10-20): Alloy is now a **root system service on
  `Network=host`**, reading `system.journal` + `user-*.journal` and pushing to
  Loki's `127.0.0.1:3100` loopback publish — a throwaway root probe read 31,825
  lines and pushed 26,966 with zero drops. The **live** instance is still the old
  rootless unit until `deploy.sh dankclaw --service alloy` runs. Detail in
  [[Alloy]] → Why root; Loki's store state in [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real
  uptime signal) and no Alertmanager vhost. The log pipeline's fix is in code but
  not yet deployed (see *Logs* above).

## Citations

- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter,cadvisor}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `scripts/gpu-exporter.py`, `scripts/gpu-exporter.service` (per-GPU host exporter)
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`