serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md

---
type: reference
title: "Observability on Dankclaw"
description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured."
tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw]
timestamp: 2026-10-20
---

# Observability on Dankclaw

## Summary

Six containers and one host-side process: Prometheus (metrics), Grafana
(dashboards), Loki (log store), Alloy (log shipper), node-exporter (host
metrics) and Alertmanager (alert delivery), with `signal-relay` on the host
carrying every firing group to Signal. All of it is Quadlet-managed,
boot-persistent and loopback-only, deployed by
`scripts/deploy-remote.sh` → `deploy_observability`. The metrics half works
end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version
was deleted down to its front matter by commit `93d1181` (the "OKF v0.1
compliance pass"), while sibling pages kept pointing at a "state note" on this
page that no longer existed. Every statement below carries the command or file
that proves it today.

## The stack

| Service | Container | Published | Unit | Page |
|---|---|---|---|---|
| Prometheus | `docker.io/prom/prometheus:latest` | `127.0.0.1:9090` | `prometheus.container` | [[Prometheus]] |
| Alertmanager | `docker.io/prom/alertmanager:latest` | `127.0.0.1:9093` | `alertmanager.container` | [[Alertmanager]] |
| Grafana | `docker.io/grafana/grafana:latest` | `127.0.0.1:3002` → 3000 | `grafana.container` | [[Grafana]] |
| Loki | `localhost/franksdank/loki:latest` | `127.0.0.1:3100` | `loki.container` | [[Loki]] |
| Alloy | `localhost/franksdank/alloy:latest` | `127.0.0.1:12345` | `alloy.container` | [[Alloy]] |
| node-exporter | `docker.io/prom/node-exporter:latest` | host netns: `127.0.0.1:9100` + `192.168.0.143:9100` | `node-exporter.container` | [[Node-Exporter]] |
| Signal relay | host `python3`, not a container | `192.168.0.143:18093` | `scripts/signal-relay.service` (user unit) | [[Alertmanager]] |

Every unit lives in `dankclaw/observability/quadlet/`, carries
`Restart=always`/`RestartSec=10` and `WantedBy=default.target`, is installed at
`~/.config/containers/systemd/<name>.container`, and reports `generated` +
`active` (`systemctl --user is-enabled|is-active` for all six, 2026-10-18;
linger is on for `dank`). `signal-relay.service` reports `enabled` + `active`
the same day, and `homepage-status.timer` — the root unit that probes all of
these on loopback — is `enabled` + `active` (`deploy-remote.sh:319-321` copies
the probe registry `shared/homepage/services.json` to `/srv/homepage`).
Dankserver runs one member of this family, node-exporter, as its own unit
(`dankserver/observability/quadlet/node-exporter.container`, `generated` +
`active`, binding `127.0.0.1:9100` + `192.168.0.235:9100`).

## Exposure

The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18)
shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`
`127.0.0.1:12345` and no wildcard — because each of them is unauthenticated
inside, and several carry lifecycle or query APIs. Browser access goes through
Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard
`anubis:8923`, and the post-auth handlers route by container name on
`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`)
`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`)
`prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`).
Alertmanager has no vhost at all. node-exporter is the exception that must live
in the host network namespace, and it binds exactly two addresses rather than
the wildcard — see [[Node-Exporter]].

The relay's bind is the one address a bridge container can reach on the host:
`192.168.0.143:18093`, with iptables INPUT admitting loopback, the host itself
and the podman bridge subnet `10.89.0.0/16` before the DROP
(`scripts/deploy-remote.sh:255-262`).

## Deployment

`./scripts/deploy.sh dankclaw --service <name>` runs `deploy_observability`
which installs the unit, copies that service's config
(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`
`copy_alloy_config`, `copy_grafana_provisioning`, plus
`install_signal_relay` with the alertmanager deploy), then waits on the loopback
health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`
`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`
`/node-exporter`: `/metrics`). Startup order follows the data dependencies —
`grafana` after `prometheus`, `alloy` after `loki`
(`grafana.container:32-33`, `alloy.container:38-39`). See
[[Observability Provisioning]] for what is provisioned as code and how to
verify it through the APIs.

## Current state (2026-10-20)

* **Metrics: working.** `curl -s http://127.0.0.1:9090/api/v1/targets` reports
  eight up targets — the three LLM endpoints (frank-archon on `vllm:*`,
  frank-dragoon and frank-zealot on `llamacpp:*`), grafana, loki, prometheus
  and both node exporters. The four downs are structural (the `comfyui` and
  `services` HTML probes, and Gitea's disabled metrics endpoint), not outages.
  Two rule groups in `shared/prometheus/rules/llm-fleet.yml`: `llm-fleet`
  (2 recording + 4 alert, `llamacpp:*`) and `vllm-fleet` (1 recording + 4
  alert, `vllm:*` — archon queueing, prefix collapse, preemption, p95
  time-to-first-token), each keyed only to the namespace its runtime exports.
* **Delivery: working.** Alertmanager's single `signal` receiver webhooks the
  relay, and `journalctl --user -u signal-relay` records successful
  `sent to +12062476336:` lines for every rule in the set.
* **Dashboards: working.** Provisioned datasources — Prometheus (uid
  `prometheus`) and Loki (uid `loki`) — and the `llm-fleet` dashboard, all
  installed from `shared/grafana/provisioning/`. The Loki datasource is how
  you read logs: Loki has no web UI of its own, so Grafana Explore is the
  human surface (the guarded `loki.franksdank.cloud` vhost serves only the API).
* **Logs: configured, empty.** Alloy ships zero lines and Loki holds zero
  streams (`GET /loki/api/v1/labels` returns no label names; no chunk files on
  disk). The cause is source-side access, now proven, not a config bug: a
  rootless container cannot read the journal files (`system.journal` is
  `root:systemd-journal 0640`/`adm` and a rootless read of it returns DENIED),
  and the container-log globs point at a root-storage path that does not exist
  under rootless. Closing it is a permission/architecture decision, not an
  edit — see [[Alloy]] → Delivery state and [[Loki]] → Contents.
* **Not configured:** no blackbox exporter (the structural downs have no real
  uptime signal) and no Alertmanager vhost. The log pipeline is provisioned
  but carries no data (see *Logs* above).

## Citations

- `dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container`
- `scripts/deploy-remote.sh:219-300` (copiers + relay), `:521-536` (health waits), `:477-486` (grafana env seed)
- `scripts/signal-relay.py:26-49`, `scripts/signal-relay.service`
- `shared/prometheus/prometheus.yml`, `shared/prometheus/alertmanager.yml`, `shared/prometheus/rules/llm-fleet.yml`
- `shared/grafana/provisioning/`, `shared/loki/loki-config.yaml`, `shared/alloy/config.alloy`
- `tests/observability_quadlet.bats` (boot persistence, no wildcard publish, probe addresses, delivery contract)
- Host evidence 2026-10-18: `ss -ltnp`, `podman ps`, `systemctl --user is-enabled|is-active`, `curl` on `127.0.0.1:9090/api/v1/{targets,rules,alerts}`, `127.0.0.1:3100/loki/api/v1/labels`, `127.0.0.1:12345/metrics`, `127.0.0.1:3002/api/health`, `192.168.0.143:18093/health`, `journalctl --user -u signal-relay`