dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a3bce7d123cd968c1a75bca65c97896281571fca
serverAppConfigs/serverAppConfigs-docs/Infrastructure/Prometheus.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

92 lines
8.3 KiB
Markdown
Raw Blame History

---
type: reference
description: "Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural."
tags:
- infrastructure
- prometheus
timestamp: 2026-09-21
---
# Prometheus
Prometheus collects metrics from the services listed below by scraping their `/metrics` endpoints, and it is the datasource behind the provisioned `llm-fleet` dashboard in [[Grafana]] (datasource uid `prometheus`, `shared/grafana/provisioning/datasources/prometheus.yml:14-21`). node-exporter supplies host metrics from both machines.
**Host:** Dankclaw — **Vhost:** `prometheus.franksdank.cloud` (guarded: Anubis →
Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
**Port:** 9090 (loopback) — **Docs:** https://prometheus.io/docs
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/prom/prometheus:latest` |
| Published | `127.0.0.1:9090` → 9090, loopback only — never a wildcard (`dankclaw/observability/quadlet/prometheus.container:50`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9090` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
| Unit | `dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`prometheus.container:59-64`), installed at `~/.config/containers/systemd/prometheus.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active prometheus` 2026-10-18) |
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
## Targets
`shared/prometheus/prometheus.yml` is the source of truth; the deploy copies it into the container's `/etc/prometheus/`. In outline:
| Job | Target | Why |
|-----|--------|-----|
| `prometheus` | `localhost:9090` | itself |
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] |
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
| Job | Target | `health` | `lastError`, verbatim |
|-----|--------|----------|------------------------|
| `gitea` | `http://192.168.0.143:3000/metrics` | down | `server returned HTTP status 404 Not Found` |
| `comfyui` | `http://comfyui:8188/` | down | `received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:8081/` | down | `received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target` |
| `services` | `http://192.168.0.143:3005/` | down | same `unsupported Content-Type "text/html; charset=utf-8"` error |
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
The remedy for a probe-shaped question is the **blackbox exporter**; the
classification rule and the honesty law for it live in [[Stale Signal Audit]].
Until the exporter is deployed, these three stay red for a structural reason:
deleting them would
hide a real service, and fixing them by publishing databases or enabling metrics
would trade a wrong signal for a real exposure. Targets are `host:port`, never
URLs.
## Volumes
| Container path | Host path |
|----------------|-----------|
| `/etc/prometheus` | `/opt/prometheus/config` (`prometheus.container:51`) |
| `/prometheus` | `/opt/prometheus/data` (`prometheus.container:52`) |
## Healthcheck
Prometheus responds on `/-/healthy` with 200; the unit's health command is the same `wget --spider` the deploy waits on (`prometheus.container:53`, `scripts/deploy-remote.sh` — the `prometheus)` case of `deploy_observability` waits on `http://127.0.0.1:9090/-/healthy`).
## Gotchas
- **Keep the node-exporter address in step with the unit.** `node-dankclaw` scrapes `host.containers.internal:9100` (`prometheus.yml:37-39`), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; `node-exporter.container:39` binds `127.0.0.1:9100` + `192.168.0.143:9100`. Change one side without the other and host metrics stop with no error anywhere. The `tests/observability_quadlet.bats:108-131` seam pins the pairing in both directions.
- The container name `prometheus` is what Caddy's post-auth handler proxies to, and `grafana`/`loki` are addressed by name here. Renaming a container breaks both silently.
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Set a retention policy or the data volume grows without bound (30 days here — `--storage.tsdb.retention.time=30d`, `prometheus.container:49`).
- The lifecycle endpoint is enabled (`--web.enable-lifecycle`, `prometheus.container:49`), which is why this API must never be published beyond loopback.
## See Also
- [[Grafana]] — the dashboard frontend (datasource and LLM dashboard provisioned as code since 2026-09-16, commit `6a19143`, installed on the host by `scripts/deploy-remote.sh:286-300`)
- [[Node-Exporter]] — the host metrics source
- [[Observability]] — the whole stack, including what is still unconfigured
Reference in New Issue View Git Blame Copy Permalink