Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

8.3 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Prometheus

Prometheus collects metrics from the services listed below by scraping their /metrics endpoints, and it is the datasource behind the provisioned llm-fleet dashboard in Grafana (datasource uid prometheus, shared/grafana/provisioning/datasources/prometheus.yml:14-21). node-exporter supplies host metrics from both machines.

Host: Dankclaw — Vhost: prometheus.franksdank.cloud (guarded: Anubis → Authentik; post-auth the Caddy vhost routes straight to prometheus:9090) — Port: 9090 (loopback) — Docs: https://prometheus.io/docs

Facts

Item	Value
Image	`docker.io/prom/prometheus:latest`
Published	`127.0.0.1:9090` → 9090, loopback only — never a wildcard (`dankclaw/observability/quadlet/prometheus.container:50`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9090` and no other bind; `tests/observability_quadlet.bats:63` pins it)
Unit	`dankclaw/observability/quadlet/prometheus.container` → `prometheus.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`prometheus.container:59-64`), installed at `~/.config/containers/systemd/prometheus.container`, reporting `generated` + `active` (`systemctl --user is-enabled
Config	`/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`)
Data	`/opt/prometheus/data` (TSDB, `prometheus.container:52`)
Retention	`--storage.tsdb.retention.time=30d` (`prometheus.container:49`)
Alerting	4 alert rules + 2 recording rules (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-."} == 0`, so only the `frank-` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family (Stale Signal Audit). Evaluated and delivered since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see Alertmanager)

Targets

shared/prometheus/prometheus.yml is the source of truth; the deploy copies it into the container's /etc/prometheus/. In outline:

Job	Target	Why
`prometheus`	`localhost:9090`	itself
`node-dankclaw`	`host.containers.internal:9100`	host metrics from Node-Exporter — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container)
`node-dankserver`	`192.168.0.235:9100`	host metrics from Node-Exporter — a different machine, so the LAN address works
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on the `dankclaw_default` network; both serve real `/metrics`
`frank-archon`	`192.168.0.235:7770`	the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on frank-archon
`frank-dragoon`, `frank-zealot`	`host.containers.internal:7770`, `:7771`	the LLM fleet. They are addressed through the gateway because they are same-host — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on frank-dragoon and frank-zealot

The frank-medium job is retired: no frank-medium job exists in the config or in /api/v1/targets (2026-10-18), and the endpoint itself left the fleet — its history is on frank-archon, which holds the port today.

Three jobs read down today, and they are not outages (re-audited against curl -s http://127.0.0.1:9090/api/v1/targets, 2026-10-18; first audited 2026-09-16):

Job	Target	`health`	`lastError`, verbatim
`gitea`	`http://192.168.0.143:3000/metrics`	down	`server returned HTTP status 404 Not Found`
`comfyui`	`http://comfyui:8188/`	down	`received unsupported Content-Type "text/html" and no fallback_scrape_protocol specified for target`
`services`	`http://192.168.0.143:8081/`	down	`received unsupported Content-Type "text/html; charset=utf-8" and no fallback_scrape_protocol specified for target`
`services`	`http://192.168.0.143:3005/`	down	same `unsupported Content-Type "text/html; charset=utf-8"` error

Gitea's exporter stays off until [metrics] ENABLED = true in its own config (no metrics env is set in the container, and curl -s http://127.0.0.1:3000/metrics → 404, 2026-10-18); ComfyUI and the two services probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: prometheus, node-dankclaw, node-dankserver, grafana, loki, frank-zealot, frank-dragoon, frank-archon.

frank-zealot reads up since 2026-10-18, and did not before. llama.cpp only answers /metrics when started with --metrics, so up{job="frank-zealot"} was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered /health 200 the whole time; the flag and its opt-out are on frank-zealot. Read lastError before believing an outage: a target can read down beside a healthy service.

Alert state is a live property, not a page fact: curl -s http://127.0.0.1:9090/api/v1/alerts returned no alerts at the 2026-10-18 morning census, one firing LlmPrefixCacheCollapse (info, frank-zealot, activeAt 11:10:48Z) minutes later, and none again by 11:56Z.

The remedy for a probe-shaped question is the blackbox exporter; the classification rule and the honesty law for it live in Stale Signal Audit. Until the exporter is deployed, these three stay red for a structural reason: deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are host:port, never URLs.

Volumes

Container path	Host path
`/etc/prometheus`	`/opt/prometheus/config` (`prometheus.container:51`)
`/prometheus`	`/opt/prometheus/data` (`prometheus.container:52`)

Healthcheck

Prometheus responds on /-/healthy with 200; the unit's health command is the same wget --spider the deploy waits on (prometheus.container:53, scripts/deploy-remote.sh — the prometheus) case of deploy_observability waits on http://127.0.0.1:9090/-/healthy).

Gotchas

Keep the node-exporter address in step with the unit. node-dankclaw scrapes host.containers.internal:9100 (prometheus.yml:37-39), because this Prometheus is a bridge container and a host-netns bind is unreachable at the host's LAN address from inside it; node-exporter.container:39 binds 127.0.0.1:9100 + 192.168.0.143:9100. Change one side without the other and host metrics stop with no error anywhere. The tests/observability_quadlet.bats:108-131 seam pins the pairing in both directions.
The container name prometheus is what Caddy's post-auth handler proxies to, and grafana/loki are addressed by name here. Renaming a container breaks both silently.
Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
Set a retention policy or the data volume grows without bound (30 days here — --storage.tsdb.retention.time=30d, prometheus.container:49).
The lifecycle endpoint is enabled (--web.enable-lifecycle, prometheus.container:49), which is why this API must never be published beyond loopback.

8.3 KiB Raw Blame History