Files

T

Frank 49e8dfff66 docs: the vault and AGENTS.md describe two Dankclaw endpoints

halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.

AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.

Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.

2026-09-16 14:52:51 -07:00

11 KiB

Raw Blame History

type: reference title: Observability on Dankclaw description: The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy and node-exporter: what runs, how it is deployed, how it is exposed, and what is still unconfigured. tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] timestamp: 2026-09-11

Observability

The observability stack runs on Dankclaw: Prometheus for metrics, Loki for logs, Alloy as the shipper, Grafana for dashboards and node-exporter for host metrics. This is a small homelab — no Kubernetes.

Stack

graph LR
    S[Services] -->|/metrics| P[Prometheus]
    H[Host] -->|":9100"| P
    S -->|logs| A[Alloy]
    J[journald] --> A
    A --> L[Loki]
    P --> G[Grafana]
    L --> G

Component	Role	Image	Published
Prometheus	metrics collection	`prom/prometheus`	`127.0.0.1:9090`
Grafana	dashboards	`grafana/grafana`	`127.0.0.1:3002`
Loki	log storage & query	`franksdank/loki` (local build)	`127.0.0.1:3100`
Alloy	log shipping	`franksdank/alloy` (local build)	`127.0.0.1:12345`
Node-Exporter	host metrics	`prom/node-exporter`	`192.168.0.143:9100`, host netns

Current state: installed, healthy, and mostly unconfigured

Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:

Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither. shared/grafana/provisioning/ is installed to /opt/grafana/provisioning by the deployer and read by the container: the Prometheus datasource (http://prometheus:9090, fixed uid prometheus) and the LLM fleet — capacity and reuse dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers vllm:kv_cache_usage_perc for frank-medium, and GET /api/search?type=dash-db returns uid llm-fleet. One trap worth knowing: a provisioned dashboard does not appear in the legacy dashboard table on this Grafana version (it lives in unified storage behind dashboard.grafana.app), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
Grafana's dashboard now also surfaces firing alerts. The Alerts firing (ALERTS) panel graphs ALERTS{alertstate="firing"} for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
Prometheus now evaluates rules; nothing delivers them (2026-09-16). shared/prometheus/rules/llm-fleet.yml ships with the config (the deployer's copy_prometheus_config installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the job:llm_prefix_cache_reuse:rate5m recording rule the alert and the dashboard share. Verified with promtool check rules (SUCCESS, 6 rules) and /api/v1/rules (all six health=ok). There is still no Alertmanager and no contact point anywhere: a firing alert appears on Prometheus' /alerts, in the ALERTS metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
Loki is configured but unread. Storage is filesystem with 30-day retention, auth_enabled: false, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
node-exporter runs its default collector set — no collector tuning, no textfile collector for custom metrics.
Alloy is the one configured piece. It ships the host journal and podman container logs to Loki with job labels, and that pipeline is live.

The honest summary: metrics and logs are collected and retained, one dashboard visualizes the LLM fleet, six rules evaluate over it, and no rule reaches a person — there is no Alertmanager and no contact point, so correlation stays manual and every signal has to be looked at.

Deployment

Six units now, across two hosts. Dankclaw runs five (Prometheus, Grafana, Loki, Alloy, node-exporter) from dankclaw/observability/quadlet/; Dankserver runs its own node-exporter from dankserver/observability/quadlet/, added 2026-09-16 after the same reboot argument took it down a second time — it had been a compose-only service, died Exited (143) at a reboot, and left Prometheus' node-dankserver job scraping nothing for a day and a half (its compose form also bound the wildcard address). The deployer installs from "$TARGET/observability/quadlet", so each host owns its own units.

The Dankclaw five left dankclaw/compose.yaml on 2026-09-11, after a reboot proved that compose-only containers do not come back on their own — the same gap that had already taken Caddy and Pi-hole down at earlier reboots, and the one that finally caught Dankserver's node-exporter. What runs is now the unit; the compose file is the historical record.

./scripts/deploy.sh dankclaw --service loki          # or prometheus|grafana|alloy|node-exporter
./scripts/deploy.sh dankserver --service node-exporter

deploy_observability in scripts/deploy-remote.sh installs the unit, restarts it and waits on its endpoint. Ordering lives in the units themselves (Grafana wants Prometheus, Alloy wants Loki), and SERVICE_ORDER in the planner puts each dependency first.

Every unit carries a healthcheck, so a wedged process reads unhealthy in podman ps rather than merely Up. See Loki for the one unit that cannot declare it inline.

Exposure

The rule is the same as for every other service: the only off-host path is a guarded vhost (<name>.franksdank.cloud → Anubis → Authentik embedded outpost). Two consequences are easy to get wrong:

Prometheus, Grafana, Loki and Alloy publish on 127.0.0.1 only. Caddy reaches them by container name over the dankclaw_default network, and the host-side homepage status timer probes loopback. Publishing on 0.0.0.0 — what the compose services did — put an unauthenticated metrics API (with the lifecycle endpoint enabled), dashboard, log store and log-ingest endpoint on every host interface, which is a bypass around the guard chain.
node-exporter is the one deliberate LAN listener, and there are two of them now — one per host. Prometheus runs in a container and cannot reach either host's loopback, so each unit binds exactly two addresses: loopback (127.0.0.1:9100) for host-local probes and its own LAN address (192.168.0.143:9100 on Dankclaw, 192.168.0.235:9100 on Dankserver) for Prometheus. ss -ltn on Dankserver shows exactly those two listeners and no wildcard. Neither authenticates anything, so neither gets a vhost.

Scrape surface

shared/prometheus/prometheus.yml is the source of truth (installed to /opt/prometheus/config/ on every deploy). It scrapes:

Job	Target	Notes
`prometheus`	`localhost:9090`	itself
`node-dankclaw`, `node-dankserver`	`192.168.0.143:9100`, `192.168.0.235:9100`	one node-exporter per host
`gitea`	`192.168.0.143:3000`	published LAN port (Gitea is on an isolated network)
`grafana`, `loki`	`grafana:3000`, `loki:3100`	container names on `dankclaw_default`
`comfyui`	`comfyui:8188`	UI probe, no native exporter
`services`	`192.168.0.143:8081`, `:3005`	SearXNG, OpenWebUI up/down probes
`frank-medium`	`192.168.0.235:7770`	vLLM on Dankserver — 418 `vllm:` series
`frank-small`	`192.168.0.143:7771`	llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair

The LLM endpoints were added on 2026-09-16, after a night spent diagnosing frank-medium by hand, and re-cut the same day when Dankclaw split into two endpoints. vLLM exports exactly what that diagnosis needed: vllm:kv_cache_usage_perc (how full the per-card pool is), vllm:num_requests_waiting (queue depth), the request-latency histograms, and the prefix-cache counters vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give the reuse rate that says whether a pinned conversation is being served warm or re-prefilled cold. What those numbers mean on this hardware is in frank-medium-radiance-mxfp4; llama.cpp's counters (prompt and decode throughput, KV fill, and the spec_decode_* pair that proves MTP is running) are read against frank-small-qwen38-gsq-rco.

frank-large is deliberately NOT scraped. The halogen engine publishes /health, /cache and /v1/* and no Prometheus text endpoint at all, so a target there would be red by construction and would fire LlmEndpointDown forever — the same class of defect as the page-only probes below. Its liveness is on the homepage (a loopback probe) and in the engine's own boot ledger.

Both scraped endpoints are reached on their LAN addresses, never through the guarded vhosts: Prometheus cannot solve an Anubis challenge.

For services without a native exporter, use the Prometheus blackbox exporter in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.

Four targets read "down" — three jobs, and every one of them is structural (found 2026-09-16 while auditing why the target list was not green):

job	error	what it means
`comfyui`	`unsupported Content-Type "text/html"`	the service is up (this is what it says since ComfyUI was fixed — before that it read `connection refused`); Prometheus refuses to parse an HTML page as metrics, so this job cannot succeed as configured
`services` (SearXNG, OpenWebUI)	`unsupported Content-Type "text/html; charset=utf-8"`	same defect: "blackbox-style probes" that scrape a page instead of metrics
`gitea`	`HTTP status 404 Not Found`	`/metrics` does not exist on that Gitea (metrics disabled); the service itself is fine

None of these is an outage, and none should be silenced by deleting the job — an absent scrape and a healthy one look identical. The fix the repository already names is a blackbox exporter in probe mode for the page-only targets, and either enabling Gitea's metrics or pointing the job at the right path. The LLM alert rules (LlmEndpointDown) are scoped to frank-*, so these probes generate no false pages.

Log aggregation choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
Loki stores compressed logs with labels; queries run in Grafana's Explore.
No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain journald (no centralized query).

11 KiB Raw Blame History