Files

T

Frank 5829a0bfe8 docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention

All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.

2026-08-28 02:20:06 -07:00

1.9 KiB

Raw Blame History

Observability

The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.

Stack

graph LR
    S[Services] -->|/metrics| P[Prometheus]
    S -->|logs| T[Alloy]
    T --> L[Loki]
    P --> G[Grafana]
    L --> G

Component	Role	Image	Port
Prometheus	metrics collection	`prom/prometheus`	9090
Grafana	dashboards	`grafana/grafana`	3002
Loki	log storage & query	`grafana/loki`	3100
Alloy	log shipping (per host)	`grafana/alloy`	—

Log Aggregation Choice

Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:

Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
Loki stores compressed logs with labels; queries run in Grafana's Explore.
No Elasticsearch memory cost, no Kubernetes required.

Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain journald (no centralized query).

Scrape Surface

Target	Source
Prometheus self	`localhost:9090/metrics`
ComfyUI	`/system_stats` (limited) or blackbox probe
Gitea	`/metrics` (built-in exporter)
Grafana	`/metrics` (built-in)
Host metrics	node-exporter `:9100`

For services without native exporters, use the Prometheus blackbox exporter in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.

Healthchecks

Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See Deployment Flow for the verify step.

1.9 KiB Raw Blame History