Files

T

Frank 701ab14f60 docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.

2026-09-11 16:10:27 -07:00

2.6 KiB

Raw Blame History

Loki

Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs. It replaces the need for a full ELK stack.

Host: Dankclaw — Port: 3100 (loopback) — Docs: https://grafana.com/docs/loki/latest

Facts

Item	Value
Image	`franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck)
Published	`127.0.0.1:3100` → 3100, loopback only
Unit	`dankclaw/observability/quadlet/loki.container` → `loki.service`
Mode	single-binary (all components in one process)
Storage	filesystem, `/opt/loki/data`
Auth	`auth_enabled: false` (single-tenant homelab)
Schema	v13 / TSDB
Retention	30 days, enforced by the compactor

Configuration

shared/loki/loki-config.yaml is the source of truth, installed to /opt/loki/config/loki-config.yaml on every deploy. Key settings:

schema_config v13 with store: tsdb and a filesystem object store.
compactor with retention_enabled: true so logs do not grow without bound.
replication_factor: 1 for a single instance.

Volumes

Container path	Host path
`/loki`	`/opt/loki/data`
`/etc/loki`	`/opt/loki/config`

Healthcheck

Loki responds on /ready with 200. The health command is not in the unit: a unit HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and this image is distroless with no shell at all. It ships in dankclaw/loki/Containerfile as an exec-form HEALTHCHECK instead (which is why the image carries busybox, and why the build must pass --format docker — the OCI image format has no HEALTHCHECK field and silently drops it).

Log Shipping

The Alloy agent on Dankclaw ships container and journal logs to http://loki:3100/loki/api/v1/push — by container name on the dankclaw_default network, never through the vhost (a non-browser client cannot solve the Anubis challenge).

Gotchas

Filesystem storage grows; keep compactor retention on.
Loki and Alloy must agree on the push URL: http://loki:3100/loki/api/v1/push.
The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
Nothing queries Loki yet: Grafana has no datasource configured. See the state note in Observability.

2.6 KiB Raw Blame History