dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(observability): record the quadlet cutover, the exposure posture, and the unconfigured state

Browse Source 
Rewrites the [[Observability]] umbrella page and the five component pages for
what is actually deployed: rootless Quadlet units instead of compose services,
the loopback-only publish posture (node-exporter's LAN binding explained), the
loki healthcheck living in its image because the image has no shell, and the
Home=/root trap that killed the first grafana start.

Adds the note that matters most to a reader: the stack is installed, healthy
and UNCONFIGURED. Grafana reports zero datasources and zero dashboards with an
empty provisioning directory; Prometheus has no rule files and no Alertmanager;
Loki is configured but has no reader; node-exporter runs its default
collectors. Alloy is the only configured piece. Metrics and logs are collected
and retained, and nothing is visualized or alerted on.

AGENTS.md's boot-persistence gap note is replaced by the resolution.
This commit is contained in:
Frank committed 2026-09-11 16:10:27 -07:00
1 parent 25c93984ad
commit 701ab14f60
8 files changed
+220 -147
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Infrastructure/Loki.md
+20 -17
View File
@@ -1,51 +1,54 @@
# Loki
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs, and it feeds the log panels in Grafana. It replaces the need for a full ELK stack.
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs. It replaces the need for a full ELK stack.
**Host:** Dankclaw — **Docs:** https://grafana.com/docs/loki/latest
**Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest
## Facts (from research)
## Facts
| Item | Value |
|------|-------|
| Image | `franksdank/loki:latest` (local build with a static busybox healthcheck) |
| Port | 3100 (host) → 3100 (container) |
| Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) |
| Published | `127.0.0.1:3100` → 3100, loopback only |
| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service` |
| Mode | single-binary (all components in one process) |
| Storage | filesystem (bind mount) |
| Storage | filesystem, `/opt/loki/data` |
| Auth | `auth_enabled: false` (single-tenant homelab) |
| Schema | v13 / TSDB |
| Retention | 30 days, enforced by the compactor |
## Configuration
Config lives at `shared/loki/loki-config.yaml`. Key settings:
`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy. Key settings:
- `schema_config` v13 with `store: tsdb` and filesystem object store.
- `schema_config` v13 with `store: tsdb` and a filesystem object store.
- `compactor` with `retention_enabled: true` so logs do not grow without bound.
- `replication_factor: 1` for a single instance.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| Container path | Host path |
|----------------|-----------|
| `/loki` | `/opt/loki/data` |
| `/etc/loki/loki-config.yaml` | `shared/loki/loki-config.yaml` (bind mount) |
| `/etc/loki` | `/opt/loki/config` |
## Healthcheck
Loki responds on `/ready` with 200.
Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile` as an exec-form `HEALTHCHECK` instead (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it).
## Log Shipping
The [[Alloy]] agent on each host ships container and journal logs to Loki. Grafana queries Loki through the datasource in [[Observability]].
The [[Alloy]] agent on Dankclaw ships container and journal logs to `http://loki:3100/loki/api/v1/push` — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge).
## Gotchas
- Filesystem storage grows; enable compactor retention.
- Loki and Promtail/Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
- For multi-host shipping, Loki's HTTP listener binds `0.0.0.0`.
- Filesystem storage grows; keep compactor retention on.
- Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
- The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
- Nothing queries Loki yet: Grafana has no datasource configured. See the state note in [[Observability]].
## See Also
- [[Observability]]
- [[Alloy]] — the log shipper
- [[Grafana]] — the query frontend
- [[Observability]] — the whole stack