Frank
e7f0d93c86
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
80 lines
5.7 KiB
Markdown
80 lines
5.7 KiB
Markdown
---
|
|
type: reference
|
|
description: "Loki is the log aggregation backend of the observability stack."
|
|
tags:
|
|
- infrastructure
|
|
- loki
|
|
timestamp: 2026-09-11
|
|
---
|
|
|
|
# Loki
|
|
|
|
Loki is the log aggregation backend of the observability stack. It is configured to store and query container and system logs, replacing the need for a full ELK stack — and as measured on 2026-10-18 the store is **empty**: the service is healthy and queryable, and it holds zero streams. See "Contents" below.
|
|
|
|
**Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) |
|
|
| Published | `127.0.0.1:3100` → 3100, loopback only (`dankclaw/observability/quadlet/loki.container:43`; `ss -ltnp` 2026-10-18 shows no other bind) |
|
|
| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`loki.container:52-57`), installed at `~/.config/containers/systemd/loki.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active loki` 2026-10-18) |
|
|
| Mode | single-binary (all components in one process — the `Exec` line passes no `-target`, `loki.container:42`) |
|
|
| Storage | filesystem, `/opt/loki/data` → `/loki` (`loki.container:45`; `loki-config.yaml:10-16`) |
|
|
| Auth | `auth_enabled: false` (single-tenant homelab, `loki-config.yaml:4`) |
|
|
| Schema | v13 / TSDB (`loki-config.yaml:22-30`) |
|
|
| Retention | 30 days, enforced by the compactor (`shared/loki/loki-config.yaml:33` `retention_period: 720h`, `:37-44` `retention_enabled: true`) |
|
|
| Contents (2026-10-18) | **zero streams** — see "Contents" |
|
|
|
|
## Configuration
|
|
|
|
`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy (`scripts/deploy-remote.sh:276-279`). Key settings:
|
|
|
|
- `schema_config` v13 with `store: tsdb` and a filesystem object store (`loki-config.yaml:22-30`).
|
|
- `compactor` with `retention_enabled: true` so logs do not grow without bound (`loki-config.yaml:37-44`).
|
|
- `replication_factor: 1` for a single instance (`loki-config.yaml:16`).
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path |
|
|
|----------------|-----------|
|
|
| `/loki` | `/opt/loki/data` (`loki.container:45`) |
|
|
| `/etc/loki` | `/opt/loki/config` (`loki.container:44`) |
|
|
|
|
## Healthcheck
|
|
|
|
Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile:29-30` as an exec-form `HEALTHCHECK` instead, probing `http://localhost:3100/ready` (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it: `dankclaw/loki/Containerfile:13-17`, `scripts/deploy-remote.sh:558-562`). The deployer waits on the same endpoint through the published loopback, `curl http://127.0.0.1:3100/ready` (`deploy-remote.sh`, the `loki)` case of `deploy_observability`), and `podman ps` reports the container `healthy` (2026-10-18).
|
|
|
|
## Log Shipping
|
|
|
|
The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge). **Nothing arrives today:** Alloy's own counters read zero lines read and zero files tailed, with both file globs refused at start (`journalctl --user -u alloy` 2026-10-18); see [[Alloy]] → "Delivery state".
|
|
|
|
## Contents (2026-10-18): healthy, empty
|
|
|
|
The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
|
|
|
|
- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
|
|
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`,
|
|
`chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
|
|
`find /opt/loki/data -type f` counted **8 files**, all WAL segments,
|
|
cluster-seed and delete-request bookkeeping, **no chunk objects**
|
|
(2026-10-18; the byte total drifts as the WAL rolls, so it is not a
|
|
useful invariant — the file census is).
|
|
- Prometheus-side health is green: the `loki` job scrapes `loki:3100/metrics` and reports up (`shared/prometheus/prometheus.yml:57-61`), and the homepage card probes `127.0.0.1:3100` (`shared/homepage/services.json:207`).
|
|
|
|
So "30-day retention" is a policy that has never had anything to delete, and no page should describe log history as available. The cause is upstream of Loki — the shipper ships nothing — and no fix is claimed here; the check that matters is a label query, not `/ready`.
|
|
|
|
## Gotchas
|
|
|
|
- Filesystem storage grows; keep compactor retention on.
|
|
- Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
|
|
- The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
|
|
- Nothing queries Loki: Grafana's only provisioned datasource is Prometheus (`shared/grafana/provisioning/datasources/prometheus.yml:14-21`, uid `prometheus`), and there is no Loki datasource file in that tree. Adding one is a YAML file, not a UI click — see [[Grafana]] → Datasource.
|
|
|
|
## See Also
|
|
|
|
- [[Alloy]] — the log shipper
|
|
- [[Grafana]] — the query frontend
|
|
- [[Observability]] — the whole stack
|