---
type: reference
description: "Loki is the log aggregation backend of the observability stack."
tags:
  - infrastructure
  - loki
timestamp: 2026-09-11
---

# Loki

Loki is the log aggregation backend of the observability stack. It is configured to store and query container and system logs, replacing the need for a full ELK stack — and as measured on 2026-10-18 the store is **empty**: the service is healthy and queryable, and it holds zero streams. See "Contents" below.

**Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest

## Facts

| Item | Value |
|------|-------|
| Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) |
| Published | `127.0.0.1:3100` → 3100, loopback only (`dankclaw/observability/quadlet/loki.container:43`; `ss -ltnp` 2026-10-18 shows no other bind) |
| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`loki.container:52-57`), installed at `~/.config/containers/systemd/loki.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active loki` 2026-10-18) |
| Mode | single-binary (all components in one process — the `Exec` line passes no `-target`, `loki.container:42`) |
| Storage | filesystem, `/opt/loki/data` → `/loki` (`loki.container:45`; `loki-config.yaml:10-16`) |
| Auth | `auth_enabled: false` (single-tenant homelab, `loki-config.yaml:4`) |
| Schema | v13 / TSDB (`loki-config.yaml:22-30`) |
| Retention | 30 days, enforced by the compactor (`shared/loki/loki-config.yaml:33` `retention_period: 720h`, `:37-44` `retention_enabled: true`) |
| Contents (2026-10-18) | **zero streams** — see "Contents" |

## Configuration

`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy (`scripts/deploy-remote.sh:276-279`). Key settings:

- `schema_config` v13 with `store: tsdb` and a filesystem object store (`loki-config.yaml:22-30`).
- `compactor` with `retention_enabled: true` so logs do not grow without bound (`loki-config.yaml:37-44`).
- `replication_factor: 1` for a single instance (`loki-config.yaml:16`).

## Volumes

| Container path | Host path |
|----------------|-----------|
| `/loki` | `/opt/loki/data` (`loki.container:45`) |
| `/etc/loki` | `/opt/loki/config` (`loki.container:44`) |

## Healthcheck

Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile:29-30` as an exec-form `HEALTHCHECK` instead, probing `http://localhost:3100/ready` (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it: `dankclaw/loki/Containerfile:13-17`, `scripts/deploy-remote.sh:558-562`). The deployer waits on the same endpoint through the published loopback, `curl http://127.0.0.1:3100/ready` (`deploy-remote.sh`, the `loki)` case of `deploy_observability`), and `podman ps` reports the container `healthy` (2026-10-18).

## Log Shipping

The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge). **Nothing arrives today:** Alloy's own counters read zero lines read and zero files tailed, with both file globs refused at start (`journalctl --user -u alloy` 2026-10-18); see [[Alloy]] → "Delivery state".

## Contents (2026-10-18): healthy, empty

The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:

- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`
  `chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
  `find /opt/loki/data -type f` counted **8 files**, all WAL segments,
  cluster-seed and delete-request bookkeeping, **no chunk objects**
  (2026-10-18; the byte total drifts as the WAL rolls, so it is not a
  useful invariant — the file census is).
- Prometheus-side health is green: the `loki` job scrapes `loki:3100/metrics` and reports up (`shared/prometheus/prometheus.yml:57-61`), and the homepage card probes `127.0.0.1:3100` (`shared/homepage/services.json:207`).

So "30-day retention" is a policy that has never had anything to delete, and no page should describe log history as available. The cause is upstream of Loki — the shipper ships nothing — and no fix is claimed here; the check that matters is a label query, not `/ready`.

## Gotchas

- Filesystem storage grows; keep compactor retention on.
- Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
- The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
- Nothing queries Loki: Grafana's only provisioned datasource is Prometheus (`shared/grafana/provisioning/datasources/prometheus.yml:14-21`, uid `prometheus`), and there is no Loki datasource file in that tree. Adding one is a YAML file, not a UI click — see [[Grafana]] → Datasource.

## See Also

- [[Alloy]] — the log shipper
- [[Grafana]] — the query frontend
- [[Observability]] — the whole stack