Files

T

Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.

2026-10-18 13:15:43 -07:00

5.7 KiB

Raw Permalink Blame History

type, description, tags, timestamp

type

description

Loki

Loki is the log aggregation backend of the observability stack. It is configured to store and query container and system logs, replacing the need for a full ELK stack — and as measured on 2026-10-18 the store is empty: the service is healthy and queryable, and it holds zero streams. See "Contents" below.

Host: Dankclaw — Port: 3100 (loopback) — Docs: https://grafana.com/docs/loki/latest

Facts

Item	Value
Image	`franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck)
Published	`127.0.0.1:3100` → 3100, loopback only (`dankclaw/observability/quadlet/loki.container:43`; `ss -ltnp` 2026-10-18 shows no other bind)
Unit	`dankclaw/observability/quadlet/loki.container` → `loki.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`loki.container:52-57`), installed at `~/.config/containers/systemd/loki.container`, reporting `generated` + `active` (`systemctl --user is-enabled
Mode	single-binary (all components in one process — the `Exec` line passes no `-target`, `loki.container:42`)
Storage	filesystem, `/opt/loki/data` → `/loki` (`loki.container:45`; `loki-config.yaml:10-16`)
Auth	`auth_enabled: false` (single-tenant homelab, `loki-config.yaml:4`)
Schema	v13 / TSDB (`loki-config.yaml:22-30`)
Retention	30 days, enforced by the compactor (`shared/loki/loki-config.yaml:33` `retention_period: 720h`, `:37-44` `retention_enabled: true`)
Contents (2026-10-18)	zero streams — see "Contents"

Configuration

shared/loki/loki-config.yaml is the source of truth, installed to /opt/loki/config/loki-config.yaml on every deploy (scripts/deploy-remote.sh:276-279). Key settings:

schema_config v13 with store: tsdb and a filesystem object store (loki-config.yaml:22-30).
compactor with retention_enabled: true so logs do not grow without bound (loki-config.yaml:37-44).
replication_factor: 1 for a single instance (loki-config.yaml:16).

Volumes

Container path	Host path
`/loki`	`/opt/loki/data` (`loki.container:45`)
`/etc/loki`	`/opt/loki/config` (`loki.container:44`)

Healthcheck

Loki responds on /ready with 200. The health command is not in the unit: a unit HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and this image is distroless with no shell at all. It ships in dankclaw/loki/Containerfile:29-30 as an exec-form HEALTHCHECK instead, probing http://localhost:3100/ready (which is why the image carries busybox, and why the build must pass --format docker — the OCI image format has no HEALTHCHECK field and silently drops it: dankclaw/loki/Containerfile:13-17, scripts/deploy-remote.sh:558-562). The deployer waits on the same endpoint through the published loopback, curl http://127.0.0.1:3100/ready (deploy-remote.sh, the loki) case of deploy_observability), and podman ps reports the container healthy (2026-10-18).

Log Shipping

The Alloy agent on Dankclaw is wired to ship container and journal logs to http://loki:3100/loki/api/v1/push (shared/alloy/config.alloy:13-17) — by container name on the dankclaw_default network, never through the vhost (a non-browser client cannot solve the Anubis challenge). Nothing arrives today: Alloy's own counters read zero lines read and zero files tailed, with both file globs refused at start (journalctl --user -u alloy 2026-10-18); see Alloy → "Delivery state".

Contents (2026-10-18): healthy, empty

The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:

curl -s http://127.0.0.1:3100/loki/api/v1/labels → {"status":"success"} with no data key (no label names exist), and a 7-day query_range on {job="systemd-journal"} returns "result":[].
du -sh /opt/loki/data → a few hundred KB of directory scaffolding (wal, chunks, compactor, tsdb-shipper-active, tsdb-shipper-cache) — find /opt/loki/data -type f counted 8 files, all WAL segments, cluster-seed and delete-request bookkeeping, no chunk objects (2026-10-18; the byte total drifts as the WAL rolls, so it is not a useful invariant — the file census is).
Prometheus-side health is green: the loki job scrapes loki:3100/metrics and reports up (shared/prometheus/prometheus.yml:57-61), and the homepage card probes 127.0.0.1:3100 (shared/homepage/services.json:207).

So "30-day retention" is a policy that has never had anything to delete, and no page should describe log history as available. The cause is upstream of Loki — the shipper ships nothing — and no fix is claimed here; the check that matters is a label query, not /ready.

Gotchas

Filesystem storage grows; keep compactor retention on.
Loki and Alloy must agree on the push URL: http://loki:3100/loki/api/v1/push.
The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
Nothing queries Loki: Grafana's only provisioned datasource is Prometheus (shared/grafana/provisioning/datasources/prometheus.yml:14-21, uid prometheus), and there is no Loki datasource file in that tree. Adding one is a YAML file, not a UI click — see Grafana → Datasource.

5.7 KiB Raw Permalink Blame History