Frank
327e78b950
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
80 lines
5.7 KiB
Markdown
80 lines
5.7 KiB
Markdown
---
|
|
type: reference
|
|
description: "Loki is the log aggregation backend of the observability stack."
|
|
tags:
|
|
- infrastructure
|
|
- loki
|
|
timestamp: 2026-09-11
|
|
---
|
|
|
|
# Loki
|
|
|
|
Loki is the log aggregation backend of the observability stack. It is configured to store and query container and system logs, replacing the need for a full ELK stack — and as measured on 2026-10-18 the store is **empty**: the service is healthy and queryable, and it holds zero streams. See "Contents" below.
|
|
|
|
**Host:** Dankclaw — **Port:** 3100 (loopback) — **Docs:** https://grafana.com/docs/loki/latest
|
|
|
|
## Facts
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Image | `franksdank/loki:latest` (local build: upstream + a static busybox for the healthcheck) |
|
|
| Published | `127.0.0.1:3100` → 3100, loopback only (`dankclaw/observability/quadlet/loki.container:43`; `ss -ltnp` 2026-10-18 shows no other bind) |
|
|
| Unit | `dankclaw/observability/quadlet/loki.container` → `loki.service`; `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`loki.container:52-57`), installed at `~/.config/containers/systemd/loki.container`, reporting `generated` + `active` (`systemctl --user is-enabled|is-active loki` 2026-10-18) |
|
|
| Mode | single-binary (all components in one process — the `Exec` line passes no `-target`, `loki.container:42`) |
|
|
| Storage | filesystem, `/opt/loki/data` → `/loki` (`loki.container:45`; `loki-config.yaml:10-16`) |
|
|
| Auth | `auth_enabled: false` (single-tenant homelab, `loki-config.yaml:4`) |
|
|
| Schema | v13 / TSDB (`loki-config.yaml:22-30`) |
|
|
| Retention | 30 days, enforced by the compactor (`shared/loki/loki-config.yaml:33` `retention_period: 720h`, `:37-44` `retention_enabled: true`) |
|
|
| Contents (2026-10-18) | **zero streams** — see "Contents" |
|
|
|
|
## Configuration
|
|
|
|
`shared/loki/loki-config.yaml` is the source of truth, installed to `/opt/loki/config/loki-config.yaml` on every deploy (`scripts/deploy-remote.sh:276-279`). Key settings:
|
|
|
|
- `schema_config` v13 with `store: tsdb` and a filesystem object store (`loki-config.yaml:22-30`).
|
|
- `compactor` with `retention_enabled: true` so logs do not grow without bound (`loki-config.yaml:37-44`).
|
|
- `replication_factor: 1` for a single instance (`loki-config.yaml:16`).
|
|
|
|
## Volumes
|
|
|
|
| Container path | Host path |
|
|
|----------------|-----------|
|
|
| `/loki` | `/opt/loki/data` (`loki.container:45`) |
|
|
| `/etc/loki` | `/opt/loki/config` (`loki.container:44`) |
|
|
|
|
## Healthcheck
|
|
|
|
Loki responds on `/ready` with 200. The health command is **not** in the unit: a unit `HealthCmd` reaches podman as `CMD-SHELL`, i.e. `/bin/sh -c`, and this image is distroless with no shell at all. It ships in `dankclaw/loki/Containerfile:29-30` as an exec-form `HEALTHCHECK` instead, probing `http://localhost:3100/ready` (which is why the image carries busybox, and why the build must pass `--format docker` — the OCI image format has no `HEALTHCHECK` field and silently drops it: `dankclaw/loki/Containerfile:13-17`, `scripts/deploy-remote.sh:558-562`). The deployer waits on the same endpoint through the published loopback, `curl http://127.0.0.1:3100/ready` (`deploy-remote.sh`, the `loki)` case of `deploy_observability`), and `podman ps` reports the container `healthy` (2026-10-18).
|
|
|
|
## Log Shipping
|
|
|
|
The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to `http://loki:3100/loki/api/v1/push` (`shared/alloy/config.alloy:13-17`) — by container name on the `dankclaw_default` network, never through the vhost (a non-browser client cannot solve the Anubis challenge). **Nothing arrives today:** Alloy's own counters read zero lines read and zero files tailed, with both file globs refused at start (`journalctl --user -u alloy` 2026-10-18); see [[Alloy]] → "Delivery state".
|
|
|
|
## Contents (2026-10-18): healthy, empty
|
|
|
|
The service answers and the API works; there is simply no data in it. Measured on 2026-10-18:
|
|
|
|
- `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`.
|
|
- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`
|
|
`chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) —
|
|
`find /opt/loki/data -type f` counted **8 files**, all WAL segments,
|
|
cluster-seed and delete-request bookkeeping, **no chunk objects**
|
|
(2026-10-18; the byte total drifts as the WAL rolls, so it is not a
|
|
useful invariant — the file census is).
|
|
- Prometheus-side health is green: the `loki` job scrapes `loki:3100/metrics` and reports up (`shared/prometheus/prometheus.yml:57-61`), and the homepage card probes `127.0.0.1:3100` (`shared/homepage/services.json:207`).
|
|
|
|
So "30-day retention" is a policy that has never had anything to delete, and no page should describe log history as available. The cause is upstream of Loki — the shipper ships nothing — and no fix is claimed here; the check that matters is a label query, not `/ready`.
|
|
|
|
## Gotchas
|
|
|
|
- Filesystem storage grows; keep compactor retention on.
|
|
- Loki and Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
|
|
- The listener is loopback-only now. Shipping logs from a second host would need the unit's publish changed (or a shared network), not just an Alloy install — Loki has no authentication of its own, so that decision is a security one, not a config one.
|
|
- Nothing queries Loki: Grafana's only provisioned datasource is Prometheus (`shared/grafana/provisioning/datasources/prometheus.yml:14-21`, uid `prometheus`), and there is no Loki datasource file in that tree. Adding one is a YAML file, not a UI click — see [[Grafana]] → Datasource.
|
|
|
|
## See Also
|
|
|
|
- [[Alloy]] — the log shipper
|
|
- [[Grafana]] — the query frontend
|
|
- [[Observability]] — the whole stack
|