docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
This commit is contained in:
Frank committed
1 parent
6d4a96a423
commit
5829a0bfe8
140 files changed
+1820
-86
No files matched your search
@@ -0,0 +1,47 @@
|
||||
# Alloy
|
||||
|
||||
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). One Alloy instance runs per host and forwards container and journal logs to Loki, and optionally metrics to Prometheus.
|
||||
|
||||
**Hosts:** Dankclaw + Dankserver (one per host) — **Docs:** https://grafana.com/docs/alloy/latest
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `franksdank/alloy:latest` (local build with a static busybox healthcheck) |
|
||||
| Port | none published (push-only) |
|
||||
| Config | River syntax, `config.alloy` |
|
||||
| Successor to | Promtail (EOL 2026-03-02) |
|
||||
| Inputs | journald socket, container log files |
|
||||
| Output | Loki push endpoint |
|
||||
|
||||
## Why Alloy and not Promtail
|
||||
|
||||
Promtail reached end-of-life on 2026-03-02. Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
|
||||
|
||||
## Configuration
|
||||
|
||||
River config at `shared/alloy/config.alloy`:
|
||||
|
||||
- Read journald logs from the host's `/run/systemd/journal/socket`.
|
||||
- Read container logs from `/var/lib/containers/...` (podman) or the docker path.
|
||||
- Forward to `http://loki:3100/loki/api/v1/push` with `host` and `job` labels.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/etc/alloy/config.alloy` | `shared/alloy/config.alloy` (bind mount) |
|
||||
| `/run/systemd/journal/socket` | host journald socket (read-only) |
|
||||
| `/var/log` | host logs (read-only) |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The journald socket mount is required for journal scraping.
|
||||
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
|
||||
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Loki]] — the log backend
|
||||
- [[Observability]] — the full stack
|
||||
@@ -0,0 +1,152 @@
|
||||
# Authentik — the homelab's SSO / OIDC provider
|
||||
|
||||
Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
|
||||
homelab. It fronts the Caddy vhosts that do not carry their own
|
||||
application-level authentication. Apps that have their own login (Grafana,
|
||||
Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped in
|
||||
forward_auth — they handle their own session.
|
||||
|
||||
- **Version:** `2026.8.0` (pinned in `dankclaw/compose.yaml`)
|
||||
- **Host:** dankclaw (`192.168.0.143`)
|
||||
- **Name:** `auth.franksdank.cloud`
|
||||
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
|
||||
`authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)
|
||||
|
||||
## Why Authentik
|
||||
|
||||
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
|
||||
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
|
||||
for all configuration, a native Caddy `forward_auth` outpost, and a footprint
|
||||
that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10,
|
||||
so the stack is 3 services, not 4.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Browser
|
||||
U[User]
|
||||
end
|
||||
subgraph Caddy [Caddy - auth gate]
|
||||
FA[forward_auth]
|
||||
end
|
||||
subgraph AK [Authentik]
|
||||
SRV[server :9000]
|
||||
WKR[worker]
|
||||
PG[(postgres)]
|
||||
OUT[outpost /auth/caddy]
|
||||
end
|
||||
subgraph App [Target app]
|
||||
A1[SearXNG / Hermes / Prometheus<br/>RomM / Copyparty / homepage]
|
||||
end
|
||||
|
||||
U -->|1. GET /| FA
|
||||
FA -->|2. GET /outpost.../auth/caddy| OUT
|
||||
OUT -->|3. no session: 302 login| U
|
||||
U -->|4. login at auth.franksdank.cloud| SRV
|
||||
SRV --> PG
|
||||
WKR --> PG
|
||||
SRV -->|5. 2xx: session valid| FA
|
||||
FA -->|6. proxy with headers| A1
|
||||
```
|
||||
|
||||
1. The browser hits a protected vhost (e.g. `search.franksdank.cloud`).
|
||||
2. Caddy `forward_auth` clones the request to the Authentik outpost
|
||||
(`/outpost.goauthentik.io/auth/caddy`).
|
||||
3. With no valid session, the outpost returns a `302` to the Authentik login
|
||||
page (`auth.franksdank.cloud`).
|
||||
4. The user logs in at Authentik (email + password, TOTP/WebAuthn MFA, or an
|
||||
upstream provider).
|
||||
5. On the next request the outpost returns `2xx`; Caddy copies the
|
||||
`X-Authentik-*` headers and proxies to the app.
|
||||
|
||||
## The three compose services
|
||||
|
||||
| Service | Role | Image |
|
||||
|---|---|---|
|
||||
| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
|
||||
| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.0` |
|
||||
| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.0` |
|
||||
|
||||
- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
|
||||
moved to PostgreSQL in 2025.10 (Redis removed).
|
||||
- **Server** and **worker** run the same image; the `command` selects the role
|
||||
(`server` / `worker`).
|
||||
- The worker runs as `0:0` (rootless: container root maps to the host user that
|
||||
owns the data dir).
|
||||
- Data persists under `/home/dank/authentik/` (bind mounts, repo policy — no
|
||||
named volumes).
|
||||
|
||||
## Caddy integration
|
||||
|
||||
The Caddyfile wraps the no-auth vhosts in a `route` block. The pattern (one
|
||||
example; the others are identical):
|
||||
|
||||
```caddyfile
|
||||
search.franksdank.cloud {
|
||||
route {
|
||||
# 1. The outpost path is proxied to Authentik (login UI + session).
|
||||
reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
|
||||
# 2. forward_auth checks the session before serving the app.
|
||||
forward_auth authentik-server:9000 {
|
||||
uri /outpost.goauthentik.io/auth/caddy
|
||||
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
|
||||
trusted_proxies private_ranges
|
||||
}
|
||||
# 3. The app.
|
||||
reverse_proxy 192.168.0.143:8081
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Vhosts protected by forward_auth: the homepage (`franksdank.cloud`,
|
||||
`www.franksdank.cloud`), SearXNG, Hermes, Prometheus, RomM, Copyparty. The
|
||||
`auth.franksdank.cloud` vhost proxies to the server directly (no forward_auth —
|
||||
it *is* the login).
|
||||
|
||||
## Configuration (secrets)
|
||||
|
||||
Two secrets live in the host `.env` (git-ignored; placeholders in
|
||||
`.env.example`):
|
||||
|
||||
- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
|
||||
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
|
||||
|
||||
## First-time setup
|
||||
|
||||
1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
|
||||
health-gated), reloads Caddy.
|
||||
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
|
||||
creates the superuser (email + password) and generates the DB schema.
|
||||
3. In the admin interface: **Directory → Applications** → add a **Proxy**
|
||||
application (or an **OAuth2/OpenID Connect** provider) for each vhost you
|
||||
protect. The Caddy outpost reads the application's policy.
|
||||
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
|
||||
Connect** provider, create the app with the correct redirect URI, and paste
|
||||
the `client_id` / `client_secret` into the app's SSO settings.
|
||||
|
||||
## Operations
|
||||
|
||||
- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
|
||||
probes this).
|
||||
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
|
||||
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
|
||||
truth). The `/home/dank/authentik/` dir holds media + static files.
|
||||
- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run
|
||||
`./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
|
||||
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
|
||||
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
|
||||
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- [Caddy](Caddy.md) — the reverse proxy that carries the forward_auth directives
|
||||
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
|
||||
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
|
||||
- Research: `.docaudit/research/auth.md` — the full provider comparison
|
||||
@@ -0,0 +1,112 @@
|
||||
# Caddy
|
||||
|
||||
Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
|
||||
|
||||
**Docs:** https://caddyserver.com/docs
|
||||
|
||||
## Facts (live inventory)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Runtime | Quadlet unit `caddy.service` (`dankclaw/caddy/quadlet/caddy.container`) — boot-persistent, self-healing |
|
||||
| Image | `franksdank/caddy-porkbun:latest` (local build, `dankclaw/caddy/Containerfile`) |
|
||||
| Config | `/home/dank/Containers/caddy/Caddyfile` (bind mount, rendered by deploy) |
|
||||
| Data | `/home/dank/Containers/caddy/data` (certs) and `.../config` (ACME state) |
|
||||
| Ports | 80, 443 on 192.168.0.143 |
|
||||
| TLS | Let's Encrypt, DNS-01 via the Porkbun module (one cert per vhost) |
|
||||
| Runs on | Dankclaw only — Dankserver has no Caddy |
|
||||
|
||||
The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before start. See [[Deployment Flow]].
|
||||
|
||||
## Quadlet (boot persistence)
|
||||
|
||||
Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").
|
||||
|
||||
- **Install path on Dankclaw:** `~/.config/containers/systemd/caddy.container` (unit `caddy.service`)
|
||||
- **Network:** `dankclaw_default` (the compose bridge — the Caddyfile's container-name upstreams must keep resolving; if no compose service has created the network yet at boot, quadlet creates it first and compose reuses it by name later)
|
||||
- **Published ports:** 80/tcp, 443/tcp
|
||||
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy writes it from the checkout's `.env`; secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
|
||||
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all (302) inside the container
|
||||
- **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
|
||||
- **Enable state:** `systemctl --user is-enabled caddy` reports `generated` — that is the healthy state for a quadlet. The podman generator creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start (linger covers boot); an explicit `systemctl enable` fails on generated units, so the deploy does not run it
|
||||
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs and starts the unit
|
||||
|
||||
## The porkbun image
|
||||
|
||||
Stock Caddy cannot solve the ACME challenge because the router exposes no inbound port 80/443. The local image adds the `github.com/caddy-dns/porkbun` module, which answers the DNS-01 challenge by creating and deleting a `_acme-challenge` TXT record through the Porkbun API. No inbound port is needed. Rebuild only when Caddy or the module needs an update:
|
||||
|
||||
```
|
||||
podman build -t franksdank/caddy-porkbun:latest -f dankclaw/caddy/Containerfile .
|
||||
```
|
||||
|
||||
The Caddyfile references the credentials through the environment:
|
||||
|
||||
```
|
||||
acme_dns porkbun {
|
||||
api_key {env.PORKBUN_API_KEY}
|
||||
api_secret_key {env.PORKBUN_API_SECRET_KEY}
|
||||
}
|
||||
```
|
||||
|
||||
The deploy writes both variables from the host `.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `.env.example` (and `dankclaw/caddy/env.example`) hold placeholders. The Porkbun account holds only `franksdank.cloud`.
|
||||
|
||||
## Vhost map
|
||||
|
||||
Every vhost uses the flat `franksdank.cloud` convention. The full list lives in `dankclaw/Caddyfile`. Routing by host:
|
||||
|
||||
| Upstream | Reach |
|
||||
|----------|-------|
|
||||
| Dankclaw services (compose network) | by container name, e.g. `comfyui:8188` |
|
||||
| Dankclaw quadlet services | by published LAN port, e.g. `192.168.0.143:3000` (Gitea) |
|
||||
| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM) |
|
||||
|
||||
The bare-IP catch-all `:80` 302-redirects to `https://dankclaw.franksdank.cloud/`. The legacy name `dankserver.franksdank.cloud` 302-redirects there too. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.
|
||||
|
||||
## Basic auth
|
||||
|
||||
The Caddyfile in git carries a **bare** placeholder; deploy.sh substitutes the **full** hash from the host `.env` (`CADDY_BASICAUTH_HASH`):
|
||||
|
||||
```
|
||||
chat.franksdank.cloud {
|
||||
basicauth {
|
||||
dank REPLACE_WITH_HASH
|
||||
}
|
||||
reverse_proxy 192.168.0.143:3005
|
||||
}
|
||||
```
|
||||
|
||||
After rendering, the line reads `dank $2a$14$xxxx…` (one complete hash). Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password). `tests/deployment_flow.bats` pins the render contract.
|
||||
|
||||
The homepage (and SearXNG, Hermes, Prometheus, RomM, Copyparty) use `forward_auth` (Authentik SSO) rather than basicauth; see [[Homepage]] and [[Authentik]]. The basicauth hash above is still rendered for the vhosts that keep it (Grafana, Pi-hole).
|
||||
|
||||
## Operations
|
||||
|
||||
Validate before a reload:
|
||||
|
||||
```
|
||||
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||
```
|
||||
|
||||
The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile change hot-reloads within seconds via the file watcher. deploy.sh also issues an explicit reload after rendering. Manual reload from the host:
|
||||
|
||||
```
|
||||
podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||
```
|
||||
|
||||
Certificates and ACME state live in the caddy data volume. Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit — do not re-add it to `compose.yaml`.
|
||||
- `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
|
||||
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
|
||||
- A new vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]].
|
||||
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]].
|
||||
- Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[LAN-DNS]] — how names resolve on the LAN and WAN
|
||||
- [[Network Architecture]] — topology, access policy
|
||||
- [[Homepage]]
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,46 @@
|
||||
# Grafana
|
||||
|
||||
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, with basicauth).
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/grafana/grafana:latest` |
|
||||
| Port | 3002 (host) → 3000 (container) |
|
||||
| Data volume | dashboards, users, datasource config |
|
||||
| Datasource | Prometheus (on the same host) |
|
||||
| Provisioning | dashboards in `/etc/grafana/provisioning/dashboards` |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: "${GRAFANA_ADMIN_USER}"
|
||||
GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
|
||||
GF_SERVER_ROOT_URL: "https://grafana.franksdank.cloud"
|
||||
volumes:
|
||||
- /opt/grafana/data:/var/lib/grafana
|
||||
- /opt/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
```
|
||||
|
||||
## Datasource
|
||||
|
||||
Point Grafana at the Prometheus service by name (`http://prometheus:9090`) on the shared podman network.
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Grafana responds on `/api/health` with 200 and `"database":"ok"`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Change the default admin credentials at first boot (env or UI).
|
||||
- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Prometheus]] — the datasource
|
||||
- [[Observability]] — the whole stack
|
||||
@@ -0,0 +1,96 @@
|
||||
# Homepage Design Grilling
|
||||
|
||||
The decision log for the homepage design. All questions are settled (2026-08-23). This page records the final decisions and the places where the operator's answer changed the initial recommendation.
|
||||
|
||||
## Final decisions
|
||||
|
||||
| # | Decision | Value |
|
||||
|---|----------|-------|
|
||||
| Q1 | How the page learns up/down | Server-side checker writes `status.json`; the page reads only that file. (Documented future intent: client-side probing.) |
|
||||
| Q2 | How the checker runs | systemd timer, rewrites `status.json` every 3 minutes. |
|
||||
| Q3 | Page model | Single site on Dankclaw. Root = user services (no host labels). `/admin/` = admin services, second operator-only password. |
|
||||
| Q5 | Service split | User: SearXNG, ComfyUI, OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin. Admin: Hermes, Grafana, Prometheus, Loki, Caddy, dnsmasq, Gluetun, LLM fleet, low-level backends (status-only). The *arr stack is excluded. |
|
||||
| Q6 | Tourist `/how/` page | **Defer** — it is extra. |
|
||||
| Q7 | `status.json` | One file; each service carries a `category` field; each page filters. |
|
||||
| Q8 | `services.json` | One file with a `category` field. |
|
||||
| Q9 | Checker language | python3. |
|
||||
| Q10 | Probe method | **Per service** — `tcp` or `http`, set per entry. |
|
||||
| Q11 | Probe timeout | **5 s** per service. |
|
||||
| Q12 | LLM loading | Three states; a listening port is Up (health 503 = Up). |
|
||||
| Q13 | Checker location | `scripts/homepage-status.py` + `.service` + `.timer`, tracked in git. |
|
||||
| Q14 | Reach backends | `probe` = full `host:port` (`127.0.0.1:PORT` or `192.168.0.235:PORT`). |
|
||||
| Q15 | WIP | static `"wip": true`; the checker skips the probe. |
|
||||
| Q16 | Staleness | `generated_at` in `status.json`; the page warns when it is older than 10 min. |
|
||||
| Q17 | Writes | atomic (write a temp file, then rename). |
|
||||
| Q18 | Admin password | `CADDY_ADMIN_HASH` in the host `.env` (git-ignored), operator-only. |
|
||||
| Q19 | Per-path basicauth | `@admin path /admin/*` with a second `basicauth` block. |
|
||||
| Q20 | Descriptions | hand-written one-liners in `services.json`. |
|
||||
| Q21 | Open link | the Caddy site URL; status-only backends set `showLink: false`. |
|
||||
| Q22 | Docs link | **a per-service `/docs/` page. Do NOT link the existing reference docs (they may contain secrets).** |
|
||||
| Q23 | Layout | **TUI-style cards/panels** (a grid of panels), not a flat list. |
|
||||
| Q24 | Summary line | yes — a placeholder for now; refine later. |
|
||||
| Q25 | Terminus | bundle the font (woff2) + a monospace fallback. |
|
||||
| Q26 | Palette | as proposed (green `#33ff66`, red `#ff5555`, amber `#ffcc33`, cyan `#33ccff`, bg `#050805`). |
|
||||
| Q27 | Styling | **toggleable: default = full theme; a toggle strips to bare unstyled HTML (motherfuckingwebsite.com style).** |
|
||||
| Q28 | Cursor | **do not alter the user's cursor.** |
|
||||
| Q29 | Docs content | how a regular user uses the service; **no maintenance details.** |
|
||||
| Q30 | Docs path | **`/docs/<svc>/`** (not `/usage/`). |
|
||||
| Q31 | Docs auth | **the same auth as the homepage section** (user docs = shared password, admin docs = admin password). |
|
||||
| Q32 | Docs workstream | a separate effort; built as each service goes active. |
|
||||
| Q33 | Deploy | **git-driven; master always reflects the deployed state.** |
|
||||
| Q34 | `/srv/homepage/` | `deploy.sh` creates it and copies the assets. |
|
||||
| Q35 | Timer | enabled at boot. |
|
||||
| Q36 | Checker tests | **full TDD.** |
|
||||
| Q37 | Page/config tests | **full TDD.** |
|
||||
| Q38 | Security test | **full TDD.** |
|
||||
| Q39 | Backend move | a config edit (`probe` host:port). |
|
||||
| Q40 | LAN timeout | **5 s.** |
|
||||
| Q41 | Auto-refresh | **no** — the page reads `status.json` once on load. |
|
||||
| Q42 | Mobile | responsive. |
|
||||
| Q43 | Footer timestamp | **last master deployment datetime** (not last-check). |
|
||||
|
||||
Styling (settled): terminal TUI, Terminus (monospace fallback), green-on-black, 1995 phosphor palette. Default = full theme; a toggle strips to bare unstyled HTML. Do not alter the user's cursor.
|
||||
|
||||
## Where the operator changed the recommendation
|
||||
|
||||
These are the deltas from the initial recommendations. Everything else was accepted as recommended.
|
||||
|
||||
- **Q10** — the probe method is **per service** (`tcp` or `http`), not TCP-only.
|
||||
- **Q11 / Q40** — the probe timeout is **5 s** (not 3 s).
|
||||
- **Q22** — the docs link points to a **per-service `/docs/` page**, never the existing reference docs (they may contain secrets).
|
||||
- **Q23** — the layout is **TUI-style cards/panels**, not a flat list.
|
||||
- **Q27** — styling is **toggleable**: the full theme by default, with a toggle to bare unstyled HTML.
|
||||
- **Q28** — **do not alter the user's cursor** (no fake blinking cursor).
|
||||
- **Q29** — the docs are **user-facing only**; no maintenance details.
|
||||
- **Q30** — the docs path is **`/docs/`** (not `/usage/`).
|
||||
- **Q31** — the docs use **the same auth as the homepage section**.
|
||||
- **Q33** — **git-driven; master always reflects the deployed state.**
|
||||
- **Q36–Q38** — **full TDD.**
|
||||
- **Q41** — **no client-side auto-refresh.**
|
||||
- **Q43** — the footer shows the **last master deployment datetime** (not last-check).
|
||||
|
||||
## Build status — complete (2026-08-23)
|
||||
|
||||
The build is done, slice by slice, full TDD (each slice red → green, committed
|
||||
and pushed). 116 tests pass. The render logic is verified with a Node DOM shim.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[write the checker tests] --> B[write the checker]
|
||||
B --> C[write services.json]
|
||||
C --> D[write the pages + theme + toggle]
|
||||
D --> E[extend deploy.sh + Caddyfile]
|
||||
E --> F[tests green]
|
||||
F --> G[deploy to Dankclaw]
|
||||
G --> H[verify in browser]
|
||||
```
|
||||
|
||||
Slices: 1 the checker core (tcp/wip/atomic/generated_at), 2 the http probe, 3
|
||||
the real `services.json` + schema, 4 the pages + theme + toggle, 5 the Caddyfile
|
||||
auth split, 6 the git-driven deploy (both hashes, `deployed.js`, the timer).
|
||||
|
||||
**Remaining to go live:** the operator adds `CADDY_ADMIN_HASH` (the
|
||||
operator-only password, `caddy hash-password`) to the target `.env`, then runs
|
||||
`./scripts/deploy.sh dankclaw`. Until then the admin hash stays a placeholder.
|
||||
|
||||
The per-service docs pages (Q22/Q29/Q30/Q31) are a separate workstream, built as each service goes active.
|
||||
@@ -0,0 +1,203 @@
|
||||
# Homepage
|
||||
|
||||
A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation.
|
||||
|
||||
**Served by:** Caddy, `file_server` + `forward_auth` (Authentik SSO) — **Docs:** this page
|
||||
|
||||
## Status — 2026-08-23: deployed to production
|
||||
|
||||
The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the static assets to `/srv/homepage/`, installs the status checker (systemd timer, every 3 min, writes `status.json`), writes `deployed.js` (the last master deployment datetime), renders the Caddyfile (the shared basicauth hash from the host `.env`), reloads Caddy, and checks health. 146 tests pass. The Caddyfile auth uses a `route` block with `forward_auth` (see [Auth](#auth)); the caddy container mounts the host's `/srv/homepage` (not the repo source) so the generated files serve.
|
||||
|
||||
**Moved to the naked domain (2026-08-24):** the homepage now serves `franksdank.cloud` and `www.franksdank.cloud`. The old names `dankclaw.franksdank.cloud` and `dankserver.franksdank.cloud` redirect to it. Authentication moved from per-page basicauth to Authentik `forward_auth` (see [Authentik](Authentik.md)).
|
||||
|
||||
The per-service `/docs/` pages, the *arr stack, and the optional `/how/` page remain separate threads.
|
||||
|
||||
## Design decision — 2026-08-22
|
||||
|
||||
The live up/down check runs server-side. A checker on Dankclaw probes each backend and writes a same-origin `status.json`. The page fetches only that file and renders the badges. The browser never calls a backend directly. The checker runs as a systemd timer every 3 minutes.
|
||||
|
||||
The site has two pages. The root lists user-facing services only, with no host labels. An `/admin/` subpage lists the admin and infrastructure services behind a separate password for the operator. Gitea is a user service. Hermes is an admin service. A "how it is built" page for visitors is deferred (Q6).
|
||||
|
||||
The client-side probe design is blocked in this topology. The page origin is `https://franksdank.cloud`. The service URLs are `https://<name>.franksdank.cloud`. That is cross-origin and auth-gated. Both fail a browser `fetch`.
|
||||
|
||||
Intended direction: move to client-side probing. Serve every service as a same-origin `https` path under `franksdank.cloud`. Add a health endpoint and CORS headers to each backend. Then the page probes each service directly. Keep this note until that shift is complete.
|
||||
|
||||
## Structure
|
||||
|
||||
The site has two pages. The root page lists the user services. The `/admin/` page lists the admin and infrastructure services. The optional `/how/` page is deferred (Q6).
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph site["dankclaw.franksdank.cloud (Caddy)"]
|
||||
ROOT["/ user services<br/>(shared password)"]
|
||||
ADMIN["/admin/ admin services<br/>(operator password)"]
|
||||
HOW["/how/ how it is built<br/>(deferred)"]
|
||||
STATUS["/status.json"]
|
||||
end
|
||||
TIMER["systemd timer<br/>every 3 minutes"]
|
||||
CHECKER["checker script<br/>probes each backend"]
|
||||
BACKENDS["backends on both hosts"]
|
||||
TIMER --> CHECKER
|
||||
CHECKER -->|writes| STATUS
|
||||
STATUS -->|polled| ROOT
|
||||
STATUS -->|polled| ADMIN
|
||||
CHECKER -->|probes| BACKENDS
|
||||
```
|
||||
|
||||
## Status model
|
||||
|
||||
Each service shows one of three states.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Up
|
||||
[*] --> Down
|
||||
[*] --> WIP
|
||||
Up --> Down: probe fails
|
||||
Down --> Up: probe succeeds
|
||||
WIP --> Up: work completes
|
||||
```
|
||||
|
||||
- **Up** — the probe reached the backend.
|
||||
- **Down** — the probe failed (connection refused or timeout).
|
||||
- **WIP** — a static override. The service is intentionally not finished. The checker does not probe it.
|
||||
|
||||
## Checker
|
||||
|
||||
The checker runs on Dankclaw as a systemd timer. Every run probes each backend directly, then writes `status.json`. It probes the backend `host:port`, not the Caddy site. A Caddy `502` or `401` cannot tell up from down. The probe method is per service (`tcp` or `http`). The timeout is 5 s. The page reads `status.json` once on load. It does not auto-refresh.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[timer fires] --> B[read service config]
|
||||
B --> C{for each service}
|
||||
C -->|wip| D[mark WIP]
|
||||
C -->|probe| E[probe: tcp or http]
|
||||
E -->|connected| F[mark Up]
|
||||
E -->|refused / timeout| G[mark Down]
|
||||
D --> H[write status.json]
|
||||
F --> H
|
||||
G --> H
|
||||
```
|
||||
|
||||
## Service classification
|
||||
|
||||
**User services** (root page): SearXNG, ComfyUI, OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin.
|
||||
|
||||
**Admin services** (`/admin/`): Hermes, Grafana, Prometheus, Loki, Caddy, Pi-hole, Gluetun, the LLM fleet, and the low-level backends (mariadb, redis, Postgres, node-exporter, alloy). The backends show status only. They get a link only where a UI is meaningful.
|
||||
|
||||
**Excluded for now:** the *arr stack (Radarr, Sonarr, Lidarr, Readarr, Prowlarr, qBittorrent). It is not started. It is a separate effort, possibly on a different host.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
shared/homepage/
|
||||
├── index.html # root page (user services)
|
||||
├── admin.html # admin + infrastructure services
|
||||
├── how.html # deferred (Q6)
|
||||
├── style.css # full theme; a toggle strips to bare HTML
|
||||
├── app.js # loads status.json once, renders the panels
|
||||
├── services.json # service config (user + admin)
|
||||
└── fonts/ # Terminus (woff2) for @font-face
|
||||
```
|
||||
|
||||
`status.json` is generated at runtime in `/srv/homepage/`. It is not committed.
|
||||
|
||||
## Rendering
|
||||
|
||||
The page renders a **grid of TUI-style panels** — one panel per service. Each panel shows the state badge, the name, the description, and the `Open` and `Docs` links. A summary line at the top counts the states. The footer shows the last master deployment datetime (Q43).
|
||||
|
||||
The page reads `status.json` **once on load**. It does not auto-refresh (Q41).
|
||||
|
||||
Styling is **toggleable** (Q27). The default is the full terminal theme. A toggle strips the page to bare, unstyled HTML (motherfuckingwebsite.com style). The page does not alter the user's cursor (Q28).
|
||||
|
||||
## Auth
|
||||
|
||||
Caddy wraps the site with `forward_auth` (Authentik SSO). A single Authentik
|
||||
session covers the whole site: the user page, the `/admin/` page, and the
|
||||
shared data + assets. The browser's Authentik cookie is reused for the data
|
||||
fetches, so no per-page credential is needed. See [Authentik](Authentik.md) for
|
||||
the provider, the outpost, and the first-time setup.
|
||||
|
||||
The Caddyfile pattern (one of the forward_auth vhosts):
|
||||
|
||||
```
|
||||
franksdank.cloud www.franksdank.cloud {
|
||||
root * /srv/homepage
|
||||
|
||||
route {
|
||||
# 1. The outpost path is proxied to Authentik (login UI + session).
|
||||
reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
|
||||
|
||||
# 2. forward_auth checks the session before serving the page.
|
||||
forward_auth authentik-server:9000 {
|
||||
uri /outpost.goauthentik.io/auth/caddy
|
||||
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
|
||||
trusted_proxies private_ranges
|
||||
}
|
||||
|
||||
# 3. The pages, in file order. 'handle' preserves order; a top-level
|
||||
# 'rewrite' would be sorted first and break the path matchers.
|
||||
handle /admin* {
|
||||
try_files /admin.html
|
||||
file_server
|
||||
}
|
||||
|
||||
@data path /services.json /status.json /app.js /style.css /deployed.js
|
||||
handle @data {
|
||||
file_server
|
||||
}
|
||||
|
||||
handle {
|
||||
try_files /index.html
|
||||
file_server
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A `handle` inline matcher is a single token; the multi-path `@data` match is a
|
||||
named matcher. The caddy container mounts the host's `/srv/homepage` (where the
|
||||
deploy writes the static files and the checker writes `status.json`), not the
|
||||
repo's `shared/homepage`. A compose change needs a container recreate (`podman
|
||||
compose up -d caddy`), not a reload. The Caddyfile syntax is documented in the
|
||||
agent knowledge base (`~/.pi/agent/docs/caddy/caddyfile.md`).
|
||||
|
||||
## Service entry format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "SearXNG",
|
||||
"category": "user",
|
||||
"description": "Privacy-respecting metasearch engine.",
|
||||
"url": "http://search.franksdank.cloud",
|
||||
"docs": "/docs/searxng/",
|
||||
"probe": "127.0.0.1:8081",
|
||||
"method": "http",
|
||||
"wip": false,
|
||||
"showLink": true
|
||||
}
|
||||
```
|
||||
|
||||
- `category` — `user` or `admin`. Selects the page.
|
||||
- `url` — the "Open" link (the Caddy site).
|
||||
- `docs` — the per-service docs page (`/docs/<svc>/`).
|
||||
- `probe` — the backend `host:port` the checker probes.
|
||||
- `method` — `tcp` or `http`. The probe type for this service (Q10).
|
||||
- `wip` — when true, the badge shows WIP and the checker skips the probe.
|
||||
- `showLink` — when false, the panel shows status only (no "Open" link).
|
||||
|
||||
## Deployment and testing
|
||||
|
||||
The deploy is **git-driven**. Master always reflects the currently deployed state (Q33). `deploy.sh` copies the assets to `/srv/homepage/`, installs the checker timer, renders the Caddyfile (both hashes), and reloads Caddy. It stamps the deployment datetime the footer shows (Q43).
|
||||
|
||||
The build is **full TDD** (Q36–Q38). Tests exist before the code: the checker (a fake up port, a down port, a WIP service, and the JSON schema), the pages and config (the `services.json` schema, `caddy validate`, the `status.json` reference), and the auth check (the forward_auth block and the renderable basicauth placeholder).
|
||||
|
||||
## Per-service docs (separate workstream)
|
||||
|
||||
Each service gets a docs page at `/docs/<svc>/` (Q30). The content is a **how-to for a regular user** — no maintenance details (Q29). The pages are password-protected by the same auth as the homepage section they belong to (Q31). They are built as each service goes active (Q32). The `docs` link on a service points to its page. The existing reference docs in `serverAppConfigs-docs/` are not linked from the homepage. They may contain secrets (Q22).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Homepage Design Grilling]] — the decision log
|
||||
- [[Caddy]] — the proxy that serves it
|
||||
- [[Dankclaw Stack]], [[Dankserver Stack]] — the services it lists
|
||||
@@ -0,0 +1,145 @@
|
||||
# LAN DNS
|
||||
|
||||
The homelab gives every service a name under `franksdank.cloud`. Users type a name; they never type a port number. This page covers how names reach users: the Pi-hole server, the G36 router hand-off, and the split between LAN and WAN resolution.
|
||||
|
||||
See [[Caddy]] for the reverse proxy that serves the names, and [[Network Architecture]] for the full topology.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Where | Role |
|
||||
|-----------|-------|------|
|
||||
| Pi-hole | Dankclaw quadlet (`dankclaw/pihole/`) | Answers the flat service names for LAN clients |
|
||||
| SURFboard G36 | 192.168.0.1 | Hands dnsmasq to clients as the DHCP DNS server |
|
||||
| Porkbun | Registrar for `franksdank.cloud` | Answers the same names for the WAN |
|
||||
| Caddy | Dankclaw (192.168.0.143:443) | Serves every name over HTTPS |
|
||||
|
||||
## How a name reaches a client
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as LAN client (laptop, phone)
|
||||
participant G as G36 router (192.168.0.1)
|
||||
participant D as Pi-hole (192.168.0.143:53)
|
||||
participant P as Porkbun (WAN)
|
||||
participant X as Caddy (192.168.0.143:443)
|
||||
|
||||
C->>G: DHCP request
|
||||
G-->>C: lease + DNS = 192.168.0.143
|
||||
C->>D: dig romm.franksdank.cloud
|
||||
D-->>C: 192.168.0.143
|
||||
Note over D: addn-hosts answer<br/>(dankclaw/pihole/addn-hosts)
|
||||
C->>X: https://romm.franksdank.cloud
|
||||
X-->>C: RomM (192.168.0.235:8085 upstream)
|
||||
C->>P: dig franksdank.cloud (unknown name)
|
||||
P-->>C: 71.227.227.172
|
||||
```
|
||||
|
||||
Pi-hole answers only the names in `addn-hosts`. Every other query goes to the upstream resolvers `1.1.1.1` and `1.0.0.1`. The zone apex and unknown subdomains fall through to Porkbun. This avoids a wildcard override.
|
||||
|
||||
## Why split DNS
|
||||
|
||||
The G36 has no NAT loopback (hairpin). A LAN client that resolves a name to the WAN IP and then connects to it would time out. Split DNS removes the hairpin: LAN clients get the LAN IP, WAN clients get the public IP. Both use the same name and the same Let's Encrypt certificate. See [[Caddy]] for the TLS side.
|
||||
|
||||
## Pi-hole quadlet
|
||||
|
||||
Pi-hole FTL is a dnsmasq fork. It serves the same `addn-hosts` file the retired dnsmasq quadlet used, mounted into `/etc/pihole/hosts/` read-only. It retired the dnsmasq quadlet on 2026-08-24.
|
||||
|
||||
- **Location:** `dankclaw/pihole/` — `quadlet/pihole.container`, `env.example`, `addn-hosts`
|
||||
- **Ports:** 53/udp + 53/tcp (DNS) and 8083/tcp (web UI) on 192.168.0.143
|
||||
- **Host prerequisite 1:** `sysctl net.ipv4.ip_unprivileged_port_start=53` (persisted in `/etc/sysctl.d/99-dnsmasq.conf`)
|
||||
- **Host prerequisite 2:** `DNSStubListener=no` in `/etc/systemd/resolved.conf` (the resolved stub would otherwise hold port 53 inside the container netns)
|
||||
- **Env file:** `/home/dank/Containers/pihole/env` (never committed) — `ServerIP`, `TZ`, `DNS1` + `DNS2` (pinned upstreams `1.1.1.1` + `1.0.0.1`), `WEBPASSWORD` (the shared family password)
|
||||
- **Web UI:** `pihole.franksdank.cloud` (Caddy vhost → 8083), family login
|
||||
- **Deploy + restart after a change:** `./scripts/deploy-pihole-quadlet.sh dankclaw`
|
||||
|
||||
Add a new service name in two places:
|
||||
|
||||
1. `dankclaw/pihole/addn-hosts` — `192.168.0.143 <name>.franksdank.cloud`
|
||||
2. Porkbun — A record `<name> 71.227.227.172` (so WAN clients and the DNS-01 challenge work)
|
||||
|
||||
Then add the vhost to [[Caddy]] and run `./scripts/deploy.sh dankclaw`. See [[Pi-hole]] for the full configuration.
|
||||
|
||||
## G36 router DNS hand-off
|
||||
|
||||
The router must tell DHCP clients to use Pi-hole. Until that step is done, new clients get the router's own DNS relay and the flat names do not resolve for them. Existing clients that already use Pi-hole are unaffected.
|
||||
|
||||
**Manual step (web GUI at `https://192.168.0.1`):**
|
||||
|
||||
1. Open **Gateway → Connection → Local IP Network → IPv4**.
|
||||
2. Uncheck **DNS Proxy** (disable the router DNS relay).
|
||||
3. Set **DNS** to **Statically configured**.
|
||||
4. Set **Primary DNS** to `192.168.0.143`.
|
||||
5. Leave the secondary field empty or set `0.0.0.0`.
|
||||
6. Apply, then re-lease DHCP on each client (`sudo dhclient -r && sudo dhclient`).
|
||||
|
||||
The router admin interface has no programmatic API in this firmware. The web GUI is the only control path. Keep the credentials out of this repo; the repo tracks no router secrets.
|
||||
|
||||
Verify after the change:
|
||||
|
||||
```
|
||||
# on a LAN client
|
||||
resolv.conf # nameserver 192.168.0.143
|
||||
dig romm.franksdank.cloud +short
|
||||
# expected: 192.168.0.143
|
||||
```
|
||||
|
||||
## Dankclaw host DNS
|
||||
|
||||
The Dankclaw host itself resolves the flat names through Pi-hole:
|
||||
|
||||
- `/etc/systemd/resolved.conf`: `DNS=192.168.0.143 192.168.0.1` (Pi-hole first, router as fallback)
|
||||
|
||||
Without this, the host would resolve the names through the router to the WAN IP and time out (no hairpin).
|
||||
|
||||
## Dankware host DNS
|
||||
|
||||
`dankware` is a LAN client, not a Pi-hole server. Its flat names resolve through Pi-hole (`192.168.0.143`), but its resolver setup differs from Dankclaw's because it carries two `+DefaultRoute` links.
|
||||
|
||||
- **Two default links:** `enp47s0` (ethernet, metric 100) and `wlan0` (wifi, metric 600), each getting the same DHCP DNS list `[192.168.0.143, 1.1.1.1, …]`.
|
||||
- **The flap:** `systemd-resolved` picks a different "current" resolver per link — `enp47s0` settled on the Comcast v6 upstream (→ WAN IP, times out on the LAN), `wlan0` on Pi-hole (→ LAN IP). Flat-name resolution flapped between the two on every query.
|
||||
- **The fix:** pin `192.168.0.143` as the static IPv4 resolver on both NetworkManager connections. `resolved.conf`'s own `DNS=` is a per-link fallback, so the link itself must be pinned:
|
||||
|
||||
```bash
|
||||
for c in "Wired connection 1" FRANKFI-5G; do
|
||||
nmcli connection modify "$c" ipv4.method auto \
|
||||
ipv4.dns 192.168.0.143 \
|
||||
ipv4.ignore-auto-dns yes \
|
||||
ipv4.ignore-auto-routes no
|
||||
done
|
||||
nmcli connection up "Wired connection 1"
|
||||
nmcli connection up FRANKFI-5G
|
||||
systemctl restart systemd-resolved
|
||||
```
|
||||
|
||||
`method:auto` keeps DHCP for the address/gateway/route (dual-default traffic path preserved); `ignore-auto-dns=yes` uses only the static resolver; `ignore-auto-routes=no` keeps the DHCP routes. Verify with a cache-clear loop — `romm.franksdank.cloud` should return `192.168.0.143` on every run:
|
||||
|
||||
```bash
|
||||
for i in $(seq 1 10); do resolvectl flush-caches; dig romm.franksdank.cloud +short; done
|
||||
```
|
||||
|
||||
The apex and unknown subdomains still fall through to Porkbun (→ `71.227.227.172`), so WAN clients and a bare `dig franksdank.cloud` keep working.
|
||||
|
||||
## WAN records (Porkbun)
|
||||
|
||||
The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the apex, all pointing at `71.227.227.172`. TTL is 300. The zone has no wildcard record: unknown subdomains return NXDOMAIN. Caddy's DNS-01 challenges add and remove `_acme-challenge.*` TXT records through the same API during issuance. See [[Caddy]] for the TLS flow.
|
||||
|
||||
`franksdank.casa` is not in this Porkbun account. The `plex.franksdank.casa` vhost stays plain HTTP for that reason. The canonical Plex name is `plex.franksdank.cloud` (HTTPS).
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | First check |
|
||||
|---------|-------------|
|
||||
| Name does not resolve on a new client | `resolv.conf` shows 192.168.0.1, not .143 — the G36 hand-off step is missing |
|
||||
| Name resolves to 71.227.x and times out on the LAN | Client used the router DNS — same root cause |
|
||||
| Flat name flaps between 192.168.0.143 and 71.227.x on one client | Two `+DefaultRoute` links; resolved picks a different "current" resolver per link. Pin 192.168.0.143 as static DNS on both NM connections — see [[#Dankware host DNS]] |
|
||||
| `dig @192.168.0.143` fails entirely | Pi-hole quadlet down: `systemctl --user status pihole` |
|
||||
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`). FTL serves it. Use the FQDN, or an `/etc/hosts` entry on the client |
|
||||
| HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
|
||||
| Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
|
||||
- [[Caddy]] — TLS, vhost list, DNS-01
|
||||
- [[Network Architecture]] — topology, access policy
|
||||
- [[Dankclaw Stack]] — the host that carries Pi-hole and Caddy
|
||||
@@ -0,0 +1,51 @@
|
||||
# Loki
|
||||
|
||||
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs, and it feeds the log panels in Grafana. It replaces the need for a full ELK stack.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://grafana.com/docs/loki/latest
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `franksdank/loki:latest` (local build with a static busybox healthcheck) |
|
||||
| Port | 3100 (host) → 3100 (container) |
|
||||
| Mode | single-binary (all components in one process) |
|
||||
| Storage | filesystem (bind mount) |
|
||||
| Auth | `auth_enabled: false` (single-tenant homelab) |
|
||||
| Schema | v13 / TSDB |
|
||||
|
||||
## Configuration
|
||||
|
||||
Config lives at `shared/loki/loki-config.yaml`. Key settings:
|
||||
|
||||
- `schema_config` v13 with `store: tsdb` and filesystem object store.
|
||||
- `compactor` with `retention_enabled: true` so logs do not grow without bound.
|
||||
- `replication_factor: 1` for a single instance.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/loki` | `/opt/loki/data` |
|
||||
| `/etc/loki/loki-config.yaml` | `shared/loki/loki-config.yaml` (bind mount) |
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Loki responds on `/ready` with 200.
|
||||
|
||||
## Log Shipping
|
||||
|
||||
The [[Alloy]] agent on each host ships container and journal logs to Loki. Grafana queries Loki through the datasource in [[Observability]].
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Filesystem storage grows; enable compactor retention.
|
||||
- Loki and Promtail/Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
|
||||
- For multi-host shipping, Loki's HTTP listener binds `0.0.0.0`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Observability]]
|
||||
- [[Alloy]] — the log shipper
|
||||
- [[Grafana]] — the query frontend
|
||||
@@ -0,0 +1,55 @@
|
||||
# Node-Exporter
|
||||
|
||||
Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host inside its own container.
|
||||
|
||||
**Hosts:** Dankserver + Dankclaw — **Docs:** https://prometheus.io/docs/guides/node-exporter
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `prom/node-exporter:latest` |
|
||||
| Port | 9100 (host) |
|
||||
| Network | `host` mode |
|
||||
| Config | `--path.procfs=/host/proc`, `--path.sysfs=/host/sys`, `--path.rootfs=/rootfs` |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
services:
|
||||
node-exporter:
|
||||
image: prom/node-exporter:latest
|
||||
restart: always
|
||||
network_mode: host
|
||||
pid: host
|
||||
volumes:
|
||||
- /proc:/host/proc:ro
|
||||
- /sys:/host/sys:ro
|
||||
- /:/rootfs:ro
|
||||
command:
|
||||
- --path.procfs=/host/proc
|
||||
- --path.sysfs=/host/sys
|
||||
- --path.rootfs=/rootfs
|
||||
```
|
||||
|
||||
## Prometheus Targets
|
||||
|
||||
Prometheus scrapes each host's node-exporter directly:
|
||||
|
||||
| Host | Target |
|
||||
|------|--------|
|
||||
| Dankserver | `192.168.0.235:9100` |
|
||||
| Dankclaw | `192.168.0.143:9100` |
|
||||
|
||||
These scrape jobs live in the Prometheus config; see [[Prometheus]] and [[Observability]].
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Requires `network_mode: host` (or a published port) so Prometheus can reach 9100.
|
||||
- The `/proc`, `/sys`, and root mounts must be `read-only` (`:ro`).
|
||||
- Do not mount the same paths on both hosts' containers; each node-exporter reflects only its own host.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Prometheus]] — the scraper
|
||||
- [[Observability]] — the full stack
|
||||
@@ -0,0 +1,53 @@
|
||||
# Observability
|
||||
|
||||
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
|
||||
|
||||
## Stack
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
S[Services] -->|/metrics| P[Prometheus]
|
||||
S -->|logs| T[Alloy]
|
||||
T --> L[Loki]
|
||||
P --> G[Grafana]
|
||||
L --> G
|
||||
```
|
||||
|
||||
| Component | Role | Image | Port |
|
||||
|-----------|------|-------|------|
|
||||
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
|
||||
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
|
||||
| Loki | log storage & query | `grafana/loki` | 3100 |
|
||||
| Alloy | log shipping (per host) | `grafana/alloy` | — |
|
||||
|
||||
## Log Aggregation Choice
|
||||
|
||||
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
||||
|
||||
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
||||
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
||||
- No Elasticsearch memory cost, no Kubernetes required.
|
||||
|
||||
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
||||
|
||||
## Scrape Surface
|
||||
|
||||
| Target | Source |
|
||||
|--------|--------|
|
||||
| Prometheus self | `localhost:9090/metrics` |
|
||||
| ComfyUI | `/system_stats` (limited) or blackbox probe |
|
||||
| Gitea | `/metrics` (built-in exporter) |
|
||||
| Grafana | `/metrics` (built-in) |
|
||||
| Host metrics | node-exporter `:9100` |
|
||||
|
||||
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
|
||||
|
||||
## Healthchecks
|
||||
|
||||
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Grafana]], [[Prometheus]]
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Homepage]] — the user-facing status surface
|
||||
@@ -0,0 +1,73 @@
|
||||
# Pi-hole
|
||||
|
||||
Pi-hole is the LAN DNS server for the homelab. It runs as a quadlet container on Dankclaw (192.168.0.143). It answers the flat `franksdank.cloud` / `franksdank.casa` service names and filters ad traffic on the LAN. It retired the dnsmasq quadlet on 2026-08-24. See [[LAN-DNS]] for the name-resolution flow and the router hand-off.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Where | Role |
|
||||
|-----------|-------|------|
|
||||
| `pihole` quadlet | `dankclaw/pihole/quadlet/pihole.container` | Boot-persistent container unit |
|
||||
| FTL | `docker.io/pihole/pihole:latest` | DNS engine (a dnsmasq fork) + ad blocking |
|
||||
| `addn-hosts` | `dankclaw/pihole/addn-hosts` | One A record per Caddy vhost |
|
||||
| sysctl drop-in | `dankclaw/pihole/sysctl/99-dnsmasq.conf` (repo) → `/etc/sysctl.d/` | Rootless port-53 threshold, applied at boot |
|
||||
| env file | `/home/dank/Containers/pihole/env` (host) | `ServerIP`, `TZ`, `DNS1`, `DNS2`, `WEBPASSWORD` |
|
||||
| Web UI | `pihole.franksdank.cloud` (Caddy → 8083) | Dashboard + admin, family login |
|
||||
|
||||
## Quadlet
|
||||
|
||||
- **Install path on Dankclaw:** `~/.config/containers/systemd/pihole.container` (unit `pihole.service`)
|
||||
- **Network:** `slirp4netns` (the same pattern as open-webui and gitea)
|
||||
- **Published ports:** 53/udp + 53/tcp (DNS), 8083/tcp (web UI)
|
||||
- **Mounts:**
|
||||
- `/home/dank/Containers/pihole/etc-pihole` → `/etc/pihole` (config + state, preserved across restarts)
|
||||
- `/home/dank/Containers/pihole/etc-dockers` → `/etc/dockers` (blocklists)
|
||||
- `/home/dank/Containers/pihole/addn-hosts` → `/etc/pihole/hosts/addn-hosts` (read-only; the record file from this repo)
|
||||
- **Restart:** `Restart=always`, `RestartSec=5`
|
||||
- **Deploy:** `./scripts/deploy-pihole-quadlet.sh dankclaw` (idempotent; preserves the data dirs)
|
||||
|
||||
## Records
|
||||
|
||||
FTL does not read `/etc/dnsmasq.d/` — its generated `dnsmasq.conf` has no `conf-dir`. The only file-based local-record mechanism is `hostsdir=/etc/pihole/hosts`: FTL parses every file in that directory as an `/etc/hosts` file. The quadlet therefore mounts the repo record file there, read-only:
|
||||
|
||||
```
|
||||
/home/dank/Containers/pihole/addn-hosts -> /etc/pihole/hosts/addn-hosts
|
||||
```
|
||||
|
||||
The file format is hosts-style (`<IP> <name> [<name> ...]`), one A record per Caddy vhost, all pointing at `192.168.0.143`. The file stays in sync with the Caddyfile (`tests/pihole.bats` cross-checks the two). Add a name there and in Porkbun, then deploy. See [[LAN-DNS]] for the two-place rule.
|
||||
|
||||
**Pitfall (2026-08-24):** the first quadlet mounted the file into `/etc/dnsmasq.d/`. FTL never read it; names only resolved from the query cache. A fresh name returned NXDOMAIN. Mount into `/etc/pihole/hosts/`, not `/etc/dnsmasq.d/`.
|
||||
|
||||
## Upstream
|
||||
|
||||
The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `DNS1=1.1.1.1`, `DNS2=1.0.0.1` (mirrors the retired `dnsmasq.conf` `server=` lines). Every query that is not a local record goes to those resolvers. The zone apex and unknown subdomains fall through to Porkbun.
|
||||
|
||||
## Web UI and API
|
||||
|
||||
- **URL:** `https://pihole.franksdank.cloud/` (Caddy vhost, family basicauth → 8083)
|
||||
- **Password:** `WEBPASSWORD` in the env file (the shared family password, the same credential as `CADDY_BASICAUTH_HASH`)
|
||||
- **API:** `https://pihole.franksdank.cloud/api/...` — authenticate with the session cookie, then query the blocklist, the local DNS records, and the query log
|
||||
- **Port on the LAN:** `http://192.168.0.143:8083/` (direct, no Caddy)
|
||||
|
||||
## Host prerequisites
|
||||
|
||||
1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any drop-in that sets a value above 53 in the home trash, comments out a stale active line in `/etc/sysctl.conf`, and applies everything with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** the value was reset to 80 by *two* sources — a later-sorted drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl processes **after** all of `sysctl.d/` and therefore overrides every drop-in. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until both conflicts were removed.
|
||||
2. `DNSStubListener=no` in `/etc/systemd/resolved.conf` — the resolved stub would otherwise hold port 53 inside the container netns.
|
||||
|
||||
Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | First check |
|
||||
|---------|-------------|
|
||||
| `dig @192.168.0.143` fails entirely | Quadlet down: `systemctl --user status pihole` |
|
||||
| A name does not resolve for a new client | The client's `resolv.conf` does not show `192.168.0.143` — the G36 hand-off is missing. See [[LAN-DNS]] |
|
||||
| The web UI asks for a password loop | `WEBPASSWORD` in the env file is the placeholder `changeme` — fill the real value, restart the unit |
|
||||
| Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong |
|
||||
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line |
|
||||
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[LAN-DNS]] — name resolution flow, router hand-off, split DNS
|
||||
- [[Caddy]] — the `pihole.franksdank.cloud` vhost
|
||||
- [[Dankclaw Stack]] — the host that carries Pi-hole
|
||||
@@ -0,0 +1,57 @@
|
||||
# Prometheus
|
||||
|
||||
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `prometheus.dankclaw.local` — **Port:** 9090 — **Docs:** https://prometheus.io/docs
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/prom/prometheus:latest` |
|
||||
| Port | 9090 (host) → 9090 (container) |
|
||||
| Config | `/etc/prometheus/prometheus.yml` (bind mount) |
|
||||
| Data volume | TSDB |
|
||||
| Retention | `--storage.tsdb.retention.time=30d` |
|
||||
|
||||
## prometheus.yml Targets
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: "prometheus"
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
|
||||
- job_name: "comfyui"
|
||||
static_configs:
|
||||
- targets: ["comfyui:8188"]
|
||||
|
||||
- job_name: "node"
|
||||
static_configs:
|
||||
- targets: ["node-exporter:9100"]
|
||||
```
|
||||
|
||||
Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/etc/prometheus` | `/opt/prometheus/config` |
|
||||
| `/prometheus` | `/opt/prometheus/data` |
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Prometheus responds on `/-/healthy` with 200.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
|
||||
- Use recording rules for heavy queries (see [[Grafana]]).
|
||||
- Set a retention policy or the data volume grows without bound.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Grafana]] — the dashboard frontend
|
||||
- [[Observability]] — the whole stack
|
||||
@@ -0,0 +1,14 @@
|
||||
# authentik-postgres
|
||||
|
||||
The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /
|
||||
OIDC provider. It is the only stateful store in the stack (users, flows,
|
||||
sessions, events). Redis was removed in Authentik 2025.10; sessions and caching
|
||||
now live here.
|
||||
|
||||
- **Compose service:** `authentik-postgres` (in `dankclaw/compose.yaml`)
|
||||
- **Image:** `docker.io/library/postgres:16-alpine`
|
||||
- **Data:** `/home/dank/authentik/postgres` (bind mount; no named volumes)
|
||||
- **Health:** `pg_isready -d authentik -U authentik`
|
||||
|
||||
See [Authentik.md](Authentik.md) for the full architecture, setup, backups
|
||||
(`pg_dump`), and failure modes.
|
||||
@@ -0,0 +1,15 @@
|
||||
# authentik-server
|
||||
|
||||
The Authentik server — the core, the admin API, and the embedded Caddy outpost.
|
||||
This is the service the Caddy `forward_auth` vhosts proxy to
|
||||
(`authentik-server:9000`). It also serves the login page at
|
||||
`https://auth.franksdank.cloud`.
|
||||
|
||||
- **Compose service:** `authentik-server` (in `dankclaw/compose.yaml`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `server`)
|
||||
- **Port:** `9000` (published; the outpost + admin + login all live here)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker)
|
||||
- **Health:** `http://localhost:9000/-/health/`
|
||||
|
||||
See [Authentik.md](Authentik.md) for the full architecture, the Caddy
|
||||
forward_auth pattern, first-time setup, and failure modes.
|
||||
@@ -0,0 +1,14 @@
|
||||
# authentik-worker
|
||||
|
||||
The Authentik worker — background tasks: event processing, provider
|
||||
synchronization, scheduled flows, and certificate renewal. It runs the same
|
||||
image as the server; the `command: worker` selects the role.
|
||||
|
||||
- **Compose service:** `authentik-worker` (in `dankclaw/compose.yaml`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `worker`)
|
||||
- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server)
|
||||
- **No port** — the worker has no HTTP endpoint.
|
||||
|
||||
See [Authentik.md](Authentik.md) for the full architecture, setup, and failure
|
||||
modes.
|
||||
Reference in new issue
Block a user