dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initialize serverAppConfigs with podman stacks for Dankserver and Dankclaw

Browse Source 
- Dankserver stack: RomM, Copyparty, Calibre-Web, node-exporter
- Dankclaw stack: Gluetun VPN (SearXNG tunnel), SearXNG+Valkey, ComfyUI
  (CPU backend), OpenWebUI, Hermes, Gitea+Postgres (migrated), Caddy
  reverse proxy, Grafana+Prometheus+Loki+Alloy observability
- Caddyfiles with .local hostname convention and basicauth homepage
- bind-mount storage policy, no named volumes
- deploy.sh mirroring vllmScripts git workflow (push -> pull -> restart -> health)
- docs/ Obsidian vault: per-service docs from frank-default-small research
  subagents (promtail EOL -> Alloy; gitea SECRET_KEY preservation; ComfyUI --cpu)
- bats test suite at 6 seams — 22 tests green
This commit is contained in:
Frank
2026-08-13 20:19:41 -07:00
commit e3eb5334cb
57 changed files with 3033 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Infrastructure/Alloy.md
+47
View File
@@ -0,0 +1,47 @@
# Alloy
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). One Alloy instance runs per host and forwards container and journal logs to Loki, and optionally metrics to Prometheus.
**Hosts:** Dankclaw + Dankserver (one per host) — **Docs:** https://grafana.com/docs/alloy/latest
## Facts (from research)
| Item | Value |
|------|-------|
| Image | `grafana/alloy:latest` |
| Port | none published (push-only) |
| Config | River syntax, `config.alloy` |
| Successor to | Promtail (EOL 2026-03-02) |
| Inputs | journald socket, container log files |
| Output | Loki push endpoint |
## Why Alloy and not Promtail
Promtail reached end-of-life on 2026-03-02. Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
## Configuration
River config at `shared/alloy/config.alloy`:
- Read journald logs from the host's `/run/systemd/journal/socket`.
- Read container logs from `/var/lib/containers/...` (podman) or the docker path.
- Forward to `http://loki:3100/loki/api/v1/push` with `host` and `job` labels.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/etc/alloy/config.alloy` | `shared/alloy/config.alloy` (bind mount) |
| `/run/systemd/journal/socket` | host journald socket (read-only) |
| `/var/log` | host logs (read-only) |
## Gotchas
- The journald socket mount is required for journal scraping.
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
## See Also
- [[Loki]] — the log backend
- [[Observability]] — the full stack
docs/Infrastructure/Caddy.md
+71
View File
@@ -0,0 +1,71 @@
# Caddy
Caddy is the reverse proxy for both hosts. It terminates TLS, routes hostnames to containers, and protects the homepage with HTTP basic auth.
**Hosts:** Dankserver + Dankclaw — **Docs:** https://caddyserver.com/docs
## Facts (from live inventory + knowledge base)
| Item | Value |
|------|-------|
| Image | `docker.io/library/caddy:latest` |
| Config | `/etc/caddy/Caddyfile` (bind mount) |
| Data | `/root/.local/share/caddy` (certs, ACME state) |
| Ports | 80, 443 |
| Existing on Dankserver | systemd `caddy` service (native), config at `/etc/caddy/Caddyfile` — being migrated into this repo |
## Existing Dankserver Caddyfile (live, to migrate)
The live Caddyfile proxies:
- `plex.franksdank.casa` / `plex.franksdank.cloud` / `plex.dankserver.local` → `localhost:32400` (Plex)
- `books.franksdank.cloud` → `localhost:8083` (Calibre-Web)
- `llms.franksdank.cloud` / `llms.dankserver.local` → `localhost:3000` (OpenWebUI — being moved to Dankclaw)
- `search.dankserver.local` → `localhost:8081` (SearXNG — being moved to Dankclaw)
## Hostname Convention
Every service gets a `.dankserver.local` or `.dankclaw.local` hostname so no one has to remember port numbers:
```
romm.dankserver.local
books.dankserver.local
files.dankserver.local
search.dankclaw.local
chat.dankclaw.local
gitea.dankclaw.local
comfyui.dankclaw.local
grafana.dankclaw.local
prometheus.dankclaw.local
```
## Basic Auth
Generate a bcrypt hash with `caddy hash-password`, then:
```
chat.dankclaw.local {
basicauth {
dank $2a$14$xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
}
reverse_proxy localhost:3000
}
```
The homepage uses basic auth too; see [[Homepage]].
## Homepage
A static page lists all services with live status (up/down) and a short description plus a link to each service's documentation. Served by Caddy with `file_server` and protected by basic auth.
## Gotchas
- Caddy auto-provisions Let's Encrypt certs for public domains; `.local` names stay internal (no external DNS).
- Validate before reload: `caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile`.
- Certificates live in the caddy data volume; back it up with the stack.
## See Also
- [[Dankserver Stack]]
- [[Dankclaw Stack]]
- [[Homepage]]
- Knowledge base: [[Gitea Architecture]] (the existing caddy chain on Dankclaw)
docs/Infrastructure/Grafana.md
+46
View File
@@ -0,0 +1,46 @@
# Grafana
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.dankclaw.local`.
**Host:** Dankclaw — **Hostname:** `grafana.dankclaw.local` — **Port:** 3002 — **Docs:** https://grafana.com/docs
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/grafana/grafana:latest` |
| Port | 3002 (host) → 3000 (container) |
| Data volume | dashboards, users, datasource config |
| Datasource | Prometheus (on the same host) |
| Provisioning | dashboards in `/etc/grafana/provisioning/dashboards` |
## Configuration
```yaml
environment:
GF_SECURITY_ADMIN_USER: "${GRAFANA_ADMIN_USER}"
GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
GF_SERVER_ROOT_URL: "http://grafana.dankclaw.local"
volumes:
- /opt/grafana/data:/var/lib/grafana
- /opt/grafana/provisioning:/etc/grafana/provisioning:ro
```
## Datasource
Point Grafana at the Prometheus service by name (`http://prometheus:9090`) on the shared podman network.
## Healthcheck
Grafana responds on `/api/health` with 200 and `"database":"ok"`.
## Gotchas
- Change the default admin credentials at first boot (env or UI).
- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
## See Also
- [[Dankclaw Stack]]
- [[Prometheus]] — the datasource
- [[Observability]] — the whole stack
docs/Infrastructure/Homepage.md
+56
View File
@@ -0,0 +1,56 @@
# Homepage
A password-protected homepage on each host lists the services that are up, with a short description of each and a link to both the service and its documentation.
**Served by:** Caddy, `file_server` + `basicauth` — **Docs:** n/a (static page in this repo)
## Purpose
- One landing page per host instead of a bookmark folder.
- Shows at a glance which services are currently up and active.
- Answers "what is this?" for every service with a one-line description.
- Links to the service and to this repo's docs page for that service.
## Layout
```
shared/
└── homepage/
├── index.html # the page, shared logic, per-host service list injected
├── style.css
└── services.json # service list: name, hostname, port, description, docs link
```
The page fetches `services.json` and pings each service's health endpoint (or the proxy hostname) to set an up/down badge. It is static; the status check runs in the browser via `fetch`.
## Auth
Caddy wraps the homepage with HTTP basic auth:
```
dankclaw.local {
basicauth {
dank $2a$14$HASH
}
root * /srv/homepage
file_server
}
```
Generate the hash with `caddy hash-password` at deploy time. Do not commit the hash or the password.
## Service Entry Format
```json
{
"name": "SearXNG",
"hostname": "http://search.dankclaw.local",
"description": "Privacy-respecting metasearch engine.",
"docs": "/docs/Dankclaw/SearXNG.md"
}
```
## See Also
- [[Caddy]] — the proxy that serves it
- [[Dankclaw Stack]], [[Dankserver Stack]] — the services it lists
docs/Infrastructure/Loki.md
+51
View File
@@ -0,0 +1,51 @@
# Loki
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs, and it feeds the log panels in Grafana. It replaces the need for a full ELK stack.
**Host:** Dankclaw — **Docs:** https://grafana.com/docs/loki/latest
## Facts (from research)
| Item | Value |
|------|-------|
| Image | `grafana/loki:latest` |
| Port | 3100 (host) → 3100 (container) |
| Mode | single-binary (all components in one process) |
| Storage | filesystem (bind mount) |
| Auth | `auth_enabled: false` (single-tenant homelab) |
| Schema | v13 / TSDB |
## Configuration
Config lives at `shared/loki/loki-config.yaml`. Key settings:
- `schema_config` v13 with `store: tsdb` and filesystem object store.
- `compactor` with `retention_enabled: true` so logs do not grow without bound.
- `replication_factor: 1` for a single instance.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/loki` | `/opt/loki/data` |
| `/etc/loki/loki-config.yaml` | `shared/loki/loki-config.yaml` (bind mount) |
## Healthcheck
Loki responds on `/ready` with 200.
## Log Shipping
The [[Alloy]] agent on each host ships container and journal logs to Loki. Grafana queries Loki through the datasource in [[Observability]].
## Gotchas
- Filesystem storage grows; enable compactor retention.
- Loki and Promtail/Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
- For multi-host shipping, Loki's HTTP listener binds `0.0.0.0`.
## See Also
- [[Observability]]
- [[Alloy]] — the log shipper
- [[Grafana]] — the query frontend
docs/Infrastructure/Node-Exporter.md
+55
View File
@@ -0,0 +1,55 @@
# Node-Exporter
Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host inside its own container.
**Hosts:** Dankserver + Dankclaw — **Docs:** https://prometheus.io/docs/guides/node-exporter
## Facts (from research)
| Item | Value |
|------|-------|
| Image | `prom/node-exporter:latest` |
| Port | 9100 (host) |
| Network | `host` mode |
| Config | `--path.procfs=/host/proc`, `--path.sysfs=/host/sys`, `--path.rootfs=/rootfs` |
## Configuration
```yaml
services:
node-exporter:
image: prom/node-exporter:latest
restart: always
network_mode: host
pid: host
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
command:
- --path.procfs=/host/proc
- --path.sysfs=/host/sys
- --path.rootfs=/rootfs
```
## Prometheus Targets
Prometheus scrapes each host's node-exporter directly:
| Host | Target |
|------|--------|
| Dankserver | `192.168.0.235:9100` |
| Dankclaw | `192.168.0.143:9100` |
These scrape jobs live in the Prometheus config; see [[Prometheus]] and [[Observability]].
## Gotchas
- Requires `network_mode: host` (or a published port) so Prometheus can reach 9100.
- The `/proc`, `/sys`, and root mounts must be `read-only` (`:ro`).
- Do not mount the same paths on both hosts' containers; each node-exporter reflects only its own host.
## See Also
- [[Prometheus]] — the scraper
- [[Observability]] — the full stack
docs/Infrastructure/Observability.md
+53
View File
@@ -0,0 +1,53 @@
# Observability
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
## Stack
```mermaid
graph LR
S[Services] -->|/metrics| P[Prometheus]
S -->|logs| T[Promtail]
T --> L[Loki]
P --> G[Grafana]
L --> G
```
| Component | Role | Image | Port |
|-----------|------|-------|------|
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
| Loki | log storage & query | `grafana/loki` | 3100 |
| Alloy | log shipping (per host) | `grafana/alloy` | — |
## Log Aggregation Choice
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
- No Elasticsearch memory cost, no Kubernetes required.
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
## Scrape Surface
| Target | Source |
|--------|--------|
| Prometheus self | `localhost:9090/metrics` |
| ComfyUI | `/system_stats` (limited) or blackbox probe |
| Gitea | `/metrics` (built-in exporter) |
| Grafana | `/metrics` (built-in) |
| Host metrics | node-exporter `:9100` |
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
## Healthchecks
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
## See Also
- [[Grafana]], [[Prometheus]]
- [[Dankclaw Stack]]
- [[Homepage]] — the user-facing status surface
docs/Infrastructure/Prometheus.md
+57
View File
@@ -0,0 +1,57 @@
# Prometheus
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.
**Host:** Dankclaw — **Hostname:** `prometheus.dankclaw.local` — **Port:** 9090 — **Docs:** https://prometheus.io/docs
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/prom/prometheus:latest` |
| Port | 9090 (host) → 9090 (container) |
| Config | `/etc/prometheus/prometheus.yml` (bind mount) |
| Data volume | TSDB |
| Retention | `--storage.tsdb.retention.time=30d` |
## prometheus.yml Targets
```yaml
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "comfyui"
static_configs:
- targets: ["comfyui:8188"]
- job_name: "node"
static_configs:
- targets: ["node-exporter:9100"]
```
Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.
## Volumes
| Container path | Host path (convention) |
|----------------|------------------------|
| `/etc/prometheus` | `/opt/prometheus/config` |
| `/prometheus` | `/opt/prometheus/data` |
## Healthcheck
Prometheus responds on `/-/healthy` with 200.
## Gotchas
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
- Use recording rules for heavy queries (see [[Grafana]]).
- Set a retention policy or the data volume grows without bound.
## See Also
- [[Dankclaw Stack]]
- [[Grafana]] — the dashboard frontend
- [[Observability]] — the whole stack