fix: basicauth double-prefix render; open-webui off slirp4netns
basicauth (2026-08-22 incident): - the Caddyfile template carried its own $2a$14$ prefix and deploy.sh substituted the full hash (prefix included), rendering an invalid double-prefixed hash. Every basicauth vhost rejected every password with 401 and Caddy logged nothing. - template is now a bare REPLACE_WITH_HASH token; tests pin the render contract (no prefix in template; rendered line is exactly 'dank <hash>'). open-webui (2026-08-22 incident): - its slirp4netns helper process died, leaving the container running with only 'lo' in its netns: port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days, unlogged. - move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24, gw 10.89.3.1) — no per-container helper to die. - LLM endpoint reached via the gateway (host.containers.internal is slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625. - add a quadlet healthcheck: /health (app) + non-loopback interface (network liveness; the dead-tap state leaves only 'lo'). - deploy script now copies the .network file too. - docs updated (OpenWebUI History section, Caddy basic-auth contract).
This commit is contained in:
Frank
@@ -11,8 +11,8 @@ OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compat
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Default backend | vLLM on Dankclaw `:52625` = `frank-default-large` (via `host.containers.internal:52625`) |
|
||||
| Network | `slirp4netns` (isolated) |
|
||||
| Default backend | vLLM on Dankclaw `:52625` = `frank-default-large` (via bridge gateway `10.89.3.1:52625`) |
|
||||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||||
|
||||
@@ -43,7 +43,12 @@ The data dir holds the database, RAG indexes, and uploads. The deploy script cre
|
||||
|
||||
## Healthcheck
|
||||
|
||||
OpenWebUI responds on `/health` with 200. Probe it at `http://dankclaw:3005/health`.
|
||||
The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start period). It checks two things from inside the container:
|
||||
|
||||
1. **App liveness** — `GET http://127.0.0.1:8080/health` returns 200 (`{"status":true}`).
|
||||
2. **Network liveness** — a non-loopback interface exists in `/sys/class/net`. The dead-tap zombie state (History) leaves only `lo` behind, and this is the check that catches it.
|
||||
|
||||
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
|
||||
|
||||
## Retired instances
|
||||
|
||||
@@ -56,7 +61,11 @@ OpenWebUI responds on `/health` with 200. Probe it at `http://dankclaw:3005/heal
|
||||
- Use the non-CUDA image on this CPU-only host.
|
||||
- The default admin account is created on first login; secure it immediately.
|
||||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
||||
- `host.containers.internal` reaches only the podman host. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
|
||||
- The LLM is reached via the bridge gateway `10.89.3.1`, not `host.containers.internal` (that name is slirp4netns-only and does not resolve on a bridge). To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
@@ -50,17 +50,19 @@ The bare-IP catch-all `:80` 302-redirects to `https://dankclaw.franksdank.cloud/
|
||||
|
||||
## Basic auth
|
||||
|
||||
Generate a bcrypt hash, then reference it:
|
||||
The Caddyfile in git carries a **bare** placeholder; deploy.sh substitutes the **full** hash from the host `.env` (`CADDY_BASICAUTH_HASH`):
|
||||
|
||||
```
|
||||
chat.franksdank.cloud {
|
||||
basicauth {
|
||||
dank $2a$14$xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
dank REPLACE_WITH_HASH
|
||||
}
|
||||
reverse_proxy 192.168.0.143:3005
|
||||
}
|
||||
```
|
||||
|
||||
After rendering, the line reads `dank $2a$14$xxxx…` (one complete hash). Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password). `tests/deployment_flow.bats` pins the render contract.
|
||||
|
||||
The homepage also uses basic auth; see [[Homepage]].
|
||||
|
||||
## Operations
|
||||
@@ -75,6 +77,7 @@ Certificates and ACME state live in the caddy data volume. Back it up with the s
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
|
||||
- A new vhost needs three things: a dnsmasq `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]].
|
||||
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the dnsmasq role in [[LAN-DNS]].
|
||||
- Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
|
||||
|
||||
Reference in New Issue
Block a user