Frank
da68b5ce1a
basicauth (2026-08-22 incident): - the Caddyfile template carried its own $2a$14$ prefix and deploy.sh substituted the full hash (prefix included), rendering an invalid double-prefixed hash. Every basicauth vhost rejected every password with 401 and Caddy logged nothing. - template is now a bare REPLACE_WITH_HASH token; tests pin the render contract (no prefix in template; rendered line is exactly 'dank <hash>'). open-webui (2026-08-22 incident): - its slirp4netns helper process died, leaving the container running with only 'lo' in its netns: port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days, unlogged. - move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24, gw 10.89.3.1) — no per-container helper to die. - LLM endpoint reached via the gateway (host.containers.internal is slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625. - add a quadlet healthcheck: /health (app) + non-loopback interface (network liveness; the dead-tap state leaves only 'lo'). - deploy script now copies the .network file too. - docs updated (OpenWebUI History section, Caddy basic-auth contract).
29 lines
1.2 KiB
SYSTEMD
29 lines
1.2 KiB
SYSTEMD
# Quadlet network: private bridge for open-webui (rootless).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/open-webui.network
|
|
# Unit name: open-webui-network.service (podman network name: systemd-open-webui)
|
|
#
|
|
# Why a bridge instead of slirp4netns (2026-08-22 incident):
|
|
# slirp4netns runs a user-space helper process per container. When that
|
|
# helper died, the container kept running but lost its tap interface,
|
|
# becoming a zombie that accepted TCP and reset every request. chat
|
|
# served empty responses for ~2 days and nothing logged an error.
|
|
# A kernel bridge has no per-container helper, so the failure mode
|
|
# cannot occur.
|
|
#
|
|
# The subnet is pinned so the gateway IP is deterministic: the LLM
|
|
# backend (llama-server :52625, bound 0.0.0.0 on the host) is reached
|
|
# from the container via the gateway. host.containers.internal is a
|
|
# slirp4netns-only name and does not resolve on a bridge.
|
|
#
|
|
# Subnet registry on dankclaw (rootless):
|
|
# 10.88.0.0/16 default "podman" bridge
|
|
# 10.89.0.0/24 gitea (systemd-gitea)
|
|
# 10.89.1.0/24 compose stack (dankclaw_default)
|
|
# 10.89.2.0/24 searxng (systemd-searxng)
|
|
# 10.89.3.0/24 open-webui (systemd-open-webui) <- this file
|
|
|
|
[Network]
|
|
Driver=bridge
|
|
Subnet=10.89.3.0/24
|
|
Gateway=10.89.3.1 |