fix: basicauth double-prefix render; open-webui off slirp4netns
basicauth (2026-08-22 incident): - the Caddyfile template carried its own $2a$14$ prefix and deploy.sh substituted the full hash (prefix included), rendering an invalid double-prefixed hash. Every basicauth vhost rejected every password with 401 and Caddy logged nothing. - template is now a bare REPLACE_WITH_HASH token; tests pin the render contract (no prefix in template; rendered line is exactly 'dank <hash>'). open-webui (2026-08-22 incident): - its slirp4netns helper process died, leaving the container running with only 'lo' in its netns: port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days, unlogged. - move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24, gw 10.89.3.1) — no per-container helper to die. - LLM endpoint reached via the gateway (host.containers.internal is slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625. - add a quadlet healthcheck: /health (app) + non-loopback interface (network liveness; the dead-tap state leaves only 'lo'). - deploy script now copies the .network file too. - docs updated (OpenWebUI History section, Caddy basic-auth contract).
This commit is contained in:
1 parent
5c9ade4e7a
commit
da68b5ce1a
9 files changed
+129
-27
No files matched your search
@@ -5,10 +5,18 @@
|
||||
#
|
||||
# Chat UI for the LLM fleet. It runs on the CPU image (dankclaw is CPU-only, so
|
||||
# the :cuda image would waste a large pull for a GPU it cannot use) and points
|
||||
# its OpenAI connection at the live vLLM backend on this host
|
||||
# (host.containers.internal:52625, model frank-default-large). It is isolated on
|
||||
# slirp4netns and publishes 3005 (3000 is Gitea's HTTP port). Data persists under
|
||||
# its OpenAI connection at the live vLLM backend on this host via the bridge
|
||||
# gateway (10.89.3.1:52625, model frank-default-large). It publishes 3005
|
||||
# (3000 is Gitea's HTTP port). Data persists under
|
||||
# /home/dank/Containers/open-webui/data. See docs/Dankclaw/OpenWebUI.md.
|
||||
#
|
||||
# Network history (2026-08-22): this unit used Network=slirp4netns. When the
|
||||
# slirp4netns helper process died, the container kept running but lost its tap
|
||||
# interface: a zombie that accepted TCP and reset every request, and chat
|
||||
# served empty responses for ~2 days with no error anywhere. It now runs on a
|
||||
# pinned bridge network (open-webui.network), which has no per-container
|
||||
# helper to die. The healthcheck probes /health (app liveness) and requires a
|
||||
# non-loopback interface (network liveness).
|
||||
|
||||
[Unit]
|
||||
Description=open-webui chat UI container
|
||||
@@ -16,14 +24,22 @@ After=network.target
|
||||
|
||||
[Container]
|
||||
ContainerName=open-webui
|
||||
Network=slirp4netns
|
||||
Network=open-webui.network
|
||||
Image=ghcr.io/open-webui/open-webui:main
|
||||
PublishPort=3005:8080
|
||||
Environment=OPENAI_API_BASE_URL=http://host.containers.internal:52625
|
||||
Environment=OPENAI_API_BASE_URL=http://10.89.3.1:52625
|
||||
Environment=ANONYMIZED_TELEMETRY=False
|
||||
Environment=DO_NOT_TRACK=True
|
||||
Volume=/home/dank/Containers/open-webui/data:/app/backend/data
|
||||
|
||||
# App liveness (/health) AND network liveness (a non-loopback interface must
|
||||
# exist; a dead tap leaves only "lo" behind).
|
||||
HealthCmd=CMD-SHELL python -c 'import os,urllib.request,sys; urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=5).read(); sys.exit(0 if any(i!="lo" for i in os.listdir("/sys/class/net")) else 1)'
|
||||
HealthInterval=30s
|
||||
HealthTimeout=10s
|
||||
HealthRetries=3
|
||||
HealthStartPeriod=300s
|
||||
|
||||
[Service]
|
||||
TimeoutStartSec=900
|
||||
Restart=always
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
# Quadlet network: private bridge for open-webui (rootless).
|
||||
#
|
||||
# Install path on dankclaw: ~/.config/containers/systemd/open-webui.network
|
||||
# Unit name: open-webui-network.service (podman network name: systemd-open-webui)
|
||||
#
|
||||
# Why a bridge instead of slirp4netns (2026-08-22 incident):
|
||||
# slirp4netns runs a user-space helper process per container. When that
|
||||
# helper died, the container kept running but lost its tap interface,
|
||||
# becoming a zombie that accepted TCP and reset every request. chat
|
||||
# served empty responses for ~2 days and nothing logged an error.
|
||||
# A kernel bridge has no per-container helper, so the failure mode
|
||||
# cannot occur.
|
||||
#
|
||||
# The subnet is pinned so the gateway IP is deterministic: the LLM
|
||||
# backend (llama-server :52625, bound 0.0.0.0 on the host) is reached
|
||||
# from the container via the gateway. host.containers.internal is a
|
||||
# slirp4netns-only name and does not resolve on a bridge.
|
||||
#
|
||||
# Subnet registry on dankclaw (rootless):
|
||||
# 10.88.0.0/16 default "podman" bridge
|
||||
# 10.89.0.0/24 gitea (systemd-gitea)
|
||||
# 10.89.1.0/24 compose stack (dankclaw_default)
|
||||
# 10.89.2.0/24 searxng (systemd-searxng)
|
||||
# 10.89.3.0/24 open-webui (systemd-open-webui) <- this file
|
||||
|
||||
[Network]
|
||||
Driver=bridge
|
||||
Subnet=10.89.3.0/24
|
||||
Gateway=10.89.3.1
|
||||
Reference in new issue
Block a user