fix: basicauth double-prefix render; open-webui off slirp4netns

basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
  substituted the full hash (prefix included), rendering an invalid
  double-prefixed hash. Every basicauth vhost rejected every password
  with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
  contract (no prefix in template; rendered line is exactly 'dank <hash>').

open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
  with only 'lo' in its netns: port 3005 accepted TCP and reset every
  request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
  gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
  slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
  (network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
This commit is contained in:
Frank committed 2026-08-22 13:07:43 -07:00
1 parent 5c9ade4e7a
commit da68b5ce1a
9 files changed
+129 -27

No files matched your search

@@ -5,10 +5,18 @@
#
# Chat UI for the LLM fleet. It runs on the CPU image (dankclaw is CPU-only, so
# the :cuda image would waste a large pull for a GPU it cannot use) and points
# its OpenAI connection at the live vLLM backend on this host
# (host.containers.internal:52625, model frank-default-large). It is isolated on
# slirp4netns and publishes 3005 (3000 is Gitea's HTTP port). Data persists under
# its OpenAI connection at the live vLLM backend on this host via the bridge
# gateway (10.89.3.1:52625, model frank-default-large). It publishes 3005
# (3000 is Gitea's HTTP port). Data persists under
# /home/dank/Containers/open-webui/data. See docs/Dankclaw/OpenWebUI.md.
#
# Network history (2026-08-22): this unit used Network=slirp4netns. When the
# slirp4netns helper process died, the container kept running but lost its tap
# interface: a zombie that accepted TCP and reset every request, and chat
# served empty responses for ~2 days with no error anywhere. It now runs on a
# pinned bridge network (open-webui.network), which has no per-container
# helper to die. The healthcheck probes /health (app liveness) and requires a
# non-loopback interface (network liveness).
[Unit]
Description=open-webui chat UI container
@@ -16,14 +24,22 @@ After=network.target
[Container]
ContainerName=open-webui
Network=slirp4netns
Network=open-webui.network
Image=ghcr.io/open-webui/open-webui:main
PublishPort=3005:8080
Environment=OPENAI_API_BASE_URL=http://host.containers.internal:52625
Environment=OPENAI_API_BASE_URL=http://10.89.3.1:52625
Environment=ANONYMIZED_TELEMETRY=False
Environment=DO_NOT_TRACK=True
Volume=/home/dank/Containers/open-webui/data:/app/backend/data
# App liveness (/health) AND network liveness (a non-loopback interface must
# exist; a dead tap leaves only "lo" behind).
HealthCmd=CMD-SHELL python -c 'import os,urllib.request,sys; urllib.request.urlopen("http://127.0.0.1:8080/health",timeout=5).read(); sys.exit(0 if any(i!="lo" for i in os.listdir("/sys/class/net")) else 1)'
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=300s
[Service]
TimeoutStartSec=900
Restart=always
@@ -0,0 +1,29 @@
# Quadlet network: private bridge for open-webui (rootless).
#
# Install path on dankclaw: ~/.config/containers/systemd/open-webui.network
# Unit name: open-webui-network.service (podman network name: systemd-open-webui)
#
# Why a bridge instead of slirp4netns (2026-08-22 incident):
# slirp4netns runs a user-space helper process per container. When that
# helper died, the container kept running but lost its tap interface,
# becoming a zombie that accepted TCP and reset every request. chat
# served empty responses for ~2 days and nothing logged an error.
# A kernel bridge has no per-container helper, so the failure mode
# cannot occur.
#
# The subnet is pinned so the gateway IP is deterministic: the LLM
# backend (llama-server :52625, bound 0.0.0.0 on the host) is reached
# from the container via the gateway. host.containers.internal is a
# slirp4netns-only name and does not resolve on a bridge.
#
# Subnet registry on dankclaw (rootless):
# 10.88.0.0/16 default "podman" bridge
# 10.89.0.0/24 gitea (systemd-gitea)
# 10.89.1.0/24 compose stack (dankclaw_default)
# 10.89.2.0/24 searxng (systemd-searxng)
# 10.89.3.0/24 open-webui (systemd-open-webui) <- this file
[Network]
Driver=bridge
Subnet=10.89.3.0/24
Gateway=10.89.3.1