Frank
33eae71d33
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
239 lines
15 KiB
Markdown
239 lines
15 KiB
Markdown
---
|
||
type: reference
|
||
description: "OpenWebUI is a self-hosted chat interface for LLMs."
|
||
tags:
|
||
- dankclaw
|
||
- openwebui
|
||
timestamp: 2026-09-21
|
||
---
|
||
|
||
# OpenWebUI
|
||
|
||
OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compatible endpoints (vLLM, llama.cpp, Ollama) and adds RAG, plugins, tool calling, and multi-user support.
|
||
|
||
**Host:** Dankclaw — **Port:** 3005 (host) → 8080 (container) — **Vhost:**
|
||
`ai.franksdank.cloud` routes straight to `192.168.0.143:3005` (unguarded —
|
||
OpenWebUI enforces its own auth) — **Boot-persistent:** Quadlet — **Docs:**
|
||
https://github.com/open-webui/open-webui
|
||
|
||
## Facts
|
||
|
||
| Item | Value |
|
||
|------|-------|
|
||
| Image | `ghcr.io/open-webui/open-webui:main` (verified from the live quadlet unit) |
|
||
| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
|
||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||
| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) |
|
||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||
|
||
## The LLM backends
|
||
|
||
OpenWebUI serves the three fleet LLM endpoints. The model picker is built from
|
||
the `model` table in `webui.db` (one row per model), NOT from live connection
|
||
discovery — so the picker shows exactly the models seeded there.
|
||
|
||
| Picker model | Connection URL | Runtime | Host |
|
||
|--------------|----------------|---------|------|
|
||
| `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw |
|
||
| `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver |
|
||
| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw |
|
||
|
||
Two gotchas make the wiring non-obvious:
|
||
|
||
- **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
|
||
`{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
|
||
OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
|
||
the llama.cpp endpoints (frank-large, frank-small). The fix is to set
|
||
`model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
|
||
builds the model list directly and never fetches `/v1/models`. frank-medium
|
||
(vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
|
||
too for consistency.
|
||
- **The host's own services are reached via `host.containers.internal`.** The
|
||
container sits on a netavark bridge; it can hairpin to a *different* host's
|
||
LAN IP (dankserver `192.168.0.235`) but not to its *own* host's non-published
|
||
ports via the host LAN IP (`192.168.0.143:7770/7771` is refused).
|
||
`host.containers.internal` (169.254.1.2) resolves to the host on the bridge
|
||
and reaches both llama.cpp services.
|
||
|
||
The Ollama connection is disabled (`ollama.enable=false`).
|
||
|
||
> All three endpoints are bare-metal systemd services managed by this
|
||
> repository. Model swaps, context sizing, and spec tuning happen there. See
|
||
> the [LLM knowledge base](../LLM/index.md).
|
||
|
||
## Container introspection (DB and backend python)
|
||
|
||
- Working dir for imports: `/app/backend`; package name `open_webui`
|
||
(underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set
|
||
BEFORE importing any `open_webui` module (import-time hard requirement);
|
||
the secret lives at `/app/backend/.webui_secret_key`.
|
||
- **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3`
|
||
module (`podman exec open-webui python3 -c "import sqlite3;
|
||
conn = sqlite3.connect('/app/backend/data/webui.db'); …"`). For config
|
||
keys use parameterized LIKE queries (avoids shell quoting).
|
||
- Backend python: `podman exec -w /app/backend open-webui python3 -c "…"` —
|
||
`Models.get_all_models()` is a coroutine (await it); model objects are
|
||
pydantic `ModelModel` instances (attribute access, not `.get()`), with NO
|
||
`status`/`enabled` attribute — use `is_active` from the DB.
|
||
- **API auth for verification:** `create_token` lives in
|
||
`open_webui.utils.auth` (NOT `open_webui.utils.utils`);
|
||
`get_verified_user` requires an `id` claim (not just `sub`);
|
||
`SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the
|
||
container:
|
||
|
||
```
|
||
podman exec -i open-webui sh -c 'export WEBUI_SECRET_KEY=$(cat /app/backend/.webui_secret_key); \
|
||
python -c "from open_webui.utils.auth import create_token; \
|
||
print(create_token({\"id\":\"<UID>\",\"sub\":\"<UID>\",\"email\":\"…\",\"role\":\"admin\"}))"'
|
||
```
|
||
|
||
(admin UID from the `user` table).
|
||
|
||
## Verification battery
|
||
|
||
1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended
|
||
models. Response key is `items`, not `data`.
|
||
2. `POST /openai/chat/completions` per model with a trivial prompt ("Reply
|
||
with exactly one word: pong") → each returns content `pong`. Reasoning
|
||
models need `max_tokens>=100` — the thinking block eats the budget.
|
||
3. A 500 "Server Connection Error" → `podman logs open-webui` for
|
||
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
|
||
`host.containers.internal`), not a model problem.
|
||
|
||
**Durability:** the model table + connections are DB-only (not git-managed);
|
||
a fresh data volume loses them. For reproducibility, add an idempotent
|
||
first-start seed script or document the setup in the service doc.
|
||
|
||
## Web search
|
||
|
||
Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
|
||
(`ENABLE_WEB_SEARCH`, `WEB_SEARCH_ENGINE=searxng`, `SEARXNG_QUERY_URL`).
|
||
|
||
- **Access path:** the backend calls `http://192.168.0.143:8081/search?format=json`
|
||
directly — gluetun's published port, NOT the `search.franksdank.cloud` vhost.
|
||
The Anubis→Authentik guard chain is a browser mechanism (proof-of-work +
|
||
session flow); a plain HTTP client cannot solve it, and the internal port
|
||
bypasses Caddy entirely.
|
||
- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`,
|
||
a hardcoded Python constant — no config knob). The `pass_ip` list is checked
|
||
FIRST and short-circuits the whole botdetection chain, and OpenWebUI's bridge
|
||
egress (`10.89.x`) is in `pass_ip`, so the cap never applies. Verified: an
|
||
8-request burst from the container returned 8×200.
|
||
- **Config storage:** OpenWebUI keeps web-search settings per-key in its SQLite
|
||
config table (`webui.db`, `web.search.*`). The quadlet env vars seed that
|
||
table on first start (fill-missing only); afterwards the DB is authoritative
|
||
and Admin-UI edits persist there.
|
||
|
||
## SSO (authentik)
|
||
|
||
Login is authentik via the OIDC authorization-code flow (wired 2026-09-24).
|
||
The vhost stays exempt from the Anubis guard chain: the OAuth dance is
|
||
browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
|
||
|
||
- **authentik objects** (created via the admin API, 2026-09-24): application
|
||
`open-webui` (slug `open-webui`) with the OAuth2 provider `Open WebUI`
|
||
(confidential client, `authorization_code` + `refresh_token`, Strict
|
||
redirect URI `https://ai.franksdank.cloud/oauth/oidc/callback`, implicit
|
||
consent flow). The client ID is in the quadlet; the client secret lives in
|
||
the git-ignored host file `/home/dank/Containers/open-webui/env` (mode 600,
|
||
loaded via `EnvironmentFile=`).
|
||
- **Roles:** the scope mapping `Open WebUI roles` (scope `roles`) reads the
|
||
application entitlements `Open WebUI Users` → `user` and `Open WebUI
|
||
Admins` → `admin`. The entitlements are bound to the groups
|
||
`openwebui_users` / `openwebui_admins`. `akadmin` (frank8465@protonmail.com)
|
||
is in `openwebui_admins`; `qa-e2e-admin` is in `openwebui_users`.
|
||
- **Account merge:** `OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true` folds an SSO login
|
||
into the existing local account with the same email — the pre-SSO admin
|
||
(frank8465@protonmail.com) keeps its data. `ENABLE_OAUTH_SIGNUP=true` lets
|
||
other `openwebui_users` create accounts on first login.
|
||
- **IdP reachability:** the container is on its own bridge (10.89.3.0/24) and
|
||
cannot reach `authentik-server:9000` by name. It reaches the IdP over the
|
||
public URL `https://auth.franksdank.cloud` — Caddy publishes `0.0.0.0:443`
|
||
(netavark DNATs published ports from other containers) and Pi-hole resolves
|
||
the name to the host LAN IP. The server-side code exchange rides that path.
|
||
- **Config authority:** the OAuth env vars in the quadlet seed the per-key
|
||
config DB on first run (fill-missing only); on the live data dir the Admin
|
||
Panel is authoritative. To rotate the client secret: create a new token in
|
||
authentik, update the host env file, and set the new value in the Admin
|
||
Panel (or the DB), then restart the container.
|
||
- **SSO-only (2026-09-24):** after the OIDC round-trip was verified, the local
|
||
username/password path was retired. `ENABLE_LOGIN_FORM=false` hides the form
|
||
and `ENABLE_PASSWORD_AUTH=false` refuses password logins, so the only way in
|
||
is the authentik OIDC flow. The `ui.enable_login_form` config-DB override was
|
||
flipped to `false` in the same change — the per-key DB wins over env, so both
|
||
must agree, and the DB value is the one to change on a live data dir. To
|
||
restore the password form: set `ui.enable_login_form` back to `true` in the
|
||
Admin Panel (or DB) and drop the two env vars from the quadlet.
|
||
|
||
|
||
## Boot persistence (Quadlet)
|
||
|
||
The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
|
||
|
||
## Volumes
|
||
|
||
| Container path | Host path |
|
||
|----------------|-----------|
|
||
| `/app/backend/data` | `/home/dank/Containers/open-webui/data` |
|
||
|
||
The data dir holds the database, RAG indexes, and uploads. The deploy script creates it if missing and preserves it across reruns.
|
||
|
||
## Healthcheck
|
||
|
||
The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start period). It checks two things from inside the container:
|
||
|
||
1. **App liveness** — `GET http://127.0.0.1:8080/health` returns 200 (`{"status":true}`).
|
||
2. **Network liveness** — a non-loopback interface exists in `/sys/class/net`. The dead-tap zombie state (History) leaves only `lo` behind, and this is the check that catches it.
|
||
|
||
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
|
||
|
||
## Image generation (ComfyUI)
|
||
|
||
Image generation rides the local [[ComfyUI]] integration, not the chat
|
||
picker — ComfyUI's models never appear in the chat model picker (that is
|
||
chat LLMs only), so ComfyUI needs its own image-generation connection
|
||
configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
|
||
serves `:8188`.
|
||
|
||
Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
|
||
route to `POST /api/v1/images/generations`; the `comfyui` engine is still
|
||
upstream) with `Authorization: Bearer <token>` and body
|
||
`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
|
||
carries base64 image data or a URL.
|
||
|
||
Failure modes: connection refused ⇒ OpenWebUI or ComfyUI is down; timeouts
|
||
on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||
|
||
## Retired instances
|
||
|
||
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
|
||
- The bare-container `open-webui` on Dankserver (port 3000) pointed at a dead backend (`:9090`) and was removed. Its data dir is preserved on Dankserver in case it is needed.
|
||
- The bare-container `searxng` on Dankserver (port 8081) was removed. SearXNG now runs only on Dankclaw behind Caddy.
|
||
|
||
## Gotchas
|
||
|
||
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
|
||
- The default admin account is created on first login; secure it immediately.
|
||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
||
- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`.
|
||
|
||
## History
|
||
|
||
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
|
||
|
||
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
|
||
|
||
- **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
|
||
|
||
- **2026-09-24 — flipped to SSO-only.** The OIDC round-trip verified in a real browser (login view renders no password form, only "Continue with authentik", and the flow lands on a signed-in session), so the local username/password path was retired: `ENABLE_LOGIN_FORM=false` + `ENABLE_PASSWORD_AUTH=false` in the quadlet and `ui.enable_login_form` set to `false` in the config DB (the DB wins over env, so both were changed together). `qa-e2e-admin` was demoted out of the `authentik Admins` group (it keeps `openwebui_users` / `homepage-admins` for app access).
|
||
|
||
## See Also
|
||
|
||
- [[Dankclaw Stack]]
|
||
- [[Hermes]] — the agent OpenWebUI can talk to
|
||
- [[SearXNG]] — the search engine on this host
|
||
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
|