diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 693fb39..26a0f36 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -14,30 +14,45 @@ https://github.com/open-webui/open-webui | Image | `ghcr.io/open-webui/open-webui:main` (verified from the live quadlet unit) | | Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. | | Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` | -| Default backend | `frank-large` on Dankclaw `:7770` (ROCmFPX llama.cpp fork — see the LLM knowledge base below) — reached via host LAN IP `192.168.0.143:7770` | +| Backends | Three fleet endpoints: `frank-large` (host.containers.internal:7770), `frank-medium` (192.168.0.235:7770), `frank-small` (host.containers.internal:7771) — see [The LLM backends](#the-llm-backends) | | Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) | | Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` | | Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` | ## The LLM backends -The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** -state (verified 2026-09-03). Both services listen on `:7770`, distinguished by -host; `frank-large` moved from `:52625` to `:7770` on 2026-08-22. +OpenWebUI serves the three fleet LLM endpoints. The model picker is built from +the `model` table in `webui.db` (one row per model), NOT from live connection +discovery — so the picker shows exactly the models seeded there. -| Endpoint | Model name | Host | Capability | -|----------|-----------|------|------------| -| `:7770` (ROCmFPX llama.cpp fork) | `frank-large` | Dankclaw | completion | -| `:7770` (ROCmFPX llama.cpp fork) | `frank-medium` | Dankserver | completion | +| Picker model | Connection URL | Runtime | Host | +|--------------|----------------|---------|------| +| `frank-large` | `http://host.containers.internal:7770/v1` | unsloth llama.cpp (Strix Halo iGPU, 200k ctx) | Dankclaw | +| `frank-medium` | `http://192.168.0.235:7770/v1` | vllm-radiance TP=2 (FP8, DFlash2, 262k ctx) | Dankserver | +| `frank-small` | `http://host.containers.internal:7771/v1` | llama.cpp (RTX 3090, Gemma4-12B, 262k ctx) | Dankclaw | -The default connection points at the on-host endpoint (`frank-large`). -To add the other backend, open **Admin → Connections** in the UI and point an -OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`). +Two gotchas make the wiring non-obvious: -> Both endpoints are bare-metal systemd services managed by this repository -(the former `vllmScripts` repo, merged in 2026-08-22). Model swaps, context -sizing, and spec tuning happen there. See the [LLM knowledge -base](../LLM/index.md). +- **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns + `{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the + OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses + the llama.cpp endpoints (frank-large, frank-small). The fix is to set + `model_ids` in each connection's `api_config` — when non-empty, OpenWebUI + builds the model list directly and never fetches `/v1/models`. frank-medium + (vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids` + too for consistency. +- **The host's own services are reached via `host.containers.internal`.** The + container sits on a netavark bridge; it can hairpin to a *different* host's + LAN IP (dankserver `192.168.0.235`) but not to its *own* host's non-published + ports via the host LAN IP (`192.168.0.143:7770/7771` is refused). + `host.containers.internal` (169.254.1.2) resolves to the host on the bridge + and reaches both llama.cpp services. + +The Ollama connection is disabled (`ollama.enable=false`). + +> All three endpoints are bare-metal systemd services managed by this +> repository. Model swaps, context sizing, and spec tuning happen there. See +> the [LLM knowledge base](../LLM/index.md). ## Web search @@ -91,12 +106,14 @@ The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].) - The default admin account is created on first login; secure it immediately. - If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready. -- The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`. +- The on-host LLM services (frank-large `:7770`, frank-small `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend (frank-medium) is a different host, reached via its LAN IP `192.168.0.235:7770`. ## History - **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns. +- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-small` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`. + ## See Also - [[Dankclaw Stack]]