dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(frank-zealot): Bonsai-2-9B PQ2_0 with the MTP head at 3 x 262,144

Browse Source 
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.

MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.

Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.

contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
This commit is contained in:
Frank committed 2026-10-18 13:15:18 -07:00
1 parent 2b127163c8
commit 466fd477d8
14 files changed
+1001 -314
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/Dankclaw/OpenWebUI.md
+55 -26
View File
@@ -21,10 +21,10 @@ https://github.com/open-webui/open-webui
| Item | Value |
|------|-------|
| Image | `ghcr.io/open-webui/open-webui:main` (verified from the live quadlet unit) |
| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
| Deployed build | `0.11.4` — `GET http://127.0.0.1:3005/api/version` → `{"version":"0.11.4"}` (2026-10-18). `:main` is the rolling image, not a stable pin, so the running build lags upstream: releases `0.12.0` (2026-10-10) through `0.12.3` (2026-10-16) exist and are not deployed (https://github.com/open-webui/open-webui/releases). The 2026-09-21 `0.11.4` security/access-control advisory is what pinned this era of the page. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
| Backends | Picker exposes exactly one model — `frank-zealot` (host.containers.internal:7771). The `frank-dragoon` and `frank-archon` connections remain configured in the DB, disabled (2026-10-07) — see [The LLM backends](#the-llm-backends) |
| Backends | The picker's content is `webui.db` state, not repo state: as recorded on 2026-10-07 it exposes **one** model, `frank-zealot`, with `frank-dragoon`/`frank-archon` configured but disabled — *unverified 2026-10-18* (the DB is the authority and is not readable from the repo). What the repo does fix is the container's own OpenAI target: `OPENAI_API_BASE_URL=http://192.168.0.143:7770` (`dankclaw/open-webui/quadlet/open-webui.container:32`, same value in the live container env) — see [The LLM backends](#the-llm-backends) |
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
@@ -34,19 +34,30 @@ https://github.com/open-webui/open-webui
The picker is built from the `model` table in `webui.db` (one row per model)
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
connection discovery — so the picker shows exactly what is seeded there.
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`**.
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unverified
2026-10-18: `webui.db` is the authority; the fleet itself serves all three ids
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
`shared/homepage/services.json`)*.
| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host |
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host |
|---|---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 3×262,144 per `scripts/frank-zealot.env`) | Dankclaw |
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
bridge move; on this network they are wrong — see the reachability gotcha below, which
is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771`
closed the same day (production unit stopped 03:38 per the repo Benchmarking
Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
PQ2_0 with the MTP head, 3 × 262,144 — was **deployed as production**, so the
picker row below was re-keyed to it (see the next paragraph).
The `model` table carries exactly one decoration row, `frank-zealot`
(context length 200,704 = the per-slot window; description since 2026-10-17:
Ternary-Bonsai-2-27B PQ2_0 + Q8_0 vision mmproj — the row's `description` is
live DB state, so a payload swap must re-key it in the database too, not only
in the repo).
(context length 262,144 = the per-slot window since 2026-10-18; description
re-keyed the same day to Ternary-Bonsai-2-9B PQ2_0 + MTP + Q8_0 vision mmproj —
the row's `description` is live DB state, so a payload swap must re-key it in
the database too, not only in the repo).
A third gate sits *after* the table, and it is the one that bites family
accounts (2026-10-07): a model row is only visible to a non-admin user when
@@ -66,15 +77,22 @@ Two gotchas make the wiring non-obvious:
OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
the llama.cpp endpoints (frank-dragoon, frank-zealot). The fix is to set
`model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
builds the model list directly and never fetches `/v1/models`. frank-medium
(vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
too for consistency.
- **The host's own services are reached via `host.containers.internal`.** The
container sits on a netavark bridge; it can hairpin to a *different* host's
LAN IP (dankserver `192.168.0.235`) but not to its *own* host's non-published
ports via the host LAN IP (`192.168.0.143:7770/7771` is refused).
`host.containers.internal` (169.254.1.2) resolves to the host on the bridge
and reaches both llama.cpp services.
builds the model list directly and never fetches `/v1/models`. The old
`frank-medium` (vLLM) endpoint — retired, no longer in the fleet — served the
OpenAI shape and would have discovered fine, but used `model_ids` too.
- **The host's own services are reached via the host LAN IP, not
`host.containers.internal`.** The container sits on the pinned kernel bridge
`systemd-open-webui` (10.89.3.0/24, gateway 10.89.3.1, container IP 10.89.3.2 —
`podman inspect open-webui` 2026-10-18), and `host.containers.internal`
(169.254.1.2) is a **slirp4netns/pasta-only name that does not resolve on this
bridge** (`dankclaw/open-webui/quadlet/open-webui.network:15-19`). The deployed
unit therefore points at the host LAN IP: `OPENAI_API_BASE_URL=http://192.168.0.143:7770`
and `SEARXNG_QUERY_URL=http://192.168.0.143:8081/search`
(`open-webui.container:32,78`, both present in the live container env, 2026-10-18).
Those ports are published on all interfaces, so the LAN IP answers:
`ss -ltn` shows `*:7770` and `0.0.0.0:7771`, and `curl http://192.168.0.143:7771/health`
→ 200 (2026-10-18). The Dankserver backend is a different host and is reached
through its LAN IP `192.168.0.235:7770` either way.
The Ollama connection is disabled (`ollama.enable=false`).
@@ -140,8 +158,14 @@ role (`ui.default_user_role=pending`), which cannot chat. Verified
`GET /api/v1/models/list`, `GET /api/v1/models` and
`POST /api/chat/completions` all return **401** with no key (also with a
bogus bearer token); only `/api/config`, `/api/version` and `/health`
answer publicly, and none of them expose the model list. An external
program needs an account + API key — there is no keyless access.
`POST /api/chat/completions` all return **401** with no key (also with a bogus bearer
token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of
them expose the model list. An external program needs an account + API key — there is
no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`,
`/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and
`POST /api/chat/completions` → 401; `/api/config` additionally publishes
`features.enable_login_form=false`, `features.enable_signup=false` and
`oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside.
## Web search
@@ -149,8 +173,11 @@ program needs an account + API key — there is no keyless access.
Chat web search is served by [[SearXNG]] on this host, wired in the quadlet
(`ENABLE_WEB_SEARCH`, `WEB_SEARCH_ENGINE=searxng`, `SEARXNG_QUERY_URL`).
- **Access path:** the backend calls `http://192.168.0.143:8081/search?format=json`
directly — gluetun's published port, NOT the `search.franksdank.cloud` vhost.
- **Access path:** the quadlet sets `SEARXNG_QUERY_URL=http://192.168.0.143:8081/search`
(`dankclaw/open-webui/quadlet/open-webui.container:78`, live in the container env) —
gluetun's published port, NOT the `search.franksdank.cloud` vhost. The JSON format
parameter is appended by OpenWebUI itself *(unverified 2026-10-18: OpenWebUI-side
behaviour; the env value carries no `format=json`)*.
The Anubis→Authentik guard chain is a browser mechanism (proof-of-work +
session flow); a plain HTTP client cannot solve it, and the internal port
bypasses Caddy entirely.
@@ -247,9 +274,11 @@ chat LLMs only), so ComfyUI needs its own image-generation connection
configured in OpenWebUI. ComfyUI itself runs GPU-enabled on Dankclaw and
serves `:8188`.
Endpoint: `POST /api/images/generations` (image `0.11.x`: upstream moved the
route to `POST /api/v1/images/generations`; the `comfyui` engine is still
upstream) with `Authorization: Bearer <token>` and body
Endpoint: `POST /api/v1/images/generations` — measured on the running build
(`0.11.4`): the legacy `POST /api/images/generations` answers **405** while the `/v1`
route answers **401** without a token, i.e. only the `/v1` path exists (2026-10-18).
The `comfyui` engine is still upstream. Call it with `Authorization: Bearer <token>`
and body
`{"model":"imagegen","prompt":"…","n":1,"size":"1024x1024"}`; the response
carries base64 image data or a URL.
@@ -267,7 +296,7 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
- The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via `host.containers.internal`, NOT the host LAN IP: the container's netavark bridge cannot hairpin to its own host's non-published ports via `192.168.0.143` (connection refused), and the bridge gateway (`10.89.3.1`) lives in the rootless netns while llama-server runs in the host netns, so it refuses too. `host.containers.internal` (169.254.1.2) resolves to the host on the bridge and works — verified end-to-end for both endpoints. The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
## History