Compare commits

...
Author SHA1 Message Date
dank 9ff7ce6a12 Merge pull request 'docs(frank-archon): record the crash signature — clean-looking exit, silent engine' (#147) from docs/frank-archon-crash-signature into main 2026-11-04 13:19:50 -08:00
Frank f25be876fb docs(frank-archon): record the crash signature — clean-looking exit, silent engine
A 2026-11-04 crash left no trace in the engine's own log and reported
Result=success with NRestarts=1: the unit's ExecStart ends in 'exec podman
wait', whose exit status is 0, while the container's real code is printed to
stdout. The only copy of the reason was the client-facing error text. Both
traps now live in the endpoint page's Laws, with the observed trigger shape
(concurrent triple-digit-thousand-token prefills) and the instruments that
ruled out OOM, GPU hang, MCE and swap pressure.
2026-11-04 13:18:43 -08:00
dank fee8121d47 Merge pull request 'docs(openwebui): re-key the model rows at swap time; default model is frank-archon' (#146) from docs/openwebui-model-rows-default into main 2026-11-04 13:08:52 -08:00
Frank 1b5c11fb60 docs(openwebui): re-key the model rows at swap time; default is frank-archon
The picker's decoration rows carried two payload-swap drifts: frank-dragoon
still named halogen 0.16.2 (live 0.18.20) and frank-zealot build 10777 /
3x262,144 / MTP off (live build 12118 / 3x200,704 / draft-mtp armed). Worse,
their params.context.length values had swapped, so OpenWebUI advertised more
window than the zealot slot grants and gave up 61k tokens on dragoon. Live
rows re-keyed through the admin API and ui.default_models set to
frank-archon; the swap procedure in AGENTS.md now names the re-key step.
2026-11-04 13:06:54 -08:00
3 changed files with 67 additions and 1 deletions

No files matched your search

+4 -1
View File
@@ -278,7 +278,10 @@ archive.
**Swap the model. Do not instrument it.** A payload change is: edit the trio
(env / serve script / unit) → `scripts/deploy.sh <host> --service <svc>` →
`/health` answers → one chat turn, one tool call, and one vision probe on an
endpoint that serves images. That is the entire procedure.
endpoint that serves images → re-key that endpoint's OpenWebUI `model` row
(`meta.description` + `params.context.length`, API door in [[OpenWebUI]]). That is
the entire procedure — no deployer step touches `webui.db`, and a stale
`context.length` makes OpenWebUI advertise a window the endpoint does not grant.
- **No benchmarking unless Frank orders it.** No pre-swap baseline battery,
no A/B arms, no decode/prefill ladders, no acceptance tables, no numbers
@@ -109,6 +109,39 @@ The Ollama connection is disabled (`ollama.enable=false`).
> connection here only hides it from this picker — the endpoint itself keeps
> serving every other client (the harness, Hermes, API consumers).
## Default model, descriptions, and the admin API door
**The default model is `ui.default_models`** — one row of the config table, a
comma-separated **string**, not a list. Live value `frank-archon` since
2026-11-04 (per Frank). The front end resolves the picker as *the browser's own
stored pick, else `config.default_models.split(",")`*, so the value decides what
a fresh profile or a cleared-storage session lands on; after a user picks
something themselves, their pick keeps winning. Verified 2026-11-04 in a clean
browser: signed in with `localStorage` empty → the pill reads
`Selected model: frank-archon`.
Two admin-token doors, both driven against loopback `:3005` — service-to-service
traffic never rides the public vhost:
- **Config:** `POST /api/v1/configs/import` with
`{"config":{"ui.default_models":"frank-archon"}}` — the general per-key setter
(no `PATCH /configs/{key}` route exists in 0.12.5; verified against
`routers/` + `main.py`), and it returns the whole config so the write reads
back in the same call.
- **A model row:** `POST /api/v1/models/model/update` with a full `ModelForm`
(`id`, `name`, `meta`, `params`, `is_active`). **`meta` and `params` REPLACE,
they do not merge:** read the row first, change one field, send everything
back — a partial payload silently drops `meta.capabilities` and the
profile/background images.
**An API key is not accepted everywhere.** Routes guarded by `get_human_user`
(`/api/v1/auths/`, `utils/auth.py`) answer **403 `A human session is required.`**
for any token whose `typ` is not `session` — that is not a dead key. The two
mutation doors above take `get_admin_user` / `get_verified_user`, which do accept
the key. Prove a key on `GET /api/v1/models` with **no trailing slash**:
`/api/v1/models/` falls through to the SPA and answers `200` + `text/html`, which
looks like success and parses as nothing.
## Container introspection (DB and backend python)
- Working dir for imports: `/app/backend`; package name `open_webui`
@@ -431,6 +464,21 @@ with frank-tts logging `speech voice=wilwheaton`.
app process env; the `.webui_secret_key` file happens to equal it — use it
explicitly, `podman exec -e`).
- **2026-11-04 — decoration rows drifted again; default model set to
`frank-archon`.** Two payload swaps had gone by without anyone touching the
picker: `frank-dragoon`'s row still advertised halogen `0.16.2`, and
`frank-zealot`'s still said `build 10777` / `3×262,144` / *MTP off* — live is
`0.18.20`, `build 12118`, `3 × 200,704`, MTP armed. The prose is cosmetic; the
`params.context.length` values had **swapped**, and that one bites:
`frank-zealot` advertised 262,144 while the endpoint refuses anything past its
200,704 slot, so a long zealot chat rides the endpoint's 400 instead of
compacting, and `frank-dragoon` advertised 200,704 against a 262,144 ceiling,
giving up 61k tokens of window for nothing. Both rows re-keyed through the API
door in [Default model, descriptions, and the admin API door](#default-model-descriptions-and-the-admin-api-door),
descriptions rewritten from the env files, and `ui.default_models` set to
`frank-archon`. **Law: a model swap is not finished until that endpoint's
`model` row is re-keyed as well — no deployer step touches `webui.db`.**
## See Also
- [[Dankclaw Stack]]
+15
View File
@@ -525,6 +525,21 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
the disk tier silently never opens. The tier does survive a payload swap when arch and
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
- **A radiance crash reads as a clean exit unless you know where to look, and the engine never
says why.** The unit's `ExecStart` ends in `exec podman wait`, so `systemctl show -p
ExecMainStatus` reports **0** after a container death — that is `podman wait`'s own status, not
the engine's — and `Result=success` beside `NRestarts=1` is the *normal* fingerprint of a
crash. The container's real code is the bare digit `podman wait` prints to stdout
(`serve-frank-archon.sh[NNNN]: 1`). The engine's fatal path writes **nothing** to its log: no
stderr line, no message before the exit and no core. The only copy of the reason is the error
the client received — `the engine stopped on an error and is not serving (<errno>)`, where the
parenthesis is the failing call's errno text (`invalid argument` = EINVAL, 2026-11-04). So
grep the *client* logs before concluding the journal is empty: omp keeps the provider error in
`~/.omp/logs/omp.<date>.<pid>.log` (`agent turn ended with provider error`), and OpenWebUI
shows it as the chat's error toast. Seen once (2026-11-04, restart counter 1 after 3d20h up):
three concurrent triple-digit-thousand-token agent turns prefilled at once, prefill rate
collapsed, all in-flight streams failed at the same instant, then a silent teardown — kernel
logged only the KFD SVM workqueue churning, with no OOM, GPU hang, MCE or swap pressure.
### Dead ends — do not re-try