docs(vault): service and infrastructure pages stop restating endpoint facts

18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
Frank
2026-10-19 00:56:02 -07:00
parent 219a097b39
commit 6bdfd2853e
18 changed files with 169 additions and 169 deletions
+25 -27
View File
@@ -14,16 +14,16 @@ registry; single source of truth — there is no copy on the servers) and
this file is repo-managed; edits are harness-side only. this file is repo-managed; edits are harness-side only.
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire **Model ids are the StarCraft unit names (official 2026-10-08, on the wire
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance), since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw `frank-zealot` (Dankclaw) — engine, port and card behind each id live on
:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`, [[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`,
`frank-zealot`. The SERVICES keep the old names (units, scripts, env `frank-zealot`. The SERVICES keep the old names (units, scripts, env
files, runbook pages); pages written in the old era name the service, files, runbook pages); pages written in the old era name the service,
the harness names the model. The wire now matches: each endpoint's the harness names the model. The wire now matches: each endpoint's
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI `/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
picker/grants, the homepage docs and the raw API all use one id per model picker/grants, the homepage docs and the raw API all use one id per model
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]] (see [[OpenWebUI]] for the grant re-key and the [[frank-archon]]
note on radiance echoing the requested name). note on the endpoint echoing the requested name).
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
192.168.0.143:7771 each answer with the unit name (`frank-archon`, 192.168.0.143:7771 each answer with the unit name (`frank-archon`,
@@ -50,10 +50,11 @@ Where the contract lives:
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js` `~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
The arithmetic (the coupled contract every endpoint's maxTokens comment The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes). Recomputed from the live files 2026-10-18 — contextWindows encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
200,000 (archon) / 155,648 (dragoon) / 200,704 (zealot) at endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
`models.yml:48,131,206`; maxTokens 16,384 / 45,056 / 16,384 at follows is a **worked example** recomputed from the live files on 2026-10-18 —
`models.yml:56,143,212`: a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The 95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
@@ -141,28 +142,25 @@ Use **wire names**, not OMP camelCase. Two traps:
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy 2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
also sets it — `Object.assign` is shallow and replaces the whole object, also sets it — `Object.assign` is shallow and replaces the whole object,
silently dropping the `reasoning_effort` kwarg the compat policy set. silently dropping the `reasoning_effort` kwarg the compat policy set.
Top-level `preserve_thinking` is honoured by halogen. Live state 2026-10-18: Top-level `preserve_thinking` is honoured by frank-dragoon
frank-archon's radiance entry carries `chat_template_kwargs: ([[frank-dragoon]]). Live state 2026-10-18: frank-archon's `extraBody`
{enable_thinking, preserve_thinking}` in `extraBody` (`models.yml:89-91`) carries `chat_template_kwargs: {enable_thinking, preserve_thinking}`
and nothing else sets that object at that endpoint, so the shallow-merge (`models.yml:89-91`) and nothing else sets that object at that endpoint, so
hazard is latent rather than firing; the FreeToken-era "verify-and-prune at the shallow-merge hazard is latent rather than firing.
the next swap" note is retired with that swap (2026-10-05,
`scripts/serve-frank-archon.sh:9-10`).
**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier flash-server **Gotcha (retired 2026-10-17): silent greedy decode.** Earlier frank-dragoon
builds carried **no server-side sampler defaults**, so an absent builds carried **no server-side sampler defaults**, so an absent
`temperature` decoded greedy while the endpoint looked configured. That era is `temperature` decoded greedy while the endpoint looked configured. That era is
over: `HALOGEN_TEMPERATURE/TOP_P/TOP_K` now set the card's thinking recipe over — the endpoint now ships its own defaults ([[frank-dragoon]]), so an
(1.0 / 0.95 / 20) as engine defaults (`scripts/frank-dragoon.env:98-103`, absent temperature is no longer greedy. frank-dragoon's `extraBody` still
exported at `scripts/serve-frank-dragoon.sh:186-188`), so an absent temperature carries the same trio — consistent, not load-bearing. Treat any older "do not
is no longer greedy. frank-dragoon's `extraBody` still carries the same trio — drop it or you get greedy" wording as history.
consistent, not load-bearing. Treat any older "do not drop it or you get
greedy" wording as history.
**Reasoning-effort folding.** Qwen3.8's template accepts only **Reasoning-effort folding.** The set `reasoning_effort` accepts is a property
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything of the endpoint, not of the harness: [[frank-dragoon]] and [[frank-zealot]]
else ("Unexpected reasoning effort high"). Every endpoint's raise on anything outside `xhigh|medium|low`, while [[frank-archon]] does not
`reasoningEffortMap` folds all harness levels onto that set; a role whose gate the set. Every endpoint's `reasoningEffortMap` folds all harness levels
onto whichever set that endpoint ships; a role whose
level is unmapped fails on its FIRST request, and the failure names Jinja, level is unmapped fails on its FIRST request, and the failure names Jinja,
not the model. not the model.
@@ -22,8 +22,10 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
"behavior pending". "behavior pending".
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service **Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of whose marker lags. On the big-payload endpoints that is a full model reload
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the measured in tens of minutes — each endpoint's boot budget is on its own page
([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon
(dankserver:7770) is also the
harness's own model backend, so a blind deploy is a self-inflicted outage for harness's own model backend, so a blind deploy is a self-inflicted outage for
possibly zero behavioral change. possibly zero behavioral change.
@@ -46,9 +48,8 @@ possibly zero behavioral change.
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>` - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'` tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
- confirm the launcher maps each tunable to its flag - confirm the launcher maps each tunable to its flag
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 → (`grep scripts/launch.sh`).
--enable-prefix-caching`.
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is **Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
live — **do NOT restart**; reconcile the marker (below). Only if the live live — **do NOT restart**; reconcile the marker (below). Only if the live
@@ -46,7 +46,7 @@ graph LR
./scripts/deploy.sh dankserver ./scripts/deploy.sh dankserver
./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target ./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target
./scripts/deploy.sh dankclaw --service searxng ./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-medium ./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run ./scripts/deploy.sh dankclaw --dry-run
``` ```
@@ -62,13 +62,13 @@ service on the other host.
|---|---|---| |---|---|---|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. | | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. | | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. | | `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw iGPU inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-dragoon]]. |
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. | | `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw 3090 inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-zealot]]. |
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. | | `frank-archon` | systemd + `systemctl` (root unit) | `deploy_llm_archon` plans from repo paths and gates on the port; it does NOT see host-side unit drift (see [[frank-archon]] → Deploy / verify). |
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. | | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
The shared chat template maps to the inference service on the selected target. The shared chat template maps to the inference service on the selected target.
The `frank-npu` endpoint is retired (2026-09-11) and has no deploy handler; The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
an unrelated large-model deployment never touches it. an unrelated large-model deployment never touches it.
## Caddy safety ## Caddy safety
@@ -34,8 +34,9 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
`/metrics` that is disabled (Gitea 404). `/metrics` that is disabled (Gitea 404).
- A scrape cannot express "is this HTTP service answering?" — the right probe - A scrape cannot express "is this HTTP service answering?" — the right probe
is the **blackbox exporter** in probe mode (which also gives guarded vhosts is the **blackbox exporter** in probe mode
a real uptime signal, since Prometheus cannot solve an Anubis challenge). (`/probe?target=…&module=http_2xx`), which also gives guarded vhosts
a real uptime signal, since Prometheus cannot solve an Anubis challenge.
- **Fix in the direction of honesty, never of silence:** unreachable target → - **Fix in the direction of honesty, never of silence:** unreachable target →
mark the homepage card `wip: true` with the reason in `description` (the mark the homepage card `wip: true` with the reason in `description` (the
checker skips `wip`; `wip` is the existing convention for "real service, checker skips `wip`; `wip` is the existing convention for "real service,
@@ -43,8 +44,8 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
database port on the LAN to make a badge green, and never delete a scrape database port on the LAN to make a badge green, and never delete a scrape
job — an absent scrape and a healthy one are indistinguishable. job — an absent scrape and a healthy one are indistinguishable.
- Alert rules must not depend on always-red signals — scope them to services - Alert rules must not depend on always-red signals — scope them to services
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for that can actually be healthy; the `frank-*` scope `LlmEndpointDown` carries
exactly that reason). is recorded on [[Prometheus]].
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from - A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
`connection refused` to `unsupported Content-Type "text/html"` means the `connection refused` to `unsupported Content-Type "text/html"` means the
service came up and only the scrape is wrong. Read the error, not just service came up and only the scrape is wrong. Read the error, not just
@@ -13,11 +13,10 @@ timestamp: 2026-08-26
# Dankclaw Stack # Dankclaw Stack
Dankclaw (192.168.0.143) runs the AI, search, development, and observability Dankclaw (192.168.0.143) runs the AI, search, development, and observability
stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX stack. Its LLM inference endpoints are bare-metal host services, not part of
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on the container stack; each has its own page — shape, payload, eras and wire
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`, contract live there, nowhere else. See [[frank-dragoon]] and [[frank-zealot]]
FLM v1.0.1) serves sight+sound models. The completion endpoint (both on this host) and [[frank-archon]] (Dankserver, 192.168.0.235).
`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235).
```mermaid ```mermaid
graph LR graph LR
@@ -28,9 +27,8 @@ graph LR
C --> F[ComfyUI] C --> F[ComfyUI]
C --> D[Grafana] C --> D[Grafana]
S --> V[Gluetun VPN] S --> V[Gluetun VPN]
O --> L[llama.cpp dankserver :7770] O --> L[LLM endpoints]
O --> H[Hermes] O --> H[Hermes]
O --> E[frank-dragoon :7770]
``` ```
## Services ## Services
@@ -48,18 +46,17 @@ graph LR
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] | | Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — | | Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] | | Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
| frank-dragoon (host service) | — | `7770` | [[deployment]] | | frank-dragoon (host service) | — | `7770` | [[frank-dragoon]] |
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] | | frank-zealot (host service) | — | `7771` | [[frank-zealot]] |
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] | | frank-archon (host service, Dankserver) | — | `7770` | [[frank-archon]] |
The LLM rows are bare-metal systemd services, not containers. This The LLM rows are bare-metal systemd services, not containers, and this repo
repo owns their scripts, units, and knowledge base (see owns their scripts, units and knowledge base (see
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03): [LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on
`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF each port is era state and lives on the endpoint page, not here. `frank-npu`
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build; (the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon`
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU; on the Dankserver port) are both retired — see [[frank-npu]] and
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving [[frank-medium-radiance-mxfp4]].
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver.
## Boot Persistence ## Boot Persistence
+4 -4
View File
@@ -29,7 +29,7 @@ render needs the app installed, not this endpoint.
| | | | | |
| --- | --- | | --- | --- |
| Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` | | Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
| Port | **7772**, published (`PublishPort=7772:7772`); the family is 7770 dragoon, 7771 zealot, 7772 tts | | Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] |
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only | | Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only |
| Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` | | Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) | | Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
@@ -142,9 +142,9 @@ types: [[Prometheus]] records that the existing HTML probes (`comfyui`, the two
`services` targets) sit **down** with `services` targets) sit **down** with
`received unsupported Content-Type … and no fallback_scrape_protocol specified`. `received unsupported Content-Type … and no fallback_scrape_protocol specified`.
A `/health` probe job would therefore be structurally never-green, and because A `/health` probe job would therefore be structurally never-green, and because
`LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, a permanently red `LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
`frank-tts` target would page on Signal every few minutes for an endpoint that lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
is fine — the frank-archon lesson pointed the other way. The engine cannot be Signal every few minutes for an endpoint that is fine. The engine cannot be
scraped at all: it publishes nothing. Status lives in `systemctl --user status`, scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
`podman healthcheck run`, the two unit journals, and the homepage card, which `podman healthcheck run`, the two unit journals, and the homepage card, which
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
+11 -10
View File
@@ -78,9 +78,9 @@ is not durable. The tracked, sanitized configuration baseline is
`/data/config.yaml` is not readable without `sudo`): `/data/config.yaml` is not readable without `sudo`):
`dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes `dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
`frank-dragoon` — verified from the host with `curl `frank-dragoon` — verified from the host with `curl
http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is `127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
a rootless pasta publish and is IPv4-only); and is IPv4-only);
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of `dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
2026-10-18, that endpoint serves only `frank-archon`** (`GET 2026-10-18, that endpoint serves only `frank-archon`** (`GET
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
@@ -88,7 +88,8 @@ is not durable. The tracked, sanitized configuration baseline is
2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so 2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
if the live `config.yaml` still names `frank-medium` the provider is dead if the live `config.yaml` still names `frank-medium` the provider is dead
weight and requests will 404 on a nonexistent model. Diff the live config weight and requests will 404 on a nonexistent model. Diff the live config
before trusting either name. before trusting either name. Which payload sits behind each port, and the
served id it answers to, is on [[frank-dragoon]] and [[frank-archon]].
- The live `config.yaml` contains secrets and user state and is not committed; - The live `config.yaml` contains secrets and user state and is not committed;
the tracked example is the sanitized backup of its non-secret settings. the tracked example is the sanitized backup of its non-secret settings.
@@ -227,9 +228,10 @@ resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
What actually breaks, and the fix: What actually breaks, and the fix:
- **`response_format` (json_object / json_schema) is a hard 400 on every - **`response_format` (json_object / json_schema) is a hard 400 on the local
local backend here** — neither FreeToken nor halogen implements constrained backends this task was written against** — what each endpoint accepts on the
decoding. Config cannot override it: `call_llm` merges the task's wire is on [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Config
cannot override it: `call_llm` merges the task's
`extra_body` first and then `update()`s the caller's, so the hardcoded one `extra_body` first and then `update()`s the caller's, so the hardcoded one
wins. Only temperature and max_tokens have reactive retry paths. wins. Only temperature and max_tokens have reactive retry paths.
- **Fix: move that one task to the Responses transport**, which never - **Fix: move that one task to the Responses transport**, which never
@@ -246,9 +248,8 @@ What actually breaks, and the fix:
effort: none # mandatory, see below effort: none # mandatory, see below
``` ```
Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw Before copying this: which id is served on which port is endpoint state — see
`:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider [[frank-zealot]] and [[frank-archon]]. The provider/model
name is recorded against — answers only `frank-archon`. The provider/model
pairing above is the shape that was landed in 2026; re-derive it from the live pairing above is the shape that was landed in 2026; re-derive it from the live
config before reusing it. config before reusing it.
@@ -61,7 +61,7 @@ podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/model
**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap. **The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout. **Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). The decision rests on each endpoint's documented request surface ([[frank-zealot]], [[frank-dragoon]]): zealot's is a known-good target for that call shape, dragoon's structured-output behaviour is **unverified**, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
## Speakers and voices: 1–4, and the IDs are free text ## Speakers and voices: 1–4, and the IDs are free text
+22 -26
View File
@@ -39,30 +39,26 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in `frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
`shared/homepage/services.json`)*. `shared/homepage/services.json`)*.
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host | | Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
|---|---|---|---|---|---| |---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw | | 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver | | 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 2×262,144 exact f16 KV per `scripts/frank-zealot.env`) | Dankclaw | | 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
The URLs in the first column are the DB's recorded values and predate the 2026-08-22 The URLs in the first column are the DB's recorded values and predate the 2026-08-22
bridge move; on this network they are wrong — see the reachability gotcha below, which bridge move; on this network they are wrong — see the reachability gotcha below, which
is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771` is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
closed the same day (production unit stopped 03:38 per the repo Benchmarking that day's payload swap (see [[frank-zealot]] for what was deployed and when).
Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
PQ2_0 with the MTP head, deployed that afternoon at 3 × 262,144 q8_0 KV and
resliced the same evening to **2 × 262,144 over exact f16 KV** — was **deployed
as production**, so the
picker row below was re-keyed to it (see the next paragraph).
The `model` table carries exactly two decoration rows, `frank-zealot` and The `model` table carries exactly two decoration rows, `frank-zealot` and
`frank-dragoon`. Both the context length and the description live **inside `frank-dragoon`. Both the context length and the description live **inside
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
`{"description": …}`; there is no `context_length` or `description` column, so `{"description": …}`; there is no `context_length` or `description` column, so
`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18: `sqlite3` one-liners against those names fail. `params.context.length` is a copy
`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon` of the endpoint's per-slot window, so the authoritative number lives on the
→ 200,704 (its `params` had kept the pre-lowering 262,144 window from endpoint page ([[frank-zealot]], [[frank-dragoon]]) and the row must follow it —
2026-10-17 while its own description already said 200,704). These rows are the 2026-10-18 re-key existed because `frank-dragoon`'s row still carried the
window its payload had before that day's swap. These rows are
live DB state, so a payload swap must re-key the database too, not only the live DB state, so a payload swap must re-key the database too, not only the
repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
@@ -106,10 +102,10 @@ Two gotchas make the wiring non-obvious:
The Ollama connection is disabled (`ollama.enable=false`). The Ollama connection is disabled (`ollama.enable=false`).
> All three endpoints are bare-metal systemd services managed by this > All three endpoints are bare-metal systemd services managed by this
> repository. Model swaps, context sizing, and spec tuning happen there. See > repository; their shape, swaps and tuning live on
> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only > [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Disabling a
> hides it from this picker — the endpoint itself keeps serving every other > connection here only hides it from this picker — the endpoint itself keeps
> client (the harness, Hermes, API consumers). > serving every other client (the harness, Hermes, API consumers).
## Container introspection (DB and backend python) ## Container introspection (DB and backend python)
@@ -145,9 +141,9 @@ The Ollama connection is disabled (`ollama.enable=false`).
models. Response key is `items`, not `data`. models. Response key is `items`, not `data`.
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply 2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
with exactly one word: pong") → each returns content `pong`. Reasoning with exactly one word: pong") → each returns content `pong`. Reasoning
models need a generous budget: frank-zealot (Qwen3.5 template) returns endpoints need a generous budget: thinking consumes `max_tokens`, so a short
empty content + `finish_reason: "length"` at `max_tokens 200` (the one returns empty content + `finish_reason: "length"` (see
thinking block eats it) and answers `pong` at 1500. [[frank-zealot]]); `frank-zealot` answers `pong` at 1500.
3. A 500 "Server Connection Error" → `podman logs open-webui` for 3. A 500 "Server Connection Error" → `podman logs open-webui` for
`ClientConnectorError`: a network-reachability problem (host LAN IP vs `ClientConnectorError`: a network-reachability problem (host LAN IP vs
`host.containers.internal`), not a model problem. `host.containers.internal`), not a model problem.
@@ -340,8 +336,8 @@ neither.
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].) - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
- The default admin account is created on first login; secure it immediately. - The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready. - If the backend is loading, OpenWebUI shows connection errors until the endpoint is ready; each endpoint's boot budget is on its own page ([[frank-dragoon]], [[frank-zealot]], [[frank-archon]]).
- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`. - Repeated from *The LLM backends*: the on-host endpoints are reached via the **host LAN IP**, never `host.containers.internal`.
## History ## History
@@ -390,4 +386,4 @@ neither.
- [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)* - [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)*
- [[Hermes]] — the agent OpenWebUI can talk to - [[Hermes]] — the agent OpenWebUI can talk to
- [[SearXNG]] — the search engine on this host - [[SearXNG]] — the search engine on this host
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints - [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]] — the endpoints behind the picker
@@ -36,14 +36,14 @@ it is not invisible.
## Why 64 GiB ## Why 64 GiB
Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1` **103.5 GiB of the 121.2 GiB** host, leaving roughly **7.1 GiB** for the then-28
working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose co-tenant containers whose working set was ~4.3 GiB. How that total is split
working set was ~4.3 GiB. That is a decision-time reading, not today's: on between weights, KV pool and working memory is the engine's own ledger — it lives
on [[frank-dragoon]]. That is a decision-time reading, not today's: on
2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB** 2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the (`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
`frank-dragoon-halogen` container accounts for **70.33 GiB** `frank-dragoon-halogen` container accounts for **70.33 GiB**
(`podman stats --no-stream`, same day) on the current flash-server payload — (`podman stats --no-stream`, same day).
a different model and KV shape than the one measured in September.
The engine itself warned at the time that when the headroom runs out, The engine itself warned at the time that when the headroom runs out,
frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
no disk activity and no output". Swap is the valve that turns an OOM kill into slow no disk activity and no output". Swap is the valve that turns an OOM kill into slow
@@ -59,7 +59,7 @@ like this, **never `swapoff` before the replacement is online.** The old 8 GiB
held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was
genuinely free — the pinned model weights are counted as reclaimable page genuinely free — the pinned model weights are counted as reclaimable page
cache, so the headline number is a lie by roughly the size of the model cache, so the headline number is a lie by roughly the size of the model
(see [[halogen-flash-server]]). Swapping off into that apparent space, with (see [[frank-dragoon]]). Swapping off into that apparent space, with
nothing else to receive the pages, is how you invite the OOM killer. nothing else to receive the pages, is how you invite the OOM killer.
The ordering that avoids it — create and enable the new swap **first**, so the The ordering that avoids it — create and enable the new swap **first**, so the
@@ -113,10 +113,10 @@ frank-dragoon `health=200` throughout.
## Caveats ## Caveats
- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are - **Swap cannot help the pinned weights.** frank-dragoon's weights are pinned
held deliberately; paging them is a stall, not a relief. More swap does not deliberately; paging them is a stall, not a relief. More swap does not make a
make the doubled KV pool affordable — see the reserve/pool ceiling in larger KV pool affordable — see the reserve/pool ceiling in
[[halogen-flash-server]]. [[frank-dragoon]].
- **`vm.swappiness=60` is untested against this workload.** Under real - **`vm.swappiness=60` is untested against this workload.** Under real
pressure the kernel will happily page anonymous memory, and frank-dragoon's KV pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
pool and working set *are* anonymous. Swapping the KV pool would be far worse pool and working set *are* anonymous. Swapping the KV pool would be far worse
@@ -151,6 +151,6 @@ The old swapfile has therefore been purged; nothing on the host still holds it.
## See Also ## See Also
- [[halogen-flash-server]] — the memory model that motivated this - [[frank-dragoon]] — the memory model that motivated this
- [[Dankclaw System Specs]] — the Strix Halo box - [[Dankclaw System Specs]] — the Strix Halo box
- [[Dankclaw Improvements]] — other host-level changes - [[Dankclaw Improvements]] — other host-level changes
@@ -233,12 +233,12 @@ loud, not a bug. Work through it in order:
they have to be copied there once before a build. they have to be copied there once before a build.
- `VOICE_PACK_TOOL` points at the trial's build tree - `VOICE_PACK_TOOL` points at the trial's build tree
(`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out (`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
at exactly `5916bf7c`), and the two GGUFs at at exactly `5916bf7c`), and the two GGUFs live at
`/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's `/home/dank/models/Qwen3-TTS-GGUF/`, the engine slice's fetch target — that
download script puts them — that directory does not exist yet, so until it path and its fetch state belong to [[Frank TTS Engine]].
does the models have to come from the trial's `models/` directory. - Whether the façade (7772) and the engine (8880) are up and serving is deploy
- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs state on [[Frank TTS]] / [[Frank TTS Engine]] — read it there, not from this
is possible on that host while serving them is not. census.
## Known gaps, on purpose ## Known gaps, on purpose
+1 -1
View File
@@ -30,7 +30,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover | | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration | | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` | | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` | | GPU | none by decision — CPU decode only; the two R9700s stay dedicated to [[frank-archon]] |
| Deploy | `./scripts/deploy.sh dankserver --service plex` | | Deploy | `./scripts/deploy.sh dankserver --service plex` |
| Docs | https://support.plex.tv | | Docs | https://support.plex.tv |
@@ -159,10 +159,9 @@ Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
entirely — admin tooling; Seerr is presented as the front door for entirely — admin tooling; Seerr is presented as the front door for
requests. Each fleet model gained a Hugging Face source link: card field requests. Each fleet model gained a Hugging Face source link: card field
`model.hf` rendered as the cyan `model on huggingface ↗` line under the `model.hf` rendered as the cyan `model on huggingface ↗` line under the
role, and the docs `Model:` bullets link the checkpoint names (zealot → role, and the docs `Model:` bullets link the checkpoint names — which
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon → checkpoint each endpoint runs is payload state on
Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon → [[frank-dragoon]], [[frank-archon]] and [[frank-zealot]].
Qwen/Qwen3.8-Flash-Next-FP8).
**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth **Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
wave):** the amber CRT palette is retired. The page is now deep space wave):** the amber CRT palette is retired. The page is now deep space
@@ -235,11 +234,10 @@ house-DNS advice; the bullet now just says the names resolve on the
home network. home network.
**Archon card runs the fleet's own model (2026-10-08):** the **Archon card runs the fleet's own model (2026-10-08):** the
`frank-archon` homepage card and docs Model bullet now say the payload `frank-archon` homepage card and docs Model bullet swapped the FP8-era
is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link base-model link for our own HF publication, and the capacity line for the
our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`) endpoint's own concurrency figure; the payload, its publication and the slot
in place of the FP8-era base-model link; capacity line says three count all live on [[frank-archon]].
submissions (`--max-num-seqs 3`).
## Design decision — 2026-08-22 ## Design decision — 2026-08-22
@@ -130,13 +130,14 @@ the blackbox exporter. See [[Prometheus]] → Targets.
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18, **A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric — `up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
the oldest sample, 2026-10-11, is already 0 — while the service itself answered the oldest sample, 2026-10-11, is already 0 — while the service itself answered
`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented** `/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
on `/metrics` unless the server is started with `--metrics`, so the scrape was is started with the flag, so the scrape was refused at the protocol layer (the
refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing: flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
stayed firing:
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02 -u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target metrics opt-in landed that day; the target
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18). reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
How many payload generations it spanned is not readable from any host How many payload generations it spanned is not readable from any host
artifact. *(unverified 2026-10-18)* artifact. *(unverified 2026-10-18)*
@@ -151,7 +152,8 @@ notification log — `dankclaw/observability/quadlet/alertmanager.container:39-4
`/opt/alertmanager/data`.) `/opt/alertmanager/data`.)
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
metrics, ComfyUI), and halogen serves `/metrics` natively, which is why metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
([[frank-dragoon]]), which is why
`frank-dragoon` has always been green. The active-alert count is a `frank-dragoon` has always been green. The active-alert count is a
moment-in-time value and should never be documented as a constant: moment-in-time value and should never be documented as a constant:
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing `/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
@@ -167,8 +169,8 @@ arm (`systemctl is-active frank-zealot` → `failed` after
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past `stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
`TimeoutStopSec`, the known shutdown shape for these engines). So `TimeoutStopSec`, the known shutdown shape for these engines). So
`up{job="frank-zealot"} == 1` that day measured the trial, not the production `up{job="frank-zealot"} == 1` that day measured the trial, not the production
shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408 shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract. and it is the repo contract.
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot` The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
reinstalls and starts it, restoring boot persistence and self-healing. Never reinstalls and starts it, restoring boot persistence and self-healing. Never
quote a candidate arm's flags or VRAM as fleet state — the process and its shape quote a candidate arm's flags or VRAM as fleet state — the process and its shape
@@ -25,7 +25,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) | | Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) | | Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) | | Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) | | Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
## Targets ## Targets
@@ -37,10 +37,10 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) | | `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works | | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` | | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works | | `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) | | `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] |
The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18). The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16): **Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
@@ -53,15 +53,14 @@ The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`. Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service. **`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z. Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
A scrape cannot express "is this HTTP service answering?" — that is what a The remedy for a probe-shaped question is the **blackbox exporter**; the
probe is for, and the remedy is the **blackbox exporter** in probe mode classification rule and the honesty law for it live in [[Stale Signal Audit]].
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real Until the exporter is deployed, these three stay red for a structural reason:
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it deleting them would
is deployed, these three stay red for a structural reason: deleting them would
hide a real service, and fixing them by publishing databases or enabling metrics hide a real service, and fixing them by publishing databases or enabling metrics
would trade a wrong signal for a real exposure. Targets are `host:port`, never would trade a wrong signal for a real exposure. Targets are `host:port`, never
URLs. URLs.
@@ -94,8 +94,8 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals `pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
seconds later** under `Restart=on-failure`. The failure looks like it failed, seconds later** under `Restart=on-failure`. The failure looks like it failed,
then looks fine, and the missing gate stays invisible because the retry masks then looks fine, and the missing gate stays invisible because the retry masks
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless it. First seen on a `frank-dragoon` unit cutover; which supervision shape that
Quadlet. era was is in [[frank-dragoon]]'s era table.
Gate procedure: stop the old unit AND the new one (a redeploy's running Gate procedure: stop the old unit AND the new one (a redeploy's running
replacement is itself the port-holder — without that stop the gate deadlocks replacement is itself the port-holder — without that stop the gate deadlocks
@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
## Endpoint down, connection refused — GPU holder reclaim ## Endpoint down, connection refused — GPU holder reclaim
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
service was not crashed. Applies to `frank-medium` (dankserver, root unit), service was not crashed. Applies to the root-unit fleet endpoints:
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless `frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
Quadlet). unit identity per endpoint is on its own page ([[frank-archon]],
[[frank-dragoon]], [[frank-zealot]]).
**1. Classify the stop before touching anything** **1. Classify the stop before touching anything**
```bash ```bash
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \ ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts" systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
``` ```
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
prod unit → get the holder, and compare its bytes against the service's prod unit → get the holder, and compare its bytes against the service's
requirement before concluding a start would even fit (frank-medium 35:30 requirement before concluding a start would even fit. The per-boot requirement
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
restart on allocation, not config). [[frank-zealot]]; the retired `frank-medium` split is measured in
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
allocation, not config.
**3. Identify the holder and prove it is abandoned** **3. Identify the holder and prove it is abandoned**
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
freshly restarted endpoint under live agent-queue backlog may return empty freshly restarted endpoint under live agent-queue backlog may return empty
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
timeout. First request after a halogen (`frank-dragoon`) restart can go silent timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`: [[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
read logs **without** sudo (per-user storage). If the restore never happened, log lives (and whether sudo is needed to read it) is on its page. If the
restore never happened,
the sudo journal names who and when: `journalctl -b | grep sudo`. the sudo journal names who and when: `journalctl -b | grep sudo`.
## Scratch vLLM boot on a fleet GPU host — stop-first law ## Scratch vLLM boot on a fleet GPU host — stop-first law
@@ -110,13 +114,14 @@ failure modes if skipped:
1. **Expert-offload planner dies on host RAM.** 1. **Expert-offload planner dies on host RAM.**
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at `vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
plan time. A production rank worker holds tens of GiB of unevictable plan time. A production rank worker holds tens of GiB of unevictable
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
boot sees ~86 GiB instead of ~239 and dies at plan time: RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
boot sees far less than the bare host and dies at plan time:
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`. `ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
Looks like a planner bug; is pure RAM contention. Signature: container Looks like a planner bug; is pure RAM contention. Signature: container
exits within ~60 s, after `Resolved architecture: …` but before worker exits within ~60 s, after `Resolved architecture: …` but before worker
init; `free -g` showing `available ≈ 86` while prod runs is the smoking init; `free -g` showing an `available` far below the endpoint page's
gun. loaded-host figure while prod runs is the smoking gun.
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless 2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine (the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
only once the prod unit is down. only once the prod unit is down.
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
working tree files would be overwritten by merge" — exactly the file you working tree files would be overwritten by merge" — exactly the file you
shipped); `git checkout --` silently eats uncommitted work without step 4; a shipped); `git checkout --` silently eats uncommitted work without step 4; a
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`, runtime may **mount files straight out of the checkout** (`scripts/**` — the
serve scripts and the chat template mount from there, see [[frank-archon]] — and
`shared/grafana/provisioning/**`) so a hand copy takes effect at next `shared/grafana/provisioning/**`) so a hand copy takes effect at next
container restart with no deploy; root-owned files from a container write container restart with no deploy; root-owned files from a container write
need `chown` back before git operations; and **a restart is not a need `chown` back before git operations; and **a restart is not a
@@ -48,8 +48,9 @@ table. Three buckets:
re-download. re-download.
- **retired** — maps to a retired runtime. Safe candidates. - **retired** — maps to a retired runtime. Safe candidates.
Trap: a directory can hold **both** a dead target and a live drafter Trap: a directory can hold **both** a dead target and a live asset — an era's
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the drafter can sit beside the retired target it came with (per-endpoint era tables:
[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the
single file, never the directory, and say so in the manifest. single file, never the directory, and say so in the manifest.
## 3. The live-reference check — before moving anything ## 3. The live-reference check — before moving anything
@@ -149,13 +150,13 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
## Refresh per engine — know the recovery path BEFORE deleting ## Refresh per engine — know the recovery path BEFORE deleting
- **frank-dragoon (halogen)**: the engine fetches its own payload via - **frank-dragoon**: self-fetching — clear the directory but
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but leave it **existing and empty**, then start the unit; the download knobs and
leave it **existing and empty**, then start the unit. **Bind-mount trap:** the payload's byte count (which is what the re-download costs) are on
[[frank-dragoon]]. **Bind-mount trap:**
podman refuses a mount whose source is missing podman refuses a mount whose source is missing
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and (`Error: statfs /home/dank/models/halogen: no such file or directory`) and
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir.
at 130–240 MB/s ≈ 10–20 min, then load.
- **llama.cpp services**: no self-fetch; the serve script refuses to start - **llama.cpp services**: no self-fetch; the serve script refuses to start
without weights. Pull yourself and verify: without weights. Pull yourself and verify: