diff --git a/serverAppConfigs-docs/Agent/OMP Harness.md b/serverAppConfigs-docs/Agent/OMP Harness.md index e2f96f6..48eafda 100644 --- a/serverAppConfigs-docs/Agent/OMP Harness.md +++ b/serverAppConfigs-docs/Agent/OMP Harness.md @@ -14,16 +14,16 @@ registry; single source of truth — there is no copy on the servers) and this file is repo-managed; edits are harness-side only. **Model ids are the StarCraft unit names (official 2026-10-08, on the wire -since 2026-10-09):** `frank-archon` (dankserver :7770, radiance), -`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw -:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`, +since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and +`frank-zealot` (Dankclaw) — engine, port and card behind each id live on +[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`, `frank-zealot`. The SERVICES keep the old names (units, scripts, env files, runbook pages); pages written in the old era name the service, the harness names the model. The wire now matches: each endpoint's `/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI picker/grants, the homepage docs and the raw API all use one id per model -(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]] -note on radiance echoing the requested name). +(see [[OpenWebUI]] for the grant re-key and the [[frank-archon]] +note on the endpoint echoing the requested name). Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and 192.168.0.143:7771 each answer with the unit name (`frank-archon`, @@ -50,10 +50,11 @@ Where the contract lives: `~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js` The arithmetic (the coupled contract every endpoint's maxTokens comment -encodes). Recomputed from the live files 2026-10-18 — contextWindows -200,000 (archon) / 155,648 (dragoon) / 200,704 (zealot) at -`models.yml:48,131,206`; maxTokens 16,384 / 45,056 / 16,384 at -`models.yml:56,143,212`: +encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the +endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what +follows is a **worked example** recomputed from the live files on 2026-10-18 — +a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and +maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot): 1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live 95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The @@ -141,28 +142,25 @@ Use **wire names**, not OMP camelCase. Two traps: 2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy also sets it — `Object.assign` is shallow and replaces the whole object, silently dropping the `reasoning_effort` kwarg the compat policy set. - Top-level `preserve_thinking` is honoured by halogen. Live state 2026-10-18: - frank-archon's radiance entry carries `chat_template_kwargs: - {enable_thinking, preserve_thinking}` in `extraBody` (`models.yml:89-91`) - and nothing else sets that object at that endpoint, so the shallow-merge - hazard is latent rather than firing; the FreeToken-era "verify-and-prune at - the next swap" note is retired with that swap (2026-10-05, - `scripts/serve-frank-archon.sh:9-10`). + Top-level `preserve_thinking` is honoured by frank-dragoon + ([[frank-dragoon]]). Live state 2026-10-18: frank-archon's `extraBody` + carries `chat_template_kwargs: {enable_thinking, preserve_thinking}` + (`models.yml:89-91`) and nothing else sets that object at that endpoint, so + the shallow-merge hazard is latent rather than firing. -**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier flash-server +**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier frank-dragoon builds carried **no server-side sampler defaults**, so an absent `temperature` decoded greedy while the endpoint looked configured. That era is -over: `HALOGEN_TEMPERATURE/TOP_P/TOP_K` now set the card's thinking recipe -(1.0 / 0.95 / 20) as engine defaults (`scripts/frank-dragoon.env:98-103`, -exported at `scripts/serve-frank-dragoon.sh:186-188`), so an absent temperature -is no longer greedy. frank-dragoon's `extraBody` still carries the same trio — -consistent, not load-bearing. Treat any older "do not drop it or you get -greedy" wording as history. +over — the endpoint now ships its own defaults ([[frank-dragoon]]), so an +absent temperature is no longer greedy. frank-dragoon's `extraBody` still +carries the same trio — consistent, not load-bearing. Treat any older "do not +drop it or you get greedy" wording as history. -**Reasoning-effort folding.** Qwen3.8's template accepts only -`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything -else ("Unexpected reasoning effort high"). Every endpoint's -`reasoningEffortMap` folds all harness levels onto that set; a role whose +**Reasoning-effort folding.** The set `reasoning_effort` accepts is a property +of the endpoint, not of the harness: [[frank-dragoon]] and [[frank-zealot]] +raise on anything outside `xhigh|medium|low`, while [[frank-archon]] does not +gate the set. Every endpoint's `reasoningEffortMap` folds all harness levels +onto whichever set that endpoint ships; a role whose level is unmapped fails on its FIRST request, and the failure names Jinja, not the model. diff --git a/serverAppConfigs-docs/Architecture/Deploy State Triage.md b/serverAppConfigs-docs/Architecture/Deploy State Triage.md index 56402ad..4bdc511 100644 --- a/serverAppConfigs-docs/Architecture/Deploy State Triage.md +++ b/serverAppConfigs-docs/Architecture/Deploy State Triage.md @@ -22,8 +22,10 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT "behavior pending". **Why it matters:** a plain `deploy.sh` run re-selects and restarts a service -whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of -a 100+ GiB payload — and frank-archon (dankserver:7770) is also the +whose marker lags. On the big-payload endpoints that is a full model reload +measured in tens of minutes — each endpoint's boot budget is on its own page +([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon +(dankserver:7770) is also the harness's own model backend, so a blind deploy is a self-inflicted outage for possibly zero behavioral change. @@ -46,9 +48,8 @@ possibly zero behavioral change. - `podman inspect --format '{{.Args}}' | tr ' ' '\n' | grep -i ` - engine env: `podman exec sh -c 'for f in /proc/[0-9]*/environ; do tr "\0" "\n" <$f 2>/dev/null; done | grep -i '` - - confirm the launcher maps each tunable to its flag - (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 → - --enable-prefix-caching`. + - confirm the launcher maps each tunable to its flag + (`grep scripts/launch.sh`). **Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is live — **do NOT restart**; reconcile the marker (below). Only if the live diff --git a/serverAppConfigs-docs/Architecture/Deployment Flow.md b/serverAppConfigs-docs/Architecture/Deployment Flow.md index 38e7370..2b56363 100644 --- a/serverAppConfigs-docs/Architecture/Deployment Flow.md +++ b/serverAppConfigs-docs/Architecture/Deployment Flow.md @@ -46,7 +46,7 @@ graph LR ./scripts/deploy.sh dankserver ./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target ./scripts/deploy.sh dankclaw --service searxng -./scripts/deploy.sh dankserver --service frank-medium +./scripts/deploy.sh dankserver --service frank-archon ./scripts/deploy.sh dankclaw --dry-run ``` @@ -62,13 +62,13 @@ service on the other host. |---|---|---| | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. | | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. | -| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. | -| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. | -| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. | +| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw iGPU inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-dragoon]]. | +| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw 3090 inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-zealot]]. | +| `frank-archon` | systemd + `systemctl` (root unit) | `deploy_llm_archon` plans from repo paths and gates on the port; it does NOT see host-side unit drift (see [[frank-archon]] → Deploy / verify). | | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver//quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/` to `/home/dank/Containers/` (mapped-uid roots move without chown). Steady state is restart-only. | The shared chat template maps to the inference service on the selected target. -The `frank-npu` endpoint is retired (2026-09-11) and has no deploy handler; +The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler; an unrelated large-model deployment never touches it. ## Caddy safety diff --git a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md index 7347336..37c16cb 100644 --- a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md +++ b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md @@ -34,8 +34,9 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a `/metrics` that is disabled (Gitea 404). - A scrape cannot express "is this HTTP service answering?" — the right probe - is the **blackbox exporter** in probe mode (which also gives guarded vhosts - a real uptime signal, since Prometheus cannot solve an Anubis challenge). + is the **blackbox exporter** in probe mode + (`/probe?target=…&module=http_2xx`), which also gives guarded vhosts + a real uptime signal, since Prometheus cannot solve an Anubis challenge. - **Fix in the direction of honesty, never of silence:** unreachable target → mark the homepage card `wip: true` with the reason in `description` (the checker skips `wip`; `wip` is the existing convention for "real service, @@ -43,8 +44,8 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a database port on the LAN to make a badge green, and never delete a scrape job — an absent scrape and a healthy one are indistinguishable. - Alert rules must not depend on always-red signals — scope them to services - that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for - exactly that reason). + that can actually be healthy; the `frank-*` scope `LlmEndpointDown` carries + is recorded on [[Prometheus]]. - A target's failure STRING can prove a fix: a `/metrics` scrape flipping from `connection refused` to `unsupported Content-Type "text/html"` means the service came up and only the scrape is wrong. Read the error, not just diff --git a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md index b9bd8f4..9123e12 100644 --- a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md +++ b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md @@ -13,11 +13,10 @@ timestamp: 2026-08-26 # Dankclaw Stack Dankclaw (192.168.0.143) runs the AI, search, development, and observability -stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX -llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on -the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`, -FLM v1.0.1) serves sight+sound models. The completion endpoint -`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235). +stack. Its LLM inference endpoints are bare-metal host services, not part of +the container stack; each has its own page — shape, payload, eras and wire +contract live there, nowhere else. See [[frank-dragoon]] and [[frank-zealot]] +(both on this host) and [[frank-archon]] (Dankserver, 192.168.0.235). ```mermaid graph LR @@ -28,9 +27,8 @@ graph LR C --> F[ComfyUI] C --> D[Grafana] S --> V[Gluetun VPN] - O --> L[llama.cpp dankserver :7770] + O --> L[LLM endpoints] O --> H[Hermes] - O --> E[frank-dragoon :7770] ``` ## Services @@ -48,18 +46,17 @@ graph LR | Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] | | Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — | | Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] | -| frank-dragoon (host service) | — | `7770` | [[deployment]] | -| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] | -| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] | +| frank-dragoon (host service) | — | `7770` | [[frank-dragoon]] | +| frank-zealot (host service) | — | `7771` | [[frank-zealot]] | +| frank-archon (host service, Dankserver) | — | `7770` | [[frank-archon]] | -The LLM rows are bare-metal systemd services, not containers. This -repo owns their scripts, units, and knowledge base (see -[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03): -`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF -on the card's pinned `kingjones30/ROCmFPX` llama.cpp build; -`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU; -`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving -kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver. +The LLM rows are bare-metal systemd services, not containers, and this repo +owns their scripts, units and knowledge base (see +[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on +each port is era state and lives on the endpoint page, not here. `frank-npu` +(the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon` +on the Dankserver port) are both retired — see [[frank-npu]] and +[[frank-medium-radiance-mxfp4]]. ## Boot Persistence diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS.md b/serverAppConfigs-docs/Dankclaw/Frank TTS.md index aa65bb6..a3c3df0 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS.md @@ -29,7 +29,7 @@ render needs the app installed, not this endpoint. | | | | --- | --- | | Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` | -| Port | **7772**, published (`PublishPort=7772:7772`); the family is 7770 dragoon, 7771 zealot, 7772 tts | +| Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] | | Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only | | Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` | | Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) | @@ -142,9 +142,9 @@ types: [[Prometheus]] records that the existing HTML probes (`comfyui`, the two `services` targets) sit **down** with `received unsupported Content-Type … and no fallback_scrape_protocol specified`. A `/health` probe job would therefore be structurally never-green, and because -`LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, a permanently red -`frank-tts` target would page on Signal every few minutes for an endpoint that -is fine — the frank-archon lesson pointed the other way. The engine cannot be +`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set +lives on [[Prometheus]]), a permanently red `frank-tts` target would page on +Signal every few minutes for an endpoint that is fine. The engine cannot be scraped at all: it publishes nothing. Status lives in `systemctl --user status`, `podman healthcheck run`, the two unit journals, and the homepage card, which probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the diff --git a/serverAppConfigs-docs/Dankclaw/Hermes.md b/serverAppConfigs-docs/Dankclaw/Hermes.md index a704dc4..461ab30 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes.md @@ -78,9 +78,9 @@ is not durable. The tracked, sanitized configuration baseline is `/data/config.yaml` is not readable without `sudo`): `dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes `frank-dragoon` — verified from the host with `curl - http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len - 200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is - a rootless pasta publish and is IPv4-only); + http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write + `127.0.0.1`, not `localhost`, because that port is a rootless pasta publish + and is IPv4-only); `dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of 2026-10-18, that endpoint serves only `frank-archon`** (`GET http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name @@ -88,7 +88,8 @@ is not durable. The tracked, sanitized configuration baseline is 2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so if the live `config.yaml` still names `frank-medium` the provider is dead weight and requests will 404 on a nonexistent model. Diff the live config - before trusting either name. + before trusting either name. Which payload sits behind each port, and the + served id it answers to, is on [[frank-dragoon]] and [[frank-archon]]. - The live `config.yaml` contains secrets and user state and is not committed; the tracked example is the sanitized backup of its non-secret settings. @@ -227,9 +228,10 @@ resp = call_llm(task="title_generation", messages=[...], max_tokens=64, What actually breaks, and the fix: -- **`response_format` (json_object / json_schema) is a hard 400 on every - local backend here** — neither FreeToken nor halogen implements constrained - decoding. Config cannot override it: `call_llm` merges the task's +- **`response_format` (json_object / json_schema) is a hard 400 on the local + backends this task was written against** — what each endpoint accepts on the + wire is on [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Config + cannot override it: `call_llm` merges the task's `extra_body` first and then `update()`s the caller's, so the hardcoded one wins. Only temperature and max_tokens have reactive retry paths. - **Fix: move that one task to the Responses transport**, which never @@ -246,9 +248,8 @@ What actually breaks, and the fix: effort: none # mandatory, see below ``` - Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw - `:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider - name is recorded against — answers only `frank-archon`. The provider/model + Before copying this: which id is served on which port is endpoint state — see + [[frank-zealot]] and [[frank-archon]]. The provider/model pairing above is the shape that was landed in 2026; re-derive it from the live config before reusing it. diff --git a/serverAppConfigs-docs/Dankclaw/Open Notebook.md b/serverAppConfigs-docs/Dankclaw/Open Notebook.md index 6607442..96cb5ca 100644 --- a/serverAppConfigs-docs/Dankclaw/Open Notebook.md +++ b/serverAppConfigs-docs/Dankclaw/Open Notebook.md @@ -61,7 +61,7 @@ podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/model **The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap. -**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout. +**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). The decision rests on each endpoint's documented request surface ([[frank-zealot]], [[frank-dragoon]]): zealot's is a known-good target for that call shape, dragoon's structured-output behaviour is **unverified**, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout. ## Speakers and voices: 1–4, and the IDs are free text diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 45db45d..42e277d 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -39,30 +39,26 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve `frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in `shared/homepage/services.json`)*. -| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host | -|---|---|---|---|---|---| -| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw | -| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver | -| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 2×262,144 exact f16 KV per `scripts/frank-zealot.env`) | Dankclaw | +| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page | +|---|---|---|---|---| +| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] | +| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] | +| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] | The URLs in the first column are the DB's recorded values and predate the 2026-08-22 bridge move; on this network they are wrong — see the reachability gotcha below, which -is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771` -closed the same day (production unit stopped 03:38 per the repo Benchmarking -Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B** -PQ2_0 with the MTP head, deployed that afternoon at 3 × 262,144 q8_0 KV and -resliced the same evening to **2 × 262,144 over exact f16 KV** — was **deployed -as production**, so the -picker row below was re-keyed to it (see the next paragraph). +is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after +that day's payload swap (see [[frank-zealot]] for what was deployed and when). The `model` table carries exactly two decoration rows, `frank-zealot` and `frank-dragoon`. Both the context length and the description live **inside JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds `{"description": …}`; there is no `context_length` or `description` column, so -`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18: -`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon` -→ 200,704 (its `params` had kept the pre-lowering 262,144 window from -2026-10-17 while its own description already said 200,704). These rows are +`sqlite3` one-liners against those names fail. `params.context.length` is a copy +of the endpoint's per-slot window, so the authoritative number lives on the +endpoint page ([[frank-zealot]], [[frank-dragoon]]) and the row must follow it — +the 2026-10-18 re-key existed because `frank-dragoon`'s row still carried the +window its payload had before that day's swap. These rows are live DB state, so a payload swap must re-key the database too, not only the repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the @@ -106,10 +102,10 @@ Two gotchas make the wiring non-obvious: The Ollama connection is disabled (`ollama.enable=false`). > All three endpoints are bare-metal systemd services managed by this -> repository. Model swaps, context sizing, and spec tuning happen there. See -> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only -> hides it from this picker — the endpoint itself keeps serving every other -> client (the harness, Hermes, API consumers). +> repository; their shape, swaps and tuning live on +> [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Disabling a +> connection here only hides it from this picker — the endpoint itself keeps +> serving every other client (the harness, Hermes, API consumers). ## Container introspection (DB and backend python) @@ -145,9 +141,9 @@ The Ollama connection is disabled (`ollama.enable=false`). models. Response key is `items`, not `data`. 2. `POST /api/chat/completions` per model with a trivial prompt ("Reply with exactly one word: pong") → each returns content `pong`. Reasoning - models need a generous budget: frank-zealot (Qwen3.5 template) returns - empty content + `finish_reason: "length"` at `max_tokens 200` (the - thinking block eats it) and answers `pong` at 1500. + endpoints need a generous budget: thinking consumes `max_tokens`, so a short + one returns empty content + `finish_reason: "length"` (see + [[frank-zealot]]); `frank-zealot` answers `pong` at 1500. 3. A 500 "Server Connection Error" → `podman logs open-webui` for `ClientConnectorError`: a network-reachability problem (host LAN IP vs `host.containers.internal`), not a model problem. @@ -340,8 +336,8 @@ neither. - The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].) - The default admin account is created on first login; secure it immediately. -- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready. -- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`. +- If the backend is loading, OpenWebUI shows connection errors until the endpoint is ready; each endpoint's boot budget is on its own page ([[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). +- Repeated from *The LLM backends*: the on-host endpoints are reached via the **host LAN IP**, never `host.containers.internal`. ## History @@ -390,4 +386,4 @@ neither. - [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)* - [[Hermes]] — the agent OpenWebUI can talk to - [[SearXNG]] — the search engine on this host -- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints +- [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]] — the endpoints behind the picker diff --git a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md index 7beb1cc..e057e24 100644 --- a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md +++ b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md @@ -36,14 +36,14 @@ it is not invisible. ## Why 64 GiB Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding -**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1` -working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose -working set was ~4.3 GiB. That is a decision-time reading, not today's: on +**103.5 GiB of the 121.2 GiB** host, leaving roughly **7.1 GiB** for the then-28 +co-tenant containers whose working set was ~4.3 GiB. How that total is split +between weights, KV pool and working memory is the engine's own ledger — it lives +on [[frank-dragoon]]. That is a decision-time reading, not today's: on 2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB** (`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the `frank-dragoon-halogen` container accounts for **70.33 GiB** -(`podman stats --no-stream`, same day) on the current flash-server payload — -a different model and KV shape than the one measured in September. +(`podman stats --no-stream`, same day). The engine itself warned at the time that when the headroom runs out, frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with no disk activity and no output". Swap is the valve that turns an OOM kill into slow @@ -59,7 +59,7 @@ like this, **never `swapoff` before the replacement is online.** The old 8 GiB held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was genuinely free — the pinned model weights are counted as reclaimable page cache, so the headline number is a lie by roughly the size of the model -(see [[halogen-flash-server]]). Swapping off into that apparent space, with +(see [[frank-dragoon]]). Swapping off into that apparent space, with nothing else to receive the pages, is how you invite the OOM killer. The ordering that avoids it — create and enable the new swap **first**, so the @@ -113,10 +113,10 @@ frank-dragoon `health=200` throughout. ## Caveats -- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are - held deliberately; paging them is a stall, not a relief. More swap does not - make the doubled KV pool affordable — see the reserve/pool ceiling in - [[halogen-flash-server]]. +- **Swap cannot help the pinned weights.** frank-dragoon's weights are pinned + deliberately; paging them is a stall, not a relief. More swap does not make a + larger KV pool affordable — see the reserve/pool ceiling in + [[frank-dragoon]]. - **`vm.swappiness=60` is untested against this workload.** Under real pressure the kernel will happily page anonymous memory, and frank-dragoon's KV pool and working set *are* anonymous. Swapping the KV pool would be far worse @@ -151,6 +151,6 @@ The old swapfile has therefore been purged; nothing on the host still holds it. ## See Also -- [[halogen-flash-server]] — the memory model that motivated this +- [[frank-dragoon]] — the memory model that motivated this - [[Dankclaw System Specs]] — the Strix Halo box - [[Dankclaw Improvements]] — other host-level changes diff --git a/serverAppConfigs-docs/Dankclaw/Voice Packs.md b/serverAppConfigs-docs/Dankclaw/Voice Packs.md index 5ed466f..f25efc8 100644 --- a/serverAppConfigs-docs/Dankclaw/Voice Packs.md +++ b/serverAppConfigs-docs/Dankclaw/Voice Packs.md @@ -233,12 +233,12 @@ loud, not a bug. Work through it in order: they have to be copied there once before a build. - `VOICE_PACK_TOOL` points at the trial's build tree (`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out - at exactly `5916bf7c`), and the two GGUFs at - `/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's - download script puts them — that directory does not exist yet, so until it - does the models have to come from the trial's `models/` directory. -- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs - is possible on that host while serving them is not. + at exactly `5916bf7c`), and the two GGUFs live at + `/home/dank/models/Qwen3-TTS-GGUF/`, the engine slice's fetch target — that + path and its fetch state belong to [[Frank TTS Engine]]. +- Whether the façade (7772) and the engine (8880) are up and serving is deploy + state on [[Frank TTS]] / [[Frank TTS Engine]] — read it there, not from this + census. ## Known gaps, on purpose diff --git a/serverAppConfigs-docs/Dankserver/Plex.md b/serverAppConfigs-docs/Dankserver/Plex.md index 374ca2f..99cbac4 100644 --- a/serverAppConfigs-docs/Dankserver/Plex.md +++ b/serverAppConfigs-docs/Dankserver/Plex.md @@ -30,7 +30,7 @@ public traffic can be fronted by Caddy like every other vhost. | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover | | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration | | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` | -| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` | +| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to [[frank-archon]] | | Deploy | `./scripts/deploy.sh dankserver --service plex` | | Docs | https://support.plex.tv | diff --git a/serverAppConfigs-docs/Infrastructure/Homepage.md b/serverAppConfigs-docs/Infrastructure/Homepage.md index e12aea2..8836544 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage.md @@ -159,10 +159,9 @@ Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs entirely — admin tooling; Seerr is presented as the front door for requests. Each fleet model gained a Hugging Face source link: card field `model.hf` rendered as the cyan `model on huggingface ↗` line under the -role, and the docs `Model:` bullets link the checkpoint names (zealot → -HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon → -Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon → -Qwen/Qwen3.8-Flash-Next-FP8). +role, and the docs `Model:` bullets link the checkpoint names — which +checkpoint each endpoint runs is payload state on +[[frank-dragoon]], [[frank-archon]] and [[frank-zealot]]. **Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth wave):** the amber CRT palette is retired. The page is now deep space @@ -235,11 +234,10 @@ house-DNS advice; the bullet now just says the names resolve on the home network. **Archon card runs the fleet's own model (2026-10-08):** the -`frank-archon` homepage card and docs Model bullet now say the payload -is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link -our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`) -in place of the FP8-era base-model link; capacity line says three -submissions (`--max-num-seqs 3`). +`frank-archon` homepage card and docs Model bullet swapped the FP8-era +base-model link for our own HF publication, and the capacity line for the +endpoint's own concurrency figure; the payload, its publication and the slot +count all live on [[frank-archon]]. ## Design decision — 2026-08-22 diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index 21d318d..e19ec35 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -130,13 +130,14 @@ the blackbox exporter. See [[Prometheus]] → Targets. **A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18, `up{job="frank-zealot"}` was 0 for the entire retained history of the metric — the oldest sample, 2026-10-11, is already 0 — while the service itself answered -`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented** -on `/metrics` unless the server is started with `--metrics`, so the scrape was -refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing: +`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it +is started with the flag, so the scrape was refused at the protocol layer (the +flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical) +stayed firing: the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user -u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02 on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the -`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target +metrics opt-in landed that day; the target reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18). How many payload generations it spanned is not readable from any host artifact. *(unverified 2026-10-18)* @@ -151,7 +152,8 @@ notification log — `dankclaw/observability/quadlet/alertmanager.container:39-4 `/opt/alertmanager/data`.) Remaining known downs are the structural ones above (HTML-scraped probes, Gitea -metrics, ComfyUI), and halogen serves `/metrics` natively, which is why +metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively +([[frank-dragoon]]), which is why `frank-dragoon` has always been green. The active-alert count is a moment-in-time value and should never be documented as a constant: `/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing @@ -167,8 +169,8 @@ arm (`systemctl is-active frank-zealot` → `failed` after `stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past `TimeoutStopSec`, the known shutdown shape for these engines). So `up{job="frank-zealot"} == 1` that day measured the trial, not the production -shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408 -PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract. +shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state, +and it is the repo contract. The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot` reinstalls and starts it, restoring boot persistence and self-healing. Never quote a candidate arm's flags or VRAM as fleet state — the process and its shape diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index ceba08c..31540e7 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -25,7 +25,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — | Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) | | Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) | | Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) | -| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) | +| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) | ## Targets @@ -37,10 +37,10 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) — | `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) | | `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works | | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` | -| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works | -| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) | +| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] | +| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] | -The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18). +The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today. **Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16): @@ -53,15 +53,14 @@ The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`. -**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service. +**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service. Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z. -A scrape cannot express "is this HTTP service answering?" — that is what a -probe is for, and the remedy is the **blackbox exporter** in probe mode -(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real -uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it -is deployed, these three stay red for a structural reason: deleting them would +The remedy for a probe-shaped question is the **blackbox exporter**; the +classification rule and the honesty law for it live in [[Stale Signal Audit]]. +Until the exporter is deployed, these three stay red for a structural reason: +deleting them would hide a real service, and fixing them by publishing databases or enabling metrics would trade a wrong signal for a real exposure. Targets are `host:port`, never URLs. diff --git a/serverAppConfigs-docs/Operations/Container Host Reachability.md b/serverAppConfigs-docs/Operations/Container Host Reachability.md index c6088fb..edda0aa 100644 --- a/serverAppConfigs-docs/Operations/Container Host Reachability.md +++ b/serverAppConfigs-docs/Operations/Container Host Reachability.md @@ -94,8 +94,8 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies `pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals seconds later** under `Restart=on-failure`. The failure looks like it failed, then looks fine, and the missing gate stays invisible because the retry masks -it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless -Quadlet. +it. First seen on a `frank-dragoon` unit cutover; which supervision shape that +era was is in [[frank-dragoon]]'s era table. Gate procedure: stop the old unit AND the new one (a redeploy's running replacement is itself the port-holder — without that stop the gate deadlocks diff --git a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md index 7163609..ce7044a 100644 --- a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md +++ b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md @@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00 ## Endpoint down, connection refused — GPU holder reclaim Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the -service was not crashed. Applies to `frank-medium` (dankserver, root unit), -`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless -Quadlet). +service was not crashed. Applies to the root-unit fleet endpoints: +`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) — +unit identity per endpoint is on its own page ([[frank-archon]], +[[frank-dragoon]], [[frank-zealot]]). **1. Classify the stop before touching anything** ```bash -ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \ - systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts" +ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \ + systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts" ls -l /etc/systemd/system/.service # symlink to repo = unmasked; -> /dev/null = masked ``` @@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep Cards idle → `systemctl start ` and verify (step 5). Cards busy with no prod unit → get the holder, and compare its bytes against the service's -requirement before concluding a start would even fit (frank-medium 35:30 -needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the -restart on allocation, not config). +requirement before concluding a start would even fit. The per-boot requirement +is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]], +[[frank-zealot]]; the retired `frank-medium` split is measured in +[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on +allocation, not config. **3. Identify the holder and prove it is abandoned** @@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed). unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A freshly restarted endpoint under live agent-queue backlog may return empty bodies/timeouts for minutes while `/health` is 200 — re-probe with a long -timeout. First request after a halogen (`frank-dragoon`) restart can go silent -45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`: -read logs **without** sudo (per-user storage). If the restore never happened, +timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see +[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own +log lives (and whether sudo is needed to read it) is on its page. If the +restore never happened, the sudo journal names who and when: `journalctl -b | grep sudo`. ## Scratch vLLM boot on a fleet GPU host — stop-first law @@ -110,13 +114,14 @@ failure modes if skipped: 1. **Expert-offload planner dies on host RAM.** `vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at plan time. A production rank worker holds tens of GiB of unevictable - pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod - boot sees ~86 GiB instead of ~239 and dies at plan time: + pinned expert rows — frank-archon's pinned-RAM ledger, and how much host + RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod + boot sees far less than the bare host and dies at plan time: `ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`. Looks like a planner bug; is pure RAM contention. Signature: container exits within ~60 s, after `Resolved architecture: …` but before worker - init; `free -g` showing `available ≈ 86` while prod runs is the smoking - gun. + init; `free -g` showing an `available` far below the endpoint page's + loaded-host figure while prod runs is the smoking gun. 2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless (the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine only once the prod unit is down. @@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop: Traps, in the order they bite: the **pull aborts on hand copies** ("untracked working tree files would be overwritten by merge" — exactly the file you shipped); `git checkout --` silently eats uncommitted work without step 4; a -runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`, +runtime may **mount files straight out of the checkout** (`scripts/**` — the +serve scripts and the chat template mount from there, see [[frank-archon]] — and `shared/grafana/provisioning/**`) so a hand copy takes effect at next container restart with no deploy; root-owned files from a container write need `chown` back before git operations; and **a restart is not a diff --git a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md index b756208..596fd31 100644 --- a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md +++ b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md @@ -48,8 +48,9 @@ table. Three buckets: re-download. - **retired** — maps to a retired runtime. Safe candidates. -Trap: a directory can hold **both** a dead target and a live drafter -(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the +Trap: a directory can hold **both** a dead target and a live asset — an era's +drafter can sit beside the retired target it came with (per-endpoint era tables: +[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the single file, never the directory, and say so in the manifest. ## 3. The live-reference check — before moving anything @@ -149,13 +150,13 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as ## Refresh per engine — know the recovery path BEFORE deleting -- **frank-dragoon (halogen)**: the engine fetches its own payload via - `HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but - leave it **existing and empty**, then start the unit. **Bind-mount trap:** +- **frank-dragoon**: self-fetching — clear the directory but + leave it **existing and empty**, then start the unit; the download knobs and + the payload's byte count (which is what the re-download costs) are on + [[frank-dragoon]]. **Bind-mount trap:** podman refuses a mount whose source is missing (`Error: statfs /home/dank/models/halogen: no such file or directory`) and - the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB - at 130–240 MB/s ≈ 10–20 min, then load. + the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. - **llama.cpp services**: no self-fetch; the serve script refuses to start without weights. Pull yourself and verify: