docs(vault): service and infrastructure pages stop restating endpoint facts

18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
Frank
2026-10-19 00:56:02 -07:00
parent 219a097b39
commit 6bdfd2853e
18 changed files with 169 additions and 169 deletions
+25 -27
View File
@@ -14,16 +14,16 @@ registry; single source of truth — there is no copy on the servers) and
this file is repo-managed; edits are harness-side only.
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`,
since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and
`frank-zealot` (Dankclaw) — engine, port and card behind each id live on
[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`,
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
files, runbook pages); pages written in the old era name the service,
the harness names the model. The wire now matches: each endpoint's
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
picker/grants, the homepage docs and the raw API all use one id per model
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
note on radiance echoing the requested name).
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon]]
note on the endpoint echoing the requested name).
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
192.168.0.143:7771 each answer with the unit name (`frank-archon`,
@@ -50,10 +50,11 @@ Where the contract lives:
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes). Recomputed from the live files 2026-10-18 — contextWindows
200,000 (archon) / 155,648 (dragoon) / 200,704 (zealot) at
`models.yml:48,131,206`; maxTokens 16,384 / 45,056 / 16,384 at
`models.yml:56,143,212`:
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
follows is a **worked example** recomputed from the live files on 2026-10-18 —
a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
@@ -141,28 +142,25 @@ Use **wire names**, not OMP camelCase. Two traps:
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
also sets it — `Object.assign` is shallow and replaces the whole object,
silently dropping the `reasoning_effort` kwarg the compat policy set.
Top-level `preserve_thinking` is honoured by halogen. Live state 2026-10-18:
frank-archon's radiance entry carries `chat_template_kwargs:
{enable_thinking, preserve_thinking}` in `extraBody` (`models.yml:89-91`)
and nothing else sets that object at that endpoint, so the shallow-merge
hazard is latent rather than firing; the FreeToken-era "verify-and-prune at
the next swap" note is retired with that swap (2026-10-05,
`scripts/serve-frank-archon.sh:9-10`).
Top-level `preserve_thinking` is honoured by frank-dragoon
([[frank-dragoon]]). Live state 2026-10-18: frank-archon's `extraBody`
carries `chat_template_kwargs: {enable_thinking, preserve_thinking}`
(`models.yml:89-91`) and nothing else sets that object at that endpoint, so
the shallow-merge hazard is latent rather than firing.
**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier flash-server
**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier frank-dragoon
builds carried **no server-side sampler defaults**, so an absent
`temperature` decoded greedy while the endpoint looked configured. That era is
over: `HALOGEN_TEMPERATURE/TOP_P/TOP_K` now set the card's thinking recipe
(1.0 / 0.95 / 20) as engine defaults (`scripts/frank-dragoon.env:98-103`,
exported at `scripts/serve-frank-dragoon.sh:186-188`), so an absent temperature
is no longer greedy. frank-dragoon's `extraBody` still carries the same trio —
consistent, not load-bearing. Treat any older "do not drop it or you get
greedy" wording as history.
over — the endpoint now ships its own defaults ([[frank-dragoon]]), so an
absent temperature is no longer greedy. frank-dragoon's `extraBody` still
carries the same trio — consistent, not load-bearing. Treat any older "do not
drop it or you get greedy" wording as history.
**Reasoning-effort folding.** Qwen3.8's template accepts only
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
else ("Unexpected reasoning effort high"). Every endpoint's
`reasoningEffortMap` folds all harness levels onto that set; a role whose
**Reasoning-effort folding.** The set `reasoning_effort` accepts is a property
of the endpoint, not of the harness: [[frank-dragoon]] and [[frank-zealot]]
raise on anything outside `xhigh|medium|low`, while [[frank-archon]] does not
gate the set. Every endpoint's `reasoningEffortMap` folds all harness levels
onto whichever set that endpoint ships; a role whose
level is unmapped fails on its FIRST request, and the failure names Jinja,
not the model.
@@ -22,8 +22,10 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
"behavior pending".
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
whose marker lags. On the big-payload endpoints that is a full model reload
measured in tens of minutes — each endpoint's boot budget is on its own page
([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon
(dankserver:7770) is also the
harness's own model backend, so a blind deploy is a self-inflicted outage for
possibly zero behavioral change.
@@ -46,9 +48,8 @@ possibly zero behavioral change.
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
- confirm the launcher maps each tunable to its flag
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
--enable-prefix-caching`.
- confirm the launcher maps each tunable to its flag
(`grep scripts/launch.sh`).
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
live — **do NOT restart**; reconcile the marker (below). Only if the live
@@ -46,7 +46,7 @@ graph LR
./scripts/deploy.sh dankserver
./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run
```
@@ -62,13 +62,13 @@ service on the other host.
|---|---|---|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw iGPU inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-dragoon]]. |
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw 3090 inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-zealot]]. |
| `frank-archon` | systemd + `systemctl` (root unit) | `deploy_llm_archon` plans from repo paths and gates on the port; it does NOT see host-side unit drift (see [[frank-archon]] → Deploy / verify). |
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
The shared chat template maps to the inference service on the selected target.
The `frank-npu` endpoint is retired (2026-09-11) and has no deploy handler;
The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
an unrelated large-model deployment never touches it.
## Caddy safety
@@ -34,8 +34,9 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
`/metrics` that is disabled (Gitea 404).
- A scrape cannot express "is this HTTP service answering?" — the right probe
is the **blackbox exporter** in probe mode (which also gives guarded vhosts
a real uptime signal, since Prometheus cannot solve an Anubis challenge).
is the **blackbox exporter** in probe mode
(`/probe?target=…&module=http_2xx`), which also gives guarded vhosts
a real uptime signal, since Prometheus cannot solve an Anubis challenge.
- **Fix in the direction of honesty, never of silence:** unreachable target →
mark the homepage card `wip: true` with the reason in `description` (the
checker skips `wip`; `wip` is the existing convention for "real service,
@@ -43,8 +44,8 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
database port on the LAN to make a badge green, and never delete a scrape
job — an absent scrape and a healthy one are indistinguishable.
- Alert rules must not depend on always-red signals — scope them to services
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
exactly that reason).
that can actually be healthy; the `frank-*` scope `LlmEndpointDown` carries
is recorded on [[Prometheus]].
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
`connection refused` to `unsupported Content-Type "text/html"` means the
service came up and only the scrape is wrong. Read the error, not just
@@ -13,11 +13,10 @@ timestamp: 2026-08-26
# Dankclaw Stack
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
FLM v1.0.1) serves sight+sound models. The completion endpoint
`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235).
stack. Its LLM inference endpoints are bare-metal host services, not part of
the container stack; each has its own page — shape, payload, eras and wire
contract live there, nowhere else. See [[frank-dragoon]] and [[frank-zealot]]
(both on this host) and [[frank-archon]] (Dankserver, 192.168.0.235).
```mermaid
graph LR
@@ -28,9 +27,8 @@ graph LR
C --> F[ComfyUI]
C --> D[Grafana]
S --> V[Gluetun VPN]
O --> L[llama.cpp dankserver :7770]
O --> L[LLM endpoints]
O --> H[Hermes]
O --> E[frank-dragoon :7770]
```
## Services
@@ -48,18 +46,17 @@ graph LR
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
| frank-dragoon (host service) | — | `7770` | [[deployment]] |
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
| frank-dragoon (host service) | — | `7770` | [[frank-dragoon]] |
| frank-zealot (host service) | — | `7771` | [[frank-zealot]] |
| frank-archon (host service, Dankserver) | — | `7770` | [[frank-archon]] |
The LLM rows are bare-metal systemd services, not containers. This
repo owns their scripts, units, and knowledge base (see
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver.
The LLM rows are bare-metal systemd services, not containers, and this repo
owns their scripts, units and knowledge base (see
[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on
each port is era state and lives on the endpoint page, not here. `frank-npu`
(the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon`
on the Dankserver port) are both retired — see [[frank-npu]] and
[[frank-medium-radiance-mxfp4]].
## Boot Persistence
+4 -4
View File
@@ -29,7 +29,7 @@ render needs the app installed, not this endpoint.
| | |
| --- | --- |
| Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
| Port | **7772**, published (`PublishPort=7772:7772`); the family is 7770 dragoon, 7771 zealot, 7772 tts |
| Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] |
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only |
| Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
@@ -142,9 +142,9 @@ types: [[Prometheus]] records that the existing HTML probes (`comfyui`, the two
`services` targets) sit **down** with
`received unsupported Content-Type … and no fallback_scrape_protocol specified`.
A `/health` probe job would therefore be structurally never-green, and because
`LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, a permanently red
`frank-tts` target would page on Signal every few minutes for an endpoint that
is fine — the frank-archon lesson pointed the other way. The engine cannot be
`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
Signal every few minutes for an endpoint that is fine. The engine cannot be
scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
`podman healthcheck run`, the two unit journals, and the homepage card, which
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
+11 -10
View File
@@ -78,9 +78,9 @@ is not durable. The tracked, sanitized configuration baseline is
`/data/config.yaml` is not readable without `sudo`):
`dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
`frank-dragoon` — verified from the host with `curl
http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len
200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is
a rootless pasta publish and is IPv4-only);
http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
`127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
and is IPv4-only);
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
2026-10-18, that endpoint serves only `frank-archon`** (`GET
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
@@ -88,7 +88,8 @@ is not durable. The tracked, sanitized configuration baseline is
2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
if the live `config.yaml` still names `frank-medium` the provider is dead
weight and requests will 404 on a nonexistent model. Diff the live config
before trusting either name.
before trusting either name. Which payload sits behind each port, and the
served id it answers to, is on [[frank-dragoon]] and [[frank-archon]].
- The live `config.yaml` contains secrets and user state and is not committed;
the tracked example is the sanitized backup of its non-secret settings.
@@ -227,9 +228,10 @@ resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
What actually breaks, and the fix:
- **`response_format` (json_object / json_schema) is a hard 400 on every
local backend here** — neither FreeToken nor halogen implements constrained
decoding. Config cannot override it: `call_llm` merges the task's
- **`response_format` (json_object / json_schema) is a hard 400 on the local
backends this task was written against** — what each endpoint accepts on the
wire is on [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Config
cannot override it: `call_llm` merges the task's
`extra_body` first and then `update()`s the caller's, so the hardcoded one
wins. Only temperature and max_tokens have reactive retry paths.
- **Fix: move that one task to the Responses transport**, which never
@@ -246,9 +248,8 @@ What actually breaks, and the fix:
effort: none # mandatory, see below
```
Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw
`:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider
name is recorded against — answers only `frank-archon`. The provider/model
Before copying this: which id is served on which port is endpoint state — see
[[frank-zealot]] and [[frank-archon]]. The provider/model
pairing above is the shape that was landed in 2026; re-derive it from the live
config before reusing it.
@@ -61,7 +61,7 @@ podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/model
**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). The decision rests on each endpoint's documented request surface ([[frank-zealot]], [[frank-dragoon]]): zealot's is a known-good target for that call shape, dragoon's structured-output behaviour is **unverified**, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
## Speakers and voices: 1–4, and the IDs are free text
+22 -26
View File
@@ -39,30 +39,26 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
`shared/homepage/services.json`)*.
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host |
|---|---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 2×262,144 exact f16 KV per `scripts/frank-zealot.env`) | Dankclaw |
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
|---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
bridge move; on this network they are wrong — see the reachability gotcha below, which
is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771`
closed the same day (production unit stopped 03:38 per the repo Benchmarking
Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
PQ2_0 with the MTP head, deployed that afternoon at 3 × 262,144 q8_0 KV and
resliced the same evening to **2 × 262,144 over exact f16 KV** — was **deployed
as production**, so the
picker row below was re-keyed to it (see the next paragraph).
is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
that day's payload swap (see [[frank-zealot]] for what was deployed and when).
The `model` table carries exactly two decoration rows, `frank-zealot` and
`frank-dragoon`. Both the context length and the description live **inside
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
`{"description": …}`; there is no `context_length` or `description` column, so
`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18:
`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon`
→ 200,704 (its `params` had kept the pre-lowering 262,144 window from
2026-10-17 while its own description already said 200,704). These rows are
`sqlite3` one-liners against those names fail. `params.context.length` is a copy
of the endpoint's per-slot window, so the authoritative number lives on the
endpoint page ([[frank-zealot]], [[frank-dragoon]]) and the row must follow it —
the 2026-10-18 re-key existed because `frank-dragoon`'s row still carried the
window its payload had before that day's swap. These rows are
live DB state, so a payload swap must re-key the database too, not only the
repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
@@ -106,10 +102,10 @@ Two gotchas make the wiring non-obvious:
The Ollama connection is disabled (`ollama.enable=false`).
> All three endpoints are bare-metal systemd services managed by this
> repository. Model swaps, context sizing, and spec tuning happen there. See
> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only
> hides it from this picker — the endpoint itself keeps serving every other
> client (the harness, Hermes, API consumers).
> repository; their shape, swaps and tuning live on
> [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Disabling a
> connection here only hides it from this picker — the endpoint itself keeps
> serving every other client (the harness, Hermes, API consumers).
## Container introspection (DB and backend python)
@@ -145,9 +141,9 @@ The Ollama connection is disabled (`ollama.enable=false`).
models. Response key is `items`, not `data`.
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
with exactly one word: pong") → each returns content `pong`. Reasoning
models need a generous budget: frank-zealot (Qwen3.5 template) returns
empty content + `finish_reason: "length"` at `max_tokens 200` (the
thinking block eats it) and answers `pong` at 1500.
endpoints need a generous budget: thinking consumes `max_tokens`, so a short
one returns empty content + `finish_reason: "length"` (see
[[frank-zealot]]); `frank-zealot` answers `pong` at 1500.
3. A 500 "Server Connection Error" → `podman logs open-webui` for
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
`host.containers.internal`), not a model problem.
@@ -340,8 +336,8 @@ neither.
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
- The default admin account is created on first login; secure it immediately.
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
- If the backend is loading, OpenWebUI shows connection errors until the endpoint is ready; each endpoint's boot budget is on its own page ([[frank-dragoon]], [[frank-zealot]], [[frank-archon]]).
- Repeated from *The LLM backends*: the on-host endpoints are reached via the **host LAN IP**, never `host.containers.internal`.
## History
@@ -390,4 +386,4 @@ neither.
- [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)*
- [[Hermes]] — the agent OpenWebUI can talk to
- [[SearXNG]] — the search engine on this host
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
- [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]] — the endpoints behind the picker
@@ -36,14 +36,14 @@ it is not invisible.
## Why 64 GiB
Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1`
working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose
working set was ~4.3 GiB. That is a decision-time reading, not today's: on
**103.5 GiB of the 121.2 GiB** host, leaving roughly **7.1 GiB** for the then-28
co-tenant containers whose working set was ~4.3 GiB. How that total is split
between weights, KV pool and working memory is the engine's own ledger — it lives
on [[frank-dragoon]]. That is a decision-time reading, not today's: on
2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
`frank-dragoon-halogen` container accounts for **70.33 GiB**
(`podman stats --no-stream`, same day) on the current flash-server payload —
a different model and KV shape than the one measured in September.
(`podman stats --no-stream`, same day).
The engine itself warned at the time that when the headroom runs out,
frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
no disk activity and no output". Swap is the valve that turns an OOM kill into slow
@@ -59,7 +59,7 @@ like this, **never `swapoff` before the replacement is online.** The old 8 GiB
held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was
genuinely free — the pinned model weights are counted as reclaimable page
cache, so the headline number is a lie by roughly the size of the model
(see [[halogen-flash-server]]). Swapping off into that apparent space, with
(see [[frank-dragoon]]). Swapping off into that apparent space, with
nothing else to receive the pages, is how you invite the OOM killer.
The ordering that avoids it — create and enable the new swap **first**, so the
@@ -113,10 +113,10 @@ frank-dragoon `health=200` throughout.
## Caveats
- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are
held deliberately; paging them is a stall, not a relief. More swap does not
make the doubled KV pool affordable — see the reserve/pool ceiling in
[[halogen-flash-server]].
- **Swap cannot help the pinned weights.** frank-dragoon's weights are pinned
deliberately; paging them is a stall, not a relief. More swap does not make a
larger KV pool affordable — see the reserve/pool ceiling in
[[frank-dragoon]].
- **`vm.swappiness=60` is untested against this workload.** Under real
pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
pool and working set *are* anonymous. Swapping the KV pool would be far worse
@@ -151,6 +151,6 @@ The old swapfile has therefore been purged; nothing on the host still holds it.
## See Also
- [[halogen-flash-server]] — the memory model that motivated this
- [[frank-dragoon]] — the memory model that motivated this
- [[Dankclaw System Specs]] — the Strix Halo box
- [[Dankclaw Improvements]] — other host-level changes
@@ -233,12 +233,12 @@ loud, not a bug. Work through it in order:
they have to be copied there once before a build.
- `VOICE_PACK_TOOL` points at the trial's build tree
(`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
at exactly `5916bf7c`), and the two GGUFs at
`/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's
download script puts them — that directory does not exist yet, so until it
does the models have to come from the trial's `models/` directory.
- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs
is possible on that host while serving them is not.
at exactly `5916bf7c`), and the two GGUFs live at
`/home/dank/models/Qwen3-TTS-GGUF/`, the engine slice's fetch target — that
path and its fetch state belong to [[Frank TTS Engine]].
- Whether the façade (7772) and the engine (8880) are up and serving is deploy
state on [[Frank TTS]] / [[Frank TTS Engine]] — read it there, not from this
census.
## Known gaps, on purpose
+1 -1
View File
@@ -30,7 +30,7 @@ public traffic can be fronted by Caddy like every other vhost.
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` |
| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to [[frank-archon]] |
| Deploy | `./scripts/deploy.sh dankserver --service plex` |
| Docs | https://support.plex.tv |
@@ -159,10 +159,9 @@ Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
entirely — admin tooling; Seerr is presented as the front door for
requests. Each fleet model gained a Hugging Face source link: card field
`model.hf` rendered as the cyan `model on huggingface ↗` line under the
role, and the docs `Model:` bullets link the checkpoint names (zealot →
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon →
Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon →
Qwen/Qwen3.8-Flash-Next-FP8).
role, and the docs `Model:` bullets link the checkpoint names — which
checkpoint each endpoint runs is payload state on
[[frank-dragoon]], [[frank-archon]] and [[frank-zealot]].
**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
wave):** the amber CRT palette is retired. The page is now deep space
@@ -235,11 +234,10 @@ house-DNS advice; the bullet now just says the names resolve on the
home network.
**Archon card runs the fleet's own model (2026-10-08):** the
`frank-archon` homepage card and docs Model bullet now say the payload
is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link
our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)
in place of the FP8-era base-model link; capacity line says three
submissions (`--max-num-seqs 3`).
`frank-archon` homepage card and docs Model bullet swapped the FP8-era
base-model link for our own HF publication, and the capacity line for the
endpoint's own concurrency figure; the payload, its publication and the slot
count all live on [[frank-archon]].
## Design decision — 2026-08-22
@@ -130,13 +130,14 @@ the blackbox exporter. See [[Prometheus]] → Targets.
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented**
on `/metrics` unless the server is started with `--metrics`, so the scrape was
refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing:
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
is started with the flag, so the scrape was refused at the protocol layer (the
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
stayed firing:
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target
metrics opt-in landed that day; the target
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
How many payload generations it spanned is not readable from any host
artifact. *(unverified 2026-10-18)*
@@ -151,7 +152,8 @@ notification log — `dankclaw/observability/quadlet/alertmanager.container:39-4
`/opt/alertmanager/data`.)
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
([[frank-dragoon]]), which is why
`frank-dragoon` has always been green. The active-alert count is a
moment-in-time value and should never be documented as a constant:
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
@@ -167,8 +169,8 @@ arm (`systemctl is-active frank-zealot` → `failed` after
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
`TimeoutStopSec`, the known shutdown shape for these engines). So
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408
PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract.
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
and it is the repo contract.
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
reinstalls and starts it, restoring boot persistence and self-healing. Never
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
@@ -25,7 +25,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
## Targets
@@ -37,10 +37,10 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) |
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] |
The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18).
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
@@ -53,15 +53,14 @@ The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service.
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
A scrape cannot express "is this HTTP service answering?" — that is what a
probe is for, and the remedy is the **blackbox exporter** in probe mode
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
is deployed, these three stay red for a structural reason: deleting them would
The remedy for a probe-shaped question is the **blackbox exporter**; the
classification rule and the honesty law for it live in [[Stale Signal Audit]].
Until the exporter is deployed, these three stay red for a structural reason:
deleting them would
hide a real service, and fixing them by publishing databases or enabling metrics
would trade a wrong signal for a real exposure. Targets are `host:port`, never
URLs.
@@ -94,8 +94,8 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
seconds later** under `Restart=on-failure`. The failure looks like it failed,
then looks fine, and the missing gate stays invisible because the retry masks
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless
Quadlet.
it. First seen on a `frank-dragoon` unit cutover; which supervision shape that
era was is in [[frank-dragoon]]'s era table.
Gate procedure: stop the old unit AND the new one (a redeploy's running
replacement is itself the port-holder — without that stop the gate deadlocks
@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
## Endpoint down, connection refused — GPU holder reclaim
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
Quadlet).
service was not crashed. Applies to the root-unit fleet endpoints:
`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
unit identity per endpoint is on its own page ([[frank-archon]],
[[frank-dragoon]], [[frank-zealot]]).
**1. Classify the stop before touching anything**
```bash
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
```
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
prod unit → get the holder, and compare its bytes against the service's
requirement before concluding a start would even fit (frank-medium 35:30
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
restart on allocation, not config).
requirement before concluding a start would even fit. The per-boot requirement
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
[[frank-zealot]]; the retired `frank-medium` split is measured in
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
allocation, not config.
**3. Identify the holder and prove it is abandoned**
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
freshly restarted endpoint under live agent-queue backlog may return empty
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
read logs **without** sudo (per-user storage). If the restore never happened,
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
log lives (and whether sudo is needed to read it) is on its page. If the
restore never happened,
the sudo journal names who and when: `journalctl -b | grep sudo`.
## Scratch vLLM boot on a fleet GPU host — stop-first law
@@ -110,13 +114,14 @@ failure modes if skipped:
1. **Expert-offload planner dies on host RAM.**
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
plan time. A production rank worker holds tens of GiB of unevictable
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
boot sees ~86 GiB instead of ~239 and dies at plan time:
pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
boot sees far less than the bare host and dies at plan time:
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
Looks like a planner bug; is pure RAM contention. Signature: container
exits within ~60 s, after `Resolved architecture: …` but before worker
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
gun.
init; `free -g` showing an `available` far below the endpoint page's
loaded-host figure while prod runs is the smoking gun.
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
only once the prod unit is down.
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
working tree files would be overwritten by merge" — exactly the file you
shipped); `git checkout --` silently eats uncommitted work without step 4; a
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
runtime may **mount files straight out of the checkout** (`scripts/**` — the
serve scripts and the chat template mount from there, see [[frank-archon]] — and
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
container restart with no deploy; root-owned files from a container write
need `chown` back before git operations; and **a restart is not a
@@ -48,8 +48,9 @@ table. Three buckets:
re-download.
- **retired** — maps to a retired runtime. Safe candidates.
Trap: a directory can hold **both** a dead target and a live drafter
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the
Trap: a directory can hold **both** a dead target and a live asset — an era's
drafter can sit beside the retired target it came with (per-endpoint era tables:
[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the
single file, never the directory, and say so in the manifest.
## 3. The live-reference check — before moving anything
@@ -149,13 +150,13 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
## Refresh per engine — know the recovery path BEFORE deleting
- **frank-dragoon (halogen)**: the engine fetches its own payload via
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
- **frank-dragoon**: self-fetching — clear the directory but
leave it **existing and empty**, then start the unit; the download knobs and
the payload's byte count (which is what the re-download costs) are on
[[frank-dragoon]]. **Bind-mount trap:**
podman refuses a mount whose source is missing
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
at 130–240 MB/s ≈ 10–20 min, then load.
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir.
- **llama.cpp services**: no self-fetch; the serve script refuses to start
without weights. Pull yourself and verify: