docs(vault): service and infrastructure pages stop restating endpoint facts
18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).
Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.
Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
This commit is contained in:
@@ -14,16 +14,16 @@ registry; single source of truth — there is no copy on the servers) and
|
|||||||
this file is repo-managed; edits are harness-side only.
|
this file is repo-managed; edits are harness-side only.
|
||||||
|
|
||||||
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
|
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
|
||||||
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
|
since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and
|
||||||
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
|
`frank-zealot` (Dankclaw) — engine, port and card behind each id live on
|
||||||
:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`,
|
[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`,
|
||||||
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
|
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
|
||||||
files, runbook pages); pages written in the old era name the service,
|
files, runbook pages); pages written in the old era name the service,
|
||||||
the harness names the model. The wire now matches: each endpoint's
|
the harness names the model. The wire now matches: each endpoint's
|
||||||
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
|
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
|
||||||
picker/grants, the homepage docs and the raw API all use one id per model
|
picker/grants, the homepage docs and the raw API all use one id per model
|
||||||
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
|
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon]]
|
||||||
note on radiance echoing the requested name).
|
note on the endpoint echoing the requested name).
|
||||||
|
|
||||||
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
|
Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and
|
||||||
192.168.0.143:7771 each answer with the unit name (`frank-archon`,
|
192.168.0.143:7771 each answer with the unit name (`frank-archon`,
|
||||||
@@ -50,10 +50,11 @@ Where the contract lives:
|
|||||||
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
|
`~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js`
|
||||||
|
|
||||||
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
The arithmetic (the coupled contract every endpoint's maxTokens comment
|
||||||
encodes). Recomputed from the live files 2026-10-18 — contextWindows
|
encodes). The authoritative `contextWindow`/`maxTokens` pair per model is on the
|
||||||
200,000 (archon) / 155,648 (dragoon) / 200,704 (zealot) at
|
endpoint page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]); what
|
||||||
`models.yml:48,131,206`; maxTokens 16,384 / 45,056 / 16,384 at
|
follows is a **worked example** recomputed from the live files on 2026-10-18 —
|
||||||
`models.yml:56,143,212`:
|
a snapshot, not fleet state — with windows 200,000 / 155,648 / 200,704 and
|
||||||
|
maxTokens 16,384 / 45,056 / 16,384 (archon / dragoon / zealot):
|
||||||
|
|
||||||
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
|
1. **Source trigger** = `thresholdPercent × source contextWindow`; at the live
|
||||||
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
|
95 % that is 190,000 (archon), 147,866 (dragoon), 190,669 (zealot). The
|
||||||
@@ -141,28 +142,25 @@ Use **wire names**, not OMP camelCase. Two traps:
|
|||||||
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
||||||
also sets it — `Object.assign` is shallow and replaces the whole object,
|
also sets it — `Object.assign` is shallow and replaces the whole object,
|
||||||
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
||||||
Top-level `preserve_thinking` is honoured by halogen. Live state 2026-10-18:
|
Top-level `preserve_thinking` is honoured by frank-dragoon
|
||||||
frank-archon's radiance entry carries `chat_template_kwargs:
|
([[frank-dragoon]]). Live state 2026-10-18: frank-archon's `extraBody`
|
||||||
{enable_thinking, preserve_thinking}` in `extraBody` (`models.yml:89-91`)
|
carries `chat_template_kwargs: {enable_thinking, preserve_thinking}`
|
||||||
and nothing else sets that object at that endpoint, so the shallow-merge
|
(`models.yml:89-91`) and nothing else sets that object at that endpoint, so
|
||||||
hazard is latent rather than firing; the FreeToken-era "verify-and-prune at
|
the shallow-merge hazard is latent rather than firing.
|
||||||
the next swap" note is retired with that swap (2026-10-05,
|
|
||||||
`scripts/serve-frank-archon.sh:9-10`).
|
|
||||||
|
|
||||||
**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier flash-server
|
**Gotcha (retired 2026-10-17): silent greedy decode.** Earlier frank-dragoon
|
||||||
builds carried **no server-side sampler defaults**, so an absent
|
builds carried **no server-side sampler defaults**, so an absent
|
||||||
`temperature` decoded greedy while the endpoint looked configured. That era is
|
`temperature` decoded greedy while the endpoint looked configured. That era is
|
||||||
over: `HALOGEN_TEMPERATURE/TOP_P/TOP_K` now set the card's thinking recipe
|
over — the endpoint now ships its own defaults ([[frank-dragoon]]), so an
|
||||||
(1.0 / 0.95 / 20) as engine defaults (`scripts/frank-dragoon.env:98-103`,
|
absent temperature is no longer greedy. frank-dragoon's `extraBody` still
|
||||||
exported at `scripts/serve-frank-dragoon.sh:186-188`), so an absent temperature
|
carries the same trio — consistent, not load-bearing. Treat any older "do not
|
||||||
is no longer greedy. frank-dragoon's `extraBody` still carries the same trio —
|
drop it or you get greedy" wording as history.
|
||||||
consistent, not load-bearing. Treat any older "do not drop it or you get
|
|
||||||
greedy" wording as history.
|
|
||||||
|
|
||||||
**Reasoning-effort folding.** Qwen3.8's template accepts only
|
**Reasoning-effort folding.** The set `reasoning_effort` accepts is a property
|
||||||
`{xhigh (default), medium, low}` for `reasoning_effort` and 400s on anything
|
of the endpoint, not of the harness: [[frank-dragoon]] and [[frank-zealot]]
|
||||||
else ("Unexpected reasoning effort high"). Every endpoint's
|
raise on anything outside `xhigh|medium|low`, while [[frank-archon]] does not
|
||||||
`reasoningEffortMap` folds all harness levels onto that set; a role whose
|
gate the set. Every endpoint's `reasoningEffortMap` folds all harness levels
|
||||||
|
onto whichever set that endpoint ships; a role whose
|
||||||
level is unmapped fails on its FIRST request, and the failure names Jinja,
|
level is unmapped fails on its FIRST request, and the failure names Jinja,
|
||||||
not the model.
|
not the model.
|
||||||
|
|
||||||
|
|||||||
@@ -22,8 +22,10 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
|||||||
"behavior pending".
|
"behavior pending".
|
||||||
|
|
||||||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||||||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
whose marker lags. On the big-payload endpoints that is a full model reload
|
||||||
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
|
measured in tens of minutes — each endpoint's boot budget is on its own page
|
||||||
|
([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon
|
||||||
|
(dankserver:7770) is also the
|
||||||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||||||
possibly zero behavioral change.
|
possibly zero behavioral change.
|
||||||
|
|
||||||
@@ -46,9 +48,8 @@ possibly zero behavioral change.
|
|||||||
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
||||||
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
||||||
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
||||||
- confirm the launcher maps each tunable to its flag
|
- confirm the launcher maps each tunable to its flag
|
||||||
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
|
(`grep scripts/launch.sh`).
|
||||||
--enable-prefix-caching`.
|
|
||||||
|
|
||||||
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
||||||
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
||||||
|
|||||||
@@ -46,7 +46,7 @@ graph LR
|
|||||||
./scripts/deploy.sh dankserver
|
./scripts/deploy.sh dankserver
|
||||||
./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target
|
./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target
|
||||||
./scripts/deploy.sh dankclaw --service searxng
|
./scripts/deploy.sh dankclaw --service searxng
|
||||||
./scripts/deploy.sh dankserver --service frank-medium
|
./scripts/deploy.sh dankserver --service frank-archon
|
||||||
./scripts/deploy.sh dankclaw --dry-run
|
./scripts/deploy.sh dankclaw --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -62,13 +62,13 @@ service on the other host.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
||||||
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
||||||
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw iGPU inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-dragoon]]. |
|
||||||
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the Dankclaw 3090 inference service; waits up to 20 minutes for `/health`. Shape, gates and boot budget: [[frank-zealot]]. |
|
||||||
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
|
| `frank-archon` | systemd + `systemctl` (root unit) | `deploy_llm_archon` plans from repo paths and gates on the port; it does NOT see host-side unit drift (see [[frank-archon]] → Deploy / verify). |
|
||||||
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
||||||
|
|
||||||
The shared chat template maps to the inference service on the selected target.
|
The shared chat template maps to the inference service on the selected target.
|
||||||
The `frank-npu` endpoint is retired (2026-09-11) and has no deploy handler;
|
The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
|
||||||
an unrelated large-model deployment never touches it.
|
an unrelated large-model deployment never touches it.
|
||||||
|
|
||||||
## Caddy safety
|
## Caddy safety
|
||||||
|
|||||||
@@ -34,8 +34,9 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
|
|||||||
`/metrics` that is disabled (Gitea 404).
|
`/metrics` that is disabled (Gitea 404).
|
||||||
|
|
||||||
- A scrape cannot express "is this HTTP service answering?" — the right probe
|
- A scrape cannot express "is this HTTP service answering?" — the right probe
|
||||||
is the **blackbox exporter** in probe mode (which also gives guarded vhosts
|
is the **blackbox exporter** in probe mode
|
||||||
a real uptime signal, since Prometheus cannot solve an Anubis challenge).
|
(`/probe?target=…&module=http_2xx`), which also gives guarded vhosts
|
||||||
|
a real uptime signal, since Prometheus cannot solve an Anubis challenge.
|
||||||
- **Fix in the direction of honesty, never of silence:** unreachable target →
|
- **Fix in the direction of honesty, never of silence:** unreachable target →
|
||||||
mark the homepage card `wip: true` with the reason in `description` (the
|
mark the homepage card `wip: true` with the reason in `description` (the
|
||||||
checker skips `wip`; `wip` is the existing convention for "real service,
|
checker skips `wip`; `wip` is the existing convention for "real service,
|
||||||
@@ -43,8 +44,8 @@ port; a "probe" job scraping `/` (HTML → `unsupported Content-Type`); a
|
|||||||
database port on the LAN to make a badge green, and never delete a scrape
|
database port on the LAN to make a badge green, and never delete a scrape
|
||||||
job — an absent scrape and a healthy one are indistinguishable.
|
job — an absent scrape and a healthy one are indistinguishable.
|
||||||
- Alert rules must not depend on always-red signals — scope them to services
|
- Alert rules must not depend on always-red signals — scope them to services
|
||||||
that can actually be healthy (`LlmEndpointDown` is scoped to `frank-*` for
|
that can actually be healthy; the `frank-*` scope `LlmEndpointDown` carries
|
||||||
exactly that reason).
|
is recorded on [[Prometheus]].
|
||||||
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
|
- A target's failure STRING can prove a fix: a `/metrics` scrape flipping from
|
||||||
`connection refused` to `unsupported Content-Type "text/html"` means the
|
`connection refused` to `unsupported Content-Type "text/html"` means the
|
||||||
service came up and only the scrape is wrong. Read the error, not just
|
service came up and only the scrape is wrong. Read the error, not just
|
||||||
|
|||||||
@@ -13,11 +13,10 @@ timestamp: 2026-08-26
|
|||||||
# Dankclaw Stack
|
# Dankclaw Stack
|
||||||
|
|
||||||
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
|
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
|
||||||
stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX
|
stack. Its LLM inference endpoints are bare-metal host services, not part of
|
||||||
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
|
the container stack; each has its own page — shape, payload, eras and wire
|
||||||
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
|
contract live there, nowhere else. See [[frank-dragoon]] and [[frank-zealot]]
|
||||||
FLM v1.0.1) serves sight+sound models. The completion endpoint
|
(both on this host) and [[frank-archon]] (Dankserver, 192.168.0.235).
|
||||||
`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235).
|
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
graph LR
|
graph LR
|
||||||
@@ -28,9 +27,8 @@ graph LR
|
|||||||
C --> F[ComfyUI]
|
C --> F[ComfyUI]
|
||||||
C --> D[Grafana]
|
C --> D[Grafana]
|
||||||
S --> V[Gluetun VPN]
|
S --> V[Gluetun VPN]
|
||||||
O --> L[llama.cpp dankserver :7770]
|
O --> L[LLM endpoints]
|
||||||
O --> H[Hermes]
|
O --> H[Hermes]
|
||||||
O --> E[frank-dragoon :7770]
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Services
|
## Services
|
||||||
@@ -48,18 +46,17 @@ graph LR
|
|||||||
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
|
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
|
||||||
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
|
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
|
||||||
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
|
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
|
||||||
| frank-dragoon (host service) | — | `7770` | [[deployment]] |
|
| frank-dragoon (host service) | — | `7770` | [[frank-dragoon]] |
|
||||||
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
|
| frank-zealot (host service) | — | `7771` | [[frank-zealot]] |
|
||||||
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
|
| frank-archon (host service, Dankserver) | — | `7770` | [[frank-archon]] |
|
||||||
|
|
||||||
The LLM rows are bare-metal systemd services, not containers. This
|
The LLM rows are bare-metal systemd services, not containers, and this repo
|
||||||
repo owns their scripts, units, and knowledge base (see
|
owns their scripts, units and knowledge base (see
|
||||||
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
|
[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on
|
||||||
`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
|
each port is era state and lives on the endpoint page, not here. `frank-npu`
|
||||||
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
|
(the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon`
|
||||||
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
|
on the Dankserver port) are both retired — see [[frank-npu]] and
|
||||||
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
|
[[frank-medium-radiance-mxfp4]].
|
||||||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver.
|
|
||||||
|
|
||||||
## Boot Persistence
|
## Boot Persistence
|
||||||
|
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ render needs the app installed, not this endpoint.
|
|||||||
| | |
|
| | |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
|
| Host | Dankclaw (`192.168.0.143`), rootless podman, user unit `frank-tts.service` |
|
||||||
| Port | **7772**, published (`PublishPort=7772:7772`); the family is 7770 dragoon, 7771 zealot, 7772 tts |
|
| Port | **7772**, published (`PublishPort=7772:7772`) — the rest of the `frank-*` port family is in [[Dankclaw Stack]] |
|
||||||
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only |
|
| Routes | `GET /health`, `GET /v1/audio/voices`, `POST /v1/audio/speech`; everything else 404, and speech is POST-only |
|
||||||
| Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
|
| Image | `localhost/franksdank/frank-tts:latest`, built from `dankclaw/voiceservice/Containerfile`, `Pull=never` |
|
||||||
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
|
| Networks | `dankclaw_default` first (it owns the default route and the consumers), then `frank-tts-engine.network` (10.89.5.0/24, the bridge to the engine) |
|
||||||
@@ -142,9 +142,9 @@ types: [[Prometheus]] records that the existing HTML probes (`comfyui`, the two
|
|||||||
`services` targets) sit **down** with
|
`services` targets) sit **down** with
|
||||||
`received unsupported Content-Type … and no fallback_scrape_protocol specified`.
|
`received unsupported Content-Type … and no fallback_scrape_protocol specified`.
|
||||||
A `/health` probe job would therefore be structurally never-green, and because
|
A `/health` probe job would therefore be structurally never-green, and because
|
||||||
`LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, a permanently red
|
`LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set
|
||||||
`frank-tts` target would page on Signal every few minutes for an endpoint that
|
lives on [[Prometheus]]), a permanently red `frank-tts` target would page on
|
||||||
is fine — the frank-archon lesson pointed the other way. The engine cannot be
|
Signal every few minutes for an endpoint that is fine. The engine cannot be
|
||||||
scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
|
scraped at all: it publishes nothing. Status lives in `systemctl --user status`,
|
||||||
`podman healthcheck run`, the two unit journals, and the homepage card, which
|
`podman healthcheck run`, the two unit journals, and the homepage card, which
|
||||||
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
|
probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the
|
||||||
|
|||||||
@@ -78,9 +78,9 @@ is not durable. The tracked, sanitized configuration baseline is
|
|||||||
`/data/config.yaml` is not readable without `sudo`):
|
`/data/config.yaml` is not readable without `sudo`):
|
||||||
`dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
|
`dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes
|
||||||
`frank-dragoon` — verified from the host with `curl
|
`frank-dragoon` — verified from the host with `curl
|
||||||
http://127.0.0.1:7770/v1/models` → single id `frank-dragoon`, `max_model_len
|
http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write
|
||||||
200704` (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is
|
`127.0.0.1`, not `localhost`, because that port is a rootless pasta publish
|
||||||
a rootless pasta publish and is IPv4-only);
|
and is IPv4-only);
|
||||||
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
|
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of
|
||||||
2026-10-18, that endpoint serves only `frank-archon`** (`GET
|
2026-10-18, that endpoint serves only `frank-archon`** (`GET
|
||||||
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
|
http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name
|
||||||
@@ -88,7 +88,8 @@ is not durable. The tracked, sanitized configuration baseline is
|
|||||||
2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
|
2026-09-15 — is retired-era: nothing on that endpoint answers to it today, so
|
||||||
if the live `config.yaml` still names `frank-medium` the provider is dead
|
if the live `config.yaml` still names `frank-medium` the provider is dead
|
||||||
weight and requests will 404 on a nonexistent model. Diff the live config
|
weight and requests will 404 on a nonexistent model. Diff the live config
|
||||||
before trusting either name.
|
before trusting either name. Which payload sits behind each port, and the
|
||||||
|
served id it answers to, is on [[frank-dragoon]] and [[frank-archon]].
|
||||||
- The live `config.yaml` contains secrets and user state and is not committed;
|
- The live `config.yaml` contains secrets and user state and is not committed;
|
||||||
the tracked example is the sanitized backup of its non-secret settings.
|
the tracked example is the sanitized backup of its non-secret settings.
|
||||||
|
|
||||||
@@ -227,9 +228,10 @@ resp = call_llm(task="title_generation", messages=[...], max_tokens=64,
|
|||||||
|
|
||||||
What actually breaks, and the fix:
|
What actually breaks, and the fix:
|
||||||
|
|
||||||
- **`response_format` (json_object / json_schema) is a hard 400 on every
|
- **`response_format` (json_object / json_schema) is a hard 400 on the local
|
||||||
local backend here** — neither FreeToken nor halogen implements constrained
|
backends this task was written against** — what each endpoint accepts on the
|
||||||
decoding. Config cannot override it: `call_llm` merges the task's
|
wire is on [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Config
|
||||||
|
cannot override it: `call_llm` merges the task's
|
||||||
`extra_body` first and then `update()`s the caller's, so the hardcoded one
|
`extra_body` first and then `update()`s the caller's, so the hardcoded one
|
||||||
wins. Only temperature and max_tokens have reactive retry paths.
|
wins. Only temperature and max_tokens have reactive retry paths.
|
||||||
- **Fix: move that one task to the Responses transport**, which never
|
- **Fix: move that one task to the Responses transport**, which never
|
||||||
@@ -246,9 +248,8 @@ What actually breaks, and the fix:
|
|||||||
effort: none # mandatory, see below
|
effort: none # mandatory, see below
|
||||||
```
|
```
|
||||||
|
|
||||||
Before copying this: as of 2026-10-18 `frank-zealot` is served on **Dankclaw
|
Before copying this: which id is served on which port is endpoint state — see
|
||||||
`:7771`**, and `http://192.168.0.235:7770/v1` — the endpoint this provider
|
[[frank-zealot]] and [[frank-archon]]. The provider/model
|
||||||
name is recorded against — answers only `frank-archon`. The provider/model
|
|
||||||
pairing above is the shape that was landed in 2026; re-derive it from the live
|
pairing above is the shape that was landed in 2026; re-derive it from the live
|
||||||
config before reusing it.
|
config before reusing it.
|
||||||
|
|
||||||
|
|||||||
@@ -61,7 +61,7 @@ podman exec open-notebook curl -sS http://host.containers.internal:7771/v1/model
|
|||||||
|
|
||||||
**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
|
**The blocker moved; it did not vanish.** frank-tts and frank-tts-engine are **deployed** (2026-10-18) and answer on 7772, so the voice leg works end to end — `podman exec open-notebook curl -sS http://frank-tts:7772/health` is the check to run the day the app exists. What still blocks a render is open-notebook itself: not installed on this host yet. When it is, the two provider rows above are the only wiring, and the seeded speaker profiles carrying OpenAI voice names are the first trap.
|
||||||
|
|
||||||
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). llama.cpp's OpenAI-compatible endpoint is a known-good target for that; frank-dragoon's halogen runtime has **unverified** structured-output behaviour, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
|
**Why frank-zealot and not frank-dragoon:** podcast scriptwriting asks the model for a JSON-schema response (with a plain-JSON fallback). The decision rests on each endpoint's documented request surface ([[frank-zealot]], [[frank-dragoon]]): zealot's is a known-good target for that call shape, dragoon's structured-output behaviour is **unverified**, so it is the unproven alternative, not the fallback. `ESPERANTO_LLM_TIMEOUT=180` (the app's own default) covers an outline or transcript call on zealot comfortably; upstream's note is to keep it under 600 so a slow call fails with a clear error instead of a UI timeout.
|
||||||
|
|
||||||
## Speakers and voices: 1–4, and the IDs are free text
|
## Speakers and voices: 1–4, and the IDs are free text
|
||||||
|
|
||||||
|
|||||||
@@ -39,30 +39,26 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`** *(unve
|
|||||||
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
|
`frank-dragoon`/`frank-zealot`/`frank-archon`, per the cards in
|
||||||
`shared/homepage/services.json`)*.
|
`shared/homepage/services.json`)*.
|
||||||
|
|
||||||
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Runtime | Host |
|
| Connection (`openai.api_configs` idx) | URL as recorded 2026-10-09 | Enabled | `model_ids` | Endpoint page |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
|
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | [[frank-dragoon]] |
|
||||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
|
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | [[frank-archon]] |
|
||||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090; production shape 2×262,144 exact f16 KV per `scripts/frank-zealot.env`) | Dankclaw |
|
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | [[frank-zealot]] |
|
||||||
|
|
||||||
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
|
The URLs in the first column are the DB's recorded values and predate the 2026-08-22
|
||||||
bridge move; on this network they are wrong — see the reachability gotcha below, which
|
bridge move; on this network they are wrong — see the reachability gotcha below, which
|
||||||
is the corrected law. Note also that the 2026-10-18 **bench window** on `:7771`
|
is the corrected law. The picker row for `frank-zealot` was re-keyed on 2026-10-18 after
|
||||||
closed the same day (production unit stopped 03:38 per the repo Benchmarking
|
that day's payload swap (see [[frank-zealot]] for what was deployed and when).
|
||||||
Policy while candidate arms held the port): the candidate — Ternary-Bonsai-2-**9B**
|
|
||||||
PQ2_0 with the MTP head, deployed that afternoon at 3 × 262,144 q8_0 KV and
|
|
||||||
resliced the same evening to **2 × 262,144 over exact f16 KV** — was **deployed
|
|
||||||
as production**, so the
|
|
||||||
picker row below was re-keyed to it (see the next paragraph).
|
|
||||||
|
|
||||||
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
The `model` table carries exactly two decoration rows, `frank-zealot` and
|
||||||
`frank-dragoon`. Both the context length and the description live **inside
|
`frank-dragoon`. Both the context length and the description live **inside
|
||||||
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
|
JSON columns** — `params` holds `{"context": {"length": …}}` and `meta` holds
|
||||||
`{"description": …}`; there is no `context_length` or `description` column, so
|
`{"description": …}`; there is no `context_length` or `description` column, so
|
||||||
`sqlite3` one-liners against those names fail. Re-keyed 2026-10-18:
|
`sqlite3` one-liners against those names fail. `params.context.length` is a copy
|
||||||
`frank-zealot` → 262,144 with the Bonsai-2-9B description, and `frank-dragoon`
|
of the endpoint's per-slot window, so the authoritative number lives on the
|
||||||
→ 200,704 (its `params` had kept the pre-lowering 262,144 window from
|
endpoint page ([[frank-zealot]], [[frank-dragoon]]) and the row must follow it —
|
||||||
2026-10-17 while its own description already said 200,704). These rows are
|
the 2026-10-18 re-key existed because `frank-dragoon`'s row still carried the
|
||||||
|
window its payload had before that day's swap. These rows are
|
||||||
live DB state, so a payload swap must re-key the database too, not only the
|
live DB state, so a payload swap must re-key the database too, not only the
|
||||||
repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
|
repo. There is no `sqlite3` binary on Dankclaw — use `python3 -c` with the
|
||||||
stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
|
stdlib `sqlite3` module, and take a `Connection.backup()` snapshot first (the
|
||||||
@@ -106,10 +102,10 @@ Two gotchas make the wiring non-obvious:
|
|||||||
The Ollama connection is disabled (`ollama.enable=false`).
|
The Ollama connection is disabled (`ollama.enable=false`).
|
||||||
|
|
||||||
> All three endpoints are bare-metal systemd services managed by this
|
> All three endpoints are bare-metal systemd services managed by this
|
||||||
> repository. Model swaps, context sizing, and spec tuning happen there. See
|
> repository; their shape, swaps and tuning live on
|
||||||
> the [LLM knowledge base](../LLM/index.md). Disabling a connection here only
|
> [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]]. Disabling a
|
||||||
> hides it from this picker — the endpoint itself keeps serving every other
|
> connection here only hides it from this picker — the endpoint itself keeps
|
||||||
> client (the harness, Hermes, API consumers).
|
> serving every other client (the harness, Hermes, API consumers).
|
||||||
|
|
||||||
## Container introspection (DB and backend python)
|
## Container introspection (DB and backend python)
|
||||||
|
|
||||||
@@ -145,9 +141,9 @@ The Ollama connection is disabled (`ollama.enable=false`).
|
|||||||
models. Response key is `items`, not `data`.
|
models. Response key is `items`, not `data`.
|
||||||
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
|
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
|
||||||
with exactly one word: pong") → each returns content `pong`. Reasoning
|
with exactly one word: pong") → each returns content `pong`. Reasoning
|
||||||
models need a generous budget: frank-zealot (Qwen3.5 template) returns
|
endpoints need a generous budget: thinking consumes `max_tokens`, so a short
|
||||||
empty content + `finish_reason: "length"` at `max_tokens 200` (the
|
one returns empty content + `finish_reason: "length"` (see
|
||||||
thinking block eats it) and answers `pong` at 1500.
|
[[frank-zealot]]); `frank-zealot` answers `pong` at 1500.
|
||||||
3. A 500 "Server Connection Error" → `podman logs open-webui` for
|
3. A 500 "Server Connection Error" → `podman logs open-webui` for
|
||||||
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
|
`ClientConnectorError`: a network-reachability problem (host LAN IP vs
|
||||||
`host.containers.internal`), not a model problem.
|
`host.containers.internal`), not a model problem.
|
||||||
@@ -340,8 +336,8 @@ neither.
|
|||||||
|
|
||||||
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
|
- The image tracks upstream `main`; pin a digest in the quadlet unit if reproducible rollback matters more than tracking. (The old "CPU-only host" framing is stale: this APU runs the gfx1151 ROCm build — see [[ComfyUI]].)
|
||||||
- The default admin account is created on first login; secure it immediately.
|
- The default admin account is created on first login; secure it immediately.
|
||||||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
- If the backend is loading, OpenWebUI shows connection errors until the endpoint is ready; each endpoint's boot budget is on its own page ([[frank-dragoon]], [[frank-zealot]], [[frank-archon]]).
|
||||||
- The on-host LLM services (Dankclaw `:7770`, `:7771`) are reached via the **host LAN IP `192.168.0.143`**, which is exactly what the deployed unit does (`open-webui.container:32`). The older "use `host.containers.internal`" advice is retired for this container: it is a slirp4netns/pasta-only name and the unit has been on the pinned bridge `systemd-open-webui` since 2026-08-22, where that name does not resolve (`open-webui.network:15-19`). Both endpoints are published on all interfaces today (`ss -ltn` → `*:7770`, `0.0.0.0:7771`; `curl http://192.168.0.143:7771/health` → 200, 2026-10-18). The Dankserver backend is a different host, reached via its LAN IP `192.168.0.235:7770`.
|
- Repeated from *The LLM backends*: the on-host endpoints are reached via the **host LAN IP**, never `host.containers.internal`.
|
||||||
|
|
||||||
## History
|
## History
|
||||||
|
|
||||||
@@ -390,4 +386,4 @@ neither.
|
|||||||
- [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)*
|
- [[Frank TTS]] — the speech endpoint behind *Text-to-speech (frank-tts)*
|
||||||
- [[Hermes]] — the agent OpenWebUI can talk to
|
- [[Hermes]] — the agent OpenWebUI can talk to
|
||||||
- [[SearXNG]] — the search engine on this host
|
- [[SearXNG]] — the search engine on this host
|
||||||
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
|
- [[frank-dragoon]] / [[frank-zealot]] / [[frank-archon]] — the endpoints behind the picker
|
||||||
|
|||||||
@@ -36,14 +36,14 @@ it is not invisible.
|
|||||||
## Why 64 GiB
|
## Why 64 GiB
|
||||||
|
|
||||||
Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
|
Dankclaw is shared. The 2026-09-12 decision measured `frank-dragoon` alone holding
|
||||||
**103.5 GiB of the 121.2 GiB** host (`68.0` weights + `14.4` KV pool + `21.1`
|
**103.5 GiB of the 121.2 GiB** host, leaving roughly **7.1 GiB** for the then-28
|
||||||
working), leaving roughly **7.1 GiB** for the then-28 co-tenant containers whose
|
co-tenant containers whose working set was ~4.3 GiB. How that total is split
|
||||||
working set was ~4.3 GiB. That is a decision-time reading, not today's: on
|
between weights, KV pool and working memory is the engine's own ledger — it lives
|
||||||
|
on [[frank-dragoon]]. That is a decision-time reading, not today's: on
|
||||||
2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
|
2026-10-18 the host reports **91 GiB used / 29 GiB available of 121 GiB**
|
||||||
(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
|
(`free -h`), 30 containers are up (`podman ps -q | wc -l`), and the
|
||||||
`frank-dragoon-halogen` container accounts for **70.33 GiB**
|
`frank-dragoon-halogen` container accounts for **70.33 GiB**
|
||||||
(`podman stats --no-stream`, same day) on the current flash-server payload —
|
(`podman stats --no-stream`, same day).
|
||||||
a different model and KV shape than the one measured in September.
|
|
||||||
The engine itself warned at the time that when the headroom runs out,
|
The engine itself warned at the time that when the headroom runs out,
|
||||||
frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
|
frank-dragoon and its neighbours "can stall for minutes at a time at full CPU with
|
||||||
no disk activity and no output". Swap is the valve that turns an OOM kill into slow
|
no disk activity and no output". Swap is the valve that turns an OOM kill into slow
|
||||||
@@ -59,7 +59,7 @@ like this, **never `swapoff` before the replacement is online.** The old 8 GiB
|
|||||||
held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was
|
held 4 GiB of live pages, and `MemAvailable` read 73 GiB while only ~5 GiB was
|
||||||
genuinely free — the pinned model weights are counted as reclaimable page
|
genuinely free — the pinned model weights are counted as reclaimable page
|
||||||
cache, so the headline number is a lie by roughly the size of the model
|
cache, so the headline number is a lie by roughly the size of the model
|
||||||
(see [[halogen-flash-server]]). Swapping off into that apparent space, with
|
(see [[frank-dragoon]]). Swapping off into that apparent space, with
|
||||||
nothing else to receive the pages, is how you invite the OOM killer.
|
nothing else to receive the pages, is how you invite the OOM killer.
|
||||||
|
|
||||||
The ordering that avoids it — create and enable the new swap **first**, so the
|
The ordering that avoids it — create and enable the new swap **first**, so the
|
||||||
@@ -113,10 +113,10 @@ frank-dragoon `health=200` throughout.
|
|||||||
|
|
||||||
## Caveats
|
## Caveats
|
||||||
|
|
||||||
- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are
|
- **Swap cannot help the pinned weights.** frank-dragoon's weights are pinned
|
||||||
held deliberately; paging them is a stall, not a relief. More swap does not
|
deliberately; paging them is a stall, not a relief. More swap does not make a
|
||||||
make the doubled KV pool affordable — see the reserve/pool ceiling in
|
larger KV pool affordable — see the reserve/pool ceiling in
|
||||||
[[halogen-flash-server]].
|
[[frank-dragoon]].
|
||||||
- **`vm.swappiness=60` is untested against this workload.** Under real
|
- **`vm.swappiness=60` is untested against this workload.** Under real
|
||||||
pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
|
pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
|
||||||
pool and working set *are* anonymous. Swapping the KV pool would be far worse
|
pool and working set *are* anonymous. Swapping the KV pool would be far worse
|
||||||
@@ -151,6 +151,6 @@ The old swapfile has therefore been purged; nothing on the host still holds it.
|
|||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|
||||||
- [[halogen-flash-server]] — the memory model that motivated this
|
- [[frank-dragoon]] — the memory model that motivated this
|
||||||
- [[Dankclaw System Specs]] — the Strix Halo box
|
- [[Dankclaw System Specs]] — the Strix Halo box
|
||||||
- [[Dankclaw Improvements]] — other host-level changes
|
- [[Dankclaw Improvements]] — other host-level changes
|
||||||
|
|||||||
@@ -233,12 +233,12 @@ loud, not a bug. Work through it in order:
|
|||||||
they have to be copied there once before a build.
|
they have to be copied there once before a build.
|
||||||
- `VOICE_PACK_TOOL` points at the trial's build tree
|
- `VOICE_PACK_TOOL` points at the trial's build tree
|
||||||
(`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
|
(`/home/dank/voice-audition/qwentts/qwentts.cpp/build/qwen-codec`, checked out
|
||||||
at exactly `5916bf7c`), and the two GGUFs at
|
at exactly `5916bf7c`), and the two GGUFs live at
|
||||||
`/home/dank/models/Qwen3-TTS-GGUF/`, which is where the engine slice's
|
`/home/dank/models/Qwen3-TTS-GGUF/`, the engine slice's fetch target — that
|
||||||
download script puts them — that directory does not exist yet, so until it
|
path and its fetch state belong to [[Frank TTS Engine]].
|
||||||
does the models have to come from the trial's `models/` directory.
|
- Whether the façade (7772) and the engine (8880) are up and serving is deploy
|
||||||
- Nothing is deployed: nothing listens on 7772 or 8880 today, so building packs
|
state on [[Frank TTS]] / [[Frank TTS Engine]] — read it there, not from this
|
||||||
is possible on that host while serving them is not.
|
census.
|
||||||
|
|
||||||
## Known gaps, on purpose
|
## Known gaps, on purpose
|
||||||
|
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ public traffic can be fronted by Caddy like every other vhost.
|
|||||||
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
| Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
|
||||||
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
| Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
|
||||||
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
| Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
|
||||||
| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` |
|
| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to [[frank-archon]] |
|
||||||
| Deploy | `./scripts/deploy.sh dankserver --service plex` |
|
| Deploy | `./scripts/deploy.sh dankserver --service plex` |
|
||||||
| Docs | https://support.plex.tv |
|
| Docs | https://support.plex.tv |
|
||||||
|
|
||||||
|
|||||||
@@ -159,10 +159,9 @@ Sonarr, Radarr, Lidarr, Readarr, qBittorrent) is out of the family docs
|
|||||||
entirely — admin tooling; Seerr is presented as the front door for
|
entirely — admin tooling; Seerr is presented as the front door for
|
||||||
requests. Each fleet model gained a Hugging Face source link: card field
|
requests. Each fleet model gained a Hugging Face source link: card field
|
||||||
`model.hf` rendered as the cyan `model on huggingface ↗` line under the
|
`model.hf` rendered as the cyan `model on huggingface ↗` line under the
|
||||||
role, and the docs `Model:` bullets link the checkpoint names (zealot →
|
role, and the docs `Model:` bullets link the checkpoint names — which
|
||||||
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive, dragoon →
|
checkpoint each endpoint runs is payload state on
|
||||||
Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated, archon →
|
[[frank-dragoon]], [[frank-archon]] and [[frank-zealot]].
|
||||||
Qwen/Qwen3.8-Flash-Next-FP8).
|
|
||||||
|
|
||||||
**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
|
**Navy Protoss console, maximal cornering, font swap (2026-10-08, sixth
|
||||||
wave):** the amber CRT palette is retired. The page is now deep space
|
wave):** the amber CRT palette is retired. The page is now deep space
|
||||||
@@ -235,11 +234,10 @@ house-DNS advice; the bullet now just says the names resolve on the
|
|||||||
home network.
|
home network.
|
||||||
|
|
||||||
**Archon card runs the fleet's own model (2026-10-08):** the
|
**Archon card runs the fleet's own model (2026-10-08):** the
|
||||||
`frank-archon` homepage card and docs Model bullet now say the payload
|
`frank-archon` homepage card and docs Model bullet swapped the FP8-era
|
||||||
is our in-house GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next and link
|
base-model link for our own HF publication, and the capacity line for the
|
||||||
our HF publication (`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)
|
endpoint's own concurrency figure; the payload, its publication and the slot
|
||||||
in place of the FP8-era base-model link; capacity line says three
|
count all live on [[frank-archon]].
|
||||||
submissions (`--max-num-seqs 3`).
|
|
||||||
|
|
||||||
## Design decision — 2026-08-22
|
## Design decision — 2026-08-22
|
||||||
|
|
||||||
|
|||||||
@@ -130,13 +130,14 @@ the blackbox exporter. See [[Prometheus]] → Targets.
|
|||||||
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
||||||
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
||||||
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
||||||
`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented**
|
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
|
||||||
on `/metrics` unless the server is started with `--metrics`, so the scrape was
|
is started with the flag, so the scrape was refused at the protocol layer (the
|
||||||
refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing:
|
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
|
||||||
|
stayed firing:
|
||||||
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
||||||
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
||||||
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
||||||
`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target
|
metrics opt-in landed that day; the target
|
||||||
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
||||||
How many payload generations it spanned is not readable from any host
|
How many payload generations it spanned is not readable from any host
|
||||||
artifact. *(unverified 2026-10-18)*
|
artifact. *(unverified 2026-10-18)*
|
||||||
@@ -151,7 +152,8 @@ notification log — `dankclaw/observability/quadlet/alertmanager.container:39-4
|
|||||||
`/opt/alertmanager/data`.)
|
`/opt/alertmanager/data`.)
|
||||||
|
|
||||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||||
metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
|
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
|
||||||
|
([[frank-dragoon]]), which is why
|
||||||
`frank-dragoon` has always been green. The active-alert count is a
|
`frank-dragoon` has always been green. The active-alert count is a
|
||||||
moment-in-time value and should never be documented as a constant:
|
moment-in-time value and should never be documented as a constant:
|
||||||
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
||||||
@@ -167,8 +169,8 @@ arm (`systemctl is-active frank-zealot` → `failed` after
|
|||||||
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
||||||
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
||||||
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
||||||
shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408
|
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
|
||||||
PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract.
|
and it is the repo contract.
|
||||||
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
||||||
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
||||||
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
||||||
|
|||||||
@@ -25,7 +25,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
|||||||
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
|
| Config | `/opt/prometheus/config/prometheus.yml`, installed from `shared/prometheus/prometheus.yml` on every deploy, together with `/opt/prometheus/config/rules/llm-fleet.yml` from `shared/prometheus/rules/llm-fleet.yml` (`scripts/deploy-remote.sh:219-227`) |
|
||||||
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
|
| Data | `/opt/prometheus/data` (TSDB, `prometheus.container:52`) |
|
||||||
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
|
| Retention | `--storage.tsdb.retention.time=30d` (`prometheus.container:49`) |
|
||||||
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
|
| Alerting | **4 alert rules + 2 recording rules** (`shared/prometheus/rules/llm-fleet.yml`: alerts `LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, `LlmEndpointDown`; recordings `job:llm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m`), confirmed against the running server: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) lists 4 alerting + 2 recording. `LlmEndpointDown` is scoped `up{job=~"frank-.*"} == 0`, so only the `frank-*` LLM jobs can page on it — any other job that is structurally never-green must stay out of that family ([[Stale Signal Audit]]). Evaluated **and delivered** since 2026-09-17: the `alerting:` block ships to `alertmanager:9093` (`prometheus.yml:19-22`) → Signal relay → signal-cli (see [[Alertmanager]]) |
|
||||||
|
|
||||||
## Targets
|
## Targets
|
||||||
|
|
||||||
@@ -37,10 +37,10 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
|||||||
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
|
| `node-dankclaw` | `host.containers.internal:9100` | host metrics from [[Node-Exporter]] — a net=host bind, so via the gateway (the LAN address is unreachable from this bridge container) |
|
||||||
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
|
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
|
||||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
||||||
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — the radiance engine since 2026-10-05, exporting vLLM-compatible names plus `radiance:*` (`prometheus.yml:122-131`); a different machine, so the LAN address works |
|
| `frank-archon` | `192.168.0.235:7770` | the LLM fleet — a different machine, so the LAN address works (`prometheus.yml:122-131`); engine, metric families and their meaning live on [[frank-archon]] |
|
||||||
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — both export the `llamacpp:` namespace (halogen included: `curl -s http://127.0.0.1:7770/metrics` → `llamacpp:prompt_tokens_total`, `text/plain; version=0.0.4`, 2026-10-18). They are addressed through the gateway because they are **same-host** — `frank-zealot` is a host-process bind, `frank-dragoon` a container publishing `0.0.0.0:7770->8731`, not a net=host pair (`prometheus.yml:96-99`, `:108-120`) |
|
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet. They are addressed through the gateway because they are **same-host** — a host-process bind and a published container port are equally invisible to a bridge container at the host's LAN address (`prometheus.yml:96-99`, `:108-120`); what each endpoint exports is on [[frank-dragoon]] and [[frank-zealot]] |
|
||||||
|
|
||||||
The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint left the fleet, `frank-archon` briefly had no exporter at all (the FreeToken-era job scraped 404s forever) and re-joined on radiance the same day, 2026-10-05 (`prometheus.yml:101-107`). No `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18).
|
The `frank-medium` job is **retired**: no `frank-medium` job exists in the config or in `/api/v1/targets` (2026-10-18), and the endpoint itself left the fleet — its history is on [[frank-archon]], which holds the port today.
|
||||||
|
|
||||||
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
|
**Three jobs read down today, and they are not outages** (re-audited against `curl -s http://127.0.0.1:9090/api/v1/targets`, 2026-10-18; first audited 2026-09-16):
|
||||||
|
|
||||||
@@ -53,15 +53,14 @@ The `frank-medium` job and its `vllm:` namespace are **retired**: that endpoint
|
|||||||
|
|
||||||
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
|
Gitea's exporter stays off until `[metrics] ENABLED = true` in its own config (no metrics env is set in the container, and `curl -s http://127.0.0.1:3000/metrics` → 404, 2026-10-18); ComfyUI and the two `services` probes are HTML pages, which Prometheus 3 refuses as metrics. Everything else is green: `prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, `frank-zealot`, `frank-dragoon`, `frank-archon`.
|
||||||
|
|
||||||
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp answers **501 Not Implemented** on `/metrics` unless started with `--metrics`; the endpoint had no such flag, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time. The `METRICS=1` knob in `scripts/frank-zealot.env` is the fix, and the target is up as measured 2026-10-18. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
**`frank-zealot` reads up since 2026-10-18, and did not before.** llama.cpp only answers `/metrics` when started with `--metrics`, so `up{job="frank-zealot"}` was 0 across the whole retained history (the oldest sample, 2026-10-11, is already 0) while the service answered `/health` 200 the whole time; the flag and its opt-out are on [[frank-zealot]]. Read `lastError` before believing an outage: a target can read down beside a healthy service.
|
||||||
|
|
||||||
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
|
Alert state is a live property, not a page fact: `curl -s http://127.0.0.1:9090/api/v1/alerts` returned no alerts at the 2026-10-18 morning census, one firing `LlmPrefixCacheCollapse` (info, `frank-zealot`, `activeAt` 11:10:48Z) minutes later, and none again by 11:56Z.
|
||||||
|
|
||||||
A scrape cannot express "is this HTTP service answering?" — that is what a
|
The remedy for a probe-shaped question is the **blackbox exporter**; the
|
||||||
probe is for, and the remedy is the **blackbox exporter** in probe mode
|
classification rule and the honesty law for it live in [[Stale Signal Audit]].
|
||||||
(`/probe?target=…&module=http_2xx`), which also gives the guarded vhosts a real
|
Until the exporter is deployed, these three stay red for a structural reason:
|
||||||
uptime signal since Prometheus cannot solve an Anubis challenge itself. Until it
|
deleting them would
|
||||||
is deployed, these three stay red for a structural reason: deleting them would
|
|
||||||
hide a real service, and fixing them by publishing databases or enabling metrics
|
hide a real service, and fixing them by publishing databases or enabling metrics
|
||||||
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
would trade a wrong signal for a real exposure. Targets are `host:port`, never
|
||||||
URLs.
|
URLs.
|
||||||
|
|||||||
@@ -94,8 +94,8 @@ earlier can still hold listener/TIME_WAIT state, so the replacement dies
|
|||||||
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
`pasta: ... bind: Address already in use` (conmon exit 126) — then **self-heals
|
||||||
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
seconds later** under `Restart=on-failure`. The failure looks like it failed,
|
||||||
then looks fine, and the missing gate stays invisible because the retry masks
|
then looks fine, and the missing gate stays invisible because the retry masks
|
||||||
it. Observed 2026-09-16 cutting `frank-dragoon` from a root unit to a rootless
|
it. First seen on a `frank-dragoon` unit cutover; which supervision shape that
|
||||||
Quadlet.
|
era was is in [[frank-dragoon]]'s era table.
|
||||||
|
|
||||||
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
Gate procedure: stop the old unit AND the new one (a redeploy's running
|
||||||
replacement is itself the port-holder — without that stop the gate deadlocks
|
replacement is itself the port-holder — without that stop the gate deadlocks
|
||||||
|
|||||||
@@ -11,15 +11,16 @@ timestamp: 2026-10-05T00:00:00-07:00
|
|||||||
## Endpoint down, connection refused — GPU holder reclaim
|
## Endpoint down, connection refused — GPU holder reclaim
|
||||||
|
|
||||||
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
Use when `:7770`/`:7771` gives **connection refused** (not a timeout) and the
|
||||||
service was not crashed. Applies to `frank-medium` (dankserver, root unit),
|
service was not crashed. Applies to the root-unit fleet endpoints:
|
||||||
`frank-zealot` (dankclaw, root unit), `frank-dragoon` (dankclaw, rootless
|
`frank-archon` (dankserver), `frank-dragoon` and `frank-zealot` (dankclaw) —
|
||||||
Quadlet).
|
unit identity per endpoint is on its own page ([[frank-archon]],
|
||||||
|
[[frank-dragoon]], [[frank-zealot]]).
|
||||||
|
|
||||||
**1. Classify the stop before touching anything**
|
**1. Classify the stop before touching anything**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ssh dankserver "systemctl is-enabled frank-medium; systemctl is-active frank-medium; \
|
ssh dankserver "systemctl is-enabled frank-archon; systemctl is-active frank-archon; \
|
||||||
systemctl show frank-medium -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
systemctl show frank-archon -p LoadState,UnitFileState,ActiveState,SubState,NRestarts"
|
||||||
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
ls -l /etc/systemd/system/<svc>.service # symlink to repo = unmasked; -> /dev/null = masked
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -36,9 +37,11 @@ ps -eo pid,comm,etime,args | grep -Ei 'llama|bench' | grep -v grep
|
|||||||
|
|
||||||
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
Cards idle → `systemctl start <svc>` and verify (step 5). Cards busy with no
|
||||||
prod unit → get the holder, and compare its bytes against the service's
|
prod unit → get the holder, and compare its bytes against the service's
|
||||||
requirement before concluding a start would even fit (frank-medium 35:30
|
requirement before concluding a start would even fit. The per-boot requirement
|
||||||
needs ~30.8 GiB ROCm0 / ~28.9 GiB ROCm1; a stray holding 15+17 GiB blocks the
|
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
|
||||||
restart on allocation, not config).
|
[[frank-zealot]]; the retired `frank-medium` split is measured in
|
||||||
|
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
|
||||||
|
allocation, not config.
|
||||||
|
|
||||||
**3. Identify the holder and prove it is abandoned**
|
**3. Identify the holder and prove it is abandoned**
|
||||||
|
|
||||||
@@ -95,9 +98,10 @@ the local Gitea on Dankclaw — push allowed).
|
|||||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||||
timeout. First request after a halogen (`frank-dragoon`) restart can go silent
|
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
|
||||||
45–60 s and self-recover; do not restart over it. Rootless `frank-dragoon`:
|
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
|
||||||
read logs **without** sudo (per-user storage). If the restore never happened,
|
log lives (and whether sudo is needed to read it) is on its page. If the
|
||||||
|
restore never happened,
|
||||||
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
the sudo journal names who and when: `journalctl -b | grep sudo`.
|
||||||
|
|
||||||
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
## Scratch vLLM boot on a fleet GPU host — stop-first law
|
||||||
@@ -110,13 +114,14 @@ failure modes if skipped:
|
|||||||
1. **Expert-offload planner dies on host RAM.**
|
1. **Expert-offload planner dies on host RAM.**
|
||||||
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
`vllm/model_executor/offloader/expert_plan.py` reads `MemAvailable` at
|
||||||
plan time. A production rank worker holds tens of GiB of unevictable
|
plan time. A production rank worker holds tens of GiB of unevictable
|
||||||
pinned expert rows (frank-archon: ~76 GiB × 2 ranks), so a beside-prod
|
pinned expert rows — frank-archon's pinned-RAM ledger, and how much host
|
||||||
boot sees ~86 GiB instead of ~239 and dies at plan time:
|
RAM the loaded endpoint leaves, is on [[frank-archon]] — so a beside-prod
|
||||||
|
boot sees far less than the bare host and dies at plan time:
|
||||||
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
`ExpertPlanError: host holds 5113 of 12096 swappable experts per rank`.
|
||||||
Looks like a planner bug; is pure RAM contention. Signature: container
|
Looks like a planner bug; is pure RAM contention. Signature: container
|
||||||
exits within ~60 s, after `Resolved architecture: …` but before worker
|
exits within ~60 s, after `Resolved architecture: …` but before worker
|
||||||
init; `free -g` showing `available ≈ 86` while prod runs is the smoking
|
init; `free -g` showing an `available` far below the endpoint page's
|
||||||
gun.
|
loaded-host figure while prod runs is the smoking gun.
|
||||||
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
2. **GPU ownership.** Two vLLM containers cannot share the GPUs regardless
|
||||||
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
(the fleet's benchmarking law). Scratch ports (`:7771`/`:7773`) are fine
|
||||||
only once the prod unit is down.
|
only once the prod unit is down.
|
||||||
@@ -166,7 +171,8 @@ blocks the very deploy that would install it. The loop:
|
|||||||
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
Traps, in the order they bite: the **pull aborts on hand copies** ("untracked
|
||||||
working tree files would be overwritten by merge" — exactly the file you
|
working tree files would be overwritten by merge" — exactly the file you
|
||||||
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
shipped); `git checkout --` silently eats uncommitted work without step 4; a
|
||||||
runtime may **mount files straight out of the checkout** (`dankserver/frank-medium/**`,
|
runtime may **mount files straight out of the checkout** (`scripts/**` — the
|
||||||
|
serve scripts and the chat template mount from there, see [[frank-archon]] — and
|
||||||
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
`shared/grafana/provisioning/**`) so a hand copy takes effect at next
|
||||||
container restart with no deploy; root-owned files from a container write
|
container restart with no deploy; root-owned files from a container write
|
||||||
need `chown` back before git operations; and **a restart is not a
|
need `chown` back before git operations; and **a restart is not a
|
||||||
|
|||||||
@@ -48,8 +48,9 @@ table. Three buckets:
|
|||||||
re-download.
|
re-download.
|
||||||
- **retired** — maps to a retired runtime. Safe candidates.
|
- **retired** — maps to a retired runtime. Safe candidates.
|
||||||
|
|
||||||
Trap: a directory can hold **both** a dead target and a live drafter
|
Trap: a directory can hold **both** a dead target and a live asset — an era's
|
||||||
(frank-dragoon's DFlash2 drafter lives beside the previous target). Move the
|
drafter can sit beside the retired target it came with (per-endpoint era tables:
|
||||||
|
[[frank-dragoon]], [[frank-zealot]], [[frank-archon]]). Move the
|
||||||
single file, never the directory, and say so in the manifest.
|
single file, never the directory, and say so in the manifest.
|
||||||
|
|
||||||
## 3. The live-reference check — before moving anything
|
## 3. The live-reference check — before moving anything
|
||||||
@@ -149,13 +150,13 @@ name**, never `endswith('.gguf')` — mradermacher ships large quants as
|
|||||||
|
|
||||||
## Refresh per engine — know the recovery path BEFORE deleting
|
## Refresh per engine — know the recovery path BEFORE deleting
|
||||||
|
|
||||||
- **frank-dragoon (halogen)**: the engine fetches its own payload via
|
- **frank-dragoon**: self-fetching — clear the directory but
|
||||||
`HALOGEN_DOWNLOAD` when the checkpoint is absent. Clear the directory but
|
leave it **existing and empty**, then start the unit; the download knobs and
|
||||||
leave it **existing and empty**, then start the unit. **Bind-mount trap:**
|
the payload's byte count (which is what the re-download costs) are on
|
||||||
|
[[frank-dragoon]]. **Bind-mount trap:**
|
||||||
podman refuses a mount whose source is missing
|
podman refuses a mount whose source is missing
|
||||||
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
(`Error: statfs /home/dank/models/halogen: no such file or directory`) and
|
||||||
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir. ~124 GiB
|
the unit loops on `Restart=on-failure` — `mkdir -p` the empty dir.
|
||||||
at 130–240 MB/s ≈ 10–20 min, then load.
|
|
||||||
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
- **llama.cpp services**: no self-fetch; the serve script refuses to start
|
||||||
without weights. Pull yourself and verify:
|
without weights. Pull yourself and verify:
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user