docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
This commit is contained in:
@@ -47,7 +47,7 @@ network namespace**:
|
||||
|
||||
`Network=` values are read from the live Quadlet sources in
|
||||
`/home/dank/.config/containers/systemd/*.container`; runtime network names on the
|
||||
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`,
|
||||
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`
|
||||
`systemd-searxng`, `systemd-open-webui`.
|
||||
|
||||
Two consequences that are easy to miss:
|
||||
@@ -83,7 +83,7 @@ podman exec <name> cat /proc/net/route # default route → gateway IP
|
||||
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?'
|
||||
```
|
||||
|
||||
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`,
|
||||
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`
|
||||
`10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns,
|
||||
published) and `:9119` (its own published port) connect; `:8080` and `:8787`
|
||||
are **refused instantly** even though both listeners are bound to `0.0.0.0` in
|
||||
|
||||
@@ -36,7 +36,7 @@ possibly zero behavioral change.
|
||||
2. Probe the target (ssh as `dank`, passwordless sudo present):
|
||||
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||||
- `git -C ~/serverAppConfigs rev-parse HEAD`
|
||||
- `systemctl is-active/is-enabled <svc>`;
|
||||
- `systemctl is-active/is-enabled <svc>`
|
||||
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
|
||||
- the on-host env file already carries the new keys (it landed via the
|
||||
ff-pull).
|
||||
@@ -119,8 +119,8 @@ each, kill stale ones by explicit PID.
|
||||
A killed deploy leaves behind:
|
||||
|
||||
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
|
||||
explicit stop — start it yourself);
|
||||
- the state marker **pending** (pre-deploy or in-flight revision);
|
||||
explicit stop — start it yourself)
|
||||
- the state marker **pending** (pre-deploy or in-flight revision)
|
||||
- the e2e gate un-run (the deployer runs it last).
|
||||
|
||||
Reconcile the marker ONLY after the service itself is verified healthy — a
|
||||
@@ -142,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts
|
||||
numbers — which looks exactly like your change having no effect. Key on
|
||||
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
|
||||
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
|
||||
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
|
||||
See for the full capture
|
||||
procedure.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Deployment Flow]] — the deploy pipeline itself
|
||||
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
|
||||
- journal capture
|
||||
|
||||
@@ -68,7 +68,7 @@ service on the other host.
|
||||
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
||||
|
||||
The shared chat template maps to the inference service on the selected target.
|
||||
The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
|
||||
The `frank-npu` endpoint is retired () and has no deploy handler
|
||||
an unrelated large-model deployment never touches it.
|
||||
|
||||
## Caddy safety
|
||||
@@ -125,6 +125,5 @@ routing. See [[Testing Policy]].
|
||||
## See Also
|
||||
|
||||
- [[Testing Policy]]
|
||||
- [LLM deployment](../LLM/deployment.md)
|
||||
- [[Caddy]]
|
||||
- [[SearXNG]]
|
||||
|
||||
@@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior.
|
||||
*arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
|
||||
sleeping Mac mini must never fail an unrelated deploy. The project joins only
|
||||
when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
|
||||
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
|
||||
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`
|
||||
`shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
|
||||
vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
|
||||
that touches the arr links must therefore be paired with a `19-frankbook` run
|
||||
|
||||
@@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
|
||||
|
||||
As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
|
||||
Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
|
||||
else), and accepted traffic returns to Caddy's internal listener on `:8016`,
|
||||
else), and accepted traffic returns to Caddy's internal listener on `:8016`
|
||||
which proxies every request whole to the Authentik server with the Host
|
||||
header preserved; Authentik's embedded outpost routes on that Host against
|
||||
each proxy provider's `external_host`, serves the login flow in place for
|
||||
@@ -67,12 +67,12 @@ graph TB
|
||||
```
|
||||
|
||||
Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`):
|
||||
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`),
|
||||
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`)
|
||||
prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta
|
||||
(`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.`
|
||||
(the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP
|
||||
clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass
|
||||
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP),
|
||||
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP)
|
||||
books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts
|
||||
that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own
|
||||
auth"). Grafana is **not** exempt even though it signs in through authentik as
|
||||
@@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through
|
||||
`Network=searxng.network` (runtime network `systemd-searxng`), while
|
||||
`searxng.container`, `marreta.container` and `marreta-browser.container` join
|
||||
the tunnel the modern way — `Network=gluetun.container`
|
||||
(`dankclaw/searxng/quadlet/searxng.container:18`,
|
||||
`dankclaw/marreta/quadlet/marreta.container:34`,
|
||||
(`dankclaw/searxng/quadlet/searxng.container:18`
|
||||
`dankclaw/marreta/quadlet/marreta.container:34`
|
||||
`dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet
|
||||
equivalent of the retired compose `network_mode: service:gluetun`. **Five**
|
||||
containers share the VPN exit, not three, and Marreta's outbound fetches leave
|
||||
through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of
|
||||
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`);
|
||||
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`)
|
||||
everything else on Dankclaw uses the normal LAN route. This matches KB Pattern
|
||||
2 in [[Gluetun]].
|
||||
See [[Container Network Namespaces]] for which addresses are reachable from that
|
||||
|
||||
@@ -36,14 +36,14 @@ done
|
||||
## 2. Cutover: compose service → Quadlet unit
|
||||
|
||||
Checklist (precedents to copy:
|
||||
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
|
||||
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
|
||||
`dankclaw/observability/quadlet/`):
|
||||
|
||||
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
|
||||
line for line: `Image=` (`localhost/` prefix for locally built images +
|
||||
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
|
||||
containers by name), `Network=`, `User=0:0` when the image's own user
|
||||
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
|
||||
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved)
|
||||
`Exec=` for compose `command:`, `PublishPort=`, health keys,
|
||||
`[Service] Restart=always` + `RestartSec=10`, `[Install]
|
||||
WantedBy=default.target`. **Never `systemctl --user enable` the generated
|
||||
@@ -56,7 +56,7 @@ Checklist (precedents to copy:
|
||||
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
|
||||
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
|
||||
`~/.config/systemd/user/<svc>.service` that shells out to
|
||||
`podman compose up` collides BY NAME with the generated unit);
|
||||
`podman compose up` collides BY NAME with the generated unit)
|
||||
`podman rm -f <container>` so the unit's container can take the name.
|
||||
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
|
||||
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
|
||||
@@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever.
|
||||
|
||||
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
= the §2 cutover. **Verify the capability, not the status:** `podman exec
|
||||
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
|
||||
<name> python3 -c "import torch; print(torch.cuda.is_available())"`
|
||||
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
|
||||
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
|
||||
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
|
||||
@@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
|
||||
unhealthy but logs look fine, compare the baked
|
||||
`{{.Config.Healthcheck.Test}}` array against the intended command before
|
||||
believing the status.
|
||||
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
|
||||
- **Probe a new key before shipping:** drop a throwaway `probe.container`
|
||||
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
|
||||
"unsupported key"`, `systemctl --user cat probe.service` (shows the
|
||||
generated ExecStart verbatim), then remove + reload.
|
||||
|
||||
@@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con
|
||||
## See Also
|
||||
|
||||
- [[Deployment Flow]]
|
||||
- [[index]]
|
||||
|
||||
Reference in New Issue
Block a user