docs(vault): destroy the dead weight — 53 pages and the change diary

The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
This commit is contained in:
Frank
2026-10-19 10:56:37 -07:00
parent 8775e64ed0
commit 327e78b950
126 changed files with 443 additions and 14157 deletions
@@ -47,7 +47,7 @@ network namespace**:
`Network=` values are read from the live Quadlet sources in
`/home/dank/.config/containers/systemd/*.container`; runtime network names on the
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`,
host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`
`systemd-searxng`, `systemd-open-webui`.
Two consequences that are easy to miss:
@@ -83,7 +83,7 @@ podman exec <name> cat /proc/net/route # default route → gateway IP
podman exec <name> sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?'
```
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`,
Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`
`10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns,
published) and `:9119` (its own published port) connect; `:8080` and `:8787`
are **refused instantly** even though both listeners are bound to `0.0.0.0` in
@@ -36,7 +36,7 @@ possibly zero behavioral change.
2. Probe the target (ssh as `dank`, passwordless sudo present):
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
- `git -C ~/serverAppConfigs rev-parse HEAD`
- `systemctl is-active/is-enabled <svc>`;
- `systemctl is-active/is-enabled <svc>`
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
- the on-host env file already carries the new keys (it landed via the
ff-pull).
@@ -119,8 +119,8 @@ each, kill stale ones by explicit PID.
A killed deploy leaves behind:
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
explicit stop — start it yourself);
- the state marker **pending** (pre-deploy or in-flight revision);
explicit stop — start it yourself)
- the state marker **pending** (pre-deploy or in-flight revision)
- the e2e gate un-run (the deployer runs it last).
Reconcile the marker ONLY after the service itself is verified healthy — a
@@ -142,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts
numbers — which looks exactly like your change having no effect. Key on
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
See for the full capture
procedure.
## Related
- [[Deployment Flow]] — the deploy pipeline itself
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
- journal capture
@@ -68,7 +68,7 @@ service on the other host.
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
The shared chat template maps to the inference service on the selected target.
The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler;
The `frank-npu` endpoint is retired () and has no deploy handler
an unrelated large-model deployment never touches it.
## Caddy safety
@@ -125,6 +125,5 @@ routing. See [[Testing Policy]].
## See Also
- [[Testing Policy]]
- [LLM deployment](../LLM/deployment.md)
- [[Caddy]]
- [[SearXNG]]
@@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior.
*arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a
sleeping Mac mini must never fail an unrelated deploy. The project joins only
when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`,
trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`
`shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr
vhosts) or when an explicit `--project` argument names it. An `11-admin` edit
that touches the arr links must therefore be paired with a `19-frankbook` run
@@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143)
As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS,
Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone
else), and accepted traffic returns to Caddy's internal listener on `:8016`,
else), and accepted traffic returns to Caddy's internal listener on `:8016`
which proxies every request whole to the Authentik server with the Host
header preserved; Authentik's embedded outpost routes on that Host against
each proxy provider's `external_host`, serves the login flow in place for
@@ -67,12 +67,12 @@ graph TB
```
Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`):
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`),
homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`)
prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta
(`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.`
(the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP
clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP),
unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP)
books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts
that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own
auth"). Grafana is **not** exempt even though it signs in through authentik as
@@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through
`Network=searxng.network` (runtime network `systemd-searxng`), while
`searxng.container`, `marreta.container` and `marreta-browser.container` join
the tunnel the modern way — `Network=gluetun.container`
(`dankclaw/searxng/quadlet/searxng.container:18`,
`dankclaw/marreta/quadlet/marreta.container:34`,
(`dankclaw/searxng/quadlet/searxng.container:18`
`dankclaw/marreta/quadlet/marreta.container:34`
`dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet
equivalent of the retired compose `network_mode: service:gluetun`. **Five**
containers share the VPN exit, not three, and Marreta's outbound fetches leave
through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`);
exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`)
everything else on Dankclaw uses the normal LAN route. This matches KB Pattern
2 in [[Gluetun]].
See [[Container Network Namespaces]] for which addresses are reachable from that
@@ -36,14 +36,14 @@ done
## 2. Cutover: compose service → Quadlet unit
Checklist (precedents to copy:
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`,
`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`
`dankclaw/observability/quadlet/`):
1. Unit file at `<host>/<svc>/quadlet/<name>.container`, mirroring compose
line for line: `Image=` (`localhost/` prefix for locally built images +
`Pull=never`), `ContainerName=` (Caddy and scrape configs address
containers by name), `Network=`, `User=0:0` when the image's own user
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved),
cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved)
`Exec=` for compose `command:`, `PublishPort=`, health keys,
`[Service] Restart=always` + `RestartSec=10`, `[Install]
WantedBy=default.target`. **Never `systemctl --user enable` the generated
@@ -56,7 +56,7 @@ Checklist (precedents to copy:
`systemctl --user restart <name>.service` → `wait_for_health`; retire any
legacy hand-written unit into `~/.AGENTTRASH/` (a leftover
`~/.config/systemd/user/<svc>.service` that shells out to
`podman compose up` collides BY NAME with the generated unit);
`podman compose up` collides BY NAME with the generated unit)
`podman rm -f <container>` so the unit's container can take the name.
4. Planner: map `<host>/<svc>/**` → service in `scripts/deploy-plan.py`
AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies
@@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever.
Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
= the §2 cutover. **Verify the capability, not the status:** `podman exec
<name> python3 -c "import torch; print(torch.cuda.is_available())"`;
<name> python3 -c "import torch; print(torch.cuda.is_available())"`
`ss -ltn | grep <port>` for the actual bind addresses. ComfyUI incident:
2026-09-16, stale device-less container, 9,146 restarts. History repeats:
2026-08-26 Caddy (no unit), 2026-09-11 five observability services.
@@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate <svc>`. Durable fix
unhealthy but logs look fine, compare the baked
`{{.Config.Healthcheck.Test}}` array against the intended command before
believing the status.
- **Probe a new key before shipping:** drop a throwaway `probe.container`,
- **Probe a new key before shipping:** drop a throwaway `probe.container`
`daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i
"unsupported key"`, `systemctl --user cat probe.service` (shows the
generated ExecStart verbatim), then remove + reload.
@@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con
## See Also
- [[Deployment Flow]]
- [[index]]