docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
This commit is contained in:
Frank
@@ -39,7 +39,7 @@ failure as failure — use it for bulk sweeps; use `/dev/tcp` (or `nc -v`) when
|
||||
refusal-vs-timeout matters.
|
||||
|
||||
Fix: point the service's host-facing URL at
|
||||
`http://host.containers.internal:<port>` in compose/quadlet env, redeploy via
|
||||
`http://host.containers.internal:<port>` in compose/quadlet env, redeploy
|
||||
git (`deploy.sh <target> --service <svc>`), verify
|
||||
`podman exec <svc> printenv <URL_VAR>` and the connect log line (~40 s).
|
||||
**Do not publish the daemon port** to make the LAN IP work — publishing is
|
||||
|
||||
@@ -155,15 +155,15 @@ http://127.0.0.1:3002/api/org` → 200 means `admin:admin` still live
|
||||
(loopback-only; the vhost path needs Anubis + OAuth2 first). LLM binds:
|
||||
`sudo ss -tln | grep -E ':(7770|7771)'` — `0.0.0.0` = LAN-reachable with no
|
||||
auth (OpenAI API, no token). Stale-record check: current WAN IP =
|
||||
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress);
|
||||
`ssh frankbook-pro 'curl -s https://api.ipify.org'` (residential egress)
|
||||
compare against every public A record — a record at a different Comcast IP
|
||||
is stale.
|
||||
|
||||
**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`,
|
||||
**Baseline (2026-09-21):** WAN: only 80 + 443 open. Public names: `www`
|
||||
`search` (guarded) + `ai`, `gitea`, `plex`, `books`, `jelly` (own auth, all
|
||||
verified) + `auth` (IdP, by design); everything else LAN-only. Open findings:
|
||||
LLM endpoints 0.0.0.0 auth-less on the LAN; Grafana `admin:admin` loopback
|
||||
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP);
|
||||
residual; `plex.franksdank.casa` stale (73.59.75.210 = previous Comcast IP)
|
||||
`plex.franksdank.cloud` WAN-broken (Plex Web 302s to LAN origin
|
||||
192.168.0.235:32400); vestigial *arr handlers on the :8017 post-auth listener
|
||||
(dead config, no exposure).
|
||||
@@ -242,7 +242,7 @@ git stash list; git worktree list` (prune stale worktrees), and
|
||||
`du -sh .AGENTTRASH` for staged deletions awaiting purge approval. This is a
|
||||
**shared checkout with parallel instances**: uncommitted peer WIP and
|
||||
stash-list churn are expected. Never commit or drop a peer's files. To
|
||||
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`,
|
||||
deploy with a dirty tree: `git stash push -u -m "..." -- <peer paths>`
|
||||
deploy, `git stash pop`. `deploy.sh` refuses a dirty tree.
|
||||
|
||||
**5. Prometheus — expected-down targets (not outages).** Query
|
||||
@@ -253,7 +253,7 @@ down, especially a `node-*` or LLM job, is a real gap. The recurring trap: a
|
||||
same-host `net=host` bind (node-exporter :9100, llama-server :7770/:7771) is
|
||||
a *host bind*, not a published port, so the bridge-networked Prometheus
|
||||
container cannot reach it at the host's LAN IP — netavark DNATs published
|
||||
ports only. Scrape same-host services via
|
||||
ports only. Scrape same-host services
|
||||
`host.containers.internal:<port>`; verify the fix from inside the container
|
||||
before editing: `podman exec prometheus wget -qO-
|
||||
http://host.containers.internal:<port>/metrics`.
|
||||
@@ -270,7 +270,7 @@ Removing a retired host / machine / project name from a repo that holds both
|
||||
documentation and live config. Evidence-first, surgical, reversible.
|
||||
|
||||
**1. Inventory before touching anything.** Count hits **per file** (the
|
||||
count decides whether this is a 5-minute job or a user-input decision),
|
||||
count decides whether this is a 5-minute job or a user-input decision)
|
||||
excluding logs/`*.db`/runtime state, and capture `git status --short` first
|
||||
— you must know which files carry in-flight peer WIP so you never stage
|
||||
them.
|
||||
|
||||
@@ -40,7 +40,7 @@ prod unit → get the holder, and compare its bytes against the service's
|
||||
requirement before concluding a start would even fit. The per-boot requirement
|
||||
is the endpoint's own arithmetic ([[frank-archon]], [[frank-dragoon]],
|
||||
[[frank-zealot]]; the retired `frank-medium` split is measured in
|
||||
[[frank-medium-llamacpp-dflash2]]) — a stray holder blocks the restart on
|
||||
) — a stray holder blocks the restart on
|
||||
allocation, not config.
|
||||
|
||||
**3. Identify the holder and prove it is abandoned**
|
||||
@@ -98,7 +98,7 @@ the local Gitea on Dankclaw — push allowed).
|
||||
unquoted word with `=` in an ssh one-liner (`echo ===` aborts the chain). A
|
||||
freshly restarted endpoint under live agent-queue backlog may return empty
|
||||
bodies/timeouts for minutes while `/health` is 200 — re-probe with a long
|
||||
timeout. The first request after a `frank-dragoon` restart is a warmup artifact — see
|
||||
timeout. The first request after a `frank-dragoon` restart is a warmup artifact —
|
||||
[[frank-dragoon]] → Contract; do not restart over it. Where an endpoint's own
|
||||
log lives (and whether sudo is needed to read it) is on its page. If the
|
||||
restore never happened,
|
||||
@@ -146,7 +146,6 @@ there; `mask --runtime` is for scratch boots that must survive a stray
|
||||
restart. Always end a window by re-checking the conjunction before
|
||||
reporting — if a restore after failure doesn't happen automatically, prod
|
||||
stays DOWN. See the repo's benchmarking policy (AGENTS.md) and
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## Iterating on a fleet service before commit
|
||||
|
||||
@@ -249,7 +248,7 @@ import socket, time
|
||||
MAC = "de:6b:6d:c1:52:23"
|
||||
mac = bytes(int(x,16) for x in MAC.split(":"))
|
||||
pkt = b"\xff"*6 + mac*16 # 102 bytes
|
||||
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255"),
|
||||
for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0.255")
|
||||
("192.168.0.83","255.255.255.255"),("192.168.0.104","255.255.255.255")]:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
|
||||
@@ -271,7 +270,7 @@ for local, dest in [("192.168.0.83","192.168.0.255"),("192.168.0.104","192.168.0
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the bench windows this playbook arms
|
||||
- the bench windows this playbook arms
|
||||
- [[Remote Job Runner]] — supervising the long jobs that create these windows
|
||||
- [[Container Host Reachability]] — the pasta port gate at container create
|
||||
- [[Quadlet Lifecycle]] — the unit layer behind stop/mask/start
|
||||
|
||||
@@ -177,12 +177,12 @@ world-readable in `/proc/<pid>/cmdline`).
|
||||
|
||||
## Restore, then verify behaviour — not health alone
|
||||
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`),
|
||||
Put the quadlet back (`mv` from trash + `systemctl --user daemon-reload`)
|
||||
`reset-failed`, `start` (root units need `sudo`). Watch the engine's own
|
||||
fetch/load lines, then `/health`, `/v1/models`, and a **real completion** on
|
||||
every restored endpoint. Read the engine's memory ledger (kv pool / weights /
|
||||
blocks left) rather than assuming the config still fits. Sweep the vhosts too
|
||||
(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki` via
|
||||
(`www`, `grafana`, `prometheus`, `hermes`, `marreta`, `gitea`, `loki`
|
||||
`curl -sk --resolve <host>:443:192.168.0.143`) — a purge touches state the
|
||||
whole host shares.
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ start)
|
||||
touch "'"$JOBS"'/$NAME.done"
|
||||
' _ "$NAME" "$@" < /dev/null > /dev/null 2>&1 &
|
||||
echo "started $NAME"
|
||||
;;
|
||||
;
|
||||
status)
|
||||
# Glob *.out (not *.done) or running jobs are invisible.
|
||||
names="${2:-$(cd "$JOBS" && ls *.out 2>/dev/null | sed 's/\.out$//' | sort)}"
|
||||
@@ -64,7 +64,7 @@ status)
|
||||
printf ' %-18s PENDING\n' "$n"
|
||||
fi
|
||||
done
|
||||
;;
|
||||
;
|
||||
wait)
|
||||
NAME="$2"; LIMIT="${3:-3600}"; waited=0
|
||||
while [ ! -f "$JOBS/$NAME.done" ] && [ "$waited" -lt "$LIMIT" ]; do
|
||||
@@ -74,11 +74,11 @@ wait)
|
||||
&& echo "DONE $NAME rc=$(cat "$JOBS/$NAME.rc" 2>/dev/null)" \
|
||||
|| echo "TIMEOUT $NAME after ${waited}s"
|
||||
tail -"${4:-25}" "$JOBS/$NAME.out" 2>/dev/null
|
||||
;;
|
||||
out) NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;;
|
||||
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;;
|
||||
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;;
|
||||
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;;
|
||||
;
|
||||
out) NAME="$2"; tail -"${3:-40}" "$JOBS/$NAME.out" 2>/dev/null ;
|
||||
tail) [ -n "${2:-}" ] && exec tail -f "$JOBS/$2.out" || exec tail -f "$JOBS"/*.out ;
|
||||
watch) while true; do clear; date '+%H:%M:%S'; bash "$0" status; sleep 5; done ;
|
||||
*) echo "usage: job.sh {start|status|wait|out|tail|watch}" >&2; exit 2 ;
|
||||
esac
|
||||
```
|
||||
|
||||
@@ -105,7 +105,7 @@ job.sh watch # refreshing status table
|
||||
command line — it kills the session. Distinctive names or `pkill -x`.
|
||||
- **Write helper scripts to a file and `scp` them.** Triple-nested
|
||||
`ssh 'bash -c "python3 -c \"…\""'` quoting broke repeatedly (unterminated
|
||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line);
|
||||
quotes; `grep -c` returning `"0\n0"` because `|| echo 0` appended a line)
|
||||
every failure disappeared once the script was a file.
|
||||
|
||||
## Verify before trusting it (3am law)
|
||||
@@ -131,5 +131,5 @@ only proves the happy path is not trustworthy.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the sweeps this runner executes
|
||||
- the sweeps this runner executes
|
||||
- [[Deploy State Triage]] — the two-process deploy this lesson generalizes
|
||||
|
||||
@@ -40,7 +40,7 @@ flows through the **process env only** — never a file, never argv.
|
||||
proves the key works.
|
||||
|
||||
Gotchas: without `SSH_ASKPASS_REQUIRE=force`, ssh ignores SSH_ASKPASS when a
|
||||
TTY is present and prompts interactively (hangs in a non-interactive tool);
|
||||
TTY is present and prompts interactively (hangs in a non-interactive tool)
|
||||
`setsid` detaches the controlling terminal — required on older OpenSSH even
|
||||
with `force`; the helper must be executable (`chmod 700`) and print ONLY the
|
||||
password; `IdentitiesOnly yes` stops ssh offering other agent keys that could
|
||||
@@ -108,7 +108,7 @@ Use when the desktop is frozen, VT switches do nothing, but SSH still works.
|
||||
4. **Trigger window**: block start = first hung message timestamp − reported
|
||||
seconds; then inspect user-session events around it
|
||||
(`journalctl -b --since "<start-2m>" --until "<start+1m>" | grep -v audit`).
|
||||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`),
|
||||
5. **Collateral**: other `D` tasks (`ps -eo pid,stat,comm | awk '$2 ~ /D/'`)
|
||||
USB workers, etc.
|
||||
|
||||
**Recoverability verdict:** compositor in `D` state is unkillable and a
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Operations
|
||||
|
||||
Procedures for operating the fleet. The LLM-specific pages live under
|
||||
[LLM](../LLM/index.md); infrastructure and architecture under the
|
||||
; infrastructure and architecture under the
|
||||
`../Infrastructure/` and `../Architecture/` folders.
|
||||
|
||||
- [[Remote Job Runner]] — Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification.
|
||||
|
||||
Reference in New Issue
Block a user