Frank
327e78b950
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
153 lines
7.2 KiB
Markdown
153 lines
7.2 KiB
Markdown
---
|
|
type: reference
|
|
title: Deploy State Triage
|
|
description: "Reading deploy-state markers against live containers and deciding, safely, whether to redeploy."
|
|
tags: [deployment, deploy-state, systemd, podman, triage]
|
|
timestamp: 2026-10-05T00:00:00-07:00
|
|
---
|
|
|
|
# Deploy State Triage
|
|
|
|
The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
|
on each target (rootless, user `dank`, format exactly two lines
|
|
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
|
|
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
|
|
|
|
## 1. Stale marker vs live container
|
|
|
|
**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
|
|
a restart after the checkout ff-pulled to a newer revision — starts the
|
|
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
|
|
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
|
"behavior pending".
|
|
|
|
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
|
whose marker lags. On the big-payload endpoints that is a full model reload
|
|
measured in tens of minutes — each endpoint's boot budget is on its own page
|
|
([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon
|
|
(dankserver:7770) is also the
|
|
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
|
possibly zero behavioral change.
|
|
|
|
**Procedure (read-only first):**
|
|
|
|
1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
|
|
operational delta is usually one `scripts/<svc>.env`.
|
|
2. Probe the target (ssh as `dank`, passwordless sudo present):
|
|
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
|
- `git -C ~/serverAppConfigs rev-parse HEAD`
|
|
- `systemctl is-active/is-enabled <svc>`
|
|
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
|
|
- the on-host env file already carries the new keys (it landed via the
|
|
ff-pull).
|
|
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
|
|
`dank` (rootless store; bare `docker` is NOT the shim → "No such object";
|
|
`sudo podman` inspects an empty store):
|
|
- `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
|
|
the pull ⇒ it read the new env
|
|
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
|
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
|
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
|
- confirm the launcher maps each tunable to its flag
|
|
(`grep scripts/launch.sh`).
|
|
|
|
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
|
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
|
container lacks the flag/env (or predates the pull) is a real
|
|
`deploy.sh <target> --service <svc>` warranted.
|
|
|
|
**Reconcile (no restart)** — mirror the deployer's atomic idiom:
|
|
|
|
```bash
|
|
S=$HOME/.local/state/serverAppConfigs/deploy-state
|
|
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
|
|
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
|
|
```
|
|
|
|
Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
|
|
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
|
|
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
|
|
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
|
|
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
|
|
no-op).
|
|
|
|
**Anti-patterns:** trusting a commit message ("X deployed") or the marker
|
|
alone — both are bookkeeping; the container's argv/env is ground truth.
|
|
Reading container state with `sudo podman` / bare `docker`. Restarting a
|
|
healthy heavy endpoint to satisfy a stale marker.
|
|
|
|
## 2. Pending lockout
|
|
|
|
Symptom: `deploy.sh <target>` fails or silently stops with
|
|
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
|
|
appears.
|
|
|
|
1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
|
|
reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
|
|
`echo $?`.
|
|
2. Inspect the state store (no sudo):
|
|
`ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
|
|
A `status=pending` entry for a service absent from `service_is_allowed`
|
|
in `scripts/deploy-remote.sh` = retired-service lockout.
|
|
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
|
|
not-allowed pending entries to
|
|
`~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
|
|
continues. Verify the WARN line names the service and the state file is
|
|
gone.
|
|
4. On a host without the fix: move the stale entry to
|
|
`~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
|
|
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
|
|
`deploy-state/` no longer holds the retired name.
|
|
|
|
Related contracts: the superseded seam
|
|
(`deployment_scoping.bats` "remote runner recognizes a pending service at
|
|
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
|
|
`deployment_flow.bats` extracts `add_pending_services` via sed and
|
|
HOME-overrides the trash path. **When retiring a service going forward:**
|
|
delete or reconcile its deploy-state entry as part of the retirement commit —
|
|
the prune is a safety net, not the intended cleanup path.
|
|
|
|
## 3. Interrupted or raced deploys
|
|
|
|
The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
|
|
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
|
|
host-side one** — two `deploy-remote.sh` processes can end up managing the
|
|
same unit, one deploying a revision you already superseded. Check:
|
|
`pgrep -af "deploy-remote"` on the target, read the revision argument on
|
|
each, kill stale ones by explicit PID.
|
|
|
|
A killed deploy leaves behind:
|
|
|
|
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
|
|
explicit stop — start it yourself)
|
|
- the state marker **pending** (pre-deploy or in-flight revision)
|
|
- the e2e gate un-run (the deployer runs it last).
|
|
|
|
Reconcile the marker ONLY after the service itself is verified healthy — a
|
|
marker that runs ahead of reality is worse than one that lags (the planner
|
|
uses it to decide what is pending). Then close the loop:
|
|
|
|
1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
|
|
slots, argv flags, `NRestarts`), not just health 200.
|
|
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
|
|
that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
|
|
does NOT have it; copy it in first).
|
|
3. If the killed revision was restoring a known-good shape, confirm the
|
|
numbers match the last measured-good boot of that shape — equality is the
|
|
proof, not health alone.
|
|
|
|
Boot failures must be read from the journal, not `podman logs`: serve scripts
|
|
`podman rm -f` the container on every start (previous attempt lost), and
|
|
`podman logs` on a container still *Stopping* returns the OLD configuration's
|
|
numbers — which looks exactly like your change having no effect. Key on
|
|
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
|
|
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
|
|
See for the full capture
|
|
procedure.
|
|
|
|
## Related
|
|
|
|
- [[Deployment Flow]] — the deploy pipeline itself
|
|
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
|
|
- journal capture
|