--- type: reference title: Deploy State Triage description: "Reading deploy-state markers against live containers and deciding, safely, whether to redeploy." tags: [deployment, deploy-state, systemd, podman, triage] timestamp: 2026-10-05T00:00:00-07:00 --- # Deploy State Triage The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/` on each target (rootless, user `dank`, format exactly two lines `revision=` / `status=deployed|pending`, written atomically via tmp file + `mv`). It is written ONLY by a `deploy.sh` dispatch. ## 1. Stale marker vs live container **Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or a restart after the checkout ff-pulled to a newer revision — starts the container from the CURRENT `scripts/.env` but does NOT advance the marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT "behavior pending". **Why it matters:** a plain `deploy.sh` run re-selects and restarts a service whose marker lags. On the big-payload endpoints that is a full model reload measured in tens of minutes — each endpoint's boot budget is on its own page ([[frank-archon]], [[frank-dragoon]], [[frank-zealot]]) — and frank-archon (dankserver:7770) is also the harness's own model backend, so a blind deploy is a self-inflicted outage for possibly zero behavioral change. **Procedure (read-only first):** 1. Committed state: `git log --oneline main..` empty ⇒ merged; the operational delta is usually one `scripts/.env`. 2. Probe the target (ssh as `dank`, passwordless sudo present): - `cat ~/.local/state/serverAppConfigs/deploy-state/` - `git -C ~/serverAppConfigs rev-parse HEAD` - `systemctl is-active/is-enabled ` `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:/health` - the on-host env file already carries the new keys (it landed via the ff-pull). 3. **Decisive — read the LIVE container, not the marker.** Use `podman` as `dank` (rootless store; bare `docker` is NOT the shim → "No such object"; `sudo podman` inspects an empty store): - `podman inspect --format '{{.State.StartedAt}}'` — booted AFTER the pull ⇒ it read the new env - `podman inspect --format '{{.Args}}' | tr ' ' '\n' | grep -i ` - engine env: `podman exec sh -c 'for f in /proc/[0-9]*/environ; do tr "\0" "\n" <$f 2>/dev/null; done | grep -i '` - confirm the launcher maps each tunable to its flag (`grep scripts/launch.sh`). **Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is live — **do NOT restart**; reconcile the marker (below). Only if the live container lacks the flag/env (or predates the pull) is a real `deploy.sh --service ` warranted. **Reconcile (no restart)** — mirror the deployer's atomic idiom: ```bash S=$HOME/.local/state/serverAppConfigs/deploy-state H=$(git -C ~/serverAppConfigs rev-parse HEAD) t=$(mktemp "$S/..XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/" ``` Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs `tests/e2e/.env.local`; expect the documented `11-admin` red if the FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` + a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression). Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy no-op). **Anti-patterns:** trusting a commit message ("X deployed") or the marker alone — both are bookkeeping; the container's argv/env is ground truth. Reading container state with `sudo podman` / bare `docker`. Restarting a healthy heavy endpoint to satisfy a stale marker. ## 2. Pending lockout Symptom: `deploy.sh ` fails or silently stops with `service is not deployed on ` and the `[3/3]` e2e gate never appears. 1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail` reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then `echo $?`. 2. Inspect the state store (no sudo): `ssh 'cat ~/.local/state/serverAppConfigs/deploy-state/'`. A `status=pending` entry for a service absent from `service_is_allowed` in `scripts/deploy-remote.sh` = retired-service lockout. 3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes not-allowed pending entries to `~/.AGENTTRASH/-retired-deploy-state/` and the run continues. Verify the WARN line names the service and the state file is gone. 4. On a host without the fix: move the stale entry to `~/.AGENTTRASH/-retired-deploy-state/` manually (never `rm`), re-run. 5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes; `deploy-state/` no longer holds the retired name. Related contracts: the superseded seam (`deployment_scoping.bats` "remote runner recognizes a pending service at the same revision") now asserts prune-and-continue (exit 0), not exit 2; `deployment_flow.bats` extracts `add_pending_services` via sed and HOME-overrides the trash path. **When retiring a service going forward:** delete or reconcile its deploy-state entry as part of the retirement commit — the prune is a safety net, not the intended cleanup path. ## 3. Interrupted or raced deploys The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh` (runs on the target over ssh). **Killing the dev-side job does NOT kill the host-side one** — two `deploy-remote.sh` processes can end up managing the same unit, one deploying a revision you already superseded. Check: `pgrep -af "deploy-remote"` on the target, read the revision argument on each, kill stale ones by explicit PID. A killed deploy leaves behind: - the unit **stopped/failed** (`Restart=always` does NOT fire after an explicit stop — start it yourself) - the state marker **pending** (pre-deploy or in-flight revision) - the e2e gate un-run (the deployer runs it last). Reconcile the marker ONLY after the service itself is verified healthy — a marker that runs ahead of reality is worse than one that lags (the planner uses it to decide what is pending). Then close the loop: 1. Verify against MEASURED, config-specific numbers (KV capacity line, arena slots, argv flags, `NRestarts`), not just health 200. 2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree` does NOT have it; copy it in first). 3. If the killed revision was restoring a known-good shape, confirm the numbers match the last measured-good boot of that shape — equality is the proof, not health alone. Boot failures must be read from the journal, not `podman logs`: serve scripts `podman rm -f` the container on every start (previous attempt lost), and `podman logs` on a container still *Stopping* returns the OLD configuration's numbers — which looks exactly like your change having no effect. Key on container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the cause from `journalctl --since "N min ago" | grep -a "\["`. See for the full capture procedure. ## Related - [[Deployment Flow]] — the deploy pipeline itself - [[Quadlet Lifecycle]] — unit ownership and drift diagnosis - journal capture