serverAppConfigs/serverAppConfigs-docs/Architecture/Deploy State Triage.md

---
type: reference
title: Deploy State Triage
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
tags: [deployment, deploy-state, systemd, podman, triage]
timestamp: 2026-10-05T00:00:00-07:00
---

# Deploy State Triage

The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
on each target (rootless, user `dank`, format exactly two lines
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.

## 1. Stale marker vs live container

**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
a restart after the checkout ff-pulled to a newer revision — starts the
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
"behavior pending".

**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
harness's own model backend, so a blind deploy is a self-inflicted outage for
possibly zero behavioral change.

**Procedure (read-only first):**

1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
   operational delta is usually one `scripts/<svc>.env`.
2. Probe the target (ssh as `dank`, passwordless sudo present):
   - `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
   - `git -C ~/serverAppConfigs rev-parse HEAD`
   - `systemctl is-active/is-enabled <svc>`;
     `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
   - the on-host env file already carries the new keys (it landed via the
     ff-pull).
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
   `dank` (rootless store; bare `docker` is NOT the shim → "No such object";
   `sudo podman` inspects an empty store):
   - `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
     the pull ⇒ it read the new env
   - `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
   - engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
     tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
   - confirm the launcher maps each tunable to its flag
     (`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
     --enable-prefix-caching`.

**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
live — **do NOT restart**; reconcile the marker (below). Only if the live
container lacks the flag/env (or predates the pull) is a real
`deploy.sh <target> --service <svc>` warranted.

**Reconcile (no restart)** — mirror the deployer's atomic idiom:

```bash
S=$HOME/.local/state/serverAppConfigs/deploy-state
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
```

Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
no-op).

**Anti-patterns:** trusting a commit message ("X deployed") or the marker
alone — both are bookkeeping; the container's argv/env is ground truth.
Reading container state with `sudo podman` / bare `docker`. Restarting a
healthy heavy endpoint to satisfy a stale marker.

## 2. Pending lockout

Symptom: `deploy.sh <target>` fails or silently stops with
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
appears.

1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
   reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
   `echo $?`.
2. Inspect the state store (no sudo):
   `ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
   A `status=pending` entry for a service absent from `service_is_allowed`
   in `scripts/deploy-remote.sh` = retired-service lockout.
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
   not-allowed pending entries to
   `~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
   continues. Verify the WARN line names the service and the state file is
   gone.
4. On a host without the fix: move the stale entry to
   `~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
   `deploy-state/` no longer holds the retired name.

Related contracts: the superseded seam
(`deployment_scoping.bats` "remote runner recognizes a pending service at
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
`deployment_flow.bats` extracts `add_pending_services` via sed and
HOME-overrides the trash path. **When retiring a service going forward:**
delete or reconcile its deploy-state entry as part of the retirement commit —
the prune is a safety net, not the intended cleanup path.

## 3. Interrupted or raced deploys

The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
host-side one** — two `deploy-remote.sh` processes can end up managing the
same unit, one deploying a revision you already superseded. Check:
`pgrep -af "deploy-remote"` on the target, read the revision argument on
each, kill stale ones by explicit PID.

A killed deploy leaves behind:

- the unit **stopped/failed** (`Restart=always` does NOT fire after an
  explicit stop — start it yourself);
- the state marker **pending** (pre-deploy or in-flight revision);
- the e2e gate un-run (the deployer runs it last).

Reconcile the marker ONLY after the service itself is verified healthy — a
marker that runs ahead of reality is worse than one that lags (the planner
uses it to decide what is pending). Then close the loop:

1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
   slots, argv flags, `NRestarts`), not just health 200.
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
   that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
   does NOT have it; copy it in first).
3. If the killed revision was restoring a known-good shape, confirm the
   numbers match the last measured-good boot of that shape — equality is the
   proof, not health alone.

Boot failures must be read from the journal, not `podman logs`: serve scripts
`podman rm -f` the container on every start (previous attempt lost), and
`podman logs` on a container still *Stopping* returns the OLD configuration's
numbers — which looks exactly like your change having no effect. Key on
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
procedure.

## Related

- [[Deployment Flow]] — the deploy pipeline itself
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture