serverAppConfigs/serverAppConfigs-docs/Architecture/Deploy State Triage.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Deploy State Triage	Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.	deployment deploy-state systemd podman triage	2026-10-05T00:00:00-07:00

Deploy State Triage

The deploy-state store is ~/.local/state/serverAppConfigs/deploy-state/<svc> on each target (rootless, user dank, format exactly two lines revision=<sha> / status=deployed|pending, written atomically via tmp file

• mv). It is written ONLY by a deploy.sh dispatch.

1. Stale marker vs live container

Why it happens: an out-of-band relaunch — reboot, systemctl restart, or a restart after the checkout ff-pulled to a newer revision — starts the container from the CURRENT scripts/<svc>.env but does NOT advance the marker. So marker != HEAD means "not dispatched via deploy.sh", NOT "behavior pending".

Why it matters: a plain deploy.sh run re-selects and restarts a service whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of a 100+ GiB payload — and frank-archon (dankserver:7770) is also the harness's own model backend, so a blind deploy is a self-inflicted outage for possibly zero behavioral change.

Procedure (read-only first):

1. Committed state: git log --oneline main..<branch> empty ⇒ merged; the operational delta is usually one scripts/<svc>.env.
2. Probe the target (ssh as dank, passwordless sudo present):
   • cat ~/.local/state/serverAppConfigs/deploy-state/<svc>
   • git -C ~/serverAppConfigs rev-parse HEAD
   • systemctl is-active/is-enabled <svc>; curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health
   • the on-host env file already carries the new keys (it landed via the ff-pull).
3. Decisive — read the LIVE container, not the marker. Use podman as dank (rootless store; bare docker is NOT the shim → "No such object"; sudo podman inspects an empty store):
   • podman inspect <ctr> --format '{{.State.StartedAt}}' — booted AFTER the pull ⇒ it read the new env
   • podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>
   • engine env: podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'
   • confirm the launcher maps each tunable to its flag (grep scripts/launch.sh), e.g. R9V_ENABLE_PREFIX_CACHING=1 → --enable-prefix-caching.

Decision: live argv+env == target config AND /health 200 ⇒ behavior is live — do NOT restart; reconcile the marker (below). Only if the live container lacks the flag/env (or predates the pull) is a real deploy.sh <target> --service <svc> warranted.

Reconcile (no restart) — mirror the deployer's atomic idiom:

S=$HOME/.local/state/serverAppConfigs/deploy-state
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"

Then prove the gate: ./tests/run.sh + tests/e2e/run.sh (needs tests/e2e/.env.local; expect the documented 11-admin red if the FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with ping -c2 + a /dev/tcp probe: 100 % loss + timeout = host off, not a regression). Log to serverAppConfigs-docs/log.md; commit + push (docs-only ⇒ deploy no-op).

Anti-patterns: trusting a commit message ("X deployed") or the marker alone — both are bookkeeping; the container's argv/env is ground truth. Reading container state with sudo podman / bare docker. Restarting a healthy heavy endpoint to satisfy a stale marker.

2. Pending lockout

Symptom: deploy.sh <target> fails or silently stops with service <name> is not deployed on <target> and the [3/3] e2e gate never appears.

1. Get the real exit code first: ./scripts/deploy.sh X 2>&1 | tail reports TAIL's 0 — use set -o pipefail or redirect to a file, then echo $?.
2. Inspect the state store (no sudo): ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'. A status=pending entry for a service absent from service_is_allowed in scripts/deploy-remote.sh = retired-service lockout.
3. Fixed behavior (since 02a7876, 2026-09-17): add_pending_services prunes not-allowed pending entries to ~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service> and the run continues. Verify the WARN line names the service and the state file is gone.
4. On a host without the fix: move the stale entry to ~/.AGENTTRASH/<date>-retired-deploy-state/ manually (never rm), re-run.
5. Verify: deploy reaches [3/3] Running Playwright e2e gate and completes; deploy-state/ no longer holds the retired name.

Related contracts: the superseded seam (deployment_scoping.bats "remote runner recognizes a pending service at the same revision") now asserts prune-and-continue (exit 0), not exit 2; deployment_flow.bats extracts add_pending_services via sed and HOME-overrides the trash path. When retiring a service going forward: delete or reconcile its deploy-state entry as part of the retirement commit — the prune is a safety net, not the intended cleanup path.

3. Interrupted or raced deploys

The deploy is TWO processes: deploy.sh (dev box) and deploy-remote.sh (runs on the target over ssh). Killing the dev-side job does NOT kill the host-side one — two deploy-remote.sh processes can end up managing the same unit, one deploying a revision you already superseded. Check: pgrep -af "deploy-remote" on the target, read the revision argument on each, kill stale ones by explicit PID.

A killed deploy leaves behind:

• the unit stopped/failed (Restart=always does NOT fire after an explicit stop — start it yourself);
• the state marker pending (pre-deploy or in-flight revision);
• the e2e gate un-run (the deployer runs it last).

Reconcile the marker ONLY after the service itself is verified healthy — a marker that runs ahead of reality is worse than one that lags (the planner uses it to decide what is pending). Then close the loop:

1. Verify against MEASURED, config-specific numbers (KV capacity line, arena slots, argv flags, NRestarts), not just health 200.
2. Run the gate the dead deploy skipped (tests/e2e/run.sh from a checkout that has the gitignored tests/e2e/.env.local — a fresh git worktree does NOT have it; copy it in first).
3. If the killed revision was restoring a known-good shape, confirm the numbers match the last measured-good boot of that shape — equality is the proof, not health alone.

Boot failures must be read from the journal, not podman logs: serve scripts podman rm -f the container on every start (previous attempt lost), and podman logs on a container still Stopping returns the OLD configuration's numbers — which looks exactly like your change having no effect. Key on container identity (podman ps -a --format '{{.CreatedAt}}') and read the cause from journalctl --since "N min ago" | grep -a "<container-name>\[". See vLLM Boot Forensics and Config Interrogation for the full capture procedure.

Related

• Deployment Flow — the deploy pipeline itself
• Quadlet Lifecycle — unit ownership and drift diagnosis
• vLLM Boot Forensics and Config Interrogation — journal capture