dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d5934a6b470f7d82fe6e87c6a45e53bc286db8f1
serverAppConfigs/serverAppConfigs-docs/Architecture/Deploy State Triage.md
T
Frank 6bdfd2853e docs(vault): service and infrastructure pages stop restating endpoint facts 
18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).

Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.

Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
2026-10-19 00:56:02 -07:00

7.4 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
reference Deploy State Triage Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
deployment
deploy-state
systemd
podman
triage
2026-10-05T00:00:00-07:00

Deploy State Triage

The deploy-state store is ~/.local/state/serverAppConfigs/deploy-state/<svc> on each target (rootless, user dank, format exactly two lines revision=<sha> / status=deployed|pending, written atomically via tmp file

  • mv). It is written ONLY by a deploy.sh dispatch.

1. Stale marker vs live container

Why it happens: an out-of-band relaunch — reboot, systemctl restart, or a restart after the checkout ff-pulled to a newer revision — starts the container from the CURRENT scripts/<svc>.env but does NOT advance the marker. So marker != HEAD means "not dispatched via deploy.sh", NOT "behavior pending".

Why it matters: a plain deploy.sh run re-selects and restarts a service whose marker lags. On the big-payload endpoints that is a full model reload measured in tens of minutes — each endpoint's boot budget is on its own page (frank-archon, frank-dragoon, frank-zealot) — and frank-archon (dankserver:7770) is also the harness's own model backend, so a blind deploy is a self-inflicted outage for possibly zero behavioral change.

Procedure (read-only first):

  1. Committed state: git log --oneline main..<branch> empty ⇒ merged; the operational delta is usually one scripts/<svc>.env.
  2. Probe the target (ssh as dank, passwordless sudo present):
    • cat ~/.local/state/serverAppConfigs/deploy-state/<svc>
    • git -C ~/serverAppConfigs rev-parse HEAD
    • systemctl is-active/is-enabled <svc>; curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health
    • the on-host env file already carries the new keys (it landed via the ff-pull).
  3. Decisive — read the LIVE container, not the marker. Use podman as dank (rootless store; bare docker is NOT the shim → "No such object"; sudo podman inspects an empty store):
    • podman inspect <ctr> --format '{{.State.StartedAt}}' — booted AFTER the pull ⇒ it read the new env
    • podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>
    • engine env: podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'
  • confirm the launcher maps each tunable to its flag (grep scripts/launch.sh).

Decision: live argv+env == target config AND /health 200 ⇒ behavior is live — do NOT restart; reconcile the marker (below). Only if the live container lacks the flag/env (or predates the pull) is a real deploy.sh <target> --service <svc> warranted.

Reconcile (no restart) — mirror the deployer's atomic idiom:

S=$HOME/.local/state/serverAppConfigs/deploy-state
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"

Then prove the gate: ./tests/run.sh + tests/e2e/run.sh (needs tests/e2e/.env.local; expect the documented 11-admin red if the FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with ping -c2 + a /dev/tcp probe: 100 % loss + timeout = host off, not a regression). Log to serverAppConfigs-docs/log.md; commit + push (docs-only ⇒ deploy no-op).

Anti-patterns: trusting a commit message ("X deployed") or the marker alone — both are bookkeeping; the container's argv/env is ground truth. Reading container state with sudo podman / bare docker. Restarting a healthy heavy endpoint to satisfy a stale marker.

2. Pending lockout

Symptom: deploy.sh <target> fails or silently stops with service <name> is not deployed on <target> and the [3/3] e2e gate never appears.

  1. Get the real exit code first: ./scripts/deploy.sh X 2>&1 | tail reports TAIL's 0 — use set -o pipefail or redirect to a file, then echo $?.
  2. Inspect the state store (no sudo): ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'. A status=pending entry for a service absent from service_is_allowed in scripts/deploy-remote.sh = retired-service lockout.
  3. Fixed behavior (since 02a7876, 2026-09-17): add_pending_services prunes not-allowed pending entries to ~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service> and the run continues. Verify the WARN line names the service and the state file is gone.
  4. On a host without the fix: move the stale entry to ~/.AGENTTRASH/<date>-retired-deploy-state/ manually (never rm), re-run.
  5. Verify: deploy reaches [3/3] Running Playwright e2e gate and completes; deploy-state/ no longer holds the retired name.

Related contracts: the superseded seam (deployment_scoping.bats "remote runner recognizes a pending service at the same revision") now asserts prune-and-continue (exit 0), not exit 2; deployment_flow.bats extracts add_pending_services via sed and HOME-overrides the trash path. When retiring a service going forward: delete or reconcile its deploy-state entry as part of the retirement commit — the prune is a safety net, not the intended cleanup path.

3. Interrupted or raced deploys

The deploy is TWO processes: deploy.sh (dev box) and deploy-remote.sh (runs on the target over ssh). Killing the dev-side job does NOT kill the host-side one — two deploy-remote.sh processes can end up managing the same unit, one deploying a revision you already superseded. Check: pgrep -af "deploy-remote" on the target, read the revision argument on each, kill stale ones by explicit PID.

A killed deploy leaves behind:

  • the unit stopped/failed (Restart=always does NOT fire after an explicit stop — start it yourself);
  • the state marker pending (pre-deploy or in-flight revision);
  • the e2e gate un-run (the deployer runs it last).

Reconcile the marker ONLY after the service itself is verified healthy — a marker that runs ahead of reality is worse than one that lags (the planner uses it to decide what is pending). Then close the loop:

  1. Verify against MEASURED, config-specific numbers (KV capacity line, arena slots, argv flags, NRestarts), not just health 200.
  2. Run the gate the dead deploy skipped (tests/e2e/run.sh from a checkout that has the gitignored tests/e2e/.env.local — a fresh git worktree does NOT have it; copy it in first).
  3. If the killed revision was restoring a known-good shape, confirm the numbers match the last measured-good boot of that shape — equality is the proof, not health alone.

Boot failures must be read from the journal, not podman logs: serve scripts podman rm -f the container on every start (previous attempt lost), and podman logs on a container still Stopping returns the OLD configuration's numbers — which looks exactly like your change having no effect. Key on container identity (podman ps -a --format '{{.CreatedAt}}') and read the cause from journalctl --since "N min ago" | grep -a "<container-name>\[". See vLLM Boot Forensics and Config Interrogation for the full capture procedure.

Related

Reference in New Issue View Git Blame Copy Permalink