Frank
33eae71d33
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
152 lines
7.3 KiB
Markdown
152 lines
7.3 KiB
Markdown
---
|
||
type: reference
|
||
title: Deploy State Triage
|
||
description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints.
|
||
tags: [deployment, deploy-state, systemd, podman, triage]
|
||
timestamp: 2026-10-05T00:00:00-07:00
|
||
---
|
||
|
||
# Deploy State Triage
|
||
|
||
The deploy-state store is `~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||
on each target (rootless, user `dank`, format exactly two lines
|
||
`revision=<sha>` / `status=deployed|pending`, written atomically via tmp file
|
||
+ `mv`). It is written ONLY by a `deploy.sh` dispatch.
|
||
|
||
## 1. Stale marker vs live container
|
||
|
||
**Why it happens:** an out-of-band relaunch — reboot, `systemctl restart`, or
|
||
a restart after the checkout ff-pulled to a newer revision — starts the
|
||
container from the CURRENT `scripts/<svc>.env` but does NOT advance the
|
||
marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
||
"behavior pending".
|
||
|
||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
||
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
|
||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||
possibly zero behavioral change.
|
||
|
||
**Procedure (read-only first):**
|
||
|
||
1. Committed state: `git log --oneline main..<branch>` empty ⇒ merged; the
|
||
operational delta is usually one `scripts/<svc>.env`.
|
||
2. Probe the target (ssh as `dank`, passwordless sudo present):
|
||
- `cat ~/.local/state/serverAppConfigs/deploy-state/<svc>`
|
||
- `git -C ~/serverAppConfigs rev-parse HEAD`
|
||
- `systemctl is-active/is-enabled <svc>`;
|
||
`curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health`
|
||
- the on-host env file already carries the new keys (it landed via the
|
||
ff-pull).
|
||
3. **Decisive — read the LIVE container, not the marker.** Use `podman` as
|
||
`dank` (rootless store; bare `docker` is NOT the shim → "No such object";
|
||
`sudo podman` inspects an empty store):
|
||
- `podman inspect <ctr> --format '{{.State.StartedAt}}'` — booted AFTER
|
||
the pull ⇒ it read the new env
|
||
- `podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>`
|
||
- engine env: `podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do
|
||
tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'`
|
||
- confirm the launcher maps each tunable to its flag
|
||
(`grep scripts/launch.sh`), e.g. `R9V_ENABLE_PREFIX_CACHING=1 →
|
||
--enable-prefix-caching`.
|
||
|
||
**Decision:** live argv+env == target config AND `/health` 200 ⇒ behavior is
|
||
live — **do NOT restart**; reconcile the marker (below). Only if the live
|
||
container lacks the flag/env (or predates the pull) is a real
|
||
`deploy.sh <target> --service <svc>` warranted.
|
||
|
||
**Reconcile (no restart)** — mirror the deployer's atomic idiom:
|
||
|
||
```bash
|
||
S=$HOME/.local/state/serverAppConfigs/deploy-state
|
||
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
|
||
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"
|
||
```
|
||
|
||
Then prove the gate: `./tests/run.sh` + `tests/e2e/run.sh` (needs
|
||
`tests/e2e/.env.local`; expect the documented `11-admin` red if the
|
||
FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with `ping -c2` +
|
||
a `/dev/tcp` probe: 100 % loss + timeout = host off, not a regression).
|
||
Log to `serverAppConfigs-docs/log.md`; commit + push (docs-only ⇒ deploy
|
||
no-op).
|
||
|
||
**Anti-patterns:** trusting a commit message ("X deployed") or the marker
|
||
alone — both are bookkeeping; the container's argv/env is ground truth.
|
||
Reading container state with `sudo podman` / bare `docker`. Restarting a
|
||
healthy heavy endpoint to satisfy a stale marker.
|
||
|
||
## 2. Pending lockout
|
||
|
||
Symptom: `deploy.sh <target>` fails or silently stops with
|
||
`service <name> is not deployed on <target>` and the `[3/3]` e2e gate never
|
||
appears.
|
||
|
||
1. **Get the real exit code first:** `./scripts/deploy.sh X 2>&1 | tail`
|
||
reports TAIL's 0 — use `set -o pipefail` or redirect to a file, then
|
||
`echo $?`.
|
||
2. Inspect the state store (no sudo):
|
||
`ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'`.
|
||
A `status=pending` entry for a service absent from `service_is_allowed`
|
||
in `scripts/deploy-remote.sh` = retired-service lockout.
|
||
3. Fixed behavior (since 02a7876, 2026-09-17): `add_pending_services` prunes
|
||
not-allowed pending entries to
|
||
`~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service>` and the run
|
||
continues. Verify the WARN line names the service and the state file is
|
||
gone.
|
||
4. On a host without the fix: move the stale entry to
|
||
`~/.AGENTTRASH/<date>-retired-deploy-state/` manually (never `rm`), re-run.
|
||
5. Verify: deploy reaches `[3/3] Running Playwright e2e gate` and completes;
|
||
`deploy-state/` no longer holds the retired name.
|
||
|
||
Related contracts: the superseded seam
|
||
(`deployment_scoping.bats` "remote runner recognizes a pending service at
|
||
the same revision") now asserts prune-and-continue (exit 0), not exit 2;
|
||
`deployment_flow.bats` extracts `add_pending_services` via sed and
|
||
HOME-overrides the trash path. **When retiring a service going forward:**
|
||
delete or reconcile its deploy-state entry as part of the retirement commit —
|
||
the prune is a safety net, not the intended cleanup path.
|
||
|
||
## 3. Interrupted or raced deploys
|
||
|
||
The deploy is TWO processes: `deploy.sh` (dev box) and `deploy-remote.sh`
|
||
(runs on the target over ssh). **Killing the dev-side job does NOT kill the
|
||
host-side one** — two `deploy-remote.sh` processes can end up managing the
|
||
same unit, one deploying a revision you already superseded. Check:
|
||
`pgrep -af "deploy-remote"` on the target, read the revision argument on
|
||
each, kill stale ones by explicit PID.
|
||
|
||
A killed deploy leaves behind:
|
||
|
||
- the unit **stopped/failed** (`Restart=always` does NOT fire after an
|
||
explicit stop — start it yourself);
|
||
- the state marker **pending** (pre-deploy or in-flight revision);
|
||
- the e2e gate un-run (the deployer runs it last).
|
||
|
||
Reconcile the marker ONLY after the service itself is verified healthy — a
|
||
marker that runs ahead of reality is worse than one that lags (the planner
|
||
uses it to decide what is pending). Then close the loop:
|
||
|
||
1. Verify against MEASURED, config-specific numbers (KV capacity line, arena
|
||
slots, argv flags, `NRestarts`), not just health 200.
|
||
2. Run the gate the dead deploy skipped (`tests/e2e/run.sh` from a checkout
|
||
that has the gitignored `tests/e2e/.env.local` — a fresh `git worktree`
|
||
does NOT have it; copy it in first).
|
||
3. If the killed revision was restoring a known-good shape, confirm the
|
||
numbers match the last measured-good boot of that shape — equality is the
|
||
proof, not health alone.
|
||
|
||
Boot failures must be read from the journal, not `podman logs`: serve scripts
|
||
`podman rm -f` the container on every start (previous attempt lost), and
|
||
`podman logs` on a container still *Stopping* returns the OLD configuration's
|
||
numbers — which looks exactly like your change having no effect. Key on
|
||
container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the
|
||
cause from `journalctl --since "N min ago" | grep -a "<container-name>\["`.
|
||
See [[vLLM Boot Forensics and Config Interrogation]] for the full capture
|
||
procedure.
|
||
|
||
## Related
|
||
|
||
- [[Deployment Flow]] — the deploy pipeline itself
|
||
- [[Quadlet Lifecycle]] — unit ownership and drift diagnosis
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — journal capture
|