Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

7.3 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Deploy State Triage

The deploy-state store is ~/.local/state/serverAppConfigs/deploy-state/<svc> on each target (rootless, user dank, format exactly two lines revision=<sha> / status=deployed|pending, written atomically via tmp file

mv). It is written ONLY by a deploy.sh dispatch.

1. Stale marker vs live container

Why it happens: an out-of-band relaunch — reboot, systemctl restart, or a restart after the checkout ff-pulled to a newer revision — starts the container from the CURRENT scripts/<svc>.env but does NOT advance the marker. So marker != HEAD means "not dispatched via deploy.sh", NOT "behavior pending".

Why it matters: a plain deploy.sh run re-selects and restarts a service whose marker lags. On the big-payload endpoints that is a full model reload measured in tens of minutes — each endpoint's boot budget is on its own page (frank-archon, frank-dragoon, frank-zealot) — and frank-archon (dankserver:7770) is also the harness's own model backend, so a blind deploy is a self-inflicted outage for possibly zero behavioral change.

Procedure (read-only first):

Committed state: git log --oneline main..<branch> empty ⇒ merged; the operational delta is usually one scripts/<svc>.env.
Probe the target (ssh as dank, passwordless sudo present):
- cat ~/.local/state/serverAppConfigs/deploy-state/<svc>
- git -C ~/serverAppConfigs rev-parse HEAD
- systemctl is-active/is-enabled <svc>; curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health
- the on-host env file already carries the new keys (it landed via the ff-pull).
Decisive — read the LIVE container, not the marker. Use podman as dank (rootless store; bare docker is NOT the shim → "No such object"; sudo podman inspects an empty store):
- podman inspect <ctr> --format '{{.State.StartedAt}}' — booted AFTER the pull ⇒ it read the new env
- podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>
- engine env: podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'

confirm the launcher maps each tunable to its flag (grep scripts/launch.sh).

Decision: live argv+env == target config AND /health 200 ⇒ behavior is live — do NOT restart; reconcile the marker (below). Only if the live container lacks the flag/env (or predates the pull) is a real deploy.sh <target> --service <svc> warranted.

Reconcile (no restart) — mirror the deployer's atomic idiom:

S=$HOME/.local/state/serverAppConfigs/deploy-state
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"

Then prove the gate: ./tests/run.sh + tests/e2e/run.sh (needs tests/e2e/.env.local; expect the documented 11-admin red if the FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with ping -c2 + a /dev/tcp probe: 100 % loss + timeout = host off, not a regression). Log to serverAppConfigs-docs/log.md; commit + push (docs-only ⇒ deploy no-op).

Anti-patterns: trusting a commit message ("X deployed") or the marker alone — both are bookkeeping; the container's argv/env is ground truth. Reading container state with sudo podman / bare docker. Restarting a healthy heavy endpoint to satisfy a stale marker.

2. Pending lockout

Symptom: deploy.sh <target> fails or silently stops with service <name> is not deployed on <target> and the [3/3] e2e gate never appears.

Get the real exit code first: ./scripts/deploy.sh X 2>&1 | tail reports TAIL's 0 — use set -o pipefail or redirect to a file, then echo $?.
Inspect the state store (no sudo): ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'. A status=pending entry for a service absent from service_is_allowed in scripts/deploy-remote.sh = retired-service lockout.
Fixed behavior (since 02a7876, 2026-09-17): add_pending_services prunes not-allowed pending entries to ~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service> and the run continues. Verify the WARN line names the service and the state file is gone.
On a host without the fix: move the stale entry to ~/.AGENTTRASH/<date>-retired-deploy-state/ manually (never rm), re-run.
Verify: deploy reaches [3/3] Running Playwright e2e gate and completes; deploy-state/ no longer holds the retired name.

Related contracts: the superseded seam (deployment_scoping.bats "remote runner recognizes a pending service at the same revision") now asserts prune-and-continue (exit 0), not exit 2; deployment_flow.bats extracts add_pending_services via sed and HOME-overrides the trash path. When retiring a service going forward: delete or reconcile its deploy-state entry as part of the retirement commit — the prune is a safety net, not the intended cleanup path.

3. Interrupted or raced deploys

The deploy is TWO processes: deploy.sh (dev box) and deploy-remote.sh (runs on the target over ssh). Killing the dev-side job does NOT kill the host-side one — two deploy-remote.sh processes can end up managing the same unit, one deploying a revision you already superseded. Check: pgrep -af "deploy-remote" on the target, read the revision argument on each, kill stale ones by explicit PID.

A killed deploy leaves behind:

the unit stopped/failed (Restart=always does NOT fire after an explicit stop — start it yourself);
the state marker pending (pre-deploy or in-flight revision);
the e2e gate un-run (the deployer runs it last).

Reconcile the marker ONLY after the service itself is verified healthy — a marker that runs ahead of reality is worse than one that lags (the planner uses it to decide what is pending). Then close the loop:

Verify against MEASURED, config-specific numbers (KV capacity line, arena slots, argv flags, NRestarts), not just health 200.
Run the gate the dead deploy skipped (tests/e2e/run.sh from a checkout that has the gitignored tests/e2e/.env.local — a fresh git worktree does NOT have it; copy it in first).
If the killed revision was restoring a known-good shape, confirm the numbers match the last measured-good boot of that shape — equality is the proof, not health alone.

Boot failures must be read from the journal, not podman logs: serve scripts podman rm -f the container on every start (previous attempt lost), and podman logs on a container still Stopping returns the OLD configuration's numbers — which looks exactly like your change having no effect. Key on container identity (podman ps -a --format '{{.CreatedAt}}') and read the cause from journalctl --since "N min ago" | grep -a "<container-name>\[". See vLLM Boot Forensics and Config Interrogation for the full capture procedure.

Deployment Flow — the deploy pipeline itself
Quadlet Lifecycle — unit ownership and drift diagnosis
vLLM Boot Forensics and Config Interrogation — journal capture

7.3 KiB Raw Blame History

Deploy State Triage

1. Stale marker vs live container

2. Pending lockout

3. Interrupted or raced deploys

Related

7.3 KiB

Raw Blame History