Files

T

Frank 33eae71d33 docs: distill the managed skill library into the vault (waves 1-7)

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.

2026-10-05 21:19:15 -07:00

7.3 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Deploy State Triage

The deploy-state store is ~/.local/state/serverAppConfigs/deploy-state/<svc> on each target (rootless, user dank, format exactly two lines revision=<sha> / status=deployed|pending, written atomically via tmp file

mv). It is written ONLY by a deploy.sh dispatch.

1. Stale marker vs live container

Why it happens: an out-of-band relaunch — reboot, systemctl restart, or a restart after the checkout ff-pulled to a newer revision — starts the container from the CURRENT scripts/<svc>.env but does NOT advance the marker. So marker != HEAD means "not dispatched via deploy.sh", NOT "behavior pending".

Why it matters: a plain deploy.sh run re-selects and restarts a service whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of a 100+ GiB payload — and frank-primary (dankserver:7770) is also the harness's own model backend, so a blind deploy is a self-inflicted outage for possibly zero behavioral change.

Procedure (read-only first):

Committed state: git log --oneline main..<branch> empty ⇒ merged; the operational delta is usually one scripts/<svc>.env.
Probe the target (ssh as dank, passwordless sudo present):
- cat ~/.local/state/serverAppConfigs/deploy-state/<svc>
- git -C ~/serverAppConfigs rev-parse HEAD
- systemctl is-active/is-enabled <svc>; curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:<port>/health
- the on-host env file already carries the new keys (it landed via the ff-pull).
Decisive — read the LIVE container, not the marker. Use podman as dank (rootless store; bare docker is NOT the shim → "No such object"; sudo podman inspects an empty store):
- podman inspect <ctr> --format '{{.State.StartedAt}}' — booted AFTER the pull ⇒ it read the new env
- podman inspect <ctr> --format '{{.Args}}' | tr ' ' '\n' | grep -i <flag>
- engine env: podman exec <ctr> sh -c 'for f in /proc/[0-9]*/environ; do tr "\0" "\n" <$f 2>/dev/null; done | grep -i <VAR>'
- confirm the launcher maps each tunable to its flag (grep scripts/launch.sh), e.g. R9V_ENABLE_PREFIX_CACHING=1 → --enable-prefix-caching.

Decision: live argv+env == target config AND /health 200 ⇒ behavior is live — do NOT restart; reconcile the marker (below). Only if the live container lacks the flag/env (or predates the pull) is a real deploy.sh <target> --service <svc> warranted.

Reconcile (no restart) — mirror the deployer's atomic idiom:

S=$HOME/.local/state/serverAppConfigs/deploy-state
H=$(git -C ~/serverAppConfigs rev-parse HEAD)
t=$(mktemp "$S/.<svc>.XXXXXX"); printf 'revision=%s\nstatus=%s\n' "$H" deployed > "$t"; mv -f -- "$t" "$S/<svc>"

Then prove the gate: ./tests/run.sh + tests/e2e/run.sh (needs tests/e2e/.env.local; expect the documented 11-admin red if the FrankBook-Pro Mac mini 192.168.0.112 is ASLEEP — confirm with ping -c2 + a /dev/tcp probe: 100 % loss + timeout = host off, not a regression). Log to serverAppConfigs-docs/log.md; commit + push (docs-only ⇒ deploy no-op).

Anti-patterns: trusting a commit message ("X deployed") or the marker alone — both are bookkeeping; the container's argv/env is ground truth. Reading container state with sudo podman / bare docker. Restarting a healthy heavy endpoint to satisfy a stale marker.

2. Pending lockout

Symptom: deploy.sh <target> fails or silently stops with service <name> is not deployed on <target> and the [3/3] e2e gate never appears.

Get the real exit code first: ./scripts/deploy.sh X 2>&1 | tail reports TAIL's 0 — use set -o pipefail or redirect to a file, then echo $?.
Inspect the state store (no sudo): ssh <target> 'cat ~/.local/state/serverAppConfigs/deploy-state/<name>'. A status=pending entry for a service absent from service_is_allowed in scripts/deploy-remote.sh = retired-service lockout.
Fixed behavior (since 02a7876, 2026-09-17): add_pending_services prunes not-allowed pending entries to ~/.AGENTTRASH/<YYYYMMDD>-retired-deploy-state/<service> and the run continues. Verify the WARN line names the service and the state file is gone.
On a host without the fix: move the stale entry to ~/.AGENTTRASH/<date>-retired-deploy-state/ manually (never rm), re-run.
Verify: deploy reaches [3/3] Running Playwright e2e gate and completes; deploy-state/ no longer holds the retired name.

Related contracts: the superseded seam (deployment_scoping.bats "remote runner recognizes a pending service at the same revision") now asserts prune-and-continue (exit 0), not exit 2; deployment_flow.bats extracts add_pending_services via sed and HOME-overrides the trash path. When retiring a service going forward: delete or reconcile its deploy-state entry as part of the retirement commit — the prune is a safety net, not the intended cleanup path.

3. Interrupted or raced deploys

The deploy is TWO processes: deploy.sh (dev box) and deploy-remote.sh (runs on the target over ssh). Killing the dev-side job does NOT kill the host-side one — two deploy-remote.sh processes can end up managing the same unit, one deploying a revision you already superseded. Check: pgrep -af "deploy-remote" on the target, read the revision argument on each, kill stale ones by explicit PID.

A killed deploy leaves behind:

the unit stopped/failed (Restart=always does NOT fire after an explicit stop — start it yourself);
the state marker pending (pre-deploy or in-flight revision);
the e2e gate un-run (the deployer runs it last).

Reconcile the marker ONLY after the service itself is verified healthy — a marker that runs ahead of reality is worse than one that lags (the planner uses it to decide what is pending). Then close the loop:

Verify against MEASURED, config-specific numbers (KV capacity line, arena slots, argv flags, NRestarts), not just health 200.
Run the gate the dead deploy skipped (tests/e2e/run.sh from a checkout that has the gitignored tests/e2e/.env.local — a fresh git worktree does NOT have it; copy it in first).
If the killed revision was restoring a known-good shape, confirm the numbers match the last measured-good boot of that shape — equality is the proof, not health alone.

Boot failures must be read from the journal, not podman logs: serve scripts podman rm -f the container on every start (previous attempt lost), and podman logs on a container still Stopping returns the OLD configuration's numbers — which looks exactly like your change having no effect. Key on container identity (podman ps -a --format '{{.CreatedAt}}') and read the cause from journalctl --since "N min ago" | grep -a "<container-name>\[". See vLLM Boot Forensics and Config Interrogation for the full capture procedure.

Deployment Flow — the deploy pipeline itself
Quadlet Lifecycle — unit ownership and drift diagnosis
vLLM Boot Forensics and Config Interrogation — journal capture

7.3 KiB Raw Blame History Unescape Escape

Deploy State Triage

1. Stale marker vs live container

2. Pending lockout

3. Interrupted or raced deploys

Related

7.3 KiB

Raw Blame History