live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/frank-bench-supervisor-recovery/SKILL.md

@@ -0,0 +1,45 @@
+---
+name: frank-bench-supervisor-recovery
+description: "Recover a frank-intelligence-bench run after engine faults: detect poison grade rows, void them, restart supervisors with fixed code, judge cutover, watchdog for multi-day runs."
+---
+
+Recover frank-intelligence-bench runs on the dank-fleet after engine faults (poison graders, judge swaps, supervisor deaths).
+
+## Laws (friction-earned 2026-10-10/11)
+1. **A code fix is only real after the supervisor restarts.** Bun loads code at spawn; an on-disk fix never reaches the running process. Kill `_exec`, relaunch — takeover law makes resume cheap.
+2. **A judge cohort is all-or-nothing.** Never mix judges on one eval's grade rows. Void ALL grade rows for LLM-judged evals (`phase == "grade"`), KEEP all `gen` rows (target output is judge-independent) — resume rejudges from cache.
+3. **Poison census signatures:** grade rows with empty text + forced FAIL = engine, not model. `unparseable verdict` storm = judge output class (empty `content` + chain in `reasoning_content`; or 2048-tok truncation before verdict JSON). Default census sig `AssertionError` masked a poison shift — treat mass-default sigs as red flags.
+4. **Detached launch form matters:** from a short-lived harness shell, `(setsid nohup cmd & )` (subshell) gets cgroup-reaped within minutes — silent death, stale `worker.pid`, no log line. Proven form: `setsid nohup cmd >> log.txt 2>&1 < /dev/null & disown` as the tail of the same shell (survived 6 h overnight).
+
+## Void-and-resume procedure
+```bash
+pkill -f "cli.ts _exec"
+python3 - <<'PYEOF'
+import json, pathlib, shutil, time
+led = pathlib.Path("runs/<RUN_ID>/ledger.jsonl")
+rows = [l for l in led.read_text().splitlines() if l.strip()]
+keep, void = [], []
+for l in rows:
+    r = json.loads(l)
+    if r["eval"] in (<EVALS>) and r["phase"] == "grade" and r["ts"] > "<CUTOVER_ISO>":
+        void.append(l)
+    else:
+        keep.append(l)
+shutil.copy(led, f".AGENTTRASH/ledger-voided-{int(time.time())}.jsonl")
+led.write_text("\n".join(keep) + "\n")
+print(f"voided {len(void)}; kept {len(keep)}")
+PYEOF
+cd runs/<RUN_ID> && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown
+```
+- Judge cutover variant: void ALL `phase==grade` rows for the judged evals (no ts filter), edit `run.json` `judge` field, relaunch.
+- Validate the judge endpoint first: live probe both directions (correct + incorrect candidate); a good judge returns bare verdict JSON fast.
+- Resume proof = a specific known step flipping FAIL→PASS (or vice versa) in the log within minutes.
+
+## Watchdog for multi-day runs
+No cron on the Arch workstations — systemd user timer, idempotent (`cli.ts worker` exits instantly when supervisor heartbeat is fresh; takes over via pid-file 30 s freshness gate):
+`~/.config/systemd/user/fib-watchdog.service`: Type=oneshot, WorkingDirectory=repo, `ExecStart=/usr/bin/bun src/cli.ts worker`, stdout/stderr `append:.../runs/watchdog.log`.
+`fib-watchdog.timer`: OnBootSec=5min, OnUnitActiveSec=10min, Persistent=true, WantedBy=timers.target.
+`systemctl --user enable --now fib-watchdog.timer`; verify `loginctl show-user <u> -p Linger` = yes (survives logout/reboot). Prove idempotence: log line "has a live supervisor — worker exits".
+
+## Scoring truth after repair
+Report renders from ledger with latest-wins via `ok` filter (no explicit dedup): ok=false rows drop, recovered key counts once. UI `done` counts unique `(eval,instance,repeat)` completed keys. SciCode denominators: 341 steps/rep (not 1120/3-rep arithmetic traps).