command-center/agent/managed-skills/frank-bench-supervisor-recovery/SKILL.md

name, description

name	description
frank-bench-supervisor-recovery	Recover a frank-intelligence-bench run after engine faults: detect poison grade rows, void them, restart supervisors with fixed code, judge cutover, watchdog for multi-day runs.

Recover frank-intelligence-bench runs on the dank-fleet after engine faults (poison graders, judge swaps, supervisor deaths).

Laws (friction-earned 2026-10-10/11)

1. A code fix is only real after the supervisor restarts. Bun loads code at spawn; an on-disk fix never reaches the running process. Kill _exec, relaunch — takeover law makes resume cheap.
2. A judge cohort is all-or-nothing. Never mix judges on one eval's grade rows. Void ALL grade rows for LLM-judged evals (phase == "grade"), KEEP all gen rows (target output is judge-independent) — resume rejudges from cache.
3. Poison census signatures: grade rows with empty text + forced FAIL = engine, not model. unparseable verdict storm = judge output class (empty content + chain in reasoning_content; or 2048-tok truncation before verdict JSON). Default census sig AssertionError masked a poison shift — treat mass-default sigs as red flags.
4. Detached launch form matters: from a short-lived harness shell, (setsid nohup cmd & ) (subshell) gets cgroup-reaped within minutes — silent death, stale worker.pid, no log line. Proven form: setsid nohup cmd >> log.txt 2>&1 < /dev/null & disown as the tail of the same shell (survived 6 h overnight).

Void-and-resume procedure

pkill -f "cli.ts _exec"
python3 - <<'PYEOF'
import json, pathlib, shutil, time
led = pathlib.Path("runs/<RUN_ID>/ledger.jsonl")
rows = [l for l in led.read_text().splitlines() if l.strip()]
keep, void = [], []
for l in rows:
    r = json.loads(l)
    if r["eval"] in (<EVALS>) and r["phase"] == "grade" and r["ts"] > "<CUTOVER_ISO>":
        void.append(l)
    else:
        keep.append(l)
shutil.copy(led, f".AGENTTRASH/ledger-voided-{int(time.time())}.jsonl")
led.write_text("\n".join(keep) + "\n")
print(f"voided {len(void)}; kept {len(keep)}")
PYEOF
cd runs/<RUN_ID> && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown

• Judge cutover variant: void ALL phase==grade rows for the judged evals (no ts filter), edit run.json judge field, relaunch.
• Validate the judge endpoint first: live probe both directions (correct + incorrect candidate); a good judge returns bare verdict JSON fast.
• Resume proof = a specific known step flipping FAIL→PASS (or vice versa) in the log within minutes.

Watchdog for multi-day runs

No cron on the Arch workstations — systemd user timer, idempotent (cli.ts worker exits instantly when supervisor heartbeat is fresh; takes over via pid-file 30 s freshness gate): ~/.config/systemd/user/fib-watchdog.service: Type=oneshot, WorkingDirectory=repo, ExecStart=/usr/bin/bun src/cli.ts worker, stdout/stderr append:.../runs/watchdog.log. fib-watchdog.timer: OnBootSec=5min, OnUnitActiveSec=10min, Persistent=true, WantedBy=timers.target. systemctl --user enable --now fib-watchdog.timer; verify loginctl show-user <u> -p Linger = yes (survives logout/reboot). Prove idempotence: log line "has a live supervisor — worker exits".

Scoring truth after repair

Report renders from ledger with latest-wins via ok filter (no explicit dedup): ok=false rows drop, recovered key counts once. UI done counts unique (eval,instance,repeat) completed keys. SciCode denominators: 341 steps/rep (not 1120/3-rep arithmetic traps).