---
name: frank-bench-supervisor-recovery
description: "Recover a frank-intelligence-bench run after engine faults: detect poison grade rows, void them, restart supervisors with fixed code, judge cutover, watchdog for multi-day runs."
---

Recover frank-intelligence-bench runs on the dank-fleet after engine faults (poison graders, judge swaps, supervisor deaths).

## Laws (friction-earned 2026-10-10/11)
1. **A code fix is only real after the supervisor restarts.** Bun loads code at spawn; an on-disk fix never reaches the running process. Kill `_exec`, relaunch — takeover law makes resume cheap.
2. **A judge cohort is all-or-nothing.** Never mix judges on one eval's grade rows. Void ALL grade rows for LLM-judged evals (`phase == "grade"`), KEEP all `gen` rows (target output is judge-independent) — resume rejudges from cache.
3. **Poison census signatures:** grade rows with empty text + forced FAIL = engine, not model. `unparseable verdict` storm = judge output class (empty `content` + chain in `reasoning_content`; or 2048-tok truncation before verdict JSON). Default census sig `AssertionError` masked a poison shift — treat mass-default sigs as red flags.
4. **Detached launch form matters:** from a short-lived harness shell, `(setsid nohup cmd & )` (subshell) gets cgroup-reaped within minutes — silent death, stale `worker.pid`, no log line. Proven form: `setsid nohup cmd >> log.txt 2>&1 < /dev/null & disown` as the tail of the same shell (survived 6 h overnight).

## Void-and-resume procedure
```bash
pkill -f "cli.ts _exec"
python3 - <<'PYEOF'
import json, pathlib, shutil, time
led = pathlib.Path("runs/<RUN_ID>/ledger.jsonl")
rows = [l for l in led.read_text().splitlines() if l.strip()]
keep, void = [], []
for l in rows:
    r = json.loads(l)
    if r["eval"] in (<EVALS>) and r["phase"] == "grade" and r["ts"] > "<CUTOVER_ISO>":
        void.append(l)
    else:
        keep.append(l)
shutil.copy(led, f".AGENTTRASH/ledger-voided-{int(time.time())}.jsonl")
led.write_text("\n".join(keep) + "\n")
print(f"voided {len(void)}; kept {len(keep)}")
PYEOF
cd runs/<RUN_ID> && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown
```
- Judge cutover variant: void ALL `phase==grade` rows for the judged evals (no ts filter), edit `run.json` `judge` field, relaunch.
- Validate the judge endpoint first: live probe both directions (correct + incorrect candidate); a good judge returns bare verdict JSON fast.
- Resume proof = a specific known step flipping FAIL→PASS (or vice versa) in the log within minutes.

## Watchdog for multi-day runs
No cron on the Arch workstations — systemd user timer, idempotent (`cli.ts worker` exits instantly when supervisor heartbeat is fresh; takes over via pid-file 30 s freshness gate):
`~/.config/systemd/user/fib-watchdog.service`: Type=oneshot, WorkingDirectory=repo, `ExecStart=/usr/bin/bun src/cli.ts worker`, stdout/stderr `append:.../runs/watchdog.log`.
`fib-watchdog.timer`: OnBootSec=5min, OnUnitActiveSec=10min, Persistent=true, WantedBy=timers.target.
`systemctl --user enable --now fib-watchdog.timer`; verify `loginctl show-user <u> -p Linger` = yes (survives logout/reboot). Prove idempotence: log line "has a live supervisor — worker exits".

## Scoring truth after repair
Report renders from ledger with latest-wins via `ok` filter (no explicit dedup): ok=false rows drop, recovered key counts once. UI `done` counts unique `(eval,instance,repeat)` completed keys. SciCode denominators: 341 steps/rep (not 1120/3-rep arithmetic traps).