46 lines
3.5 KiB
Markdown
46 lines
3.5 KiB
Markdown
---
|
|
name: frank-bench-supervisor-recovery
|
|
description: "Recover a frank-intelligence-bench run after engine faults: detect poison grade rows, void them, restart supervisors with fixed code, judge cutover, watchdog for multi-day runs."
|
|
---
|
|
|
|
Recover frank-intelligence-bench runs on the dank-fleet after engine faults (poison graders, judge swaps, supervisor deaths).
|
|
|
|
## Laws (friction-earned 2026-10-10/11)
|
|
1. **A code fix is only real after the supervisor restarts.** Bun loads code at spawn; an on-disk fix never reaches the running process. Kill `_exec`, relaunch — takeover law makes resume cheap.
|
|
2. **A judge cohort is all-or-nothing.** Never mix judges on one eval's grade rows. Void ALL grade rows for LLM-judged evals (`phase == "grade"`), KEEP all `gen` rows (target output is judge-independent) — resume rejudges from cache.
|
|
3. **Poison census signatures:** grade rows with empty text + forced FAIL = engine, not model. `unparseable verdict` storm = judge output class (empty `content` + chain in `reasoning_content`; or 2048-tok truncation before verdict JSON). Default census sig `AssertionError` masked a poison shift — treat mass-default sigs as red flags.
|
|
4. **Detached launch form matters:** from a short-lived harness shell, `(setsid nohup cmd & )` (subshell) gets cgroup-reaped within minutes — silent death, stale `worker.pid`, no log line. Proven form: `setsid nohup cmd >> log.txt 2>&1 < /dev/null & disown` as the tail of the same shell (survived 6 h overnight).
|
|
|
|
## Void-and-resume procedure
|
|
```bash
|
|
pkill -f "cli.ts _exec"
|
|
python3 - <<'PYEOF'
|
|
import json, pathlib, shutil, time
|
|
led = pathlib.Path("runs/<RUN_ID>/ledger.jsonl")
|
|
rows = [l for l in led.read_text().splitlines() if l.strip()]
|
|
keep, void = [], []
|
|
for l in rows:
|
|
r = json.loads(l)
|
|
if r["eval"] in (<EVALS>) and r["phase"] == "grade" and r["ts"] > "<CUTOVER_ISO>":
|
|
void.append(l)
|
|
else:
|
|
keep.append(l)
|
|
shutil.copy(led, f".AGENTTRASH/ledger-voided-{int(time.time())}.jsonl")
|
|
led.write_text("\n".join(keep) + "\n")
|
|
print(f"voided {len(void)}; kept {len(keep)}")
|
|
PYEOF
|
|
cd runs/<RUN_ID> && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown
|
|
```
|
|
- Judge cutover variant: void ALL `phase==grade` rows for the judged evals (no ts filter), edit `run.json` `judge` field, relaunch.
|
|
- Validate the judge endpoint first: live probe both directions (correct + incorrect candidate); a good judge returns bare verdict JSON fast.
|
|
- Resume proof = a specific known step flipping FAIL→PASS (or vice versa) in the log within minutes.
|
|
|
|
## Watchdog for multi-day runs
|
|
No cron on the Arch workstations — systemd user timer, idempotent (`cli.ts worker` exits instantly when supervisor heartbeat is fresh; takes over via pid-file 30 s freshness gate):
|
|
`~/.config/systemd/user/fib-watchdog.service`: Type=oneshot, WorkingDirectory=repo, `ExecStart=/usr/bin/bun src/cli.ts worker`, stdout/stderr `append:.../runs/watchdog.log`.
|
|
`fib-watchdog.timer`: OnBootSec=5min, OnUnitActiveSec=10min, Persistent=true, WantedBy=timers.target.
|
|
`systemctl --user enable --now fib-watchdog.timer`; verify `loginctl show-user <u> -p Linger` = yes (survives logout/reboot). Prove idempotence: log line "has a live supervisor — worker exits".
|
|
|
|
## Scoring truth after repair
|
|
Report renders from ledger with latest-wins via `ok` filter (no explicit dedup): ok=false rows drop, recovered key counts once. UI `done` counts unique `(eval,instance,repeat)` completed keys. SciCode denominators: 341 steps/rep (not 1120/3-rep arithmetic traps).
|