---
name: frank-bench-run-ops
description: "Operate frank-intelligence-bench runs on the dank fleet: status census, full void + fresh launch, judge cutover, supervisor restart law, watchdog timer, fault-autopsy snippets."
---

Runbook for /home/blank/Projects/frank-intelligence-bench (Bun/TS). Companion to frank-bench-supervisor-recovery (row-voiding for poison); this skill is the ops loop around it.

## Status census (run from repo root)
```bash
export RID=$(ls -t runs | grep -m1 frank-archon)   # live run id
tail -2 runs/$RID/log.txt
python3 - <<'PYEOF'   # latest-wins cohort census (mirrors score.ts semantics)
import json, os, collections
led = f"runs/{os.environ['RID']}/ledger.jsonl"
best = {}
for l in open(led):
    r = json.loads(l)
    if r["phase"] != "grade": continue
    k = (r["eval"], r["instance"], r["repeat"])
    if r["ok"] and (k not in best or r["ts"] > best[k]["ts"]): best[k] = r
    elif k not in best: best[k] = r
by = collections.defaultdict(lambda: [0,0])
for k,v in best.items(): by[k[0]][0]+=bool(v["grade"]["correct"]); by[k[0]][1]+=1
for e,(c,n) in sorted(by.items()): print(f"{e}: {c}/{n} = {100*c/n:.1f}%")
PYEOF
```
`pgrep -f "cli.ts _exec"` = supervisor; `systemctl --user is-active fib-watchdog.timer` = watchdog; last ledger ts older than ~5 min = suspect stall.

## Launch laws (each earned by an outage)
- Code fix + running supervisor = FIX IS NOT LIVE. Restart supervisor: `pkill -f "cli.ts _exec"`, then relaunch.
- Correct detached form from harness bash: `cd runs/$RID && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown`. NEVER the subshell form `( ... & )` — reaped by cgroup within minutes.
- Preferred launch = `bun src/cli.ts run HOST --judge JUDGE --detach` (in-process spawn + unref, proven 6 h+).
- Watchdog = systemd user timer `fib-watchdog.{service,timer}` (~/.config/systemd/user), 10 min, Persistent, runs `bun src/cli.ts worker` (idempotent: exits when heartbeat fresh, takes over stale runs via 30 s mtime gate).

## Void all results + fresh run (model/endpoint change)
1. `pkill -f "cli.ts _exec"`; sleep 1.
2. `mv runs/<id> .AGENTTRASH/run-voided-<reason>-$(date +%s)`; `mv reports/<model>/* .AGENTTRASH/`.
3. queue.json: set `runs: []` (python json edit).
4. New run: `bun src/cli.ts run frank-archon --judge frank-dragoon --detach`.
5. Log the void in docs/log.md (reason, quarantined scope) and commit — never silently erase scores.

## Judge cutover (zealot→dragoon pattern)
Void ONLY LLM-judged evals' `phase=grade` rows (keep ALL gen rows — target output is judge-independent); set `judge` in run.json; restart supervisor; resume rejudges automatically (grade keys incomplete). SciCode = pytest, judge-independent, never voids for judge changes.

## Fault-autopsy snippets
- Engine-fault census: grade rows with `ok:false` + empty text = engine, not model (signature-default trap: "AssertionError × N" with empty text = grades never executed).
- Runaway-chain detector (HLE loops): gen rows with `finish=="length"` AND completion_tokens==max AND empty text = reasoning degenerated into repetition; measure pre/post any quantization change to separate model behavior from engine artifacts.
- Census-by-eval: `collections.Counter((r["eval"], r["phase"], r["ok"]) for r in rows if r["ts"] > <cutover>)`.
- Verdict-parser bugs: probe the live judge on a minimal case (`judgeAalcr`/`judgeHle` via `bun -`), inspect raw head+tail; llama.cpp thinking models may return content:"" with everything in reasoning_content, and prose verdicts when their JSON scaffold breaks.

## Reporting etiquette (Frank's rules)
Quote score as: per-rep %, test avg@1 / avg@3, dev, index-lite; label provisional until all reps land; publish the runaway/parse-rate census next to any handicapped number. Dashboard :8787; `bun src/cli.ts report` renders the trio; `analyze` prints failure census.