command-center/agent/managed-skills/frank-bench-run-ops/SKILL.md

name, description

name	description
frank-bench-run-ops	Operate frank-intelligence-bench runs on the dank fleet: status census, full void + fresh launch, judge cutover, supervisor restart law, watchdog timer, fault-autopsy snippets.

Runbook for /home/blank/Projects/frank-intelligence-bench (Bun/TS). Companion to frank-bench-supervisor-recovery (row-voiding for poison); this skill is the ops loop around it.

Status census (run from repo root)

export RID=$(ls -t runs | grep -m1 frank-archon)   # live run id
tail -2 runs/$RID/log.txt
python3 - <<'PYEOF'   # latest-wins cohort census (mirrors score.ts semantics)
import json, os, collections
led = f"runs/{os.environ['RID']}/ledger.jsonl"
best = {}
for l in open(led):
    r = json.loads(l)
    if r["phase"] != "grade": continue
    k = (r["eval"], r["instance"], r["repeat"])
    if r["ok"] and (k not in best or r["ts"] > best[k]["ts"]): best[k] = r
    elif k not in best: best[k] = r
by = collections.defaultdict(lambda: [0,0])
for k,v in best.items(): by[k[0]][0]+=bool(v["grade"]["correct"]); by[k[0]][1]+=1
for e,(c,n) in sorted(by.items()): print(f"{e}: {c}/{n} = {100*c/n:.1f}%")
PYEOF

pgrep -f "cli.ts _exec" = supervisor; systemctl --user is-active fib-watchdog.timer = watchdog; last ledger ts older than ~5 min = suspect stall.

Launch laws (each earned by an outage)

• Code fix + running supervisor = FIX IS NOT LIVE. Restart supervisor: pkill -f "cli.ts _exec", then relaunch.
• Correct detached form from harness bash: cd runs/$RID && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown. NEVER the subshell form ( ... & ) — reaped by cgroup within minutes.
• Preferred launch = bun src/cli.ts run HOST --judge JUDGE --detach (in-process spawn + unref, proven 6 h+).
• Watchdog = systemd user timer fib-watchdog.{service,timer} (~/.config/systemd/user), 10 min, Persistent, runs bun src/cli.ts worker (idempotent: exits when heartbeat fresh, takes over stale runs via 30 s mtime gate).

Void all results + fresh run (model/endpoint change)

1. pkill -f "cli.ts _exec"; sleep 1.
2. mv runs/<id> .AGENTTRASH/run-voided-<reason>-$(date +%s); mv reports/<model>/* .AGENTTRASH/.
3. queue.json: set runs: [] (python json edit).
4. New run: bun src/cli.ts run frank-archon --judge frank-dragoon --detach.
5. Log the void in docs/log.md (reason, quarantined scope) and commit — never silently erase scores.

Judge cutover (zealot→dragoon pattern)

Void ONLY LLM-judged evals' phase=grade rows (keep ALL gen rows — target output is judge-independent); set judge in run.json; restart supervisor; resume rejudges automatically (grade keys incomplete). SciCode = pytest, judge-independent, never voids for judge changes.

Fault-autopsy snippets

• Engine-fault census: grade rows with ok:false + empty text = engine, not model (signature-default trap: "AssertionError × N" with empty text = grades never executed).
• Runaway-chain detector (HLE loops): gen rows with finish=="length" AND completion_tokens==max AND empty text = reasoning degenerated into repetition; measure pre/post any quantization change to separate model behavior from engine artifacts.
• Census-by-eval: collections.Counter((r["eval"], r["phase"], r["ok"]) for r in rows if r["ts"] > <cutover>).
• Verdict-parser bugs: probe the live judge on a minimal case (judgeAalcr/judgeHle via bun -), inspect raw head+tail; llama.cpp thinking models may return content:"" with everything in reasoning_content, and prose verdicts when their JSON scaffold breaks.

Reporting etiquette (Frank's rules)

Quote score as: per-rep %, test avg@1 / avg@3, dev, index-lite; label provisional until all reps land; publish the runaway/parse-rate census next to any handicapped number. Dashboard :8787; bun src/cli.ts report renders the trio; analyze prints failure census.