live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/frank-bench-run-ops/SKILL.md

@@ -0,0 +1,52 @@
+---
+name: frank-bench-run-ops
+description: "Operate frank-intelligence-bench runs on the dank fleet: status census, full void + fresh launch, judge cutover, supervisor restart law, watchdog timer, fault-autopsy snippets."
+---
+
+Runbook for /home/blank/Projects/frank-intelligence-bench (Bun/TS). Companion to frank-bench-supervisor-recovery (row-voiding for poison); this skill is the ops loop around it.
+
+## Status census (run from repo root)
+```bash
+export RID=$(ls -t runs | grep -m1 frank-archon)   # live run id
+tail -2 runs/$RID/log.txt
+python3 - <<'PYEOF'   # latest-wins cohort census (mirrors score.ts semantics)
+import json, os, collections
+led = f"runs/{os.environ['RID']}/ledger.jsonl"
+best = {}
+for l in open(led):
+    r = json.loads(l)
+    if r["phase"] != "grade": continue
+    k = (r["eval"], r["instance"], r["repeat"])
+    if r["ok"] and (k not in best or r["ts"] > best[k]["ts"]): best[k] = r
+    elif k not in best: best[k] = r
+by = collections.defaultdict(lambda: [0,0])
+for k,v in best.items(): by[k[0]][0]+=bool(v["grade"]["correct"]); by[k[0]][1]+=1
+for e,(c,n) in sorted(by.items()): print(f"{e}: {c}/{n} = {100*c/n:.1f}%")
+PYEOF
+```
+`pgrep -f "cli.ts _exec"` = supervisor; `systemctl --user is-active fib-watchdog.timer` = watchdog; last ledger ts older than ~5 min = suspect stall.
+
+## Launch laws (each earned by an outage)
+- Code fix + running supervisor = FIX IS NOT LIVE. Restart supervisor: `pkill -f "cli.ts _exec"`, then relaunch.
+- Correct detached form from harness bash: `cd runs/$RID && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown`. NEVER the subshell form `( ... & )` — reaped by cgroup within minutes.
+- Preferred launch = `bun src/cli.ts run HOST --judge JUDGE --detach` (in-process spawn + unref, proven 6 h+).
+- Watchdog = systemd user timer `fib-watchdog.{service,timer}` (~/.config/systemd/user), 10 min, Persistent, runs `bun src/cli.ts worker` (idempotent: exits when heartbeat fresh, takes over stale runs via 30 s mtime gate).
+
+## Void all results + fresh run (model/endpoint change)
+1. `pkill -f "cli.ts _exec"`; sleep 1.
+2. `mv runs/<id> .AGENTTRASH/run-voided-<reason>-$(date +%s)`; `mv reports/<model>/* .AGENTTRASH/`.
+3. queue.json: set `runs: []` (python json edit).
+4. New run: `bun src/cli.ts run frank-archon --judge frank-dragoon --detach`.
+5. Log the void in docs/log.md (reason, quarantined scope) and commit — never silently erase scores.
+
+## Judge cutover (zealot→dragoon pattern)
+Void ONLY LLM-judged evals' `phase=grade` rows (keep ALL gen rows — target output is judge-independent); set `judge` in run.json; restart supervisor; resume rejudges automatically (grade keys incomplete). SciCode = pytest, judge-independent, never voids for judge changes.
+
+## Fault-autopsy snippets
+- Engine-fault census: grade rows with `ok:false` + empty text = engine, not model (signature-default trap: "AssertionError × N" with empty text = grades never executed).
+- Runaway-chain detector (HLE loops): gen rows with `finish=="length"` AND completion_tokens==max AND empty text = reasoning degenerated into repetition; measure pre/post any quantization change to separate model behavior from engine artifacts.
+- Census-by-eval: `collections.Counter((r["eval"], r["phase"], r["ok"]) for r in rows if r["ts"] > <cutover>)`.
+- Verdict-parser bugs: probe the live judge on a minimal case (`judgeAalcr`/`judgeHle` via `bun -`), inspect raw head+tail; llama.cpp thinking models may return content:"" with everything in reasoning_content, and prose verdicts when their JSON scaffold breaks.
+
+## Reporting etiquette (Frank's rules)
+Quote score as: per-rep %, test avg@1 / avg@3, dev, index-lite; label provisional until all reps land; publish the runaway/parse-rate census next to any handicapped number. Dashboard :8787; `bun src/cli.ts report` renders the trio; `analyze` prints failure census.