53 lines
3.8 KiB
Markdown
53 lines
3.8 KiB
Markdown
---
|
||
name: frank-bench-run-ops
|
||
description: "Operate frank-intelligence-bench runs on the dank fleet: status census, full void + fresh launch, judge cutover, supervisor restart law, watchdog timer, fault-autopsy snippets."
|
||
---
|
||
|
||
Runbook for /home/blank/Projects/frank-intelligence-bench (Bun/TS). Companion to frank-bench-supervisor-recovery (row-voiding for poison); this skill is the ops loop around it.
|
||
|
||
## Status census (run from repo root)
|
||
```bash
|
||
export RID=$(ls -t runs | grep -m1 frank-archon) # live run id
|
||
tail -2 runs/$RID/log.txt
|
||
python3 - <<'PYEOF' # latest-wins cohort census (mirrors score.ts semantics)
|
||
import json, os, collections
|
||
led = f"runs/{os.environ['RID']}/ledger.jsonl"
|
||
best = {}
|
||
for l in open(led):
|
||
r = json.loads(l)
|
||
if r["phase"] != "grade": continue
|
||
k = (r["eval"], r["instance"], r["repeat"])
|
||
if r["ok"] and (k not in best or r["ts"] > best[k]["ts"]): best[k] = r
|
||
elif k not in best: best[k] = r
|
||
by = collections.defaultdict(lambda: [0,0])
|
||
for k,v in best.items(): by[k[0]][0]+=bool(v["grade"]["correct"]); by[k[0]][1]+=1
|
||
for e,(c,n) in sorted(by.items()): print(f"{e}: {c}/{n} = {100*c/n:.1f}%")
|
||
PYEOF
|
||
```
|
||
`pgrep -f "cli.ts _exec"` = supervisor; `systemctl --user is-active fib-watchdog.timer` = watchdog; last ledger ts older than ~5 min = suspect stall.
|
||
|
||
## Launch laws (each earned by an outage)
|
||
- Code fix + running supervisor = FIX IS NOT LIVE. Restart supervisor: `pkill -f "cli.ts _exec"`, then relaunch.
|
||
- Correct detached form from harness bash: `cd runs/$RID && setsid nohup bun run ../../src/cli.ts _exec run.json >> log.txt 2>&1 < /dev/null & disown`. NEVER the subshell form `( ... & )` — reaped by cgroup within minutes.
|
||
- Preferred launch = `bun src/cli.ts run HOST --judge JUDGE --detach` (in-process spawn + unref, proven 6 h+).
|
||
- Watchdog = systemd user timer `fib-watchdog.{service,timer}` (~/.config/systemd/user), 10 min, Persistent, runs `bun src/cli.ts worker` (idempotent: exits when heartbeat fresh, takes over stale runs via 30 s mtime gate).
|
||
|
||
## Void all results + fresh run (model/endpoint change)
|
||
1. `pkill -f "cli.ts _exec"`; sleep 1.
|
||
2. `mv runs/<id> .AGENTTRASH/run-voided-<reason>-$(date +%s)`; `mv reports/<model>/* .AGENTTRASH/`.
|
||
3. queue.json: set `runs: []` (python json edit).
|
||
4. New run: `bun src/cli.ts run frank-archon --judge frank-dragoon --detach`.
|
||
5. Log the void in docs/log.md (reason, quarantined scope) and commit — never silently erase scores.
|
||
|
||
## Judge cutover (zealot→dragoon pattern)
|
||
Void ONLY LLM-judged evals' `phase=grade` rows (keep ALL gen rows — target output is judge-independent); set `judge` in run.json; restart supervisor; resume rejudges automatically (grade keys incomplete). SciCode = pytest, judge-independent, never voids for judge changes.
|
||
|
||
## Fault-autopsy snippets
|
||
- Engine-fault census: grade rows with `ok:false` + empty text = engine, not model (signature-default trap: "AssertionError × N" with empty text = grades never executed).
|
||
- Runaway-chain detector (HLE loops): gen rows with `finish=="length"` AND completion_tokens==max AND empty text = reasoning degenerated into repetition; measure pre/post any quantization change to separate model behavior from engine artifacts.
|
||
- Census-by-eval: `collections.Counter((r["eval"], r["phase"], r["ok"]) for r in rows if r["ts"] > <cutover>)`.
|
||
- Verdict-parser bugs: probe the live judge on a minimal case (`judgeAalcr`/`judgeHle` via `bun -`), inspect raw head+tail; llama.cpp thinking models may return content:"" with everything in reasoning_content, and prose verdicts when their JSON scaffold breaks.
|
||
|
||
## Reporting etiquette (Frank's rules)
|
||
Quote score as: per-rep %, test avg@1 / avg@3, dev, index-lite; label provisional until all reps land; publish the runaway/parse-rate census next to any handicapped number. Dashboard :8787; `bun src/cli.ts report` renders the trio; `analyze` prints failure census.
|