live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/bench-harness-fault-census/SKILL.md

@@ -0,0 +1,22 @@
+---
+name: bench-harness-fault-census
+description: Attribute benchmark failures to harness vs model via exception-signature census before trusting any score; python-grader fault signatures and fixes
+---
+
+# Rule (from 2026-10-10 frank-intelligence-bench incident, 3 harness faults found)
+An LLM benchmark score is VOID until every failure signature is attributed to the MODEL. Run the census before quoting any number.
+
+# Procedure
+1. Aggregate graded rows per eval; for each failed grade extract the LAST python exception signature:
+   regex `(\w+Error|\w+Exception): (?:name '([^']+)' is not defined|No module named '([^']+)')?` → sig = `Type: detail`.
+2. Indict the harness (not the model) when:
+   - `NameError: target` / `NameError: targets` in any count → expected-value bindings never reached the tests (gold data file missing/unstaged).
+   - `ModuleNotFoundError: <grader package>` (e.g. `scicode`) → grading deps missing from the execution image.
+   - ≥10 failures sharing ONE signature at >50% share → systematic fault; audit the protocol (e.g. markdown fences reaching an exec-based extractor — `SyntaxError` storm; strip ``` fences before feeding model text to graders that exec raw code).
+   - >50% of generations fail → endpoint/transport fault.
+3. Real model failures look heterogeneous: AssertionError/AxisError/NameError of the model's OWN undefined names.
+
+# Companion checks
+- Prove the grading path with a gold-solution integration test BEFORE trusting live scores: gold must PASS end-to-end (FIB_INTEGRATION=1 pattern).
+- Quarantine poisoned ledgers (.AGENTTRASH) rather than deleting; census them to root-cause.
+- Expect-score sanity: mixed PASS/FAIL with model-side signatures = real signal, even if the % looks low.