command-center/agent/managed-skills/bench-harness-fault-census/SKILL.md

name, description

name	description
bench-harness-fault-census	Attribute benchmark failures to harness vs model via exception-signature census before trusting any score; python-grader fault signatures and fixes

Rule (from 2026-10-10 frank-intelligence-bench incident, 3 harness faults found)

An LLM benchmark score is VOID until every failure signature is attributed to the MODEL. Run the census before quoting any number.

Procedure

1. Aggregate graded rows per eval; for each failed grade extract the LAST python exception signature: regex (\w+Error|\w+Exception): (?:name '([^']+)' is not defined|No module named '([^']+)')? → sig = Type: detail.
2. Indict the harness (not the model) when:
   • NameError: target / NameError: targets in any count → expected-value bindings never reached the tests (gold data file missing/unstaged).
   • ModuleNotFoundError: <grader package> (e.g. scicode) → grading deps missing from the execution image.
   • ≥10 failures sharing ONE signature at >50% share → systematic fault; audit the protocol (e.g. markdown fences reaching an exec-based extractor — SyntaxError storm; strip ``` fences before feeding model text to graders that exec raw code).

   • 50% of generations fail → endpoint/transport fault.

3. Real model failures look heterogeneous: AssertionError/AxisError/NameError of the model's OWN undefined names.

Companion checks

• Prove the grading path with a gold-solution integration test BEFORE trusting live scores: gold must PASS end-to-end (FIB_INTEGRATION=1 pattern).
• Quarantine poisoned ledgers (.AGENTTRASH) rather than deleting; census them to root-cause.
• Expect-score sanity: mixed PASS/FAIL with model-side signatures = real signal, even if the % looks low.