---
name: bench-harness-fault-census
description: Attribute benchmark failures to harness vs model via exception-signature census before trusting any score; python-grader fault signatures and fixes
---

# Rule (from 2026-10-10 frank-intelligence-bench incident, 3 harness faults found)
An LLM benchmark score is VOID until every failure signature is attributed to the MODEL. Run the census before quoting any number.

# Procedure
1. Aggregate graded rows per eval; for each failed grade extract the LAST python exception signature:
   regex `(\w+Error|\w+Exception): (?:name '([^']+)' is not defined|No module named '([^']+)')?` → sig = `Type: detail`.
2. Indict the harness (not the model) when:
   - `NameError: target` / `NameError: targets` in any count → expected-value bindings never reached the tests (gold data file missing/unstaged).
   - `ModuleNotFoundError: <grader package>` (e.g. `scicode`) → grading deps missing from the execution image.
   - ≥10 failures sharing ONE signature at >50% share → systematic fault; audit the protocol (e.g. markdown fences reaching an exec-based extractor — `SyntaxError` storm; strip ``` fences before feeding model text to graders that exec raw code).
   - >50% of generations fail → endpoint/transport fault.
3. Real model failures look heterogeneous: AssertionError/AxisError/NameError of the model's OWN undefined names.

# Companion checks
- Prove the grading path with a gold-solution integration test BEFORE trusting live scores: gold must PASS end-to-end (FIB_INTEGRATION=1 pattern).
- Quarantine poisoned ledgers (.AGENTTRASH) rather than deleting; census them to root-cause.
- Expect-score sanity: mixed PASS/FAIL with model-side signatures = real signal, even if the % looks low.