23 lines
1.7 KiB
Markdown
23 lines
1.7 KiB
Markdown
---
|
|
name: bench-harness-fault-census
|
|
description: Attribute benchmark failures to harness vs model via exception-signature census before trusting any score; python-grader fault signatures and fixes
|
|
---
|
|
|
|
# Rule (from 2026-10-10 frank-intelligence-bench incident, 3 harness faults found)
|
|
An LLM benchmark score is VOID until every failure signature is attributed to the MODEL. Run the census before quoting any number.
|
|
|
|
# Procedure
|
|
1. Aggregate graded rows per eval; for each failed grade extract the LAST python exception signature:
|
|
regex `(\w+Error|\w+Exception): (?:name '([^']+)' is not defined|No module named '([^']+)')?` → sig = `Type: detail`.
|
|
2. Indict the harness (not the model) when:
|
|
- `NameError: target` / `NameError: targets` in any count → expected-value bindings never reached the tests (gold data file missing/unstaged).
|
|
- `ModuleNotFoundError: <grader package>` (e.g. `scicode`) → grading deps missing from the execution image.
|
|
- ≥10 failures sharing ONE signature at >50% share → systematic fault; audit the protocol (e.g. markdown fences reaching an exec-based extractor — `SyntaxError` storm; strip ``` fences before feeding model text to graders that exec raw code).
|
|
- >50% of generations fail → endpoint/transport fault.
|
|
3. Real model failures look heterogeneous: AssertionError/AxisError/NameError of the model's OWN undefined names.
|
|
|
|
# Companion checks
|
|
- Prove the grading path with a gold-solution integration test BEFORE trusting live scores: gold must PASS end-to-end (FIB_INTEGRATION=1 pattern).
|
|
- Quarantine poisoned ledgers (.AGENTTRASH) rather than deleting; census them to root-cause.
|
|
- Expect-score sanity: mixed PASS/FAIL with model-side signatures = real signal, even if the % looks low.
|