command-center/agent/managed-skills/llm-bench-harness-fault-census/SKILL.md

name, description

name	description
llm-bench-harness-fault-census	Diagnose implausible LLM benchmark scores in frank-intelligence-bench: failure-signature census, harness-fault signatures (target NameError, scicode ModuleNotFound, markdown-fence SyntaxError), and official-protocol grading law.

Use when benchmark scores look impossible (0% across the board, sudden cliffs) in frank-intelligence-bench.

Law

Scores are untrusted until bun run src/cli.ts analyze <run-id> shows a failure-signature census with ZERO harness suspects. Model-side signatures (AssertionError on real values, AxisError, model's own NameErrors) are fine; harness signatures indict the engine.

Census

• Reads runs/<id>/ledger.jsonl grade rows, extracts last *Error: name 'X' / No module named 'X' per failing row, aggregates per eval.
• harnessSuspects() flags: NameError: target(s) (gold target binding missing), ModuleNotFoundError: scicode (pinned pkg absent from sandbox image), >50% shared signature ≥10 samples (systematic fault), >50% gen failures (endpoint fault).

Known harness faults (friction-earned, 2026-10-10)

1. SciCode gold values live ONLY in test_data.h5 (authors' Drive folder 1W5GZW6_bdiDAiipuFMqdUhvUaHIj6-pR); tests reference bare target. Fix = stageH5() (sha256-gated upload to Dankclaw, mounted /w-data/test_data.h5:ro) + process_hdf5_to_tuple(step, n_cases, h5py_file=...) per official eval/scripts/test_generated_code.py.
2. Sandbox image must contain h5py + scicode @ git+https://github.com/scicode-bench/SciCode.git@<pin> (+ git in python:3.12-slim for pip-git). Image tag carries the fix epoch: frank-bench-py:3.12-scicode1.
3. Official extractor get_function_from_code execs raw submission text → markdown fences are guaranteed SyntaxError. Official pipelines store CLEAN code: strip fences (stripFences(): join all ``` blocks) before extraction, for submissions AND prior-step context.
4. Gold CODE ships only for the dev split (50 steps) on HF AI-SCODE/SciCode; problems_test.jsonl gold=None; GitHub pinned tree ships no gold jsonl; scicode pip package ships no data. Leaderboard (AA) SciCode uses reference-solution context — compare only against the gold-context slice, never the self-cumulative test split.
5. Quarantine poisoned ledgers to .AGENTTRASH/ (never blend void rows into reports), reset runs/queue.json to {"active": null, "runs": []} (active must be null, not "").

Verification gates

• Real-infra proof: FIB_INTEGRATION=1 bun test tests/integration.test.ts (gold solution must PASS official protocol; process_hdf5_to_tuple("77.1", 3) → 3 targets). Integration files must delete process.env.DRY_RUN and point FIB_DATA_CACHE at the real cache (preload isolates it to TMP fixtures).
• After a protocol fix, fresh live run must show mixed PASS/FAIL with model-side-only signatures before any score is reported.