25 lines
2.8 KiB
Markdown
25 lines
2.8 KiB
Markdown
---
|
|
name: llm-bench-harness-fault-census
|
|
description: "Diagnose implausible LLM benchmark scores in frank-intelligence-bench: failure-signature census, harness-fault signatures (target NameError, scicode ModuleNotFound, markdown-fence SyntaxError), and official-protocol grading law."
|
|
---
|
|
|
|
Use when benchmark scores look impossible (0% across the board, sudden cliffs) in frank-intelligence-bench.
|
|
|
|
# Law
|
|
Scores are untrusted until `bun run src/cli.ts analyze <run-id>` shows a failure-signature census with ZERO harness suspects. Model-side signatures (AssertionError on real values, AxisError, model's own NameErrors) are fine; harness signatures indict the engine.
|
|
|
|
# Census
|
|
- Reads `runs/<id>/ledger.jsonl` grade rows, extracts last `*Error: name 'X'` / `No module named 'X'` per failing row, aggregates per eval.
|
|
- `harnessSuspects()` flags: `NameError: target(s)` (gold target binding missing), `ModuleNotFoundError: scicode` (pinned pkg absent from sandbox image), >50% shared signature ≥10 samples (systematic fault), >50% gen failures (endpoint fault).
|
|
|
|
# Known harness faults (friction-earned, 2026-10-10)
|
|
1. SciCode gold values live ONLY in `test_data.h5` (authors' Drive folder `1W5GZW6_bdiDAiipuFMqdUhvUaHIj6-pR`); tests reference bare `target`. Fix = `stageH5()` (sha256-gated upload to Dankclaw, mounted `/w-data/test_data.h5:ro`) + `process_hdf5_to_tuple(step, n_cases, h5py_file=...)` per official `eval/scripts/test_generated_code.py`.
|
|
2. Sandbox image must contain `h5py` + `scicode @ git+https://github.com/scicode-bench/SciCode.git@<pin>` (+ `git` in python:3.12-slim for pip-git). Image tag carries the fix epoch: `frank-bench-py:3.12-scicode1`.
|
|
3. Official extractor `get_function_from_code` `exec`s raw submission text → markdown fences are guaranteed SyntaxError. Official pipelines store CLEAN code: strip fences (`stripFences()`: join all ``` blocks) before extraction, for submissions AND prior-step context.
|
|
4. Gold CODE ships only for the dev split (50 steps) on HF `AI-SCODE/SciCode`; `problems_test.jsonl` gold=None; GitHub pinned tree ships no gold jsonl; scicode pip package ships no data. Leaderboard (AA) SciCode uses reference-solution context — compare only against the gold-context slice, never the self-cumulative test split.
|
|
5. Quarantine poisoned ledgers to `.AGENTTRASH/` (never blend void rows into reports), reset `runs/queue.json` to `{"active": null, "runs": []}` (active must be `null`, not `""`).
|
|
|
|
# Verification gates
|
|
- Real-infra proof: `FIB_INTEGRATION=1 bun test tests/integration.test.ts` (gold solution must PASS official protocol; `process_hdf5_to_tuple("77.1", 3)` → 3 targets). Integration files must `delete process.env.DRY_RUN` and point `FIB_DATA_CACHE` at the real cache (preload isolates it to TMP fixtures).
|
|
- After a protocol fix, fresh live run must show mixed PASS/FAIL with model-side-only signatures before any score is reported.
|