25 lines
2.8 KiB
Markdown
25 lines
2.8 KiB
Markdown
---
|
||
name: scicode-official-grading-protocol
|
||
description: "Grade SciCode in frank-intelligence-bench the official way: scicode pkg pin, test_data.h5 targets, fence stripping, dev/test split semantics, harness-fault census."
|
||
---
|
||
|
||
Grade SciCode in frank-intelligence-bench the official way.
|
||
|
||
## Protocol law (mirrors eval/scripts/test_generated_code.py @ e3158ea011d4235245a547460d3688d7ccbf9900)
|
||
- Submitted function extracted via `scicode.parse.parse.extract_function_name` + `get_function_from_code` inside the Dankclaw sandbox image `frank-bench-py:3.12-scicode1` (python:3.12-slim + git + numpy/scipy/pandas/matplotlib/h5py + `scicode @ git+...@pin`; git is REQUIRED for pip git+https on slim).
|
||
- Gold expected values live ONLY in `test_data.h5` (1.05 GB, authors' Drive folder `1W5GZW6_bdiDAiipuFMqdUhvUaHIj6-pR`, fetch with `uvx --with gdown gdown --folder <url> -O <dir>` — lands in a `SciCode_test/` subdir, move up). Bind per test case: `targets = process_hdf5_to_tuple(step_number, len(cases), h5py_file="/w-data/test_data.h5")`, then `ns["target"] = targets[i]` before exec'ing each case. Mount read-only at `/w-data`.
|
||
- STRIP MARKDOWN FENCES before extraction. The official pipeline stores clean code; the grader `exec`s text verbatim, so "```python" lines = guaranteed SyntaxError = 100% false FAIL (live census caught 7/7). Apply to submissions AND prior-step texts.
|
||
- Splits: dev (15 problems / 50 steps, ships `ground_truth_code` + self-contained tests with inline `target = ...`) grades with REFERENCE-solution context; test split (65 problems / 288 steps, gold is None, tests reference h5 `target`/`refN`) is SELF-CUMULATIVE (model's own prior generations, inspect-ai convention). Scope = 338 steps = AA design count. Loader reads `problems_test.jsonl` + `problems_dev.jsonl` from HF `SciCode1/SciCode`.
|
||
|
||
## Harness-fault census (bench analyze)
|
||
Before trusting any SciCode score, run the failure-signature census. Indict the harness (not the model) when:
|
||
- `NameError: name 'target' is not defined` — h5 targets not reaching tests (never staged / not mounted / bare-target fixtures).
|
||
- `ModuleNotFoundError: No module named 'scicode'` — sandbox image lacks the pinned package.
|
||
- 100% shared signature across ≥10 failures — systematic fault, audit protocol first.
|
||
Historic incident 2026-10-10: first live run "0% pass" = 251× target-NameError + 36× scicode-module: all harness faults.
|
||
|
||
## Sandbox ops
|
||
- Stage h5 with sha256 compare (`ssh sha256sum` vs local) before transfer; stream via `ssh 'cat > file'`.
|
||
- DRY sandbox must emit `SCICODE_PASS (DRY_RUN)` so DRY grading passes.
|
||
- podman short-names: an image tag that was never built falls through to registry lookup (exit 125) — ensure the tag exists (build) before `podman run`; tests that call `execPython` directly must `ensureSandbox()` first.
|