command-center/agent/managed-skills/scicode-official-grading-protocol/SKILL.md

name, description

name	description
scicode-official-grading-protocol	Grade SciCode in frank-intelligence-bench the official way: scicode pkg pin, test_data.h5 targets, fence stripping, dev/test split semantics, harness-fault census.

Grade SciCode in frank-intelligence-bench the official way.

Protocol law (mirrors eval/scripts/test_generated_code.py @ e3158ea011d4235245a547460d3688d7ccbf9900)

• Submitted function extracted via scicode.parse.parse.extract_function_name + get_function_from_code inside the Dankclaw sandbox image frank-bench-py:3.12-scicode1 (python:3.12-slim + git + numpy/scipy/pandas/matplotlib/h5py + scicode @ git+...@pin; git is REQUIRED for pip git+https on slim).
• Gold expected values live ONLY in test_data.h5 (1.05 GB, authors' Drive folder 1W5GZW6_bdiDAiipuFMqdUhvUaHIj6-pR, fetch with uvx --with gdown gdown --folder <url> -O <dir> — lands in a SciCode_test/ subdir, move up). Bind per test case: targets = process_hdf5_to_tuple(step_number, len(cases), h5py_file="/w-data/test_data.h5"), then ns["target"] = targets[i] before exec'ing each case. Mount read-only at /w-data.
• STRIP MARKDOWN FENCES before extraction. The official pipeline stores clean code; the grader execs text verbatim, so "```python" lines = guaranteed SyntaxError = 100% false FAIL (live census caught 7/7). Apply to submissions AND prior-step texts.
• Splits: dev (15 problems / 50 steps, ships ground_truth_code + self-contained tests with inline target = ...) grades with REFERENCE-solution context; test split (65 problems / 288 steps, gold is None, tests reference h5 target/refN) is SELF-CUMULATIVE (model's own prior generations, inspect-ai convention). Scope = 338 steps = AA design count. Loader reads problems_test.jsonl + problems_dev.jsonl from HF SciCode1/SciCode.

Harness-fault census (bench analyze)

Before trusting any SciCode score, run the failure-signature census. Indict the harness (not the model) when:

• NameError: name 'target' is not defined — h5 targets not reaching tests (never staged / not mounted / bare-target fixtures).
• ModuleNotFoundError: No module named 'scicode' — sandbox image lacks the pinned package.
• 100% shared signature across ≥10 failures — systematic fault, audit protocol first. Historic incident 2026-10-10: first live run "0% pass" = 251× target-NameError + 36× scicode-module: all harness faults.

Sandbox ops

• Stage h5 with sha256 compare (ssh sha256sum vs local) before transfer; stream via ssh 'cat > file'.
• DRY sandbox must emit SCICODE_PASS (DRY_RUN) so DRY grading passes.
• podman short-names: an image tag that was never built falls through to registry lookup (exit 125) — ensure the tag exists (build) before podman run; tests that call execPython directly must ensureSandbox() first.