command-center/agent/managed-skills/judge-empty-content-parse-repair/SKILL.md

name, description

name	description
judge-empty-content-parse-repair	Diagnose and repair LLM-judge "unparseable verdict" storms: judge-fail rows store no raw → reproduce live on the failing instance; root causes include empty model response (runaway reasoning chains, finish=length + content:"") and judge prose-form verdicts when JSON scaffold breaks; fix parser fallback + prove via restart rejudge.

Use when LLM-judge grades show an elevated "unparseable verdict" rate (baseline ~1%; storm = ≥5-10%).

Law

Judge rows with ok:false store NO raw judge output (meta empty) — you are blind in the ledger. NEVER guess the cause from the ledger alone; reproduce the exact failing instance live.

Procedure

1. Census the rate: filter ledger grade rows since a timestamp; rate = bad/total. Compare against the eval's baseline (≈1%). HLE with a 180B bare-JSON judge showed 11.8%.
2. Pull the failing instance: pick the newest failing grade row; get its instance id and the latest gen row for that instance (latest-wins; ledger is append-only).
3. Reproduce live: one-shot bun - (or equivalent) script: load registry host by judge name, call the judge function with that instance's exact question/answer/response, print raw head+tail. Judge raw text is ground truth.
4. Root-cause A — empty model response: if resp_len: 0, the judge is fine; the MODEL produced no answer content. Census gens: text empty + finish=length + completion_tokens == max_tokens ⇒ runaway reasoning chain: the model wrote the whole budget into reasoning and never emitted content. Loop proof: 60-char block Counter over the reasoning tail (49 identical blocks = degeneration, e.g. w4 w? w4 w?…).
5. Root-cause B — prose verdict: judges (esp. bare-JSON 180B class) drop the JSON scaffold when judging degenerate inputs (empty response) and emit plain correct: no\nconfidence: 100. Parser knowing only quoted "correct": "no" forms reports null = "unparseable" while the verdict is perfectly valid. Fix = add prose fallback regex (?:^|\n)\s*correct\s*:\s*(yes|no)\b after the JSON/quoted attempts, never before (JSON must win).
6. Baseline comparison before blaming the engine change: for any model-behavior rate (runaway %, empty-answer %), census the previous (voided/.AGENTTRASH/) ledger the same way. Pre/post identical (18.5% vs 18.0%) ⇒ pre-existing model behavior, not the quantization/config under investigation. This keeps causal claims honest.
7. Verify the fix live: restart the supervisor (poison law); pending ok:false grade rows re-grade on resume — the previously-blind rows flipping to real fail verdicts in the log is the proof.

Empty-content scoring law

Runaway-chain empties score 0 honestly (AA-style truncating harness). Keep the score, publish the runaway census beside every number. Sampling changes (repetition_penalty, num_predict cap) deviate from AA posture and void+restart the eval — user decision, not an assistant fix.

Pitfalls

• podman logs 2>/dev/null-style stderr discards make failure rows text-empty — check the capture path before inventing model-side causes (sandbox.ts fixed to 2>&1).
• export RID=... for python heredocs (plain RID= is invisible to os.environ).
• Never cd x && setsid ... & disown expecting the cd to persist: & backgrounds the whole compound; foreground cwd is unchanged.