live: heartbeat sync 2026-10-18T16:47:59-07:00 (Refs #11)

agent/managed-skills/judge-empty-content-parse-repair/SKILL.md

@@ -0,0 +1,26 @@
+---
+name: judge-empty-content-parse-repair
+description: "Diagnose and repair LLM-judge \"unparseable verdict\" storms: judge-fail rows store no raw → reproduce live on the failing instance; root causes include empty model response (runaway reasoning chains, finish=length + content:\"\") and judge prose-form verdicts when JSON scaffold breaks; fix parser fallback + prove via restart rejudge."
+---
+
+Use when LLM-judge grades show an elevated "unparseable verdict" rate (baseline ~1%; storm = ≥5-10%).
+
+## Law
+Judge rows with `ok:false` store NO raw judge output (meta empty) — you are blind in the ledger. NEVER guess the cause from the ledger alone; reproduce the exact failing instance live.
+
+## Procedure
+1. **Census the rate**: filter ledger grade rows since a timestamp; rate = bad/total. Compare against the eval's baseline (≈1%). HLE with a 180B bare-JSON judge showed 11.8%.
+2. **Pull the failing instance**: pick the newest failing grade row; get its `instance` id and the latest `gen` row for that instance (latest-wins; ledger is append-only).
+3. **Reproduce live**: one-shot `bun -` (or equivalent) script: load registry host by judge name, call the judge function with that instance's exact question/answer/response, print `raw` head+tail. Judge raw text is ground truth.
+4. **Root-cause A — empty model response**: if `resp_len: 0`, the judge is fine; the MODEL produced no answer content. Census gens: `text` empty + `finish=length` + `completion_tokens == max_tokens` ⇒ **runaway reasoning chain**: the model wrote the whole budget into `reasoning` and never emitted `content`. Loop proof: 60-char block Counter over the reasoning tail (49 identical blocks = degeneration, e.g. `w4 w? w4 w?…`).
+5. **Root-cause B — prose verdict**: judges (esp. bare-JSON 180B class) drop the JSON scaffold when judging degenerate inputs (empty response) and emit plain `correct: no\nconfidence: 100`. Parser knowing only quoted `"correct": "no"` forms reports null = "unparseable" while the verdict is perfectly valid. Fix = add prose fallback regex `(?:^|\n)\s*correct\s*:\s*(yes|no)\b` after the JSON/quoted attempts, never before (JSON must win).
+6. **Baseline comparison before blaming the engine change**: for any model-behavior rate (runaway %, empty-answer %), census the previous (voided/`.AGENTTRASH/`) ledger the same way. Pre/post identical (18.5% vs 18.0%) ⇒ pre-existing model behavior, not the quantization/config under investigation. This keeps causal claims honest.
+7. **Verify the fix live**: restart the supervisor (poison law); pending `ok:false` grade rows re-grade on resume — the previously-blind rows flipping to real `fail` verdicts in the log is the proof.
+
+## Empty-content scoring law
+Runaway-chain empties score 0 honestly (AA-style truncating harness). Keep the score, publish the runaway census beside every number. Sampling changes (repetition_penalty, num_predict cap) deviate from AA posture and void+restart the eval — user decision, not an assistant fix.
+
+## Pitfalls
+- `podman logs 2>/dev/null`-style stderr discards make failure rows text-empty — check the capture path before inventing model-side causes (sandbox.ts fixed to `2>&1`).
+- `export RID=...` for python heredocs (plain `RID=` is invisible to `os.environ`).
+- Never `cd x && setsid ... & disown` expecting the cd to persist: `&` backgrounds the whole compound; foreground cwd is unchanged.