command-center/agent/managed-skills/fleet-judge-swap-calibrate/SKILL.md

name, description

name	description
fleet-judge-swap-calibrate	Swap the frank-intelligence-bench judge default and prove judge quality: fleet alias→weights map, known-truth calibration via scripts/judge-calibration.ts, per-backend reasoning-off dialect traps.

Fleet judge swap + calibration (frank-intelligence-bench)

Fleet alias → weights (verified 2026-10-10 via ps on dankclaw; endpoints only serve aliases)

• frank-archon = Qwen3.8-Flash-Next quant (radiance TP=2, dankserver:7770)
• frank-dragoon = qwen38-flash-next-v2-swift15-abliterated.hgn, 180B MoE (halogen container frank-large-halogen, dankclaw:7770) — check podman inspect frank-large-halogen --format {{json .Config.Env}} | grep HALOGEN_CHECKPOINT
• frank-zealot = Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K (llama.cpp, 3090, dankclaw:7771) — check ps aux | grep <port>

Swap procedure

1. config/hosts.yml: target host gains roles: [target, judge]; set judge_default. No-self-judge law: judge == target throws at enqueue; rotation = next fleet model.
2. Judge load = AA-LCR + HLE only (SciCode/Terminal-Bench are unit-tested, judge-free).
3. Mid-run: run.json pins its judge (provenance); default swap affects future runs only.
4. Docs: docs/endpoints.md judge law section; commit with gates.

Calibration — the ONLY accepted proof

bun scripts/judge-calibration.ts [n] [judges...] — known-truth floor test: gold answers must be judged CORRECT, deterministically mutated (first number +7) must be INCORRECT. Reports accuracy, mean s/verdict, projected AA-LCR/HLE judge hours. Safe beside a live run (judge-only traffic).

• Known-truth is a FLOOR test: catches broken judges, not lenient ones. Gold-vs-mutation is easy mode; small-model leniency on model-vs-gold edges needs real-verdict spot audits.

Traps (all fired at least once)

• False-accuracy via transport death: a 400-dying judge scores ~50% on this test (all verdicts collapse to INCORRECT, matching the mutation half by luck). ALWAYS inspect err on misses before trusting any agreement/accuracy number. Fix: if all misses share one err=HTTP 4xx, it is the harness, not the judge.
• Reasoning-off dialect split: llama.cpp/radiance want chat_template_kwargs:{thinking:false}; halogen/vLLM-style servers 400 on thinking and honour enable_thinking. HostSpec field reasoning_off: thinking|enable_thinking (default thinking; dragoon needs enable_thinking). Wire-test via mock bodies capture.
• No historical judged rows? Quarantined ledgers may pre-date judge usage (census first: failureCensus rows/phase), then use known-truth synthesis instead of re-judging.

Baselines measured (for future swaps)

• zealot (9B): 25/25 known-truth, 13 s/verdict → AA-LCR 1.1h, HLE 9.3h; leniency risk on hard edges (small judge + compliance tune).
• dragoon (180B): 2 s/verdict → AA-LCR 0.2h, HLE 1.4h; family-bias caveat when target shares its base family.