31 lines
2.9 KiB
Markdown
31 lines
2.9 KiB
Markdown
---
|
|
name: fleet-judge-swap-calibrate
|
|
description: "Swap the frank-intelligence-bench judge default and prove judge quality: fleet alias→weights map, known-truth calibration via scripts/judge-calibration.ts, per-backend reasoning-off dialect traps."
|
|
---
|
|
|
|
# Fleet judge swap + calibration (frank-intelligence-bench)
|
|
|
|
## Fleet alias → weights (verified 2026-10-10 via ps on dankclaw; endpoints only serve aliases)
|
|
- frank-archon = Qwen3.8-Flash-Next quant (radiance TP=2, dankserver:7770)
|
|
- frank-dragoon = qwen38-flash-next-v2-swift15-abliterated.hgn, 180B MoE (halogen container `frank-large-halogen`, dankclaw:7770) — check `podman inspect frank-large-halogen --format {{json .Config.Env}} | grep HALOGEN_CHECKPOINT`
|
|
- frank-zealot = Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K (llama.cpp, 3090, dankclaw:7771) — check `ps aux | grep <port>`
|
|
|
|
## Swap procedure
|
|
1. `config/hosts.yml`: target host gains `roles: [target, judge]`; set `judge_default`. No-self-judge law: `judge == target` throws at enqueue; rotation = next fleet model.
|
|
2. Judge load = AA-LCR + HLE only (SciCode/Terminal-Bench are unit-tested, judge-free).
|
|
3. Mid-run: `run.json` pins its judge (provenance); default swap affects future runs only.
|
|
4. Docs: `docs/endpoints.md` judge law section; commit with gates.
|
|
|
|
## Calibration — the ONLY accepted proof
|
|
`bun scripts/judge-calibration.ts [n] [judges...]` — known-truth floor test: gold answers must be judged CORRECT, deterministically mutated (first number +7) must be INCORRECT. Reports accuracy, mean s/verdict, projected AA-LCR/HLE judge hours. Safe beside a live run (judge-only traffic).
|
|
- Known-truth is a FLOOR test: catches broken judges, not lenient ones. Gold-vs-mutation is easy mode; small-model leniency on model-vs-gold edges needs real-verdict spot audits.
|
|
|
|
## Traps (all fired at least once)
|
|
- **False-accuracy via transport death:** a 400-dying judge scores ~50% on this test (all verdicts collapse to INCORRECT, matching the mutation half by luck). ALWAYS inspect `err` on misses before trusting any agreement/accuracy number. Fix: if all misses share one `err=HTTP 4xx`, it is the harness, not the judge.
|
|
- **Reasoning-off dialect split:** llama.cpp/radiance want `chat_template_kwargs:{thinking:false}`; halogen/vLLM-style servers 400 on `thinking` and honour `enable_thinking`. HostSpec field `reasoning_off: thinking|enable_thinking` (default `thinking`; dragoon needs `enable_thinking`). Wire-test via mock `bodies` capture.
|
|
- **No historical judged rows?** Quarantined ledgers may pre-date judge usage (census first: `failureCensus` rows/phase), then use known-truth synthesis instead of re-judging.
|
|
|
|
## Baselines measured (for future swaps)
|
|
- zealot (9B): 25/25 known-truth, 13 s/verdict → AA-LCR 1.1h, HLE 9.3h; leniency risk on hard edges (small judge + compliance tune).
|
|
- dragoon (180B): 2 s/verdict → AA-LCR 0.2h, HLE 1.4h; family-bias caveat when target shares its base family.
|