command-center/agent/managed-skills/aa-reference-scores-seeding/SKILL.md

name, description

name	description
aa-reference-scores-seeding	Seed/refresh frank-intelligence-bench config/references/aa-scores.yml from artificialanalysis.ai: authoritative top-3 sentences from eval pages, Recharts SVG zip extraction for per-model chart values, cross-check law, Elo normalization, schema/repair loop.

Seed frank-intelligence-bench config/references/aa-scores.yml from AA's public pages, then re-render the report trio.

Provenance law (non-negotiable)

• Every row: { eval, pct, source_url, fetched_at, index_version } (Zod-enforced; top-level schema_version: 1, fetched_at, index_version required).
• Eval ids MUST match report result ids: scicode, aa-lcr, hle, terminal-bench, gdp-pdf, aa-briefcase, gdpval-aa, automationbench-aa, aa-omniscience, critpt (render refFor(evalId) joins on exact id).
• Never ship an order-guessed chart↔eval mapping; only cross-checked ones.

Data tiers (authority order)

1. Eval-page sentence (authoritative, SSR, per eval page /evaluations/<slug>): "X scores the highest on <eval> with a score of N%, followed by Y ... and Z ..." → top-3 rows with verbatim model names. Regex trap: model names contain 5.5 — capture names with [^.;] truncates at decimals; use targeted extraction per page, verify each value against the page's 66.9-style raw text.
2. Model-page chart labels (per-model values, incl. the page's own model): charts are client-side Recharts (NOT in raw HTML, NOT via /api/ — keyless API 404s). Browser: browser.open({name,url,wait_until}), scroll full page in ~0.15 steps (lazy render), then tab.evaluate (NOT tab.run — that's Bun world):
   • names = <foreignObject> div text, x from parent g[transform] translate(x,...);
   • values = g.recharts-label-list text attrs;
   • zip by sorted-x index (nearest-x pairing is off by half a bar; index-zip verified against published tops).
3. sync-aa free Data API for headline indices — needs AA_API_KEY (unset in ~/.secrets as of 2026-10-11).

Cross-check law (mandatory before shipping chart-derived rows)

Chart's top-3 zipped values must match the eval page's published sentence (e.g. Kimi 88.7→89, Elo 1866→68.3, Opus index 46). Two independent methods disagreeing = stop, re-anchor; geometry pairing (link-top→svg-top proximity) collides on adjacent cards — prefer sentence-vs-zip corroboration.

Normalizations

• Elo evals (GDPval-AA, AA-Briefcase): sentences publish raw Elo; store AA's published (Elo−500)/2000×100 (1866→68.3). Document in file header.
• AA-Omniscience eval: aa-omniscience = the −100..100 Index (not accuracy); accuracy/non-hallucination charts corroborate (17%/7% ⇒ −60 arithmetically consistent).

Known identities (2026-10-11 census, v4.3.2)

• frank-archon lineage base = Qwen3.8-Flash-Next (/models/qwen3-8-flash-next), AA index 40, class median 18. NOT Qwen3-Next-80B-A3B-Instruct (index 10, 2025 family) — first seed got this wrong; verify lineage in docs/endpoints.md before labeling kind: base.
• Base per-eval (chart): SciCode 51, HLE 38, AA-LCR 80, TB-4.0 25, GDP.pdf 16, CritPt 11, Briefcase 54, GDPval 56, AutoBench 56, Omniscience −10.
• Frontier anchors: SciCode Opus 5.5 Max 66.9 (board flagged "under review" — travel this caveat with any SciCode delta); HLE Opus 61.4; AA-LCR Kimi K3 88.7; TB Sonnet 63.6; CritPt GPT-6.1 Sol 32.3.

Edit traps

• YAML surgery via edit: replacing a line range that included a top-level key (schema_version/models:) silently duplicates or deletes the anchor → Zod error expected 1 / received undefined. After any header edit, grep -n "^schema_version\|^fetched_at\|^index_version\|^models:" must show exactly one each.

Ship loop

1. bun -e 'import { loadAaFile, loadAaRefs } from "./src/aa"; ...' schema-parse gate.
2. bun src/cli.ts report <run-id> (takes bare id, NOT a path) → trio in reports/<host>/<id>/.
3. Verify data.json .aa_reference non-empty, per-eval coverage, delta columns.
4. log entry in docs/log.md + atomic commit + push (local gitea OK).
5. browser.close({name}) — headless tabs are shared/retained.