42 lines
4.1 KiB
Markdown
42 lines
4.1 KiB
Markdown
---
|
||
name: aa-reference-scores-seeding
|
||
description: "Seed/refresh frank-intelligence-bench config/references/aa-scores.yml from artificialanalysis.ai: authoritative top-3 sentences from eval pages, Recharts SVG zip extraction for per-model chart values, cross-check law, Elo normalization, schema/repair loop."
|
||
---
|
||
|
||
Seed frank-intelligence-bench `config/references/aa-scores.yml` from AA's public pages, then re-render the report trio.
|
||
|
||
## Provenance law (non-negotiable)
|
||
- Every row: `{ eval, pct, source_url, fetched_at, index_version }` (Zod-enforced; top-level `schema_version: 1, fetched_at, index_version` required).
|
||
- Eval ids MUST match report result ids: `scicode, aa-lcr, hle, terminal-bench, gdp-pdf, aa-briefcase, gdpval-aa, automationbench-aa, aa-omniscience, critpt` (render `refFor(evalId)` joins on exact id).
|
||
- Never ship an order-guessed chart↔eval mapping; only cross-checked ones.
|
||
|
||
## Data tiers (authority order)
|
||
1. **Eval-page sentence** (authoritative, SSR, per eval page `/evaluations/<slug>`): `"X scores the highest on <eval> with a score of N%, followed by Y ... and Z ..."` → top-3 rows with verbatim model names. Regex trap: model names contain `5.5` — capture names with `[^.;]` truncates at decimals; use targeted extraction per page, verify each value against the page's `66.9`-style raw text.
|
||
2. **Model-page chart labels** (per-model values, incl. the page's own model): charts are client-side Recharts (NOT in raw HTML, NOT via /api/ — keyless API 404s). Browser: `browser.open({name,url,wait_until})`, scroll full page in ~0.15 steps (lazy render), then `tab.evaluate` (NOT `tab.run` — that's Bun world):
|
||
- names = `<foreignObject>` div text, x from parent `g[transform]` `translate(x,...)`;
|
||
- values = `g.recharts-label-list text` attrs;
|
||
- **zip by sorted-x index** (nearest-x pairing is off by half a bar; index-zip verified against published tops).
|
||
3. `sync-aa` free Data API for headline indices — needs `AA_API_KEY` (unset in ~/.secrets as of 2026-10-11).
|
||
|
||
## Cross-check law (mandatory before shipping chart-derived rows)
|
||
Chart's top-3 zipped values must match the eval page's published sentence (e.g. Kimi 88.7→89, Elo 1866→68.3, Opus index 46). Two independent methods disagreeing = stop, re-anchor; geometry pairing (link-top→svg-top proximity) collides on adjacent cards — prefer sentence-vs-zip corroboration.
|
||
|
||
## Normalizations
|
||
- Elo evals (GDPval-AA, AA-Briefcase): sentences publish raw Elo; store AA's published `(Elo−500)/2000`×100 (1866→68.3). Document in file header.
|
||
- AA-Omniscience `eval: aa-omniscience` = the −100..100 Index (not accuracy); accuracy/non-hallucination charts corroborate (17%/7% ⇒ −60 arithmetically consistent).
|
||
|
||
## Known identities (2026-10-11 census, v4.3.2)
|
||
- frank-archon lineage base = **Qwen3.8-Flash-Next** (`/models/qwen3-8-flash-next`), AA index 40, class median 18. NOT Qwen3-Next-80B-A3B-Instruct (index 10, 2025 family) — first seed got this wrong; verify lineage in `docs/endpoints.md` before labeling `kind: base`.
|
||
- Base per-eval (chart): SciCode 51, HLE 38, AA-LCR 80, TB-4.0 25, GDP.pdf 16, CritPt 11, Briefcase 54, GDPval 56, AutoBench 56, Omniscience −10.
|
||
- Frontier anchors: SciCode Opus 5.5 Max 66.9 (board flagged "under review" — travel this caveat with any SciCode delta); HLE Opus 61.4; AA-LCR Kimi K3 88.7; TB Sonnet 63.6; CritPt GPT-6.1 Sol 32.3.
|
||
|
||
## Edit traps
|
||
- YAML surgery via `edit`: replacing a line range that included a top-level key (`schema_version`/`models:`) silently duplicates or deletes the anchor → Zod error `expected 1 / received undefined`. After any header edit, `grep -n "^schema_version\|^fetched_at\|^index_version\|^models:"` must show exactly one each.
|
||
|
||
## Ship loop
|
||
1. `bun -e 'import { loadAaFile, loadAaRefs } from "./src/aa"; ...'` schema-parse gate.
|
||
2. `bun src/cli.ts report <run-id>` (takes bare id, NOT a path) → trio in `reports/<host>/<id>/`.
|
||
3. Verify `data.json .aa_reference` non-empty, per-eval coverage, delta columns.
|
||
4. log entry in `docs/log.md` + atomic commit + push (local gitea OK).
|
||
5. `browser.close({name})` — headless tabs are shared/retained.
|