25 lines
2.8 KiB
Markdown
25 lines
2.8 KiB
Markdown
---
|
||
name: aa-chart-model-score-extraction
|
||
description: "Extract per-model benchmark scores from artificialanalysis.ai pages: charts are client-rendered Recharts SVGs; zip label-list texts to foreignObject names by x-sorted index, validate chart identity against the page's published top-3 sentence, apply Elo/Omniscience normalization laws, seed frank-intelligence-bench aa-scores.yml."
|
||
---
|
||
|
||
Extract AI benchmark scores for a specific model from artificialanalysis.ai. Per-model per-eval numbers are NOT in SSR HTML (JS charts, API needs key); eval-page SSR text DOES carry the top-3 sentence.
|
||
|
||
## Procedure (validated 5x on this repo's model pages)
|
||
1. `browser.open({ name, url: "https://artificialanalysis.ai/models/<slug>", wait_until: "domcontentloaded" })`, then scroll in ~6 steps (0.15..0.9 of scrollHeight, ~1.3 s pauses) — charts lazy-render.
|
||
2. Extract each chart: pair `g.recharts-label-list text` (values, take `x` attr) with `foreignObject div` (model names, x from parent `g[transform] translate(x,y)`). **Sort both lists by x and zip by index — NOT nearest-x** (nearest-x is off by one bar; index-zip matches published tops exactly).
|
||
3. Self row: match name loosely (`Qwen3.8-Flash-Next` has hyphens — test both forms).
|
||
|
||
## Identity law (never ship an order-guessed chart→eval mapping)
|
||
- Model pages have ~14 charts, 18 eval cards; DOM order and geometry pairing BOTH proved ambiguous.
|
||
- Anchor every chart by its top-3 names/values vs the eval page's published sentence: `https://artificialanalysis.ai/evaluations/<slug>` body text: "X scores the highest on <eval> with a score of N%, followed by …". Regex must tolerate decimals: value pattern `(-?[\\d.]+%?)`; build page scripts with `String.raw` (bun template literals eat `\\d`).
|
||
- A chart's `self` value on the MODEL page is page-local → trustworthy once the top-3 anchor confirms the eval (e.g. AA-LCR: Kimi 88.7 → chart top "Kimi K3 (max)=89").
|
||
|
||
## Normalization laws (frank-intelligence-bench aa-scores.yml)
|
||
- Elo evals (AA-Briefcase, GDPval-AA): store `(Elo−500)/2000×100` (1866 → 68.3), AA's published normalization.
|
||
- AA-Omniscience: score = Accuracy (w10) + 1−Hallucination (w5), both 0–100 → composite `(10·acc + 5·nonhall)/15`. NEVER feed the −100..100 Index chart value (pre-fix cost ~7 pts; calibration then reproduces AA's official index ±0.6: Flash-Next 39.85≈40, Haiku 43.4≈43, 27B 33.85≈34, 3.6-27B 21.55≈21).
|
||
- Every seeded row: `source_url` + `fetched_at` + `index_version`.
|
||
|
||
## Seed + validate loop
|
||
Append model block to `config/references/aa-scores.yml` (block-insert via python string replace on the ` - name: <anchor>` line — beware: dropping rows from a block loses the 2-space indent on the anchor line; verify with `yaml.safe_load`), then `bun -e loadAaRefs` parse-check, `bun src/cli.ts report $RID`, calibration check (full-row models: lite vs AA official), `bun test`, commit.
|