25 lines
2.7 KiB
Markdown
25 lines
2.7 KiB
Markdown
---
|
||
name: aa-chart-label-browser-extraction
|
||
description: "Extract per-model benchmark scores from artificialanalysis.ai pages: charts are client-rendered recharts SVGs invisible to plain HTML fetch; pair sorted label/name x-coords, then cross-check against published sentences before trusting any chart↔eval identity."
|
||
---
|
||
|
||
Extract per-eval scores for a model from artificialanalysis.ai when no API key exists. Plain HTML fetch yields NO chart data (client-side recharts; SSR has only meta + FAQ sentences; `/api/*` = 401).
|
||
|
||
## Procedure
|
||
1. Managed browser tab: `browser.open({name, url, wait_until: "domcontentloaded"})`.
|
||
2. **Per-eval top-3 (authoritative):** open `https://artificialanalysis.ai/evaluations/<slug>`; regex `document.body.innerText` (collapse whitespace): `([A-Z][^.;]{0,60}?) scores the highest on (.+?) with a score of (-?[\d.]+%?), followed by (.+?) with a score of ... and ...` — mind decimal points truncating naive `[^.]+` regexes. Build the page script as a plain string (`String.raw`), not an interpolated template, so `\\d` survives.
|
||
3. **All-eval per-model rows:** on `https://artificialanalysis.ai/models/<slug>`: scroll in 6 steps (0.15→0.9, ~1.3 s each) to trigger lazy charts, then per `svg` with ≥3 bars: labels = `[...svg.querySelectorAll("g.recharts-label-list text")]` (x attr), names = `[...svg.querySelectorAll("foreignObject")]` divs sorted by `g[transform] translate(x,y)`. Sort BOTH by x, **zip by index** — x-center pairing is off by a systematic half-bar.
|
||
4. Self-row filter: exact hyphenated name (e.g. `Qwen3.8-Flash-Next`, not "Qwen3.8 Flash").
|
||
5. Elo evals (GDPval-AA, AA-Briefcase): charts show AA's published `(Elo−500)/2000` normalization (1866→68.3).
|
||
|
||
## Verification law (non-negotiable — two plausible mappings disagreed twice)
|
||
- Zip top-3 of every chart against the eval page's published sentence (e.g. Kimi 88.7→89, Opus 66.9→67); ship only cross-checked rows.
|
||
- Arithmetic corroboration helps: Omniscience accuracy 17% + non-hallucination 7% ↔ index −60.
|
||
- Record provenance per row: sentence rows → eval-page URL; chart-label rows → model-page URL; `fetched_at` + `index_version`.
|
||
- Geometry pairing (link→nearest chart below) is AMBIGUOUS when chart height > gap to next card — adjacent links share a chart. Use order + sentence tops, not geometry.
|
||
|
||
## Traps
|
||
- `tab.run` = Bun runtime; page-world JS needs `tab.evaluate`. Eval-cell `log()` does not reach display; write to a /tmp file and `cat`.
|
||
- "Add model from specific provider" picker is not in the DOM (renders after chart data loads); UI drill-through × N evals is fragile — prefer the sentences.
|
||
- Headline index FAQ: "scores N on the Artificial Analysis Intelligence Index" — check for "(estimated)" qualifier; est. = "independent evaluation forthcoming".
|