diff --git a/agent/managed-skills/aa-chart-label-browser-extraction/SKILL.md b/agent/managed-skills/aa-chart-label-browser-extraction/SKILL.md new file mode 100644 index 0000000..e90b044 --- /dev/null +++ b/agent/managed-skills/aa-chart-label-browser-extraction/SKILL.md @@ -0,0 +1,24 @@ +--- +name: aa-chart-label-browser-extraction +description: "Extract per-model benchmark scores from artificialanalysis.ai pages: charts are client-rendered recharts SVGs invisible to plain HTML fetch; pair sorted label/name x-coords, then cross-check against published sentences before trusting any chart↔eval identity." +--- + +Extract per-eval scores for a model from artificialanalysis.ai when no API key exists. Plain HTML fetch yields NO chart data (client-side recharts; SSR has only meta + FAQ sentences; `/api/*` = 401). + +## Procedure +1. Managed browser tab: `browser.open({name, url, wait_until: "domcontentloaded"})`. +2. **Per-eval top-3 (authoritative):** open `https://artificialanalysis.ai/evaluations/`; regex `document.body.innerText` (collapse whitespace): `([A-Z][^.;]{0,60}?) scores the highest on (.+?) with a score of (-?[\d.]+%?), followed by (.+?) with a score of ... and ...` — mind decimal points truncating naive `[^.]+` regexes. Build the page script as a plain string (`String.raw`), not an interpolated template, so `\\d` survives. +3. **All-eval per-model rows:** on `https://artificialanalysis.ai/models/`: scroll in 6 steps (0.15→0.9, ~1.3 s each) to trigger lazy charts, then per `svg` with ≥3 bars: labels = `[...svg.querySelectorAll("g.recharts-label-list text")]` (x attr), names = `[...svg.querySelectorAll("foreignObject")]` divs sorted by `g[transform] translate(x,y)`. Sort BOTH by x, **zip by index** — x-center pairing is off by a systematic half-bar. +4. Self-row filter: exact hyphenated name (e.g. `Qwen3.8-Flash-Next`, not "Qwen3.8 Flash"). +5. Elo evals (GDPval-AA, AA-Briefcase): charts show AA's published `(Elo−500)/2000` normalization (1866→68.3). + +## Verification law (non-negotiable — two plausible mappings disagreed twice) +- Zip top-3 of every chart against the eval page's published sentence (e.g. Kimi 88.7→89, Opus 66.9→67); ship only cross-checked rows. +- Arithmetic corroboration helps: Omniscience accuracy 17% + non-hallucination 7% ↔ index −60. +- Record provenance per row: sentence rows → eval-page URL; chart-label rows → model-page URL; `fetched_at` + `index_version`. +- Geometry pairing (link→nearest chart below) is AMBIGUOUS when chart height > gap to next card — adjacent links share a chart. Use order + sentence tops, not geometry. + +## Traps +- `tab.run` = Bun runtime; page-world JS needs `tab.evaluate`. Eval-cell `log()` does not reach display; write to a /tmp file and `cat`. +- "Add model from specific provider" picker is not in the DOM (renders after chart data loads); UI drill-through × N evals is fragile — prefer the sentences. +- Headline index FAQ: "scores N on the Artificial Analysis Intelligence Index" — check for "(estimated)" qualifier; est. = "independent evaluation forthcoming". diff --git a/agent/managed-skills/aa-chart-model-score-extraction/SKILL.md b/agent/managed-skills/aa-chart-model-score-extraction/SKILL.md new file mode 100644 index 0000000..f1fcd28 --- /dev/null +++ b/agent/managed-skills/aa-chart-model-score-extraction/SKILL.md @@ -0,0 +1,24 @@ +--- +name: aa-chart-model-score-extraction +description: "Extract per-model benchmark scores from artificialanalysis.ai pages: charts are client-rendered Recharts SVGs; zip label-list texts to foreignObject names by x-sorted index, validate chart identity against the page's published top-3 sentence, apply Elo/Omniscience normalization laws, seed frank-intelligence-bench aa-scores.yml." +--- + +Extract AI benchmark scores for a specific model from artificialanalysis.ai. Per-model per-eval numbers are NOT in SSR HTML (JS charts, API needs key); eval-page SSR text DOES carry the top-3 sentence. + +## Procedure (validated 5x on this repo's model pages) +1. `browser.open({ name, url: "https://artificialanalysis.ai/models/", wait_until: "domcontentloaded" })`, then scroll in ~6 steps (0.15..0.9 of scrollHeight, ~1.3 s pauses) — charts lazy-render. +2. Extract each chart: pair `g.recharts-label-list text` (values, take `x` attr) with `foreignObject div` (model names, x from parent `g[transform] translate(x,y)`). **Sort both lists by x and zip by index — NOT nearest-x** (nearest-x is off by one bar; index-zip matches published tops exactly). +3. Self row: match name loosely (`Qwen3.8-Flash-Next` has hyphens — test both forms). + +## Identity law (never ship an order-guessed chart→eval mapping) +- Model pages have ~14 charts, 18 eval cards; DOM order and geometry pairing BOTH proved ambiguous. +- Anchor every chart by its top-3 names/values vs the eval page's published sentence: `https://artificialanalysis.ai/evaluations/` body text: "X scores the highest on with a score of N%, followed by …". Regex must tolerate decimals: value pattern `(-?[\\d.]+%?)`; build page scripts with `String.raw` (bun template literals eat `\\d`). +- A chart's `self` value on the MODEL page is page-local → trustworthy once the top-3 anchor confirms the eval (e.g. AA-LCR: Kimi 88.7 → chart top "Kimi K3 (max)=89"). + +## Normalization laws (frank-intelligence-bench aa-scores.yml) +- Elo evals (AA-Briefcase, GDPval-AA): store `(Elo−500)/2000×100` (1866 → 68.3), AA's published normalization. +- AA-Omniscience: score = Accuracy (w10) + 1−Hallucination (w5), both 0–100 → composite `(10·acc + 5·nonhall)/15`. NEVER feed the −100..100 Index chart value (pre-fix cost ~7 pts; calibration then reproduces AA's official index ±0.6: Flash-Next 39.85≈40, Haiku 43.4≈43, 27B 33.85≈34, 3.6-27B 21.55≈21). +- Every seeded row: `source_url` + `fetched_at` + `index_version`. + +## Seed + validate loop +Append model block to `config/references/aa-scores.yml` (block-insert via python string replace on the ` - name: ` line — beware: dropping rows from a block loses the 2-space indent on the anchor line; verify with `yaml.safe_load`), then `bun -e loadAaRefs` parse-check, `bun src/cli.ts report $RID`, calibration check (full-row models: lite vs AA official), `bun test`, commit. diff --git a/agent/managed-skills/aa-reference-scores-seeding/SKILL.md b/agent/managed-skills/aa-reference-scores-seeding/SKILL.md new file mode 100644 index 0000000..af9eca0 --- /dev/null +++ b/agent/managed-skills/aa-reference-scores-seeding/SKILL.md @@ -0,0 +1,41 @@ +--- +name: aa-reference-scores-seeding +description: "Seed/refresh frank-intelligence-bench config/references/aa-scores.yml from artificialanalysis.ai: authoritative top-3 sentences from eval pages, Recharts SVG zip extraction for per-model chart values, cross-check law, Elo normalization, schema/repair loop." +--- + +Seed frank-intelligence-bench `config/references/aa-scores.yml` from AA's public pages, then re-render the report trio. + +## Provenance law (non-negotiable) +- Every row: `{ eval, pct, source_url, fetched_at, index_version }` (Zod-enforced; top-level `schema_version: 1, fetched_at, index_version` required). +- Eval ids MUST match report result ids: `scicode, aa-lcr, hle, terminal-bench, gdp-pdf, aa-briefcase, gdpval-aa, automationbench-aa, aa-omniscience, critpt` (render `refFor(evalId)` joins on exact id). +- Never ship an order-guessed chart↔eval mapping; only cross-checked ones. + +## Data tiers (authority order) +1. **Eval-page sentence** (authoritative, SSR, per eval page `/evaluations/`): `"X scores the highest on with a score of N%, followed by Y ... and Z ..."` → top-3 rows with verbatim model names. Regex trap: model names contain `5.5` — capture names with `[^.;]` truncates at decimals; use targeted extraction per page, verify each value against the page's `66.9`-style raw text. +2. **Model-page chart labels** (per-model values, incl. the page's own model): charts are client-side Recharts (NOT in raw HTML, NOT via /api/ — keyless API 404s). Browser: `browser.open({name,url,wait_until})`, scroll full page in ~0.15 steps (lazy render), then `tab.evaluate` (NOT `tab.run` — that's Bun world): + - names = `` div text, x from parent `g[transform]` `translate(x,...)`; + - values = `g.recharts-label-list text` attrs; + - **zip by sorted-x index** (nearest-x pairing is off by half a bar; index-zip verified against published tops). +3. `sync-aa` free Data API for headline indices — needs `AA_API_KEY` (unset in ~/.secrets as of 2026-10-11). + +## Cross-check law (mandatory before shipping chart-derived rows) +Chart's top-3 zipped values must match the eval page's published sentence (e.g. Kimi 88.7→89, Elo 1866→68.3, Opus index 46). Two independent methods disagreeing = stop, re-anchor; geometry pairing (link-top→svg-top proximity) collides on adjacent cards — prefer sentence-vs-zip corroboration. + +## Normalizations +- Elo evals (GDPval-AA, AA-Briefcase): sentences publish raw Elo; store AA's published `(Elo−500)/2000`×100 (1866→68.3). Document in file header. +- AA-Omniscience `eval: aa-omniscience` = the −100..100 Index (not accuracy); accuracy/non-hallucination charts corroborate (17%/7% ⇒ −60 arithmetically consistent). + +## Known identities (2026-10-11 census, v4.3.2) +- frank-archon lineage base = **Qwen3.8-Flash-Next** (`/models/qwen3-8-flash-next`), AA index 40, class median 18. NOT Qwen3-Next-80B-A3B-Instruct (index 10, 2025 family) — first seed got this wrong; verify lineage in `docs/endpoints.md` before labeling `kind: base`. +- Base per-eval (chart): SciCode 51, HLE 38, AA-LCR 80, TB-4.0 25, GDP.pdf 16, CritPt 11, Briefcase 54, GDPval 56, AutoBench 56, Omniscience −10. +- Frontier anchors: SciCode Opus 5.5 Max 66.9 (board flagged "under review" — travel this caveat with any SciCode delta); HLE Opus 61.4; AA-LCR Kimi K3 88.7; TB Sonnet 63.6; CritPt GPT-6.1 Sol 32.3. + +## Edit traps +- YAML surgery via `edit`: replacing a line range that included a top-level key (`schema_version`/`models:`) silently duplicates or deletes the anchor → Zod error `expected 1 / received undefined`. After any header edit, `grep -n "^schema_version\|^fetched_at\|^index_version\|^models:"` must show exactly one each. + +## Ship loop +1. `bun -e 'import { loadAaFile, loadAaRefs } from "./src/aa"; ...'` schema-parse gate. +2. `bun src/cli.ts report ` (takes bare id, NOT a path) → trio in `reports///`. +3. Verify `data.json .aa_reference` non-empty, per-eval coverage, delta columns. +4. log entry in `docs/log.md` + atomic commit + push (local gitea OK). +5. `browser.close({name})` — headless tabs are shared/retained. diff --git a/agent/managed-skills/audit-subagent-deliverables/SKILL.md b/agent/managed-skills/audit-subagent-deliverables/SKILL.md new file mode 100644 index 0000000..e3ec436 --- /dev/null +++ b/agent/managed-skills/audit-subagent-deliverables/SKILL.md @@ -0,0 +1,61 @@ +--- +name: audit-subagent-deliverables +description: "Integrate subagent-written code safely: line-by-line audit, mechanical gates (bash test/lint), adversarial-review-report triage, resource-safety pass (fd/memory), and cutover with running-process restarts." +--- + +# Audit subagent deliverables + +A subagent that wrote code is a source of plausible output, not a verified change. Integrate it the way you'd integrate an untrusted PR — with extra suspicion, because the author is an agent that shares your blind spots and its prose is more confident than its evidence. + +Order of operations: read everything → gates → tests (as evidence, not proof) → resource-safety pass → fix in-place → cutover. + +## 1. Read every produced file, line by line +Full file, not a diff skim. You are the reviewer who must catch the line the author never thought about. + +## 2. Mechanical gates before logic review +```bash +bash -n