command-center/agent/managed-skills/frank-model-bench-grilling-state/SKILL.md

name, description

name	description
frank-model-bench-grilling-state	Resume the frank-model-bench design grilling: AA Intelligence Index v4.3.2 research brief, fleet endpoint facts, dataset gating, and all settled design decisions for /home/blank/Projects/frank-model-bench.

frank-model-bench — Grilling State (design settled 2026-10-09, awaiting build go-ahead)

Target dir /home/blank/Projects/frank-model-bench is EMPTY (no git). Init from zero.

Settled design decisions

• Scope: Tiered reproduction of AA Intelligence Index v4.3.2 (10 evals, 4 categories; Agents 30 / Coding 20 / General 30 / Science 20).
  • Tier A (local, self-graded): SciCode, Terminal-Bench 4.0 (podman on Dankclaw via ssh one-shot), AA-LCR, HLE incl. image questions (vision towers).
  • Tier B (external dep): GDP.pdf (LiteParse+vision+judge).
  • Tier C (never run; AA reference rows only): AA-Briefcase, GDPval-AA, AutomationBench-AA, AA-Omniscience (public set is 600/6000 — demoted), CritPt (key-gated grader API 10/24h — demoted).
  • Locally-run weight ≈ 45% of AA index weight. Headline = "Index-lite" (category-weighted over run evals, formula on-page).
• Stack: Bun/TS orchestrator (UI, queue, ledger, reports) shelling out to commit-pinned vendored uv venvs per upstream harness (inspect_ai, Harbor, SciCode/HLE scripts). evals.yml manifest pins git URL+SHA+entrypoint+args+repeats per eval.
• UI: local web dashboard, localhost bind only, SSE live tail, start wizard; CLI frank-bench.
• Endpoints: registry hosts.yml (name, base_url, api_key_ref, model, ctx, backend, role, cap). Discovery = probe known fleet hosts ports 7770-7799 /v1/models + /props, confirm-then-append. Fleet: frank-archon dankserver:7770 (radiance TP=2 R9700, 200k), frank-dragoon dankclaw:7770 (halogen, 200k×2), frank-zealot dankclaw:7771 (llama.cpp 3090, 200k×2). All OpenAI-compat + API key + vision.
• Judge: self-hosted fleet model ONLY, default frank-zealot; hard no-self-judge rotation rule. Scores labeled local-judge, drift vs AA hosted graders expected/documented.
• Execution: per-endpoint runs, strictly serial queue, one request in flight per eval (per-endpoint cap), temp 0.6 reasoning ON, max_tokens cap 32,768, AA-style 30-attempt retry, per-sample timeout = ctx ÷ measured decode rate.
• Durability: per-sample JSONL ledger is canonical artifact; resume mid-eval; reports commit to git (reports///{index.html,data.json,hf-card.md,manifest+ledger SHA-256}), ledgers gitignored.
• AA comparison: free Data API key → bench sync-aa pins headline indices; per-benchmark refs hand-seeded in references/aa-scores.yml (source URL + fetched-at + index version per row). Free=100 req/24h headline-only; Pro=500 req/24h per-benchmark (stubbed for later).
• Datasets: bench init-data checklist (hf auth login + accept terms cais/hle, surgeai/GDP.pdf, Harbor Hub); checksum cache ~/.cache/frank-bench/datasets; gated evals refuse with fix-URL and continue suite.
• Methodology updates: version-pinned manifests + immutable history; update = one commit: evals.yml bump + docs/methodology rewrite + log.md + resync snapshot. AA cadence ~2-3 minor versions/week; changelog artificialanalysis.ai/changelog.
• KB: docs/ inside repo, OKF v0.1 (type required; index.md per dir; log.md newest-first; .okf/config.yaml from omp-docs); Obsidian-wikilink convention per serverAppConfigs-docs.
• Git: Gitea on Dankclaw remote, master base, worktree branch per milestone, pre-commit: bun test + shellcheck + OKF validate. Secrets: ~/.config/frank-bench/secrets.env 600.
• Tests: bun test w/ scripted mock OpenAI endpoint (full run→ledger→HTML offline); Playwright dashboard e2e; bench smoke manual real-endpoint.
• Milestones: M1 scaffold+registry+mock; M2 SciCode E2E w/ real report trio; M3 rest of Tier A + queue/resume; M4 dashboard UI + discovery wizard; M5 GDP.pdf + AA snapshot + peer tables.
• Reports: single-page HTML, offline-openable, no external assets; per-benchmark rows: local | AA ref | Δ | repeats | n | graded-by; Tier C rows "AA reference only"; speed strip captured (no AA speed compare).

Key research facts

• AA Data API: base https://artificialanalysis.ai/api/v2, x-api-key header, OpenAPI at /api/v2/openapi; free = /language/models/free headline indices.
• Eval repos: SciCode1/SciCode (HF, Apache), cais/hle (gated, MIT, 2500 q, ~0.27GB), ArtificialAnalysis/AA-LCR (public, Equality Checker prompt public), surgeai/GDP.pdf (gated, Gemini 3.5 judge, inspect_ai), harbor-framework/terminal-bench 4.0 (66 tasks, CONTAINERS_BACKEND=podman, Harbor Hub gated), AA-Omniscience-Public 600q, CritPt public grading API 10/24h.
• AA org repos: Stirrup (MIT agent harness), aa-agentperf-local, aiperf, optima, StirrupJS.
• AA settings: temp 0/0.6, 16k+ max tokens, 30 retries, 1-5 repeats pass@1; Omniscience index [-100,100] accuracy−hallucination.
• Search chain web/kagi→web/searxng (search.franksdank.cloud) verified working 2026-10-09.

Accepted risks (user-informed)

• Elo-pairwise agentic evals unreproducible locally; self-hosted judge drift; Omniscience (top quant-damage signal) not run; HLE vision inclusion diverges from AA text-based posture.