40 lines
5.1 KiB
Markdown
40 lines
5.1 KiB
Markdown
---
|
||
name: frank-model-bench-grilling-state
|
||
description: "Resume the frank-model-bench design grilling: AA Intelligence Index v4.3.2 research brief, fleet endpoint facts, dataset gating, and all settled design decisions for /home/blank/Projects/frank-model-bench."
|
||
---
|
||
|
||
# frank-model-bench — Grilling State (design settled 2026-10-09, awaiting build go-ahead)
|
||
|
||
Target dir `/home/blank/Projects/frank-model-bench` is EMPTY (no git). Init from zero.
|
||
|
||
## Settled design decisions
|
||
- **Scope**: Tiered reproduction of AA Intelligence Index v4.3.2 (10 evals, 4 categories; Agents 30 / Coding 20 / General 30 / Science 20).
|
||
- **Tier A (local, self-graded)**: SciCode, Terminal-Bench 4.0 (podman on Dankclaw via ssh one-shot), AA-LCR, HLE incl. image questions (vision towers).
|
||
- **Tier B (external dep)**: GDP.pdf (LiteParse+vision+judge).
|
||
- **Tier C (never run; AA reference rows only)**: AA-Briefcase, GDPval-AA, AutomationBench-AA, AA-Omniscience (public set is 600/6000 — demoted), CritPt (key-gated grader API 10/24h — demoted).
|
||
- Locally-run weight ≈ 45% of AA index weight. Headline = "Index-lite" (category-weighted over run evals, formula on-page).
|
||
- **Stack**: Bun/TS orchestrator (UI, queue, ledger, reports) shelling out to commit-pinned vendored uv venvs per upstream harness (inspect_ai, Harbor, SciCode/HLE scripts). `evals.yml` manifest pins git URL+SHA+entrypoint+args+repeats per eval.
|
||
- **UI**: local web dashboard, localhost bind only, SSE live tail, start wizard; CLI `frank-bench`.
|
||
- **Endpoints**: registry `hosts.yml` (name, base_url, api_key_ref, model, ctx, backend, role, cap). Discovery = probe known fleet hosts ports 7770-7799 `/v1/models` + `/props`, confirm-then-append. Fleet: frank-archon dankserver:7770 (radiance TP=2 R9700, 200k), frank-dragoon dankclaw:7770 (halogen, 200k×2), frank-zealot dankclaw:7771 (llama.cpp 3090, 200k×2). All OpenAI-compat + API key + vision.
|
||
- **Judge**: self-hosted fleet model ONLY, default frank-zealot; hard no-self-judge rotation rule. Scores labeled `local-judge`, drift vs AA hosted graders expected/documented.
|
||
- **Execution**: per-endpoint runs, strictly serial queue, one request in flight per eval (per-endpoint cap), temp 0.6 reasoning ON, max_tokens cap 32,768, AA-style 30-attempt retry, per-sample timeout = ctx ÷ measured decode rate.
|
||
- **Durability**: per-sample JSONL ledger is canonical artifact; resume mid-eval; reports commit to git (reports/<model>/<run-id>/{index.html,data.json,hf-card.md,manifest+ledger SHA-256}), ledgers gitignored.
|
||
- **AA comparison**: free Data API key → `bench sync-aa` pins headline indices; per-benchmark refs hand-seeded in `references/aa-scores.yml` (source URL + fetched-at + index version per row). Free=100 req/24h headline-only; Pro=500 req/24h per-benchmark (stubbed for later).
|
||
- **Datasets**: `bench init-data` checklist (hf auth login + accept terms cais/hle, surgeai/GDP.pdf, Harbor Hub); checksum cache ~/.cache/frank-bench/datasets; gated evals refuse with fix-URL and continue suite.
|
||
- **Methodology updates**: version-pinned manifests + immutable history; update = one commit: evals.yml bump + docs/methodology rewrite + log.md + resync snapshot. AA cadence ~2-3 minor versions/week; changelog artificialanalysis.ai/changelog.
|
||
- **KB**: `docs/` inside repo, OKF v0.1 (type required; index.md per dir; log.md newest-first; .okf/config.yaml from omp-docs); Obsidian-wikilink convention per serverAppConfigs-docs.
|
||
- **Git**: Gitea on Dankclaw remote, master base, worktree branch per milestone, pre-commit: bun test + shellcheck + OKF validate. Secrets: ~/.config/frank-bench/secrets.env 600.
|
||
- **Tests**: bun test w/ scripted mock OpenAI endpoint (full run→ledger→HTML offline); Playwright dashboard e2e; `bench smoke` manual real-endpoint.
|
||
- **Milestones**: M1 scaffold+registry+mock; M2 SciCode E2E w/ real report trio; M3 rest of Tier A + queue/resume; M4 dashboard UI + discovery wizard; M5 GDP.pdf + AA snapshot + peer tables.
|
||
- **Reports**: single-page HTML, offline-openable, no external assets; per-benchmark rows: local | AA ref | Δ | repeats | n | graded-by; Tier C rows "AA reference only"; speed strip captured (no AA speed compare).
|
||
|
||
## Key research facts
|
||
- AA Data API: base https://artificialanalysis.ai/api/v2, `x-api-key` header, OpenAPI at /api/v2/openapi; free = /language/models/free headline indices.
|
||
- Eval repos: SciCode1/SciCode (HF, Apache), cais/hle (gated, MIT, 2500 q, ~0.27GB), ArtificialAnalysis/AA-LCR (public, Equality Checker prompt public), surgeai/GDP.pdf (gated, Gemini 3.5 judge, inspect_ai), harbor-framework/terminal-bench 4.0 (66 tasks, CONTAINERS_BACKEND=podman, Harbor Hub gated), AA-Omniscience-Public 600q, CritPt public grading API 10/24h.
|
||
- AA org repos: Stirrup (MIT agent harness), aa-agentperf-local, aiperf, optima, StirrupJS.
|
||
- AA settings: temp 0/0.6, 16k+ max tokens, 30 retries, 1-5 repeats pass@1; Omniscience index [-100,100] accuracy−hallucination.
|
||
- Search chain web/kagi→web/searxng (search.franksdank.cloud) verified working 2026-10-09.
|
||
|
||
## Accepted risks (user-informed)
|
||
- Elo-pairwise agentic evals unreproducible locally; self-hosted judge drift; Omniscience (top quant-damage signal) not run; HLE vision inclusion diverges from AA text-based posture.
|