33 lines
3.9 KiB
Markdown
33 lines
3.9 KiB
Markdown
---
|
||
name: snapcompact-reader-recall-probe
|
||
description: Measure bitmap-frame OCR recall and true per-frame token billing of a custom vision endpoint to pick snapcompact shape and price it correctly.
|
||
---
|
||
|
||
# Snapcompact reader recall probe
|
||
|
||
Decide snapcompact settings for a non-frontier (custom/local) vision model: which frame shape its tower actually reads, and what the endpoint really bills per frame (vs the harness's blind `FRAME_TOKEN_ESTIMATE`=5024 floor).
|
||
|
||
## Fact sources (in trust order)
|
||
1. `omp://compaction.md` (Snapcompact method section) + `cfg://snapcompact` + `cfg://compaction` — mechanism + resolved values/defaults.
|
||
2. Upstream ground truth: `raw.githubusercontent.com/can1357/oh-my-pi/main/packages/snapcompact/CHANGELOG.md` — per-model eval winners and the `toolbench` re-tune (spacing-tuned cells beat dense; ~16px/char OCR floor). `research/` dir has the eval scripts.
|
||
3. Live bundle truth: grep minified `dist/cli.js` (`~/.bun/install/global/node_modules/@oh-my-pi/pi-coding-agent/dist/`) for string literals: `grep -o '\[\[/[^;]\{0,400\}' cli.js` shows the ideal-shape regex table (`claude→11on16-bw, gemini→8on22-bw@2048, gpt|codex→8on22-bw, kimi→8on22-bw, glm→8on16-bw`); `unknown:"8on22-bw"` is the openai-compat wire fallback. Unmeasured ids (no qwen matcher) get the wire-family shape.
|
||
4. Reachability of pricing knobs (proved 2026-10-27 on 18.12.5): the user-row merge chain (`grep -o '.\{80\}t\.imageCountBudget.\{200\}' cli.js`) forwards `imageCountBudget` ONLY — `imageTokenization` resolves through the compiled catalog identity (no user-row key reaches it, so custom endpoints can never mark frames `verified` and always eat the 5024 floor in sizing). Regimes: `openai-patch | anthropic-patch | deepseek-patch | fixed` (`"fixed":return e.tokens`) — catalog-side only.
|
||
|
||
## Probe recipe
|
||
Scratch dir: `bun init -y && bun add @oh-my-pi/snapcompact` (ships TS source; native renderer via pi-natives). npm version may drift a patch ahead of the harness bundle — geometry caveat, note it.
|
||
|
||
`probe.ts` legs against the endpoint's `/v1/chat/completions`:
|
||
- Build ~120k-char tool-traffic transcript in snapcompact scope format (`¶user:`, `¶call:` + `<out>…</out>`, `¶ai:`) with N planted unique spelled-out facts (spell digits as words so filler can't collide; keep filler digit-free); score recall by `content.includes(value)`.
|
||
- Legs: plain-text baseline, then images per candidate shape via `resolveShape({api, id}, variant)` + `renderMany(text, {shape})`; send as `{type:"image_url", image_url:{url:"data:<mime>;base64,<data>", detail:"original"}}` with a reading-guide text part.
|
||
- Replicate the harness's real request shape: same `chat_template_kwargs` (enable_thinking/preserve_thinking) and sampling from the models.yml row.
|
||
- True per-frame billing = `(prompt_tokens − text-scaffold) / frames`, compare vs `plannedFrameTokens(shape)`/`frameBilling`.
|
||
|
||
Then: declare `imageCountBudget: <proven count>` on the models.yml row (custom providers inherit the unknown-provider floor of 5 images/request; llama.cpp has no such limit) — that lifts the count clamp; token pricing stays floor-bound until upstream prices the wire family. Measured baseline (frank-archon, omp 18.12.5): auto `8on22-bw` 8/8 recall at 47 % of text's billed tokens, ~2,370 tok/frame true vs 5,024 charged; dense grids (8on16/11on16) drop characters — keep `shape: auto`.
|
||
|
||
## Gotchas
|
||
- `log()` is an eval-kernel global, NOT a bun global — define a console wrapper or the script dies at first call.
|
||
- bash `async` + `name` = service mode (rejects async); drop `name` for finite jobs; `timeout: 0` disables the 600s deadline for multi-leg vision runs.
|
||
- edit tool needs the absolute path (not bare filename) for /tmp files; `PUT <N` inserts BEFORE line N.
|
||
- Vision gate: row must declare `input: [text, image]` or snapcompact is skipped silently.
|
||
- Fleet follow-ups live in `omp-docs/core/snapcompact.md` (dank/command-center).
|