Frank
8af40c402b
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
32 lines
4.0 KiB
Markdown
32 lines
4.0 KiB
Markdown
---
|
||
name: harness-model-config-wire-capture
|
||
description: "Verify what the omp harness actually sends to a local inference endpoint (reasoning_effort, sampler keys, thinking-off) by pointing a scratch HOME at a logging proxy, instead of inferring it from models.yml comments."
|
||
---
|
||
|
||
A `models.yml` comment claiming "the wire sends X" is a claim about the harness, not the engine, and the harness rewrites between the two: compat maps pass unmapped values through, effort ladders clamp before the map applies, and a "disable reasoning" role flag can be implemented as *send the lowest effort* rather than a switch. Read only the wire.
|
||
|
||
## Recipe (minutes, no production state touched)
|
||
|
||
1. **Logging proxy** — python `ThreadingHTTPServer` on `127.0.0.1:7799` forwarding body+path to the real engine and appending the request body's interesting keys (`model, reasoning_effort, max_tokens, temperature, top_p, top_k, chat_template_kwargs`) as one JSON line per request. Run it as a named bash service with `ready.port` so the port is proven before use.
|
||
2. **Scratch HOME** — `mkdir -p /tmp/omph/.omp/agent`, copy the *candidate* `models.yml` + `config.yml` there, rewrite that provider's `baseUrl` to `http://127.0.0.1:7799/v1`. Never edit `~/.omp/agent/` — in `dank/command-center` the checkout is the deployment.
|
||
3. **One-shot runs** — `HOME=/tmp/omph omp --model '<provider>/<id>:<effort>' --no-tools --no-session --no-skills --no-rules --no-extensions --no-pty --no-lsp -p 'Reply with exactly: OK'`, once per effort (`off minimal low medium high xhigh max`). Add `--print-thoughts` when the question is "did thinking actually happen".
|
||
4. Read the log, and assert the value **after** the change. A config edit that never engaged is the failure mode this exists to catch.
|
||
|
||
## Traps found doing this on frank-dragoon (2026-10-21)
|
||
|
||
- `--model X:off` does **not** mean `reasoning_effort: off` on the wire. Reasoning is switched through `compat.reasoningDisableMode` (enum includes `lowest-effort`, `none-effort`, `qwen-enable-thinking-false`, `qwen-template-false`, `chat-template-thinking-false`); the default derives from `thinkingFormat`, and `lowest-effort` quietly keeps thinking alive (`:off` arrived as `"minimal"`, with a canned 26-token thinking block on every "off" request). `none-effort` sent `"none"` → `reasoning_tokens: 0`.
|
||
- Effort levels are clamped to the harness's per-model ladder **before** `reasoningEffortMap` applies, so a map row for a level the ladder lacks is inert (`:max` arrived as `xhigh` even with `max: max`). Capture before promising a level works.
|
||
- Map lookup is `compat.reasoningEffortMap?.[effort] ?? thinking?.effortMap?.[effort] ?? effort` — removing a row is a pass-through, never a 400. Claims about rejection must be proven against the engine.
|
||
- Engine-side claims: ask for a deliberately bad value and read its 400 — the engine prints its own accepted set (`reasoning_effort 'maximum' unsupported; use minimal|low|medium|high|xhigh|max|none`).
|
||
|
||
## Engine-side probes that transfer to any OpenAI-compatible endpoint
|
||
|
||
- **Accepted parameters**: `{model, messages, max_tokens:1, <param>:<value>}` → status + `error.message`. Acceptance is not honouring: report it as acceptance.
|
||
- **Level honoured?** same prompt at `max_tokens 2048` per level, compare `usage.completion_tokens_details.reasoning_tokens`. Equal values across levels = the parameter is ignored; `reasoning_closed_at` reveals a per-level thinking ceiling.
|
||
- **Thinking off**: `chat_template_kwargs {"enable_thinking": false}` and top-level `enable_thinking: false` → expect `reasoning_tokens: 0`.
|
||
- **Vision**: build a solid-colour PNG in-script (`zlib`+`struct`, no PIL), 512×512, ask for the colour in one word **with thinking disabled** — otherwise the answer is cut off by the thinking block and the probe is inconclusive. A 1×1 image only proves token accounting.
|
||
|
||
## Teardown
|
||
|
||
Kill the proxy service, remove the scratch HOME, the proxy script and its log. Never leave copies of `config.yml`/`models.yml` in `/tmp`: they carry provider endpoints and sometimes tokens.
|