chore(agent): commit the machine tier and the doctrine text the checkout was already running
Landing blocker for #61 and #63: the deployment checkout is 12 commits behind and `merge --ff-only` refuses while tracked paths are dirty. Discard is not available here (workspace policy: no `checkout`/`stash` on shared state), so the dirt is committed. - `agent/APPEND_SYSTEM.md`: the point-of-use / one-work-log / PR-link doctrine, already byte-identical to `origin/main` (`cmp` against the trunk blob: no difference). No new prose — the commit exists so the trunk merge can proceed. - `agent/config.yml` and `agent/managed-skills/**`: the machine-live tier the heartbeat owns (AGENTS.md § Live-sync tiers). `ALLOW_SECRETS=1` was used, deliberately and after auditing the staged diff by hand: the only credential-shaped strings in it are two copies of `AKIA0000000000000EXAMPLE` — the inert all-zero AWS example id — and they appear inside the two skill pages that teach that exact placeholder shape. This checkout is still running the pre-#61 hook; the merged gate (value-scoped placeholder filter, `71601f7`) exempts this value, so the refusal dies with this landing. No `--no-verify` was used: the gate ran and its verdict is recorded here. Refs #61
This commit is contained in:
Frank committed
1 parent
004962ffce
commit
8af40c402b
79 files changed
+2771
-332
No files matched your search
@@ -1,29 +1,48 @@
|
||||
---
|
||||
name: omp-extension-stale-module-audit
|
||||
description: "Diagnose recurring omp extension handler timeouts/silence when the on-disk fix is already deployed: harness has no extension hot-reload, so running sessions freeze the module at session start (zombie modules)."
|
||||
description: "Diagnose omp extension misbehavior (handler timeouts/silence, cross-instance arbitration failures like simultaneous speech) when the on-disk fix is already deployed: running sessions hold pre-upgrade in-memory bundles; compare ps lstart against artifact mtimes and merge dates."
|
||||
---
|
||||
|
||||
Use when a fix for an omp extension is merged/deployed yet the SAME errors persist ("handler timed out after 30000ms", silence, wrong behavior) in running sessions.
|
||||
# Stale in-memory omp extension bundles
|
||||
|
||||
**Core fact:** the harness loads extension `.ts` modules at session start and never reloads them (no `/reload` for extensions; bundle `reload` hits are LSP/MCP/browser only). A session older than the fix keeps executing stale code regardless of the file on disk.
|
||||
Symptom class: an extension behaves as if unfixed even though the disk artifact
|
||||
contains the fix — recurring handler timeouts, silence, or (multi-instance
|
||||
coordination features) arbitration failures that look like code bugs. Root
|
||||
cause: **omp loads the bundle at session start; `omp plugin upgrade` on disk
|
||||
does not touch the code already running in live sessions.**
|
||||
|
||||
**Procedure:**
|
||||
1. Census the errors from logs, not guesses — timeout lines carry the culprit pids:
|
||||
`grep -h "handler timed out" ~/.omp/logs/omp.<today>*.log | python3 -c 'import sys,json; rows=[json.loads(l) for l in sys.stdin]; rows.sort(key=lambda r:r["timestamp"]); [print(r["timestamp"], r["event"], r["pid"]) for r in rows]'`
|
||||
(`~/.omp/logs/omp.<date>.<pid>.log` — the pid in the filename is the session.)
|
||||
2. Session start times vs deployed file versions:
|
||||
`ps -eo pid,lstart,args | grep '[o]mp'`
|
||||
`git log --format='%h %ad %s' --date=format:'%m-%d %H:%M' -- <extension path>`
|
||||
`git reflog --format='%gd %ad %gs' --date=format:'%m-%d %H:%M' | grep merge` (live-sync pull times = when the file content changed).
|
||||
Any session whose start time precedes the fix's arrival has the stale module.
|
||||
3. Confirm the stale revision actually contains the bad pattern — read what that session loaded, not HEAD:
|
||||
`git show <old-rev>:<path> | grep -n 'await speak\|async (event'`
|
||||
(e.g. an `async agent_end` awaiting a >30 s chain is exactly what the harness's per-handler `Kzc` 30 s race logs; fire-and-forget `void` is fine — detached promises are NOT tracked by the dispatch window.)
|
||||
4. Optional empirical proof when static analysis says "impossible": probe extensions through the real harness:
|
||||
`omp -p --no-extensions -e /tmp/probe.ts "Reply OK"` with a probe that logs handler start/return and `void`s a 40 s detached promise + spawns a long child. Fast exit + no timeout log ⇒ detachment is untracked ⇒ blame stale modules, not the harness.
|
||||
5. Remediation: restart the affected sessions (kill the listed pids or their terminals — the operator session itself may be one of the zombies; say so explicitly). There is no other fix.
|
||||
## Recipe 1: date the sessions against the artifact
|
||||
|
||||
**Gotchas:**
|
||||
- Per-handler attribution in the log is accurate per dispatch; don't blame other extensions before checking the pid's start time.
|
||||
- Endpoints healthy + silence persisting usually means TWO stacked bugs (e.g. stale awaiting module in old sessions AND a mute bug in current code) — diagnose each pid cohort separately.
|
||||
- Extension chain code must use self-owned `AbortSignal.timeout(...)`, never a ctx-scoped signal: harness dispatch signals die when the dispatch window closes.
|
||||
```bash
|
||||
date
|
||||
stat -c '%y %n' ~/.omp/plugins/node_modules/<pkg>/dist/index.js # last upgrade
|
||||
ps -eo pid,lstart,etime,args | grep 'omp' | grep -v grep # each session's boot
|
||||
```
|
||||
|
||||
Any session whose START predates the artifact mtime holds an older bundle —
|
||||
for it, the "fix" never shipped. When the feature landed in git, compare
|
||||
against the merge date, not the last upgrade (`git log --format='%h %cI %s'
|
||||
main | grep -i <feature>`): a session can predate the feature *existing*.
|
||||
|
||||
Fix is always: restart (`/quit`, then `omp --resume <id>`) — resume is
|
||||
lossless. It is the user's live session; never kill it yourself.
|
||||
|
||||
## Recipe 2: cross-process arbitration — a bypasser explains "simultaneous X"
|
||||
|
||||
When a feature serializes across sessions via files (e.g. dankvoice stage:
|
||||
`~/.cache/dankvoice/stage/` with `lease.json` = `{holder,pid,expires}`, one
|
||||
`<ISO>-<session>.json` entry per queued item, FIFO = filename order, lease TTL
|
||||
6 s), overlap between two actors proves **at least one actor never consults
|
||||
the directory** — the mechanism can't serialize against a bypasser. Census the
|
||||
queue live (`watch -g -d 'ls <dir>; cat <dir>/lease.json'`): a live, renewing
|
||||
lease proves the new sessions coordinate; then Recipe 1 finds the bypasser.
|
||||
|
||||
Known residual hazard to note, not silently fix: a lease heartbeat is a timer
|
||||
on the holder's own event loop; a stall longer than the TTL lets a peer steal
|
||||
the lease mid-action. Mitigation = pid-liveness check in the steal path.
|
||||
|
||||
## Verification
|
||||
|
||||
After restarts, the symptom must vanish while the queue shows traffic. If it
|
||||
persists, look for actors on OTHER hosts (file arbitration is per-box), or
|
||||
the TTL-steal hazard above.
|
||||
Reference in new issue
Block a user