diff --git a/agent/managed-skills/agenttrash-purger-invocation/SKILL.md b/agent/managed-skills/agenttrash-purger-invocation/SKILL.md index 2164e93..f357416 100644 --- a/agent/managed-skills/agenttrash-purger-invocation/SKILL.md +++ b/agent/managed-skills/agenttrash-purger-invocation/SKILL.md @@ -1,9 +1,9 @@ --- name: agenttrash-purger-invocation -description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) — plus its two refusal traps: ENOSPC masquerading as a stale lock, and an unreadable container-storage directory poisoning the whole root scan." +description: "Invoke the .AGENTTRASH purger (scripts/purge-agenttrash.sh) — plus its refusal traps (ENOSPC as a fake stale lock, unreadable dirs poisoning a root) and the per-root scope trap." --- -# Purging .AGENTTRASH — invocation, and the two traps that make it refuse +# Purging .AGENTTRASH — invocation, and the traps that make it refuse or over-reach `scripts/purge-agenttrash.sh` (serverAppConfigs, merged in PR #79 `fad87aa`) is the tool form of the fleet's deletion policy: files land in `.AGENTTRASH`, and an approved purge runs this instead of being improvised per session. @@ -19,6 +19,10 @@ sh scripts/purge-agenttrash.sh --execute /some/root # scope to one root (defaul - Every printable line is fixed at plan time in `$HOME/.agenttrash-purge.lock/plan`, so dry-run and `--execute` output cannot drift. Exit `0` ok · `1` any refusal or failure · `2` usage. - Deleting a big staged item by hand (e.g. `find -depth -delete`) is the fallback **only** when the tool cannot run at all — see trap 1 — and the operator's purge approval covers exactly that. +## Trap 0: the approval is per item, the scope is per root + +The plan verb `empty /.AGENTTRASH` means **the whole trash directory**, not the one item the operator approved. So before `--execute`, list the root and prove it holds only the approved staged item — `ls -A /.AGENTTRASH && du -sh /.AGENTTRASH/*` — and note that a second `.AGENTTRASH` deeper in the tree is also in scope (`plan remove` on it = it was empty, safe). Measured 2026-10-30 on dankserver: `/mnt/entropy/.AGENTTRASH` held exactly the approved `20261030-movies-monk` (124 files, 32 G) plus an empty `qwen38-w5nl64/.AGENTTRASH`; plan → `--execute /mnt/entropy` reported `removed=2 deleted=2 refused=0` and free space rose 187 G → 218 G. + ## Trap 1: a full filesystem makes it lie about a lock (measured 2026-10-30) With **0 bytes free**, three invocations refused with @@ -39,4 +43,4 @@ and no lock directory existed anywhere. The script writes its lock *and* its pla ## Verified good path (dankserver, 2026-10-30) -After freeing space by hand: `/tmp`-scoped plan → `--execute` deleted the staged item and reported `removed=1 deleted=1`; a second `--execute /mnt/entropy` cleaned that volume's root (`refused=0`). The cleanup log carries one dated line per run; read it instead of re-deriving what happened. +After freeing space by hand: `/tmp`-scoped plan → `--execute` deleted the staged item and reported `removed=1 deleted=1`; a second `--execute /mnt/entropy` cleaned that volume's root (`refused=0`). A 32 G staged item on an 87 T volume scans + purges in ~90 s per pass (the scan, not the deletion, is the slow part). The cleanup log carries one dated line per run — both the dry run and the execute log the verb `plan`, so tell them apart by `removed=`; read it instead of re-deriving what happened. diff --git a/agent/managed-skills/line-edit-anchor-discipline/SKILL.md b/agent/managed-skills/line-edit-anchor-discipline/SKILL.md index 3f8c643..57e143a 100644 --- a/agent/managed-skills/line-edit-anchor-discipline/SKILL.md +++ b/agent/managed-skills/line-edit-anchor-discipline/SKILL.md @@ -1,24 +1,42 @@ --- name: line-edit-anchor-discipline -description: "Avoid the recurring edit-tool line-op traps: empty PUT deletes lines, PUT ranges swallowing multi-line block delimiters, and anchors landing mid-expression. Use whenever issuing CUT/PUT line edits on tagged files." +description: "Avoid the recurring edit-tool line-op traps: empty PUT deletes lines, PUT ranges swallowing neighbours, wrong-anchored PUTs replacing the wrong line, literal markdown bullet rows arriving verbatim as \"+-\"." --- -# Line-edit anchor discipline (CUT/PUT ops on tagged files) +# Line-edit anchor discipline -Five self-inflicted syntax wounds in one session (franksign, 2026-10-21) all trace to three rules. +The `edit` tool addresses lines, not intent. Three traps cost real time; all three +fail *silently* or with a repair notice that is easy to skim past. -## Rules -1. **An empty PUT body DELETES the targeted lines** (tool warns: "Interpreted an empty PUT body as deletion"). Twice it ate a whole log entry / a closing `}`. For no-ops use nothing; for deletes use explicit `CUT`. -2. **Never PUT a range whose endpoints cut through a multi-line block.** If a `| {`-style union variant, a frontmatter `---` fence, or a `test(...)` closer straddles the range boundary, the replacement swallows the opener/closer and strands the body. Pre-condition: line N-1 and M+1 must be block-complete. Fix-after-the-fact pattern: read the broken region raw, then `PUT k.=k:` to restore the single swallowed line (e.g. the missing `---`, the orphaned ` | {`, `}` → `});`). -3. **Search-hit reads do not authorize PUT.** A grep/folded-summary read marks lines as "never displayed"; the edit tool refuses and echoes actual content. That refusal is a free verification — re-check the echoed lines match intent, then straight-retry with the same tag. +## 1. Empty PUT deletes the range -## Costly mistakes seen -- Empty `PUT 7-7:` deleted a just-written log entry; restoring required re-issuing the full entry text. -- `PUT 6.=7` replacing `timestamp`+`---` dropped the frontmatter close fence → lint-breaking file silently. -- `PUT 10.=11` on a union type swallowed ` | {` opening a 12-line variant → cascade of LSP "unexpected label" warnings. -- `CUT` of a stranded opener left its dangling `.json())` behind; balance errors surface only at EOF (`expected ')' but file ends`) — locate by reading regions raw, not by widening the payload. +`PUT n.=m:` with no body rows deletes lines n..m. To insert, use `PUT >n:` +(after line n) or `PUT /.omp/extensions/probe.js` +(default-export `function (pi)`), log to an absolute path, then + +```bash +: > /tmp/probe.log +timeout 220 omp -p --thinking off --cwd \ + "Run these bash commands strictly one after the other: first: sleep 7; echo FIRST. then: sleep 7; echo SECOND. Then reply: FIRST SECOND" +``` + +Register the tick inside `session_start` (`ctx.setInterval(() => stamp(info(ctx)), 1000)`), +and give every entry you expect to exclude a speech-shaped payload with a sentinel word — +an absence assertion with an empty producer proves nothing. Do NOT pass `--no-session`: +the journal is the thing under test. Teardown: nothing to undo (no plugin link, no +`~/.omp/plugins/package.json` write) — delete the run dir. diff --git a/agent/managed-skills/radiance-endpoint-silent-wedge-triage/SKILL.md b/agent/managed-skills/radiance-endpoint-silent-wedge-triage/SKILL.md new file mode 100644 index 0000000..cdfcd47 --- /dev/null +++ b/agent/managed-skills/radiance-endpoint-silent-wedge-triage/SKILL.md @@ -0,0 +1,32 @@ +--- +name: radiance-endpoint-silent-wedge-triage +description: "Diagnose a radiance (frank-archon) endpoint that is wedged or crawling while systemd says active(running) and /health answers 200 — separate engine-state wedge from config, hardware, swap and clock causes, and prove the restart fixed it." +--- + +# Silent wedge: `active (running)` + `/health` 200, but no throughput + +For radiance endpoints (`frank-archon`, root unit whose `ExecStart` is `exec podman wait`). The unit can only exit when the container dies, so a stalled engine loop **never** triggers `Restart=always`, `up` stays `1`, and Alertmanager stays empty (every rule keys on `up==0`, `num_requests_waiting>0`, or TTFT p95). Verified 2026-10-30 21:00 on dankserver. + +## Triage ladder (in this order — each step kills a class) + +1. **Front end vs engine split.** `curl -w '%{time_total}' /health` vs `/v1/models`. A 0.4 ms `/health` beside a 20 s `/v1/models` timeout = front end alive, engine loop stalled. +2. **Unit truth, not the port.** `systemctl status frank-archon -n 0` + `systemctl show -p MainPID,DropInPaths,ExecMainStartTimestamp`. A 4 h-old unit on a 2-week-old host means a restart happened today — cross-check `journalctl -u --since ` for `Starting/Stopping/exit` and `git log -- scripts/.*` to see whether a deploy caused it. Non-empty `DropInPaths` = leftover bench arm (separate trap). +3. **Container storage namespace.** The unit is `User=dank`, so the container lives in **rootless** storage: `sudo podman ps -a` lists nothing and `podman inspect` says `no such object` — query as the user, without sudo. +4. **Engine's own voice (the decisive artifact).** `podman logs --timestamps frank-archon | grep "E device:"`. `a synchronisation has seen no queue advance for 5 s` naming `barrier-and dep[0]=1` on both ranks = the TP all-reduce never completed. Timestamp the first occurrence; that is the incident start. +5. **Rule out the engine's own suggested causes before believing anything else:** `cat /sys/module/amdgpu/parameters/cwsr_enable` (its message blames `cwsr_enable=0`; `1` falsifies it). Swap: read `/proc//status` VmSwap — do NOT infer from `free`/`/proc/swaps` totals (host-wide swap can be 32 GiB while the engine holds 378 MiB). Clocks during load: fire a request, then `rocm-smi --showclocks --showperflevel --showpower` + `/sys/class/drm/cardN/device/gpu_busy_percent` — 100 % busy at full sclk means neither throttle nor starvation. +6. **Quantify the collapse from counters, not vibes.** `radiance:engine_step_seconds_decode_total` ÷ `radiance:engine_steps_decode_total` = mean decode step time; diff two scrapes (Δseconds ÷ Δsteps) for the current rate. Compare to the shape's recorded baseline in the endpoint's vault page (15.5 ms at frank-archon's 4×200000 fp8 shape; 209 ms = 13× collapse). + +## Proof of fix (same-request A/B, not a health check) + +Submit one fixed request before and after the restart and diff the engine's own `timings` object — an 8-token probe measures only first-token noise, use `max_tokens>=128`: + +``` +curl -sS -m 240 -o /tmp/q.json -X POST :7770/v1/chat/completions \ + -H 'Content-Type: application/json' -d @/tmp/query.json # then read timings.predicted_per_token_ms / predicted_per_second / ttft_ms / draft_n / draft_n_accepted +``` + +Before/after on the identical request: 90.76 → 11.06 ms/token, 11.02 → 90.40 t/s, ttft 420 → 148 ms, acceptance unchanged (266/266). Then re-read the boot ledger (`elastic / kv cache / vram_weights / unclaimed / sampler rows`) and confirm it matches the page's recorded ledger — a successful restart proves nothing about the flags. Close with the restore oracle: `is-active` **and** `is-enabled` **and** no `/run/systemd/system/` mask symlink **and** container owned by the unit **and** `DropInPaths` empty. + +## Law + +A wedged radiance engine has **no in-place recovery** (its own wording: "a queue the driver evicted and could not restore") — `systemctl restart` is the only cure, and the fix must be proven by a rate, never by `/health`. Durable gap: no alert fires on a live-but-stalled endpoint; the missing rule is a decode-step-time ratio gated on `up==1` in `shared/prometheus/rules/llm-fleet.yml`. diff --git a/agent/memories/sharpshooter/dvcursor-run-33tna8tpxqqm8/state.json b/agent/memories/sharpshooter/dvcursor-run-33tna8tpxqqm8/state.json new file mode 100644 index 0000000..3ecf2c8 --- /dev/null +++ b/agent/memories/sharpshooter/dvcursor-run-33tna8tpxqqm8/state.json @@ -0,0 +1,4 @@ +{ + "v": 1, + "lastConsolidatedAt": 1793417535449 +} diff --git a/agent/memories/sharpshooter/serverAppConfig-eu5lc7uqff33/state.json b/agent/memories/sharpshooter/serverAppConfig-eu5lc7uqff33/state.json index e35771d..62768d7 100644 --- a/agent/memories/sharpshooter/serverAppConfig-eu5lc7uqff33/state.json +++ b/agent/memories/sharpshooter/serverAppConfig-eu5lc7uqff33/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1793408441828 + "lastConsolidatedAt": 1793419146014 } diff --git a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json index ed7f006..77398b3 100644 --- a/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json +++ b/agent/memories/sharpshooter/wt-20261027-142335-6d1fc1f-1hpiehxnq73s0/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1793417334882, + "lastConsolidatedAt": 1793420934894, "lastResult": { "at": 1793403514528, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261030-163847-6d1fc1f-p5mp3j807mvf/state.json b/agent/memories/sharpshooter/wt-20261030-163847-6d1fc1f-p5mp3j807mvf/state.json index 3c44b4d..281b094 100644 --- a/agent/memories/sharpshooter/wt-20261030-163847-6d1fc1f-p5mp3j807mvf/state.json +++ b/agent/memories/sharpshooter/wt-20261030-163847-6d1fc1f-p5mp3j807mvf/state.json @@ -1,10 +1,10 @@ { "v": 1, - "lastConsolidatedAt": 1793417207358, + "lastConsolidatedAt": 1793420867382, "lastResult": { - "at": 1793405994161, + "at": 1793418430591, "sessions": 1, - "deltas": 2, + "deltas": 1, "model": "frank-zealot" } } diff --git a/agent/memories/sharpshooter/wt-20261030-164316-1b9ba40-gg5su7w1ks2p/state.json b/agent/memories/sharpshooter/wt-20261030-164316-1b9ba40-gg5su7w1ks2p/state.json index 57eb15a..b995a7f 100644 --- a/agent/memories/sharpshooter/wt-20261030-164316-1b9ba40-gg5su7w1ks2p/state.json +++ b/agent/memories/sharpshooter/wt-20261030-164316-1b9ba40-gg5su7w1ks2p/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1793417116362 + "lastConsolidatedAt": 1793421016379 } diff --git a/agent/memories/sharpshooter/wt-20261030-180042-6d1fc1f-3s0rhbwqcda42/state.json b/agent/memories/sharpshooter/wt-20261030-180042-6d1fc1f-3s0rhbwqcda42/state.json index f7e2c1f..bb6a9d2 100644 --- a/agent/memories/sharpshooter/wt-20261030-180042-6d1fc1f-3s0rhbwqcda42/state.json +++ b/agent/memories/sharpshooter/wt-20261030-180042-6d1fc1f-3s0rhbwqcda42/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1793417202830 + "lastConsolidatedAt": 1793417502833 } diff --git a/agent/memories/sharpshooter/wt-20261030-205906-6d1fc1f-34xg0pon1v54z/state.json b/agent/memories/sharpshooter/wt-20261030-205906-6d1fc1f-34xg0pon1v54z/state.json new file mode 100644 index 0000000..3eecb56 --- /dev/null +++ b/agent/memories/sharpshooter/wt-20261030-205906-6d1fc1f-34xg0pon1v54z/state.json @@ -0,0 +1,4 @@ +{ + "v": 1, + "lastConsolidatedAt": 1793421007031 +}