diff --git a/agent/managed-skills/docs-dedupe-verify-campaign/SKILL.md b/agent/managed-skills/docs-dedupe-verify-campaign/SKILL.md index 82aaefc..b77a6f7 100644 --- a/agent/managed-skills/docs-dedupe-verify-campaign/SKILL.md +++ b/agent/managed-skills/docs-dedupe-verify-campaign/SKILL.md @@ -1,35 +1,60 @@ --- name: docs-dedupe-verify-campaign -description: "Run an exhaustive docs dedupe + fact-verification campaign in dank/command-center: scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR." +description: "Run an exhaustive docs dedupe + fact-verification campaign (command-center or any OKF vault, e.g. franksign): scout cluster ledgers, override scouts with evidence, gate every commit, and sweep scaffolding before PR." --- -# Exhaustive docs dedupe + verification campaign (dank/command-center) +# Exhaustive docs dedupe + verification campaign (dank/command-center; transplants to any OKF vault — proven on franksign 2026-10-21, #64 2026-10-21) ## Frame -- Read `omp-docs/references/okf-spec.md` + root `omp-docs/index.md` first; inventory with `find omp-docs -name '*.md' | wc -l`. -- Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir). -- Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/ -b docs/-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees. +- Read `omp-docs/references/okf-spec.md` (or the vault's vendored OKF contract, e.g. `franksign-docs/read-only/okf-v0.2-spec.md`) + root `index.md` first; inventory with `find -name '*.md' | wc -l`. +- Run `scripts/kb-lint.sh omp-docs` baseline; index-coverage gate is `scripts/kb-coverage.py omp-docs` (each dir's `index.md` must link every sibling file/subdir). On franksign the whole mechanical gate is one seam: `bun tests/lib/okf-lint.mjs` (frontmatter, links, index coverage, version-once) — run it before and after. +- Branch off fresh `origin/main` in a NEW worktree (`git -C ~/.omp worktree add ~/.omp/wt/ -b docs/-slug origin/main`) — workspace policy blocks `git checkout -b` in session worktrees. In a session worktree, `git checkout -b` is intercepted anyway: commit on the session branch, then named-ref push `HEAD:refs/heads/docs/-slug` (verify `git log origin/main..HEAD` shows ONLY your commit first). - Open the Gitea issue first (multi-moving-parts law); branch name `docs/-`. ## Scout fleet (overlap detection) -- One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task (skills trees, llama trees, pi vs packages, niri, sleep/power, core, gitea, gitea+cross-cutting sweep). Cap 2 concurrent; queue rest. -- Ledger format per cluster → `local://dedupe/.md`: inventory table (file | lines | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE), overlaps, facts-to-keep, inbound links, staleness. -- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. NEVER apply a scout verdict without re-checking the file, the tests, and the changelog yourself. +- One `task` call, `tasks[]` batch, agent `scout`, read-only; one cluster per task. **Fan out 2-3 clusters at a time, never seven**: `task.maxConcurrency` is 2 on this fleet, so the excess only queues, and `proc://` reports queued tasks as `running` — identical uptimes with zero deliveries means thrash, not progress (#64: seven scouts, nothing in 17 min; killing the five redundant ones let the two survivors land within minutes). +- Do NOT scout a cluster you have already audited inline. Kill the redundant jobs early with `write proc:///kill` — the endpoint is the scarce resource, and your own session shares it. +- Ledger format per cluster: inventory rows (file | verdict KEEP|MERGE|DELETE|VENDOR|UPDATE | evidence line/command). **Demand the ledger as the yield TEXT, never a `local://` file reference**: #64 produced one 0-byte ledger plus three files whose names contained the scout's own serialized parameter block (`gitea-caddy-inventory.md` + newline + a parameter tag), recoverable only by `ls`-ing `~/.omp/agent/sessions//local//`. +- **Scouts lie.** Verified misses in the 2026-10-19 campaign: fabricated ledger entries ("peer-reviewed BrowseComp" claim that exists in no file), a DELETE verdict for a generated artifact that is checked in by design AND under test (`tools/niri/keybindings.md`, see `scripts/gen-keybinds.sh` + `tests/gen-keybinds.bats`), and a KEEP for a leftover the scout misread its own changelog about. Franksign 2026-10-21: 2 of 3 cluster scouts fabricated — "7 missing component stories" refuted by the pairing-law seam that runs green on every push, invented entry/page counts, DELETE verdicts on live-box census numbers the purpose test KEEPS; a third hit the 30-min runtime cap producing zero output. #64 same day: three pages "missing an H1" that have one on their first heading line; a 215-line `omp-docs/AGENTS.md` that does not exist; a "wikilinks + 700-char wall" defect in `core/settings.md` that a commit from that same morning had already removed; `core/subagent-limits.md` called superseded while `agent/config.yml` carries the cited `maxRuntimeMs` / `softRequestBudget` / `maxRecursionDepth` values verbatim. +- **Four pre-checks before a scout row enters the diff**: (1) the file exists; (2) the named heading/marker is actually absent (`grep -c '^# '`, `grep -c '^# Citations'`); (3) `git log --oneline -3 -- ` — a same-day commit may already have fixed it; (4) if the claim is "config supersedes the doc", grep the config for the cited keys. +- Cluster sizing: for a vault under ~20 pages, scout fleets are net-negative — audit inline yourself (reading 5 pages beats babysitting a scout). Reserve scouts for multi-hundred-page trees. + +## Mechanical sweeps that pay (run on every vault) +- Empty-text links: `grep -rnE '(^|[ ,])\]\(' ` — a generated "See Also" pass left five `- ](index.md)` links (#64); rendered, an empty bullet reads as style, never as a broken reference. +- Citation convention census BEFORE adding anything: count `grep -rl '^resource:'` against `grep -rl '^# Citations'`. command-center's convention is `resource:` frontmatter (12 vs 3) — "add Citations to six pages" is a convention change, not a fix; only the pages missing `resource:` were real hits. +- Long-line walls: `awk 'length>300 {print FILENAME":"NR"("length")"}'` — reflow prose to ~100 columns, wording intact; leave tables (they cannot wrap) and the text of verbatim blockquotes alone. +- Dated log headings: three `## YYYY-MM-DD` sections for one day is three logs wearing one date. Collapse to one `##` plus one `###` per campaign, and write the shape into the spec page so nobody has to infer it. ## Fact verification -- Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32). +- Fleet facts: read-only probes over `ssh -o BatchMode=yes dankclaw|dankserver` (podman ps/inspect/top, stat, /etc/os-release, curl localhost APIs, `podman exec --version`). Live state beats docs even when docs claim recent verification — the 2026-10-18 "audit" still drifted (zealot 27B→9B, build 10754→10777, container census 14→32; #64 then found `8C/16T` where SMT is off, a "masked" unit that is simply absent, Pi-hole missing from a DNS list, and a weekly-drifting engine count that became a size class with `/config` named as the only authority). +- Local box facts too: `pacman -Qs ` / `command -v` before trusting a page's "installed tool" claim (#64: `wlr-randr` documented as available, never installed). +- Unit health is not endpoint health: an `:7770` that answers `/v1/models` while its systemd unit sits inactive `Result=exit-code` means a container escaped its unit. Report the drift to the operator; never record transient state as docs, and never touch a production unit without a per-run OK. +- Code-bound facts (franksign-style): every doc claim about routes/columns/shipped status gets grepped against `apps/*/src`, `packages/db/migrations/` — docs narrating features that never shipped ("page ships in the slice" while only the API landed) get truthed with a status pointer to the owning issue. - External claims: web_search/web read yourself (no subagents per Frank's rule). Kill fabrications outright (invented distro `FreeToken`, nonexistent `ornith-llama-cpp-setup` tree, unattributable "44.5% peer-reviewed CUA") — delete/drop with an explicit note, never keep "plausible" numbers. - Policy drift (live ≠ tuned docs): flag in the doc + PR body as an operator decision; NEVER silently re-apply policy. +- Deletion proposals from scouts die by default: archived trees carrying an honest archive banner (`pi/**`, banner 2026-10-18) stay, a directory `index.md` cannot be deleted (index-coverage gate), and incident logs are laws, not bookkeeping. Deletion is a pruning session with Frank present. + +## OKF §9 / frontmatter audit (cheap wins, check every vault) +- Log entries filed under the wrong ISO-date heading (crossed midnight): cut the entry (`CUT N.=M @reg`), insert the new date heading above the old one (newest-first headings), paste the entry under it. +- Missing trust family (`generated` on machine-written pages) — lint may not catch it; grep every frontmatter block. +- Laws living only in `log.md`: promote each to its owner page (mechanism law → architecture page owning the thing), leave the log entry as the incident record; the HANDOFF/README boundary then points at the vault instead of restating. +- A law restated in two tiers (vault spec page + always-loaded `AGENTS.md`) is not a duplicate to cut: name the co-ownership in the spec page ("change one, change the other") instead of silently dropping one copy. ## Log compaction (log.md) - Entry law: dated entry = what changed + durable lesson + concept-page pointer; narratives recoverable via `git log --follow`. Record the law at the top of the log. Tag historical-era entries. +- One `## YYYY-MM-DD` per day, `###` per campaign. Put the fan-out and ledger-verification lesson in the entry — that is what stops the next campaign from trusting a scout. ## Gates + landing -- Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands. +- Gate every commit: `bash scripts/kb-lint.sh omp-docs && python3 scripts/kb-coverage.py omp-docs && bats tests/`. Pre-commit hook prints `error: command not found: caddy` on Caddyfile-touching commits — warning only, commit still lands. On franksign: kill the :3000 demo stack first (a live API hangs the gate ~10 min; law lives in `franksign-docs/architecture/testing.md`), then `bun tests/lib/okf-lint.mjs && tests/run.sh docs`. - Before opening the PR: `git diff --stat origin/main..HEAD | grep -i 'tmp\|scratch\|\.bak'` — scaffolding merged under a `.tmp-` name in the 2026-10-19 campaign and needed a follow-up PR to promote. -- PR via API (skill `gitea-api-pr-operations` for token-extraction + merge payload); `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue. -- After Frank's merge OK: propagate deployment `git -C ~/.omp fetch && git merge --ff-only origin/main`; reap campaign worktree (`git worktree remove --force`) + delete branch local/remote; close issue with landing comment. +- PR via API (skills `gitea-api-pr-operations`, `gitea-merge-form-encoded`): token through a 600-perm `-K` curl config, never `-H` (`/proc//cmdline` is world-readable); create/PATCH = JSON WITH Content-Type, merge = form body with NO Content-Type, `do=merge&merge_style=no-ff`, success = HTTP 200 empty body verified by landed state, not by re-reading the PR. `shred -u` the cfg and payload afterwards. `xdg-open` the PR the moment it opens; comment the campaign ledger on the issue — ownership-move table, re-read facts with the command that read them, and rejected findings with their evidence. +- After Frank's merge OK: propagate the deployment with `git -C ~/.omp fetch && git merge --ff-only origin/main`, but first prove the FF cannot touch live machine-tier state: `comm -12 <(git -C ~/.omp status --porcelain | grep -v '^??' | awk '{print $2}' | sort) <(git diff --name-only HEAD..origin/main | sort)` must print nothing, and re-check the dirty count after the merge. +- Worktree reaping is the live-sync heartbeat's job (`wt/*` older than 7 days) unless Frank says otherwise — no `rm`, no deleting a session branch you did not create. -## Line-edit op syntax trap (bit three times) +## Line-edit op syntax trap (bit five times) - Valid ops: `PUT N.=M:` (body `+text` lines), `CUT N.=M` / `CUT N.=M @reg`, `PUT N:` / `PUT N:` or `PUT ]}` + - replace: `PUT /issues/{n}/labels` with `{"labels":[ids]}` (works without Content-Type) + - remove: `DELETE /issues/{n}/labels/{id}` +2. **Any JSON body without `header = "Content-Type: application/json"` in the curl config** — e.g. closing with `PATCH {"state":"closed"}` returns **201 and no-ops**; identical request with the header closes. curl's default `-d` type (`application/x-www-form-urlencoded`) makes this silent. + +## Rules +- Mutating JSON calls: always send `Content-Type: application/json` alongside `Authorization: token` in the same `-K` config file. +- **Never trust the status code for mutations — follow up with `GET` and read the field** (`labels`, `state`). Both traps look like success otherwise. +- Token stays out of `-H` args (world-readable `/proc//cmdline`): extract from `~/.config/secrets/gitea` into a 600-perm curl config, use `-K`. + +## Quick verification idiom +```bash +curl -sK "$cfg" "http://192.168.0.143:3000/api/v1/repos/dank/franksign/issues/22" \ + | jq '{state, labels: [.labels[].name]}' +``` + +Franksign's vault page `franksign-docs/agents/issue-tracker.md` owns the repo-local copy of this law; this skill exists so other repos (serverAppConfigs, command-center) don't re-burn the cycle. diff --git a/agent/managed-skills/halogen-npu-xrt-third-prefix/SKILL.md b/agent/managed-skills/halogen-npu-xrt-third-prefix/SKILL.md new file mode 100644 index 0000000..d94e59e --- /dev/null +++ b/agent/managed-skills/halogen-npu-xrt-third-prefix/SKILL.md @@ -0,0 +1,44 @@ +--- +name: halogen-npu-xrt-third-prefix +description: "Triage a halogen frank-dragoon (flash-server) unit crash-loop caused by the NPU lane: signature \"No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2\", dead-lane-kills-container law, third-prefix mount fix, and the FRANK_DRAGOON_ENV_FILE scratch-boot proof recipe." +--- + +# Halogen NPU lane: dead lane kills the container; XRT needs THREE mount prefixes + +Friction-earned 2026-10-21 on `frank-dragoon` (Dankclaw): unit boot-looped 88× (`activating`, exit 1 in ~200 ms) with the GPU/serve_api lines looking healthy; chat dead beside a fine-looking ledger. + +## Signatures (in order of appearance) + +``` +halogen-npu: No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2' +halogen npu: the NPU engine exited before it was ready (rc=1; ...) +``` +then `systemctl show -p ExecMainStatus` = 1, `NRestarts` climbing, and `/home/dank/logs/frank_dragoon.log` truncated at ~10 lines each boot (serve script truncates the log per start). + +**Laws:** a dead NPU lane is NOT survivable — the engine supervisor takes the WHOLE container down on it. And `Restart=always` makes each restart race its own GPU/NPU debris: the FIRST failure's message is the only real evidence; later loop entries are noise. Stop the unit immediately (`sudo systemctl stop frank-dragoon`) before probing. + +## Root cause class + +The NPU child `dlopen`s XRT from `/opt/xilinx/lib//` IN ADDITION to `/opt/xilinx/xrt/lib` and the install path. Host package churn (dpkg.log reconfigure of `libxrt2` the previous day) made the child demand the third prefix; our mounts filled only two. A lane that worked for days with byte-identical mounts can die purely from host package changes — diff the mounts last, check `grep libxrt /var/log/dpkg.log` first. + +Fix: mount each REAL file (never a symlink) at all three prefixes; derive the multiarch name from `FRANK_DRAGOON_XRT_DIR`'s basename (`MULTIARCH="${XRT_DIR##*/}"`). + +## Proof recipe (production-shaped, no scratch port) + +One card holds one server — never bench on a side port; use the sanctioned env-file seam: + +1. Scratch env on the host: `source /home/dank/serverAppConfigs/scripts/frank-dragoon.env` + knob overrides (e.g. `HALOGEN_NPU_MODELS=...`). +2. Copy the serve script with the candidate change applied via `sed` into `/tmp` (or edit + `FRANK_DRAGOON_ENV_FILE`); export `FRANK_DRAGOON_ENV_FILE=` and `FRANK_DRAGOON_LOG=/tmp/.log`; run under `timeout 1200`. The seam exists precisely for this. +3. Judge from `/tmp/.log`: `npu_api: ... buckets u0…u9` = lane alive; `No such library` = still broken. Then `curl :7770/health` → `npu.status`. +4. Retire the same session: `podman stop -t 60` (weights unpin slowly), `podman rm -f`, kill wrapper procs BY PID (`pgrep -f` the scratch serve script — `pkill -f` a pattern containing your own ssh line kills the session), move scratch files to `~/.AGENTTRASH/-/`, then prove `pgrep -f serve-` is empty. Leftover scratch state silently hijacks the next deploy (same class as leftover bench drop-ins). + +## Traps hit en route + +- Rootless `podman mount` fails ("cannot run in rootless mode") and leaves `$M` EMPTY — a later `grep -r "$M/..."` then reads the HOST cwd and "finds" the answer in Loki WAL files. Never grep an unverified mount var; `podman unshare` or `--entrypoint` probe instead. +- Image ships NO `/opt/xilinx` at all: all XRT comes from host mounts; `docs/NPU.md` is not inside the image. +- `ssh host 'tail '` runs on the DEV box if you forgot `ssh` — a missing-file error is not "the engine wrote nothing". +- Health `context` field echoes the RUNNING engine's ceiling — use it to prove which shape is live, unlike `/v1/models` name echo. + +## Verification that the fix landed + +`systemctl show frank-dragoon -p DropInPaths` empty; boot ledger shows `reserving the KV pool … (about 11.0 GiB)`, `… in all`, an `npu_api:` line, no clamp line; deployer prints `NPU lane verified — embedded`. diff --git a/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json b/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json index b725d97..cc203b4 100644 --- a/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json +++ b/agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792575105798, + "lastConsolidatedAt": 1792578705824, "lastResult": { "at": 1792573818108, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json b/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json index 305e7e9..032ed95 100644 --- a/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json +++ b/agent/memories/sharpshooter/wt-20261020-125416-6d1fc1f-3cjcsma2m7gk5/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792574961654, + "lastConsolidatedAt": 1792578921668, "lastResult": { "at": 1792574016725, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json b/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json index 379f3e6..69c3dca 100644 --- a/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json +++ b/agent/memories/sharpshooter/wt-20261020-144605-e7c60f6-31xsw83t7yjc6/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792575038705, + "lastConsolidatedAt": 1792578938739, "lastResult": { "at": 1792556461435, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json b/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json index 1340df5..e5c7a86 100644 --- a/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json +++ b/agent/memories/sharpshooter/wt-20261020-155034-1b9ba40-iffz605wjz2v/state.json @@ -1,6 +1,6 @@ { "v": 1, - "lastConsolidatedAt": 1792574970034, + "lastConsolidatedAt": 1792578870072, "lastResult": { "at": 1792566095222, "sessions": 1, diff --git a/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json b/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json index f2f29c6..79a1fb7 100644 --- a/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json +++ b/agent/memories/sharpshooter/wt-20261021-021543-6d1fc1f-o9a35ouamphx/state.json @@ -1,4 +1,4 @@ { "v": 1, - "lastConsolidatedAt": 1792575103668 + "lastConsolidatedAt": 1792578883678 }