dank/command-center
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

live: heartbeat sync 2026-10-19T14:21:41-07:00 (Refs #11)

Browse Source
This commit is contained in:
Frank
2026-10-19 14:21:41 -07:00
parent 36ff3742b4
commit 6e3e5594eb
24 changed files with 657 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/managed-skills/amd-apu-vulkan-tenant-onboarding/SKILL.md
+89
View File
@@ -0,0 +1,89 @@
---
name: amd-apu-vulkan-tenant-onboarding
description: "Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, the measured RADV heap gate, render-node resolution on hybrid hosts, build/runtime dependency surface, co-tenancy measurement."
---
Add a ggml/Vulkan inference tenant to an AMD APU (gfx1151 / Strix Halo iGPU) that already serves another engine: APU memory census, the RADV heap gate, the build/runtime dependency surface, and co-tenancy measurement.
Use when a ggml-based engine (llama.cpp, qwentts.cpp, anything built on ggml) must move from CPU to the **iGPU** of an AMD APU (gfx115x / Strix Halo / Ryzen AI) that **already serves a production tenant**. Answers: what budget actually exists, what gate decides feasibility before a model loads, and exactly which files a Vulkan build needs.
## Why the usual VRAM reasoning is wrong on an APU
`/sys/class/drm/card*/device/mem_info_vram_total` reports a small **carve-out** (2 GiB class), while the real working set lives in **GTT** — and GTT pages are *system pages*. A second GPU tenant therefore consumes host RAM from the same pool the first tenant and the OS draw from; there is no second memory to size against. The aperture is almost never the constraint; available RAM and shader time are.
## Census (read-only; run before proposing anything)
```bash
for c in /sys/class/drm/card*; do
v=$(cat $c/device/mem_info_vram_used 2>/dev/null); g=$(cat $c/device/mem_info_gtt_used 2>/dev/null)
[ -n "$v$g" ] && echo "$(basename $c) vram_used=$((v/1048576))MiB gtt_used=$((g/1048576))MiB vram_total=$(($(cat $c/device/mem_info_vram_total)/1048576))MiB"
done
cat /sys/class/drm/card*/device/mem_info_gtt_total # the aperture, in bytes
free -g # available is the number that bites
```
**Resolve the render node by driver, never by number** — on a hybrid host (dGPU + APU)
`renderD128` is often the *other* vendor's node:
```bash
for r in /sys/class/drm/renderD*; do
echo "$(basename $r) $(basename $(readlink -f $r/device/driver))"
done # gfx1151 example on a box with an RTX 3090: renderD128 = nvidia, renderD129 = amdgpu
```
Also check what the *userspace* offers before promising a backend:
```bash
dpkg -l | grep -iE "mesa-vulkan|vulkan-tools" # ICD present? probe installed?
ls /usr/share/vulkan/icd.d/ # radeon_icd.json = RADV available
ls <image>/usr/share/vulkan/icd.d/ # the ICD must exist INSIDE the container
```
## Gate 1 — the heap, before any model loads
Ask what heap the Vulkan driver hands a tenant on this APU, **in the same container image
that will host the engine**, with `/dev/dri` passed through:
```bash
podman run --rm --device /dev/dri -v <dir>:/probe:rw <engine-build-base> bash -lc '
apt-get update -qq && apt-get install -y -qq --no-install-recommends mesa-vulkan-drivers vulkan-tools
vulkaninfo --text > /probe/full.txt' # NOT --batch: that flag does not exist and prints help, rc=1
awk "/memoryHeaps: count/,/memoryTypes: count/" /probe/full.txt # sizes + budgets + DEVICE_LOCAL flag
```
Measured on gfx1151 (Strix Halo, 116 GiB aperture) with a production ROCm tenant already
loaded: `Radeon 8060S Graphics (RADV GFX1151)`, driver `radv`, API 1.4.318, heap 0
host-visible **38.67 GiB** (budget 32.58) and heap 1 `DEVICE_LOCAL` **77.33 GiB** (budget
**65.16**). Mesa splits the aperture roughly ⅓/⅔ on APUs, so **`DEVICE_LOCAL` is the
aperture, not the 2 GiB carve-out** — the "if the heap is the carve-out, give up" branch
does not trigger on this silicon. ggml asks for `DEVICE_LOCAL`, so a few-GiB engine lands
inside a tens-of-GiB budget and memory stops being the question. Both heaps are GTT: the
same host RAM, so a budget is a ceiling on *mappings*, not free silicon memory — `free`
still decides whether it can be touched.
Two probe traps: `vulkaninfo --batch` is not a flag in Ubuntu 24.04's `vulkan-tools` (it
prints the help page and exits 1), and a slim container has **no `nvidia_icd.json`**, so
Vulkan device 0 happens to be the iGPU there — an image that grows an NVIDIA ICD silently
reorders the list. Select by device *name*, never by index.
This is enumeration: no allocation, no compute — but it opens the render node on a
production card, so it is still a decision for the operator, not a routine probe.
## Gate 2 — contention, measured on the co-tenant
If the target shape is **co-tenancy** (both resident), the honest measurement is the *existing* tenant's TTFT and decode with the new tenant's work in flight — one request each, no ladder. Stopping the co-tenant is for benchmarks, not for a deployment shape. Expect compute-unit sharing, not memory pressure.
## The dependency surface people miss
1. **Build stage needs more than the flag.** ggml compiles its Vulkan shaders at build time: `-DGGML_VULKAN=ON` alone fails at configure without `libvulkan-dev` **and** `glslang-dev` (or `glslang-tools`). A CPU-only image's build stage typically installs only `cmake g++ git make`.
2. **The runtime stage needs an ICD, not just a device node.** `mesa-vulkan-drivers` must be installed *in the image*; `/dev/dri` without `radeon_icd.json` in the container buys nothing.
3. **Rootless/quadlet device access**: `AddDevice=/dev/dri` (or the iGPU's own node — see the census: it is not necessarily `renderD128`) + `GroupAdd=keep-groups`, and the host needs the world-writable udev rule (`KERNEL=="renderD*", MODE="0666"`) — systemd user-unit spawns carry no supplementary groups, so the render group never arrives.
4. Runtime selection is a device flag on the engine (`--device Vulkan0` for ggml-based tools), not an env var — and confirm which physical device that index resolves to after the image gains its ICD.
## Laws
- Price the tenant from a **booted** footprint, never from `weights + KV + buffers`: on an APU the compute buffers and any projector ride on GTT, which is host RAM — check `free` after the load, not before.
- Never treat the `vram_total` carve-out as capacity, and never subtract a "dedicated" figure from it to find headroom.
- Don't reason from `nvidia-smi`-style intuition: on this platform the useful numbers are `mem_info_gtt_used`, `free`, and RADV's heap budgets.
- Unheld fabric clock (`pp_dpm_fclk` = `auto`) is a hazard only when the NPU joins too; a pure iGPU pair is unaffected — check it before adding a third tenant.
- Swap on a host with GPU tenants protects co-tenants; paging a KV pool is worse than the OOM it prevents.
agent/managed-skills/dankvoice-silence-triage/SKILL.md
+60
View File
@@ -0,0 +1,60 @@
---
name: dankvoice-silence-triage
description: "Triage silent or late dankvoice auto-speak on the fleet: arming key, /say grammar, endpoint/sink, event granularity, humanizer latency, plus the black-box bats driver."
---
# Triage "dankvoice is silent" on the fleet
Symptom that recurs: auto-speak only speaks when `/say <text>` is typed
explicitly. Work the gates in this order; each has an objective check.
1. **Armed?** `cat ~/.omp/cache/dankvoice.json | jq .auto` — must be `true`
(or `DANKVOICE_AUTO=1`). Arming is read at **startup only** and `/say on`
never persists, so a session started before the key was added is silent even
though fresh sessions work. `/say start` is *not* a subcommand: unknown
arguments are spoken verbatim (it read out the word "start"). `/say on` is
the verb; `/say status` must print `auto-speak on`.
2. **Endpoint + sink?** `/say status` must show a real endpoint (not
`NO ENDPOINT`) and the saved sink must exist:
`pw-dump | grep -o 'alsa_output\.[^"]*' | sort -u`.
3. **Did it synthesize?** `pgrep -a pw-play` during a turn: one process per
segment, `--target <sink>`, wav under `/tmp/dankvoice-*/say-<epochms>.wav`.
One temp **directory per utterance chain** — count dirs to count utterances,
pids to count segments.
4. **Event granularity (the structural cause).** `agent_end` fires once per
*prompt* when the loop settles and is skipped when the harness maps a
non-terminal settle (`willContinue`) — long runs with background jobs or
subagents therefore narrate nothing. Mid-run narration needs `turn_end`
(fires per assistant message + tool results). Verify delivery, don't infer
it: skill `omp-extension-event-probe`.
5. **Latency is not silence.** Narration passes through the humanizer
(frank-zealot) before synthesis, so an utterance lands several seconds after
its turn; under endpoint load that stretches to tens of seconds.
`/say humanize off` = mechanical voice, near-instant. Judge cadence from
probe-log timestamps, never from when audio starts.
## Design rules the plugin now encodes (dank/dankvoice#2→#3)
- Narrate `turn_end` per turn **plus** `agent_end` for the settled answer;
dedupe identical text per run (reset on `agent_start`) so the answer speaks once.
- A turn landing while the player is alive is **dropped** — no queue, no
interrupting; a tool loop would cut every sentence off after a word.
Busy test: `state.child !== undefined && state.child.exitCode === null`.
- Tool-only turns (no prose, no narration-flagged thinking) stay silent.
- `print`/`json` modes stay silent, so benches never chatter.
- Settings writes must **round-trip the file** (`{...onDisk, ...owned keys}`).
Rebuilding from owned keys deleted `"auto"` on any menu choice and silently
un-armed every later session — the classic cause of "it worked yesterday".
## Black-box test harness (repo `dank/dankvoice`)
`tests/behavior.bats` drives the built artifact through an in-process driver:
`const say = DV_EXT(ctx, cfg)[0]`, handlers via `const events = new Map();
const pi = { on: (n,h)=>events.set(n,h) }`, then `events.get("turn_end")?.({
turnIndex:0, message:{role:"assistant",content:[{type:"text",text:"…"}]},
toolResults:[]}, ctx)`. Assert on `CALL`/`PLAYS`/`NOTIFY` lines and
`SETTINGS <file contents>`; stubs are `DV_PLAYER_CMD` (logs voice+input, honours
`DV_HOLD`) and `DV_REWRITE_URL` (404 → falls back to mechanical text). `setup()`
runs `bun run build` per test → serialize suites on the shared `dist/`.
`tests/packaging.bats` requires `dist/` **committed** (compares to HEAD), so
build → commit → the drift pin goes green.
agent/managed-skills/fixture-delay-window-race-test/SKILL.md
+35
View File
@@ -0,0 +1,35 @@
---
name: fixture-delay-window-race-test
description: Pin a race/abort bug in a fixture-server test by delaying only the Nth request so the interruption lands inside the in-flight window; assert via a driver-installed unhandledRejection reporter.
---
# Reproducing a timing-window bug against a fixture server
Use when a defect only appears in a window — a promise created before it is awaited, a kill/abort landing mid-flight, a prefetch interrupted — and a fixture HTTP server answers instantly, so the window never opens. Earned in `dankvoice` issue #7 (`[Unhandled Rejection] AbortError` from `/say stop` during a pipelined synthesis).
## Why naive attempts pass vacuously
1. **Uniform delay on the fixture route** (`DV_LATENCY_MS` for every request) makes *all* requests resolve at the same time, so by the time the interruption fires nothing is in flight. The test goes green against the broken code — worse than no test.
2. Asserting on a log line count that also counts interruption noise: `dankvoice`'s driver prints `PLAYS <n>` as the number of **player log lines**, and the stub appends `KILLED` on SIGTERM — so a single playback with one interrupt reports `PLAYS 2`. Count the payload lines instead (`grep -c 'PLAYER .*\.wav$'`).
## The shape that works
- **Delay only requests after the first.** The first response gets the pipeline into playback; the second stays in flight, which is the window:
```ts
const spoken = received.filter((c) => c.url?.endsWith("/audio/speech")).length; // already pushed
const delay = spoken > 1 ? Number(process.env.DV_SYNTH_MS ?? 0) : 0;
if (delay > 0) setTimeout(answer, delay); else answer();
```
- **Make the interruption land inside it.** Player holds long (`DV_HOLD=2`), interruption at a fixed offset (`DV_STOP_AT=400`) with the delayed response far later (`DV_SYNTH_MS=1200`): margins wide enough that the abort is neither before the request nor after the response.
- **Turn unhandled rejections into observable output** in the test driver instead of letting the runtime crash on them — that is also exactly what the real harness does to the user:
```ts
process.on("unhandledRejection", (reason) => {
console.log(`UNHANDLED ${reason instanceof Error ? `${reason.name}: ${reason.message}` : String(reason)}`);
});
```
Then assert `[[ "$output" != *"UNHANDLED"* ]]` plus the concrete behavior (segments actually played, player killed).
- **Leak checks measure a delta, not an absolute:** count `dankvoice-*` temp dirs before and after the run and compare; an absolute `== 0` fails on another test's leftovers.
## Order
Write the scenario first and watch it fail **on the rejection assertion specifically** — if it fails on some other assertion (a count, a kill line), the window is wrong, not the bug. Then fix, then confirm red→green on that one assertion before running the full suite.
agent/managed-skills/gitea-merge-form-encoded/SKILL.md
+22 -8
View File
@@ -1,27 +1,41 @@
---
name: gitea-merge-form-encoded
description: "Use when a Gitea API merge on Dankclaw (192.168.0.143:3000) returns HTTP 422 \"[Do]: Required\" — the merge endpoint needs form-encoded fields, not the JSON body that PR creation accepts."
description: "Use when a Gitea API merge on Dankclaw (192.168.0.143:3000) returns HTTP 422 \"[Do]: Required\" — the merge endpoint needs form-encoded fields with the lowercase field name do, not the JSON body that PR creation accepts."
---
Gitea on Dankclaw (192.168.0.143:3000, `dank/serverAppConfigs`) — merge quirk, measured 2026-10-19.
Gitea on Dankclaw (192.168.0.143:3000) — merge quirk, measured 2026-10-19 on `dank/serverAppConfigs` and again on `dank/dankvoice`.
Creating a PR accepts a JSON body. **Merging does not**: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge` with a JSON body (`{"merge_style":"no-ff", "Do-Not-Reply-From":…, "Reply-To":…}`) or with those values as request headers returns `422 {"message":"[Do]: Required"}` — four variants of that all fail identically, so do not "fix" it by changing the style name or moving the headers around.
Creating a PR accepts a JSON body. **Merging does not**: `POST /api/v1/repos/{owner}/{repo}/pulls/{n}/merge` with a JSON body (`{"merge_style":"no-ff", ...}`) or with those values as request headers returns `422 {"message":"[Do]: Required"}` — four variants of that all fail identically, so do not "fix" it by changing the style name or moving the headers around.
**The field name is lowercase `do`.** `--data-urlencode 'Do=merge'` (capital D, which the error message spells) returns the same `422 [Do]: Required`. Do not retry with case variants — the web-UI form field is `do`.
Working call (form-encoded):
```bash
tok=$(sed -n 's|http://[^:]*:\([^@]*\)@.*|\1|p' ~/.config/secrets/gitea | head -1) # git credential.helper store file
curl -sS -X POST -K <(printf 'header = "Authorization: token %s"\n' "$tok") \
curl -sS -X POST -w 'http %{http_code}\n' -K <(printf 'header = "Authorization: token %s"\n' "$tok") \
--data-urlencode 'do=merge' \
--data-urlencode 'merge_style=no-ff' \
--data-urlencode 'Do-Not-Reply-From=chief@franksdank.cloud' \
--data-urlencode 'Reply-To=chief@franksdank.cloud' \
http://192.168.0.143:3000/api/v1/repos/dank/serverAppConfigs/pulls/<N>/merge
http://192.168.0.143:3000/api/v1/repos/dank/<repo>/pulls/<N>/merge
# 200 with an empty body == merged
```
- `do=merge` is the field the 422 was demanding (the web-UI form field name, not an API doc field).
- `merge_style` values are the dashed names: `no-ff`, `merge`, `rebase`, `squash`.
- Keep the token out of argv: `-K <(printf …)` feeds the `Authorization` header through stdin (a `-H "Authorization: token …"` lands in `/proc/<pid>/cmdline`, world-readable — the 2026-09-16 HF-token leak class).
- Verify by fast-forwarding a checkout (`git fetch && git merge --ff-only origin/main`) rather than re-reading the PR.
- Fleet law still applies: never merge without an explicit current OK from Frank, and open the PR URL in his browser (`xdg-open`) the moment it exists.
## Credential-store token extraction (same file, adjacent trap)
`~/.config/secrets/gitea` holds one line shaped `http://user:token@192.168.0.143:3000`. Splitting it with `awk -F'/'` puts the host in `$3`, not `$2`, and `awk -F'/' '{print $NF}'` yields the host+path, not the token — an empty/garbage token makes **private repos answer `404 not found`** (Gitea hides them from anonymous), which reads like a wrong URL rather than an auth failure. Extract with:
```bash
tok=$(sed -n 's|^https\{0,1\}://[^:]*:\([^@]*\)@192\.168\.0\.143.*$|\1|p' ~/.config/secrets/gitea | head -1)
echo "token length: ${#tok}" # 40 — print the length, never the value
```
Other API quirks measured the same day:
- `GET /pulls?state=open` can return an error object where `state=all` returns the list; `state=all` list entries may omit `head`/`base` — fetch `/pulls/{n}` for those.
- POST `/pulls` for a head/base pair that already has a PR → `409` naming `issue_id: N`; the PR number is that `issue_id`, so PATCH it instead of POSTing again.
- Issues and PRs share the issue index per repo, so `#36`-style numbers are repo-scoped: a footer fix aimed at `dank/command-center#36` 404s when pointed at `dank/dankvoice`.
agent/managed-skills/gitea-pr-api-cycle/SKILL.md
+37 -19
View File
@@ -1,29 +1,47 @@
---
name: gitea-pr-api-cycle
description: "Open/update/merge PRs and issues on the Dankclaw Gitea API (192.168.0.143:3000, dank/command-center) with the token-safe curl+jq sequence, including the {\"Do\":\"merge\"} merge payload and post-merge live-checkout fast-forward."
description: "Open/update/merge PRs and issues on the Dankclaw Gitea API (192.168.0.143:3000) with the token-safe sequence, the form-encoded merge field, the empty-token-returns-404 trap, and post-merge live-checkout fast-forward."
---
# Gitea PR/issue cycle — dank/command-center
# Gitea PR/issue cycle — Dankclaw API
Friction-earned 2026-10-18 across PRs #15–#20 and issue #18: three separate fumbles (jq trailer flag, 422 merge payload, silent -o/-w swallowing) codified once.
Friction-earned 2026-10-18 (PRs #15–#20, issue #18) and re-earned 2026-10-19 on `dank/dankvoice` PRs #4–#6: the merge payload claim below was wrong, and an empty token masqueraded as a missing repo.
## Where
- API base: `http://192.168.0.143:3000/api/v1/repos/dank/command-center/{pulls,issues}`
- Token: never inline, never echoed. Extract from the git credential file into a 600-perm temp curl config:
`tok=$(sed -nE 's#^http://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)`
`cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"` → `curl -K "$cfg"`; `rm -f "$cfg"` when done.
- Unauthenticated GET works for state reads (`GET /pulls/N` → `jq -c '{state,merged,merge_commit_sha}'`).
- API base: `http://192.168.0.143:3000/api/v1/repos/dank/<repo>/{pulls,issues}` — repo varies (`command-center`, `dankvoice`, `serverAppConfigs`); numbering is per-repo, so a footer citing `#36` means nothing without the repo.
- Token: never inline, never in argv (`-H "Authorization: token …"` lands in `/proc/<pid>/cmdline`, world-readable). Extract into a 600-perm curl config and hand it to curl with `-K`:
```bash
tok=$(sed -nE 's#^https?://[^:/@]+:([^@]+)@192\.168\.0\.143.*#\1#p' ~/.config/secrets/gitea | head -1)
[ "${#tok}" -ge 40 ] || { echo "token extraction failed"; exit 1; }
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: token %s"\n' "$tok" > "$cfg"
curl -sS -K "$cfg" … ; rm -f "$cfg"
```
- **Empty token answers `404 {"message":"not found"}`, NOT 401** — Gitea hides the private repo. A 404 on a path you have used before means auth, not a typo'd repo. The trap that produces the empty token: `awk -F'/' … '{print $NF}'` on the store line (`scheme://user:token@host/`) — the host is in `$3`, the match never fires, you get a silent empty variable. Always assert the length before POSTing.
- Multi-line bodies: build JSON in python (`json.dumps({"title":…, "body":open(f).read(),…})`) or `jq -n --arg b "$(cat b.txt)"`. NEVER pass `---` as a bare jq argument (parsed as a flag). Escaped quotes inside `python3 -c '…'` break on `\"` — write the script to a file or use a heredoc.
- Always capture status: `-o r.json -w '%{http_code}'`; a bare `-o` hides failures.
## Body construction (the two traps)
- Build JSON with `jq -n --arg t <title> --arg b "$(cat b.txt)" '{title:$t, head:<branch>, base:"main", body:$b}'`; write the body to a temp file first (`printf '%s' … > b.txt`). NEVER pass `---` as a bare jq argument — jq parses it as a flag (`unknown flag: ---`). The AGENTS.md attribution trailer (`---` + `cwd: … · session: …`) goes INSIDE the body string.
- Always capture status explicitly: `-o r.json -w '%{http_code}'` and print it; a bare `-o` hides failures (a 422 looked like success-with-no-output once).
## Create
`POST /pulls` with `{title, body, base:"main", head:<branch>}` — JSON is accepted here (unlike merge).
- **409 `pull request already exists for these targets [issue_id: N, …]`** — a PR for that head/base exists already. Common cause: the push message printed `/pulls/new/<branch>` and someone (Frank, or an `xdg-open` you ran) clicked Create. `GET /pulls?state=all`, find it, then `PATCH /pulls/N` with your title+body instead of re-posting.
- `GET /pulls?state=all` on this Gitea omits `head`/`base` per entry — read `/pulls/N` for those, or key off `number`/`user`.
- Issues: same shape, `POST /issues` `{title, body}`; `PATCH /issues/N` fixes a body retroactively (works for PRs too, since a PR is an issue).
## Merge (the 422 trap)
`POST /pulls/N/merge` requires body `{"Do":"merge"}` — `Do` is the Required field on this Gitea. `{}`, `{"DoNotMerge":false}`, `{"merge_style":…}` all return 422 (`"[Do]: Required"`). Success may return an EMPTY body — verify by GET afterwards (`state=closed, merged=true`), not by the POST response.
- Merge only with Frank's explicit per-PR OK (repo law).
- Post-merge the live deployment must follow: `git -C ~/.omp fetch -q origin && git -C ~/.omp merge --ff-only origin/main` (the checkout IS the deployment; branch is `main`).
## Merge (the 422 trap) — corrected
`POST /pulls/N/merge` needs **form-encoded** fields; the merge form field name is lowercase **`do`**:
## PR etiquette per AGENTS.md
- Immediately after opening: `xdg-open <PR-URL>` lands it in Frank's Firefox.
- Redesign mid-review: amend + `git push -f` the branch, then PATCH the PR body (`/pulls/N`, `{"body":…}`) so title/body match the net diff — verify `{number,state,mergeable}` in the response.
- Issue creation identical shape: `POST /issues` with `{title, body}`, expect 201; PATCH same endpoint `/issues/N` to fix bodies retroactively, then GET-readback verify.
```bash
curl -sS -X POST -K "$cfg" \
--data-urlencode 'do=merge' --data-urlencode 'merge_style=no-ff' \
http://192.168.0.143:3000/api/v1/repos/dank/<repo>/pulls/N/merge
```
JSON `{"Do":"merge"}` (what this skill used to claim) and header variants all return `422 {"message":"[Do]: Required"}`. `merge_style` ∈ `no-ff|merge|rebase|squash`. Success returns **200 with an empty body**; verify by landing a checkout rather than re-reading the PR. Merge only with Frank's explicit per-PR OK.
## Post-merge landing
- Command-center: the checkout IS the deployment — `git -C ~/.omp fetch -q origin && git -C ~/.omp merge --ff-only origin/main`.
- Package repos (`~/Projects/<pkg>`): same `--ff-only`, then reinstall (`omp plugin upgrade <name>`) and `cmp` the installed `dist/index.js` against the repo `dist/index.js` to prove the live artifact is the merged one.
- **A fast-forward that adds or removes a tracked `node_modules` changes it on disk.** Landing a PR that deleted the tracked symlink wiped the dev checkout's installed deps, and `bun run typecheck` then died with `tsc: command not found` (exit 127); `bun install` restored them. Check `ls -ld node_modules` after landing anything that touched it.
- Prune: `git worktree remove --force <path>` (force: untracked deps link), `git branch -D`, `git push origin --delete <branch>` (allowed: local Gitea).
## Attribution footer (AGENTS.md)
Every issue/PR body ends with `---`, then `cwd: <session cwd> · session: <session id>`. Read the real values, never invent an id: the session id is the suffix of the transcript dir name (`ls -t ~/.omp/agent/sessions/` → `…_01a155bb-…`), and `cwd` is that session's own working directory. `local://<name>.md` artifacts live under that same session dir in `local/` — resolve with `read local://<name>.md` (its header prints the absolute path) instead of guessing.
agent/managed-skills/gitignore-trailing-slash-symlink-contamination/SKILL.md
+41
View File
@@ -0,0 +1,41 @@
---
name: gitignore-trailing-slash-symlink-contamination
description: "Trailing-slash gitignore rules don't match symlinks, so worktree dependency symlinks get committed; detect with git ls-tree mode 120000, fix with git rm --cached."
---
A `.gitignore` entry written with a **trailing slash matches directories only**. A symlink of the same name is not a directory, so it slips past the rule, and `git add -A` in a `git worktree` checkout silently **tracks** it. Cost measured 2026-10-19: a machine-specific symlink (`node_modules -> /home/blank/Projects/dankvoice/node_modules`) landed in `main` with a merged PR.
Why worktrees are the trigger: a worktree has no `node_modules` (or other installed-deps dir) of its own, so the cheap way to make build/test work there is a symlink into the primary checkout. With `node_modules/` ignored, nothing complains.
## Detect
```bash
git ls-tree HEAD -- node_modules # mode 120000 == tracked symlink
git ls-files -s | awk '$1==120000' # every tracked symlink in the index
```
## Fix (branch, never on main directly)
```bash
git rm -q --cached node_modules # untrack; leaves the file on disk
```
plus relax the ignore rule to the slash-free form, with the reason inline so nobody re-adds the slash:
```
# Trailing slash would only match directories: a node_modules symlink (the
# shape `git worktree` checkouts end up with) slips past it and gets tracked.
node_modules
```
Then `git check-ignore -v node_modules` must print the rule, and the suite runs after (the symlink still resolves, it is just ignored).
## Prevent
In a fresh worktree, create the deps symlink **after** confirming the ignore rule has no trailing slash:
```bash
git check-ignore -v node_modules || sed -i 's|^node_modules/$|node_modules|' .gitignore
```
Same class for any ignored-by-convention dir that a worktree might provide as a link: `node_modules`, `vendor`, `.venv`, `target`, `dist` when built artifacts are ignored.
agent/managed-skills/land-merged-pr-with-blocked-tree-ops/SKILL.md
+27
View File
@@ -0,0 +1,27 @@
---
name: land-merged-pr-with-blocked-tree-ops
description: "Use when a merged PR must land in a checkout whose working tree is dirty and workspace policy blocks git switch/restore/stash/reset-to-commit: prove identity, commit forward, merge, verify."
---
# Landing a merged PR into a checkout with blocked tree operations
Shared-workspace policy can block `git switch`, `git checkout <branch>`, `git restore`, `git stash`, and `git reset <commit>` ("uncommitted changes other concurrent agents rely on"). A `git pull --ff-only` then aborts with *"Your local changes to the following files would be overwritten by merge"* — **even when the local content is byte-identical to the incoming commit**, because git compares the index, not intent. Observed twice on 2026-10-19 (dank/command-center, dank/dankvoice).
## Procedure
1. **Prove identity first — never assume it.**
`git fetch && git diff --stat origin/main -- <the dirty paths>` → **empty output** means the working tree already equals the merged result, so committing it forward destroys nothing. If it is not empty, stop: there is real local work; do not proceed.
2. **Commit forward** the identical paths (message says exactly that, e.g. *"land merged #N content; policy blocks restore/stash, so the identical copies are committed rather than discarded"*). Hooks still run.
3. **Merge, don't ff:** `git merge --no-edit origin/main` (ort resolves identical changes on both sides cleanly). `--ff-only` is impossible once step 2 created a commit.
4. **Untracked-collision case:** if the abort says *"Please move or remove them before you merge"*, an untracked local file occupies a path the merge adds. `mv` it aside, merge, then `cmp` the restored file against the copy — identical → drop the copy (if the copy came from `.AGENTTRASH`, delete it and `rmdir`, since the file is now tracked).
5. **Verify:** repo lint/tests, `git status --porcelain -- <paths>` empty, and `git rev-parse --short HEAD origin/main` equal **after a fresh `git fetch`**.
## Gotchas
- `git merge … | tail -3` masks the exit status — `tail` always exits 0 and an `&&` chain then continues past an aborted merge. Run the merge unmasked, or check `${PIPESTATUS[0]}`.
- Immediately after a push, `git rev-list --count main..origin/main` can report phantom commits-behind from a stale remote-tracking snapshot. Re-`fetch` and compare `rev-parse` before believing any drift.
- Do not push redundant commits into a *deployment* checkout without checking the repo's tier law: machine/live tiers commit straight to `main`, human/agent prose goes by PR.
## Never
No `--no-verify`, no `reset --hard`, no `checkout -- <path>`, no "temporary" stash: those are exactly the operations the policy blocks because they discard other agents' work.
agent/managed-skills/omp-extension-event-probe/SKILL.md
+73
View File
@@ -0,0 +1,73 @@
---
name: omp-extension-event-probe
description: "Prove which lifecycle events the omp harness actually dispatches to extensions, with a throwaway timestamping probe plugin, instead of inferring delivery from side effects."
---
# Prove harness→extension event delivery with a probe plugin
Use when an extension handler seems never to fire, or when you must know *when* an
event arrives relative to another (mid-run vs settle). Never infer dispatch from
the feature's side effects: async work in the handler (LLM rewrites, synthesis,
network) lags seconds and destroys the timing signal.
## Shape (no build step needed)
`/tmp/probe/package.json` — the installer validates the declared entry on disk:
```json
{
"name": "event-probe",
"version": "0.0.0",
"type": "module",
"exports": "./index.js",
"pi": { "extensions": ["./index.js"] }
}
```
`/tmp/probe/index.js` — append arrival times, not counts:
```js
import { appendFileSync } from "node:fs";
const LOG = "/tmp/probe.log";
const stamp = (line) => appendFileSync(LOG, `${new Date().toISOString().slice(11, 23)} ${line}\n`);
export default function probe(pi) {
pi.on("agent_start", () => stamp("agent_start"));
pi.on("turn_end", (e) => stamp(`turn_end turn=${e.turnIndex} toolResults=${(e.toolResults ?? []).length}`));
pi.on("agent_end", (e) => stamp(`agent_end messages=${(e.messages ?? []).length} willContinue=${String(e.willContinue)}`));
}
```
## Run
1. `cd ~/.omp/plugins && omp plugin link /tmp/probe` (link, never edit
`node_modules`), then `omp plugin doctor` → expect a `✔ plugin:event-probe` line.
2. Launch a **fresh** TUI session in a pty (`omp --cwd /tmp/probe-run`, `pty: true`).
Extensions load at startup only — an already-running session never sees the probe.
3. PTY gotcha: input sent before the splash clears is swallowed. Read the screen
first, then submit text and finish with a separate write containing a bare
`\r` (LF alone left the text sitting in the composer).
4. Drive a task with ≥2 tool rounds so turns and settles separate in time
(sequential reads work and need no approval; ask for steps "strictly one
after the other" or the model parallelises them into one turn).
5. Wait for a terminal line with an async bounded loop (`grep -q agent_end`,
`sleep 1` inside one backgrounded bash call) — never poll across turns.
## Read
`turn_end` timestamps strictly earlier than `agent_end` ⇒ per-turn delivery
works. `willContinue=true` on `agent_end` marks a non-terminal settle (harness
maps `isTerminal === false` to it) — handlers that `return` on it never speak.
## Teardown (mandatory)
`omp plugin uninstall event-probe`; `omp plugin doctor`; delete `/tmp/probe*`.
The link writes into `~/.omp/plugins/package.json` (tracked, live tier) —
uninstall restores it; confirm with `grep` for the probe name returning nothing.
## Ground truth read for free
Event union + payloads: `~/.omp/plugins/node_modules/@oh-my-pi/pi-coding-agent/dist/types/extensibility/shared-events.d.ts`;
the session→extension mapping lives in `src/extensibility/extensions/lifecycle-mirror.ts`
(`extensionEventFromSessionEvent`). Extension-level `turn_end` carries
`turnIndex`/`message`/`toolResults` but **no `willContinue`** — that field exists
only on `agent_end`, so "is this the last turn" cannot be decided from `turn_end`.
agent/managed-skills/omp-lifecycle-event-selection/SKILL.md
+27
View File
@@ -0,0 +1,27 @@
---
name: omp-lifecycle-event-selection
description: "Pick the right omp harness lifecycle event for narration/telemetry: agent_end is a settle, turn_end waits for tool results, message_end is earliest with content, after_provider_response carries no content."
---
# Choosing an omp lifecycle event (narration, telemetry, hooks)
Re-derived twice from the installed typings/sources (`@oh-my-pi/pi-coding-agent`, `@oh-my-pi/pi-agent-core`) while fixing dankvoice. Read this before wiring a handler; verify against the installed version's `src/extensibility/extensions/types.ts` and `pi-agent-core/src/agent-loop.ts` when precision matters.
| event | payload you can use | when it fires | traps |
|---|---|---|---|
| `after_provider_response` | `status`, `headers`, `requestId`, `metadata` **only** | response headers arrive, **before the stream body is consumed** | No message content at all: useless for anything that speaks or summarizes text. Right for latency telemetry, non-200/abort detection, `requestId` correlation. |
| `message_end` | full `message` (assistant, user, tool-result, gate-stop, error) | the moment that message is built (`agent-loop.ts`, assistant case) | Earliest event carrying assistant text. Fires for non-assistant messages too → gate `message.role === "assistant"`. Payload is cloned for extensions. |
| `turn_end` | `turnIndex`, `message`, `toolResults` | after that turn's **tool results** come back | Stale by the tool duration: a 14 s tool delays a `turn_end` trigger 14 s. Extension event has **no `willContinue`** (the `turnIndex` is numbered by the lifecycle mirror), so you cannot tell the terminal turn from here. |
| `agent_end` | `messages`, `willContinue`, `awaitingAsyncWork` | once per user prompt, when the loop settles | `willContinue: true` = the harness scheduled a continuation (auto-retry / empty-stop), not a user-visible settle — narrating it speaks an answer the model is about to redo. Silence-prone on long-running sessions. |
| `agent_start` | — | run begin | Use to reset per-run dedupe state, so a later answer with identical text is not swallowed. |
## Narration cadence pattern (what dankvoice ships)
- `turn_end` (or `message_end` if the tool-time lag matters) for mid-run commentary; `agent_end` only for the settled answer.
- Drop a mid-run trigger while audio/IO is busy — never queue, never interrupt.
- Dedupe the terminal text per run (`agent_start` resets), because the last turn and the settle carry the same string.
- Skip `ctx.mode === "print" | "json"` so benches stay silent.
## Prove delivery, don't infer it
Audio/UI timing cannot show which events arrive. Link a throwaway extension that appends timestamps per event and run one real session: `omp plugin link /tmp/probe` (package.json with `"pi": { "extensions": ["./index.js"] }`), prompt a task with a slow tool, read the log, then `omp plugin uninstall probe`. Confirmed live 2026-10-19: extensions receive `agent_start`, `turn_end`, `agent_end`; `turn_end` landed seconds before `agent_end`.
agent/managed-skills/omp-local-artifact-path/SKILL.md
+40
View File
@@ -0,0 +1,40 @@
---
name: omp-local-artifact-path
description: Resolve a local:// artifact to a real filesystem path (for scripts/curl) without guessing the session-slug directory.
---
`local://<name>.md` artifacts live in a per-cwd, per-session directory whose name you cannot guess. Three invented paths failed before this worked (2026-10-19).
Actual layout:
```
~/.omp/agent/sessions/<cwd-slug>/<timestamp>Z_<session-id>/local/<name>.md
```
`<cwd-slug>` is the working directory with `/` replaced by `-`, **keeping the leading dash** — `/home/blank/.omp/wt/wt-20261019-125509-7c8a99d` under `/home/blank` becomes `-.omp-wt-wt-20261019-125509-7c8a99d`. Dropping the leading `-.` is the mistake that produces `FileNotFoundError`.
## Get the real filesystem path
Ask the tool, don't reconstruct it — a ranged read of the artifact reports the resolved path in its header:
```
read local://pr-body-msgend.md:1-1 -> [~/.omp/agent/sessions/-.../local/pr-body-msgend.md#TAG]
```
Or locate it by basename (needs enough depth; `maxdepth 5` misses it — the file sits 6–7 levels down):
```bash
f=$(find ~/.omp/agent/sessions -maxdepth 7 -name 'pr-body-msgend.md' | head -1)
cp "$f" /tmp/prbody.md # stable path for the python/curl step
```
Copy to `/tmp` before handing the content to a script: one cheap command, and every later tool call stops depending on session-scoped paths.
## Related lookups in the same directory tree
- Session id for the current cwd — `~/.omp/agent/history.db` (prompts table has `session_id`, `cwd`; the column is `prompt`, **not** `text`, and raw `?q=` queries cannot be combined with a table selector or pagination):
```
read history.db?q=SELECT session_id, cwd, substr(prompt,1,50) AS p FROM history WHERE cwd LIKE '%wt-20261019%' ORDER BY rowid DESC LIMIT 3
```
Use the real id in issue/PR attribution footers; never reuse an id remembered from a compacted transcript.
- Transcript dir for the same session sits beside `local/`.
agent/managed-skills/parallel-worktree-path-hazard/SKILL.md
+22 -56
View File
@@ -1,68 +1,34 @@
---
name: parallel-worktree-path-hazard
description: "\"Use when several agents edit the same repo in different git worktrees at once, or when an edit/read may have landed in the wrong checkout: which path forms bind where, how to detect with git -C status, and the repair protocol (cp + git show HEAD blob write-back) because git checkout/restore is policy-blocked in a shared workspace.\""
description: "Use when several agents edit the same repo in different git worktrees at once, or when an edit lands in the wrong checkout: relative tool paths resolve to the session cwd, not the shell cwd."
---
# Concurrent-worktree path hazard (shared repo, many agents)
Use when several agents (or you across two checkouts) edit the same repo in different git worktrees at once, or when an edit/query lands in the wrong checkout.
Measured 2026-10-18 on `serverAppConfigs` with two agents live at once. The
failure is **silent**: the tool reports success, the diff looks right, and the
change lives in a *different* worktree.
## The silent failure: relative tool paths resolve to the SESSION cwd
## Which path form binds where
- **Parent/top-level session**: a **relative** `path` resolves against the
session CWD — its own worktree — never against the worktree named in the
prompt. This is how a change meant for `wt-tts-trial-docs` landed in
`wt-20261017-…`.
- **Subagent**: an **absolute** path to its own assigned worktree can resolve
into the parent's worktree when the same *relative* path exists there
(`edit` twice onto `dankclaw/.env.example`). Its `write` with the same
absolute path was fine — the bug is `edit`/`read` path resolution, not `write`.
`read` / `edit` / `write` / `grep` resolve a **relative** `path` against the **session working directory** — never against the `cd` inside a neighboring `bash` call, and not against a `../other-worktree/...` prefix either (that gets normalized back into the session tree). So after `cd ../wt-B && grep …` you can `read path="docs/page.md"` and get **wt-A's** copy.
Rules that hold:
1. A subagent passes paths as `../<its-worktree>/<rel>` (relative to the parent
worktree) for `read` **and** `edit`; the parent passes **absolute** paths to
any worktree that is not its own CWD.
2. After every edit batch, prove the landing site, do not trust the tool's echo:
`git -C <intended-worktree> status --short` must list the file. A file you
edited that is missing from that list went somewhere else.
3. Keep each work item on its own branch/worktree, and keep ONE integration
owner (the parent) for shared files — `scripts/deploy-remote.sh`,
`scripts/deploy-plan.py`, `tests/`, `serverAppConfigs-docs/index.md`,
`AGENTS.md` are the collision hot spots in this repo. Never rebase a
sibling's branch.
Why it survives "I just read it to check": the returned path is printed workspace-relative and the file name is identical in every checkout, so the output looks like a correct read of the tree you meant. Only content that differs between the checkouts betrays it.
## Repair protocol (no `git checkout`, no `git restore`)
`git checkout -- <path>` and `git restore <path>` are **policy-blocked** in a
shared workspace ("overwriting working-tree files … discards changes other
concurrent agents rely on"). Unstaging (`git reset [<path>]`,
`git restore --staged <path>`) is allowed but does not undo content. So:
Rules:
- **Absolute paths for every tool call that targets a worktree other than the session's.** No exceptions for quick reads.
- Confirm the checkout from content you know differs (a heading, a status line), not from the echoed path.
- A `read` error like "Did you mean .AGENTTRASH/<name>" is a signal the relative path resolved into the session tree — re-issue absolute.
- Same trap for `bash`: it starts each call in the session cwd, so `cd` twice per call or use absolute paths there too.
```bash
F="path/with spaces if any"
# 1. move the change to the worktree that should own it (both at the same base commit)
cp "<wrong-wt>/$F" "<right-wt>/$F"
# 2. undo in the wrong worktree by writing the committed blob back
git -C "<wrong-wt>" show "HEAD:$F" | tee "<wrong-wt>/$F" > /dev/null
# 3. prove both sides
md5sum <(git -C "<wrong-wt>" show "HEAD:$F") "<wrong-wt>/$F" "<right-wt>/$F"
git -C "<wrong-wt>" status --short # must be empty for that file
git -C "<right-wt>" diff --stat # must show the intended delta
```
## Landing a change when the session worktree is stale
Never fix it blind: if the wrong worktree had its own uncommitted edits to that
file, `cp`/blob-write-back would destroy them — check
`git -C <wrong-wt> status --short` FIRST and reconcile content, don't overwrite.
`git diff --name-only HEAD origin/main` first. If main moved a file you are editing, your edit's base is wrong even though the file "looks fine" — do not commit on the stale base.
## Tidy after a cancelled sibling
A killed subagent can leave a worktree plus an empty branch. Removing them is
safe only when verified empty: `git -C <main> log --oneline -3 <branch>` (no
unique commits) and `git -C <wt> status --short` (clean), then
`git worktree remove <wt>` + `git branch -D <branch>`. An orphan worktree at a
stale commit is the same class of trap as a leftover bench drop-in: the next
agent reads it as unfinished work.
1. `git worktree add ../wt-clean -b <type>/<slug> origin/main`; copy the new files in.
2. Re-apply the edit against the clean tree's version **with absolute paths**, then read it back absolutely to prove which tree changed.
3. Run the suite inside the clean worktree; commit there.
4. Remove your own duplicate edit from the session worktree by editing it away (`CUT`/`PUT`). `git checkout -- <path>` and branch switches are policy-blocked in shared workspaces and should stay blocked — they discard edits other agents rely on.
5. Never carry the same uncommitted edit in two worktrees at once; if you must keep both trees, keep a task that reconciles them.
## Report it
File the misresolution to `xd://report_issue` (false positives are fine) and
record the incident in the repo worklog — a repair nobody wrote down looks like
a hijack later.
## Multi-agent hazards in one shared worktree
- Another agent can stage, commit, or check out files under you; re-read a file immediately before editing it if any tool result hinted at intervening change.
- Untracked scratch you drop in a shared tree (reports, notes, `.AGENTTRASH/` entries) is visible to everyone and can be committed by them. Put throwaway artifacts outside the checkout.
- Never `git clean -f`, `rm -rf`, or branch switching to "get a clean state" in a tree another agent is using.
agent/managed-skills/ryzenai-generic-failure-loader-order/SKILL.md
+4 -4
View File
@@ -28,14 +28,14 @@ site-packages/flexml/flexml_extras/lib : site-packages/onnxruntime/capi : site-p
1. `ldd <deployment>/lib/libonnxruntime_providers_ryzenai.so | grep -E "libxrt_coreutil|libryzen_mm"` — the resolved path settles it alone: `/opt/xilinx/xrt/lib` = good, `.../voe/lib` = this bug.
2. `XILINX_XRT=/opt/xilinx/xrt /opt/xilinx/xrt/bin/xrt-smi examine` → `RyzenAI-npu5 | aie2p | 6x8` + `NPU Firmware Version`. Needs `libboost_filesystem1.83.0`, `libboost_program_options1.83.0`, `libuuid1` (26.04 names; the 24.04 docs' 1.74.0 does not exist). A "No such file or directory" for a file that exists = missing interpreter/dep, not a missing binary.
3. `strace -f -e trace=openat` — `/dev/accel/accel0` never opened ⇒ failure is pre-device ⇒ loader-level.
4. `gdb -q -batch -x <(echo 'catch throw'; echo run; echo 'bt 10')` style catch-throw loop → names `shim_loader` / `Allocator::Alloc`.
4. `gdb -q -batch` with `catch throw` + `bt 10` loop → names `shim_loader` / `Allocator::Alloc`.
Known dead ends (each costs one run to rule out, then stop): `--ulimit memlock`, XRT INI verbosity, `DD_LOG_LEVEL` / `RYZENAI_TRACE` / `RYZENAI_EP_TRACING` (print nothing before the throw), byte-scanning XRT `.so` for PCI IDs (ambiguous), libboost presence for the OGA path (only `xrt-smi` needs it), flag-by-flag container matrices (`--memory`, `--security-opt`, `--name`, extra mounts — all innocent).
Known dead ends (one run each, then stop): `--ulimit memlock`; XRT INI verbosity; `DD_LOG_LEVEL` / `RYZENAI_TRACE` / `RYZENAI_EP_TRACING` (print nothing before the throw); byte-scanning XRT `.so` for PCI IDs (ambiguous); libboost presence for the OGA path (only `xrt-smi` needs it); flag-by-flag container matrices (`--memory`, `--security-opt`, `--name`, extra mounts — all innocent).
## Verified facts on this stack
- Vendor XRT 2.25.37 userspace (`dpkg-deb -x`, never dpkg-installed) works against Ubuntu's **in-tree `amdxdna` 0.7.0`; no DKMS module needed. `1022:17f0` rev 11 = XDNA2, handled as `aie2p 6x8`.
- Vendor XRT 2.25.37 userspace (`dpkg-deb -x`, never dpkg-installed) works against Ubuntu's in-tree `amdxdna` 0.7.0; no DKMS module needed. PCI `1022:17f0` rev 11 = XDNA2, reported as `aie2p 6x8`.
- AMD's `quicktest` passing does **not** prove NPU use (it can silently run on CPU); trust `xrt-smi` + logits, not "it didn't crash".
- `/dev/accel/accel0` is `root:render 0660`; rootless podman cannot read it (userns drops the gid; `keep-groups`, `keep-id:gid=990`, `--group-add render` all fail) → rootful container, or a udev rule like the GPU nodes'.
- OGA 0.14 (RAI 1.8) contract: `GeneratorParams` has only `set_search_options` (no `input_ids`, no `model.generate`); feed with `Generator.append_tokens(ids)`, loop `generate_next_token()`, read `get_sequence(0)`; `max_length` counts **prompt + generated**. Full Fusion (`*_npu_4K`) caps input+output at 4096.
- Validate the *content*, not just the exit code: `np.argsort(logits[0])[-8:]`. `append_tokens` returning a real logits tensor proves the graph ran; if argmax is the sequence terminator (`
- Validate content, not just exit code: `np.argsort(logits[0])[-8:]`. A real logits tensor after `append_tokens` proves the graph executed (prefill is folded into step 1, so a sub-millisecond first step + a 1-token sequence means it did not); if argmax is the sequence terminator (`
agent/managed-skills/ryzenai-npu-oga-linux-spike/SKILL.md
+25 -7
View File
@@ -1,6 +1,6 @@
---
name: ryzenai-npu-oga-linux-spike
description: "Run a vendor-compiled AMD Ryzen AI (XDNA2) ONNX-GenAI model on a Linux Strix Halo host from public artifacts: staging, rootless NPU device-access wall, container recipe, and the NuTTS-air speech-token contract."
description: "Run an AMD-precompiled AMD Ryzen AI (XDNA2) ONNX-GenAI model on a Linux Strix Halo host from public artifacts: staging, rootless NPU device-access wall, container recipe, and the NuTTS-air speech-token contract (65,536-code vocab, 50 codes/s real-time line)."
---
Run an AMD-precompiled XDNA2 model (OGA / onnxruntime-genai) on a Linux Strix Halo host (Dankclaw-class) without installing the vendor stack on the host.
@@ -19,10 +19,12 @@ Tarball has 30 top-level entries: cp312/linux_x86_64 wheels (`onnxruntime_genai_
- **Rootless containers cannot read `/dev/accel/accel0`** (`root:render 0660`): the host gid never enters the userns gid map, so `--device` yields `nobody:nogroup`. Verified failing 2026-10-19: `--group-add keep-groups`, `--userns=keep-id`, `--userns=keep-id:gid=990` (process gets gid 990, node still unreadable), `--group-add render`, `--group-add gid:990` (both error). Options: run rootful (`sudo podman`, separate storage → rebuild the image rootful, ~1 min), or add a world-writable udev rule for `accel0` (the fleet already does exactly this for `kfd`/`renderD*` in `99-comfyui-gpu-access.rules`, whose header explains why: systemd user-unit spawns carry no supplementary groups).
- **Rootless image build trap:** the vendor installer's `tar -xf` preserves recorded uids (95777 etc.) and aborts under `set -e` with `Cannot change ownership ... Invalid argument`, silently skipping the `deployment`/`onnxruntime` extraction. Patch it in the Containerfile: `sed -i 's/tar -xf/tar --no-same-owner -xf/g' install_ryzen_ai.sh`.
- Installer usage: `./install_ryzen_ai.sh -a yes -p /opt/rai -c /opt/rai` (python **3.12** required; base `python:3.12-slim`; needs `linux-libc-dev zip pciutils patch libgomp1`). It builds a venv from the local wheels, reinstalls `onnxruntime-vitisai` then `--no-deps onnxruntime-genai-ryzenai`, installs `requirements_extras.txt` (transformers 4.57.6), patches `bin/activate` (sets `RYZEN_AI_INSTALLATION_PATH=$VIRTUAL_ENV` + `LD_LIBRARY_PATH` into site-packages), and untars `LLM/deployment/onnxruntime/voe/quicktest` into the cpp path.
- `xrt-smi` in a container needs `libboost-filesystem1.83.0` + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names) and `apt update` first; boost is **only** needed by `xrt-smi`, never by the OGA/EP path.
## HF artifact staging
- Gated (401 on API, 403 on git) for a plain account: `neuphonic/neucodec` (the **only** NeuCodec encoder), `neuphonic/neucodec-onnx-decoder`, `neuphonic/neucodec-onnx-decoder-int8`, `neuphonic/neutts-air`. `amd/neutts-air_rai_1.8.0_npu_4K` is public. Public mirrors of the codec exist (`eugenehp/neucodec` has the full 1.16 GB `pytorch_model.bin`) — treat a mirror of a gated release as a licensing decision for the operator, not a workaround to take silently.
- Gated-repo loader trap: `HF_HUB_OFFLINE=1` + `HF_HOME` pointed at a cache laid out as `hub/models--<org>--<repo>/snapshots/<rev>/…` works, but `NeuCodec._from_pretrained` **asserts the repo id** is in `["neuphonic/neucodec", "neuphonic/distill-neucodec"]` — a local directory path is refused outright, so pass the repo id and satisfy it from the cache.
- `github.com/neuphonic/neutts` is public and ships `samples/{dave,emily,greta,jo}.{pt,txt,wav}` = **pre-encoded reference codes + transcripts** → a spike needs zero gated bytes.
- Auth without leaking the token into `ps`: build a 600-perm credential file with a shell builtin and point git at it —
`umask 077; printf 'https://oauth2:%s@huggingface.co\n' "$(cat ~/.cache/huggingface/token)" > ~/.git-credentials-hf`
@@ -30,14 +32,30 @@ Tarball has 30 top-level entries: cp312/linux_x86_64 wheels (`onnxruntime_genai_
## NuTTS-air on XDNA2 (the model-side contract)
- The package is an OGA model dir: `genai_config.json` (`type: qwen2`, 24 layers, hidden 896, 14 heads / 2 KV, ctx 32768, `provider_options: [{"RyzenAI": {"hybrid_opt_token_backend": "npu"}}]`), plus `dd_metastate_*` AIE artifacts and `model.onnx` + `model.onnx.data`.
- **Speech tokens live in the text vocab**: `<|speech_0|>` = id 151671 (10 001 codes), with `<|TEXT_PROMPT_START|>` 151666, `<|TEXT_PROMPT_END|>` 151667, `<|TEXT_REPLACE|>` 151665, `<|SPEECH_REPLACE|>` 151668, `<|SPEECH_GENERATION_START|>` 151669, `<|SPEECH_GENERATION_END|>` 151670. Generation is a plain `og.Generator` loop; the graph's own `eos_token_id` is `[151645, 151643]`, so pass 151670 as an extra eos or stop on it manually.
- **Real-time line = 50 speech tokens/s** (`sample_rate=24000`, `hop_length=480`). Judge any NPU/iGPU TTS number against 50.
- The package is an OGA model dir: `genai_config.json` (`type: qwen2`, 24 layers, hidden 896, 14 heads / 2 KV, ctx 32768, `vocab_size` **217,652**, `provider_options: [{"RyzenAI": {"hybrid_opt_token_backend": "npu"}}]`), plus `dd_metastate_*` AIE artifacts and `model.onnx` + `model.onnx.data`.
- **Speech tokens live in the text vocab, and the codebook is 65,536 — not 10,001.** `tokenizer.json` `added_tokens` holds exactly 65,536 `<|speech_N|>` entries, ids **151,671–217,206** contiguous, over a base BPE vocab of 151,643; specials are 151,644–151,670 (`<|SPEECH_GENERATION_END|>` = 151,670). The vendor template maps a codec index straight to its id: `"".join(f"<|speech_{i}|>" for i in ref_codes)`. An encoded reference really does contain codes in the 60k range, so a 10k assumption silently truncates the space.
- The graph's own `eos_token_id` is `[151645, 151643]` (the **chat turn** stop) and `search.eos_token_ids` is unset — pass 151,670 as an extra eos or speech ends after one token.
- **Real-time line = 50 speech codes/s** — NeuCodec `sample_rate 24000 / hop_length 480`, measured exactly (49.96–50.00 across three 20–25 s references). Judge any NPU/iGPU TTS number against 50. (Qwen3-TTS, the fleet's deployed arm, is 12.5 codes/s — do not conflate the two bars.)
- Reference encode is cheap and cacheable: `librosa.load(sr=16000, mono=True)` → `[1,1,T]` float tensor → `codec.encode_code(audio_or_path=…)` (mirrors `NeuTTS.encode_reference`); CPU encoder RTF 0.28–0.38 on 8 threads. `neucodec` needs `torchtune` (`RotaryPositionalEmbeddings`) — install both `--no-deps --target <dir>` so a second torch never shadows the venv's.
- Build the prompt by *importing* the vendor function, never by copying the template:
`sys.path.insert(0, <neutts repo>); from neutts.neutts import NeuTTS`, then
`NeuTTS._apply_chat_template(SimpleNamespace(tokenizer=hf_tok, input_format="text"), ref_codes, ref_text, text, None)` — works on a duck-typed `self`, so no backbone/codec load. Template: `user\n<|TEXT_REPLACE|>\nassistant\n<|SPEECH_REPLACE|>`; splice `TEXT_PROMPT_START + encode(f"{ref_text} {text}") + TEXT_PROMPT_END` at TEXT_REPLACE, then truncate at SPEECH_REPLACE and append `[SPEECH_GENERATION_START] + encode("".join(f"<|speech_{i}|>"))`.
- Voice identity = reference codes + transcript (request-time conditioning). No trained artifact exists to preserve; packs holding only `reference.wav` require the gated encoder once, offline.
- Voice identity = reference codes + transcript (request-time conditioning). No trained artifact exists to preserve; packs holding only `reference.wav` require the gated encoder once, offline — **and** a transcript, which a voice pack does not carry (`manifest.json` holds provenance/crop geometry only) and a host without ASR cannot produce.
- Full-Fusion input ceiling: 4096 tokens **total**, input + output.
## Status caveat
## Status caveat (2026-10-19)
Measured on 2026-10-19 up to and including: staged artifacts, image build, device-access matrix, prompt contract. The token-rate measurement (does the vendor userspace drive the in-tree 0.7.0 driver, and does it clear 50 tok/s) is **not** captured here — re-measure, never trust a remembered rate.
Staged artifacts, image build, device-access matrix, prompt contract, reference pre-encode:
**done and reproducible.** Token generation on the NPU: **NO-GO, and the loader is the
wall.** The device is healthy (`xrt-smi validate` PASSED incl. `throughput`,
`RyzenAI-npu5 / aie2p / 6×8`, `pyxrt.device(0)` constructs), and `strace` shows the
failing load reading `genai_config.json`, opening `model.onnx`, loading
`libxrt_driver_xdna.so.2` and opening `/dev/accel/accel0` with successful GEM ioctls —
then the RyzenAI EP **declines the graph before touching any `dd_metastate_*`**, silently:
`GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`,
`set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT` all emit
zero lines. Hardlinked mirror dirs of the model tree were ruled out as shadowing. One run
loaded and produced finite logits; byte-identical runs never reproduced it. **No token rate
was ever measured, so quote none**, and remember `quicktest` success ≠ NPU residency (that
build falls back to CPU silently — prove residency via
`/sys/class/accel/accel0/device/mem_usage`).
agent/managed-skills/ryzenai-oga-model-dir-and-eos-override/SKILL.md
+116
View File
@@ -0,0 +1,116 @@
---
name: ryzenai-oga-model-dir-and-eos-override
description: "Debug OGA on XDNA2 when the RyzenAI EP never claims the If-rooted graph (shadow model dirs, bind-mounted configs, missing device), or a TTS/LLM graph stops on the wrong eos token."
---
# OGA on XDNA2: graph not claimed, or generation stops at once
Two independent failure families in the AMD `onnxruntime-genai` (OGA) + RyzenAI EP stack on
Linux XDNA2 (Strix Halo / Ryzen AI Max). Diagnose them in this order — they are separate
gates and each has its own instrument.
## Gate 0 — is the NPU path alive at all?
Run the vendor's own instrument before touching anything else:
```bash
cd /opt/rai/quicktest && RYZEN_AI_INSTALLATION_PATH=/opt/rai python quicktest.py
# want: "Setting environment for STX/KRK" ... "Session successfully initialized" ... "Test Finished"
```
It finds the device by shelling out to `lspci -nn` and grepping `1022:17f0`, so it also
proves `lspci` is present in the container. Pass ⇒ the in-tree `amdxdna` driver and the
extracted userspace pair correctly; stop blaming the host.
Separate **XRT enumeration** from **OGA's device probe** — they are not the same check:
```python
import sys; sys.path.insert(0, "/opt/xilinx/xrt/python")
import pyxrt; dev = pyxrt.device(0) # constructs => XRT sees the device
```
`pyxrt.device(0)` succeeding while OGA says `No RyzenAI devices detected` means the
refusal is OGA's own predicate, inside `libonnxruntime-genai.so` (the string lives there,
next to `RyzenAI`, `max_length_for_kv_cache`, `library_path`). EP-side env levers worth
pulling: `RYZENAI_EP_TRACING`, `RYZENAI_EP_PERFORMANCE_COUNTERS`, `RYZENAI_LLM_ASYNC_INIT`,
`RYZENAI_LLM_PROFILE_INIT`.
## Gate 1 — `invalid model … Node (If) cond_full`
This is **ONNX Runtime verifying the raw graph**, which happens when no provider claimed
it: the EP registered nothing and the CPU provider was left holding an `If` whose branches
mismatch. It is never a corrupted model.
```bash
LD_DEBUG=files LD_DEBUG_OUTPUT=/tmp/ld python loadprobe.py
grep -a "calling init:.*ryzenai\|undefined symbol" /tmp/ld.*
# observed: libonnxruntime_providers_ryzenai.so loaded, then
# symbol lookup error: undefined symbol: GetProvider (fatal)
```
`GetProvider` is not a dynamic symbol in the ORT cores, the shared-provider stub, or the
EP (`nm -D` shows neither defines nor wants it), so the failure is registration, not an
ELF mismatch. Confirm the cores are identical before chasing versions: both shipped
`libonnxruntime.so.1.27.0` copies hash equal.
Separate the two gates with an explicit global-scope load before importing OGA:
```python
import ctypes
ctypes.CDLL("/opt/rai/deployment/lib/libonnxruntime_providers_ryzenai.so",
mode=ctypes.RTLD_GLOBAL)
import onnxruntime_genai as og
og.Model(MODEL) # registration fixed => error MOVES to "No RyzenAI devices detected"
```
An error that **moves** proves you cleared one gate and exposes the next. Silence in the
first form means the provider list was empty.
### Dead ends (measured, do not retry)
- **Shadow model dirs never work.** Hardlink mirror, symlink mirror, and even a
read-only single-file bind mount of a patched `genai_config.json` over the export's own
`genai_config.json` all produce the same `If` rejection. The graph is only claimable
when loaded from the export's own directory, and there is no config-path override in
`0.14.0-dev`: the only env names in the shipped binaries are `GENAI_LOG_ORT_LIB`,
`GENAI_ORT_VERBOSE_LOGGING`, `OG_ORT_LIB`. Carrying a config change therefore means
either editing the download in place or a **single-file bind mount over the real dir**
(`-v file.json:/spike/<model>/genai_config.json:ro`) — and if the EP still refuses, the
gate is not the config.
- **Bisect the image before believing "environment drift".** Run the same load probe
against every local tag; three tags failing identically rules the image out.
- **Rule out host state with instruments, not memory:** `fuser -v /dev/accel/accel0`,
`dmesg | grep -i xdna`, `stat -c "%i %y"` on the model files, `df -h`, `free -g`.
- `sysfs` class is `accel`, not `misc`: glob `/sys/class/accel*/*`. An empty
`/sys/class/misc/accel*` is your glob, not a missing device.
- `GENAI_ORT_VERBOSE_LOGGING=1` and `og.set_log_options(...)` produce nothing useful in
this build; `set_log_options` is a pybind builtin (no `inspect.signature`) and rejects
`severity` as an int. Don't burn turns trying to make it talk — use `LD_DEBUG`.
- `strace` is absent from the RAI image and `apt` may have no mirror: `LD_DEBUG=files` is
the loader tracer that needs nothing installed.
## Gate 2 — generation stops after one token
OGA reads its stop list from the **model** section: `model.eos_token_id`. Chat exports
carry the chat-turn-close id (e.g. 151645) there, so the model emits it first and the
speech/tool budget is never spent — while `search` has **no** eos key: writing
`search.eos_token_id` is a parse refusal (`Unknown value "eos_token_id"`), not a no-op.
- Fix: rewrite `model.eos_token_id` to only the task's own end token (TTS: the
speech-generation-end id), or pass it through `set_search_options` if the build accepts
that key name.
- A sub-millisecond `generate_next_token()` **with finite, high-confidence logits** means
the graph ran and chose a stop token — it is not evidence of a CPU fallback.
- Verify a token-range census against the export's `added_tokens.json`, not against a
guessed vocabulary size: NeuCodec codes reach 65,536 entries (ids 151671…217206), and a
census bounded at 10k silently miscounts every code as a text token.
- A hardlinked run dir is a loaded gun: `open(dst, "w")` on a hardlink truncates the
**original** through the shared inode. `os.unlink` the link before writing a patched
file. Also skip `.git/` and mirror directories when walking an HF checkout.
## Rate measurement
Once a load succeeds, measure at ≥300 steps with a host-side `%CPU` sampler beside the
container (`ps -C python -o pid=,pcpu=,rss=` every 2 s): a device-bound decode waits on
`accel0` and stays under one core, a silent CPU fallback pegs at least one. Never report
request #1 after a boot — graph capture makes it a warmup artifact.
agent/managed-skills/strix-halo-npu-readiness-census/SKILL.md
+14 -2
View File
@@ -1,13 +1,13 @@
---
name: strix-halo-npu-readiness-census
description: "Engine-agnostic census of whether an AMD Strix Halo / Ryzen AI host can run anything on its XDNA2 NPU: driver bind, firmware, device perms, IOMMU, distro XRT, missing AIE compiler, fabric clock, RAM headroom."
description: "Engine-agnostic census of whether an AMD Strix Halo / Ryzen AI host can run anything on its XDNA2 NPU: driver bind, firmware, device perms, IOMMU, distro XRT, missing AIE compiler, fabric clock, RAM headroom — plus the strace test that separates a dead device from a runtime that declines the graph."
---
Use when asked "can we run <engine/model> on the Strix Halo / Ryzen AI NPU?" for a runtime that is **not** halogen (halogen has its own gate + vendor catalog: use `halogen-npu-feasibility-census` for that). This answers only: is the *host* able, and what is missing.
## Why it is usually the wrong door first
XDNA2 executes **precompiled AIE partition programs** loaded through XRT → `amdxdna`. There is no generic tensor dispatch: an engine built on ggml/llama.cpp (or any GGUF loader) cannot "enable" the NPU — the port must be a re-export + vendor compile, not a flag. So pair this census with the accelerator the same engine *can* select by build flag (Vulkan/HIP on the gfx1151 iGPU) before proposing NPU work.
XDNA2 executes **precompiled AIE partition programs** loaded through XRT → `amdxdna`. There is no generic tensor dispatch: an engine built on ggml/llama.cpp (or any GGUF loader) cannot "enable" the NPU — the port must be a re-export + vendor compile, not a flag. So pair this census with the accelerator the same engine *can* select by build flag (Vulkan/HIP on the gfx1151 iGPU) before proposing NPU work. On Dankclaw the iGPU flag was ours to flip all along (`-DGGML_VULKAN=OFF` written out in the fleet's own engine Containerfile) — check the engine's own build file before treating the NPU as the only hardware door.
## Census (all read-only, over ssh from the operator box)
@@ -37,8 +37,20 @@ nproc; free -g | head -2 # RAM headroom + t
- No `ryzenai-compile`/ONNX Ryzen AI EP ⇒ nothing on the host can produce a partition program. That is the enablement delta, and it is a toolchain install + a model-specific export/quantize/compile project, not a config change.
- `free` headroom matters: NPU partitions want a few GB of host RAM each and can log benign `Failed to alloc dev bo memory, ret -28` under pressure.
## Device healthy, load still fails? Classify before blaming the hardware
Two different failures, one call apart. Run the load under `strace -f -e trace=openat`:
- **Never opens `/dev/accel/accel0`** ⇒ pre-device: loader/shim/env (see `ryzenai-npu-oga-linux-run` — the `Generic Failure` in ~2 ms is this class).
- **Opens `/dev/accel/accel0`, GEM ioctls succeed, then the EP declines** ⇒ post-device: the *runtime* rejects the graph. The device is innocent; stop the hardware investigation and fix the last file it opened (`model.onnx` = graph rejected; `dd_metastate_*` = partition artifact problem).
Ground truth that the device itself is sound, inside the container: `xrt-smi examine` (expect `RyzenAI-npu5 | aie2p | 6x8` + firmware version) then `xrt-smi validate` — `PASSED` incl. `throughput`. Needs `libboost-filesystem1.83.0` + `libboost-program-options1.83.0` (**hyphenated** Ubuntu names, after `apt update`); boost is required by `xrt-smi` only, never by the OGA/EP path. Vendor `quicktest` printing success does **not** prove NPU residency (that build falls back to CPU silently) — sample `/sys/class/accel/accel0/device/mem_usage` during generation instead.
When the vendor runtime declines silently, exhaust its own levers and then say so: `GENAI_ORT_VERBOSE_LOGGING`, `GENAI_LOG_ORT_LIB`, `set_log_options`, `set_log_callback(callable)`, `RYZENAI_EP_TRACING`, `RYZENAI_LLM_PROFILE_INIT` all produced zero lines on the 2026-10-19 NuTTS-Air decline. Rule out shadowed artifacts by moving hardlinked mirror copies out of the scanned tree (glob-based sibling scans) — if the failure is byte-identical afterwards, that theory is dead. **If a load ever succeeded once and the byte-identical command never reproduces it (files pristine by inode+mtime), that is a NO-GO, not a lead: no rate may be quoted for a run that cannot be repeated.**
## Laws / traps
- Read state only. Never probe or load the NPU beside a live inference service — stop + mask the service first (fleet benching policy); this host class has a precedent of a concurrent GPU/NPU experiment needing a power cycle.
- `sudo podman ...` finds nothing (per-user rootless storage); run podman as the deploy user.
- `/dev/accel/accel0` existing does not mean a context/PASID can be created; with IOMMU off the failure is at context creation, not at the node.
- Rootless containers cannot read `accel0` (`root:render 0660`) by any documented flag combination — rootful, or a world-writable udev rule.
agent/memories/sharpshooter/dv-live-2n3rewmadkx8c/state.json
+4
View File
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1792441535971
}
agent/memories/sharpshooter/dv-live2-1b4uyv3f0j33c/state.json
+4
View File
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1792441746823
}
agent/memories/sharpshooter/dv-p2-h9uexa4ehfqv/state.json
+4
View File
@@ -0,0 +1,4 @@
{
"v": 1,
"lastConsolidatedAt": 1792443092036
}
agent/memories/sharpshooter/wt-20261019-020309-6d1fc1f-8knxmfcrfa1k/state.json
+1 -1
View File
@@ -1,6 +1,6 @@
{
"v": 1,
"lastConsolidatedAt": 1792440990901,
"lastConsolidatedAt": 1792444614846,
"lastResult": {
"at": 1792433353188,
"sessions": 1,
agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/architecture.md
View File
agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/product.md
+1
View File
@@ -0,0 +1 @@
- Merge it: product decision.
agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/state.json
+7 -1
View File
@@ -1,4 +1,10 @@
{
"v": 1,
"lastConsolidatedAt": 1792441029706
"lastConsolidatedAt": 1792444862643,
"lastResult": {
"at": 1792444218045,
"sessions": 1,
"deltas": 2,
"model": "frank-zealot"
}
}
agent/memories/sharpshooter/wt-20261019-125509-7c8a99d-me8scwe1t2nf/style.md
View File
dotfiles/noctalia/.local/state/noctalia/settings.toml
+4 -4
View File
@@ -275,13 +275,13 @@ transition = [ "fade" ]
interval_seconds = 1200
[wallpaper.default]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9w8r5.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_ymp267.jpg"
[wallpaper.last]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9w8r5.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_ymp267.jpg"
[wallpaper.monitors.DP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9w8r5.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_ymp267.jpg"
[wallpaper.monitors.DP-2]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxex69.jpg"
@@ -290,7 +290,7 @@ transition = [ "fade" ]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9e2w3.jpg"
[wallpaper.monitors.eDP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9w8r5.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_ymp267.jpg"
[weather]
unit = "imperial"