live: heartbeat sync 2026-10-19T14:21:41-07:00 (Refs #11)

agent/managed-skills/ryzenai-oga-model-dir-and-eos-override/SKILL.md

@@ -0,0 +1,116 @@
+---
+name: ryzenai-oga-model-dir-and-eos-override
+description: "Debug OGA on XDNA2 when the RyzenAI EP never claims the If-rooted graph (shadow model dirs, bind-mounted configs, missing device), or a TTS/LLM graph stops on the wrong eos token."
+---
+
+# OGA on XDNA2: graph not claimed, or generation stops at once
+
+Two independent failure families in the AMD `onnxruntime-genai` (OGA) + RyzenAI EP stack on
+Linux XDNA2 (Strix Halo / Ryzen AI Max). Diagnose them in this order — they are separate
+gates and each has its own instrument.
+
+## Gate 0 — is the NPU path alive at all?
+
+Run the vendor's own instrument before touching anything else:
+
+```bash
+cd /opt/rai/quicktest && RYZEN_AI_INSTALLATION_PATH=/opt/rai python quicktest.py
+# want: "Setting environment for STX/KRK" ... "Session successfully initialized" ... "Test Finished"
+```
+
+It finds the device by shelling out to `lspci -nn` and grepping `1022:17f0`, so it also
+proves `lspci` is present in the container. Pass ⇒ the in-tree `amdxdna` driver and the
+extracted userspace pair correctly; stop blaming the host.
+
+Separate **XRT enumeration** from **OGA's device probe** — they are not the same check:
+
+```python
+import sys; sys.path.insert(0, "/opt/xilinx/xrt/python")
+import pyxrt; dev = pyxrt.device(0)   # constructs => XRT sees the device
+```
+
+`pyxrt.device(0)` succeeding while OGA says `No RyzenAI devices detected` means the
+refusal is OGA's own predicate, inside `libonnxruntime-genai.so` (the string lives there,
+next to `RyzenAI`, `max_length_for_kv_cache`, `library_path`). EP-side env levers worth
+pulling: `RYZENAI_EP_TRACING`, `RYZENAI_EP_PERFORMANCE_COUNTERS`, `RYZENAI_LLM_ASYNC_INIT`,
+`RYZENAI_LLM_PROFILE_INIT`.
+
+## Gate 1 — `invalid model … Node (If) cond_full`
+
+This is **ONNX Runtime verifying the raw graph**, which happens when no provider claimed
+it: the EP registered nothing and the CPU provider was left holding an `If` whose branches
+mismatch. It is never a corrupted model.
+
+```bash
+LD_DEBUG=files LD_DEBUG_OUTPUT=/tmp/ld python loadprobe.py
+grep -a "calling init:.*ryzenai\|undefined symbol" /tmp/ld.*
+# observed: libonnxruntime_providers_ryzenai.so loaded, then
+#   symbol lookup error: undefined symbol: GetProvider (fatal)
+```
+
+`GetProvider` is not a dynamic symbol in the ORT cores, the shared-provider stub, or the
+EP (`nm -D` shows neither defines nor wants it), so the failure is registration, not an
+ELF mismatch. Confirm the cores are identical before chasing versions: both shipped
+`libonnxruntime.so.1.27.0` copies hash equal.
+
+Separate the two gates with an explicit global-scope load before importing OGA:
+
+```python
+import ctypes
+ctypes.CDLL("/opt/rai/deployment/lib/libonnxruntime_providers_ryzenai.so",
+            mode=ctypes.RTLD_GLOBAL)
+import onnxruntime_genai as og
+og.Model(MODEL)   # registration fixed => error MOVES to "No RyzenAI devices detected"
+```
+
+An error that **moves** proves you cleared one gate and exposes the next. Silence in the
+first form means the provider list was empty.
+
+### Dead ends (measured, do not retry)
+
+- **Shadow model dirs never work.** Hardlink mirror, symlink mirror, and even a
+  read-only single-file bind mount of a patched `genai_config.json` over the export's own
+  `genai_config.json` all produce the same `If` rejection. The graph is only claimable
+  when loaded from the export's own directory, and there is no config-path override in
+  `0.14.0-dev`: the only env names in the shipped binaries are `GENAI_LOG_ORT_LIB`,
+  `GENAI_ORT_VERBOSE_LOGGING`, `OG_ORT_LIB`. Carrying a config change therefore means
+  either editing the download in place or a **single-file bind mount over the real dir**
+  (`-v file.json:/spike/<model>/genai_config.json:ro`) — and if the EP still refuses, the
+  gate is not the config.
+- **Bisect the image before believing "environment drift".** Run the same load probe
+  against every local tag; three tags failing identically rules the image out.
+- **Rule out host state with instruments, not memory:** `fuser -v /dev/accel/accel0`,
+  `dmesg | grep -i xdna`, `stat -c "%i %y"` on the model files, `df -h`, `free -g`.
+- `sysfs` class is `accel`, not `misc`: glob `/sys/class/accel*/*`. An empty
+  `/sys/class/misc/accel*` is your glob, not a missing device.
+- `GENAI_ORT_VERBOSE_LOGGING=1` and `og.set_log_options(...)` produce nothing useful in
+  this build; `set_log_options` is a pybind builtin (no `inspect.signature`) and rejects
+  `severity` as an int. Don't burn turns trying to make it talk — use `LD_DEBUG`.
+- `strace` is absent from the RAI image and `apt` may have no mirror: `LD_DEBUG=files` is
+  the loader tracer that needs nothing installed.
+
+## Gate 2 — generation stops after one token
+
+OGA reads its stop list from the **model** section: `model.eos_token_id`. Chat exports
+carry the chat-turn-close id (e.g. 151645) there, so the model emits it first and the
+speech/tool budget is never spent — while `search` has **no** eos key: writing
+`search.eos_token_id` is a parse refusal (`Unknown value "eos_token_id"`), not a no-op.
+
+- Fix: rewrite `model.eos_token_id` to only the task's own end token (TTS: the
+  speech-generation-end id), or pass it through `set_search_options` if the build accepts
+  that key name.
+- A sub-millisecond `generate_next_token()` **with finite, high-confidence logits** means
+  the graph ran and chose a stop token — it is not evidence of a CPU fallback.
+- Verify a token-range census against the export's `added_tokens.json`, not against a
+  guessed vocabulary size: NeuCodec codes reach 65,536 entries (ids 151671…217206), and a
+  census bounded at 10k silently miscounts every code as a text token.
+- A hardlinked run dir is a loaded gun: `open(dst, "w")` on a hardlink truncates the
+  **original** through the shared inode. `os.unlink` the link before writing a patched
+  file. Also skip `.git/` and mirror directories when walking an HF checkout.
+
+## Rate measurement
+
+Once a load succeeds, measure at ≥300 steps with a host-side `%CPU` sampler beside the
+container (`ps -C python -o pid=,pcpu=,rss=` every 2 s): a device-bound decode waits on
+`accel0` and stays under one core, a silent CPU fallback pegs at least one. Never report
+request #1 after a boot — graph capture makes it a warmup artifact.