117 lines
6.1 KiB
Markdown
117 lines
6.1 KiB
Markdown
---
|
|
name: ryzenai-oga-model-dir-and-eos-override
|
|
description: "Debug OGA on XDNA2 when the RyzenAI EP never claims the If-rooted graph (shadow model dirs, bind-mounted configs, missing device), or a TTS/LLM graph stops on the wrong eos token."
|
|
---
|
|
|
|
# OGA on XDNA2: graph not claimed, or generation stops at once
|
|
|
|
Two independent failure families in the AMD `onnxruntime-genai` (OGA) + RyzenAI EP stack on
|
|
Linux XDNA2 (Strix Halo / Ryzen AI Max). Diagnose them in this order — they are separate
|
|
gates and each has its own instrument.
|
|
|
|
## Gate 0 — is the NPU path alive at all?
|
|
|
|
Run the vendor's own instrument before touching anything else:
|
|
|
|
```bash
|
|
cd /opt/rai/quicktest && RYZEN_AI_INSTALLATION_PATH=/opt/rai python quicktest.py
|
|
# want: "Setting environment for STX/KRK" ... "Session successfully initialized" ... "Test Finished"
|
|
```
|
|
|
|
It finds the device by shelling out to `lspci -nn` and grepping `1022:17f0`, so it also
|
|
proves `lspci` is present in the container. Pass ⇒ the in-tree `amdxdna` driver and the
|
|
extracted userspace pair correctly; stop blaming the host.
|
|
|
|
Separate **XRT enumeration** from **OGA's device probe** — they are not the same check:
|
|
|
|
```python
|
|
import sys; sys.path.insert(0, "/opt/xilinx/xrt/python")
|
|
import pyxrt; dev = pyxrt.device(0) # constructs => XRT sees the device
|
|
```
|
|
|
|
`pyxrt.device(0)` succeeding while OGA says `No RyzenAI devices detected` means the
|
|
refusal is OGA's own predicate, inside `libonnxruntime-genai.so` (the string lives there,
|
|
next to `RyzenAI`, `max_length_for_kv_cache`, `library_path`). EP-side env levers worth
|
|
pulling: `RYZENAI_EP_TRACING`, `RYZENAI_EP_PERFORMANCE_COUNTERS`, `RYZENAI_LLM_ASYNC_INIT`,
|
|
`RYZENAI_LLM_PROFILE_INIT`.
|
|
|
|
## Gate 1 — `invalid model … Node (If) cond_full`
|
|
|
|
This is **ONNX Runtime verifying the raw graph**, which happens when no provider claimed
|
|
it: the EP registered nothing and the CPU provider was left holding an `If` whose branches
|
|
mismatch. It is never a corrupted model.
|
|
|
|
```bash
|
|
LD_DEBUG=files LD_DEBUG_OUTPUT=/tmp/ld python loadprobe.py
|
|
grep -a "calling init:.*ryzenai\|undefined symbol" /tmp/ld.*
|
|
# observed: libonnxruntime_providers_ryzenai.so loaded, then
|
|
# symbol lookup error: undefined symbol: GetProvider (fatal)
|
|
```
|
|
|
|
`GetProvider` is not a dynamic symbol in the ORT cores, the shared-provider stub, or the
|
|
EP (`nm -D` shows neither defines nor wants it), so the failure is registration, not an
|
|
ELF mismatch. Confirm the cores are identical before chasing versions: both shipped
|
|
`libonnxruntime.so.1.27.0` copies hash equal.
|
|
|
|
Separate the two gates with an explicit global-scope load before importing OGA:
|
|
|
|
```python
|
|
import ctypes
|
|
ctypes.CDLL("/opt/rai/deployment/lib/libonnxruntime_providers_ryzenai.so",
|
|
mode=ctypes.RTLD_GLOBAL)
|
|
import onnxruntime_genai as og
|
|
og.Model(MODEL) # registration fixed => error MOVES to "No RyzenAI devices detected"
|
|
```
|
|
|
|
An error that **moves** proves you cleared one gate and exposes the next. Silence in the
|
|
first form means the provider list was empty.
|
|
|
|
### Dead ends (measured, do not retry)
|
|
|
|
- **Shadow model dirs never work.** Hardlink mirror, symlink mirror, and even a
|
|
read-only single-file bind mount of a patched `genai_config.json` over the export's own
|
|
`genai_config.json` all produce the same `If` rejection. The graph is only claimable
|
|
when loaded from the export's own directory, and there is no config-path override in
|
|
`0.14.0-dev`: the only env names in the shipped binaries are `GENAI_LOG_ORT_LIB`,
|
|
`GENAI_ORT_VERBOSE_LOGGING`, `OG_ORT_LIB`. Carrying a config change therefore means
|
|
either editing the download in place or a **single-file bind mount over the real dir**
|
|
(`-v file.json:/spike/<model>/genai_config.json:ro`) — and if the EP still refuses, the
|
|
gate is not the config.
|
|
- **Bisect the image before believing "environment drift".** Run the same load probe
|
|
against every local tag; three tags failing identically rules the image out.
|
|
- **Rule out host state with instruments, not memory:** `fuser -v /dev/accel/accel0`,
|
|
`dmesg | grep -i xdna`, `stat -c "%i %y"` on the model files, `df -h`, `free -g`.
|
|
- `sysfs` class is `accel`, not `misc`: glob `/sys/class/accel*/*`. An empty
|
|
`/sys/class/misc/accel*` is your glob, not a missing device.
|
|
- `GENAI_ORT_VERBOSE_LOGGING=1` and `og.set_log_options(...)` produce nothing useful in
|
|
this build; `set_log_options` is a pybind builtin (no `inspect.signature`) and rejects
|
|
`severity` as an int. Don't burn turns trying to make it talk — use `LD_DEBUG`.
|
|
- `strace` is absent from the RAI image and `apt` may have no mirror: `LD_DEBUG=files` is
|
|
the loader tracer that needs nothing installed.
|
|
|
|
## Gate 2 — generation stops after one token
|
|
|
|
OGA reads its stop list from the **model** section: `model.eos_token_id`. Chat exports
|
|
carry the chat-turn-close id (e.g. 151645) there, so the model emits it first and the
|
|
speech/tool budget is never spent — while `search` has **no** eos key: writing
|
|
`search.eos_token_id` is a parse refusal (`Unknown value "eos_token_id"`), not a no-op.
|
|
|
|
- Fix: rewrite `model.eos_token_id` to only the task's own end token (TTS: the
|
|
speech-generation-end id), or pass it through `set_search_options` if the build accepts
|
|
that key name.
|
|
- A sub-millisecond `generate_next_token()` **with finite, high-confidence logits** means
|
|
the graph ran and chose a stop token — it is not evidence of a CPU fallback.
|
|
- Verify a token-range census against the export's `added_tokens.json`, not against a
|
|
guessed vocabulary size: NeuCodec codes reach 65,536 entries (ids 151671…217206), and a
|
|
census bounded at 10k silently miscounts every code as a text token.
|
|
- A hardlinked run dir is a loaded gun: `open(dst, "w")` on a hardlink truncates the
|
|
**original** through the shared inode. `os.unlink` the link before writing a patched
|
|
file. Also skip `.git/` and mirror directories when walking an HF checkout.
|
|
|
|
## Rate measurement
|
|
|
|
Once a load succeeds, measure at ≥300 steps with a host-side `%CPU` sampler beside the
|
|
container (`ps -C python -o pid=,pcpu=,rss=` every 2 s): a device-bound decode waits on
|
|
`accel0` and stays under one core, a silent CPU fallback pegs at least one. Never report
|
|
request #1 after a boot — graph capture makes it a warmup artifact.
|