command-center/agent/managed-skills/ryzenai-generic-failure-loader-order/SKILL.md

name, description

name	description
ryzenai-generic-failure-loader-order	Debug AMD Ryzen AI OGA "Exception during initialization: Generic Failure" on Linux XDNA2 — the LD_LIBRARY_PATH-vs-activate ordering trap, its proof instruments, and the OGA 0.14 run contract.

Symptom (any onnxruntime_genai run on a Ryzen AI NPU, container or bare):

[E:onnxruntime:, inference_session.cc:2811 operator()] Exception during initialization: Generic Failure

Fails ~2 ms after og.Model(). No openat of /dev/accel/accel0, nothing in the kernel log, no XRT log (the INI logger is not built into AMD's shipped XRT), no core dump. It is NOT a platform, model-export, or driver-ABI failure.

Root cause

source <venv>/bin/activate from install_ryzen_ai.sh prepends to LD_LIBRARY_PATH:

site-packages/flexml/flexml_extras/lib : site-packages/onnxruntime/capi : site-packages/voe/lib

voe/lib is the compiler-side bundle: it ships its own libxrt_coreutil.so.2 and libryzen_mm.so.1. Set the path before activating and those copies win, XRT's XDNA shim binds against the wrong coreutil, xrt_core::shim_loader::shim_loader() → get_userpf_device() → xrt::device(0) throws, ryzenai::Allocator::Alloc throws next, and ORT flattens both into the bare string above.

Rule: activate first, THEN set LD_LIBRARY_PATH, with /opt/xilinx/xrt/lib first. Passing the var via podman -e and sourcing activate afterwards in the same bash -lc is the broken shape — that is exactly how a runner script differs from a hand-typed probe, and it is why diagnostics pass while the runner fails. Mounting XRT is not enough either: the tree must be on the path.

Proof instruments, cheapest first

1. ldd <deployment>/lib/libonnxruntime_providers_ryzenai.so | grep -E "libxrt_coreutil|libryzen_mm" — the resolved path settles it alone: /opt/xilinx/xrt/lib = good, .../voe/lib = this bug.
2. XILINX_XRT=/opt/xilinx/xrt /opt/xilinx/xrt/bin/xrt-smi examine → RyzenAI-npu5 | aie2p | 6x8 + NPU Firmware Version. Needs libboost_filesystem1.83.0, libboost_program_options1.83.0, libuuid1 (26.04 names; the 24.04 docs' 1.74.0 does not exist). A "No such file or directory" for a file that exists = missing interpreter/dep, not a missing binary.
3. strace -f -e trace=openat — /dev/accel/accel0 never opened ⇒ failure is pre-device ⇒ loader-level.
4. gdb -q -batch -x <(echo 'catch throw'; echo run; echo 'bt 10') style catch-throw loop → names shim_loader / Allocator::Alloc.

Known dead ends (each costs one run to rule out, then stop): --ulimit memlock, XRT INI verbosity, DD_LOG_LEVEL / RYZENAI_TRACE / RYZENAI_EP_TRACING (print nothing before the throw), byte-scanning XRT .so for PCI IDs (ambiguous), libboost presence for the OGA path (only xrt-smi needs it), flag-by-flag container matrices (--memory, --security-opt, --name, extra mounts — all innocent).

Verified facts on this stack

• Vendor XRT 2.25.37 userspace (dpkg-deb -x, never dpkg-installed) works against Ubuntu's **in-tree amdxdna 0.7.0; no DKMS module needed. 1022:17f0rev 11 = XDNA2, handled asaie2p 6x8`.
• AMD's quicktest passing does not prove NPU use (it can silently run on CPU); trust xrt-smi + logits, not "it didn't crash".
• /dev/accel/accel0 is root:render 0660; rootless podman cannot read it (userns drops the gid; keep-groups, keep-id:gid=990, --group-add render all fail) → rootful container, or a udev rule like the GPU nodes'.
• OGA 0.14 (RAI 1.8) contract: GeneratorParams has only set_search_options (no input_ids, no model.generate); feed with Generator.append_tokens(ids), loop generate_next_token(), read get_sequence(0); max_length counts prompt + generated. Full Fusion (*_npu_4K) caps input+output at 4096.
• Validate the content, not just the exit code: np.argsort(logits[0])[-8:]. append_tokens returning a real logits tensor proves the graph ran; if argmax is the sequence terminator (`