command-center/agent/managed-skills/strix-halo-npu-readiness-census/SKILL.md

name, description

name	description
strix-halo-npu-readiness-census	Engine-agnostic census of whether an AMD Strix Halo / Ryzen AI host can run anything on its XDNA2 NPU: driver bind, firmware, device perms, IOMMU, distro XRT, missing AIE compiler, fabric clock, RAM headroom — plus the strace test that separates a dead device from a runtime that declines the graph.

Use when asked "can we run <engine/model> on the Strix Halo / Ryzen AI NPU?" for a runtime that is not halogen (halogen has its own gate + vendor catalog: use halogen-npu-feasibility-census for that). This answers only: is the host able, and what is missing.

Why it is usually the wrong door first

XDNA2 executes precompiled AIE partition programs loaded through XRT → amdxdna. There is no generic tensor dispatch: an engine built on ggml/llama.cpp (or any GGUF loader) cannot "enable" the NPU — the port must be a re-export + vendor compile, not a flag. So pair this census with the accelerator the same engine can select by build flag (Vulkan/HIP on the gfx1151 iGPU) before proposing NPU work. On Dankclaw the iGPU flag was ours to flip all along (-DGGML_VULKAN=OFF written out in the fleet's own engine Containerfile) — check the engine's own build file before treating the NPU as the only hardware door.

Census (all read-only, over ssh from the operator box)

ssh <host> '
cat /proc/cmdline; ls /sys/kernel/iommu_groups | wc -l      # iommu=pt required; groups==0 => IOMMU off => NPU dead
lsmod | grep -i amdxdna                                      # in-tree driver
sudo dmesg | grep -iE "xdna|npu" | tail                      # "Initialized amdxdna_accel_driver <ver> for 0000:c6:00.1"
lspci -nn | grep -iE "npu|processing"                        # Strix Halo = 1022:17f0
ls /sys/bus/pci/drivers/amdxdna                              # must list the BDF
ls -l /dev/accel                                             # accel0, root:render 660
id -nG <deploy-user> | tr " " "\n" | grep -E "render|video"  # rootless podman needs render, else --device fails at create
find /usr/lib/firmware -iname "*npu*" | head                 # firmware is /usr/lib/firmware/amdnpu/<devid>_<rev>/npu.sbin*.zst — match the PCI rev
dpkg -l | grep -iE "xrt|xdna"; ls /opt/xilinx/xrt/lib 2>&1   # distro libxrt-npu2 = XDNA2 runtime; vendor /opt/xilinx/xrt usually absent
apt-cache policy libxrt2 libxrt-npu2 | grep Candidate
apt-cache search onnxruntime | head                          # distro ONNX RT exists; the Ryzen AI EP does not
python3 -c "import onnxruntime" 2>&1 | tail -1               # absence of the EP + of ryzenai-compile is the real blocker
for c in /sys/class/drm/card*/device; do echo "-- $c"; cat $c/power_dpm_force_performance_level 2>&1; cat $c/pp_dpm_fclk 2>&1 | tr "\n" " "; echo; done
nproc; free -g | head -2                                      # RAM headroom + thread count the engine will pick
'

Reading it

• Driver bound + accel0 + firmware dir + iommu=pt ⇒ kernel side is ready; do not re-litigate "is the NPU disabled". A missing device node is not the modern failure mode.
• auto on pp_dpm_fclk / power_dpm_force_performance_level ⇒ fabric clock unheld. GPU+NPU concurrently with a moving fclk hangs these hosts and corrupts NPU results; any NPU plan must pin the top fclk step (or run in a container holding it) first.
• No ryzenai-compile/ONNX Ryzen AI EP ⇒ nothing on the host can produce a partition program. That is the enablement delta, and it is a toolchain install + a model-specific export/quantize/compile project, not a config change.
• free headroom matters: NPU partitions want a few GB of host RAM each and can log benign Failed to alloc dev bo memory, ret -28 under pressure.

Device healthy, load still fails? Classify before blaming the hardware

Two different failures, one call apart. Run the load under strace -f -e trace=openat:

• Never opens /dev/accel/accel0 ⇒ pre-device: loader/shim/env (see ryzenai-npu-oga-linux-run — the Generic Failure in ~2 ms is this class).
• Opens /dev/accel/accel0, GEM ioctls succeed, then the EP declines ⇒ post-device: the runtime rejects the graph. The device is innocent; stop the hardware investigation and fix the last file it opened (model.onnx = graph rejected; dd_metastate_* = partition artifact problem).

Ground truth that the device itself is sound, inside the container: xrt-smi examine (expect RyzenAI-npu5 | aie2p | 6x8 + firmware version) then xrt-smi validate — PASSED incl. throughput. Needs libboost-filesystem1.83.0 + libboost-program-options1.83.0 (hyphenated Ubuntu names, after apt update); boost is required by xrt-smi only, never by the OGA/EP path. Vendor quicktest printing success does not prove NPU residency (that build falls back to CPU silently) — sample /sys/class/accel/accel0/device/mem_usage during generation instead.

When the vendor runtime declines silently, exhaust its own levers and then say so: GENAI_ORT_VERBOSE_LOGGING, GENAI_LOG_ORT_LIB, set_log_options, set_log_callback(callable), RYZENAI_EP_TRACING, RYZENAI_LLM_PROFILE_INIT all produced zero lines on the 2026-10-19 NuTTS-Air decline. Rule out shadowed artifacts by moving hardlinked mirror copies out of the scanned tree (glob-based sibling scans) — if the failure is byte-identical afterwards, that theory is dead. If a load ever succeeded once and the byte-identical command never reproduces it (files pristine by inode+mtime), that is a NO-GO, not a lead: no rate may be quoted for a run that cannot be repeated.

Laws / traps

• Read state only. Never probe or load the NPU beside a live inference service — stop + mask the service first (fleet benching policy); this host class has a precedent of a concurrent GPU/NPU experiment needing a power cycle.
• sudo podman ... finds nothing (per-user rootless storage); run podman as the deploy user.
• /dev/accel/accel0 existing does not mean a context/PASID can be created; with IOMMU off the failure is at context creation, not at the node.
• Rootless containers cannot read accel0 (root:render 0660) by any documented flag combination — rootful, or a world-writable udev rule.