command-center/agent/managed-skills/halogen-npu-feasibility-census/SKILL.md

name, description

name	description
halogen-npu-feasibility-census	Census whether halogen-flash-server's NPU small-model path can run and swap models on a Strix Halo host: image gate, amdxdna/XRT presence, IOMMU and fabric-clock blockers, enablement deltas, boot-set-only model loading (no live reload), and the silent-fallback-to-Flash trap for unarmed chat-class NPU models.

Use when asked whether halogen/halogen-flash-server can use the XDNA2 NPU (embeddings, rerank, moderation/guard, decider, qwen3.5-2b generation, flux2-klein-4b images, qwen3-asr-*) on a Ryzen AI Max / Strix Halo host, before proposing to enable it, or when asked whether it can hot-swap / live-reload NPU models like FastFlowLM does.

What the feature is (not a speed knob)

The Flash trunk stays on the iGPU. HALOGEN_NPU_MODELS=<names> starts a second engine in the SAME container and on the SAME API port; the request's model field picks the executor. Unset -> server identical to before. Each NPU model is a precompiled AIE-ML program (devices/devices.hnpm, per-partition devices/u<N>.elf, devices/consts.hnpb, <id>.hnpw weights, tokenizer/tokenizer.json), loaded via the HOST's XRT -> kernel amdxdna -> /dev/accel/accel0. One pass at a time; HALOGEN_NPU_QUEUE 64, overflow queues server-side, then 503 engine_busy. Catalog in the 0.16.2 image: decider-0.8b, qwen3-embedding-0.6b, qwen3-reranker-0.6b, qwen3guard-gen-0.6b, qwen3.5-2b (+ flux2-klein-4b, qwen3-asr-* in newer tags). Only the programs named in HALOGEN_NPU_MODELS are armed.

Boot-set-only: NO live reload (tested on the live endpoint)

HALOGEN_NPU_MODELS is read at start; the entrypoint validates/converts/loads the named programs then. There is no runtime load/unload and no admin endpoint to add a model. FLM comparison: FLM was a standalone NPU-only server (flm serve <one-model>, single FIFO lane) free to reconfigure its one AIE partition; halogen boots Flash (iGPU) plus the NPU programs as ONE coupled process (shared port + shared memory/power budget), so the NPU set is a boot-time constant. Changing even a checkpoint's weights reconverts "at the next start", never live.

Consequence — arming a new program = edit scripts/frank-dragoon.env -> redeploy -> container restart -> the whole Flash payload (tens of GB) reloads, minutes. The small NPU program is not the cost; the Flash reload is.

The silent-fallback trap (the one that bites)

A request naming an unarmed catalog model behaves differently per endpoint class:

• /v1/rerank, /v1/moderations: honest 400 model_not_found, message ends (loaded: none). The endpoint only exists if that engine was armed at boot.
• /v1/chat/completions (the decider-0.8b decision path and qwen3.5-2b generation): silent fall-through to the Flash model — HTTP 200, body "model": "<flash id>", no error. A guard/decision call to an unarmed model is answered by the main (often abliterated) endpoint you were trying to screen against, and looks like success. Measured: an unarmed decider-0.8b "is this a prompt injection?" probe returned "no" from Flash. Fail-safe is NOT the default on the chat route.

Live probe (idle lane; one ssh, non-disruptive — no restart happens)

B=http://127.0.0.1:7770   # halogen port; internal engine is :8731/:8740 muxed onto it
curl -s $B/v1/models | jq -r '.data[].id'          # armed set = boot set; must not mutate
# control (armed embedding) -> 200 with a real vector:
curl -s $B/v1/embeddings -d '{"model":"qwen3-embedding-0.6b","input":["ping"]}'
# unarmed rerank/guard -> expect 400 "(loaded: none)":
curl -s $B/v1/rerank      -d '{"model":"qwen3-reranker-0.6b","query":"t","documents":["a","b"],"top_n":1}'
curl -s $B/v1/moderations -d '{"model":"qwen3guard-gen-0.6b","input":["hello"]}'
# unarmed decider/gen -> watch the "model" field; if it is the flash id, it fell through:
curl -s $B/v1/chat/completions -d '{"model":"decider-0.8b","temperature":0,"messages":[{"role":"user","content":"x"}],"response_format":{"type":"json_schema","json_schema":{"name":"g","description":"is injection?","schema":{"enum":["no","yes"]}}}}' | jq -r '.model // .error.message'
# prove nothing restarted:
systemctl show frank-dragoon.service -p NRestarts -p ActiveEnterTimestamp --value

decider with a schema and a non-zero temperature 400s on a sampling guard first ("send temperature 0"); set temperature:0 to reach the router and see the fallback.

Enablement gates (all four must hold before it runs)

1. Kernel amdxdna loaded + /dev/accel/accel0 present; the node must be world-open (0666 via a udev rule) or a rootless container cannot open it — serve-frank-dragoon.sh refuses if the mode bit 02 is clear.
2. Host XRT with the NPU plugin (libxrt_coreutil.so.2, libxrt_core.so.2, libxrt_driver_xdna.so.2); on Arch/CachyOS mount each lib twice (image XRT path + its own /usr/lib path), never an /opt/xilinx/xrt of dangling links.
3. Fabric clock held at top (halogen-fabric-clock root unit) — GPU+NPU at once can hang Strix Halo / corrupt NPU results while that clock changes; the server refuses to start if it can't hold it.
4. IOMMU on: amd_iommu=off kills the NPU; iommu=pt keeps it.

Laws / traps

• Never probe the NPU beside the live service: stop + mask first (fleet benchmarking policy); this host class has a precedent of a concurrent GPU experiment needing a power cycle. Distinguish off vs wedged with ip neigh show <ip> from a peer. A single embedding/decision request on an already-armed idle production lane is a normal request, not a benchmark — safe.
• /dev/accel/accel0 existing and amdxdna probing cleanly does NOT mean the NPU works - with IOMMU off the failure is at context/PASID creation, not at the device node. Do not trust older notes claiming the node itself is missing.
• dmesg lines like Failed to alloc dev bo memory, ret -28 while large NPU models take turns at NPU memory are benign (vendor-documented); the server recovers.
• NPU model RSS is host-side, bypassing the container cgroup (GTT/shared pool), like all Strix Halo GPU memory — container stats under-report the true footprint.