command-center/agent/managed-skills/halogen-npu-xrt-third-prefix/SKILL.md

name, description

name	description
halogen-npu-xrt-third-prefix	Triage a halogen frank-dragoon (flash-server) unit crash-loop caused by the NPU lane: signature "No such library /opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2", dead-lane-kills-container law, third-prefix mount fix, and the FRANK_DRAGOON_ENV_FILE scratch-boot proof recipe.

Halogen NPU lane: dead lane kills the container; XRT needs THREE mount prefixes

Friction-earned 2026-10-21 on frank-dragoon (Dankclaw): unit boot-looped 88× (activating, exit 1 in ~200 ms) with the GPU/serve_api lines looking healthy; chat dead beside a fine-looking ledger.

Signatures (in order of appearance)

halogen-npu: No such library '/opt/xilinx/lib/x86_64-linux-gnu/libxrt_core.so.2'
halogen npu: the NPU engine exited before it was ready (rc=1; ...)

then systemctl show <svc> -p ExecMainStatus = 1, NRestarts climbing, and /home/dank/logs/frank_dragoon.log truncated at ~10 lines each boot (serve script truncates the log per start).

Laws: a dead NPU lane is NOT survivable — the engine supervisor takes the WHOLE container down on it. And Restart=always makes each restart race its own GPU/NPU debris: the FIRST failure's message is the only real evidence; later loop entries are noise. Stop the unit immediately (sudo systemctl stop frank-dragoon) before probing.

Root cause class

The NPU child dlopens XRT from /opt/xilinx/lib/<multiarch>/ IN ADDITION to /opt/xilinx/xrt/lib and the install path. Host package churn (dpkg.log reconfigure of libxrt2 the previous day) made the child demand the third prefix; our mounts filled only two. A lane that worked for days with byte-identical mounts can die purely from host package changes — diff the mounts last, check grep libxrt /var/log/dpkg.log first.

Fix: mount each REAL file (never a symlink) at all three prefixes; derive the multiarch name from FRANK_DRAGOON_XRT_DIR's basename (MULTIARCH="${XRT_DIR##*/}").

Proof recipe (production-shaped, no scratch port)

One card holds one server — never bench on a side port; use the sanctioned env-file seam:

1. Scratch env on the host: source /home/dank/serverAppConfigs/scripts/frank-dragoon.env + knob overrides (e.g. HALOGEN_NPU_MODELS=...).
2. Copy the serve script with the candidate change applied via sed into /tmp (or edit + FRANK_DRAGOON_ENV_FILE); export FRANK_DRAGOON_ENV_FILE=<scratch> and FRANK_DRAGOON_LOG=/tmp/<name>.log; run under timeout 1200. The seam exists precisely for this.
3. Judge from /tmp/<name>.log: npu_api: <model> ... buckets u0…u9 = lane alive; No such library = still broken. Then curl :7770/health → npu.status.
4. Retire the same session: podman stop -t 60 (weights unpin slowly), podman rm -f, kill wrapper procs BY PID (pgrep -f the scratch serve script — pkill -f a pattern containing your own ssh line kills the session), move scratch files to ~/.AGENTTRASH/<date>-<topic>/, then prove pgrep -f serve-<scratch> is empty. Leftover scratch state silently hijacks the next deploy (same class as leftover bench drop-ins).

Traps hit en route

• Rootless podman mount fails ("cannot run in rootless mode") and leaves $M EMPTY — a later grep -r "$M/..." then reads the HOST cwd and "finds" the answer in Loki WAL files. Never grep an unverified mount var; podman unshare or --entrypoint probe instead.
• Image ships NO /opt/xilinx at all: all XRT comes from host mounts; docs/NPU.md is not inside the image.
• ssh host 'tail <log>' runs on the DEV box if you forgot ssh — a missing-file error is not "the engine wrote nothing".
• Health context field echoes the RUNNING engine's ceiling — use it to prove which shape is live, unlike /v1/models name echo.

Verification that the fix landed

systemctl show frank-dragoon -p DropInPaths empty; boot ledger shows reserving the KV pool … (about 11.0 GiB), … in all, an npu_api: line, no clamp line; deployer prints NPU lane verified — <model> embedded.