command-center/agent/managed-skills/halogen-npu-mount-prefix-proof/SKILL.md

name, description

name	description
halogen-npu-mount-prefix-proof	Prove and fix a halogen frank-dragoon NPU-lane mount failure: crash-loop where a dead lane kills the whole container, dlopen-prefix census, and the sanctioned FRANK_DRAGOON_ENV_FILE proof-boot.

Halogen NPU lane: dead lane kills the container — prove the mount prefix, don't guess

Applies when a frank-dragoon unit crash-loops in seconds (exit 1 in ~200 ms, NRestarts climbing) and the engine log's last lines are:

halogen npu: <model>: 15 files checked against this image's record
halogen-npu: No such library '<some/prefix>/libxrt_core.so.2'
halogen npu: the NPU engine exited before it was ready (rc=1)

Laws this incident wrote (2026-10-21, frank-dragoon, 88-restart loop)

• A dead NPU lane is fatal to the whole container: the flash-server supervisor exits with the lane. Chat dies beside a healthy GPU preflight — do not read the GTT/weights lines as "engine fine, side feature broken".
• The NPU child dlopens distro XRT from three container prefixes per library: /opt/xilinx/xrt/lib/, the library's own install path, and /opt/xilinx/lib/<multiarch>/. The missing one is exactly what the error names.
• Host state can flip this under a byte-identical container: the lane ran for days with two prefixes, then a host libxrt2/libxrt-npu2 reconfigure (check grep libxrt /var/log/dpkg.log) made the child demand the third. Image digest, serve script, payload all unchanged ⇒ suspect HOST package state, not the deploy.
• Fastest discriminator vs engine faults: exit in <1 s with the engine's preflight lines present = serve/lane gate failure, not model load. Model-load failures take minutes and print pin/KV lines.

Proof procedure (sanctioned seam, no scratch ports)

1. Read the error's path verbatim — it names the missing prefix.
2. sudo systemctl stop frank-dragoon (unit must be down: the proof binds :7770 at container CREATE).
3. Scratch env file on the host using the bench-sanctioned seam:

   /tmp/dragoon-nputest.env:
     source /home/dank/serverAppConfigs/scripts/frank-dragoon.env
     HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
   

4. Scratch serve copy that appends the candidate mount to the NPU_ARGS+=(--device line (sed), REAL files only (readlink -f; vendor forbids dangling symlinks under /opt/xilinx), and run with: FRANK_DRAGOON_ENV_FILE=/tmp/... FRANK_DRAGOON_LOG=/tmp/... timeout 1200 /tmp/serve-nputest.sh
5. Success = boot ledger (… in all) plus npu_api: <model> … buckets u0…u9 plus /health npu.status ok. Then codify the mount in the serve script, pin it in tests/frank_dragoon_unit.bats (the fixture XRT dir's basename IS the multiarch name — name it x86_64-linux-gnu), update the endpoint page's NPU prerequisite to the three-prefix law, and cut over via PR + deploy.sh dankclaw --service frank-dragoon.
6. Retire scratch: the timeout kills the proof container; podman rm -f, move /tmp scripts to ~/.AGENTTRASH/<date>-<topic>/. A proof instance serving production while you wait is drift — deploy the merged fix promptly.

Traps

• Greping image contents rootless: podman mount fails (needs podman unshare) and leaves $M EMPTY, so grep -r $M/opt silently greps the HOST cwd — that is how host /opt/fastflowlm masqueraded as image content. Always check $M non-empty, or use podman unshare bash -c 'M=$(podman mount C); …'.
• The entrypoint swallows podman run IMG sh -c … as a mode name; use --entrypoint sh/cat/ls.
• The proof container inherits production name/port unless the override is set AFTER sourcing the env (the env file re-exports FRANK_DRAGOON_CONTAINER).
• Restart=always re-runs the lane into its own GPU/NPU debris; stop the unit before any manual boot, don't fight the loop.