command-center/agent/managed-skills/halogen-npu-lane-boot-loop-triage/SKILL.md

name, description

name	description
halogen-npu-lane-boot-loop-triage	Triage a frank-dragoon (halogen-flash-server) unit crash-loop caused by the NPU lane: signature classification, restart-debris trap, XRT mount-path census, and the transient env-file proof boot.

Triage when frank-dragoon (or any halogen-flash-server unit with HALOGEN_NPU_MODELS armed) sits in activating (auto-restart) with ExecStart … code=exited, status=1/FAILURE seconds after start, while /health stays dark.

1. Classify the exit BEFORE touching the config

• NRestarts climbing + exit in <5 s = refusal, not an engine fault. Read journalctl -u frank-dragoon --since "-30m" AND the engine log /home/dank/logs/frank_dragoon.log (engine stdout never goes to the journal).
• Lane signature: log ends with halogen npu: … <name>: N files checked against this image's record then halogen-npu: No such library '<path>' + the NPU engine exited before it was ready (rc=1) → the NPU lane exit is fatal to the whole container (supervisor kills the Flash engine with it). Chat-down is the lane's fault even though chat code never changed.
• KV/pre-flight lines present but no locked in RAM line = died before weight pinning.

2. The restart-debris trap

Restart=always re-runs the failing boot every ~10 s; each attempt leaves GPU/NPU handles being released, and the next boot logs GTT in use before this start: N GiB … held by 1 process(es). That NOTE is debris from your own crash-loop — do not chase it as a foreign co-tenant (real co-tenants: ComfyUI python on /dev/kfd, find with sudo fuser -v /dev/dri/renderD128 /dev/kfd). sudo systemctl stop <unit> FIRST, then any diagnosis.

3. XRT path census (the No such library '/opt/xilinx/lib/<multiarch>/libxrt_*.so.2' case)

• The image ships NO XRT at all (find / -xdev -name "libxrt*" inside the image = empty); the NPU child dlopens the host's libs through the serve script's mounts, which land at TWO destinations only: /opt/xilinx/xrt/lib/<L> and $XRT_DIR/<L>. A child searching /opt/xilinx/lib/<multiarch>/ names a THIRD prefix nobody mounts.
• The serve loop resolves REAL="$(readlink -f …)" and mounts resolved files — if a host package upgrade retargets the symlinks (check grep libxrt /var/log/dpkg.log + ldconfig -p | grep xrt mtimes vs the last good boot), the file set the child expects can change under byte-identical mounts.
• Rootless gotcha: podman mount fails outside podman unshare and leaves $M EMPTY — grep -r pat "$M/opt" then greps your CWD = HOST tree (false hits like /opt/fastflowlm). Always podman unshare bash -c 'M=$(podman mount $C); …' or --entrypoint ls|cat probes instead, and distrust hits that name host-only paths.

4. Proof boot without touching production

The serve script honors FRANK_DRAGOON_ENV_FILE (sanctioned seam). Throwaway script on the host:

1. scratch env: source /home/dank/serverAppConfigs/scripts/frank-dragoon.env + the knob under test.
2. sed the serve script to inject the candidate extra mounts into the NPU_ARGS+=(--device … line — mount RESOLVED real files (.so.2.25.00), never symlinks (vendor: dangling links inside the container).
3. export FRANK_DRAGOON_ENV_FILE=/tmp/… FRANK_DRAGOON_LOG=/tmp/…; FRANK_DRAGOON_CONTAINER=<scratch-name> timeout 1200 /tmp/serve-test.sh — the container still publishes 7770, so systemctl stop the unit first (port binds at CREATE, exit 126 on race).
4. Retire same session: podman rm -f <scratch-name>, trash /tmp scripts; the unit stays stopped until the real fix deploys.

5. Restore chat while the lane is unfixed

HALOGEN_NPU_MODELS= (blank) is the documented one-line lane-off switch — lane absent = endpoint exactly as before the lane existed. Land through git + deploy.sh dankclaw --service frank-dragoon; never hand-edit the remote env.