---
name: halogen-npu-mount-prefix-proof
description: "Prove and fix a halogen frank-dragoon NPU-lane mount failure: crash-loop where a dead lane kills the whole container, dlopen-prefix census, and the sanctioned FRANK_DRAGOON_ENV_FILE proof-boot."
---

# Halogen NPU lane: dead lane kills the container — prove the mount prefix, don't guess

Applies when a `frank-dragoon` unit crash-loops in seconds (`exit 1` in ~200 ms,
`NRestarts` climbing) and the engine log's last lines are:

```
halogen npu: <model>: 15 files checked against this image's record
halogen-npu: No such library '<some/prefix>/libxrt_core.so.2'
halogen npu: the NPU engine exited before it was ready (rc=1)
```

## Laws this incident wrote (2026-10-21, frank-dragoon, 88-restart loop)
- **A dead NPU lane is fatal to the whole container**: the flash-server supervisor
  exits with the lane. Chat dies beside a healthy GPU preflight — do not read the
  GTT/weights lines as "engine fine, side feature broken".
- The NPU child `dlopen`s distro XRT from **three** container prefixes per library:
  `/opt/xilinx/xrt/lib/`, the library's own install path, **and
  `/opt/xilinx/lib/<multiarch>/`**. The missing one is exactly what the error names.
- Host state can flip this under a byte-identical container: the lane ran for days
  with two prefixes, then a host `libxrt2`/`libxrt-npu2` reconfigure (check
  `grep libxrt /var/log/dpkg.log`) made the child demand the third. Image digest,
  serve script, payload all unchanged ⇒ suspect HOST package state, not the deploy.
- Fastest discriminator vs engine faults: exit in <1 s with the engine's preflight
  lines present = serve/lane gate failure, not model load. Model-load failures take
  minutes and print pin/KV lines.

## Proof procedure (sanctioned seam, no scratch ports)
1. Read the error's path verbatim — it names the missing prefix.
2. `sudo systemctl stop frank-dragoon` (unit must be down: the proof binds :7770
   at container CREATE).
3. Scratch env file on the host using the bench-sanctioned seam:
   ```
   /tmp/dragoon-nputest.env:
     source /home/dank/serverAppConfigs/scripts/frank-dragoon.env
     HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
   ```
4. Scratch serve copy that appends the candidate mount to the `NPU_ARGS+=(--device`
   line (`sed`), REAL files only (`readlink -f`; vendor forbids dangling symlinks
   under /opt/xilinx), and run with:
   `FRANK_DRAGOON_ENV_FILE=/tmp/... FRANK_DRAGOON_LOG=/tmp/... timeout 1200 /tmp/serve-nputest.sh`
5. Success = boot ledger (`… in all`) **plus** `npu_api: <model> … buckets u0…u9`
   plus `/health` `npu.status ok`. Then codify the mount in the serve script, pin
   it in `tests/frank_dragoon_unit.bats` (the fixture XRT dir's basename IS the
   multiarch name — name it `x86_64-linux-gnu`), update the endpoint page's NPU
   prerequisite to the three-prefix law, and cut over via PR +
   `deploy.sh dankclaw --service frank-dragoon`.
6. Retire scratch: the `timeout` kills the proof container; `podman rm -f`,
   move /tmp scripts to `~/.AGENTTRASH/<date>-<topic>/`. A proof instance
   serving production while you wait is drift — deploy the merged fix promptly.

## Traps
- Greping image contents rootless: `podman mount` fails (needs `podman unshare`)
  and leaves `$M` EMPTY, so `grep -r $M/opt` silently greps the HOST cwd — that
  is how host `/opt/fastflowlm` masqueraded as image content. Always check `$M`
  non-empty, or use `podman unshare bash -c 'M=$(podman mount C); …'`.
- The entrypoint swallows `podman run IMG sh -c …` as a mode name; use
  `--entrypoint sh/cat/ls`.
- The proof container inherits production name/port unless the override is set
  AFTER sourcing the env (the env file re-exports `FRANK_DRAGOON_CONTAINER`).
- `Restart=always` re-runs the lane into its own GPU/NPU debris; stop the unit
  before any manual boot, don't fight the loop.