live: heartbeat sync 2026-10-21T02:32:51-07:00 (Refs #11)

agent/managed-skills/halogen-npu-mount-prefix-proof/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: halogen-npu-mount-prefix-proof
+description: "Prove and fix a halogen frank-dragoon NPU-lane mount failure: crash-loop where a dead lane kills the whole container, dlopen-prefix census, and the sanctioned FRANK_DRAGOON_ENV_FILE proof-boot."
+---
+
+# Halogen NPU lane: dead lane kills the container — prove the mount prefix, don't guess
+
+Applies when a `frank-dragoon` unit crash-loops in seconds (`exit 1` in ~200 ms,
+`NRestarts` climbing) and the engine log's last lines are:
+
+```
+halogen npu: <model>: 15 files checked against this image's record
+halogen-npu: No such library '<some/prefix>/libxrt_core.so.2'
+halogen npu: the NPU engine exited before it was ready (rc=1)
+```
+
+## Laws this incident wrote (2026-10-21, frank-dragoon, 88-restart loop)
+- **A dead NPU lane is fatal to the whole container**: the flash-server supervisor
+  exits with the lane. Chat dies beside a healthy GPU preflight — do not read the
+  GTT/weights lines as "engine fine, side feature broken".
+- The NPU child `dlopen`s distro XRT from **three** container prefixes per library:
+  `/opt/xilinx/xrt/lib/`, the library's own install path, **and
+  `/opt/xilinx/lib/<multiarch>/`**. The missing one is exactly what the error names.
+- Host state can flip this under a byte-identical container: the lane ran for days
+  with two prefixes, then a host `libxrt2`/`libxrt-npu2` reconfigure (check
+  `grep libxrt /var/log/dpkg.log`) made the child demand the third. Image digest,
+  serve script, payload all unchanged ⇒ suspect HOST package state, not the deploy.
+- Fastest discriminator vs engine faults: exit in <1 s with the engine's preflight
+  lines present = serve/lane gate failure, not model load. Model-load failures take
+  minutes and print pin/KV lines.
+
+## Proof procedure (sanctioned seam, no scratch ports)
+1. Read the error's path verbatim — it names the missing prefix.
+2. `sudo systemctl stop frank-dragoon` (unit must be down: the proof binds :7770
+   at container CREATE).
+3. Scratch env file on the host using the bench-sanctioned seam:
+   ```
+   /tmp/dragoon-nputest.env:
+     source /home/dank/serverAppConfigs/scripts/frank-dragoon.env
+     HALOGEN_NPU_MODELS=qwen3-embedding-0.6b
+   ```
+4. Scratch serve copy that appends the candidate mount to the `NPU_ARGS+=(--device`
+   line (`sed`), REAL files only (`readlink -f`; vendor forbids dangling symlinks
+   under /opt/xilinx), and run with:
+   `FRANK_DRAGOON_ENV_FILE=/tmp/... FRANK_DRAGOON_LOG=/tmp/... timeout 1200 /tmp/serve-nputest.sh`
+5. Success = boot ledger (`… in all`) **plus** `npu_api: <model> … buckets u0…u9`
+   plus `/health` `npu.status ok`. Then codify the mount in the serve script, pin
+   it in `tests/frank_dragoon_unit.bats` (the fixture XRT dir's basename IS the
+   multiarch name — name it `x86_64-linux-gnu`), update the endpoint page's NPU
+   prerequisite to the three-prefix law, and cut over via PR +
+   `deploy.sh dankclaw --service frank-dragoon`.
+6. Retire scratch: the `timeout` kills the proof container; `podman rm -f`,
+   move /tmp scripts to `~/.AGENTTRASH/<date>-<topic>/`. A proof instance
+   serving production while you wait is drift — deploy the merged fix promptly.
+
+## Traps
+- Greping image contents rootless: `podman mount` fails (needs `podman unshare`)
+  and leaves `$M` EMPTY, so `grep -r $M/opt` silently greps the HOST cwd — that
+  is how host `/opt/fastflowlm` masqueraded as image content. Always check `$M`
+  non-empty, or use `podman unshare bash -c 'M=$(podman mount C); …'`.
+- The entrypoint swallows `podman run IMG sh -c …` as a mode name; use
+  `--entrypoint sh/cat/ls`.
+- The proof container inherits production name/port unless the override is set
+  AFTER sourcing the env (the env file re-exports `FRANK_DRAGOON_CONTAINER`).
+- `Restart=always` re-runs the lane into its own GPU/NPU debris; stop the unit
+  before any manual boot, don't fight the loop.